Sensu is an open source monitoring service for the cloud by Sonian the diagram below explains most of how it works quite well, but it has several components in order to make it scalable:
RabbitMQ is really the central “server” in terms of where checks come from, and where the results go. Here’s the flow of a rabbitmq check.
- Server gets a new check that the client needs to execute. It puts that into rabbitmq.
- Client checks rabbitmq for any checks to execute, it sees a check it should perform, so it gets the data from rabbitmq, and executes the command.
- Client takes the results of the command, and puts that into rabbitmq
- Server checks for command results, sees that a client put in a result. It posts it to the dashboard.
- Server sees a metric, as directed by the handler, it puts it into the rabbitmq for Graphite’s carbon service.
- Carbon takes that data from rabbitmq and puts it into graph.
Built for the cloud. Sensu is made to have clients just magically appear. There is no individual client specification in the config files in Sensu. Likewise, Sensu has a REST based API where clients can be just as easily removed.
Scalable. Since the central service uses rabbitmq, which itself is quite scalable and can be run HA if necessary, It also has discrete components which can all also be made redundant.
Integration with Graphite. Sensu checks by design, integrate with graphing engines like graphite. Also no clients need to be registered in graphite for them to appear.
Sensu server has 4 chief server components:
The server initiates checks on clients, receives the output of the checks feeds them to handlers. (As of version 0.9.2, clients can also execute checks that the server doesn’t know about and the server will still process their results, more on these ‘standalone checks’ in a future article.)
Sensu-server relies on a Redis instance to keep persistent data. It also relies heavily (as do most sensu components) on access to rabbitmq for passing data between itself and sensu-client nodes.
A REST API that provides access to various pieces of data maintained on the sensu-server in Redis. You will typically run this on the same server as your sensu-server or Redis instance. It is mostly used by internal sensu components at this time.
A minimal dashboard providing an overview of the current state of your Sensu infrastructure and the ability to perform actions, such as temporarily silencing alerts.
A better web dashboard providing an overview of the current state of your Sensu infrastructure and the ability to perform actions, such as temporarily silencing alerts.
- There are two types of plugins that checks run: metrics (handled by graphite), and checks (handled by sensu).
- Clients get groups of checks called subscriptions. Subscriptions are defined on the server. Clients can get multiple subscriptions.
- Clients must have the plugin that they are running locally. The plugin must by executable.
- Clients must be able to reach the rabbitmq server defined in /etc/sensu/config.json
- Client’s definitions are in /etc/conf.d/client.json. Any *json files in this directory will be parsed.
- Client definitions are ruled by chef.
“subscriptions”: [ “linux” ]
When adding server checks you’ll need to restart the server. At times this can be painful and you need to kill -9 the process.
Fast facts about checks:
- Server Check Configurations: netmon1.pointinside.com:/etc/sensu/conf.d
- Check Format: JSON
Sample metric check (JSON):
“command”: “/etc/sensu/plugins/vmstat-metrics.rb –scheme stats.:::name:::”,
“subscribers”: [ “linux” ]
Sample alert check:
“command”: “/etc/sensu/plugins/check-mem.sh -w 10 -c 5”,
“subscribers”: [ “linux” ]
The handler defines what happens with the check output. There are many custom handlers out there. Of course the standard is an e-mail handler:
“command”: “mail -s ‘sensu alert’ firstname.lastname@example.org”