Yesterday was very frustrating at work. I’ve finally gotten my proof-of-concept Nagios/Merlin install up to the point where I can turn on real service checks and test with a small subset of the production monitoring load. While doing this, I went back to check one of the last things on the “just make sure it works list”, which was notification handling. In the current setup, there are a pair of front-end nodes that handle the reporting, graphing and alerting functions, while a separate set of servers do the bulk of the checks. The check results are passed back via NSCA, and Everything Just Works.

The trouble is that in the Nagios/Merlin world, the checks are not passed back and forth. A state database and a NEB make sure every node knows the state of all services, but the actual notification is done on the host that runs the checks. This means the SMS gateways installed on the front end boxes won’t see most of the notifications destined for them, until some glue is written. And the whole point of using Merlin was to get away from these edge cases.

It also means I need to go back and look at perfdata processing. I’d assumed it was done everywhere as the perfdata is also shared, but looks like I can’t assume that either.