xinetd is the super-daemon in linux that spawns other processes as a result of network requests. The most well know examples are ftp and ssh. When a user connects on the correct port, xinetd grabs the connection, starts the ssh or ftp server, and connects the session to the proper program. It is much more efficient than having every process watch for connections and spawn off new threads to handle the connections. xinetd is well tested and well tuned. Most of the performance issues have been ironed out over the years.

Which left me very confused when xinetd stopped handing connections for the Nagios Service Check Acceptor from time to time. Load on the system seemed fine, free memory seemed fine, and there were no entires in the log that suggested a problem. I was at my wits end, having just raised the limit for simultaneous connections to 1000. And I was still having problems.

It wasn’t that xinetd would stop entirely – it would still accept connections, but on a reduced rate. Finally, our resident Linux Guru pointed out that I could use strace on some of the hung nsca connections to see what they were doing. Turns out, they were waiting to write to Nagios’s named fifo pipe, a file that is destroyed and recreated every time Nagios quits to reload configuration data. Every time Nagios shut down and deletes this file, there is a chance that nsca processes would hang trying to write to the file. With between 500 and 600 checks coming in every minute, it was happening pretty frequently.

It’s now been several days without issues – adding a /sbin/service xinetd stop and start to my configuration generation script has fixed the problem.