xinetd is the super-daemon in linux that spawns other processes as a result of network requests. The most well know examples are ftp and ssh. When a user connects on the correct port, xinetd grabs the connection, starts the ssh or ftp server, and connects the session to the proper program. It is much more efficient than having every process watch for connections and spawn off new threads to handle the connections. xinetd is well tested and well tuned. Most of the performance issues have been ironed out over the years.

Which left me very confused when xinetd stopped handing connections for the Nagios Service Check Acceptor from time to time. Load on the system seemed fine, free memory seemed fine, and there were no entires in the log that suggested a problem. I was at my wits end, having just raised the limit for simultaneous connections to 1000. And I was still having problems.

It wasn’t that xinetd would stop entirely – it would still accept connections, but on a reduced rate. Finally, our resident Linux Guru pointed out that I could use strace on some of the hung nsca connections to see what they were doing. Turns out, they were waiting to write to Nagios’s named fifo pipe, a file that is destroyed and recreated every time Nagios quits to reload configuration data. Every time Nagios shut down and deletes this file, there is a chance that nsca processes would hang trying to write to the file. With between 500 and 600 checks coming in every minute, it was happening pretty frequently.

It’s now been several days without issues – adding a /sbin/service xinetd stop and start to my configuration generation script has fixed the problem.

My current project is to rebuild the monitoring and notification system for the University. The old system was built on Nagios 1.4, which has served it’s purpose very well since it was setup, but is starting to get long in the tooth, and the previous owner of the project left the employ of the University in 2007. I have spent the last few months learning his code and getting a new back end ready for the system. One of the things that had been postponing the move away from Nagios 1.4 was the removal of the database backend from the core of the Nagios system. The web interface provided by Nagios has always stunk, so we had been running our own set of front end tools for managing the configuration files and keeping an eye on the status of the servers monitored. Thus, the database is critical to our monitoring needs.

There is a new-ish database backend for Nagios, called NDOUtils (Nagios Data Output Utility) that runs a daemon that dumps all the Nagios event data to a MySQL database. The problem is that NDO was never designed for scale, as best I can tell. The first major sin it commits is not generating any SQL indexes by default when you install the database. This is a Very Bad Thing, and leads to horrible database performance as the data builds up in the tables. There is an optional schema file you can process and run to get indexes, but the installer doesn’t apply them for you. The other problem is that all event data is logged to the database. This includes state changes, notification messages, performance data – everything. Admittedly, this isn’t a bug, it’s a feature. There is even a handy pruning feature that will clean up entries older than a given age. This pruning age is set to a week for most of the event types.

The best post I could find about tuning NDOUtils talks about having 400 hosts and 3000 service checks. It’s a very good article, but I’m trying to solve a different problem. We have 1500 hosts, and over 4000 service checks, and NSCA isn’t overloaded (although xinetd is, but that’s a whole other story). Not all checks are run every minute, or even every 5 minutes, so I wrote a small script to troll through the nagios log looking for external events submitted by the servers performing the actual checks. My baseline is around 600 checks per minute coming into our front end Nagios server. Let’s do the math – 600 checks * 10,080 minutes in a week = 6 million possible events in the log. This leads us to the second major sin that is being committed by NDO – it frequently calls a DELETE statement to clear any data older than the specified time. In SQL, the DELETE command is one of the most expensive commands you can run, in terms of CPU usage. It’s made even worse when there is no index on the table you are deleting from. It’s made into a 3 Act Greek tragedy when every 5 minutes it tries to trim the last 3,000 entries from a table with 6 million rows. It starts to approach EPIC FAIL when you consider that we are going to be roughly doubling the number of servers and checks in the next few weeks as we bring the Oracle/HR environment under our monitoring, and maybe double again if we absorb network monitoring.

I of course discover all of this a few hours after I had sent the demo page for the new system out to the various teams of system administrators to look at, I knew I needed to fix the database performance problem, and quickly. I knew that it would be intensive to clear the bad data, but I didn’t have any real idea of how bad it would be. I turned the prune timer down from 10080 minutes to 60 (just keeping an hour’s worth of data), and restarted the database connector.

75 minutes later, it finished cleaning up the first of 6 multi-million entry tables. I waited till 5PM, stopped MySQL, dropped the database, and reloaded the schema (and the optional indexes), and brought everything back up. It took a few minutes for enough data to get back into the database to get accurate states to show up again, but it was far less than the 8+ hours it would have taken to prune away the old data. I shudder to think what would have happened had I loaded all the other devices, and was trying to process 6 tables with 20 million rows each.

Mr. Murdoch is thinking people will pay $2/week to view WSJ content on their iPhones and Blackberries. Why would they, though, when they could pay $2.29/week to get the damn thing delivered, to their door, every day.

Personally, if I cared about the content and the quality of their articles, I’d shuck out the extra $1.50 per month to get a copy delivered to my door every morning that I can read without having to poke and zoom with my fingers. Don’t get me wrong – I love media delivery via the iPhone – but if I’m going to read a lot of text, I don’t want to stare at something the size of a pack of playing cards in my hand. It’s handy when you’re on the go, but I don’t know anyone who would prefer reading text off an iPhone to reading the equivalent material on a full size computer monitor.

I finally installed Mac OS X 10.6 (Snow Leopard) on my primary desktop. On launching my trusty process watcher ‘top‘ for the first time, I noticed it has a lot more columns of output than it used to. The best part – resizing the window gave me even more output. Even wilder – I use transparent terminal windows, so I can keep an eye on what else is going on with my computer. Now, the png files generated by shift-command-4 preserve the transparency.

I’m getting close to another breaking point. I now have upwards of 40,000 RAW files in my Aperture library, which is close to 400GB in disk usage for my library, which lives on a 750 GB SATA drive. Each time I go to update the Aperture Vault that is associated with the library, it takes longer and longer to run. Part of that is due to the fact that I’m backing up to a network mounted sparseimage, and it has to trawl through tens of thousands of files and hundreds of GB of data. Sadly, the 750 GB drive is filling up, though, as I keep my iTunes library and a full set of my home folder backups on that drive as well. Something’s gotta give.

The best solution that keeps my current workflow intact is to get a pair of 1TB or 1.5 TB SATA drives and put them in the two unused drive bays of my MacPro. I would use one of them for the Aperture library, and the other for the Aperture Vault. This unchains me from using the aforementioned spareseimage, as well as freeing up a 750 GB SATA drive to move into one of my ZFS file servers. However, the purchase of two drives right now is unlikely at best.

All of the other options I have involve not storing all of the images in Aperture. This would be especially true for the rapid-fire sequence shots, when I manage to get a gig of pictures of Qais eating avocado during a single meal. With some careful filtering, I could probably move %70 of my Aperture library to individual files on my networked file servers, which have room to spare at this point. The trouble then is access. How do I easily browse the images? Find all images taken in 2008 that I tagged as ‘Oman’? This setup would require a that I setup a networked media server application on the file server to sort, search and deliver the images. It would have to be able to manage NEF files from a D60, a D70, a D90 and a D200 (the four cameras %99 of the raw files are from). Ideally, it would be web-based, so anyone at all could at least look at the images. And of course, it needs to be free – not necessarily ‘free as in freedom’, but ‘free as in beer’. If I could afford the hundreds of dollars media management programs cost, I could just as easily get the hard drives in the ideal solution above.

I have yet to find a free one. (If I had found one, would I have written this long a post about it? I think not.) Really, what I want jinzora, but for photos, not music and videos. Coppermine looks nice and everything, but I loaded it up last night and it won’t eat my NEF’s – and it’s dog slow to import 500KB JPG’s. I wonder how many weeks it would take to ingest my photo library exported at 16 bit TIFFs?

So, suggestions? If I find something, I’ll post about it. If you find something, and tell me about it, I’ll post about that too, and say really nice things about you.

Recently the FBI rounded up the heads of the largest music copyright infringement ring in the world and taken them into custody. They are known to have distributed over 25,000 albums, most of them before their release to the public. Each member is being charged with a single count of ‘conspiracy to commit copyright infringement’ that carries a maximum penalty of $250,000 USD fine and up to 5 years in jail. That’s $10 per album – the price you’d pay on the iTunes Music Store – or about $1/song, assuming a typical album has 10 songs. Now for the fun part. If they had been popped by RIAA and MediaSentry, and had been found guilty on the sliding scale that is civil copyright law, they would have been paying $22,500 USD per song. That’s $5.6 billion USD in damages.

The lesson learned here? It’s better to run a criminal conspiracy to with intent to commit copyright fraud than it is to use Kazaa or Limewire. If you get caught, the punishment is orders of magnitude less. Please, Recording Industry Association of America, die in a fire.

Bill of materials:

• 1 quarter inch ‘2 ¹⁄₂’ bolt.
• 1 matching nut
• string

Tie string to bolt. Thread nut and tighten. Thread bolt into bottom of camera. Stand on string. Lift camera until string is tight. Stabilization! …almost. Sadly, it only buys me about ¹⁄₂ a stop of shutter speed, which I can get out of being lucky and running on high speed capture. Time to work on that DIY ringlight, now that I have the support bolts purchased.