The Guru College

The Practical Operations Podcast

Myself, Jack Neely and Jarod Watkins have started a podcast about system operations and engineering topics, called the Practical Operations Podcast. It’s a weekly show where the three of us discuss pragmatic and practical topics in the field of operations. With the Thanksgiving holiday we were a little delayed releasing the second episode about the best approaches to get monitoring and alerting under control, and we’ve already recorded episode 3.

We are currently trying to do a weekly show, and we are trying to keep it to about 30 minutes per show.

If you have questions or comments about the show, or would like to ask us to cover specific topics, please let us know via twitter @operationsfm or [email protected]

ttytter is dead, long live oysttyer!

My favorite command-line twitter client is dead. It’s been replaced by the open source oysttyer, as the original author lost interest in twitter as a platform and decided to let the community run with it.

iCloud Photo Library

An exciting tale about what happens when you max out your asymmetric upload.

A few weeks ago I decided to enable iCloud Photo Library and start using Photos for OS X. In the past, I’ve had a patchy history with Apple’s cloud services, especially the ones that shuffle photos from your device to your “real” computer and vice versa. After enabling the iCloud Photo Library on my phone and desktop, my internet connection crawled to a halt. I was uploading photos to Apple at a good clip, but nothing else worked. In the entire house. We couldn’t stream Netflix, couldn’t load reddit and couldn’t use FaceTime while on WiFi. What had happened: due to the asymmetrical nature of most residential internet connections, the upload connection was saturated with photo uploads. This prevented any other inbound connection from ack’ing traffic to it’s source, which in plain terms meant nothing else worked.

Luckily, I run a decent router, so I was able to put traffic limiting in place, and put in rules that no host could use more than 3mbps of the 5.5mbps we get from our provider. This kept part of the upstream open, and life went back to normal. Until last night, when I turned on iCloud Photo Library for my wife. And then imported a large chunk of photos from the DSLR on my computer. Each computer happily started using 3mbps of the connection, and all other traffic became unreasonably slow – bordering on failure conditions again.

As I love data, here’s the graph of my connection, and it’s pretty clear when I started my DSLR import/upload and when I updated the traffic limiter:


Inside Photos for OS X, the only control you have is “Disable uploads for 24 hours”. Which is another way of saying “Please wait until this time tomorrow to destroy my connection once again.” I like iCloud Photo Library and Photos for OS X… but Apple needs to address this. A simple internal rate limiter, like the ones used by every other cloud sync or cloud backup provider would be sufficient.

A Great Week For the NSA

It’s been a great week for the NSA. First, we get the news that they are (effectively) behind the hacker collective known as the Equation Group (which does insane things, like deploy malware into the firmware of hard drives, so it survives drive formatting). Now we learn that they’ve essentially pwned all cell phone SIM cards.

The bottom line is that people around the world, regardless of their nationality, should know that the United States is not spying on ordinary people who don’t threaten our national security and that we take their privacy concerns into account in our policies and procedures.

Barack Obama, Jan 17, 2014

Between this and the fact that the private key for the Lenovo adware/malware was cracked in 3 hours by a single man…

Superfish, or how to MITM everyone

Lenovo was just added to the list of companies I’m hesitant to ever buy anything from ever again, in any capacity. As Ars Technica reports, the Superfish adware that was installed by default on Lenovo machines presents a self-signed root CA certificate in the Trusted Roots for the system’s SSL keys. This certificate was also trivially cracked. This means, if you are running Windows as shipped on a Lenovo machine, you may well be subject to insane security breaches.

Best bet: backup your personal data, return the Lenovo, and get a new laptop from a different vendor. And when you get the new machine, wipe the drive and reinstall your OS of choice from a vendor-supplied DVD. Only then should you put your data back on the machine.

Oh, yeah, and never trust OEM supplied OS images, ever again.


New Year News: I will be posting more here and on GitHub in 2015. I have taken a new job, due to start in late January, which encourages open source project work. I will be working for a company called 42 Lines, doing systems administration and Operations work.


FreeNAS is becoming more and more unusable for me. Part of the reason for using it was access to the jails and plugins – notably CrashPlan and OwnCloud. Due to the moving target nature of the system, I have had to repeatedly login and restart processes, manage memory, kill zombies, and deal with other issues in the jail setup. Further, the performance of AFP in FreeNAS (for me) has not been something to write home about, even with the newer bits (3.1.x branch, last I looked). The last straw is that the ZPOOL version is 5000 with a number of feature flags that nobody else has implemented yet. This means that the pools on my system are unreadable without the very newest releases of FreeNAS or FreeBSD.

I’m backing up now, headed for the better understood lands of ZFS on Linux. It looks stable enough to keep me afloat, and I know how to keep CrashPlan from spawning 65000 zombies in the background.

Newer Posts | Home | Older Posts