The Guru College

Job Transitions and Loadbalancers

The most recent Practical Operations Podcast episodes are about job transitions and load balancers, both things near and dear to our hearts. Give a listen, let me know what you think! We’d like to know what we should cover better - so topic ideas are always welcome - and what we’ve covered poorly, so comments are encouraged.

XML feed

With the move from Wordpress to Hugo, the RSS feed for this site has changed to something more universally understood and common place: The old address of will still work for some time, but should go away soon.


Switching this site from WordPress to Hugo, a static site generator written in Go. This is the first post written entirely in hugo and not imported from wordpress.

The benefits of using a static site generator include not needing a database or complicated caching, as static files can be cached very effectively, and there are no round trips to a database for content. It also removes the two biggest security issues with WordPress - attacks on the database and on PHP functions on the pages. It is, however, more complicated to setup initially, and changing things means regenerating all the static pages in the site.

The Moment

I know the exact moment I decided to leave the job before last. I didn’t know it at the time. It took months to figure it out. But I happened. Reading an article on Rands In Repose made me remember how distinctly that moment stands out as the moment. I was sitting in a 3 hour meeting with the senior management, and I was told that I wasn’t part of the group that was making decisions. Even more so, my group was being explicitly excluded from that process. There was another team that was driving decisions, and we were simply there to implement and support them.

In reflection, the next job I left was for a similar reason. At the time, I was happy with the work, my boss and my coworkers, but when an old colleague asked if I was interested in something bigger… I responded that I was interested. It wasn’t until I was sitting in the interview itself with my new employer that I realized what it was about my current job that I was unhappy with – again, I was not being consulted on architecture and other forward-looking aspects of the stack. The work we were doing was fascinating and full of technology I was delighted to be learning more about. However, there were lots of legacy bits we held onto for Reasons, and the folks in charge of leading the platform had no context on how to production-ize the code they wrote.

These are both jobs I loved. I learned a lot at them, I worked with amazing people, and I did what I felt was important work – not just helping post cat pictures to the internet. In the end, not having a sense of ownership of the stack can be a very discouraging thing to deal with – just as bad in many ways as having abusive coworkers, being underpaid or being bored.

The Practical Operations Podcast

Myself, Jack Neely and Jarod Watkins have started a podcast about system operations and engineering topics, called the Practical Operations Podcast. It’s a weekly show where the three of us discuss pragmatic and practical topics in the field of operations. With the Thanksgiving holiday we were a little delayed releasing the second episode about the best approaches to get monitoring and alerting under control, and we’ve already recorded episode 3.

We are currently trying to do a weekly show, and we are trying to keep it to about 30 minutes per show.

If you have questions or comments about the show, or would like to ask us to cover specific topics, please let us know via twitter @operationsfm or [email protected]

ttytter is dead, long live oysttyer!

My favorite command-line twitter client is dead. It’s been replaced by the open source oysttyer, as the original author lost interest in twitter as a platform and decided to let the community run with it.

iCloud Photo Library

An exciting tale about what happens when you max out your asymmetric upload.

A few weeks ago I decided to enable iCloud Photo Library and start using Photos for OS X. In the past, I’ve had a patchy history with Apple’s cloud services, especially the ones that shuffle photos from your device to your “real” computer and vice versa. After enabling the iCloud Photo Library on my phone and desktop, my internet connection crawled to a halt. I was uploading photos to Apple at a good clip, but nothing else worked. In the entire house. We couldn’t stream Netflix, couldn’t load reddit and couldn’t use FaceTime while on WiFi. What had happened: due to the asymmetrical nature of most residential internet connections, the upload connection was saturated with photo uploads. This prevented any other inbound connection from ack’ing traffic to it’s source, which in plain terms meant nothing else worked.

Luckily, I run a decent router, so I was able to put traffic limiting in place, and put in rules that no host could use more than 3mbps of the 5.5mbps we get from our provider. This kept part of the upstream open, and life went back to normal. Until last night, when I turned on iCloud Photo Library for my wife. And then imported a large chunk of photos from the DSLR on my computer. Each computer happily started using 3mbps of the connection, and all other traffic became unreasonably slow – bordering on failure conditions again.

As I love data, here’s the graph of my connection, and it’s pretty clear when I started my DSLR import/upload and when I updated the traffic limiter:


Inside Photos for OS X, the only control you have is “Disable uploads for 24 hours”. Which is another way of saying “Please wait until this time tomorrow to destroy my connection once again.” I like iCloud Photo Library and Photos for OS X… but Apple needs to address this. A simple internal rate limiter, like the ones used by every other cloud sync or cloud backup provider would be sufficient.

Home | Older Posts