The Moment

I know the exact moment I decided to leave the job before last. I didn’t know it at the time. It took months to figure it out. But I happened. Reading this article by Rands In Repose made me remember how distinctly that moment stands out as the moment. I was sitting in a 3 hour meeting with the senior management, and I was told that I wasn’t part of the group that was making decisions. Even more so, my group was being explicitly excluded from that process. There was another team that was driving decisions, and we were simply there to implement and support them.

In reflection, the next job I left was for a similar reason. At the time, I was happy with the work, my boss and my coworkers, but when an old colleague asked if I was interested in something bigger… I responded that I was interested. It wasn’t until I was sitting in the interview itself with my new employer that I realized what it was about my current job that I was unhappy with – again, I was not being consulted on architecture and other forward-looking aspects of the stack. The work we were doing was fascinating and full of technology I was delighted to be learning more about. However, there were lots of legacy bits we held onto for Reasons, and the folks in charge of leading the platform had no context on how to production-ize the code they wrote.

These are both jobs I loved. I learned a lot at them, I worked with amazing people, and I did what I felt was important work – not just helping post cat pictures to the internet. In the end, not having a sense of ownership of the stack can be a very discouraging thing to deal with – just as bad in many ways as having abusive coworkers, being underpaid or being bored.

http://randsinrepose.com/archives/shields-down/

The Practical Operations Podcast

Myself, Jack Neely and Jarod Watkins have started a podcast about system operations and engineering topics, called the Practical Operations Podcast. It’s a weekly show where the three of us discuss pragmatic and practical topics in the field of operations. With the Thanksgiving holiday we were a little delayed releasing the second episode about the best approaches to get monitoring and alerting under control, and we’ve already recorded episode 3.

We are currently trying to do a weekly show, and we are trying to keep it to about 30 minutes per show.

If you have questions or comments about the show, or would like to ask us to cover specific topics, please let us know via twitter @operationsfm or [email protected]

iCloud Photo Library

An exciting tale about what happens when you max out your asymmetric upload.

A few weeks ago I decided to enable iCloud Photo Library and start using Photos for OS X. In the past, I’ve had a patchy history with Apple’s cloud services, especially the ones that shuffle photos from your device to your “real” computer and vice versa. After enabling the iCloud Photo Library on my phone and desktop, my internet connection crawled to a halt. I was uploading photos to Apple at a good clip, but nothing else worked. In the entire house. We couldn’t stream Netflix, couldn’t load reddit and couldn’t use FaceTime while on WiFi. What had happened: due to the asymmetrical nature of most residential internet connections, the upload connection was saturated with photo uploads. This prevented any other inbound connection from ack’ing traffic to it’s source, which in plain terms meant nothing else worked.

Luckily, I run a decent router, so I was able to put traffic limiting in place, and put in rules that no host could use more than 3mbps of the 5.5mbps we get from our provider. This kept part of the upstream open, and life went back to normal. Until last night, when I turned on iCloud Photo Library for my wife. And then imported a large chunk of photos from the DSLR on my computer. Each computer happily started using 3mbps of the connection, and all other traffic became unreasonably slow – bordering on failure conditions again.

As I love data, here’s the graph of my connection, and it’s pretty clear when I started my DSLR import/upload and when I updated the traffic limiter:

bandwidth-graph

Inside Photos for OS X, the only control you have is “Disable uploads for 24 hours”. Which is another way of saying “Please wait until this time tomorrow to destroy my connection once again.” I like iCloud Photo Library and Photos for OS X… but Apple needs to address this. A simple internal rate limiter, like the ones used by every other cloud sync or cloud backup provider would be sufficient.

A Great Week For the NSA

It’s been a great week for the NSA. First, we get the news that they are (effectively) behind the hacker collective known as the Equation Group (which does insane things, like deploy malware into the firmware of hard drives, so it survives drive formatting). Now we learn that they’ve essentially pwned all cell phone SIM cards.

“The bottom line is that people around the world, regardless of their nationality, should know that the United States is not spying on ordinary people who don’t threaten our national security and that we take their privacy concerns into account in our policies and procedures” — Barack Obama, Jan 17, 2014

Between this and the fact that the private key for the Lenovo adware/malware was cracked in 3 hours by a single man…