The Guru College

Speaking At SurgeCon 2016

Earlier this year I submitted a talk proposal to SurgeCon, based on the work I have been doing scaling the log searching infrastructure at work. I was notified recently that my proposal has been accepted, and I’m going to be speaking!

It’s been a long an interesting road going from 30,000 to 200,000+ logs per second, as well as replacing or upgrading almost every piece of the logging infrastructure from pipelines to data storage to presentation. There were some nasty bumps along the way, as when the index of logs is measured in the hundreds of billions scaling limitations come in to play quickly - and we’re on track to be at or near a trillion logs in the search indexes in the coming months.

Registration is open at the moment, but the early bird pricing goes away soon, so if you are interested, sign up!

I'm Never Sleeping Again

Life Begins At 8 Bar

In the past few months, my morning routine has changed. Instead of firing up the kettle and grinding out ~25 grams of coarse coffee, I’m find myself praying that my son has turned the rocker switch so I can begin the ritual of the manual espresso machine.

My wonderful wife found, quite by accident, a La Pavoni Europiccola Millenium in almost mint condition for $20. These machines are quite picky about almost everything - the grind of the bean, distribution in the basket, the humidity of the air, the level of water in the boiler, and then of course the temperature of the boiler, grouphead, portafilter and cup. This is before we even get to steaming milk – and I’ve skipped several important considerations that I won’t bore you with now.


I’ve learned more in the last few months about the picky little details of espresso making than I ever thought I’d know. I purchased a used espresso grinder for 10 times what we paid for the Pavoni, and it would still have been a deal at 2x the price. We’re into gear (cups, steaming pitchers, tampers, you name it) up to our elbows, and we’re almost where we need to be. Drinks are consistently good, at times rising to the level of actual excellence.

I can’t lie and say we’re saving any money compared to our hipster pour over days. Before this I was drinking my coffee black, and the quality of the bean was far less important. Now, we run through a little over 3 gallons of milk and 2 pounds of coffee a week, considering our needs and the needs of our guests. But my latte’s cost about a dollar ten a cup in consumables, and even when amortizing the cost of the hardware, we’re looking at $1.50 a cup if we stopped tomorrow.

And there’s something that’s just delightful about watching the syrupy caffeinated magic flow into a waiting cup.

Job Transitions and Loadbalancers

The most recent Practical Operations Podcast episodes are about job transitions and load balancers, both things near and dear to our hearts. Give a listen, let me know what you think! We’d like to know what we should cover better - so topic ideas are always welcome - and what we’ve covered poorly, so comments are encouraged.

XML feed

With the move from Wordpress to Hugo, the RSS feed for this site has changed to something more universally understood and common place: The old address of will still work for some time, but should go away soon.


Switching this site from WordPress to Hugo, a static site generator written in Go. This is the first post written entirely in hugo and not imported from wordpress.

The benefits of using a static site generator include not needing a database or complicated caching, as static files can be cached very effectively, and there are no round trips to a database for content. It also removes the two biggest security issues with WordPress - attacks on the database and on PHP functions on the pages. It is, however, more complicated to setup initially, and changing things means regenerating all the static pages in the site.

The Moment

I know the exact moment I decided to leave the job before last. I didn’t know it at the time. It took months to figure it out. But I happened. Reading an article on Rands In Repose made me remember how distinctly that moment stands out as the moment. I was sitting in a 3 hour meeting with the senior management, and I was told that I wasn’t part of the group that was making decisions. Even more so, my group was being explicitly excluded from that process. There was another team that was driving decisions, and we were simply there to implement and support them.

In reflection, the next job I left was for a similar reason. At the time, I was happy with the work, my boss and my coworkers, but when an old colleague asked if I was interested in something bigger… I responded that I was interested. It wasn’t until I was sitting in the interview itself with my new employer that I realized what it was about my current job that I was unhappy with – again, I was not being consulted on architecture and other forward-looking aspects of the stack. The work we were doing was fascinating and full of technology I was delighted to be learning more about. However, there were lots of legacy bits we held onto for Reasons, and the folks in charge of leading the platform had no context on how to production-ize the code they wrote.

These are both jobs I loved. I learned a lot at them, I worked with amazing people, and I did what I felt was important work – not just helping post cat pictures to the internet. In the end, not having a sense of ownership of the stack can be a very discouraging thing to deal with – just as bad in many ways as having abusive coworkers, being underpaid or being bored.

The Practical Operations Podcast

Myself, Jack Neely and Jarod Watkins have started a podcast about system operations and engineering topics, called the Practical Operations Podcast. It’s a weekly show where the three of us discuss pragmatic and practical topics in the field of operations. With the Thanksgiving holiday we were a little delayed releasing the second episode about the best approaches to get monitoring and alerting under control, and we’ve already recorded episode 3.

We are currently trying to do a weekly show, and we are trying to keep it to about 30 minutes per show.

If you have questions or comments about the show, or would like to ask us to cover specific topics, please let us know via twitter @operationsfm or

Home | Older Posts