The Guru College Recent content on The Guru College Hugo -- en-us Sat, 16 Jul 2016 12:59:55 -0400 Speaking At SurgeCon 2016 Sat, 16 Jul 2016 12:59:55 -0400 <p>Earlier this year I submitted a talk proposal to <label for="mn-example" class="margin-toggle">&#8853;</label> <input type="checkbox" id="mn-example" class="margin-toggle"/> <span class="marginnote"><a href=""></a></span> SurgeCon, based on the work I have been doing scaling the log searching infrastructure at work. I was notified recently that my proposal has been accepted, and I&rsquo;m going to be <label for="mn-example" class="margin-toggle">&#8853;</label> <input type="checkbox" id="mn-example" class="margin-toggle"/> <span class="marginnote"><a href=""></a></span> speaking!</p> <p>It&rsquo;s been a long an interesting road going from 30,000 to 200,000+ logs per second, as well as replacing or upgrading almost every piece of the logging infrastructure from pipelines to data storage to presentation. There were some nasty bumps along the way, as when the index of logs is measured in the hundreds of billions scaling limitations come in to play quickly - and we&rsquo;re on track to be at or near a trillion logs in the search indexes in the coming months.</p> <p>Registration is open at the moment, but the early bird pricing goes away soon, so if you are interested, sign up!</p> I'm Never Sleeping Again Tue, 12 Jul 2016 22:00:00 -0400 <p>In the past few months, my morning routine has changed. Instead of firing up the kettle and grinding out ~25 grams of coarse coffee, I&rsquo;m find myself praying that my son has turned the rocker switch so I can begin the ritual of the manual espresso machine.</p> <p>My wonderful wife found, quite by accident, a La Pavoni Europiccola Millenium in almost mint condition for $20. These machines are quite picky about almost everything - the grind of the bean, distribution in the basket, the humidity of the air, the level of water in the boiler, and then of course the temperature of the boiler, grouphead, portafilter and cup. This is before we even get to steaming milk &ndash; and I&rsquo;ve skipped several important considerations that I won&rsquo;t bore you with now.</p> <p><a data-flickr-embed="true" href="" title="DSC_7033.jpg"><img src="" width="2048" height="1368" alt="DSC_7033.jpg"></a><script async src="//" charset="utf-8"></script></p> <p>I&rsquo;ve learned more in the last few months about the picky little details of espresso making than I ever thought I&rsquo;d know. I purchased a used espresso grinder for 10 times what we paid for the Pavoni, and it would still have been a deal at 2x the price. We&rsquo;re into gear (cups, steaming pitchers, tampers, you name it) up to our elbows, and we&rsquo;re <em>almost</em> where we need to be. Drinks are consistently good, at times rising to the level of actual excellence.</p> <p>I can&rsquo;t lie and say we&rsquo;re saving any money compared to our hipster pour over days. Before this I was drinking my coffee black, and the quality of the bean was far less important. Now, we run through a little over 3 gallons of milk and 2 pounds of coffee a week, considering our needs and the needs of our guests. But my latte&rsquo;s cost about a dollar ten a cup in consumables, and even when amortizing the cost of the hardware, we&rsquo;re looking at $1.50 a cup if we stopped tomorrow.</p> <p>And there&rsquo;s something that&rsquo;s just delightful about watching the syrupy caffeinated magic flow into a waiting cup.</p> Job Transitions and Loadbalancers Wed, 23 Mar 2016 20:27:04 -0400 <p>The most recent Practical Operations Podcast episodes are about <label for="mn-example" class="margin-toggle">&#8853;</label> <input type="checkbox" id="mn-example" class="margin-toggle"/> <span class="marginnote"><a href=""></a></span> job transitions and load balancers,<label for="mn-example" class="margin-toggle">&#8853;</label> <input type="checkbox" id="mn-example" class="margin-toggle"/> <span class="marginnote"><a href=""></a></span> both things near and dear to our hearts. Give a listen, let me know what you think! We&rsquo;d like to know what we should cover better - so topic ideas are always welcome - and what we&rsquo;ve covered poorly, so comments are encouraged.</p> XML feed Tue, 22 Mar 2016 15:59:04 -0400 <p>With the move from Wordpress to Hugo, the RSS feed for this site has changed to something more universally understood and common place: <a href=""></a> The old address of <a href=""></a> will still work for some time, but should go away soon.</p> Hugo Tue, 22 Mar 2016 14:44:27 -0400 <p>Switching this site from WordPress to Hugo,<label for="mn-example" class="margin-toggle">&#8853;</label> <input type="checkbox" id="mn-example" class="margin-toggle"/> <span class="marginnote"><a href=""></a></span> a static site generator written in Go. This is the first post written entirely in hugo and not imported from wordpress.</p> <p>The benefits of using a static site generator include not needing a database or complicated caching, as static files can be cached very effectively, and there are no round trips to a database for content. It also removes the two biggest security issues with WordPress - attacks on the database and on PHP functions on the pages. It is, however, more complicated to setup initially, and changing things means regenerating all the static pages in the site.</p> The Moment Sat, 02 Jan 2016 00:00:00 +0000 <p>I know the exact moment I decided to leave the job before last. I didn&rsquo;t know it at the time. It took months to figure it out. But I happened. Reading an article on Rands In Repose <label for="mn-example" class="margin-toggle">&#8853;</label> <input type="checkbox" id="mn-example" class="margin-toggle"/> <span class="marginnote"><a href=""></a></span> made me remember how distinctly that moment stands out as <em>the</em> moment. I was sitting in a 3 hour meeting with the senior management, and I was told that I wasn&rsquo;t part of the group that was making decisions. Even more so, my group was being explicitly excluded from that process. There was another team that was driving decisions, and we were simply there to implement and support them.</p> <p>In reflection, the next job I left was for a similar reason. At the time, I was happy with the work, my boss and my coworkers, but when an old colleague asked if I was interested in something bigger&#8230; I responded that I was interested. It wasn&rsquo;t until I was sitting in the interview itself with my new employer that I realized what it was about my current job that I was unhappy with &#8211; again, I was not being consulted on architecture and other forward-looking aspects of the stack. The work we were doing was fascinating and full of technology I was delighted to be learning more about. However, there were lots of legacy bits we held onto for Reasons, and the folks in charge of leading the platform had no context on how to production-ize the code they wrote.</p> <p>These are both jobs I loved. I learned a lot at them, I worked with amazing people, and I did what I felt was important work &#8211; not just helping post cat pictures to the internet. In the end, not having a sense of ownership of the stack can be a very discouraging thing to deal with &#8211; just as bad in many ways as having abusive coworkers, being underpaid or being bored.</p> The Practical Operations Podcast Mon, 30 Nov 2015 00:00:00 +0000 <p>Myself, Jack Neely and Jarod Watkins have started a podcast about system operations and engineering topics, called the <a href="">Practical Operations Podcast</a>. It&rsquo;s a weekly show where the three of us discuss pragmatic and practical topics in the field of operations. With the Thanksgiving holiday we were a little delayed releasing <a href="">the second episode</a> about the best approaches to get monitoring and alerting under control, and we&rsquo;ve already recorded episode 3.</p> <p>We are currently trying to do a weekly show, and we are trying to keep it to about 30 minutes per show.</p> <p>If you have questions or comments about the show, or would like to ask us to cover specific topics, please let us know via twitter <a href="">@operationsfm</a> or <a href=""></a></p> ttytter is dead, long live oysttyer! Sun, 29 Nov 2015 00:00:00 +0000 <p>My favorite command-line twitter client is dead. It&rsquo;s been replaced by the open source <a href="">oysttyer</a>, as the original author lost interest in twitter as a platform and decided to let the community run with it.</p> iCloud Photo Library Sat, 27 Jun 2015 00:00:00 +0000 <p><strong>An exciting tale about what happens when you max out your asymmetric upload.</strong></p> <p>A few weeks ago I decided to enable iCloud Photo Library and start using Photos for OS X. In the past, I&rsquo;ve had a patchy history with Apple&rsquo;s cloud services, especially the ones that shuffle photos from your device to your &#8220;real&#8221; computer and vice versa. After enabling the iCloud Photo Library on my phone and desktop, my internet connection crawled to a halt. I was uploading photos to Apple at a good clip, but nothing else worked. In the entire house. We couldn&rsquo;t stream Netflix, couldn&rsquo;t load reddit and couldn&rsquo;t use FaceTime while on WiFi. What had happened: due to the asymmetrical nature of most residential internet connections, the upload connection was saturated with photo uploads. This prevented any other inbound connection from ack&rsquo;ing traffic to it&rsquo;s source, which in plain terms meant nothing else worked.</p> <p>Luckily, I run a decent router, so I was able to put traffic limiting in place, and put in rules that no host could use more than 3mbps of the 5.5mbps we get from our provider. This kept part of the upstream open, and life went back to normal. Until last night, when I turned on iCloud Photo Library for my wife. And then imported a large chunk of photos from the DSLR on my computer. Each computer happily started using 3mbps of the connection, and all other traffic became unreasonably slow &#8211; bordering on failure conditions again.</p> <p>As I love data, here&rsquo;s the graph of my connection, and it&rsquo;s pretty clear when I started my DSLR import/upload and when I updated the traffic limiter:</p> <p><a href=""><img src="" alt="bandwidth-graph" width="300" height="293" class="aligncenter size-medium wp-image-3294" /></a></p> <p>Inside Photos for OS X, the only control you have is &#8220;Disable uploads for 24 hours&#8221;. Which is another way of saying &#8220;Please wait until this time tomorrow to destroy my connection once again.&#8221; I like iCloud Photo Library and Photos for OS X&#8230; but Apple needs to address this. A simple internal rate limiter, like the ones used by every other cloud sync or cloud backup provider would be sufficient.</p> A Great Week For the NSA Fri, 20 Feb 2015 00:00:00 +0000 <p>It&rsquo;s been a great week for the NSA. First, we get the news that they are (effectively) behind the hacker collective known as the Equation Group (which does insane things, like deploy malware into the firmware of hard drives, so it survives drive formatting). <a href="">Now we learn that they&rsquo;ve essentially pwned all cell phone SIM cards</a>.</p> <blockquote cite=""> <p>The bottom line is that people around the world, regardless of their nationality, should know that the United States is not spying on ordinary people who don’t threaten our national security and that we take their privacy concerns into account in our policies and procedures.</p> <footer>Barack Obama, Jan 17, 2014</footer> </blockquote> <p>Between this and the fact that the private key for the Lenovo adware/malware was cracked in 3 hours by a single man&#8230;</p> Superfish, or how to MITM everyone Thu, 19 Feb 2015 00:00:00 +0000 <p>Lenovo was just added to the list of companies I&rsquo;m hesitant to ever buy anything from ever again, in any capacity. As Ars Technica <a href="">reports</a>, the Superfish adware that was installed by default on Lenovo machines presents a self-signed root CA certificate in the Trusted Roots for the system&rsquo;s SSL keys. This certificate was also trivially cracked. This means, if you are running Windows as shipped on a Lenovo machine, you may well be subject to insane security breaches.</p> <p>Best bet: backup your personal data, return the Lenovo, and get a new laptop from a different vendor. And when you get the new machine, wipe the drive and reinstall your OS of choice from a vendor-supplied DVD. Only then should you put your data back on the machine.</p> <p>Oh, yeah, and never trust OEM supplied OS images, ever again.</p> Changes Tue, 30 Dec 2014 00:00:00 +0000 <p>New Year News: I will be posting more here and on <a href="">GitHub</a> in 2015. I have taken a new job, due to start in late January, which encourages open source project work. I will be working for a company called <a href="">42 Lines</a>, doing systems administration and Operations work.</p> FreeNAS Tue, 18 Nov 2014 00:00:00 +0000 <p>FreeNAS is becoming more and more unusable for me. Part of the reason for using it was access to the jails and plugins &#8211; notably CrashPlan and OwnCloud. Due to the moving target nature of the system, I have had to repeatedly login and restart processes, manage memory, kill zombies, and deal with other issues in the jail setup. Further, the performance of AFP in FreeNAS (for me) has not been something to write home about, even with the newer bits (3.1.x branch, last I looked). The last straw is that the ZPOOL version is 5000 with a number of feature flags that nobody else has implemented yet. This means that the pools on my system are unreadable without the very newest releases of FreeNAS or FreeBSD.</p> <p>I&rsquo;m backing up now, headed for the better understood lands of <a href="">ZFS on Linux</a>. It looks stable enough to keep me afloat, and I know how to keep CrashPlan from spawning 65000 zombies in the background.</p> Dropbox Pro Quota Levels Wed, 27 Aug 2014 00:00:00 +0000 <p>I was just notified that Dropbox Pro is <a href="">10x&rsquo;ing their paid storage plans</a> &#8211; this is good news for my family and I. I had been thinking of moving off Dropbox and over to iCloud Drive this fall when Mac OS X Yosemite ships (assuming it works), but I&rsquo;m less likely to move now.</p> <p>Which is probably exactly the reason Dropbox is doing this.</p> Motorola Surfboard Metrics Sat, 12 Jul 2014 00:00:00 +0000 <p>Announcing another quick-and-dirty perl script today: <a href="">surfboard-metrics</a></p> <p>I have been having a lot of trouble with my ISP over the last few weeks &#8211; constant disconnects triggered by a modem reboot &#8211; which were taking us offline for 3-5 minutes at a time (or longer). This would be a mere annoyance, other than the fact that I work from home pretty frequently, and the reboot takes the connection out long enough to drop VPN and ssh sessions. The longest outage to date is 4 hours. I finally decided to start collecting data to see if I needed to add a powered amplifier to my cable system.</p> <p>I&rsquo;m using a <a href="">Motorola Surfboard 6121</a>, and while the spec sheet lists SNMP v2 and v3, they appear to only allow SNMP access over the coax interface. It&rsquo;s for the ISP to use, not for the end user. This means screen-scraping the web interfaces, which by default live at <a href=""></a>. There is also a log event page at <a href=""></a> but I&rsquo;m not dealing with that yet.</p> <p>The important thing to watch for are power levels and the signal-to-noise ratios for the upstream and downstream channels. If upstream goes above 55dBvM, or if downstream gets much above 40dB, performance will go to crap, and you will start seeing <code>No Ranging Response received - T3 time-out</code> or <code>Unicast Ranging Received Abort Response - initializing MAC</code> and the modem will eventually reboot itself.</p> <p>The script I wrote screen scrapes the pages, and outputs to a graphite/carbon server all the various metrics. I&rsquo;ve tried to make it reasonably flexible &#8211; if you have a higher speed connection and have multiple bonded upstream or downstream channels, the script should be able to create a metric for each channel &#8211; but as I only have my modem for reference, I can&rsquo;t verify some of that. As it might be helpful, I&rsquo;ve put it on github. If you have a Motorola Surfboard, give it a spin. I welcome pull reuquests that add support for ohter modems or other outputs to metric engines.</p> <p>Here&rsquo;s a screen capture of my local grafana instance, looking at recent data:</p> <p><a href=""><img src="" alt="Screen Shot 2014-07-12 at 12.15.51 PM" width="1024" height="439" class="alignright size-large wp-image-3256" /></a></p> <p>You can see a reboot at <sup>7</sup>&frasl;<sub>11</sub>, around 14:30, and again on <sup>7</sup>&frasl;<sub>12</sub> at 11:30. I think the next step is to correlate temperature and humidity readings, as well as internet traffic levels, and see if any patterns emerge.</p>