The Guru College

Changes

New Year News: I will be posting more here and on GitHub in 2015. I have taken a new job, due to start in late January, which encourages open source project work. I will be working for a company called 42 Lines, doing systems administration and Operations work.

FreeNAS

FreeNAS is becoming more and more unusable for me. Part of the reason for using it was access to the jails and plugins – notably CrashPlan and OwnCloud. Due to the moving target nature of the system, I have had to repeatedly login and restart processes, manage memory, kill zombies, and deal with other issues in the jail setup. Further, the performance of AFP in FreeNAS (for me) has not been something to write home about, even with the newer bits (3.1.x branch, last I looked). The last straw is that the ZPOOL version is 5000 with a number of feature flags that nobody else has implemented yet. This means that the pools on my system are unreadable without the very newest releases of FreeNAS or FreeBSD.

I’m backing up now, headed for the better understood lands of ZFS on Linux. It looks stable enough to keep me afloat, and I know how to keep CrashPlan from spawning 65000 zombies in the background.

Dropbox Pro Quota Levels

I was just notified that Dropbox Pro is 10x’ing their paid storage plans – this is good news for my family and I. I had been thinking of moving off Dropbox and over to iCloud Drive this fall when Mac OS X Yosemite ships (assuming it works), but I’m less likely to move now.

Which is probably exactly the reason Dropbox is doing this.

Motorola Surfboard Metrics

Announcing another quick-and-dirty perl script today: surfboard-metrics

I have been having a lot of trouble with my ISP over the last few weeks – constant disconnects triggered by a modem reboot – which were taking us offline for 3-5 minutes at a time (or longer). This would be a mere annoyance, other than the fact that I work from home pretty frequently, and the reboot takes the connection out long enough to drop VPN and ssh sessions. The longest outage to date is 4 hours. I finally decided to start collecting data to see if I needed to add a powered amplifier to my cable system.

I’m using a Motorola Surfboard 6121, and while the spec sheet lists SNMP v2 and v3, they appear to only allow SNMP access over the coax interface. It’s for the ISP to use, not for the end user. This means screen-scraping the web interfaces, which by default live at http://192.168.100.1/cmSignal.htm. There is also a log event page at http://192.168.100.1/cmLogs.htm but I’m not dealing with that yet.

The important thing to watch for are power levels and the signal-to-noise ratios for the upstream and downstream channels. If upstream goes above 55dBvM, or if downstream gets much above 40dB, performance will go to crap, and you will start seeing No Ranging Response received - T3 time-out or Unicast Ranging Received Abort Response - initializing MAC and the modem will eventually reboot itself.

The script I wrote screen scrapes the pages, and outputs to a graphite/carbon server all the various metrics. I’ve tried to make it reasonably flexible – if you have a higher speed connection and have multiple bonded upstream or downstream channels, the script should be able to create a metric for each channel – but as I only have my modem for reference, I can’t verify some of that. As it might be helpful, I’ve put it on github. If you have a Motorola Surfboard, give it a spin. I welcome pull reuquests that add support for ohter modems or other outputs to metric engines.

Here’s a screen capture of my local grafana instance, looking at recent data:

Screen Shot 2014-07-12 at 12.15.51 PM

You can see a reboot at 711, around 14:30, and again on 712 at 11:30. I think the next step is to correlate temperature and humidity readings, as well as internet traffic levels, and see if any patterns emerge.

Aperture is Dead

Apple has decided to kill Aperture. I left Aperture years ago due to performance issues when the library got large enough. Now, it will be time to get my wife off of it. I hope that “Photos for OS X”, coming in the spring, is good enough. I loathe the idea of giving Adobe any money for Creative Cloud, but I don’t think that DarkTable is friendly enough for my wife to use every day.

FreeNAS: First Impressions

I’ve been “production” on my new FreeNAS based file server for a few days now. I finished the data migration over the weekend. I’m still drinking in the differences, and trying to get a handle on all the ways FreeNAS (and recent FreeBSD builds) are different than an years-old install of OpenSolaris. I am mostly impressed. I am going to miss FMA on Solaris and a few of those sort of features that made the OS so incredibly good (cfgadm comes to mind), and I’m not sure how FreeNAS handles drive failures and removals – I’m sure I will find out over the next weeks, months and years.

The cautionary tale I have for today is using “AFP” shares in FreeNAS, which I assume is netatalk. My photo library lives on the NAS, and has for years. One of the things I was looking forward to moving off OpenSolaris for was a better stack for CIFS/AFP file services. This evening, I finally settled down to get some image editing done – and the system was so incredibly slow as to be unusable. Going into the Finder, pulling a directory listing was taking 20-30 seconds. I was starting to lose my mind, but I decided to fall back to NFS, which I had been using to move data around, and had seemed quick for bulk transfers. Everything was lightning fast.

I don’t have anything conclusive yet, but it seems that NFS is an order of magnitude faster than AFP, out of the box, on FreeNAS. I’ll look into this further, but I’m tired, and I don’t want to try to dig up numbers tonight.

Personal Projects: Moving to FreeNAS

Another personal technology project I have put into motion this summer is abandoning OpenSolaris once and for all and moving to FreeNAS. This involves a totally new machine, which has been ordered, and will move the data from 4 pairs of mirrored disks to an 8 disk RAIDZ2. This gives me much better data protection and a lot more usable space.

I’m buying NAS-rated drives for the RAIDZ2 pool, which is for general file and media storage. NAS-rated 2TB drives are now $99 on NewEgg. Using RAIDZ2 lets me lose any two drives simultaneously and not have a loss of data. I can currently lose 4 drives in my mirrored pair setup, but if any two drives are from the same pair, the whole pool is lost. Considering the age of the oldest drives, I’m not liking my odds any more. I was looking at 3TB and 4TB drives, but I’m not comfortable with the URE rates and rebuild times when larger drives are involved, so I’ve stuck with the smaller drives.

I have also moved to a Xeon processor, 16 GB of ECC RAM and a new PCIe 2.0 LSI HBA. This is going to give me significant throughput gains over my current setup, and the 16GB of RAM should help buffer when I move from mirrored pairs to RAID. The smallest files I do work with are Nikon NEF files, which are 12MB+ each, so I don’t have as much issue with small file I/O as some people do.

I toyed with the idea of getting a SSD for a slog or an L2ARC, but I decided against it for now. The ZIL/slog is most useful for doing random NFS writes, which I do very little of, and the L2ARC has a memory cost that is pretty steep – 5GB of L2ARC consumes 1GB of system RAM. A 60GB SSD would chew through 12GB of RAM, and I’m only buying 16GB to start with. There are two slots open on the new motherboard, so I can add more RAM if needed, and that is a much more performant place to add ARC.

The order is placed and the machine will be here over a couple of days next week. I’m looking forward to building a new system. I’ll take pictures as I build and post them here.

Newer Posts | Home | Older Posts