Happy New Year! It’s 2012, finally, and we’ve have a new member of the family to celebrate. It’s also possible to now use the phrase “election year” correctly when referring to the 2012 US Presidential Election. 2011 was an interesting year – I learned how to make hot sauce, hard cider and take much better photographs with off camera flash. I also expanded my professional abilities, doing a lot with Nagios I never thought possible, and diving deeper into Linux. All in all, it was a good year.

This coming year, I only have one resolution, that my almost-three-year-old said two days ago – “Follow the Me”. I’m going to do that, and follow wherever it leads. I do have some time off at the beginning of this year to spend time bonding with the newest Dezendorf, and I’m also going to use this time to also work on some photography and cooking projects I’ve been putting off.

Please help me welcome the newest member of the Dezendorf/Shoneman family: Nora Anne Dezendorf. She was born on December 22nd, 2011, at 11:46 PM, weighing 9 lbs and 10 oz. There were no complications in labor. She is at home with her loving family, resting from the ordeal of being born. Pics on gurufoto.net and flickr.

I’ve noticed an irritating trend in its apps – the apparent desire of app developers to keep the user in the app for as long as possible. Facebook, Twitter and Flipboard are the worst offenders I’ve seen recently. Every attempt to visit a linked URL opens a WebKit view, but everything stays within the app. UI controls are what the app wants, and you are unable to save bookmarks or otherwise surf the web in the normal way. The user can then choose to open the page in safari – sometimes. Even that can be tricky, as Facebook for a long time was adding an IFRAME to each page, making you click through once you got to Mobile Safari. All so you could add the article to your reading list, Instapaper account, or email it to a friend and not use Facebook’s internal messaging functionality. All of this is maddening.

It reminds me of the horrible ‘news’ sites that break a 600 word article into three pages to maximize the ad views, sites that take an image gallery and make it into a slideshow, or the randomly highlight words and phrases in their content and have all of those links go to other parts of their sites. All of them tend to bury the link to the source material as deep as possible, and those are the times they even bother to link. It’s the Scumbag Steve of SEO, and it is all done with the intention of keeping the user on the site for as long as humanly possible.

Sadly, it seems to be a sustainable business model for some. Facebook is just as popular as ever.

I’m a long time Aperture user. I’ve been in the habit of saving my Master images (be them .NEF, .TIFF, .PSD or .JPG) to secondary disks or NAS devices for years. This lets me keep my Aperture library, containing the metadata about the images, the preview thumbnails, and the project organization relatively small. (1.4 TB of images, 97 GB library). However, I’ve been running without a secondary file server for months now, which I was using for a backup of the primary image collection, and I’m not terribly comfortable without a reliable backup. CrashPlan is running on my machines at home, but literally 8 months remain until all my images are backed up to the cloud. I would also prefer to have backups at home.

This has lead me to do a bunch of research on distributed and replicated network file systems. The idea being that you buy a number of relatively cheap computers to be used as file server nodes, you join them in some sort of namespace, and then you let the servers act as redundant copies of each other in case one is down with hardware issues or needs to be patched, or whatever. The popular options seem to be the RedHat backed GlusterFS, the network file system side of Hadoop, and Lustre, the somewhat failed distributed file system that Sun purchased and Oracle seems to have killed off. In the developmental space, there is Ceph, which looks promising but really isn’t ready for prime time.

Of course, none of these support OS X natively, and have lots of their own internal problems. Hadoop can only be accessed over it’s JSON/REST calls, or via a FUSE plugin; Lustre development is dying and similarly needs to be shared via FUSE; GlusterFS is modern and supported, but you have to re-share the namespace via SMB or NFS if you want to use it on your Mac, and the syntax for growing pools of storage seem archaic. Ceph is dynamic, POSIX compliant, and apparently awesome, but it’s not done, and there’s again no native OS X support, so it’s FUSE or SMB/NFS time.

I don’t want to set up a cluster of nodes just to have the one that is serving over SMB die, and have all my connections get reset. I want to access the file systems natively, and in a way that Aperture can actually use the space in the way the file system designers intended. And, much to my surprise, it seems that all roads are pointing me to OpenAFS, which I just so happen to use at work.

To start, it doesn’t do some of what I want, which is the automatic server-to-server replication that you see in Ceph or GlusterFS. But apart from that, it seems to rock along nicely. Volumes can be moved between servers, or even partitions on servers, while they are in active use by clients. All authentication is done over Kerberos or IP ACLs. Backups are automatically run nightly, and with a little scripting, volumes can be balanced between nodes without much effort. But the best of all is that it’s “native” to OSX – there’s a kernel-level driver to install, sure, but once that’s done, the global AFS namespace shows up under /afs – and by global, it covers every publicly available AFS installation in the world, if you want it to. It’s been around for 25+ years, and it just works. There are robust clients for OSX, Windows, Linux and Solaris, which are all the machines I’ll ever run it on.

Even better is the idea that if I were to partition my photo repository properly, I could move lesser used volumes to nodes with larger, slower disks, and move recently used files to faster ones – perhaps SSDs. With a working knowledge of the AFS command set, perl and find, this could even be done somewhat automatically, where every 30 minutes or so “hot” volumes could be moved to faster storage.

A lot of this is still a pipe dream – I need to replace my desktop before I turn my eye to storage solutions – but planning is King, right? Comments or questions, please sound off in the comments.

This is a feature suggestion for Apple.

I want the ability to turn off the Camera Roll on my iPhone. Not the extra albums that I’ve lovingly created and filled by hand, but the primary photo album. The one everything goes into by default. I want to turn it off and replace it with a better integrated Photo Stream. A Photo Stream that acts just like the basic Camera Roll until you turn Photo Stream on a Mac hooked to your iCloud account. iCloud is smart enough to see this already. When there is another computer that can sync, it starts to download photos from your iCloud account, leaving the last 30 days or 1000 pictures in iCloud. The next time you snap another picture, or the next time the date rolls over, the iOS device would trim down to the 30 days/1000 photo limit. The Mac would already have the images, so nothing would need to be done there.

Occasionally, your Mac wouldn’t be online for a long period of time. Perhaps you are on an extended vacation. Again, iCloud is smart enough to see this, and would have your iPhone retain more images than usual (to keep Photo Stream at it’s 1000 image limit). As soon as that computer came back online, and caught up downloading images, your iPhone would get the go-ahead to prune the images on your phone.

Simple, direct, Apple-like, and importantly, even less work or hassle for the end user.

Netatalk 2.2.1 happily doesn’t blow up when the volume gets full. They have this in their notes, but I had to check it myself. I tested it last night by setting the quota on the volume pretty low, and then having a script generate random 2.5GB files (and overwrite previous files when it made a new one). This ensured a constant flow of new data that would need to be backed up, and correctly, TimeMachine started deleting my backups from the netatalk volume.

Now that this is done, I’ve turned compression back on and upped the quota on the volume to 300GB. This also means I can recover the 750GB drive I’m currently using for TimeMachine.

I now have OpenSolaris (b151a), Mac OS X Lion (10.7.2) and netatalk v2.2.1 working together in harmony. What follows is a brief setup guide:

Download, build and install a current version of BerkleyDB. I used 5.2.36, the current version on Oracle’s website.

1. cd /tmp
2. tar xvzf db-5.2.36.tar.gz
3. cd db-5.2.36/build_unix
4. ../dist/configure
5. make && pfexec make install

Next, install gettext, from the IPS repo. If you are still using OpenSolaris like I am, you will need to unset the opensolaris.org repo, as it no longer responds.

1. pfexec pkg unset-publisher opensolaris.org
2. pfexec pkg install gettext

Next, download build and install the current version of netatalk, using the bdb you installed earlier:

1. cd /tmp
2. tar xvf netatalk-2.2.1.tar
3. cd netatalk-2.2.1
4. pfexec ./configure --with-bdb=/usr/local/BerkeleyDB.5.2
5. make && pfexec make install

You are almost done. Create your ZFS filesystem for AFP sharing, setup your config files, and fire up the daemons:

1. pfexec create -o compress=on tank/timemachine
2. pfexec chown -R <em>username</em> /tank/timemachine
3. pfexec chmod -R u+rwx /tank/timemachine
4. echo "/tank/timemachine \"Time Machine Backups\" allow:<em>username</em> options:tm" >> /usr/local/etc/netatalk/AppleVolumes.default
5. pfexec /etc/init.d/netatalk start

Netatalk should fire up a cnid_metad process as well as an afpd process, and you can now connect to the OpenSolaris server with afp://hostname. Once it’s mounted, Lion can use the volume “Time Machine Backups” as a TimeMachine disk, as it runs the full suite of AFP3.3 commands, including the fast/force sync stuff to make sure the writes happen even if the network connection is severed, which allows TimeMachine to resume backups.

As a word of warning: DO NOT kill -9 THE afpd OR cnid_metad PROCESSES. Use SIGTERM or SIGHUP, or just use the init script. If you kill -9, you run the very real risk of killing the CNID DB backend, which keeps track of all the Apple specific file relations, which make things like aliases work. Also, try very hard not to use hard links in AFP volumes – it’s a world of pain waiting for you.