In my last post about file systems, I talked about options that were replicated, shared-nothing, and distributed. One of the options that came up was OpenAFS, and while it doesn't yet support replicated read/write volumes, it does have a lot going for it, including a healthy perl module that allows a lot of operations to be done from scripts, and the fact that I administer OpenAFS at work, so I have a passing familiarity with it.
OpenAFS also allows a seamless mounting of logical volumes anywhere inside the filesystem root, and allows for the live migration of volumes between storage nodes with no user downtime - users are still able to write to live volumes while they are being moved. This in turn creates an opportunity for some neat tricks: if one of the nodes is setup with smaller but fast SSD's, frequently accessed volumes could be migrated to the faster nodes seamlessly, and less frequently used volumes could be moved off to slower, cheaper drives.
OpenAFS still doesn't give me one the the features I really want, however: real time, distributed protection of data between nodes. It seems essential at this point that I have the ability to shut down any node at any time and not suffer any outages. This is a tall order, I know, but this is the modern world of technology, damnit.