Yay! All my data has been migrated, and the pool contents balanced out over the VDEVs, so I should be running at optimal performance. All that is left is running through my weekly cronjobs to make sure that nothing misfires (like my weekly scrubs). This is a major victory for me! I’ve actually upgraded the server, on time and under budget.

But in doing so, I discovered something irritating about ZFS that I really should have known. When you have a pool made of VDEVs of different sizes, the VDEVs all fill at the same rate, not proportionally to their size. That is, if you have a 2 TB VDEV and a 1 TB VDEV, and you write 600 GB of data, 300 GB goes to each VDEV, not 400 GB to the 2 TB VDEV and 200 GB to the 1 TB.

Yes, this makes sense in many ways, as it won’t bais the pool to a specific VDEV, but it also means that after I write a little over 1 TB of extra data, I’m going to be down to the 2 TB VDEVs – so I’ll have two disks worth of IOPS again, not 4.

You win some, you lose some, I guess.