Running quite nicely. It's doing nightly backups of 79 servers at the moment, creating a snapshot each night before the rsync run starts. For those 79 servers, we are averaging only 2 GB of changed data per night, with the occasional spike to 10 GB. We have 64 snapshots so far, using just under 4 TB of data in total. The normal (no new servers added) rsync run takes under 2 hours. When we add a new server, the run can take up to 8 hours, depending on how much data needs to be transfered.
You really need to tune the kernel memory map and the ZFS ARC size. Both of these are set in /boot/loader.conf. The max kmem size in FreeBSD 6 and 7 is 1536 MB, although few people can get it to work about 1500. For our purposes (rsync, massive reads every night) we found that you want to push the kmem as high as you can (in small increments), then set the ARC to about 1/2 of the kmem size. That gives ZFS lots of RAM to cache things in, but still leaves the kernel lots of room to play. And the more RAM you can put in the system, the better.
We experienced lockups almost daily during the tuning period. That calmed down to about once every 2 weeks while we played with how often to add new servers to the backup list (heavy network and file I/O across 20+ rsync processes will lock it up). Now we're at about 1 every 6 weeks. With a little more fine-tuning, we should be able to get to 1 every 6 months or so.
I did have to hack the /etc/rc.d/zfs script to force it to unconditionally attempt to import the pool. During all the lockups and initial tuning, we would lose the ability to import the pool automatically during boot. Adding zfs import -f <poolname> to the start function was needed. We could probably remove it now, but it doesn't harm anything to be there.
And, if you use zvols for swap, you'll have to add a swapon command to /etc/rc.local as it doesn't seem to always enable it via the rc.d scripts.
Beyond that, we are *extremely* pleased with it. Being able to browse through the .zfs/snapshot/<snapshotname>/ folder hierarchy to look for files is just awesome! No mounting needed, as zfs does it for you automatically. And doing system restores from any snapshot is just as easy.
Our next step is to build a duplicate server, house it off-site, and use the snapshot stream feature to have redundant backups.
After that we want to build some kind of web management interface around it so that people can browse through the snapshots for their servers to download files as needed.
|