ZFS stability compared to Solaris

[tanked]

Does anyone know why ZFS on 32 bit Solaris with (fairly) limited memory is stable, but on FreeBSD under the same hardware conditions it is not? Its not massively important, its just something I've been wondering for some time.

[Oliver_H]

>but on FreeBSD under the same hardware conditions it is not

I'm using OpenSolaris (Build 99) and FreeBSD stable (build from yesterday) on 32bit (Athlon X2, Intel C2D) together with ZFS (2G of memory). At home I'm using ZFS for fileserver purposes (backups of all family-members, pictures, videos etc. pp.) - without any problems. But it could be very different under high load on a production server.

[ninjatux]

You need to tune on FreeBSD, where as on Solaris, you generally don't. The ZFS support in STABLE is listed as experimental. That's not in respect to its implementation, but more so the tuning aspect and such. The implementation is sound, otherwise it wouldn't be in STABLE.

[phoenix]

The big difference is in the way the memory allocations work. I don't know the specifics on the Solaris side of things, but on the FreeBSD side, the ZFS ARC uses kernel memory (compared to the unified buffer cache that the rest of the OS uses for accessing UFS, FAT, ISO9660, etc, which is non-kernel memory). There's a finite amount of kernel memory on 32-bit systems (25% of RAM up to 1 GB by default, max 2 GB).

So, if you don't tune the kernel memory and ZFS ARC settings on 32-bit systems, then ZFS will try to use all your RAM for the ARC, and cause kernel panics ("kmem_map too small" errors).

Things are better on the 64-bit side of things, as there aren't the same limitations on kernel memory. FreeBSD 7.x still has a 2 GB kmem max size (actually about 1596 MB in practise), but FreeBSD 8.x will have a 512 GB max. There are quite a few updates and fixes for ZFS coming with FreeBSD 7.1. And FreeBSD 8.0 will have ZFS version 11, with a bunch more features, better performance, and more stability.

So give it time. It's not like ZFS appeared in Solaris working perfectly from the get-go. They've had 10+ years to get things to this point. FreeBSD has had less than 2.

[Oliver_H]

ZFS _is_ a work in process in Solaris and of course in FreeBSD too. But it runs quiet well, again I wouldn't try it on a production server whether it's marked experimental or stable. The more people test it now the earlier it will be ready for production.

>So give it time.

But without real world tests it will be never ready and FreeBSD has not got a plethora of users. Pjd cannot do it all alone, he can do a thorough port, but the users are for fine-tuning.

>They've had 10+ years to get things to this point.

And as a commercial company they did tests on real world machines to iron ZFS, but there is no company which will take this burden for FreeBSD. Ten years to mature as filesystem per se, it doesn't need another ten years for a "mere" port ;-)


http://wiki.freebsd.org/ZFSKnownProblems
http://wiki.freebsd.org/ZFSTuningGuide
http://wiki.freebsd.org/ZFS

[tanked]

Originally Posted by phoenix

The big difference is in the way the memory allocations work. I don't know the specifics on the Solaris side of things, but on the FreeBSD side, the ZFS ARC uses kernel memory (compared to the unified buffer cache that the rest of the OS uses for accessing UFS, FAT, ISO9660, etc, which is non-kernel memory). There's a finite amount of kernel memory on 32-bit systems (25% of RAM up to 1 GB by default, max 2 GB).

Why does ZFS on FreeBSD use kernel memory but the other file systems don't?