Misc. BSD/UNIX ZFS just got built-in deduplication

[vermaden]

Deduplication is the process of eliminating duplicate copies of data. Dedup is generally either file-level, block-level, or byte-level. Chunks of data -- files, blocks, or byte ranges -- are checksummed using some hash function that uniquely identifies data with very high probability.

(...)

Chunks of data are remembered in a table of some sort that maps the data's checksum to its storage location and reference count. When you store another copy of existing data, instead of allocating new space on disk, the dedup code just increments the reference count on the existing data. When data is highly replicated, which is typical of backup servers, virtual machine images, and source code repositories, deduplication can reduce space consumption not just by percentages, but by multiples.

MORE: http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup

[TerryP]

Sounds like pushing the same concept as hardlinks, down into a file system smart enough to optimize for the occasion; I'd love to see how dedup and zfs could be used to tune database servers tbh, but lack the resources to experiment.

[vermaden]

@TerryP

ZFS deduplication is block based, so if you have two different files, but some blocks are the same, you still gain space.

[TerryP]

Take a look at git-clone or ln documentation sometime, you might understand what I meant by concept .

git clone is able to exploit hard links as part of cloning repositories through the file system rather then the network, as a way to save disk space and time used; chosen, I assume for the same reason that FreeBSD uses hardlinks instead of symlinks to create programs with multiple names pointing to one binary (like ex, vi, nex, and nvi).


Being able to apply the same concept as a hard link, automagically at a lower level then whole files [sic], is a good thing :-D.