DaemonForums  

Go Back   DaemonForums > DaemonForums.org > News

News News regarding BSD and related.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 3rd November 2009
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,052
Thanked 118 Times in 93 Posts
Cool ZFS just got built-in deduplication

Quote:
Deduplication is the process of eliminating duplicate copies of data. Dedup is generally either file-level, block-level, or byte-level. Chunks of data -- files, blocks, or byte ranges -- are checksummed using some hash function that uniquely identifies data with very high probability.

(...)

Chunks of data are remembered in a table of some sort that maps the data's checksum to its storage location and reference count. When you store another copy of existing data, instead of allocating new space on disk, the dedup code just increments the reference count on the existing data. When data is highly replicated, which is typical of backup servers, virtual machine images, and source code repositories, deduplication can reduce space consumption not just by percentages, but by multiples.
MORE: http://blogs.sun.com/bonwick/en_US/entry/zfs_dedup
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
  #2   (View Single Post)  
Old 3rd November 2009
TerryP's Avatar
TerryP TerryP is offline
Arp Constable
 
Join Date: May 2008
Location: USofA
Posts: 1,547
Thanked 112 Times in 104 Posts
Default

Sounds like pushing the same concept as hardlinks, down into a file system smart enough to optimize for the occasion; I'd love to see how dedup and zfs could be used to tune database servers tbh, but lack the resources to experiment.
__________________
My Journal

Thou shalt check the array bounds of all strings (indeed, all arrays), for surely where thou typest ``foo'' someone someday shall type ``supercalifragilisticexpialidocious''.
Reply With Quote
  #3   (View Single Post)  
Old 3rd November 2009
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,052
Thanked 118 Times in 93 Posts
Default

@TerryP

ZFS deduplication is block based, so if you have two different files, but some blocks are the same, you still gain space.
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
  #4   (View Single Post)  
Old 3rd November 2009
TerryP's Avatar
TerryP TerryP is offline
Arp Constable
 
Join Date: May 2008
Location: USofA
Posts: 1,547
Thanked 112 Times in 104 Posts
Default

Take a look at git-clone or ln documentation sometime, you might understand what I meant by concept .

git clone is able to exploit hard links as part of cloning repositories through the file system rather then the network, as a way to save disk space and time used; chosen, I assume for the same reason that FreeBSD uses hardlinks instead of symlinks to create programs with multiple names pointing to one binary (like ex, vi, nex, and nvi).


Being able to apply the same concept as a hard link, automagically at a lower level then whole files [sic], is a good thing :-D.
__________________
My Journal

Thou shalt check the array bounds of all strings (indeed, all arrays), for surely where thou typest ``foo'' someone someday shall type ``supercalifragilisticexpialidocious''.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
a novel idea: a sh with a `fork built-in? TerryP Programming 4 25th March 2009 07:17 PM
Keyboard with built in mouse trackball blodan FreeBSD General 2 21st June 2008 07:44 PM


All times are GMT. The time now is 01:39 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick