DaemonForums  

Go Back   DaemonForums > FreeBSD > FreeBSD General

FreeBSD General Other questions regarding FreeBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 29th December 2008
terryd terryd is offline
Port Guard
 
Join Date: Nov 2008
Posts: 22
Thanked 0 Times in 0 Posts
Thumbs up ZFS

Just a report more than a question. Over the last few weeks I have put it into production on about 3 servers + plus mine at home.
All 3 have just been a mirror setup just for the data running on HP ProLiant ML115 G5's all are used just for samba shares. With between 20 to 30 users during working hours which is no great work load I know.
But I have yet to see one single error or hear of any complaints on the 3 servers.
Mine at home is an old i386 with 1gb of ram which I ran with out tweaks. When I rsynced 160gb of data to it from a windows pc it would stop responding and reboot. After applying the tweaks from http://wiki.freebsd.org/ZFSTuningGuide

vm.kmem_size="512M"
vm.kmem_size_max="512M"

in /boot/loader.conf I can hammer it as hard as I like and no more problems
The main downfall of zfs is it just works and takes minutes to setup so theres no chance to play with it much
Reply With Quote
  #2   (View Single Post)  
Old 30th December 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 699
Thanked 90 Times in 81 Posts
Default

You'll want to limit the ZFS ARC as well, to no more than 50% of the kmem_size_max.

It's the ARC that causes all the lockup issues, as it uses kernel memory. If you don't limit it, it tries to use up to 90-some-odd% of kmem_size_max, which means, if you have a lot of network traffic, disk traffic, zfs traffic, and so on all at once, you'll get the "kmem_map too small" issues. ZFS uses all the kmem, so there's nothing left for the rest of the kernel services to use.

vfs.zfs.arc_max is the tunable to set in /boot/loader.conf.

The fun comes when you start to play with all the ZFS properties, like compression (gzip-9 to save space, lzjb to save space without sacrifising CPU), sharenfs (configures the exports file automatically), and shareiscsi (can be used with zvols and the iscsi-target port).

Then you can start in on the fun that is "zfs send" and "zfs recv", which lets you atomically transfer snapshots between systems (either the entire snapshot is created, or nothing is saved). That's what I'm currently playing with, in an effort to create a remote, off-site replica of our backup server.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
  #3   (View Single Post)  
Old 30th December 2008
terryd terryd is offline
Port Guard
 
Join Date: Nov 2008
Posts: 22
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by phoenix View Post
You'll want to limit the ZFS ARC as well, to no more than 50% of the kmem_size_max.

It's the ARC that causes all the lockup issues, as it uses kernel memory. If you don't limit it, it tries to use up to 90-some-odd% of kmem_size_max, which means, if you have a lot of network traffic, disk traffic, zfs traffic, and so on all at once, you'll get the "kmem_map too small" issues. ZFS uses all the kmem, so there's nothing left for the rest of the kernel services to use.

vfs.zfs.arc_max is the tunable to set in /boot/loader.conf.

The fun comes when you start to play with all the ZFS properties, like compression (gzip-9 to save space, lzjb to save space without sacrifising CPU), sharenfs (configures the exports file automatically), and shareiscsi (can be used with zvols and the iscsi-target port).

Then you can start in on the fun that is "zfs send" and "zfs recv", which lets you atomically transfer snapshots between systems (either the entire snapshot is created, or nothing is saved). That's what I'm currently playing with, in an effort to create a remote, off-site replica of our backup server.

Now that does sound worth doing. Funny you should mention zfs send" and "zfs recv a friend on irc just suggested i go play with that.
How is the off site stuff going ? We have a few offsite servers that I would like to backup to the main office. At the moment I use rsync but it would be nice to have other options
Reply With Quote
  #4   (View Single Post)  
Old 30th December 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 699
Thanked 90 Times in 81 Posts
Default

We use rsync to backup out remote servers (82 so far) to the backup box running FreeBSD+ZFS, taking a snapshot of the backup directory each night. We do full rsyncs of the entire remote system (everything except /proc, /sys, /tmp, and /dev), into separate directories for each server. That way, we can do full system restores as well as single-file recoveries.

After the initial sync, which can take a few days across an ADSL link, the follow-up syncs take between 30 seconds and 2 hours, depending on the amount of data that has changed and the speed of the link.

Our backup script is configured such that it starts 1 backup process per remote site, waiting 7 minutes before starting the next process. That process then runs rsync for each server at the site in sequence. So we can have up to 15-ish rsync processes running at a time. (In theory, we could have 55 running at once, as that's the number of remote sites we have, but they tend to finish fast enough that we never have more than around 15 running.)

As for the off-site replica of our backup box, I'm still running tests with the ZFS send/recv stuff. The atomic nature of the snapshot feature can be a bit touchy. For example, trying to do a # zfs send storage/backup@20080805 | ssh remoteserver "zfs recv storage/backup@20080805 would fail on me. Turns out, that was combining 4 snapshots, and trying to send 1.2 TB of data across the network. The SSH connection would fail after 36-ish hours, then the box would spend about 12 hours deleting all the data transferred so far. Breaking the process up into "zfs send pool/snapshot > file", "scp file remoteserver:", and "zfs recv pool/snapshot < file" steps makes it a lot smoother. Less automated, but more reliable.

Now I'm in the process of brining the replica box up-to-date with the 100+ snapshots we've taken so far.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 10:18 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick