DaemonForums  

Go Back   DaemonForums > Miscellaneous > General software and network

General software and network General OS-independent software and network questions, X11, MTA, routing, etc.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 6th May 2008
stukov's Avatar
stukov stukov is offline
Real Name: Jean-Michel Philippon-Nadeau
Package Pilot
 
Join Date: May 2008
Location: Sherbrooke, Qc, Canada
Posts: 167
Default Remote backup utility

Hòla,

Is anybody aware of a software that specializes in remote backups? I know I could do it with a simple implementation of rsync but if there is anything else more elegant am I willing to give it a try. The problem am I trying to solve is the bandwidth required to backup a complete machine in case of disaster or just to roll-back a few days ago files erased by mistake.


Thanks!
__________________
"Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius -- and a lot of courage -- to move in the opposite direction."
Reply With Quote
  #2   (View Single Post)  
Old 6th May 2008
anomie's Avatar
anomie anomie is offline
Local
 
Join Date: Apr 2008
Location: Texas
Posts: 445
Default

I've never had to actually implement an enterprise backup solution from scratch (I have worked with other folks' solutions), but I thought amanda looked pretty slick. Might be worth checking out to see if it meets your needs.
__________________
Kill your t.v.
Reply With Quote
  #3   (View Single Post)  
Old 6th May 2008
DrJ DrJ is offline
ISO Quartermaster
 
Join Date: Apr 2008
Location: Gold Country, CA
Posts: 507
Default

I use bacula:

http://www.freebsd.org/cgi/url.cgi?p...rver/pkg-descr

Works well.
Reply With Quote
  #4   (View Single Post)  
Old 7th May 2008
dug dug is offline
New User
 
Join Date: May 2008
Posts: 1
Default

Check out rsnapshot...
http://www.rsnapshot.org/
Reply With Quote
  #5   (View Single Post)  
Old 11th May 2008
bigb89 bigb89 is offline
Fdisk Soldier
 
Join Date: May 2008
Posts: 69
Default

BackupPC is great software that can backup *nix machines, as well as windows. The best part is that no client side software is needed and you get to choose between rsync, tar, smb (rsync being the best) as your backup method. Right now I'm backing up 48 servers using BackupPC and its being working great so far. Here's the website: http://backuppc.sourceforge.net.
Reply With Quote
  #6   (View Single Post)  
Old 11th May 2008
corey_james corey_james is offline
Uber Geek
 
Join Date: Apr 2008
Location: Brisbane, Australia
Posts: 238
Default

i'm a fan of amanda too
Reply With Quote
  #7   (View Single Post)  
Old 11th May 2008
18Googol2's Avatar
18Googol2 18Googol2 is offline
Real Name: whoami
Spam Deminer
 
Join Date: Apr 2008
Location: pwd
Posts: 283
Default

Acronis is my choice. It is superb backup application. Stable, easy (you wont make any mistake when backing up or restore. You know how costly backup mistake is). It can back up, restore from and to usb peripheral, samba, ftp.

If you back up slice which contains MBR, when you restore, it offers you the option to restore only MBR, slide or both. It can read standard windows, linux and freebsd file system.

5 and half star from me

PS: But lack of backing up in real time without rebooting and not being able to read file and dir within the backup image makes me think about another backup utility
__________________
The power of plain text? It can control an entire OS
Reply With Quote
  #8   (View Single Post)  
Old 11th May 2008
cajunman4life cajunman4life is offline
Real Name: Aaron Graves
Package Pilot
 
Join Date: May 2008
Location: Coolidge, Arizona
Posts: 203
Default

Quote:
Originally Posted by 18Googol2 View Post
PS: But lack of backing up in real time without rebooting and not being able to read file and dir within the backup image makes me think about another backup utility
Reboot to run backup? Yikes!
__________________
I just saved a bunch of money on my car insurance by fleeing the scene of the accident!
Reply With Quote
  #9   (View Single Post)  
Old 12th May 2008
windependence's Avatar
windependence windependence is offline
Real Name: Tim
Shell Scout
 
Join Date: May 2008
Location: Phoenix, Arizona
Posts: 116
Default

Those of you who use amanda, how hard is it to set up. It looks kinda difficult and keep in mind I run BSD servers and do not consider myself a noob. Is there a good tutorial out there?

-Tim
__________________
www.windependence.org
Get your Windependence today!
Reply With Quote
Old 27th May 2008
Starhost Starhost is offline
New User
 
Join Date: May 2008
Posts: 4
Default

Does Acronis work on BSD 7? I thought it was only a linux / windows solution.
Reply With Quote
Old 27th May 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Quote:
Originally Posted by stukov View Post
Is anybody aware of a software that specializes in remote backups? I know I could do it with a simple implementation of rsync but if there is anything else more elegant am I willing to give it a try. The problem am I trying to solve is the bandwidth required to backup a complete machine in case of disaster or just to roll-back a few days ago files erased by mistake.
We're in the testing stages of using rsync to clone all our Linux servers to a FreeBSD 7 system running zfs. Each server gets it's own directory under /storage, and a snapshot of /storage is taken after every run. This way, /storage/serverA contains the current files for serverA, and /storage@<date>/serverA contains the files for serverA from <date>.

We're looking at building a webpage around it that allows techs to login, see their servers, and then browse the file tree for the servers to restore individual files or directories. The nice thing about ZFS is that you can access the files in any of the snapshots without having to mount the snapshot. /storage/.zfs/snapshot contains the directory tree of all the snapshots.

So far, things look good. The initial rsync of a server takes a couple of hours and transfers a lot of data. The subsequent rsync runs transfer only the data that has changed, and takes maybe 10 minutes across an E10 link (an hour across an ADSL link due to the horrid upload speeds).

Our backup server has 9 TB of disk space, which should allow us to keep 30 days worth of snapshots for the 30 or so servers we'll be backing up, along with a few "monthly" snapshots.

The other nice thing about ZFS is that we can "clone" our backup server to an offsite server simply by running:
Code:
zfs send /storage@<date> | ssh remoteserver 'zfs receive /storage@<date>'
This sends a single stream of bits from one server to the next across an SSH connection. Much faster than rsync from one server to the next.

No software needs to be installed on the remote servers. Everything is done from the backup server, using simple sh scripts, rsync, and ssh.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 28th May 2008
windependence's Avatar
windependence windependence is offline
Real Name: Tim
Shell Scout
 
Join Date: May 2008
Location: Phoenix, Arizona
Posts: 116
Default

What are you using to snapshot the servers?

I have several servers in VMs and I want to take snapshots of them and use the snapshots of the VMs for backup. Will this work? Since VMs are just directories, I have them all under /vm and then each one has it's own directory under that like /vm/server1 etc. I would like to be able to take a hot backup of the servers using snapshots and move them to another machine later since transfer is slow even on gigabit ethernet. What are your thougts on this?

-Tim
__________________
www.windependence.org
Get your Windependence today!
Reply With Quote
Old 28th May 2008
cajunman4life cajunman4life is offline
Real Name: Aaron Graves
Package Pilot
 
Join Date: May 2008
Location: Coolidge, Arizona
Posts: 203
Default

Quote:
Originally Posted by windependence View Post
What are you using to snapshot the servers?

I have several servers in VMs and I want to take snapshots of them and use the snapshots of the VMs for backup. Will this work? Since VMs are just directories, I have them all under /vm and then each one has it's own directory under that like /vm/server1 etc. I would like to be able to take a hot backup of the servers using snapshots and move them to another machine later since transfer is slow even on gigabit ethernet. What are your thougts on this?

-Tim
I have a similar setup and use tar.gz from host system on the vm directories. That's essentially a "full" backup of the system. No need to use network resources when what you're trying to back up is on the same system. You can then send the files from the host system somewhere else if necessary.
__________________
I just saved a bunch of money on my car insurance by fleeing the scene of the accident!
Reply With Quote
Old 29th May 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

Depends how the VMs are configured. Are they file-backed, or do you export a block device via a volume manager? If file-backed, is it one large file as the "virtual harddrive"? If so, you can't to live backups outside the VM, unless you "pause" the VM first.

We're looking at using rsync from within our VMs to back them up. And plain dd to image the VMs while they are not running.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 2nd June 2008
windependence's Avatar
windependence windependence is offline
Real Name: Tim
Shell Scout
 
Join Date: May 2008
Location: Phoenix, Arizona
Posts: 116
Default

Quote:
Originally Posted by phoenix View Post
Depends how the VMs are configured. Are they file-backed, or do you export a block device via a volume manager? If file-backed, is it one large file as the "virtual harddrive"? If so, you can't to live backups outside the VM, unless you "pause" the VM first.

We're looking at using rsync from within our VMs to back them up. And plain dd to image the VMs while they are not running.
Thanks Phoenix,

All my VMs are just file based, and one large file for each virtual hard disk.

I am aware of the need to pause the VMs, hence my question on how you were snapshotting. I could possibly pause the VM as I am building a second box as a clustered backup so that box would failover and serve the web while the backup is being preformed. I am planning to rsync the vms across the netowrk to keep them in sync. That should eliminate the need to replicate the SQL databases since the vm viruall disk will be replicating all the time via rsync. Do you see any problems with this approach?
__________________
www.windependence.org
Get your Windependence today!
Reply With Quote
Old 8th June 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

We're snapshotting the backups, not the VMs.

We run rsync inside the VMs to sync back to a FreeBSD box using ZFS. Then snapshot the filesystem once all the backups are done. So far, for 6 servers, the snapshots take up just under 1 GB for 1 day of data.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 9th June 2008
windependence's Avatar
windependence windependence is offline
Real Name: Tim
Shell Scout
 
Join Date: May 2008
Location: Phoenix, Arizona
Posts: 116
Default

Quote:
Originally Posted by phoenix View Post
We're snapshotting the backups, not the VMs.

We run rsync inside the VMs to sync back to a FreeBSD box using ZFS. Then snapshot the filesystem once all the backups are done. So far, for 6 servers, the snapshots take up just under 1 GB for 1 day of data.
So I take it snapshots are not useful by themselves for backup. Forgive me for being ignorant but snapshots are not an actual copy of the data? I have always been confused by this. So if I have a 1GB file and take a snapshot of it, can I recreate the file from the snapshot?

-Tim
__________________
www.windependence.org
Get your Windependence today!
Reply With Quote
Old 10th June 2008
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 696
Default

How snapshots are implemented depends on which filesystem you are using.

For UFS, if you take a snapshot of a filesystem, it creates a hidden .snaphot directory in the root of the filesystem. Anytime a file in that filesystem is edited, a copy of the original is put in the .snapshot directory. I haven't played with UFS snapshots, so not sure how to access files in snapshots.

For ZFS, if you take a snapshot of a filesystem, you give it a name (usually based on the date, to make it easier to sort/view the list of snapshots). Since ZFS is copy-on-write (it writes files out to new locations on the disk, then updates the directory pointer to point at the new file, then later sets the previously used blocks to unallocated), when a file is updated, the original's directory pointer is added to the snapshot, and the new file is written to a new location on disk. You can either mount the snapshot by name to access the files, or you can navigate through the .zfs/ directory tree hierarchy. It's quite powerful stuff.

How we have things setup is like so:
/storage is a massive zpool covering 10 500 GB drives, configured as raidz2
/storage/remote-servers is a zfs filesystem with gzip-9 compression enabled
/storage/remote-servers/<servername> is a zfs filesystem, one for each server

Each night, the backup server runs a script that connects it to multiple servers via ssh, runs rsync against that server and the corresponding <servername> directory, then takes a snapshot of the /storage/remote-servers filesystem once the run is complete. The snapshots are named using the date in YYYYMMDD format.

This way, we have daily archives of all the files on each of the servers. If we need individual files to restore, we just mount the snapshot for the day in question, and go into the <servername> directory, find the file, and copy it to wherever it needs to be.

To restore a server, we boot it with a Knoppix or Kanotix CD, partition the drives, mount the drives/partitions, then run rsync off the backup server to put everything back to where it was. Takes about 1 hour to restore a server (3x 400 GB drives in RAID5, ~250 GB of data).

Each server uses about 1 GB of data in each daily snapshot (as only changed files are included).

Our previous method was to use DAR to do a full backup on Sunday, and incremental backups Mon-Sat, and rotate into a lastweek/ directory on Sat, to keep two weeks of backups. Restoring individual files took up to 3 hours, and restoring a full server was a full-day affair. Plus, each server took up about 80 GB on Sun, and up to 10 GB for each day of the week, times two for the previous weeks' archives.
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 13th June 2008
windependence's Avatar
windependence windependence is offline
Real Name: Tim
Shell Scout
 
Join Date: May 2008
Location: Phoenix, Arizona
Posts: 116
Default

Thanks for the detailed explanation. Since all our stuff is UFS, I'm gonna have to RTFM a bit to figure out how all this works on that file system. I am too scared to use ZFS as it is fairly new and not quite mature yet as I understand it.

-Tim
__________________
www.windependence.org
Get your Windependence today!
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Remote backup for clueless n00bs Albright FreeBSD General 7 18th May 2009 06:44 PM
backup freeBSD 7.0 using Backup Exec ccc FreeBSD General 2 25th April 2009 09:23 PM
Remote Installation of *BSD JMJ_coder Other BSD and UNIX/UNIX-like 3 21st August 2008 02:19 PM
Utility to make FreeBSD custom ISO mtx FreeBSD General 5 9th June 2008 11:13 PM
opera: disable inbuild torrent utility ephemera FreeBSD Ports and Packages 11 17th May 2008 12:26 AM


All times are GMT. The time now is 07:45 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick