DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 13th May 2009
sherekhan sherekhan is offline
New User
 
Join Date: May 2009
Location: Norway
Posts: 7
Thanked 0 Times in 0 Posts
Default Backup strategies and disaster planning

I have a home made script run by daily.local which every night dumps all file systems to a dedicated local hard drive using dump, compresses them with gzip and rotates them based on date. For the sake of preserving history, I like to keep the most recent backups for a while before deleting them to make room for more backups. Dump-level is determined by date; 0 once per month, 1 once per week, 2 all other days.

I have also tested that the backups work by doing a test restore on a second computer. I booted using an install CD, created file systems and boot loader, restored the dumps, booted, and everything worked as expected.

I also have the option of running the backup to an NFS server, at which point I will compress the dump on the fly through a pipe rather than after dumping. Later I also may consider using external USB devices, like a flash drive or external hard drive which can be swapped at regular intervals and stored off site.

I may also want to use something other than gzip for compression: lzop (for speed, especially when dumping over the network to NFS) or lzma (for high compression ratio, if dumping to flash devices with limited space).

I am dumping live file systems. I have seen warnings against this some places, but I have not really understood excactly why. If OpenBSD supported file system snapshots I would use these, but I guess that's not really possible. And going single mode every night is not something I want.

The computer in question mostly runs a personal mail server.

A few questions:

1) I would like to have the option of encrypting the dumps, especially if I am to use external devices for backups. What is my best option? I notice both aescrypt and ccrypt in ports uses Rijndael/AES, what is the difference between the two? Should I use any of them, or something else?

2) If encrypting my backups, and/or if I compress them with something else than gzip, I guess I can no longer expect to be able to access the dumps directly from the tools available from a basic install CD. What are my options here? Is there a good OpenBSD based live CD that includes such tools? Should I try to build my own boot CD or USB drive? Or should I be prepared to install a bare OpenBSD system and the needed tools before restoring my dumps when disaster strikes?

3) Any other comments, or other problems I have not foreseen?
Reply With Quote
  #2   (View Single Post)  
Old 13th May 2009
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,052
Thanked 118 Times in 93 Posts
Default

RAID1/RAID5 for storage of these dumps, if your disk fail, then you lose all backups, some external case/NAS for 2 disks or maybe just RAID1 on NFS server.

Maybe also generate your own LiveCD/ISO image that you will be able to burn, boot and bring back latest backup (system dumps in this ISO also).

About restoring, create scripts that will speed up and automate restoration process.
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
  #3   (View Single Post)  
Old 13th May 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

"Offsite" backup is critical. You could easily lose data on a single hard drive, and vermaden mentions RAID protection ... but that would not prevent data loss in the event of a loss of the RAID array -- fire/flood/earthquake/EMP/theft/you-name-it.

You might consider daily incrementals -- so that you could restore to "Wednesday" if necessary, on Saturday. Example: Monthly 0, Weekly 1, Dailies use 2-7.

On the fly compression of your backups will save disk space, locally.

You need not necessarily require a 3rd party package for encryption, depending on need. If you want to encrypt data for transit on an insecure network to secure storage, ssh(1) works fine. If you need to store the data in encrypted form, openssl(1) is also available.

Note that neither openssl nor ssh are directly available from the ramdisk kernel, so you might need alternative procedures for disaster recovery. NFS isn't available from the ramdisk kernel, either.

There are lots of ways to manage DR without live media; one common method is to install OpenBSD on a USB stick, and boot that instead of the ramdisk kernel. Another is to use the ramdisk kernel to restore / and /usr using "traditional" methods and media, then chroot into the restored environment to run nc, ssh, or other tools you might need to restore encrypted or network connected data. Yet another option is to do a re-install of the OS, then restore /usr/local, /var, /home, and /etc once you've booted the system. Live media is an option, also. See the FAQ in my link, below.

I happen to use dump with piped compression and transmission via ssh to my backup server. The data is stored offline on DVD+RW, using sysutils/shunt on the backup server.
Reply With Quote
  #4   (View Single Post)  
Old 13th May 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

Some follow up, regarding your question of "live" system backups. The key here are knowledge of the application(s) and the impact of any type of data loss.

When you are backing up *filesystems* that can change during the backup, you risk loss of data during the backup due to filesystem changes -- additions, deletions, and updates. You must determine if such loss is acceptable or not. It may be. Example: I back up /var on a running system. Even a fairly idle system will have content being added to /var/log regularly. So, I may lose log information in today's backup; but those logs will be available for me tomorrow. In the event I need to recover /var, I may lose /var/log information that was never captured in a backup. But .... even if I do a clean and high-integrity backup by stopping applications and dropping into single user mode at 01:00, I will still lose data if I have a filesystem melt-down at 09:00 later that morning.

There are any number of applications which allow for high-integrity backups of running systems: modern DBMS systems are a good example. My backup of /var, for example, begins with backing up a PostgreSQL DBMS into a flat file, first, prior to running dump(8).

RAID systems can provide redundancy and prevent individual storage device failure from causing data loss; and they can keep application availability going. But they cannot prevent data loss. I run RAID systems, yet I've had to do any number of recoveries unrelated to hardware failure. Application bugs can do harm to data. (I had one of those on Monday this week.) "Finger fumbles" are also a common reason for recovery. RAID can't help if the data on the array is wrong.

Data loss impact must be understood. I've had customers who, for some applications, would flush cache to disk and take "snapshots" of file systems hourly, because they could manually recover only an hour's worth of lost information. I've had other customers with applications that could not withstand any data loss; they required synchronous remote replication of their data, and the ability to seamlessly switch applications from one computing center to another in the event of any infrastructure problems. And, I've had customers for whom "last week's backup" was good enough. The tradeoff is availability and cost.

I even had one customer who's offsite backup strategy -- for a multibillion (US) dollar development project -- was to have a Unix admin toss a set of tapes into the back of his personal vehicle and take them home on Fridays after work. A publicly held, very large US company, too. I'm glad I wasn't their auditor.
Reply With Quote
  #5   (View Single Post)  
Old 15th May 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

Here's an example of two duplicate servers, one backing up for the other, where all data was lost (destroyed intentionally, actually).

This was reported on Slashdot today; here's a link to the actual article referenced by slashdot:

http://news.bbc.co.uk/2/hi/technology/8049780.stm

(Some of *my* data was lost, as I was a member of this group about six or eight years ago. Inactive data, but data nonetheless.)
Reply With Quote
  #6   (View Single Post)  
Old 15th May 2009
revzalot's Avatar
revzalot revzalot is offline
Shell Scout
 
Join Date: May 2008
Posts: 123
Thanked 1 Time in 1 Post
Default

Excellent topic. How about hourly snapshots of your system stored on an external server out of your dmz servers?
Reply With Quote
  #7   (View Single Post)  
Old 16th May 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

5-10 years ago, when I was consulting in this area of infrastructure, the best practice was to have remote replication distant enough to be on a separate power grid -- and for my customers in earthquake susceptible areas, distant enough to be on a separate tectonic plate, as well.
At the time, there were no specific business continuity standards requiring compliance. As I have been out of this part of the industry for some years, I do not know if standards were developed. More recently I have been involved in U.S. regulatory compliance considerations, and to the best of my knowledge there are no specific remote replication standards to be met in the regulatory acts I've dealt with since 2002. This list includes both general use and industry specific federal regulations over data management such as ITAR, Sarbanes-Oxley, TREAD, and HIPAA.
The remote replication technologies I had expertise in were usually configured with private point-to-point telecom connections; alternatively, we could allow customers to implement a dedicated single-purpose VPN. The latter is not as secure as a private connection for data replication, of course, but by limiting it to single-purpose the possible attack vectors are significantly easier to control.

EDIT: I should point out that one method for managing data latency with remote replication, if done continuously, is to have two tiers: a relatively local facility -- typically on-campus but off-site, for synchronous updates; and a second tier at distance which is replicated from the first tier asynchronously.

Last edited by jggimi; 16th May 2009 at 02:59 AM.
Reply With Quote
  #8   (View Single Post)  
Old 18th May 2009
sherekhan sherekhan is offline
New User
 
Join Date: May 2009
Location: Norway
Posts: 7
Thanked 0 Times in 0 Posts
Default

Thanks for the replies so far, especially jggimis insight

Regarding RAID: I do have a couple of computers in the house used for file servers, printer sharing, media streaming etc, but they are currently not powered 24/7. That means they are not suitable for daily backups at the moment, although that may change. But they _do_ have RAID. It`s a messy combination of RAID1, RAID5 and LVM using disks of different sizes, and the I/O-performance sucks, but the redundancy works. I have parity or mirrors for all data at all times, including GRUB and /boot. The servers are running Debian and Ubuntu atm, but I am considering looking into something with ZFS, maybe FreeBSD or Nexenta. I am also considering upgrading my mail server, and setting up some kind of disk mirroring on a new OpenBSD installation. RAIDframe or ccd? RAIDframe looks like the best and most reliable option from what I read, but it also seems to require a lot of planning and work to set up. Especially if I need a custom kernel every time I upgrade OpenBSD on the box.

Regarding boot media for recovery: Failing to find anything that has already been baked that fulfills my need, I feel that I am left with two options: 1) Make my own live CD with everything I need, and 2) use something that is _not_ OpenBSD. Following jggimis links there are tutorials and hints on how to make your own live CD, and I may end up doing this no matter what just for the experience, but I am still curious about option 2. I am currently typing this post from Frenzy, which is a live CD based on FreeBSD. It seems to have the tools I need (gzip/lzop/lzma, openssl, NFS, network support, man pages and even a basic desktop environment with Opera, plus a wonderful selection of forensics and system tools). What I am not certain about is file system compatibility and bootloader.

I have mounted a partition created with OpenBSD from Frenzy, and it seems to work just fine, including read/write, but can I be certain that a UFS file system created with FreeBSD will work 100% with OpenBSD? What about support for ccd or RAIDframe devices? FreeBSD has ccd, but I have no idea if the implementation and the tools are compatible with OpenBSD. And a Google search on "freebsd raidframe" suggests that RAIDframe on FreeBSD is dead.

Then there is the boot loader. Is it possible to restore this without "proper" OpenBSD boot media? Is it just a matter of using dd to make a backup of MBR, and restoring MBR with dd when required, or is there more to it? Or can I simply install the FreeBSD boot loader (or another boot loader for that matter) and use that to boot OpenBSD?
Reply With Quote
  #9   (View Single Post)  
Old 19th May 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

Quote:
Originally Posted by sherekhan View Post
RAIDframe or ccd?
For software RAID, use RAIDframe, for now. Its eventual replacement will be softraid(4), but today softraid does not have any way to recover from a storage device loss, the admin must back up the degraded array and recreate it, which can require many hours of down time. Plus, the only RAID available is a variant of RAID 1. (Crypto is another capability of softraid, but that doesn't apply to this discussion.)

The ccd facility is simpler to set up, but it cannot guarantee integrity after a failure, as it does not do parity checking on restart; it is my understanding that depending on the type of failure, that even mirroring (RAID 1) cannot assure integrity.
Quote:
it also seems to require a lot of planning and work to set up.
Not a lot, but yes, some planning is required. Running it in a test bed prior to making the switch gives one a great deal of confidence, however.
Quote:
Especially if I need a custom kernel every time I upgrade OpenBSD on the box.
No, that's not too all that hard either. The basic steps are to update your source tree, build the new kernel, boot it, then manually upgrade by un-tarring all but the *etcNN.tgz file sets per FAQ 4.10, then running sysmerge(8). In addition, each release's Upgrade Guide has a section on upgrading without the install kernel that you can use, with a little thought, as guidance.

In my case, I run -current on my RAIDframe system, so I "upgrade" by building the kernel and userland, then running sysmerge.
Quote:
I feel that I am left with two options: 1) Make my own live CD with everything I need, and 2) use something that is _not_ OpenBSD.
There are many other options. Assuming you can't restore directly from the ramdisk kernel, there are still other options. Here's just a few:
  • full system on USB drive/stick; directly booted or booted from install media
  • full system on LAN
  • boot ramdisk kernel, install full system on temporary storage
  • boot ramdisk kernel, dd restore of functioning system
Quote:
Following jggimis links there are tutorials and hints on how to make your own live CD, and I may end up doing this no matter what just for the experience
Making one's own is doable. I've been chatting with Stephan Rickauer (bsdanywhere.org) about jointly developing a "build-your-own-live-media" port/package for OpenBSD. We may do so in the next few months, if we can find the time.
Quote:
What I am not certain about is file system compatibility and bootloader.
There are significant differences between the filesystems -- there has been 14 years of divergent development between these systems. Data integrity/recoverability cannot be assured if you are doing filesystem restores. The same might be said of dump(8), as well.

The bootloaders are entirely different, as well. Even if you can possibly restore an OpenBSD filesystem from an OpenBSD dump from a FreeBSD system, something I doubt, you will need to install boot blocks (PBR) after the restore, and for that, you will need a running OpenBSD system. See installboot(8).
Reply With Quote
Old 2nd June 2009
mururoa mururoa is offline
Fdisk Soldier
 
Join Date: May 2008
Posts: 46
Thanked 0 Times in 0 Posts
Default

Quote:
Originally Posted by jggimi View Post
I happen to use dump with piped compression and transmission via ssh to my backup server. The data is stored offline on DVD+RW, using sysutils/shunt on the backup server.
Do you still have the scripts ?
I'm interested using compressed backup, so far (personnal) I use distant rsync (ssh). It works well but I need to have about same size HD at destination since I cant use compression at destination.
Reply With Quote
Old 2nd June 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

I have a script, yes, but the meat of it is nothing more than something like:
Code:
dump -$level -auf - $filesystem | gzip | \
ssh user@server "cat > /path/to/$file.$level.dump.gz"
I'll send it to you when I'm not behind a customer's firewall.
Reply With Quote
Old 2nd June 2009
revzalot's Avatar
revzalot revzalot is offline
Shell Scout
 
Join Date: May 2008
Posts: 123
Thanked 1 Time in 1 Post
Default

Quote:
Originally Posted by jggimi View Post
I have a script, yes, but the meat of it is nothing more than something like:
Code:
dump -$level -auf - $filesystem | gzip | \
ssh user@server "cat > /path/to/$file.$level.dump.gz"
I'll send it to you when I'm not behind a customer's firewall.
Can you send me that script, too?

I just want to add is always do a checksum (cksum) on all your backups for file authentication.
Reply With Quote
Old 2nd June 2009
phoenix's Avatar
phoenix phoenix is offline
Risen from the ashes
 
Join Date: May 2008
Posts: 699
Thanked 90 Times in 81 Posts
Default

Quote:
Originally Posted by mururoa View Post
Do you still have the scripts ?
I'm interested using compressed backup, so far (personnal) I use distant rsync (ssh). It works well but I need to have about same size HD at destination since I cant use compression at destination.
Use a compressed filesystem (like ZFS), then you don't have to worry about file compression. And you don't need the exact same size harddrive to use rsync, just one that is at least the size of your backed up data (of course, bigger is always better).
__________________
Freddie

Help for FreeBSD: Handbook, FAQ, man pages, mailing lists.
Reply With Quote
Old 2nd June 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default

Wrong OS, Phoenix.
Reply With Quote
Old 2nd June 2009
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,154
Thanked 182 Times in 149 Posts
Default

OpenBSD has no support for ZFS, Vermaden
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Old 2nd June 2009
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 3,710
Thanked 214 Times in 189 Posts
Default Example from one of my laptops

SSH Authentication is public key, without passphrase; private key resides on a softraid encrypted partition.
Code:
#!/bin/sh
#
# This script is executed via the /etc/daily infrastructure; calls
# are made from /etc/daily.local, /etc/weekly.local, and /etc/monthly.local
#
# All filesystems back up at the same dump level:

# Monthly 0-level, 1st day of month.
# Weekly  1-level, every Saturday
# Monthly and weekly are manually stored on dvd-rw
# Daily: may be stored optically, but only ad hoc.
# 
# 8 and 9 are reserved for ad-hoc dump use.
#
############################################
#
# $1  = "daily" | "weekly" | "monthly" | "8" | "9"
#
# "8" or "9" are from manual operation
############################################
level=99
if [ $1 = daily ];  then
    case `date | awk '{print $1}'` in
        Sun)
            level=3
            ;;
        Mon)
            level=2
            ;;
        Tue)
            level=5
            ;;
        Wed)
            level=4
            ;;
        Thu)
            level=7
            ;;
        Fri)
            level=6
            ;;
    esac
elif [ $1 = monthly ]; then
    level=0
elif [ $1 = weekly ]; then
    level=1
fi
if [ "$level" != "99" ]; then
    dump -$level -h 0 -auf - / | gzip -9 | \
        ssh bkup@server "umask 177 ; cat > lap-two/root.$level.gz"
    dump -$level -h 0 -auf - /usr | gzip -9 | \
        ssh bkup@server "umask 177 ; cat > lap-two/usr.$level.gz"
    dump -$level -h 0 -auf - /var | gzip -9 | \
        ssh bkup@server "umask 177 ; cat > lap-two/var.$level.gz"
   dump -$level -auf - /home/ | gzip -9 | \
        ssh bkup@server "umask 177 ; cat > lap-two/home.$level.gz"
   dump -$level -auf - /home/crypt | gzip -9 | \
        ssh bkup@server "umask 177 ; cat > lap-two/crypt.$level.gz"
fi
Reply With Quote
Old 2nd June 2009
vermaden's Avatar
vermaden vermaden is offline
Administrator
 
Join Date: Apr 2008
Location: pl_PL.lodz
Posts: 1,052
Thanked 118 Times in 93 Posts
Default

Quote:
Originally Posted by J65nko View Post
OpenBSD has no support for ZFS, Vermaden
"Something never never changes, Something never will"

Like good old days I haven't checked the category ...
__________________
religions, worst damnation of mankind
"If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds

Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”.
vermaden's: links resources deviantart spreadbsd
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Disaster recovery best practices RandomSF FreeBSD General 8 7th December 2010 06:41 AM
backup freeBSD 7.0 using Backup Exec ccc FreeBSD General 2 25th April 2009 09:23 PM
Backup strategies Oko General software and network 4 2nd February 2009 01:32 AM
Backup script(s)? giddyupman General software and network 2 3rd January 2009 02:06 PM
How to backup my system PatrickBaer FreeBSD General 4 16th July 2008 08:12 PM


All times are GMT. The time now is 02:54 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick