|
OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below. |
|
Thread Tools | Display Modes |
|
|||
Backup strategies and disaster planning
I have a home made script run by daily.local which every night dumps all file systems to a dedicated local hard drive using dump, compresses them with gzip and rotates them based on date. For the sake of preserving history, I like to keep the most recent backups for a while before deleting them to make room for more backups. Dump-level is determined by date; 0 once per month, 1 once per week, 2 all other days.
I have also tested that the backups work by doing a test restore on a second computer. I booted using an install CD, created file systems and boot loader, restored the dumps, booted, and everything worked as expected. I also have the option of running the backup to an NFS server, at which point I will compress the dump on the fly through a pipe rather than after dumping. Later I also may consider using external USB devices, like a flash drive or external hard drive which can be swapped at regular intervals and stored off site. I may also want to use something other than gzip for compression: lzop (for speed, especially when dumping over the network to NFS) or lzma (for high compression ratio, if dumping to flash devices with limited space). I am dumping live file systems. I have seen warnings against this some places, but I have not really understood excactly why. If OpenBSD supported file system snapshots I would use these, but I guess that's not really possible. And going single mode every night is not something I want. The computer in question mostly runs a personal mail server. A few questions: 1) I would like to have the option of encrypting the dumps, especially if I am to use external devices for backups. What is my best option? I notice both aescrypt and ccrypt in ports uses Rijndael/AES, what is the difference between the two? Should I use any of them, or something else? 2) If encrypting my backups, and/or if I compress them with something else than gzip, I guess I can no longer expect to be able to access the dumps directly from the tools available from a basic install CD. What are my options here? Is there a good OpenBSD based live CD that includes such tools? Should I try to build my own boot CD or USB drive? Or should I be prepared to install a bare OpenBSD system and the needed tools before restoring my dumps when disaster strikes? 3) Any other comments, or other problems I have not foreseen? |
|
||||
RAID1/RAID5 for storage of these dumps, if your disk fail, then you lose all backups, some external case/NAS for 2 disks or maybe just RAID1 on NFS server.
Maybe also generate your own LiveCD/ISO image that you will be able to burn, boot and bring back latest backup (system dumps in this ISO also). About restoring, create scripts that will speed up and automate restoration process.
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
||||
Some follow up, regarding your question of "live" system backups. The key here are knowledge of the application(s) and the impact of any type of data loss.
When you are backing up *filesystems* that can change during the backup, you risk loss of data during the backup due to filesystem changes -- additions, deletions, and updates. You must determine if such loss is acceptable or not. It may be. Example: I back up /var on a running system. Even a fairly idle system will have content being added to /var/log regularly. So, I may lose log information in today's backup; but those logs will be available for me tomorrow. In the event I need to recover /var, I may lose /var/log information that was never captured in a backup. But .... even if I do a clean and high-integrity backup by stopping applications and dropping into single user mode at 01:00, I will still lose data if I have a filesystem melt-down at 09:00 later that morning. There are any number of applications which allow for high-integrity backups of running systems: modern DBMS systems are a good example. My backup of /var, for example, begins with backing up a PostgreSQL DBMS into a flat file, first, prior to running dump(8). RAID systems can provide redundancy and prevent individual storage device failure from causing data loss; and they can keep application availability going. But they cannot prevent data loss. I run RAID systems, yet I've had to do any number of recoveries unrelated to hardware failure. Application bugs can do harm to data. (I had one of those on Monday this week.) "Finger fumbles" are also a common reason for recovery. RAID can't help if the data on the array is wrong. Data loss impact must be understood. I've had customers who, for some applications, would flush cache to disk and take "snapshots" of file systems hourly, because they could manually recover only an hour's worth of lost information. I've had other customers with applications that could not withstand any data loss; they required synchronous remote replication of their data, and the ability to seamlessly switch applications from one computing center to another in the event of any infrastructure problems. And, I've had customers for whom "last week's backup" was good enough. The tradeoff is availability and cost. I even had one customer who's offsite backup strategy -- for a multibillion (US) dollar development project -- was to have a Unix admin toss a set of tapes into the back of his personal vehicle and take them home on Fridays after work. A publicly held, very large US company, too. I'm glad I wasn't their auditor. |
|
||||
Here's an example of two duplicate servers, one backing up for the other, where all data was lost (destroyed intentionally, actually).
This was reported on Slashdot today; here's a link to the actual article referenced by slashdot: http://news.bbc.co.uk/2/hi/technology/8049780.stm (Some of *my* data was lost, as I was a member of this group about six or eight years ago. Inactive data, but data nonetheless.) |
|
||||
5-10 years ago, when I was consulting in this area of infrastructure, the best practice was to have remote replication distant enough to be on a separate power grid -- and for my customers in earthquake susceptible areas, distant enough to be on a separate tectonic plate, as well.
At the time, there were no specific business continuity standards requiring compliance. As I have been out of this part of the industry for some years, I do not know if standards were developed. More recently I have been involved in U.S. regulatory compliance considerations, and to the best of my knowledge there are no specific remote replication standards to be met in the regulatory acts I've dealt with since 2002. This list includes both general use and industry specific federal regulations over data management such as ITAR, Sarbanes-Oxley, TREAD, and HIPAA.The remote replication technologies I had expertise in were usually configured with private point-to-point telecom connections; alternatively, we could allow customers to implement a dedicated single-purpose VPN. The latter is not as secure as a private connection for data replication, of course, but by limiting it to single-purpose the possible attack vectors are significantly easier to control. EDIT: I should point out that one method for managing data latency with remote replication, if done continuously, is to have two tiers: a relatively local facility -- typically on-campus but off-site, for synchronous updates; and a second tier at distance which is replicated from the first tier asynchronously. Last edited by jggimi; 16th May 2009 at 02:59 AM. |
|
|||
Thanks for the replies so far, especially jggimis insight
Regarding RAID: I do have a couple of computers in the house used for file servers, printer sharing, media streaming etc, but they are currently not powered 24/7. That means they are not suitable for daily backups at the moment, although that may change. But they _do_ have RAID. It`s a messy combination of RAID1, RAID5 and LVM using disks of different sizes, and the I/O-performance sucks, but the redundancy works. I have parity or mirrors for all data at all times, including GRUB and /boot. The servers are running Debian and Ubuntu atm, but I am considering looking into something with ZFS, maybe FreeBSD or Nexenta. I am also considering upgrading my mail server, and setting up some kind of disk mirroring on a new OpenBSD installation. RAIDframe or ccd? RAIDframe looks like the best and most reliable option from what I read, but it also seems to require a lot of planning and work to set up. Especially if I need a custom kernel every time I upgrade OpenBSD on the box. Regarding boot media for recovery: Failing to find anything that has already been baked that fulfills my need, I feel that I am left with two options: 1) Make my own live CD with everything I need, and 2) use something that is _not_ OpenBSD. Following jggimis links there are tutorials and hints on how to make your own live CD, and I may end up doing this no matter what just for the experience, but I am still curious about option 2. I am currently typing this post from Frenzy, which is a live CD based on FreeBSD. It seems to have the tools I need (gzip/lzop/lzma, openssl, NFS, network support, man pages and even a basic desktop environment with Opera, plus a wonderful selection of forensics and system tools). What I am not certain about is file system compatibility and bootloader. I have mounted a partition created with OpenBSD from Frenzy, and it seems to work just fine, including read/write, but can I be certain that a UFS file system created with FreeBSD will work 100% with OpenBSD? What about support for ccd or RAIDframe devices? FreeBSD has ccd, but I have no idea if the implementation and the tools are compatible with OpenBSD. And a Google search on "freebsd raidframe" suggests that RAIDframe on FreeBSD is dead. Then there is the boot loader. Is it possible to restore this without "proper" OpenBSD boot media? Is it just a matter of using dd to make a backup of MBR, and restoring MBR with dd when required, or is there more to it? Or can I simply install the FreeBSD boot loader (or another boot loader for that matter) and use that to boot OpenBSD? |
|
|||||
For software RAID, use RAIDframe, for now. Its eventual replacement will be softraid(4), but today softraid does not have any way to recover from a storage device loss, the admin must back up the degraded array and recreate it, which can require many hours of down time. Plus, the only RAID available is a variant of RAID 1. (Crypto is another capability of softraid, but that doesn't apply to this discussion.)
The ccd facility is simpler to set up, but it cannot guarantee integrity after a failure, as it does not do parity checking on restart; it is my understanding that depending on the type of failure, that even mirroring (RAID 1) cannot assure integrity. Quote:
Quote:
In my case, I run -current on my RAIDframe system, so I "upgrade" by building the kernel and userland, then running sysmerge. Quote:
Quote:
Quote:
The bootloaders are entirely different, as well. Even if you can possibly restore an OpenBSD filesystem from an OpenBSD dump from a FreeBSD system, something I doubt, you will need to install boot blocks (PBR) after the restore, and for that, you will need a running OpenBSD system. See installboot(8). |
|
|||
Quote:
I'm interested using compressed backup, so far (personnal) I use distant rsync (ssh). It works well but I need to have about same size HD at destination since I cant use compression at destination. |
|
||||
I have a script, yes, but the meat of it is nothing more than something like:
Code:
dump -$level -auf - $filesystem | gzip | \ ssh user@server "cat > /path/to/$file.$level.dump.gz" |
|
||||
Quote:
I just want to add is always do a checksum (cksum) on all your backups for file authentication. |
|
||||
Use a compressed filesystem (like ZFS), then you don't have to worry about file compression. And you don't need the exact same size harddrive to use rsync, just one that is at least the size of your backed up data (of course, bigger is always better).
|
|
|||
OpenBSD has no support for ZFS, Vermaden
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
||||
Example from one of my laptops
SSH Authentication is public key, without passphrase; private key resides on a softraid encrypted partition.
Code:
#!/bin/sh # # This script is executed via the /etc/daily infrastructure; calls # are made from /etc/daily.local, /etc/weekly.local, and /etc/monthly.local # # All filesystems back up at the same dump level: # Monthly 0-level, 1st day of month. # Weekly 1-level, every Saturday # Monthly and weekly are manually stored on dvd-rw # Daily: may be stored optically, but only ad hoc. # # 8 and 9 are reserved for ad-hoc dump use. # ############################################ # # $1 = "daily" | "weekly" | "monthly" | "8" | "9" # # "8" or "9" are from manual operation ############################################ level=99 if [ $1 = daily ]; then case `date | awk '{print $1}'` in Sun) level=3 ;; Mon) level=2 ;; Tue) level=5 ;; Wed) level=4 ;; Thu) level=7 ;; Fri) level=6 ;; esac elif [ $1 = monthly ]; then level=0 elif [ $1 = weekly ]; then level=1 fi if [ "$level" != "99" ]; then dump -$level -h 0 -auf - / | gzip -9 | \ ssh bkup@server "umask 177 ; cat > lap-two/root.$level.gz" dump -$level -h 0 -auf - /usr | gzip -9 | \ ssh bkup@server "umask 177 ; cat > lap-two/usr.$level.gz" dump -$level -h 0 -auf - /var | gzip -9 | \ ssh bkup@server "umask 177 ; cat > lap-two/var.$level.gz" dump -$level -auf - /home/ | gzip -9 | \ ssh bkup@server "umask 177 ; cat > lap-two/home.$level.gz" dump -$level -auf - /home/crypt | gzip -9 | \ ssh bkup@server "umask 177 ; cat > lap-two/crypt.$level.gz" fi |
|
||||
"Something never never changes, Something never will"
Like good old days I haven't checked the category ...
__________________
religions, worst damnation of mankind "If 386BSD had been available when I started on Linux, Linux would probably never had happened." Linus Torvalds Linux is not UNIX! Face it! It is not an insult. It is fact: GNU is a recursive acronym for “GNU's Not UNIX”. vermaden's: links resources deviantart spreadbsd |
|
|||
Quote:
I tried to use your script (made some changes to fit my computer). Manually start it but got an error " [: daily: unexpected operator/operand " if [ "$level" != "99" ]; then seams to be the culprit. Any idea? |
|
||||
Wow. You're asking me if a script I posted a dozen years ago has a transcription error.
I'm currently traveling without a workstation, and have no way to test whether the error is yours or mine. I likely haven't used this script in at least 10 years. I'm a tarsnap user, these days. |
|
|||
The part you have trouble with uses "\" s, or continuation lines. These lines need to be followed immediately with a newline.
My suspicion is that one of those continuation lines is followed by a blank or space. You can check that with the vi ":set list" command: Code:
1 #/bin/sh$ 2 $ 3 level=$1$ 4 $ 5 if [ "$level" != "99" ] ; then$ 6 echo \ $ 7 ^I"Not 99!" $ 8 fi$ 9 $ 10 $ ~ ~ ~ ~ :set list Code:
adriaan@ml310e ~ $ sh test 99 adriaan@ml310e ~ $ sh test 1 test: 7: test: Not 99!: not found adriaan@ml310e ~ $
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Disaster recovery best practices | RandomSF | FreeBSD General | 8 | 7th December 2010 06:41 AM |
backup freeBSD 7.0 using Backup Exec | ccc | FreeBSD General | 2 | 25th April 2009 09:23 PM |
Backup strategies | Oko | General software and network | 4 | 2nd February 2009 01:32 AM |
Backup script(s)? | giddyupman | General software and network | 2 | 3rd January 2009 02:06 PM |
How to backup my system | PatrickBaer | FreeBSD General | 4 | 16th July 2008 08:12 PM |