Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Thread Tools Display Modes
  #1   (View Single Post)  
Old 2nd May 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
Join Date: May 2008
Location: USA
Posts: 6,980
Default Cloud service and disaster recovery

I used to have a backup infrastructure: backup management servers. Off-site media storage. Media management tools. Rescue systems for bare-metal restores.

These days, my backups are in the cloud. I use a commercial service, Tarsnap, developed and operated by Colin Percival, Security Officer Emeritus of the FreeBSD project.
Briefly: Tarsnap's service deduplicates, compresses, and encrypts data on the system being backed up. Data is stored in multiple Amazon S3 locations. Keys are managed by the customer.
It took me several months to convince myself to leave my prior backup systems behind. Before making the complete plunge to "cloud-only" backup, I practised restoring systems from scratch -- "bare-metal" restoration. I practised several times - with both servers and worksttions. It's a little different than restoring a complete system from a full backup. Because I don't take full backups any longer. I don't back up the OS, or any installed applications. So restoring a dump(8) over the network or from media has ended.

Backups are currently performed daily. I back up configuration data, application data (but not the application), and user data. For OpenBSD, that's effectively /etc, /root, /var, and /home.

I'm glad I practised, because earlier this week, I needed to follow the process for real. One of my firewalls lost its only storage media: a compact flash drive. Restoration on a new drive was fairly straightforward procedurally.
  1. Install the OS on the firewall's new, replacement drive.
  2. Install the client application and the firewall's emergency subkey.
  3. Restore configuration data and full key set.
  4. Install application set.
  5. Restore the most recent backup archives.
  6. Rebuild the backup deduplication cache
  7. Restart
1. Installing the OS from scratch will require replicating the original storage layout. On OpenBSD, that means disklabel partitions. If they were only stored in the cloud ... you'll need to restore this information first. Which means you will either use a second platform for this service, or, installing twice. I prefer having this information, like keys, already shared among systems.

2. One operational decision I had to make was how I was going to manage my keys. If keys are lost, backups are unrecoverable. Each of my systems has a master key which can backup, restore, and delete individual archives without a passphrase. And each of my systems has a subordinate key, a "subkey,", which has the same authority but requires a passphrase, for use in disaster recovery. Electronic copies of this subkey are shared among all my systems, and printed copies are stored off-site. In the event I need to restore from off-site, the paper keys will be scanned and OCRed to recover these subkeys.

3. The subkey is used to restore /root, which contains not only the master key which needs no passphrase, it contains my list of applications -- on OpenBSD, that's a list of manually installed packages.

4. I install the application set. This builds unused configuration files in /etc and empty data structures in /var, which will be overlayed. The needed software and dependencies are installed in /usr/local.

5. I now drop to single-user mode, so no applications are running, and restore the remaining archives: /etc, /var, and /home. In practice, I was restoring /etc and /var to temporary locations, but in single-user mode, it is generally safe to do so ... as long as any newly required provisions -- such as DUIDs for disk drives in /etc/fstab -- are updated before rebooting. In my case, revising /etc/fstab is the only provisioning change I need to manage.

6. I skipped this step in my practice, then discovered it in production this week. Tarsnap has a local cache for deduplication management, which must be rebuilt before new backups can be run. Oops. It's just one command: # tarsnap --fsck

7. If /etc was managed properly (either restored to a temporary location first, or, managed in single-user mode) and re-provisioned properly, rebooting restarts the fully recovered system.


Lessons learned:
  • Data loss is still a consideration. I have a few applications on this particular firewall that really need more frequent backups than once per day. I may increase the cadence to 2x or 4x per day -- the cost for increased frequency is not a factor for this particular cloud-based service, since you pay for bytes transferred and stored, after deduplication.
  • Test all your server and desktops, individually. Some things on this platform were forgotten and not backed up. I have one application which was not packaged as a port and was unlisted in the application list. The application was restored to /home, but not deployed in /usr/local where a script in /etc expected to find it. And, I also neglected to back up the /tftpboot directory used to support PXE boot on the local network. I had to recreate this directory and manually restore kernels and bootloader.
  • Michael W. Lucas's Tarsnap Mastery book helped me consider the most critical items: key management, backup cadence, what data to back up, and what to avoid backing up, and supporting tools. I'm using Another Calendar-based Tarsnap Script (ACTS), which I have ported to OpenBSD.

Last edited by jggimi; 2nd May 2015 at 01:38 AM.
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
The art of recovery -- grumble and moan jggimi General Hardware 12 25th February 2012 08:32 PM
Disaster recovery best practices RandomSF FreeBSD General 8 7th December 2010 06:41 AM
Disaster Recovery scheme using dump/restore commands. FBSD Guides 4 26th February 2010 03:34 PM
Backup strategies and disaster planning sherekhan OpenBSD General 16 2nd June 2009 10:30 PM
data recovery. LateNiteTV FreeBSD General 8 29th August 2008 08:11 PM

All times are GMT. The time now is 04:14 PM.

Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick