Cloud service and disaster recovery
I used to have a backup infrastructure: backup management servers. Off-site media storage. Media management tools. Rescue systems for bare-metal restores.
These days, my backups are in the cloud. I use a commercial service, Tarsnap, developed and operated by Colin Percival, Security Officer Emeritus of the FreeBSD project.
Briefly: Tarsnap's service deduplicates, compresses, and encrypts data on the system being backed up. Data is stored in multiple Amazon S3 locations. Keys are managed by the customer.It took me several months to convince myself to leave my prior backup systems behind. Before making the complete plunge to "cloud-only" backup, I practised restoring systems from scratch -- "bare-metal" restoration. I practised several times - with both servers and worksttions. It's a little different than restoring a complete system from a full backup. Because I don't take full backups any longer. I don't back up the OS, or any installed applications. So restoring a dump(8) over the network or from media has ended.
Backups are currently performed daily. I back up configuration data, application data (but not the application), and user data. For OpenBSD, that's effectively /etc, /root, /var, and /home.
I'm glad I practised, because earlier this week, I needed to follow the process for real. One of my firewalls lost its only storage media: a compact flash drive. Restoration on a new drive was fairly straightforward procedurally.
2. One operational decision I had to make was how I was going to manage my keys. If keys are lost, backups are unrecoverable. Each of my systems has a master key which can backup, restore, and delete individual archives without a passphrase. And each of my systems has a subordinate key, a "subkey,", which has the same authority but requires a passphrase, for use in disaster recovery. Electronic copies of this subkey are shared among all my systems, and printed copies are stored off-site. In the event I need to restore from off-site, the paper keys will be scanned and OCRed to recover these subkeys.
3. The subkey is used to restore /root, which contains not only the master key which needs no passphrase, it contains my list of applications -- on OpenBSD, that's a list of manually installed packages.
4. I install the application set. This builds unused configuration files in /etc and empty data structures in /var, which will be overlayed. The needed software and dependencies are installed in /usr/local.
5. I now drop to single-user mode, so no applications are running, and restore the remaining archives: /etc, /var, and /home. In practice, I was restoring /etc and /var to temporary locations, but in single-user mode, it is generally safe to do so ... as long as any newly required provisions -- such as DUIDs for disk drives in /etc/fstab -- are updated before rebooting. In my case, revising /etc/fstab is the only provisioning change I need to manage.
6. I skipped this step in my practice, then discovered it in production this week. Tarsnap has a local cache for deduplication management, which must be rebuilt before new backups can be run. Oops. It's just one command:
7. If /etc was managed properly (either restored to a temporary location first, or, managed in single-user mode) and re-provisioned properly, rebooting restarts the fully recovered system.
Last edited by jggimi; 2nd May 2015 at 01:38 AM.
|Thread||Thread Starter||Forum||Replies||Last Post|
|The art of recovery -- grumble and moan||jggimi||General Hardware||12||25th February 2012 08:32 PM|
|Disaster recovery best practices||RandomSF||FreeBSD General||8||7th December 2010 06:41 AM|
|Disaster Recovery scheme using dump/restore commands.||FBSD||Guides||4||26th February 2010 03:34 PM|
|Backup strategies and disaster planning||sherekhan||OpenBSD General||16||2nd June 2009 10:30 PM|
|data recovery.||LateNiteTV||FreeBSD General||8||29th August 2008 08:11 PM|