DaemonForums  

Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 6th September 2018
shep shep is offline
Real Name: Scott
Arp Constable
 
Join Date: May 2008
Location: Dry and Dusty
Posts: 1,503
Default Backup large directory to usb drive

I've been using 2 x 4.7GB dvd's to perform infrequent /home backups and just obtained a 16GB usb3.0 thumbdrive. I can dedicate the usb3.0 strictly for backups.

I would like to begin encrypting my backups and doing them more frequently and did get some search hits using pipes to gpg or p7zip.


Currently, dvd backups had to be done in 3 steps: 1) generate tar.gz files. 2) produce the *.iso and then burn the iso via growisofs. If I attempted to have growisofs burn the tar.gz directly, it would hang

Given the problems I had with single commands performing sequential operations on large files; I'm looking for advice.

Generate a ~8GB encrypted tar.gz and then copy to the usb drive vs mounting the usb drive and use it as an output target for the tar.gz? Mount the usb drive w/ encryption? Use rsync?

Last edited by shep; 8th September 2018 at 04:06 PM.
Reply With Quote
  #2   (View Single Post)  
Old 7th September 2018
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 1,027
Default

Mounting your USB drive with encryption sounds like a reasonable thing to try. It should be fairly transparent and not get in the way like a pipe. Then you could directly build the tar.gz file there, or use rsync. I use rsync for mirroring things on my LAN, the underlying devices may or may not be encrypted.

I make an annual backup of financial data onto a series of encrypted DVD+RWs, but they're not that big, maybe 250 MB. It sounds like you have ~ 8 GB and want to avoid splitting things over DVDs.

For encrypting individual files, I use openssl. Probably libressl can do the same (though I haven't used it). It seems to work pretty good, but I don't think I've done 8 GB with it. Maybe 500 MB or so.

If using tar for large amounts of data, I've read it's a good idea to compare the tar file with the original source directory. GNU tar can do this.

Last edited by IdOp; 8th September 2018 at 12:19 AM. Reason: MB --> GB
Reply With Quote
  #3   (View Single Post)  
Old 7th September 2018
ip6ix's Avatar
ip6ix ip6ix is offline
Fdisk Soldier
 
Join Date: Sep 2017
Posts: 66
Default

I'd be tempted to use a pipe, and use tar or pax with openssl for the encryption, and write directly to the disk. For example, assuming your USB disk is sd1, to backup the /home partition something like the following should work:

Code:
tar cf - /home | openssl enc -des3 -salt | dd bs=10240 of=/dev/rsd1c
You'll be prompted for a password. To restore, do the reverse:

Code:
dd bs=10240 if=/dev/rsd1c | openssl enc -d -des3 -salt | tar xpfv -
For infrequent backups the above may be enough.

Have you taken a look at sysutils/duplicity?
__________________
dc -e '[q]sa[ln0=aln256%Pln256/snlbx]sb12247225403800449909543746snlbxq'
Reply With Quote
  #4   (View Single Post)  
Old 7th September 2018
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 314
Exclamation

Please remember that thumb drives are notoriously unrealiable! I use them myself and have had some fail right under my nose. Do yourself a favour and don't use it as your only means of backup.
Reply With Quote
  #5   (View Single Post)  
Old 8th September 2018
shep shep is offline
Real Name: Scott
Arp Constable
 
Join Date: May 2008
Location: Dry and Dusty
Posts: 1,503
Default

@ip6ix

I was thinking something on the order of
Code:
tar -czf - * | openssl enc -e -aes256 -out /mnt/secured.tar.gz
with the usb drive mounted at /mnt

I would need to see if libressl has changed the encryptation syntax.

@fvgit

I am concerned about the reliability of flash memory and plan to continue DVD backups biannually - with each OpenBSD release. I was getting concerned that ~6months was a fairly big gap as I become increasingly paperless. The usb3.0 was cheap and more than large enough for all my data, even uncompressed.
Reply With Quote
  #6   (View Single Post)  
Old 8th September 2018
frcc frcc is offline
Don't Worry Be Happy!
 
Join Date: Jul 2011
Location: hot,dry,dusty,rainy,windy,straight winds, tornado,puts the fear of God in you-Texas
Posts: 335
Default

Quote:
Originally Posted by shep View Post
@ip6ix
I am concerned about the reliability of flash memory and plan to continue DVD backups biannually - with each OpenBSD release. I was getting concerned that ~6months was a fairly big gap as I become increasingly paperless. The usb3.0 was cheap and more than large enough for all my data, even uncompressed.
The last article I read concerning reliability of (usb,cd,dvd) backups indicated that cd, dvd was the better choice.
This may have changed.
I "personally" find usb easier to use for temp storage and transport of files when not using ssh . I use cd,dvd's for
archive of important data.

A related comment.....For "me" it is sometimes a little confusing maintaining versions of that backed up data, as
my knowledge level concerning backup schemes is somewhat limited.

Last edited by frcc; 8th September 2018 at 12:30 PM. Reason: add related comment
Reply With Quote
  #7   (View Single Post)  
Old 8th September 2018
ibara ibara is offline
OpenBSD language porter
 
Join Date: Jan 2014
Posts: 783
Default

If archival storage is your goal, then you still can't beat tape. And maybe never will be able to. LTO-4 seems like the sweet spot for home backup users these days; you can get an internal LTO-4 drive for right around $100 and new tapes go for about $15-$20 each. LTO-5 is also viable if you want to pay a little bit more for the initial startup (internal drives seem to go for around $250). The nice thing about an LTO-4 tape would be that you could store 50-100 16GB backups onto a single tape (depending on compression used on the tape), which is over 25 years of backups if you do a biannual snapshot! You could store even more on an LTO-5 tape. So with just two tapes, you could fulfill the old adage of having on-site and off-site backups basically forever. This is the route I would go if replacing CDROM backup media.

I am decidedly less cynical about the reliability of USB drives, but fvgit and frcc are right to note the unreliability of USB drives. The advice here would be you get what you pay for. And even then, a good USB drive can die on you unexpectedly.
Reply With Quote
  #8   (View Single Post)  
Old 8th September 2018
fvgit's Avatar
fvgit fvgit is offline
Spikes in tights
 
Join Date: May 2016
Location: perl -MMIME::Base64 -le 'print decode_base64("U2hlcndvb2QgRm9yZXN0")'
Posts: 314
Default

Quote:
Originally Posted by ibara View Post
If archival storage is your goal, then you still can't beat tape. And maybe never will be able to. LTO-4 seems like the sweet spot for home backup users these days; you can get an internal LTO-4 drive for right around $100 and new tapes go for about $15-$20 each. LTO-5 is also viable if you want to pay a little bit more for the initial startup (internal drives seem to go for around $250). The nice thing about an LTO-4 tape would be that you could store 50-100 16GB backups onto a single tape (depending on compression used on the tape), which is over 25 years of backups if you do a biannual snapshot! You could store even more on an LTO-5 tape. So with just two tapes, you could fulfill the old adage of having on-site and off-site backups basically forever. This is the route I would go if replacing CDROM backup media.
I think you've mentioned it before here on DF. Just had a peek at the LTO wikipedia page, sounds very interesting. This might very well make on my wish list for my new workstation in the not so distant future. The data I most care about is currently an ever expanding 19+GB archive which I keep copying around onto several hdds/machines and, yes, a USB thumb drive as well, for good measure.
Reply With Quote
  #9   (View Single Post)  
Old 9th September 2018
ibara ibara is offline
OpenBSD language porter
 
Join Date: Jan 2014
Posts: 783
Default

I've been burned badly by the USB flash drive backup method in the past.
Reply With Quote
Old 11th September 2018
bsd-keith bsd-keith is offline
Real Name: Keith
Open Source Software user
 
Join Date: Jun 2014
Location: Surrey/Hants Border, England
Posts: 344
Default

I don't have anything particularly important, but my main backup is to external HDD, (my movies & music especially), whilst I use pendrives for the small files, quite often on more than one.
__________________
Linux since 1999, & also a BSD user.
Reply With Quote
Old 11th September 2018
shep shep is offline
Real Name: Scott
Arp Constable
 
Join Date: May 2008
Location: Dry and Dusty
Posts: 1,503
Default

Just in F/U this
Code:
tar -czf - * | openssl enc -e -aes256 -out /mnt/secured.tar.gz
worked without problems on an amd64 system.

extraction

Code:
openssl enc -d -aes256 -in /mnt/secured.tar.gz -out /home/user/unsecured.tar.gz
Then tar xvzf unsecured.tar.gz

Modified from https://www.tecmint.com/encrypt-decr...openssl-linux/

Last edited by shep; 11th September 2018 at 06:50 PM.
Reply With Quote
Old 4th October 2018
shep shep is offline
Real Name: Scott
Arp Constable
 
Join Date: May 2008
Location: Dry and Dusty
Posts: 1,503
Default

In F/U to the Thread's title, when I tried to write the encryped tar.gz to my 16GB usb thumb drive I noticed the tar.gz was consitently the same 4GB size, even after I had added content. A little seaching found that 4GB is a file size limitation for vfat/msdos file systems. I got around it by reformating the entire drive as ufs. Since I'm dedicating the drive to Backup of my main OpenBSD workstation, I get the added benefit, in case of loss/theft, that the file system is not friendly to Windows users.
Reply With Quote
Old 19th October 2018
Prevet Prevet is offline
Shell Scout
 
Join Date: Oct 2017
Posts: 84
Default

Re the OP: xorriso looks ideal for this kind of thing if you want to use CDs/DVDs. I haven't tried it yet, but the instructions below should work on my machine, because the user uses my model Blu-ray writer. If you use MDisc it will last longer than you, and it won't be affected by EMP..

https://www.linuxquestions.org/quest...5/#post5915808
Reply With Quote
Old 19th October 2018
PapaParrot's Avatar
PapaParrot PapaParrot is offline
parrot
 
Join Date: Jul 2015
Location: Durango, Mx.
Posts: 472
Default

All though that would probably work good on Linux, I don't think it would work on OpenBsd.
__________________
My best friends are parrots
Reply With Quote
Old 19th October 2018
Prevet Prevet is offline
Shell Scout
 
Join Date: Oct 2017
Posts: 84
Default

Why not?
Reply With Quote
Old 19th October 2018
PapaParrot's Avatar
PapaParrot PapaParrot is offline
parrot
 
Join Date: Jul 2015
Location: Durango, Mx.
Posts: 472
Default

Oops, maybe it would, xorriso
When I first looked I did not see it available as a package. Sorry, about that. One would just have to try it to be sure,
__________________
My best friends are parrots
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How to backup the home directory kerasi OpenBSD General 3 15th November 2017 08:36 AM
OpenBSD not reading or not using all of large hard-drive? sacerdos_daemonis OpenBSD General 10 4th December 2014 12:11 AM
Cannot copy large files to Flash Drive sharris FreeBSD General 6 30th July 2010 09:57 AM
Hard Drive backup roundkat OpenBSD General 3 29th October 2009 07:32 PM
backup mails on NAS directory milo974 OpenBSD General 3 8th August 2008 07:39 AM


All times are GMT. The time now is 11:31 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick