Data Back Up Schemes

[frcc]

Hi Folks:

Two Questions:
(1) I have been fumbling around for years when it comes to backing data up. I ran into the typical problems of making sure the files I save are the latest and most up to date manually.
Of course I have reaped the usual results of doing so. I have had bad experiences with some back-up systems that I either didn't quite understand of simply were not up to the task.
I am looking at using "rsync grysnc" as a solution to this problem (For Desktop Use).
I will not use a cloud scheme.
What do you use? What do you recommend? Pitfalls? Pro's/Con's?

(2) I have raid hardware in my back-up server. It is older hardware so i am cautious using
hardware raid because a failure in that card might not be able to be replaced easily if at
all, (and i have no spare) therefore complicating data recovery (T-320) 3 drives approx 1T capacity. I have
looked into software raid and at this time do not want to take the chance of setting it up
with my current skill level. Therefore its the main reason for Question (1).

I am thinking of simply using the drives on the T-320 to contain data from my Desktop or
laptop using the rsync scheme --> to one or two drives on that server.

Thoughts, suggestions, Pitfalls, Tips, check on my thinking?

As Usual, Tysm

[jggimi]

I use a cloud service for backup, out-of-scope for you, so I will just state the following general backup and recovery advice back from a time when I sold business continuity services.

• Make (or take) your backups off-site. You do not want a physical disaster (flood, fire, earthquake, ...) to cause loss of both your site and your data.
• Test your recovery process. Regularly. We backup due to fear, but we restore in a panic. You want a well-understood and well-practiced recovery process that begins with "bare metal" hardware platforms.

Your idea of using on-site hardware for backup is fine for many types of data loss, but does not account for any of the many possible types of physical disasters that could occur.

[bsd-keith]

As just an ordinary desktop user, I back up to an external disk, several copies, taken at different times/months, usually keep 3 copies of my data & rotate, (that is, I overwrite the earliest copy with the latest backup).

[J65nko]

See Using rsync to synchronize or mirror your home directory files to an USB stick

In case you change your mind about the cloud service, I can highly recommend https://www.tarsnap.com/index.html

[frcc]

Originally Posted by jggimi

I use a cloud service for backup, out-of-scope for you, so I will just state the following general backup and recovery advice back from a time when I sold business continuity services.

• Make (or take) your backups off-site. You do not want a physical disaster (flood, fire, earthquake, ...) to cause loss of both your site and your data.
• Test your recovery process. Regularly. We backup due to fear, but we restore in a panic. You want a well-understood and well-practiced recovery process that begins with "bare metal" hardware platforms.

Your idea of using on-site hardware for backup is fine for many types of data loss, but does not account for any of the many possible types of physical disasters that could occur.

Thanks for that jggimi, I have dual data issues I deal with. On my small business end I have moved my Websites about a dozen or so to a hosting company. I manage those sites with C-panel. This has worked well as of this writing. Since these are "static" sites I simply back-up the "Index.html" files there or download and place on Optical disc locally. In the past I ran OpenBsd httpd which I prefer, however, noting my location, the servers were
often rebooting themselves and re-establishing connectivity to the net following storms. Although the power is cheap here it is somewhat erratic because of the weather.This can take its toll on older hardware. I loved the simplicity and reliability of httpd when run in conjunction with a well thought out pf.conf. That said I feel "relatively" comfortable with the current business data set-up. That set-up although now a bit dated was a nice OpenBsd
learning experience.

My concern is my personal data which is now at my age mostly medical, photo, notes, documents, videos, pictures, contact info etc which are either somewhat essential or sentimental. I have no tape drive and probably will not invest in that direction as my needs are shrinking with my age which is swelling. "" I do/have used Optical, HD, and or USB devices in the past and have just a slight order of preference with the three. That said my biggest issue is maintaining order with the process of syncing the data with previous methods. Now, i note the comment by J65nko referring to "Jggimi's immortal words, a script that uses plain 'rsync' actually is a "backup data loss" script. This is one of them". Therefore, before I use rsync it would be a good idea to understand the sync process of that software so that I don't delete files on the destination device during back-up by improper use. I will also look at
https://www.tarsnap.com/index.html as stated by J65nko. Daily I use a laptop which I transfer data to a usb stick and then to my office desktop or server, therefore the laptop at the end of the day has no user data on it, so to speak with the exception of .conf files etc. So the problem is actually limited to obtaining and understanding and using back-up software correctly so that my files are up to date somewhere. Right now I have a mixture of data files on optical and usb usually duplicated which translates into a ga-zillion keystrokes. For personal data storage I probably don't need my T-320 as it is dated and a huge energy consumer. It appears at this point utilizing rsync etc
correctly to an optical or usb device is the least complicated path.

[jggimi]

Just like J65nko, I'm a Tarsnap customer. For all of my systems, local and remote.

If you ever reconsider external services, I do recommend the book "Tarsnap Mastery: Online Backups for the Truly Paranoid," by Michael W. Lucas.

[frcc]

Originally Posted by bsd-keith

As just an ordinary desktop user, I back up to an external disk, several copies, taken at different times/months, usually keep 3 copies of my data & rotate, (that is, I overwrite the earliest copy with the latest backup).

bsd-keith, what back-up scheme do you use for this?

[jggimi]

Now, i note the comment by J65nko referring to "Jggimi's immortal words, a script that uses plain 'rsync' actually is a "backup data loss" script. This is one of them". Therefore, before I use rsync it would be a good idea to understand the sync process of that software so that I don't delete files on the destination device during back-up by improper use.

Read further down in that thread. That was misquote from another user.

My statement, misquoted, was that RAID is not backup.

[frcc]

Because of your recommendation and that of J65nko's I will look into the Tarsnap service to see which is easiest for me.

[frcc]

Got-it I agree at least with older machines using hardware Raid. Also, I will not put the effort into establishing software Raid as I can see one ill thought out Keystroke can make a bad day, at least on a machine that has a installed OS.

[frcc]

Yes, did read the entire post. ty

[jggimi]

I like Tarsnap as a solution and service so much that:

• I'm the OpenBSD port maintainer for tarsnap-gui, Tarsnap's graphical user interface for use on workstations. I use tarsnap-gui on my laptop.
• I have ported ACTS (another calendar-based tarsnap script) to OpenBSD, which I discovered via the Tarsnap Mastery book. My port is included in the contrib/ directory of the published source for ACTS, and also available in the OpenBSD work in progress ports collection. I use ACTS on all of my servers.

[bsd-keith]

Originally Posted by frcc

bsd-keith, what back-up scheme do you use for this?

Just simply copying my data from machine to external drive, as per the 3 copies rotation.

I picked up the idea from the way servers were backed up in the old days.

[frcc]

Thanks, for the comeback, was wondering if anything other than what i thought you were doing, which i think is adequate for most desktop users most of the time.

[dayid]

I use a mixture of: rsync (with personal script); git; dump; & tarsnap.

I mention each because they each have different use-cases for me:

• git: Yes, it's not "for backups"; but for things like /etc/ on machines I keep each in their own branch (hostname) and push them to my central server. This provides me with an easy way of comparing configurations on multiple machines (simple diff) while also giving me a super-fast way to back-out changes (checkout previous commit). It is also useful when duplicating a configuration as I can just checkout/clone files and fork. This is a backup for recovering simple files from local as well as a remote and protects from human error most readily as well as inconsistencies during upgrades and the like.
• rsync: My main method for quickly-changing filesystems (I'm looking at you /home). Particularly with a script and writing to different subdirs for "snapshots" in conjunction with --link-dest this is very powerful. Lightweight, quick, and easy to configure for both local-backup and remote this is useful for human errors and system/update errors.
• dump: Classic for a reason and since it "just works" I use dump as a nightly method to write across the network to another machine w/i the same LAN. Dump/Restore are powerful and with a small amount of learning are very useful for full-system recoveries (disk-failure, catastrophies, the like) but can also be good for human errors like "Where's that .png I .... oh.... deleted..... a long time ago.... longer than I keep in my rsync-mirror".
• tarsnap: My main offsite/third-party resource. I'd heard of it before MWL's book, but the simplicity of it (and familiarity of it after having used rsync/dump and tar for so long) made it a shoe-in. The deduplication and encrypted storage are also fantastic benefits. I use this in addition to all my local methods, but unless there's a large incident I don't restore from it as usually I can restore from a method on my own LANs and avoid fees (and network speeds) in restoring.

[frcc]

Comprehensive approach with a great reason/answer/use of the methods you employ. It provokes thought on the subject even through my needs are v. modest. Thanks for the reply..