DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 30th June 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Using rsync to synchronize or mirror your home directory files to an USB stick

Using rsync to synchronize or mirror your home directory files to an USB stick

Before you modify and start using the script please consider the following:
  • OS dependencies

    The script has been developed on OpenBSD and uses an USB stick formatted with OpenBSD UFS filesystem.
    On my system with an IDE drive the hard disk is 'wd0' and the USB stick is 'sd0'.
    When you use SATA disks 'sd0' will be your first hard disk. So please make sure you will use the correct device name for your USB stick.

    For speed purpose the USB stick is mounted with what is called softdep in OpenBSD.
    Code:
    $ mount
    /dev/wd0a on / type ffs (local)
    /dev/sd0a on /mnt type ffs (local, noatime, softdep)
    In FreeBSD this is called soft updates.

    In OpenBSD this is a option for mount(8):
    Code:
                 softdep
                         (FFS only.)  Mount the file system using soft
                         dependencies.  Instead of metadata being written
                         immediately, it is written in an ordered fashion to keep
                         the on-disk state of the file system consistent.  This
                         results in significant speedups for file create/delete
                         operations.  This option is ignored when using the -u
                         flag and a file system is already mounted read/write.
    
                         The options async and softdep are mutually exclusive.
    For FreeBSD you have to use the -U option when you create the file system with newfs(8):
    Code:
    -U      Enable soft updates on the new file system.
  • The rsync '--delete' option makes this script unsuitable for backing up purposes.

    If you accidentally delete the picture of your boy/girl friend in his/her new swimsuit from your home directory, it will also be deleted from the USB stick upon the next run of this script.
    Or in Jggimi's immortal words, a script that uses plain 'rsync' actually is a "backup data loss" script. This is one of them

    If you really need a backup script you will have to wait for an next installment, that will use rsnapshot
    From rsnapshot.org
    Quote:
    rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

    Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.
  • Security

    In the current state, the script will also sync your private SSH keys in the .ssh directory, the passwords saved by your browser etc. So please keek the USB stick in a physical secure place or bad things may happen.

My USB stick details:
  • dmesg:
    Code:
    umass0 at uhub0 port 5 configuration 1 interface 0 "SanDisk Cruzer Edge" rev 2.00/1.26 addr 2
    umass0: using SCSI over Bulk-Only
    scsibus3 at umass0: 2 targets, initiator 0
    sd0 at scsibus3 targ 1 lun 0: <SanDisk, Cruzer Edge, 1.26> SCSI3 0/direct removable serial.0781556b501E0E819601
    sd0: 15267MB, 512 bytes/sector, 31266816 sectors
  • fdisk:
    Code:
    Disk: sd0       geometry: 1946/255/63 [31266816 Sectors]
    Offset: 0       Signature: 0xAA55
                Starting         Ending         LBA Info:
     #: id      C   H   S -      C   H   S [       start:        size ]
    -------------------------------------------------------------------------------
     0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
     1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
     2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
    *3: A6      0   1   2 -   1945 254  63 [          64:    31262426 ] OpenBSD
  • disklabel:
    Code:
    # /dev/rsd0c:
    type: SCSI
    disk: SCSI disk
    label: Cruzer Edge     
    duid: 624ba8481eba7c02
    flags:
    bytes/sector: 512
    sectors/track: 63
    tracks/cylinder: 255
    sectors/cylinder: 16065
    cylinders: 1946
    total sectors: 31266816
    boundstart: 64
    boundend: 31262490
    drivedata: 0 
    
    16 partitions:
    #                size           offset  fstype [fsize bsize  cpg]
      a:         31262400               64  4.2BSD   2048 16384    1 
      c:         31266816                0  unused


Things that you will have to modify
  • Name of your home directory
  • Device name and label of your USB stick
  • The browser cache name in 'BROWSER_CACHE'
  • The rsync command to exclude more files and/or directories

Script structure
  • Initialize variables
  • Check for root privileges to mount the USB stick and create the log files
    Code:
    ./USB-rsync.sh : you are not ROOT !!
    ./USB-rsync.sh : aborting .......
  • Delete '.serverauth*' files older than 1 day
    Code:
    deleting old .serverauth files
    286167    4 -rw-------    1 adriaan  adriaan       186 Jun 28 01:38 /home/adriaan/.serverauth.29764
    286207    4 -rw-------    1 adriaan  adriaan       186 Jun 28 05:53 /home/adriaan/.serverauth.9425
    286211    4 -rw-------    1 adriaan  adriaan       186 Jun 28 22:06 /home/adriaan/.serverauth.12955
    286216    4 -rw-------    1 adriaan  adriaan       186 Jun 29 08:27 /home/adriaan/.serverauth.30490
    286208    4 -rw-------    1 adriaan  adriaan       186 Jun 29 17:38 /home/adriaan/.serverauth.24636
  • Mount USB stick
    Code:
    Mounting USB stick .......
    USB has been been mounted
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/wd0a     27.6G   12.4G   13.8G    47%    /
    /dev/sd0a     14.7G    5.1G    9.4G    35%    /mnt
  • Using the less pager perform a dry run of rsync to show the files that will be affected.

    Code:
    sending incremental file list
    adriaan/
    adriaan/Guide-usb-rynsc.txt
    adriaan/USB-rsync.sh
    adriaan/.mozilla/firefox/chmj5t7f.default/
    adriaan/.mozilla/firefox/chmj5t7f.default/.parentlock
    adriaan/.mozilla/firefox/chmj5t7f.default/_CACHE_CLEAN_
    [snip]
    adriaan/.mozilla/firefox/chmj5t7f.default/thumbnails/
    adriaan/.mozilla/firefox/chmj5t7f.default/thumbnails/14b8d30faa111187e7ff165b59b4a956.png
    adriaan/.mozilla/firefox/chmj5t7f.default/thumbnails/f297b1b7fbb3972f2e4c5bb3ed0b6aec.png
    adriaan/RCS/
    adriaan/RCS/USB-rsync.sh,v
    
    sent 41309 bytes  received 381 bytes  27793.33 bytes/sec
    total size is 5475203761  speedup is 131331.34 (DRY RUN)
  • Query the user whether the actual rsync has to be done
    Code:
    Do you want to perform the actual, real rsync now? (Y/N):
  • After the rsync has ended, show an informational message about the disk space used and the location of the log files.
    Code:
    sent 33540722 bytes  received 3884 bytes  919030.30 bytes/sec
    total size is 5475796134  speedup is 163.24
    
    ./USB-rsync.sh : has finished
    =======================================================
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/wd0a     27.6G   12.4G   13.8G    47%    /
    /dev/sd0a     14.7G    5.1G    9.4G    35%    /mnt
    
    The log file of the dry run     : /var/log/USB-rsync_dry_run.log
    The log file of the actual run  : /var/log/USB-rsync.log
    
    Please do not forget to umount your USB stick .........
    =======================================================
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 15th April 2021 at 02:14 AM. Reason: Fixed rsnapshot,org link
Reply With Quote
  #2   (View Single Post)  
Old 30th June 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default The script

The script:
Code:
#!/bin/sh
# $Id: USB-rsync.sh,v 1.5 2013/06/30 18:11:05 adriaan Exp $

#  Copyright (c) 2013 J65nko - administrator daemonforums.org
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

USB_STICK=/dev/sd0a
LOGFILE=/var/log/USB-rsync.log
LOGFILE_DRY=/var/log/USB-rsync_dry_run.log

# --- directory to exclude from rsync:
FIREFOX_CACHE='.mozilla/firefox/chmj5t7f.default/Cache'

# ---- source and destination for rysnc
# --- this will create a directory 'adriaan' on '/mnt'
SOURCE_DIR=/home/adriaan
MOUNT_DIR=/mnt

# alternative specification with identical result
# SOURCE_DIR=/home/adriaan/
# MOUNT_DIR=/mnt/adriaan
# here no directory will not be created, giving you the choice to
# specify an alternative destination directory e.g. :
# MOUNT_DIR=/mnt/j65nko

# Need root privileges to mount stick on '/mnt'
# and to create logfile in '/var/log'

if [ "$(id -u)" -ne  0 ]; then 
    echo $0 : you are not ROOT !! 
    echo $0 : aborting .......
    exit 1
fi

# --- 
echo deleting old .serverauth files

find ${SOURCE_DIR} -name '.serverauth*' -maxdepth 1 ! -ctime 1 -ls
find ${SOURCE_DIR} -name '.serverauth*' -maxdepth 1 ! -ctime 1 -exec rm {} \; 

# ---
echo Mounting USB stick .......

if mount -o noatime,softdep ${USB_STICK} ${MOUNT_DIR} ; then
    echo USB has been been mounted
    df -h
    echo ----------
else
    echo Could not mount USB stick!
    exit 2
fi
# -------- rsync options
#  -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
#  -v, --verbose               increase verbosity
#  -z, --compress              compress file data during the transfer
#  -n, --dry-run               perform a trial run with no changes made
#      --delete                delete extraneous files from dest dirs
#      --delete-excluded       also delete excluded files from dest dirs

echo ----- dry-run 
rsync -avz \
        --delete \
        --exclude=${FIREFOX_CACHE} \
        --dry-run \
        ${SOURCE_DIR} \
        ${MOUNT_DIR} 2>&1 | tee ${LOGFILE_DRY} | less 

printf "Do you want to perform the actual, real rsync now? (Y/N): "

read ANSWER
case ${ANSWER} in
[Yy] )          echo OK doing the rsync .....
                ;;

* )             echo "$0: You decided not to do the actual rsync from ${SOURCE_DIR} to ${MOUNT_DIR}"
                exit 3
                ;;
esac

# ----  perform the real rsync 
rsync -avz \
        --delete \
        --exclude=${FIREFOX_CACHE} \
        ${SOURCE_DIR} \
        ${MOUNT_DIR} 2>&1 | tee ${LOGFILE}

# ----- message to user
cat <<END

$0 : has finished
=======================================================
$(df -h)

The log file of the dry run     : ${LOGFILE_DRY}
The log file of the actual run  : ${LOGFILE}

Please do not forget to umount your USB stick .........
=======================================================
END
# ---- end of script ---
Please use the link to download the script
Attached Files
File Type: sh USB-rsync.sh (3.3 KB, 130 views)
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #3   (View Single Post)  
Old 30th June 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Backup using rsnapshot (1)

Placeholder
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #4   (View Single Post)  
Old 30th June 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Backup using rsnapshot (2)

Placeholder
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #5   (View Single Post)  
Old 1st July 2013
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 1,027
Default

Hi, just a very quick comment for the USB-rsync.sh script. I think you can combine the two find commands into one, by adding the -ls into the second one.
Reply With Quote
  #6   (View Single Post)  
Old 14th July 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Default

Thank you so much , j65nko !!
Quite a useful script.
Quote:
Jggimi's immortal words, a script that uses plain 'rsync' actually is a "backup data loss" script.
Lol ! jggimi : Pure Signals !
Reply With Quote
  #7   (View Single Post)  
Old 1st January 2014
thirdm thirdm is offline
Spam Deminer
 
Join Date: May 2009
Posts: 248
Default

This paraphrase of jggimi's statement out of context is misleading. It takes a careful reading of this tutorial, I think, as well as some interpretation or inference of the intended meaning, to avoid walking away with the impression that rsync used alone (even without one of the --delete arguments) will delete files in the destination directory and that rsnapshot is the (as opposed to a) solution to that.

I believe it doesn't intend to say that at all. Your first phrasing is clear enough in only saying rsync --delete falls into jggimi's case of backing up your deletions: "The rsync '--delete' option makes this script unsuitable for backing up purposes." Then to me it would be clearer to follow up in your next tutorial with a modification of this script with different rsync options that make it a backup script rather than only a mirroring script (no --delete, but with --link-dest?) rather than proceeding immediately to an rsnapshot tutorial.

Myself, I don't want to go to rsnapshot but like to code my back up scripts myself. This lets me do something a little weird: I have a small disk laptop that I snapshot to an old Macintosh with a larger harddrive and some older files that never fit on the laptop. After I rsync from laptop to a subdirectory on the powermac I rsync from the powermac the full set of files minus some exclusions to a USB drive. Not sure this will fit into someone else's idea of a backup or snapshotting program. Then writing it myself also helps me to remember better what my "back up" (I'm still foggy on the rigorous definition of a back up -- sysadmins seem to have a stronger definition than I do) does exactly.

I like this phrasing of jggimi's too, but have to admit I've had occasion to want to backup some of my deletions, not to mention my moves. First, there are some things you genuinely do not want to exist anywhere. Second, in my half baked "backup" schemes I sometimes run out of space on the back up media. Yes, I suppose disks are cheap but not necessarily cheap enough for my taste. My back up space is sufficient again now, but I expect whenever I get around to using --link-dest in my script this may change eventually. Hmmm, I wonder how --link-dest and --delete interact, if at all. Maybe I will only have to delete selective big, old, no longer extant on the source drive files directly off the back up medium manually. Maybe you could argue that's a safer bet than rsync --delete in the first place.

Last edited by thirdm; 1st January 2014 at 07:26 PM. Reason: correction and clarification after re-reading OP
Reply With Quote
  #8   (View Single Post)  
Old 1st January 2014
thirdm thirdm is offline
Spam Deminer
 
Join Date: May 2009
Posts: 248
Default

Oh, one other thing. I found it useful in syncing to usb to first check that the right drive was mounted. You know, you can have a lot of different things you might stick in a usb port. I was doing this before on slackware by using fstab to mount to a certain directory based on disk identifier somehow IIRC and checking in script that that directory was mounted to an sdaX or something before proceeding. I think now that I'm back to OpenBSD I can check with the disk uuid from disklabel: e.g. disklabel sd0a | grep '^duid: 2319282e388bd9dc', but haven't added this back in yet.

Last edited by thirdm; 1st January 2014 at 07:02 PM.
Reply With Quote
  #9   (View Single Post)  
Old 1st January 2014
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I believe this is the actual statement daemonfowl was referring to. Had he quoted this instead of paraphrasing, the intent of my comments would have been clearer.
Quote:
Originally Posted by jggimi View Post
Keep in mind that mirrors that synchronize -- instantly, such as in a RAID array, or on a schedule, as with the technologies discussed here -- are not backup.

Why? Once you've mirrored a data loss, you've lost data.
Reply With Quote
Reply

Tags
rsync

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow rsync backup from zfs pool with 99% of files Septic FreeBSD General 10 29th May 2013 09:45 PM
HOWTO: Find Duplicated Files in Directory Tree vermaden Guides 22 27th April 2010 07:43 PM
strange "~" directory in home directory gosha OpenBSD General 5 23rd February 2009 06:12 PM
FTP users sharing same home directory phreud FreeBSD General 6 11th November 2008 10:11 PM
Jails - mount: /usr/home: No such file or directory chris FreeBSD General 6 6th August 2008 10:47 PM


All times are GMT. The time now is 10:24 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick