View Single Post
  #1   (View Single Post)  
Old 30th June 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Using rsync to synchronize or mirror your home directory files to an USB stick

Using rsync to synchronize or mirror your home directory files to an USB stick

Before you modify and start using the script please consider the following:
  • OS dependencies

    The script has been developed on OpenBSD and uses an USB stick formatted with OpenBSD UFS filesystem.
    On my system with an IDE drive the hard disk is 'wd0' and the USB stick is 'sd0'.
    When you use SATA disks 'sd0' will be your first hard disk. So please make sure you will use the correct device name for your USB stick.

    For speed purpose the USB stick is mounted with what is called softdep in OpenBSD.
    Code:
    $ mount
    /dev/wd0a on / type ffs (local)
    /dev/sd0a on /mnt type ffs (local, noatime, softdep)
    In FreeBSD this is called soft updates.

    In OpenBSD this is a option for mount(8):
    Code:
                 softdep
                         (FFS only.)  Mount the file system using soft
                         dependencies.  Instead of metadata being written
                         immediately, it is written in an ordered fashion to keep
                         the on-disk state of the file system consistent.  This
                         results in significant speedups for file create/delete
                         operations.  This option is ignored when using the -u
                         flag and a file system is already mounted read/write.
    
                         The options async and softdep are mutually exclusive.
    For FreeBSD you have to use the -U option when you create the file system with newfs(8):
    Code:
    -U      Enable soft updates on the new file system.
  • The rsync '--delete' option makes this script unsuitable for backing up purposes.

    If you accidentally delete the picture of your boy/girl friend in his/her new swimsuit from your home directory, it will also be deleted from the USB stick upon the next run of this script.
    Or in Jggimi's immortal words, a script that uses plain 'rsync' actually is a "backup data loss" script. This is one of them

    If you really need a backup script you will have to wait for an next installment, that will use rsnapshot
    From rsnapshot.org
    Quote:
    rsnapshot is a filesystem snapshot utility for making backups of local and remote systems.

    Using rsync and hard links, it is possible to keep multiple, full backups instantly available. The disk space required is just a little more than the space of one full backup, plus incrementals.
  • Security

    In the current state, the script will also sync your private SSH keys in the .ssh directory, the passwords saved by your browser etc. So please keek the USB stick in a physical secure place or bad things may happen.

My USB stick details:
  • dmesg:
    Code:
    umass0 at uhub0 port 5 configuration 1 interface 0 "SanDisk Cruzer Edge" rev 2.00/1.26 addr 2
    umass0: using SCSI over Bulk-Only
    scsibus3 at umass0: 2 targets, initiator 0
    sd0 at scsibus3 targ 1 lun 0: <SanDisk, Cruzer Edge, 1.26> SCSI3 0/direct removable serial.0781556b501E0E819601
    sd0: 15267MB, 512 bytes/sector, 31266816 sectors
  • fdisk:
    Code:
    Disk: sd0       geometry: 1946/255/63 [31266816 Sectors]
    Offset: 0       Signature: 0xAA55
                Starting         Ending         LBA Info:
     #: id      C   H   S -      C   H   S [       start:        size ]
    -------------------------------------------------------------------------------
     0: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
     1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
     2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused      
    *3: A6      0   1   2 -   1945 254  63 [          64:    31262426 ] OpenBSD
  • disklabel:
    Code:
    # /dev/rsd0c:
    type: SCSI
    disk: SCSI disk
    label: Cruzer Edge     
    duid: 624ba8481eba7c02
    flags:
    bytes/sector: 512
    sectors/track: 63
    tracks/cylinder: 255
    sectors/cylinder: 16065
    cylinders: 1946
    total sectors: 31266816
    boundstart: 64
    boundend: 31262490
    drivedata: 0 
    
    16 partitions:
    #                size           offset  fstype [fsize bsize  cpg]
      a:         31262400               64  4.2BSD   2048 16384    1 
      c:         31266816                0  unused


Things that you will have to modify
  • Name of your home directory
  • Device name and label of your USB stick
  • The browser cache name in 'BROWSER_CACHE'
  • The rsync command to exclude more files and/or directories

Script structure
  • Initialize variables
  • Check for root privileges to mount the USB stick and create the log files
    Code:
    ./USB-rsync.sh : you are not ROOT !!
    ./USB-rsync.sh : aborting .......
  • Delete '.serverauth*' files older than 1 day
    Code:
    deleting old .serverauth files
    286167    4 -rw-------    1 adriaan  adriaan       186 Jun 28 01:38 /home/adriaan/.serverauth.29764
    286207    4 -rw-------    1 adriaan  adriaan       186 Jun 28 05:53 /home/adriaan/.serverauth.9425
    286211    4 -rw-------    1 adriaan  adriaan       186 Jun 28 22:06 /home/adriaan/.serverauth.12955
    286216    4 -rw-------    1 adriaan  adriaan       186 Jun 29 08:27 /home/adriaan/.serverauth.30490
    286208    4 -rw-------    1 adriaan  adriaan       186 Jun 29 17:38 /home/adriaan/.serverauth.24636
  • Mount USB stick
    Code:
    Mounting USB stick .......
    USB has been been mounted
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/wd0a     27.6G   12.4G   13.8G    47%    /
    /dev/sd0a     14.7G    5.1G    9.4G    35%    /mnt
  • Using the less pager perform a dry run of rsync to show the files that will be affected.

    Code:
    sending incremental file list
    adriaan/
    adriaan/Guide-usb-rynsc.txt
    adriaan/USB-rsync.sh
    adriaan/.mozilla/firefox/chmj5t7f.default/
    adriaan/.mozilla/firefox/chmj5t7f.default/.parentlock
    adriaan/.mozilla/firefox/chmj5t7f.default/_CACHE_CLEAN_
    [snip]
    adriaan/.mozilla/firefox/chmj5t7f.default/thumbnails/
    adriaan/.mozilla/firefox/chmj5t7f.default/thumbnails/14b8d30faa111187e7ff165b59b4a956.png
    adriaan/.mozilla/firefox/chmj5t7f.default/thumbnails/f297b1b7fbb3972f2e4c5bb3ed0b6aec.png
    adriaan/RCS/
    adriaan/RCS/USB-rsync.sh,v
    
    sent 41309 bytes  received 381 bytes  27793.33 bytes/sec
    total size is 5475203761  speedup is 131331.34 (DRY RUN)
  • Query the user whether the actual rsync has to be done
    Code:
    Do you want to perform the actual, real rsync now? (Y/N):
  • After the rsync has ended, show an informational message about the disk space used and the location of the log files.
    Code:
    sent 33540722 bytes  received 3884 bytes  919030.30 bytes/sec
    total size is 5475796134  speedup is 163.24
    
    ./USB-rsync.sh : has finished
    =======================================================
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/wd0a     27.6G   12.4G   13.8G    47%    /
    /dev/sd0a     14.7G    5.1G    9.4G    35%    /mnt
    
    The log file of the dry run     : /var/log/USB-rsync_dry_run.log
    The log file of the actual run  : /var/log/USB-rsync.log
    
    Please do not forget to umount your USB stick .........
    =======================================================
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 15th April 2021 at 02:14 AM. Reason: Fixed rsnapshot,org link
Reply With Quote