Samba/rsync backup, delete file problem.
Hi,
I just installed OpenBSD 5.6 i386 on a Alix 2.2.
My goal is to have a small internal file server on my local net, with a backup to a local machine. I use CF to the OS and a USB HD to the file storage (no RAID, just a single disk). I use Samba to share the disk to my computers (OpenBSD, OSX and WIn7) and everything works fine. I user rsync as the backup/clone system if the disk crash (the ”backup” is on a Mac). It works... but, if I use Swedish letters in a folder or a file, rsync delete the already existing ”backup files” (on the Mac) and the copy it again! Every time I run rsync. How come?
If I use English letters, no problem! Everything works perfect.
The rsync command runs on my mac, not on the file server. This because I don’t have my Mac on 24/7, and the Mac will trigger the fileserver to make the backups.
When this work, crontab will do everything for me.
The rsync command:
rsync -avzP --log-file=rs.log --delete backup@10.30.45.15:/disk/TheVault /Volumes/BackUp/TheVault
rsync ver. on fileserver (OpenBSD 5.6 i386): rsync-3.1.1
rsync ver. on backup (OSX 10.9.5): rsync version 2.6.9 protocol version 29
The USB HD (on OpenBSD) is auto mounted as: sd0a /disk ffs rw 1 0
I will have this disk encrypted when everything works well.
OpenBSD is uptodate and patched, also the Mac.
I build this setup to replace my existing cloud that is online (I don’t need the online stuff). It works quiet well, but I don’t want my files online any more, even if they are all encrypted. Then, it’s laging to sync via the computers. It’s very annoying.
What I don’t like with my samba/rsync solution is that I need to work on my small fileserver as my main files share. It’s only a 100Mbit/s NIC on the Alix, and a USB interface disk etc. It’s slow.. And, I would rather have the files locally on each computer, then have the file server as backup/copy machine (I have a laptop that will not have the files on it! It’s not good). Today, I have my files on all my local machines (and can work with them fast), that’s sync to the cloud, and syncs back to the other machines (with lags). I like that.
But, that’s hard to fix with my samba/rsync without lag (crontab). Or?
I have been thinking of some rsync trigger on file modification. Someone with experience of that?
I also have some thoughts on a rsync ”internal cloud” there I have all files on my computers and the Alix file server as the "master copy machine". rsync with --update --existing etc. commands. But... I see problem...
Any tips to a better solution? I don’t want to use a cloud solutions, as a local owncloud etc. Just a simple ”file copy to every computer backup system”.. that’s run on a small Alix.
/Ajla
Edit:
Problem and fix:
=============
Here is some problem and fixes that I noticed that maybe help others. This is made between OpenBSD (source) to OSX (dest.) via rsync on OSX.
1) Hidden files
I got problems with hidden files that rsync tried to sync, specially .temporaryitems/folders on the source.
I put [ --exclude-from 'rs-notcopy.txt' ] in my rsync command and created a ”rs-notcopy.txt” file that included: .temporaryitems
Why I use a file and not .temporaryitems in the rsync command directly is if I want to exclude some more files and folders in the future (that I thick I will).
If I will have some problem in the future, I will try .*
2) UTF8
As you can read above I had problem with Swedish letter, e.g. UTF8.
On *BSD, be sure to install: rsync-3.1.1-iconv
If you are using a MAC <->, you can’t use the inbuilt rsync 2.6.9 as it not support --iconv. So you have to get a new one. But.. 3.1.x is bugy. So I use 3.0.9 with no problem. Then, you have to run rsync from /usr/local/bin/rsync and not only rsync as it is 2.6.9.
Update rsync on the mac:
# mkdir tmp-rs && cd tmp-rs
# curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz && curl -O http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz
# tar xvzf rsync-3.0.9.tar.gz && tar xvzf rsync-patches-3.0.9.tar.gz
# cd rsync-3.0.9 && patch -p1 <patches/fileflags.diff && patch -p1 <patches/crtimes.diff && patch -p1 <patches/hfs-compression.diff
# ./prepare-source
# ./configure
# make && make install
# cd .. && rm -R tmp-rs
You can do a alias to the new rsync.
To notice, is a different with "--iconv=UTF8-MAC,UTF-8" and "--iconv=UTF8,UTF8-MAC" depending if you push or pull.
3) My new rsync command
/usr/local/bin/rsync --iconv=UTF8-MAC,UTF-8 --log-file=rs.log --delete --exclude-from 'rs-notcopy.txt' -avzP -e 'ssh -p 22' backup@10.30.45.15:/disk/TheVault /Volumes/BackUp/TheVault
More to come...
Last edited by Ajla; 12th February 2015 at 11:56 AM.
|