Samba/rsync backup, delete file problem.
I just installed OpenBSD 5.6 i386 on a Alix 2.2.
My goal is to have a small internal file server on my local net, with a backup to a local machine. I use CF to the OS and a USB HD to the file storage (no RAID, just a single disk). I use Samba to share the disk to my computers (OpenBSD, OSX and WIn7) and everything works fine. I user rsync as the backup/clone system if the disk crash (the ”backup” is on a Mac). It works... but, if I use Swedish letters in a folder or a file, rsync delete the already existing ”backup files” (on the Mac) and the copy it again! Every time I run rsync. How come?
If I use English letters, no problem! Everything works perfect.
The rsync command runs on my mac, not on the file server. This because I don’t have my Mac on 24/7, and the Mac will trigger the fileserver to make the backups.
When this work, crontab will do everything for me.
The rsync command:
rsync -avzP --log-file=rs.log --delete email@example.com:/disk/TheVault /Volumes/BackUp/TheVault
rsync ver. on fileserver (OpenBSD 5.6 i386): rsync-3.1.1
rsync ver. on backup (OSX 10.9.5): rsync version 2.6.9 protocol version 29
The USB HD (on OpenBSD) is auto mounted as: sd0a /disk ffs rw 1 0
I will have this disk encrypted when everything works well.
OpenBSD is uptodate and patched, also the Mac.
I build this setup to replace my existing cloud that is online (I don’t need the online stuff). It works quiet well, but I don’t want my files online any more, even if they are all encrypted. Then, it’s laging to sync via the computers. It’s very annoying.
What I don’t like with my samba/rsync solution is that I need to work on my small fileserver as my main files share. It’s only a 100Mbit/s NIC on the Alix, and a USB interface disk etc. It’s slow.. And, I would rather have the files locally on each computer, then have the file server as backup/copy machine (I have a laptop that will not have the files on it! It’s not good). Today, I have my files on all my local machines (and can work with them fast), that’s sync to the cloud, and syncs back to the other machines (with lags). I like that.
But, that’s hard to fix with my samba/rsync without lag (crontab). Or?
I have been thinking of some rsync trigger on file modification. Someone with experience of that?
I also have some thoughts on a rsync ”internal cloud” there I have all files on my computers and the Alix file server as the "master copy machine". rsync with --update --existing etc. commands. But... I see problem...
Any tips to a better solution? I don’t want to use a cloud solutions, as a local owncloud etc. Just a simple ”file copy to every computer backup system”.. that’s run on a small Alix.
Problem and fix:
Here is some problem and fixes that I noticed that maybe help others. This is made between OpenBSD (source) to OSX (dest.) via rsync on OSX.
1) Hidden files
I got problems with hidden files that rsync tried to sync, specially .temporaryitems/folders on the source.
I put [ --exclude-from 'rs-notcopy.txt' ] in my rsync command and created a ”rs-notcopy.txt” file that included: .temporaryitems
Why I use a file and not .temporaryitems in the rsync command directly is if I want to exclude some more files and folders in the future (that I thick I will).
If I will have some problem in the future, I will try .*
As you can read above I had problem with Swedish letter, e.g. UTF8.
On *BSD, be sure to install: rsync-3.1.1-iconv
If you are using a MAC <->, you can’t use the inbuilt rsync 2.6.9 as it not support --iconv. So you have to get a new one. But.. 3.1.x is bugy. So I use 3.0.9 with no problem. Then, you have to run rsync from /usr/local/bin/rsync and not only rsync as it is 2.6.9.
Update rsync on the mac:
# mkdir tmp-rs && cd tmp-rs
# curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz && curl -O http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz
# tar xvzf rsync-3.0.9.tar.gz && tar xvzf rsync-patches-3.0.9.tar.gz
# cd rsync-3.0.9 && patch -p1 <patches/fileflags.diff && patch -p1 <patches/crtimes.diff && patch -p1 <patches/hfs-compression.diff
# make && make install
# cd .. && rm -R tmp-rs
You can do a alias to the new rsync.
To notice, is a different with "--iconv=UTF8-MAC,UTF-8" and "--iconv=UTF8,UTF8-MAC" depending if you push or pull.
3) My new rsync command
/usr/local/bin/rsync --iconv=UTF8-MAC,UTF-8 --log-file=rs.log --delete --exclude-from 'rs-notcopy.txt' -avzP -e 'ssh -p 22' firstname.lastname@example.org:/disk/TheVault /Volumes/BackUp/TheVault
More to come...
Last edited by Ajla; 12th February 2015 at 11:56 AM.
Hello, and welcome!
I don't use Samba or rsync, but I have seen the issue with UTF-8 characters and MAC OS X and rsync mentioned here: https://rsync.samba.org/FAQ.html
Hi, and thank you!
Yesterday I saw something about UTF8 when I was reading about the problem, but my brain didn’t notes it! It was most there rsync didn’t copt it all.
Well... a sign that I need more coffee...
I edit my first post.
BTW, how do I remove links? The [URL] is coming back.
Edit: Ok.. saw it with URLs ..
Last edited by Ajla; 12th February 2015 at 11:57 AM.
|Thread||Thread Starter||Forum||Replies||Last Post|
|Slow rsync backup from zfs pool with 99% of files||Septic||FreeBSD General||10||29th May 2013 09:45 PM|
|Easily retrievable configuration file backup with Gmail||J65nko||Guides||5||10th February 2010 04:30 AM|
|rsync problem||paul-lkw||General software and network||6||23rd January 2009 02:39 AM|
|Advice needed: File server for VPN - samba, NFS?||sim||FreeBSD General||11||4th November 2008 10:48 PM|
|backup drive to image file||milo974||OpenBSD General||8||14th August 2008 02:01 PM|