Go Back   DaemonForums > OpenBSD > OpenBSD General

OpenBSD General Other questions regarding OpenBSD which do not fit in any of the categories below.

Thread Tools Display Modes
  #1   (View Single Post)  
Old 12th February 2015
Ajla's Avatar
Ajla Ajla is offline
Real Name: Ajla
New User
Join Date: Feb 2015
Location: 145700
Posts: 4
Default Samba/rsync backup, delete file problem.


I just installed OpenBSD 5.6 i386 on a Alix 2.2.
My goal is to have a small internal file server on my local net, with a backup to a local machine. I use CF to the OS and a USB HD to the file storage (no RAID, just a single disk). I use Samba to share the disk to my computers (OpenBSD, OSX and WIn7) and everything works fine. I user rsync as the backup/clone system if the disk crash (the ”backup” is on a Mac). It works... but, if I use Swedish letters in a folder or a file, rsync delete the already existing ”backup files” (on the Mac) and the copy it again! Every time I run rsync. How come?
If I use English letters, no problem! Everything works perfect.

The rsync command runs on my mac, not on the file server. This because I don’t have my Mac on 24/7, and the Mac will trigger the fileserver to make the backups.
When this work, crontab will do everything for me.

The rsync command:
rsync -avzP --log-file=rs.log --delete backup@ /Volumes/BackUp/TheVault

rsync ver. on fileserver (OpenBSD 5.6 i386): rsync-3.1.1
rsync ver. on backup (OSX 10.9.5): rsync version 2.6.9 protocol version 29

The USB HD (on OpenBSD) is auto mounted as: sd0a /disk ffs rw 1 0
I will have this disk encrypted when everything works well.

OpenBSD is uptodate and patched, also the Mac.

I build this setup to replace my existing cloud that is online (I don’t need the online stuff). It works quiet well, but I don’t want my files online any more, even if they are all encrypted. Then, it’s laging to sync via the computers. It’s very annoying.

What I don’t like with my samba/rsync solution is that I need to work on my small fileserver as my main files share. It’s only a 100Mbit/s NIC on the Alix, and a USB interface disk etc. It’s slow.. And, I would rather have the files locally on each computer, then have the file server as backup/copy machine (I have a laptop that will not have the files on it! It’s not good). Today, I have my files on all my local machines (and can work with them fast), that’s sync to the cloud, and syncs back to the other machines (with lags). I like that.
But, that’s hard to fix with my samba/rsync without lag (crontab). Or?
I have been thinking of some rsync trigger on file modification. Someone with experience of that?
I also have some thoughts on a rsync ”internal cloud” there I have all files on my computers and the Alix file server as the "master copy machine". rsync with --update --existing etc. commands. But... I see problem...

Any tips to a better solution? I don’t want to use a cloud solutions, as a local owncloud etc. Just a simple ”file copy to every computer backup system”.. that’s run on a small Alix.



Problem and fix:
Here is some problem and fixes that I noticed that maybe help others. This is made between OpenBSD (source) to OSX (dest.) via rsync on OSX.

1) Hidden files
I got problems with hidden files that rsync tried to sync, specially .temporaryitems/folders on the source.

I put [ --exclude-from 'rs-notcopy.txt' ] in my rsync command and created a ”rs-notcopy.txt” file that included: .temporaryitems

Why I use a file and not .temporaryitems in the rsync command directly is if I want to exclude some more files and folders in the future (that I thick I will).

If I will have some problem in the future, I will try .*

2) UTF8
As you can read above I had problem with Swedish letter, e.g. UTF8.

On *BSD, be sure to install: rsync-3.1.1-iconv

If you are using a MAC <->, you can’t use the inbuilt rsync 2.6.9 as it not support --iconv. So you have to get a new one. But.. 3.1.x is bugy. So I use 3.0.9 with no problem. Then, you have to run rsync from /usr/local/bin/rsync and not only rsync as it is 2.6.9.

Update rsync on the mac:

# mkdir tmp-rs && cd tmp-rs
# curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.9.tar.gz && curl -O http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.9.tar.gz
# tar xvzf rsync-3.0.9.tar.gz && tar xvzf rsync-patches-3.0.9.tar.gz
# cd rsync-3.0.9 && patch -p1 <patches/fileflags.diff && patch -p1 <patches/crtimes.diff && patch -p1 <patches/hfs-compression.diff
# ./prepare-source
# ./configure
# make && make install
# cd .. && rm -R tmp-rs

You can do a alias to the new rsync.

To notice, is a different with "--iconv=UTF8-MAC,UTF-8" and "--iconv=UTF8,UTF8-MAC" depending if you push or pull.

3) My new rsync command
/usr/local/bin/rsync --iconv=UTF8-MAC,UTF-8 --log-file=rs.log --delete --exclude-from 'rs-notcopy.txt' -avzP -e 'ssh -p 22' backup@ /Volumes/BackUp/TheVault

More to come...

Last edited by Ajla; 12th February 2015 at 11:56 AM.
Reply With Quote
  #2   (View Single Post)  
Old 12th February 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
Join Date: May 2008
Location: USA
Posts: 5,445

Hello, and welcome!

I don't use Samba or rsync, but I have seen the issue with UTF-8 characters and MAC OS X and rsync mentioned here: https://rsync.samba.org/FAQ.html
[Your described problem] can occur with HFS+ on Mac OS X: if you copy a directory with a file that has a UTF-8 character sequence in it, say a 2-byte umlaut-u (\0303\0274), the file will get that character stored by the filesystem using 3 bytes (\0165\0314\0210), and rsync will not know that these differing filenames are the same file (it will, in fact, remove a prior copy of the file if --delete is enabled, and then recreate it).

You can avoid a charset problem by passing an appropriate --iconv option to rsync that tells it what character-set the source files are, and what character-set the destination files get stored in. For instance, the above Mac OS X problem would be dealt with by using --iconv=UTF-8,UTF8-MAC (UTF8-MAC is a pseudo-charset recognized by Mac OS X iconv in which all characters are decomposed).
I hope this small lead helps you with problem resolution.
Reply With Quote
  #3   (View Single Post)  
Old 12th February 2015
Ajla's Avatar
Ajla Ajla is offline
Real Name: Ajla
New User
Join Date: Feb 2015
Location: 145700
Posts: 4

Hi, and thank you!

Yesterday I saw something about UTF8 when I was reading about the problem, but my brain didn’t notes it! It was most there rsync didn’t copt it all.
Well... a sign that I need more coffee...

I edit my first post.

BTW, how do I remove links? The [URL] is coming back.

Edit: Ok.. saw it with URLs ..

Last edited by Ajla; 12th February 2015 at 11:57 AM.
Reply With Quote

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow rsync backup from zfs pool with 99% of files Septic FreeBSD General 10 29th May 2013 09:45 PM
Easily retrievable configuration file backup with Gmail J65nko Guides 5 10th February 2010 04:30 AM
rsync problem paul-lkw General software and network 6 23rd January 2009 02:39 AM
Advice needed: File server for VPN - samba, NFS? sim FreeBSD General 11 4th November 2008 10:48 PM
backup drive to image file milo974 OpenBSD General 8 14th August 2008 02:01 PM

All times are GMT. The time now is 10:46 PM.

Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick