DaemonForums  

Go Back   DaemonForums > Miscellaneous > General software and network

General software and network General OS-independent software and network questions, X11, MTA, routing, etc.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 4th June 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Default your favorite mirroring tool

Hi everyone !

What is your favorite mirroring tool ? wget ? httrack ? rsync ? w3mir ? and why ?

wget is cool but if mirroring is interrupted , -k becomes useless and bye to offline browsing .. here httrack excels .. as excels in specifying wildcards ..

Please share your thoughts :-)
Reply With Quote
  #2   (View Single Post)  
Old 4th June 2013
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 1,027
Default

I use rsync. Why? It's the first one I tried, and it worked for my limited needs.
Reply With Quote
  #3   (View Single Post)  
Old 4th June 2013
shep shep is offline
Real Name: Scott
Arp Constable
 
Join Date: May 2008
Location: Dry and Dusty
Posts: 1,503
Default

I mainly mirror files within my LAN as an inexpensive form of backup.

Since my main system runs OpenBSD I just start the ftpd on the server using the defaults.

For the clients I use several tools:
On Linux systems I usually install lftp which has a mirroring mode. When I need to sort through a large number of files/directories Midnight commander (misc/mc) has an ftp mode that allows tagging of files/directories and selection of the latest/largest files along with recursively moving an entire directory. The 2 panel view is a plus.

You can run commands on files in mc. I have a "rainy weekend" thought that I may be able to show a package respository in the panel and pass the tagged files to $ xterm -e sudo pkg_install -i as a quasi graphical package manager in OpenBSD.

Last edited by shep; 4th June 2013 at 10:00 PM.
Reply With Quote
  #4   (View Single Post)  
Old 5th June 2013
PrinceCruise PrinceCruise is offline
Real Name: Prince
PrinceCruise
 
Join Date: Jun 2012
Location: India
Posts: 23
Default

rsync does what I want.

Regards.
Reply With Quote
  #5   (View Single Post)  
Old 5th June 2013
jggimi's Avatar
jggimi jggimi is online now
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Quote:
I mainly mirror files within my LAN as an inexpensive form of backup.
Keep in mind that mirrors that synchronize -- instantly, such as in a RAID array, or on a schedule, as with the technologies discussed here -- are not backup.

Why? Once you've mirrored a data loss, you've lost data.
Reply With Quote
  #6   (View Single Post)  
Old 7th June 2013
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by jggimi View Post
Once you've mirrored a data loss, you've lost data.
Which begs the following question. daemonfowl, what is it that you are attempting to accomplish? Are you really wanting to mirror files, or back-up files?
Reply With Quote
  #7   (View Single Post)  
Old 7th June 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Default

Hi all and thank you very much for sharing opinions and ideas !

Quote:
what is it that you are attempting to accomplish?
Thanks for minding , ocicat !
I'd like to know BSD folk predilection .. but also to know about other mirroring tools .. I use httrack for dynamic websites and wget for static ..some websites refuse to mirror either way (like the undeadly.org ) ..
Why mirror ?? please do accept a fool's answer : either to keep it for offline reference .. or for the sake of event chronicling -so to speak ..
I'll be happy to know if there is a way to mirror such websites when httrack and wget fail to .. :-)

Quote:
Are you really wanting to mirror files, or back-up files?
yes ocicat .. mirroring.
Reply With Quote
  #8   (View Single Post)  
Old 7th June 2013
thirdm thirdm is offline
Spam Deminer
 
Join Date: May 2009
Posts: 248
Default

I use rsync, but I haven't looked carefully at how I mirror and backup yet. There are some things I hope to do better someday.

I've generally made do with one way sync. i.e. I take whatever's on one side and replace what's on the other side with that. Eventually I want to merge the two sides taking what's changed from either (or the more appropriate with conflicts resolved interactively). While starting to learn the ML programming language, I noticed in the promotional materials for OCaml that there's a file sync program called Unison written in OCaml that's supposed to handle that well. Someday I'll try it.

Another piece I've not looked at yet is keeping old copies of files (akin to VMS, Hammer's builtin snapshotting, or a version control system). With rsync, I'm pretty sure this is simply a matter of supplying the right flag (this has come up on OpenBSD mailing lists) but I haven't tried that yet either.

Finally, I have to figure out a better way of syncing a bunch of mail folders, as created by the Gnus newsreader/mail client, on one machine and syncing them in a way that I can run Gnus on the 2nd machine and have all pointers or whatever synced up in a way that it looks the same as on the original machine. I'm guessing this will require understanding how Gnus writes its files and may not be a simple matter of syncing every file involved.

But until I can get OpenBSD working on my fast machine (someone added an ACPI fix to a recent current that looks very promising), I'm considering everything sort of interim anyway, so my syncing is kind of sloppy.
Reply With Quote
  #9   (View Single Post)  
Old 10th June 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Default

So , why do some websites refuse "wget -r -m -k --no-parent" ?
Reply With Quote
Old 10th June 2013
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

daemonfowl, you're hijacking your own thread again.

Reply With Quote
Old 10th June 2013
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Default

sorry I'll start a new thread .. despite the fact that the reason behind this very thread is :
that I cannot rely on httrack nor on wget to mirror a few websited of interest (undeadly.org) .. is it a CyberTaboo ? :-)
Reply With Quote
Old 10th June 2013
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by daemonfowl View Post
... despite the fact that the reason behind this very thread is :
that I cannot rely on httrack nor on wget to mirror a few websited of interest...
This may very well be, but the fact remains that the initial question asked for different modes of mirroring, & a number of respondents addressed the question posed.

Before posting, you should ask yourself what is the fundamental issue that you would like others to comment upon, & articulate the problem & motivation upfront.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is your favorite font? guitarscn Off-Topic 4 1st January 2011 01:46 AM
your 3 favorite games welkin FreeBSD Ports and Packages 17 5th June 2010 08:26 PM
What's your favorite Window Manager? welkin FreeBSD Ports and Packages 1 13th February 2010 10:39 AM
Favorite x86 CPU manufacturer for... JMJ_coder Off-Topic 12 12th July 2008 05:59 PM


All times are GMT. The time now is 11:18 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick