tarsnap and acts script

[stanl]

This concerns tarsnap and acts both built from /usr/ports/sysutils/

If I run tarsnap from the command line things seem to work ok. However if I run the acts script I get archives of only several Kb.

If someone could look at these 2 scripts and tell me where I am going wrong i would appreciate it.

/etc/acts.conf:

#!/bin/sh
# acts configuration file

## backuptargets MUST be set before use ##
# backuptargets
# Space-separated list of directories to backup, relative to /. This is a required option.
# Default: unset
backuptargets="var etc home root"

# tarsnap
# What command to call for 'tarsnap'.
# Anything that should be used every time (like --configfile)
# should go here.
# Default: tarsnap
#tarsnap="nice -n19 ionice -c3 tarsnap" # Linux
tarsnap="/usr/bin/nice /usr/local/bin/tarsnap" # OpenBSD

# tarsnapbackupoptions
# What options to use ONLY when backing up.
# Note: this variable will be expanded by /bin/sh.
# Default: unset
tarsnapbackupoptions="--one-file-system --humanize-numbers"

# calltarsnap
# Wrapper function that invokes the tarsnap command.
# Use this when you need to run tarsnap in a pipeline.
# The actual tarsnap arguments and commands are passed via "$@".
# Default: calltarsnap() { $tarsnap "$@"; }
#calltarsnap() { pass show tarsnap-passphrase | $tarsnap --passphrase-stdin "$@"; }

# verbose
# Log verbosity. Output is written to stderr.
# -1 silent. 0=normal. 1=verbose. 2=debug.
# Default: 0
verbose=1# hostname
# The machine name prefixed to created archives.
hostname=$(hostname -s)
#hostname=$(hostname)

# uselocaltime
# Use local time instead of UTC for the archive date and timestamps.
# Default: 0 (i.e. use UTC)
uselocaltime=1

# prebackupscript
# This script is run before backups are created. Make sure it's executable.
# Default: unset
#prebackupscript=/root/acts-pre.sh

# postbackupscript
# This script is run after acts is otherwise finished. Make sure it's executable.
# All created backups are passed as args
# Default: unset
#postbackupscript=/root/acts-post.sh

# syslog
# If set, log output will be written to syslog with the given facility.
# eg, user, local0, ...
# Default: unset
syslog=user

# lockfile
# Where acts should write its lockfile.
# Default: /var/run/acts
lockfile=/tmp/acts# How many backups of each type to keep
# For all limits, 0 means keep indefinitely
# Defaults: 31 daily, 12 monthly, and yearly indefinitely

dailybackups=31
monthlybackups=12
yearlybackups=0[/file]

/etc/tarsnap.conf:

[file]### Recommended options

# Tarsnap cache directory
cachedir /usr/local/tarsnap-cache

# Tarsnap key file
keyfile /root/tarsnap.key

# Don't archive files which have the nodump flag set.
nodump

# Print statistics when creating or deleting archives.
print-stats

# Create a checkpoint once per GB of uploaded data.
checkpoint-bytes 1G

### Commonly useful options

# Use SI prefixes to make numbers printed by --print-stats more readable.
humanize-numbers

### Other options, not applicable to most systems

# Aggressive network behaviour: Use multiple TCP connections when
# writing archives. Use of this option is recommended only in
# cases where TCP congestion control is known to be the limiting
# factor in upload performance.
#aggressive-networking

# Exclude files and directories matching specified patterns.
# Only one file or directory per command; multiple "exclude"
# commands may be given.
exclude /home/slaure/Videos/*
# Include only files and directories matching specified patterns.
# Only one file or directory per command; multiple "include"
# commands may be given.
include /home/slaurel/*
include /etc/*
include /var/*
include /root/*

# Attempt to reduce tarsnap memory consumption. This option
# will slow down the process of creating archives, but may help
# on systems where the average size of files being backed up is
# less than 1 MB.
#lowmem

# Try even harder to reduce tarsnap memory consumption. This can
# significantly slow down tarsnap, but reduces its memory usage
# by an additional factor of 2 beyond what the lowmem option does.
#verylowmem

# Snapshot time. Use this option if you are backing up files
# from a filesystem snapshot rather than from a "live" filesystem.
#snaptime <file>

# Include only files and directories matching specified patterns.
# Only one file or directory per command; multiple "include"
# commands may be given.
include /home/slaurel/*
include /etc/*
include /var/*
include /root/*

# Attempt to reduce tarsnap memory consumption. This option
# will slow down the process of creating archives, but may help
# on systems where the average size of files being backed up is
# less than 1 MB.
#lowmem

# Try even harder to reduce tarsnap memory consumption. This can
# significantly slow down tarsnap, but reduces its memory usage
# by an additional factor of 2 beyond what the lowmem option does.
#verylowmem

# Snapshot time. Use this option if you are backing up files
# from a filesystem snapshot rather than from a "live" filesystem.
#snaptime <file>

Thanks

[jggimi]

I'm an ACTS user ... but right now I'm without a workstation. I have a replacement laptop arriving in several days.

[stanl]

As I believe I said earlier, if I run tarsnap thru the acts script I'm getting archives of 1.4 -1.5 Kb.
However, if I run tarsnap thru a script provided on the tarsnap website:
#!/bin/sh

# User variables
email=slaurel
tarsnap_output_filename=/tmp/tarsnap-output-temporary.log
dirs="/home /root /etc /var"

# Run backup
tarsnap -c \
-f "tarsnap-$(date +%Y-%m-%d_%H-%M-%S)" \
${dirs} >${tarsnap_output_filename} 2>&1

# Send email
if [ $? -eq 0 ]; then
subject="Tarsnap backup success"
else
subject="Tarsnap backup FAILURE"
fi
mail -s "${subject}" ${email} < ${tarsnap_output_filename}

# Clean up
rm ${tarsnap_output_filename}

everything seems to work.

I would much rather use acts as I like the idea of seperate archives but until someone who knows more than me takes a look I am stuck.

[TronDD]

I don't use either tool but can you look and see what is in the small archive? That should give a clue about how far it's getting.

[stanl]

I should have thought of that but unfortunately I deleted those.

[jggimi]

You currently have ACTS provisioned with `syslog=user` -- so you should be able to review its output previous run results in /var/log/messages. I run it as part of daily.local(8) and get emailed output from every run on every server.

Now that I have a replacement laptop, I've been able to finally ssh(1) into one of the servers and compare my acts.conf with yours. Nothing jumps out at me as significantly different.


I've built a port of ACTS which I'd submitted to the project several times but it never got reviewed for commit. An up-to-date version can be found within the ACTS contrib/ directory.

[stanl]

jggimi - With suggestions from Colin Percival I believe I finally got it to work. In acts.conf I had:

backuptargets="var etc home root"

And in tarsnap.conf I had:
include /home/slaurel/*
include /etc/*
include /var/*
include /root/*

Now acts.conf reads:

backuptargets="/home /root /etc /var"

And tarsnap.conf reads:

include /home
include /root
include /etc
include /var

Thanks for not forgetting and looking into this. You've helped me out more times than I can keep track of.

[stanl]

Apparently I spoke too soon:

# tarsnap --print-stats -f '*' | sort:
All archives 33 GB 29 GB
cadfael-daily-2022-03-31_08:40:57-etc 96 MB 22 MB
cadfael-daily-2022-03-31_08:40:57-home 12 GB 11 GB
cadfael-daily-2022-03-31_08:40:57-root 320 MB 278 MB
cadfael-daily-2022-03-31_08:40:57-var 63 MB 12 MB
cadfael-daily-2022-04-01_00:10:04-etc 96 MB 22 MB
cadfael-daily-2022-04-01_00:10:04-home 12 GB 11 GB
cadfael-daily-2022-04-01_00:10:04-root 320 MB 278 MB
cadfael-daily-2022-04-01_00:10:04-var 63 MB 12 MB
cadfael-daily-2022-04-01_08:56:33-etc 96 MB 22 MB
cadfael-daily-2022-04-01_08:56:33-home 6.5 GB 5.6 GB
cadfael-daily-2022-04-01_08:56:33-root 320 MB 278 MB
cadfael-daily-2022-04-01_08:56:33-var 63 MB 12 MB

But then:

cd /tmp
# tarsnap -x -f cadfael-daily-2022-03-31_08:40:57-etc
tarsnap: /var: Not found in archive
tarsnap: /etc: Not found in archive
tarsnap: /root: Not found in archive
tarsnap: /home: Not found in archive
tarsnap: Error exit delayed from previous errors.

[jggimi]

Use -t, and see what files are in that archive. `# tarsnap -tf cadfael-daily-2022-03-31_08:40:57-etc`.

Leading slashes are not included in tarsnap archives. Not in standard tar archives, either.

[stanl]

cadfael# tarsnap -tf cadfael-daily-2022-03-31_08:40:57-etc
tarsnap: /var: Not found in archive
tarsnap: /etc: Not found in archive
tarsnap: /root: Not found in archive
tarsnap: /home: Not found in archive
tarsnap: Error exit delayed from previous errors.
cadfael#

[jggimi]

Here's a typical acts.conf:

backuptargets="var etc home root"
tarsnap="/usr/bin/nice /usr/local/bin/tarsnap"
verbose=1
prebackupscript=/usr/local/libexec/acts-pre.sh
postbackupscript=/usr/local/libexec/acts-post.sh

and a typical tarsnap.conf -- please note there are no "include" provisions:

cachedir /usr/local/tarsnap-cache
keyfile /root/tarsnap.key
nodump
print-stats
checkpoint-bytes 1G
humanize-numbers

[stanl]

I simply copied the 2 examples you provided and everything now seems to work.
So, no leading slashes and no include lines in /etc/tarsnap.conf as /etc/acts..conf manages that?
One last question if I may:
How about an exclude line? Would home/slaurel/Videos be correct and would that go in tarsnap.conf or acts.conf?
Thank you very much

[jggimi]

You should be able to test adding an exclude line to confirm matching files are excluded. I don't use them. I use"nodump" instead, which will honor that for entire directory structures or individual files.


See chflags(1) for setting / clearing the nodump file flag.

[stanl]

I used:

# chflags -R nodump /home/slaurel/Videos

and it seems to have worked.

[jggimi]

I believe you could just flag the top directory and tarsnap will not descend it.

[stanl]

You mean the -R option may not be necessary?

[jggimi]

More than that. You would be able to add new files to that structure without having to reissue chflags.

[stanl]

I am sorry to be so dense but are you saying:

# chflags dump /home/slaurel/Videos

would cause tarsnap to ignore everything the Videos directory and everything in it? In which case what is the point of the -R option?

[jggimi]

Individual files can be flagged. When you flag a directory with "nodump" the entire branch of files and directories beneath will not be dump(8)ed, and tarsnap follows this same behavior. The -R option is available for all flags, but it is not particularly relevant to the nodump flag.

[stanl]

jggimi - I'm really sorry to keep pestering you about this.

I do:

chflags nodump Old_Time_Radio/

and get:

~$ ls -lao Old_Time_Radio/
total 88
drwxr-xr-x 9 slaurel slaurel nodump 512 Apr 1 20:50 .
drwxr-xr-x 38 slaurel slaurel - 2560 Apr 2 06:04 ..
drwxr-xr-x 2 slaurel slaurel - 3072 Apr 1 20:56 Frontier_Gentleman
drwxr-xr-x 2 slaurel slaurel - 8192 Apr 1 23:16 Grand_Ole_Opry
drwxr-xr-x 5 slaurel slaurel - 4608 Apr 1 07:59 Horatio_Hornblower
drwxr-xr-x 2 slaurel slaurel - 1024 Apr 1 07:59 LOTR-BBC
drwxr-xr-x 2 slaurel slaurel - 1536 Apr 1 20:57 Mercury_Theater
drwxr-xr-x 2 slaurel slaurel - 1024 Apr 1 07:59 Poirot
drwxr-xr-x 2 slaurel slaurel - 512 Apr 1 21:02 Ulysses

Runnning tarsnap -tvf seems to show me that while the Old_Time_Radio directory and all sub-directories weren't dumped, the contents of those sub-directories were.
As a result I now have archives at tarsnap of over 30 GB.

What am I doing wrong and is this because according to dump (8):

"Files with this flag set will only be dumped during full backups."