DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 7th January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default Automating OpenBSD snapshot downloads with a .netrc file

While in http://www.daemonforums.org/showthread.php?t=670, another approach is used, this script uses a .netrc generated on the fly to do the complete download in a single login session.

The .netrc syntax is described in the ftp(1) man page.

Some examples
Code:
$ snapget
snapget
Destination directory: Snapshots
./snapget: Please specify one or more sets
The sets are: base x mini pxe floppy man misc iso
Selecting the 'mini' install sets as defined in the snapget script
Code:
$ snapget mini
Destination directory: Snapshots
-------------- Created .netrc-----------------------
     1  
     2  machine ftp.nluug.nl login anonymous password thanks@puffy.org
     3  
     4  macdef init
     5  prompt off
     6  epsv4 off
     7  preserve on
     8  get /pub/OpenBSD/snapshots/i386/SHA256 Snapshots/SHA256
     9  get /pub/OpenBSD/snapshots/i386/base46.tgz Snapshots/base46.tgz
    10  get /pub/OpenBSD/snapshots/i386/bsd Snapshots/bsd
    11  get /pub/OpenBSD/snapshots/i386/bsd.rd Snapshots/bsd.rd
    12  get /pub/OpenBSD/snapshots/i386/etc46.tgz Snapshots/etc46.tgz
    13  quit
    14  
------------------------------------------------------
Do you want to start 'ftp' with this '.netrc' ? (Y/N)
n
./snapget: I take this for a 'NO', aborting ...
The second line defines the ftp site, login name and password.

After an empty line a macro definition named 'init' is created. It contains some ftp commands. See the ftp man page for the details. Line 8 to 12 perform a ftp 'get' of the minimal number of filesets to install OpenBSD. By leaving out 'bsd.rd' it could be still be made smaller.

The 'init' macro will be executed immediately after the ftp server has authorized the ftp login. Because the macro ends with a 'quit' command the ftp session will terminate automatically.

Code:
$  snapget pxe flopp
Destination directory: Snapshots
./snapget : undefined set flopp
./snapget : aborting
The name of the file sets will be checked, and if an unknown file set is detected, the script will terminate with a message informing the user.

Normally a ".netrc' file is located in the home directory. Actually ftp uses the environment variable HOME to decide in which directory to look.
Code:
$ env | grep HOME
HOME=/home/j65nko
In order not to overwrite a possible existing .netrc the script stores the .netrc file in the directory defined in the ${DESTDIR{ variable.

By using env(1) utility, the value of HOME is changed into $DESTDIR}, and thus coaches ftp to locate and execute the right .netrc:
Code:
env HOME=${DESTDIR} ftp -4 ${SITE} 2>&1 | tee ${DESTDIR}/Logfile
A very simple illustration of using 'env', is to print the current date and time from another time zone by setting the TZ environment variable
Code:
# env  TZ=CET date    
Thu Jan  7 03:24:55 CET 2010

$env  TZ=EST date   
Wed Jan  6 21:25:09 EST 2010
A last example :
Code:
$snapget mini floppy
Destination directory: Snapshots
-------------- Created .netrc-----------------------
     1  
     2  machine ftp.nluug.nl login anonymous password thanks@puffy.org
     3  
     4  macdef init
     5  prompt off
     6  epsv4 off
     7  preserve on
     8  get /pub/OpenBSD/snapshots/i386/SHA256 Snapshots/SHA256
     9  get /pub/OpenBSD/snapshots/i386/base46.tgz Snapshots/base46.tgz
    10  get /pub/OpenBSD/snapshots/i386/bsd Snapshots/bsd
    11  get /pub/OpenBSD/snapshots/i386/bsd.rd Snapshots/bsd.rd
    12  get /pub/OpenBSD/snapshots/i386/etc46.tgz Snapshots/etc46.tgz
    13  get /pub/OpenBSD/snapshots/i386/floppy46.fs Snapshots/floppy46.fs
    14  get /pub/OpenBSD/snapshots/i386/floppyB46.fs Snapshots/floppyB46.fs
    15  get /pub/OpenBSD/snapshots/i386/floppyC46.fs Snapshots/floppyC46.fs
    16  quit
    17  
------------------------------------------------------
Do you want to start 'ftp' with this '.netrc' ? (Y/N)
y
Trying 192.87.102.43...
Connected to ftp.nluug.nl.
220-Welcome to the FTP archive of SURFnet BV and
220-The Netherlands Unix Users Group (NLUUG).
[snip]
331 Please specify the password.
230 Login successful.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256 Snapshots/SHA256
local: Snapshots/SHA256 remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (192,87,102,43,199,230)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/SHA256 (2162 bytes).
226 File send OK.
2162 bytes received in 0.19 seconds (10.90 KB/s)
get /pub/OpenBSD/snapshots/i386/base46.tgz Snapshots/base46.tgz
local: Snapshots/base46.tgz remote: /pub/OpenBSD/snapshots/i386/base46.tgz
227 Entering Passive Mode (192,87,102,43,239,73)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/base46.tgz (50281346 bytes).
226 File send OK.
50281346 bytes received in 64.90 seconds (756.64 KB/s)
[snip]
get /pub/OpenBSD/snapshots/i386/floppyC46.fs Snapshots/floppyC46.fs
local: Snapshots/floppyC46.fs remote: /pub/OpenBSD/snapshots/i386/floppyC46.fs
227 Entering Passive Mode (192,87,102,43,222,187)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/floppyC46.fs (1474560 bytes).
226 File send OK.
1474560 bytes received in 1.99 seconds (723.88 KB/s)
quit
221 Goodbye.
The retrieved files:
Code:
$ ls -l Snapshots/ 
total 134992
-rw-r--r--  1 j65nko  j65nko      3599 Jan  7 03:42 Logfile
-rw-r--r--  1 j65nko  j65nko      2162 Jan  5 22:59 SHA256
-rw-r--r--  1 j65nko  j65nko  50281346 Jan  5 22:58 base46.tgz
-rw-r--r--  1 j65nko  j65nko   7505333 Jan  5 22:58 bsd
-rw-r--r--  1 j65nko  j65nko   6239716 Jan  5 22:58 bsd.rd
-rw-r--r--  1 j65nko  j65nko    522035 Jan  5 22:58 etc46.tgz
-rw-r--r--  1 j65nko  j65nko   1474560 Jan  5 22:58 floppy46.fs
-rw-r--r--  1 j65nko  j65nko   1474560 Jan  5 22:58 floppyB46.fs
-rw-r--r--  1 j65nko  j65nko   1474560 Jan  5 22:58 floppyC46.fs
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #2   (View Single Post)  
Old 7th January 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,128
Default The 'snapget' sh script

Code:
#!/bin/sh
# $Id: snapget,v 1.4 2010/01/07 01:42:05 j65nko Exp $
# get snapshot files with a on the fly generated '.netrc' for ftp(1)


DESTDIR="Snapshots"
if [ -d ${DESTDIR} ] ; then
   rm -f ${DESTDIR}/*
else
   mkdir -p ${DESTDIR} || exit 1
fi

# -- destination directory 
# DESTDIR="SNAP$(date -u "+%Y-%m-%d_%H:%M_UTC")"
# mkdir ${DESTDIR} || exit 1

echo "Destination directory: ${DESTDIR}"

# variables  used in 'netrc' template
# ------------------------------------------------------------------
ARCH='i386'		# architecture name used as ftp dir
#ARCH='amd64'		# architecture name used as ftp dir

RN=46			# used in "base${RN}.tgz" etc
#VERSION='4.6'		# ftp dir for releases
VERSION='snapshots'	# ftp dir for snapshots

DIR="/pub/OpenBSD/${VERSION}/${ARCH}"  

SITE='ftp.calyx.nl'
SITE='ftp.eu.openbsd.org'
SITE='ftp.esat.net'
SITE='ftp.openbsd.org'
SITE='ftp.wu-wien.ac.at'
SITE='ftp.eu.openbsd.org'
SITE='ftp.nluug.nl'

# customizable file sets 
# if you add or remove sets, don't forget the update the ALLSETS variable 
# ------------------------------------------------------------------

#SETS_START
# ------------------------------------------------------------------
base="
INSTALL.${ARCH}
SHA256
base${RN}.tgz
bsd
bsd.mp
bsd.rd
comp${RN}.tgz
etc${RN}.tgz
man${RN}.tgz
misc${RN}.tgz
"

x="
xbase${RN}.tgz
xetc${RN}.tgz
xfont${RN}.tgz
xserv${RN}.tgz
xshare${RN}.tgz
"

mini="
SHA256
base${RN}.tgz
bsd
bsd.rd
etc${RN}.tgz
"
pxe="
pxeboot
"

floppy="
floppy${RN}.fs
floppyB${RN}.fs
floppyC${RN}.fs
"
man="
man${RN}.tgz
"

misc="
misc${RN}.tgz
"
iso="
install${RN}.iso
"
# ------------------------------------------------------------------
#SETS_END

# ------------------------------------------------------------------
ALLSETS="base x mini pxe floppy man misc iso"
# ------------------------------------------------------------------

if [ $# -eq 0 ] ; then
    echo "$0: Please specify one or more sets"
    echo "The sets are: ${ALLSETS}"
    exit 1
fi

# -- accumulate all sets in FILES
for SET in "$@" ; do
     eval CONTENTS=\"\$${SET}\"
     if [ "x${CONTENTS}" = "x" ] ; then
         echo "$0 : undefined set ${SET}"
         echo "$0 : aborting"
         exit 1
     fi 
    FILES="${FILES} ${CONTENTS} "
done

# echo ${FILES} | tr '[:blank:]' '\n'

# --- 'here document' as '.netrc' template

cat <<-TEMPLATE >${DESTDIR}/.netrc

machine ${SITE} login anonymous password thanks@puffy.org

macdef init
prompt off
epsv4 off
preserve on
$( for THIS in ${FILES} ; do 
     echo "get ${DIR}/${THIS} ${DESTDIR}/${THIS}" 
   done
)
quit

TEMPLATE

# -- An empty line is the terminator for a macdef macro
# -- So KEEP EMPTY LINE between 'quit' and here document marker 'TEMPLATE'
# -- to prevent 'Macro definition missing null line terminator' messages
# -- from 'ftp'

echo -------------- Created '.netrc'-----------------------
cat -n  $DESTDIR/.netrc
echo ------------------------------------------------------
echo "Do you want to start 'ftp' with this '.netrc' ? (Y/N)" ; read answer

case "$answer" in
[Yy] )		;;

*    )		echo "$0: I take this for a 'NO', aborting ..."
    		exit 2
		;;
esac

# -- ftp uses HOME to locate '.netrc'
env HOME=${DESTDIR} ftp -4 ${SITE} 2>&1 | tee ${DESTDIR}/Logfile

# -- EOF
At this moment the script does not prompt for the ftp site, or architecture. You will have to edit the variables in the script itself.

Because the forum software insists on a file extension, you will have to rename the downloaded script to remove the '.txt'
Code:
$ mv snapget.txt snapget

$ chmod u+x snapget
Attached Files
File Type: txt snapget.txt (3.1 KB, 111 views)
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Reply

Tags
.netrc, ftp, openbsd snapshots

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
PHP read file contents - Maximum file size cksraj Programming 1 21st September 2009 11:38 AM
Best way to upgrade from -release to snapshot Carpetsmoker OpenBSD General 5 26th July 2009 08:51 PM
file:/// mfaridi FreeBSD Security 3 27th July 2008 02:18 PM
Automating ports update with portmaster jaymax FreeBSD Ports and Packages 3 12th June 2008 06:56 AM
Customizeable FTP auto-fetch script (OpenBSD snapshot download as example) J65nko Guides 2 1st June 2008 03:29 AM


All times are GMT. The time now is 11:03 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick