Automating FreeBSD release downloads with a .netrc file

J65nko · #1 **(View Single Post)** 17th January 2010

1. Automating FreeBSD release downloads with a '.netrc' file

1.1 Introduction
1.2 File hierarchy of a FreeBSD release
1.3 A '.netrc' file to download a FreeBSD release
1.4 Transcript of '.netrc' processing
1.5 '.netrc' commands to download boot-only-iso image
1.6 A merge of the two '.netrc' files
1.7 The '.netrc' file generator script
1.8 Example of usage

1.1 Introduction

Many years ago for the first time, I did a ftp install of FreeBSD 4.x. using the two floppies. I instructed 'sysinstall' to retrieve the installation files from a FreeBSD ftp mirror on the internet. Previously I had installed FreeBSD a few times from CD. At that time I was using a new ISDN modem, which was fast with 64Kb, at least compared with the 14k4 old modem.

Because the system did not boot, I must have made a mistake somewhere. Not looking forward to a couple of repeated ftp downloads with the ISDN modem, I decided to download the release files once and for all on a FreeBSD Pentium-Pro box. Some simple configuration steps turned the Pentium-Pro into a local ftp server.

This method has become my favorite way to install FreeBSD as well as OpenBSD. First download the files making up the FreeBSD release to a ftp server in my local network. Boot 'sysinstall' with the install floppies, and install FreeBSD by fetching the installation files from the local ftp server.

A nice side effect is speed. Transfer over the network turned out to be much faster than reading from a 10x speed CD on a Pentium I box.

1.2 File hierarchy of a FreeBSD release

In the standardized ftp structure, we find the release files in '/pub/FreeBSD/releases/<arch>/<release>/', where <arch> stands for the hardware architecture like 'i386' or 'amd64', and <release> is a release tag, e.g. '7.2-RELEASE' or '8.0-RELEASE'.

Code:

lrwxr-xr-x  1 1006  3000       1 May  1  2009 7.2-RELEASE -> .
-r--r--r--  1 1006  3000    4877 May  1  2009 ERRATA.HTM
-r--r--r--  1 1006  3000    3514 May  1  2009 ERRATA.TXT
-r--r--r--  1 1006  3000  186567 May  1  2009 HARDWARE.HTM
-r--r--r--  1 1006  3000  114717 May  1  2009 HARDWARE.TXT
-r--r--r--  1 1006  3000   19736 May  1  2009 README.HTM
-r--r--r--  1 1006  3000   14311 May  1  2009 README.TXT
-r--r--r--  1 1006  3000   60037 May  1  2009 RELNOTES.HTM
-r--r--r--  1 1006  3000   31407 May  1  2009 RELNOTES.TXT
drwxrwxr-x  2 1006  3000    1024 May  1  2009 base
drwxrwxr-x  2 1006  3000     512 May  1  2009 catpages
-rw-r--r--  1 1006  3000      25 May  1  2009 cdrom.inf
drwxrwxr-x  2 1006  3000     512 May  1  2009 dict
drwxrwxr-x  2 1006  3000    1536 May  1  2009 doc
-r--r--r--  1 1006  3000    3704 May  1  2009 docbook.css
drwxrwxr-x  2 1006  3000     512 May  1  2009 floppies
drwxrwxr-x  2 1006  3000     512 May  1  2009 games
drwxrwxr-x  2 1006  3000     512 May  1  2009 info
drwxrwxr-x  2 1006  3000    1024 May  1  2009 kernels
drwxrwxr-x  2 1006  3000     512 May  1  2009 manpages
lrwxr-xr-x  1 1006  3000      40 May  2  2009 packages -> 
                       ../../../ports/i386/packages-7.2-release
drwxrwxr-x  2 1006  3000     512 May  1  2009 ports
drwxrwxr-x  2 1006  3000     512 May  1  2009 proflibs
drwxrwxr-x  2 1006  3000    2560 May  1  2009 src

A summary of the files:

There are two links. The first one, '7.2-RELEASE' points to the current directory, and identifies the release. The 'packages' link points to the pre-compiled packages.
A collection of files like 'ERRRATA', 'RELNOTES' in both text and html formats.
A 'cdrom.inf' file and 'docbook.css', a cascading style sheet for the html documents.

Directories like 'base', 'catpages' and 'src'.

A look inside 'base':

Code:

   1869 May  1  2009 CHECKSUM.MD5
   3199 May  1  2009 CHECKSUM.SHA256
1425408 May  1  2009 base.aa
1425408 May  1  2009 base.ab
1425408 May  1  2009 base.ac
1425408 May  1  2009 base.ad
    [snip]
1425408 May  1  2009 base.ax
1425408 May  1  2009 base.ay
1425408 May  1  2009 base.az
1425408 May  1  2009 base.ba
    [snip]
1425408 May  1  2009 base.bh
 202680 May  1  2009 base.bi
   1053 May  1  2009 base.inf
1315055 May  1  2009 base.mtree
    438 Apr 15  2009 install.sh

MD5 and SHA256 checksum files for each individual 'base.??'. All 'base.??' files, from 'base.aa' to 'base.bi' are chunks of a 'tar' archive. A peek into 'install.sh', reveals how these pieces are glued together to a single tar file. and how subsequently this tar archive is unpacked:

Code:

 cat base.?? | tar --unlink -xpzf - -C ${DESTDIR:-/}

Downloading all these files with the command line 'ftp' would be rather tedious. For a long time I used 'wget' to fetch the files. A couple of years later I discovered how to use a '.netrc' file to script ftp commands.

1.3 A '.netrc' file to download a FreeBSD release

A full explanation of the '.netrc' tokens can be found in the ftp(1) man page.

Code:

machine ftp2.dk.FreeBSD.org

macdef init
prompt off
cd /pub/FreeBSD/releases/amd64/8.0-RELEASE
mget  ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM 
      README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css 
$ getdir base catpages dict doc games info kernels
         manpages ports proflibs src
! ln -s . 8.0-RELEASE 
quit

macdef getdir
! mkdir $i
mget $i/*

Note that the parameters for the mget and $ getdir commands have been wrapped for display purposes. In reality both commands expect these on a single line.

In the following invocation of the ftp command, the '-4' instructs the ftp client to only use IPv4 and not try IPv6, the 'a' option prescribes an 'anonymous' login.

Code:

$ ftp -4a ftp://ftp2.dk.FreeBSD.org

The ftp(1) program will perform the following procedure:

A check whether the directory specified by the 'HOME' environment variable, contains a '.netrc' file.
Scan the found '.netrc' for a line starting with 'machine ftp2.dk.FreeBSD.org'. All information up to end of file or till another line starting with either the 'machine' or 'default' token will be processed.
The first encounter is a macro definition called "init", which is executed automatically after the ftp server has authorized us to use its services.

prompt off prevents ftp from asking us for confirmations. From the ftp man page

Code:

prompt
            Toggle interactive prompting.  Interactive prompting occurs
            during multiple file transfers to allow the user to selec-
            tively retrieve or store files.  If prompting is turned off
            (default is on), any mget or mput will transfer all files,
            and any mdelete will delete all files.

cd /pub/FreeBSD/releases/amd64/8.0-RELEASE changes the directory on the server to the standard location for the 8.0 files for the amd64 processor.
mget ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT .. docbook.css gets or retrieves the files into the current working directory of the ftp client.
$ getdir base catpages dict [snip] src executes the macro getdir, with a long list of parameters like 'base', 'catpages', 'dict' etc.

The 'getdir' macro is defined as follows:
Code:
```
macdef getdir
! mkdir $i
mget $i/*
```
! mkdir $i executes the shell command mkdir $ion the ftp client. The $i parameter will be expanded to the first parameter 'base', resulting in 'mkdir base" on the ftp client. mget $1 will be expanded to mget base/*.

Now getdir will move on to the next parameter 'catpages', creating a 'catpages' directory on the client and mgeting the 'catpages/*' directory.

This implicit loop will continue until getdirs runs out of parameters.

An equivalent shell construct for $ getdir base catpages dict would be:
Code:
```
for i in base catpages dict ; do
    mkdir $i
    mget $i/*
done
```
! ln -s . 8.0-RELEASE creates a symbolic link to the current directory '.'
quit terminates the ftp session with the remote server, and exits ftp.

1.4 Transcript of '.netrc' processing

Code:

$ ftp -4a ftp://ftp2.dk.FreeBSD.org

Connected to heset.dkuug.dk.
220 FTP server ready.
331 Guest login ok, send your email address as password.
230- Velcome to the DKUUG software archive.
230- If you encounter any problems please send mail to ftpadmin@dkuug.dk
    [snip]
230 Guest login ok, access restrictions apply.
prompt off
Interactive mode off.
cd /pub/FreeBSD/releases/amd64/8.0-RELEASE
250 CWD command successful.
mget  ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM README.TXT
RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css
local: ERRATA.HTM remote: ERRATA.HTM
150 Opening BINARY mode data connection for 'ERRATA.HTM' (4932 bytes).
226 Transfer complete.
4932 bytes received in 0.19 seconds (25.12 KB/s)
    [snip]
local: docbook.css remote: docbook.css
150 Opening BINARY mode data connection for 'docbook.css' (3704 bytes).
226 Transfer complete.
3704 bytes received in 0.19 seconds (18.88 KB/s)

The getdir macro operating on the directory 'base' as parameter.

Code:

$ getdir  base catpages dict doc games info kernels lib32 manpages ports proflibs src
! mkdir base
mget base/*
local: base/CHECKSUM.MD5 remote: base/CHECKSUM.MD5
150 Opening BINARY mode data connection for 'base/CHECKSUM.MD5' (2163 bytes).
226 Transfer complete.
2163 bytes received in 0.19 seconds (11.04 KB/s)
local: base/CHECKSUM.SHA256 remote: base/CHECKSUM.SHA256
150 Opening BINARY mode data connection for 'base/CHECKSUM.SHA256' (3703 bytes).
226 Transfer complete.
3703 bytes received in 0.19 seconds (18.93 KB/s)
local: base/base.aa remote: base/base.aa
150 Opening BINARY mode data connection for 'base/base.aa' (1425408 bytes).
226 Transfer complete.
1425408 bytes received in 1.95 seconds (713.33 KB/s)
    [snip]
local: base/install.sh remote: base/install.sh
150 Opening BINARY mode data connection for 'base/install.sh' (442 bytes).
226 Transfer complete.
442 bytes received in 0.20 seconds (2.20 KB/s)

Last iteration of getdir macro on directory 'src', the creation of the '8.0-RELEASE' symbolic link and termination with quit command.

Code:

! mkdir src
mget src/*
local: src/CHECKSUM.MD5 remote: src/CHECKSUM.MD5
150 Opening BINARY mode data connection for 'src/CHECKSUM.MD5' (5758 bytes).
226 Transfer complete.
5758 bytes received in 0.20 seconds (28.05 KB/s)
local: src/CHECKSUM.SHA256 remote: src/CHECKSUM.SHA256
150 Opening BINARY mode data connection for 'src/CHECKSUM.SHA256' (9678 bytes).
226 Transfer complete.
9678 bytes received in 0.19 seconds (49.59 KB/s)
local: src/install.sh remote: src/install.sh
150 Opening BINARY mode data connection for 'src/install.sh' (942 bytes).
226 Transfer complete.
942 bytes received in 0.19 seconds (4.81 KB/s)
local: src/sbase.aa remote: src/sbase.aa
150 Opening BINARY mode data connection for 'src/sbase.aa' (75919 bytes).
226 Transfer complete.
75919 bytes received in 0.21 seconds (349.05 KB/s)
    [snip]
local: src/susbin.ac remote: src/susbin.ac
150 Opening BINARY mode data connection for 'src/susbin.ac' (120619 bytes).
226 Transfer complete.
120619 bytes received in 0.28 seconds (427.11 KB/s)
local: src/susbin.inf remote: src/susbin.inf
150 Opening BINARY mode data connection for 'src/susbin.inf' (99 bytes).
226 Transfer complete.
99 bytes received in 0.20 seconds (0.48 KB/s)
! ln -s . 8.0-RELEASE
quit
221 Goodbye
$

1.5 '.netrc' commands to download boot-only-iso image

Code:

! mkdir iso
lcd iso
cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 
mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256

With ! mkdir iso a local shell command creates a 'iso' directory. The lcd iso, a local cd, changes the default directory on the ftp client. The ftp(1) man page explains the reason:

Code:

mget remote-files
            Expand the remote-files on the remote machine and do a get
            for each file name thus produced.
            ......
            Files are transferred into the local working directory, which
            can be changed with `lcd directory'; new local directories
            can be created with `! mkdir directory'

The default directory on the server is switched to the iso images with cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0.

After having set the correct directory for both local and remote, the mget downloads the iso, as well as the two checksum files.

A transcript:

Code:

! mkdir iso
lcd iso
Local directory now /home/j65nko/FREEBSD/amd64_80/iso
cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0
250 CWD command successful.
mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256
local: 8.0-RELEASE-amd64-bootonly.iso remote: 8.0-RELEASE-amd64-bootonly.iso
150 Opening BINARY mode data connection for '8.0-RELEASE-amd64-bootonly.iso' (47366144 bytes).
226 Transfer complete.
47366144 bytes received in 60.97 seconds (758.67 KB/s)
local: CHECKSUM.MD5 remote: CHECKSUM.MD5
150 Opening BINARY mode data connection for 'CHECKSUM.MD5' (351 bytes).
226 Transfer complete.
351 bytes received in 0.20 seconds (1.75 KB/s)
local: CHECKSUM.SHA256 remote: CHECKSUM.SHA256
150 Opening BINARY mode data connection for 'CHECKSUM.SHA256' (526 bytes).
226 Transfer complete.
526 bytes received in 0.20 seconds (2.57 KB/s

1.6 A merge of the two '.netrc' files

The boot-only-iso is fetched first, so the impatient already can burn the image file to CD, while the release files are still being downloaded.

Code:

machine ftp2.dk.FreeBSD.org

macdef init
prompt off
! mkdir iso
lcd iso
cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 
mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256
lcd ..
cd /pub/FreeBSD/releases/amd64/8.0-RELEASE
mget  ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM
      README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css 
$ getdir  base catpages dict doc games info kernels 
          lib32 manpages ports proflibs src
! ln -s . 8.0-RELEASE 
quit

macdef getdir
! mkdir $i
mget $i/*

Because it easy to make mistakes in adapting this '.netrc' to a another release or processor, a simple shell script takes care of the customization.

1.7 The '.netrc' file generator script

By only editing the shell variables SITE, RN, and ARCH, a different ftp mirror site, release or processor architecture can be chosen.

The MY_ISO_DIR variable allows customization of the destination directory of the boot-only-iso, and it's checksum files.

Code:

# -- start of user configurable settings

#SITE=ftp4.freebsd.org
#SITE=ftp2.dk.FreeBSD.org
SITE=ftp.nl.freebsd.org

RN='8.0'
#ARCH=i386
ARCH=amd64

MY_ISO_DIR="ISO"

# -- end of user configurable settings

These variables are then used to initialize some derivate variables

Code:

RELEASE="${RN}-RELEASE"
DIR=/pub/FreeBSD/releases/${ARCH}/${RELEASE}
FILE=${ARCH}_${RELEASE}files.txt

# -------- bootonly.iso -----------------------
# ftp://ftp.nl.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.0/
# 8.0-RELEASE-i386-bootonly.iso

ISO_DIR="/pub/FreeBSD/ISO-IMAGES-${ARCH}/${RN}"
BOI="${RELEASE}-${ARCH}-bootonly.iso"
MD5="CHECKSUM.MD5"
SHA256="CHECKSUM.SHA256"

By executing the script with the "-v" and "-x" options, we can see the expansion of the variables in the lines pre-fixed with a plus sign.

Code:

$ sh -vx release-fetch

# -- start of user configurable settings

#SITE=ftp4.freebsd.org
#SITE=ftp2.dk.FreeBSD.org
SITE=ftp.nl.freebsd.org
+ SITE=ftp.nl.freebsd.org

RN='8.0'
+ RN=8.0
#ARCH=i386
ARCH=amd64
+ ARCH=amd64

MY_ISO_DIR="ISO"
+ MY_ISO_DIR=ISO

# -- end of user configurable settings

RELEASE="${RN}-RELEASE"
+ RELEASE=8.0-RELEASE
DIR=/pub/FreeBSD/releases/${ARCH}/${RELEASE}
+ DIR=/pub/FreeBSD/releases/amd64/8.0-RELEASE
FILE=${ARCH}_${RELEASE}files.txt
+ FILE=amd64_8.0-RELEASEfiles.txt

# -------- bootonly.iso -----------------------
# ftp://ftp.nl.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.0/
# 8.0-RELEASE-i386-bootonly.iso

ISO_DIR="/pub/FreeBSD/ISO-IMAGES-${ARCH}/${RN}"
+ ISO_DIR=/pub/FreeBSD/ISO-IMAGES-amd64/8.0
BOI="${RELEASE}-${ARCH}-bootonly.iso"
+ BOI=8.0-RELEASE-amd64-bootonly.iso
MD5="CHECKSUM.MD5"
+ MD5=CHECKSUM.MD5
SHA256="CHECKSUM.SHA256"
+ SHA256=CHECKSUM.SHA256

Now these variables have been set, the script downloads a ls -l style listing from the ftp mirror.

The method used is not very well known, but is simple. Echo a ftp ls command to stdout. Have ftp receive this command through a pipe as standard input to execute.

Code:

get_main_listing() {
    echo "ls ${DIR} ${FILE}" | ftp -4ai ${SITE} 1>&2 
}

get_main_listing

Because the shell '-vx' option does not trace inside functions, the following is a manual expansion

Code:

echo "ls /pub/FreeBSD/releases/amd64/8.0-RELEASE \
       amd64_8.0-RELEASEfiles.txt" |  ftp -4ai ftp.nl.freebsd.org  1>&2

The file 'amd64_8.0-RELEASEfiles.txt':

Code:

lrwxrwxrwx    1 500      450             1 Nov 23 07:53 8.0-RELEASE -> .
-r--r--r--    1 500      450          4932 Nov 21 15:10 ERRATA.HTM
-r--r--r--    1 500      450          3563 Nov 21 15:10 ERRATA.TXT
-r--r--r--    1 500      450        190670 Nov 21 15:10 HARDWARE.HTM
-r--r--r--    1 500      450        115402 Nov 21 15:10 HARDWARE.TXT
-r--r--r--    1 500      450         19867 Nov 21 15:10 README.HTM
-r--r--r--    1 500      450         14336 Nov 21 15:10 README.TXT
-r--r--r--    1 500      450         10512 Nov 21 15:10 RELNOTES.HTM
-r--r--r--    1 500      450          7545 Nov 21 15:10 RELNOTES.TXT
drwxrwxr-x    2 500      450          4096 Nov 22 20:04 base
drwxrwxr-x    2 500      450          4096 Nov 22 20:06 catpages
-rw-r--r--    1 500      450            25 Nov 21 15:10 cdrom.inf
drwxrwxr-x    2 500      450          4096 Nov 22 20:06 dict
drwxrwxr-x    2 500      450          4096 Nov 22 20:06 doc
-r--r--r--    1 500      450          3704 Nov 21 15:10 docbook.css
drwxrwxr-x    2 500      450          4096 Nov 22 20:06 games
drwxrwxr-x    2 500      450          4096 Nov 22 20:07 info
drwxrwxr-x    2 500      450          4096 Nov 22 20:19 kernels
drwxrwxr-x    2 500      450          4096 Nov 22 20:21 lib32
drwxrwxr-x    2 500      450          4096 Nov 22 20:23 manpages
lrwxrwxrwx    1 500      450            41 Nov 23 07:53 packages -> 
                         ../../../ports/am d64/packages-8.0-release
drwxrwxr-x    2 500      450          4096 Nov 22 20:32 ports
drwxrwxr-x    2 500      450          4096 Nov 22 20:33 proflibs
drwxrwxr-x    2 500      450          4096 Nov 22 20:59 src

Because the entries for the files start with '-r', and the directories with 'd'. 'awk' can easily distinguish and select the release documents and directories.

Code:

# select the release file directories 
SETS=$(awk '/^d/ { printf " %s", $9 }' ${FILE} )
# select the release documents
DOCS=$(awk '/^-r/ { printf " %s" , $9 }' ${FILE} )

The trace:

Code:

# select the release file directories 
SETS=$(awk '/^d/ { printf " %s", $9 }' ${FILE} )
+ awk /^d/ { printf " %s", $9 } amd64_8.0-RELEASEfiles.txt
+ SETS= base catpages dict doc games info kernels lib32 manpages \
        ports proflibs src
# select the release documents
DOCS=$(awk '/^-r/ { printf " %s" , $9 }' ${FILE} )
+ awk /^-r/ { printf " %s" , $9 } amd64_8.0-RELEASEfiles.txt
+ DOCS= ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM \
        README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css

Note that the amd64 release has a 'lib32' file set, while the i386 release does not. Distilling the directories from a listing is a simple method to automatically include all release file sets for a certain architecture.

Finally after having gathered all necessary information, the script can generate the '.netrc' file.

Code:

# -- Create .netrc using 'here document'

cat <<END_OF_NETRC

machine ${SITE}

macdef init
prompt off
! mkdir ${MY_ISO_DIR}
lcd ${MY_ISO_DIR}
cd ${ISO_DIR} 
mget ${BOI} ${MD5} ${SHA256}
lcd ..
cd ${DIR}
mget ${DOCS} 
$ getdir ${SETS}
! ln -s . ${RELEASE} 
quit

macdef getdir
! mkdir \$i
mget \$i/*

END_OF_NETRC

The variables used in the 'here' document will be expanded. By prefixing the macro parameter '$i' with a "\", the shell is prevented from interpreting this macro parameter as an non-existing shell variable $i.

Code:

+ cat
+ << END_OF_NETRC 

machine ftp.nl.freebsd.org

macdef init
prompt off
! mkdir ISO
lcd ISO
cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 
mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256
lcd ..
cd /pub/FreeBSD/releases/amd64/8.0-RELEASE
mget  ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf do
cbook.css 
$ getdir  base catpages dict doc games info kernels lib32 manpages ports proflibs src
! ln -s . 8.0-RELEASE 
quit

macdef getdir
! mkdir $i
mget $i/*

The final action informs the user how to coach ftp into using the '.netrc' file.

Code:

# print Usage message on stderr
cat <<END >&2
-------------------------------------------------------------
Remember that ftp(1) uses the HOME environment variable to locate .netrc
Assuming you saved and/or edited .netrc in the current directory or '.',
use it as follows:

env HOME=. ftp -4a ftp://${SITE} 2>&1 | tee Logfile

END
+ cat
+ << END 
+ >&2 
-------------------------------------------------------------
Remember that ftp(1) uses the HOME environment variable to locate .netrc
Assuming you saved and/or edited .netrc in the current directory or '.',
use it as follows:

env HOME=. ftp -4a ftp://ftp.nl.freebsd.org 2>&1 | tee Logfile

Through intermediation of env(1) the client ftp program is passed a modified version of the HOME environment variable, as directive to use the proper '.netrc' file.

An excerpt from env(1):

Code:

NAME
     env - set and print environment

SYNOPSIS
     env [-i] [name=value ...] [utility [argument ...]]

DESCRIPTION
     env executes utility after modifying the environment as specified on the
     command line.  The option name=value specifies an environment variable,
     name, with a value of value.

1.8 Example of usage

Code:

$ mkdir FBSD80 ; release-fetch >FBSD80/.netrc
Trying 192.87.102.42...
-------------------------------------------------------------
Remember that ftp(1) uses the HOME environment variable to locate .netrc
Assuming you saved and/or edited .netrc in the current directory or '.',
use it as follows:

env HOME=. ftp -4a ftp://ftp.nl.freebsd.org 2>&1 | tee Logfile

$ cd FBSD80 ; ls -l .netrc
-rw-r--r--  1 j65nko  j65nko  501 Jan 16 20:45 .netrc

After editing or inspecting, paste the suggested command and press return.

Code:

$ env HOME=. ftp -4a ftp://ftp.nl.freebsd.org 2>&1 | tee Logfile

Trying 192.87.102.42...
Connected to ftp.nluug.nl.
220-Welcome to the FTP archive of SURFnet BV and
220-The Netherlands Unix Users Group (NLUUG).
     [snip]
331 Please specify the password.
230 Login successful.
prompt off
Interactive mode off.
! mkdir ISO
lcd ISO
Local directory now /home/j65nko/FREEBSD/FBSD80/ISO
cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0
250 Directory successfully changed.
mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256
local: 8.0-RELEASE-amd64-bootonly.iso remote: 8.0-RELEASE-amd64-bootonly.iso
150 Opening BINARY mode data connection for 8.0-RELEASE-amd64-bootonly.iso (47366144 bytes).
226 File send OK.
47366144 bytes received in 61.47 seconds (752.48 KB/s)
    [snip]
150 Opening BINARY mode data connection for src/susbin.inf (99 bytes).
226 File send OK.
99 bytes received in 0.20 seconds (0.49 KB/s)
! ln -s . 8.0-RELEASE 
quit
221 Goodbye.

$Id: Ftp_download_FBSD_with_netrc.xml,v 1.6 2010/01/17 03:11:32 j65nko Exp $
$Id: vbul-html.xsl,v 1.15 2010/01/16 00:58:03 j65nko Exp $

J65nko · #2 **(View Single Post)** 17th January 2010

Code:

#!/bin/sh
# $Id: release-fetch,v 1.5 2010/01/16 20:13:08 j65nko Exp $
# fetch FreeBSD release files and boot-only-iso with ftp and .netrc 

#  Copyright (c) 2006,2007,2008,2009 J65nko <Administrator daemonforums.org>
# 
#  Permission to use, copy, modify, and distribute this software for any
#  purpose with or without fee is hereby granted, provided that the above
#  copyright notice and this permission notice appear in all copies.
# 
#  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
#  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
#  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
#  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
#  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
#  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
#  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
# 

# -- start of user configurable settings

#SITE=ftp4.freebsd.org
#SITE=ftp2.dk.FreeBSD.org
SITE=ftp.nl.freebsd.org

RN='8.0'
#ARCH=i386
ARCH=amd64

MY_ISO_DIR="ISO"

# -- end of user configurable settings

RELEASE="${RN}-RELEASE"
DIR=/pub/FreeBSD/releases/${ARCH}/${RELEASE}
FILE=${ARCH}_${RELEASE}files.txt

# -------- bootonly.iso -----------------------
# ftp://ftp.nl.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.0/
# 8.0-RELEASE-i386-bootonly.iso

ISO_DIR="/pub/FreeBSD/ISO-IMAGES-${ARCH}/${RN}"
BOI="${RELEASE}-${ARCH}-bootonly.iso"
MD5="CHECKSUM.MD5"
SHA256="CHECKSUM.SHA256"

# -----------------------------------------
#
get_main_listing() {
    echo "ls ${DIR} ${FILE}" | ftp -4ai ${SITE} 1>&2 
}

get_main_listing

# select the release file directories 
SETS=$(awk '/^d/ { printf " %s", $9 }' ${FILE} )
# select the release documents
DOCS=$(awk '/^-r/ { printf " %s" , $9 }' ${FILE} )

# -- Create .netrc using 'here document'

cat <<END_OF_NETRC

machine ${SITE}

macdef init
prompt off
! mkdir ${MY_ISO_DIR}
lcd ${MY_ISO_DIR}
cd ${ISO_DIR} 
mget ${BOI} ${MD5} ${SHA256}
lcd ..
cd ${DIR}
mget ${DOCS} 
$ getdir ${SETS}
! ln -s . ${RELEASE} 
quit

macdef getdir
! mkdir \$i
mget \$i/*

END_OF_NETRC

# print Usage message on stderr
cat <<END >&2
-------------------------------------------------------------
Remember that ftp(1) uses the HOME environment variable to locate .netrc
Assuming you saved and/or edited .netrc in the current directory or '.',
use it as follows:

env HOME=. ftp -4a ftp://${SITE} 2>&1 | tee Logfile

END

J65nko · #3 **(View Single Post)** 17th January 2010

Downloads of the 'release-fetch' script and the guide in XHTML format

FBSD · #4 **(View Single Post)** 11th February 2010

When accessing one of the FreeBSD FTP services it is very common to get timed out before downloading of all the files is completed. Re-running the above shown .netrc will just start downloading from the beginning again.
The use of the mget command is what causes this. Changing mget to mreget will cause FTP to check the local host for the file and only download it if it's missing from the local host or it's date or size is different. This way you can use the same .netrc for the first execution and or to restart downloading files where your task was forced to end because the remote host timed you out.

I issue this command on the user root console command line for the first execution and all restarts.

ftp -v ftp2.jp.FreeBSD.org

This is the corrected .netrc file I have in /root directory

Code:

machine ftp2.jp.FreeBSD.org
login anonymous
password FBSD@home.com
macdef init
prompt off
cd /pub/FreeBSD/releases/i386/8.0-RELEASE
epsv4 off
mreget  ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM
mreget  README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css
$ getdir base catpages dict doc games info kernels manpages ports proflibs src
quit

macdef getdir
! mkdir $i
mreget $i/*

There are two things you may notice.
The use of the “epsv4 off” statement. FTP defaults to using extended passive mode. I could not get FTP to function through my ipfilter firewall until I forced FTP to use native passive mode by adding the "epsv4 off" which turns off extended passive mode.
Secondly I am using the mirror ftp2.jp.FreeBSD.org site. I tried a few other mirror FTP sites and was timeout after 3 to 5 minutes. This mirror has very little traffic and I got 80% done before timing out. Only one restart was necessary to complete the download. Recommend you try different mirrors located near your geographical location until you find one with low traffic.

J65nko · #5 **(View Single Post)** 11th February 2010

Re: epsv4 off

I also use epsv4 off this in the .netrc file of Automating OpenBSD snapshot downloads with a .netrc file and as well in Tracking OpenBSD snapshots with some simple shell scripts

I just wonder why I did not use it in this particular one

Re: interrrupted ftp transfers

I cannot remember to ever had an interrupted ftp transfer from my favorite ftp.nluug.nl mirror.

My rule for selecting ftp sites is: if there isn't a suitable in your own country, use one in a not so big country with a good infrastructure, like Denmark, Switzerland, Sweden, Norway etc.

If I would have an interrupted ftp transfer using a .netrc I would simply remove the lines with the files already downloaded.

But using mreget, as you suggested, is of course is a more pro-active solution.

Unfortunately the OpenBSD ftp client only supports reget. And in case you wonder what the OpenBSD ftp client has to do with this. Well, I use a local OpenBSD ftp server to install both FreeBSD and OpenBSD from, so that is why.

RE: location of .netrc

Although ftp uses the HOME environment variable to locate .netrc, this can easily be overridden by using env as I pointed out in my guide. This works on both Open- and FreeBSD.

Besides manipulating the HOME variable with env, the FreeBSD ftp client has two additional ways to select a .netrc

Using the environment variable NETRC. As described in the FBSD ftp manpage: " An alternate location of the .netrc file."
A command line switch/option described in the same man page as
Code:
```
-N netrc
	      Use netrc instead of ~/.netrc.
```

BTW I wonder why in your example you need to be root to ftp. Seen from a security viewpoint, a bad example

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Automating OpenBSD snapshot downloads with a .netrc file	J65nko	Guides	1	7th January 2010 03:09 AM
FreeBSD 8.0-RELEASE	Nightweaver	FreeBSD Installation and Upgrading	10	10th December 2009 12:12 AM
FreeBSD 7.2-RELEASE	vermaden	FreeBSD General	6	12th May 2009 08:33 PM
FreeBSD 6.2-RELEASE > 7.0-RELEASE Upgrade	Marci	FreeBSD Installation and Upgrading	2	23rd July 2008 02:10 PM
Automating ports update with portmaster	jaymax	FreeBSD Ports and Packages	3	12th June 2008 06:56 AM