![]() |
|
Guides All Guides and HOWTO's. |
|
Thread Tools | Display Modes |
|
|||
![]()
1. Automating FreeBSD release downloads with a '.netrc' file
1.1 Introduction Many years ago for the first time, I did a ftp install of FreeBSD 4.x. using the two floppies. I instructed 'sysinstall' to retrieve the installation files from a FreeBSD ftp mirror on the internet. Previously I had installed FreeBSD a few times from CD. At that time I was using a new ISDN modem, which was fast with 64Kb, at least compared with the 14k4 old modem. Because the system did not boot, I must have made a mistake somewhere. Not looking forward to a couple of repeated ftp downloads with the ISDN modem, I decided to download the release files once and for all on a FreeBSD Pentium-Pro box. Some simple configuration steps turned the Pentium-Pro into a local ftp server. This method has become my favorite way to install FreeBSD as well as OpenBSD. First download the files making up the FreeBSD release to a ftp server in my local network. Boot 'sysinstall' with the install floppies, and install FreeBSD by fetching the installation files from the local ftp server. A nice side effect is speed. Transfer over the network turned out to be much faster than reading from a 10x speed CD on a Pentium I box. 1.2 File hierarchy of a FreeBSD release In the standardized ftp structure, we find the release files in '/pub/FreeBSD/releases/<arch>/<release>/', where <arch> stands for the hardware architecture like 'i386' or 'amd64', and <release> is a release tag, e.g. '7.2-RELEASE' or '8.0-RELEASE'. Code:
lrwxr-xr-x 1 1006 3000 1 May 1 2009 7.2-RELEASE -> . -r--r--r-- 1 1006 3000 4877 May 1 2009 ERRATA.HTM -r--r--r-- 1 1006 3000 3514 May 1 2009 ERRATA.TXT -r--r--r-- 1 1006 3000 186567 May 1 2009 HARDWARE.HTM -r--r--r-- 1 1006 3000 114717 May 1 2009 HARDWARE.TXT -r--r--r-- 1 1006 3000 19736 May 1 2009 README.HTM -r--r--r-- 1 1006 3000 14311 May 1 2009 README.TXT -r--r--r-- 1 1006 3000 60037 May 1 2009 RELNOTES.HTM -r--r--r-- 1 1006 3000 31407 May 1 2009 RELNOTES.TXT drwxrwxr-x 2 1006 3000 1024 May 1 2009 base drwxrwxr-x 2 1006 3000 512 May 1 2009 catpages -rw-r--r-- 1 1006 3000 25 May 1 2009 cdrom.inf drwxrwxr-x 2 1006 3000 512 May 1 2009 dict drwxrwxr-x 2 1006 3000 1536 May 1 2009 doc -r--r--r-- 1 1006 3000 3704 May 1 2009 docbook.css drwxrwxr-x 2 1006 3000 512 May 1 2009 floppies drwxrwxr-x 2 1006 3000 512 May 1 2009 games drwxrwxr-x 2 1006 3000 512 May 1 2009 info drwxrwxr-x 2 1006 3000 1024 May 1 2009 kernels drwxrwxr-x 2 1006 3000 512 May 1 2009 manpages lrwxr-xr-x 1 1006 3000 40 May 2 2009 packages -> ../../../ports/i386/packages-7.2-release drwxrwxr-x 2 1006 3000 512 May 1 2009 ports drwxrwxr-x 2 1006 3000 512 May 1 2009 proflibs drwxrwxr-x 2 1006 3000 2560 May 1 2009 src
Downloading all these files with the command line 'ftp' would be rather tedious. For a long time I used 'wget' to fetch the files. A couple of years later I discovered how to use a '.netrc' file to script ftp commands. 1.3 A '.netrc' file to download a FreeBSD release A full explanation of the '.netrc' tokens can be found in the ftp(1) man page. Code:
machine ftp2.dk.FreeBSD.org macdef init prompt off cd /pub/FreeBSD/releases/amd64/8.0-RELEASE mget ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css $ getdir base catpages dict doc games info kernels manpages ports proflibs src ! ln -s . 8.0-RELEASE quit macdef getdir ! mkdir $i mget $i/* In the following invocation of the ftp command, the '-4' instructs the ftp client to only use IPv4 and not try IPv6, the 'a' option prescribes an 'anonymous' login. Code:
$ ftp -4a ftp://ftp2.dk.FreeBSD.org
1.4 Transcript of '.netrc' processing Code:
$ ftp -4a ftp://ftp2.dk.FreeBSD.org Connected to heset.dkuug.dk. 220 FTP server ready. 331 Guest login ok, send your email address as password. 230- Velcome to the DKUUG software archive. 230- If you encounter any problems please send mail to ftpadmin@dkuug.dk [snip] 230 Guest login ok, access restrictions apply. prompt off Interactive mode off. cd /pub/FreeBSD/releases/amd64/8.0-RELEASE 250 CWD command successful. mget ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css local: ERRATA.HTM remote: ERRATA.HTM 150 Opening BINARY mode data connection for 'ERRATA.HTM' (4932 bytes). 226 Transfer complete. 4932 bytes received in 0.19 seconds (25.12 KB/s) [snip] local: docbook.css remote: docbook.css 150 Opening BINARY mode data connection for 'docbook.css' (3704 bytes). 226 Transfer complete. 3704 bytes received in 0.19 seconds (18.88 KB/s) Code:
$ getdir base catpages dict doc games info kernels lib32 manpages ports proflibs src ! mkdir base mget base/* local: base/CHECKSUM.MD5 remote: base/CHECKSUM.MD5 150 Opening BINARY mode data connection for 'base/CHECKSUM.MD5' (2163 bytes). 226 Transfer complete. 2163 bytes received in 0.19 seconds (11.04 KB/s) local: base/CHECKSUM.SHA256 remote: base/CHECKSUM.SHA256 150 Opening BINARY mode data connection for 'base/CHECKSUM.SHA256' (3703 bytes). 226 Transfer complete. 3703 bytes received in 0.19 seconds (18.93 KB/s) local: base/base.aa remote: base/base.aa 150 Opening BINARY mode data connection for 'base/base.aa' (1425408 bytes). 226 Transfer complete. 1425408 bytes received in 1.95 seconds (713.33 KB/s) [snip] local: base/install.sh remote: base/install.sh 150 Opening BINARY mode data connection for 'base/install.sh' (442 bytes). 226 Transfer complete. 442 bytes received in 0.20 seconds (2.20 KB/s) Code:
! mkdir src mget src/* local: src/CHECKSUM.MD5 remote: src/CHECKSUM.MD5 150 Opening BINARY mode data connection for 'src/CHECKSUM.MD5' (5758 bytes). 226 Transfer complete. 5758 bytes received in 0.20 seconds (28.05 KB/s) local: src/CHECKSUM.SHA256 remote: src/CHECKSUM.SHA256 150 Opening BINARY mode data connection for 'src/CHECKSUM.SHA256' (9678 bytes). 226 Transfer complete. 9678 bytes received in 0.19 seconds (49.59 KB/s) local: src/install.sh remote: src/install.sh 150 Opening BINARY mode data connection for 'src/install.sh' (942 bytes). 226 Transfer complete. 942 bytes received in 0.19 seconds (4.81 KB/s) local: src/sbase.aa remote: src/sbase.aa 150 Opening BINARY mode data connection for 'src/sbase.aa' (75919 bytes). 226 Transfer complete. 75919 bytes received in 0.21 seconds (349.05 KB/s) [snip] local: src/susbin.ac remote: src/susbin.ac 150 Opening BINARY mode data connection for 'src/susbin.ac' (120619 bytes). 226 Transfer complete. 120619 bytes received in 0.28 seconds (427.11 KB/s) local: src/susbin.inf remote: src/susbin.inf 150 Opening BINARY mode data connection for 'src/susbin.inf' (99 bytes). 226 Transfer complete. 99 bytes received in 0.20 seconds (0.48 KB/s) ! ln -s . 8.0-RELEASE quit 221 Goodbye $ 1.5 '.netrc' commands to download boot-only-iso image Code:
! mkdir iso lcd iso cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256 Code:
mget remote-files Expand the remote-files on the remote machine and do a get for each file name thus produced. ...... Files are transferred into the local working directory, which can be changed with `lcd directory'; new local directories can be created with `! mkdir directory' After having set the correct directory for both local and remote, the mget downloads the iso, as well as the two checksum files. A transcript: Code:
! mkdir iso lcd iso Local directory now /home/j65nko/FREEBSD/amd64_80/iso cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 250 CWD command successful. mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256 local: 8.0-RELEASE-amd64-bootonly.iso remote: 8.0-RELEASE-amd64-bootonly.iso 150 Opening BINARY mode data connection for '8.0-RELEASE-amd64-bootonly.iso' (47366144 bytes). 226 Transfer complete. 47366144 bytes received in 60.97 seconds (758.67 KB/s) local: CHECKSUM.MD5 remote: CHECKSUM.MD5 150 Opening BINARY mode data connection for 'CHECKSUM.MD5' (351 bytes). 226 Transfer complete. 351 bytes received in 0.20 seconds (1.75 KB/s) local: CHECKSUM.SHA256 remote: CHECKSUM.SHA256 150 Opening BINARY mode data connection for 'CHECKSUM.SHA256' (526 bytes). 226 Transfer complete. 526 bytes received in 0.20 seconds (2.57 KB/s 1.6 A merge of the two '.netrc' files The boot-only-iso is fetched first, so the impatient already can burn the image file to CD, while the release files are still being downloaded. Code:
machine ftp2.dk.FreeBSD.org macdef init prompt off ! mkdir iso lcd iso cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256 lcd .. cd /pub/FreeBSD/releases/amd64/8.0-RELEASE mget ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css $ getdir base catpages dict doc games info kernels lib32 manpages ports proflibs src ! ln -s . 8.0-RELEASE quit macdef getdir ! mkdir $i mget $i/* 1.7 The '.netrc' file generator script By only editing the shell variables SITE, RN, and ARCH, a different ftp mirror site, release or processor architecture can be chosen. The MY_ISO_DIR variable allows customization of the destination directory of the boot-only-iso, and it's checksum files. Code:
# -- start of user configurable settings #SITE=ftp4.freebsd.org #SITE=ftp2.dk.FreeBSD.org SITE=ftp.nl.freebsd.org RN='8.0' #ARCH=i386 ARCH=amd64 MY_ISO_DIR="ISO" # -- end of user configurable settings Code:
RELEASE="${RN}-RELEASE" DIR=/pub/FreeBSD/releases/${ARCH}/${RELEASE} FILE=${ARCH}_${RELEASE}files.txt # -------- bootonly.iso ----------------------- # ftp://ftp.nl.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.0/ # 8.0-RELEASE-i386-bootonly.iso ISO_DIR="/pub/FreeBSD/ISO-IMAGES-${ARCH}/${RN}" BOI="${RELEASE}-${ARCH}-bootonly.iso" MD5="CHECKSUM.MD5" SHA256="CHECKSUM.SHA256" Code:
$ sh -vx release-fetch # -- start of user configurable settings #SITE=ftp4.freebsd.org #SITE=ftp2.dk.FreeBSD.org SITE=ftp.nl.freebsd.org + SITE=ftp.nl.freebsd.org RN='8.0' + RN=8.0 #ARCH=i386 ARCH=amd64 + ARCH=amd64 MY_ISO_DIR="ISO" + MY_ISO_DIR=ISO # -- end of user configurable settings RELEASE="${RN}-RELEASE" + RELEASE=8.0-RELEASE DIR=/pub/FreeBSD/releases/${ARCH}/${RELEASE} + DIR=/pub/FreeBSD/releases/amd64/8.0-RELEASE FILE=${ARCH}_${RELEASE}files.txt + FILE=amd64_8.0-RELEASEfiles.txt # -------- bootonly.iso ----------------------- # ftp://ftp.nl.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.0/ # 8.0-RELEASE-i386-bootonly.iso ISO_DIR="/pub/FreeBSD/ISO-IMAGES-${ARCH}/${RN}" + ISO_DIR=/pub/FreeBSD/ISO-IMAGES-amd64/8.0 BOI="${RELEASE}-${ARCH}-bootonly.iso" + BOI=8.0-RELEASE-amd64-bootonly.iso MD5="CHECKSUM.MD5" + MD5=CHECKSUM.MD5 SHA256="CHECKSUM.SHA256" + SHA256=CHECKSUM.SHA256 The method used is not very well known, but is simple. Echo a ftp ls command to stdout. Have ftp receive this command through a pipe as standard input to execute. Code:
get_main_listing() { echo "ls ${DIR} ${FILE}" | ftp -4ai ${SITE} 1>&2 } get_main_listing Code:
echo "ls /pub/FreeBSD/releases/amd64/8.0-RELEASE \ amd64_8.0-RELEASEfiles.txt" | ftp -4ai ftp.nl.freebsd.org 1>&2 Code:
lrwxrwxrwx 1 500 450 1 Nov 23 07:53 8.0-RELEASE -> . -r--r--r-- 1 500 450 4932 Nov 21 15:10 ERRATA.HTM -r--r--r-- 1 500 450 3563 Nov 21 15:10 ERRATA.TXT -r--r--r-- 1 500 450 190670 Nov 21 15:10 HARDWARE.HTM -r--r--r-- 1 500 450 115402 Nov 21 15:10 HARDWARE.TXT -r--r--r-- 1 500 450 19867 Nov 21 15:10 README.HTM -r--r--r-- 1 500 450 14336 Nov 21 15:10 README.TXT -r--r--r-- 1 500 450 10512 Nov 21 15:10 RELNOTES.HTM -r--r--r-- 1 500 450 7545 Nov 21 15:10 RELNOTES.TXT drwxrwxr-x 2 500 450 4096 Nov 22 20:04 base drwxrwxr-x 2 500 450 4096 Nov 22 20:06 catpages -rw-r--r-- 1 500 450 25 Nov 21 15:10 cdrom.inf drwxrwxr-x 2 500 450 4096 Nov 22 20:06 dict drwxrwxr-x 2 500 450 4096 Nov 22 20:06 doc -r--r--r-- 1 500 450 3704 Nov 21 15:10 docbook.css drwxrwxr-x 2 500 450 4096 Nov 22 20:06 games drwxrwxr-x 2 500 450 4096 Nov 22 20:07 info drwxrwxr-x 2 500 450 4096 Nov 22 20:19 kernels drwxrwxr-x 2 500 450 4096 Nov 22 20:21 lib32 drwxrwxr-x 2 500 450 4096 Nov 22 20:23 manpages lrwxrwxrwx 1 500 450 41 Nov 23 07:53 packages -> ../../../ports/am d64/packages-8.0-release drwxrwxr-x 2 500 450 4096 Nov 22 20:32 ports drwxrwxr-x 2 500 450 4096 Nov 22 20:33 proflibs drwxrwxr-x 2 500 450 4096 Nov 22 20:59 src Code:
# select the release file directories SETS=$(awk '/^d/ { printf " %s", $9 }' ${FILE} ) # select the release documents DOCS=$(awk '/^-r/ { printf " %s" , $9 }' ${FILE} ) Code:
# select the release file directories SETS=$(awk '/^d/ { printf " %s", $9 }' ${FILE} ) + awk /^d/ { printf " %s", $9 } amd64_8.0-RELEASEfiles.txt + SETS= base catpages dict doc games info kernels lib32 manpages \ ports proflibs src # select the release documents DOCS=$(awk '/^-r/ { printf " %s" , $9 }' ${FILE} ) + awk /^-r/ { printf " %s" , $9 } amd64_8.0-RELEASEfiles.txt + DOCS= ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM \ README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf docbook.css Finally after having gathered all necessary information, the script can generate the '.netrc' file. Code:
# -- Create .netrc using 'here document' cat <<END_OF_NETRC machine ${SITE} macdef init prompt off ! mkdir ${MY_ISO_DIR} lcd ${MY_ISO_DIR} cd ${ISO_DIR} mget ${BOI} ${MD5} ${SHA256} lcd .. cd ${DIR} mget ${DOCS} $ getdir ${SETS} ! ln -s . ${RELEASE} quit macdef getdir ! mkdir \$i mget \$i/* END_OF_NETRC Code:
+ cat + << END_OF_NETRC machine ftp.nl.freebsd.org macdef init prompt off ! mkdir ISO lcd ISO cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256 lcd .. cd /pub/FreeBSD/releases/amd64/8.0-RELEASE mget ERRATA.HTM ERRATA.TXT HARDWARE.HTM HARDWARE.TXT README.HTM README.TXT RELNOTES.HTM RELNOTES.TXT cdrom.inf do cbook.css $ getdir base catpages dict doc games info kernels lib32 manpages ports proflibs src ! ln -s . 8.0-RELEASE quit macdef getdir ! mkdir $i mget $i/* Code:
# print Usage message on stderr cat <<END >&2 ------------------------------------------------------------- Remember that ftp(1) uses the HOME environment variable to locate .netrc Assuming you saved and/or edited .netrc in the current directory or '.', use it as follows: env HOME=. ftp -4a ftp://${SITE} 2>&1 | tee Logfile END + cat + << END + >&2 ------------------------------------------------------------- Remember that ftp(1) uses the HOME environment variable to locate .netrc Assuming you saved and/or edited .netrc in the current directory or '.', use it as follows: env HOME=. ftp -4a ftp://ftp.nl.freebsd.org 2>&1 | tee Logfile An excerpt from env(1): Code:
NAME env - set and print environment SYNOPSIS env [-i] [name=value ...] [utility [argument ...]] DESCRIPTION env executes utility after modifying the environment as specified on the command line. The option name=value specifies an environment variable, name, with a value of value. 1.8 Example of usage Code:
$ mkdir FBSD80 ; release-fetch >FBSD80/.netrc Trying 192.87.102.42... ------------------------------------------------------------- Remember that ftp(1) uses the HOME environment variable to locate .netrc Assuming you saved and/or edited .netrc in the current directory or '.', use it as follows: env HOME=. ftp -4a ftp://ftp.nl.freebsd.org 2>&1 | tee Logfile $ cd FBSD80 ; ls -l .netrc -rw-r--r-- 1 j65nko j65nko 501 Jan 16 20:45 .netrc Code:
$ env HOME=. ftp -4a ftp://ftp.nl.freebsd.org 2>&1 | tee Logfile Trying 192.87.102.42... Connected to ftp.nluug.nl. 220-Welcome to the FTP archive of SURFnet BV and 220-The Netherlands Unix Users Group (NLUUG). [snip] 331 Please specify the password. 230 Login successful. prompt off Interactive mode off. ! mkdir ISO lcd ISO Local directory now /home/j65nko/FREEBSD/FBSD80/ISO cd /pub/FreeBSD/ISO-IMAGES-amd64/8.0 250 Directory successfully changed. mget 8.0-RELEASE-amd64-bootonly.iso CHECKSUM.MD5 CHECKSUM.SHA256 local: 8.0-RELEASE-amd64-bootonly.iso remote: 8.0-RELEASE-amd64-bootonly.iso 150 Opening BINARY mode data connection for 8.0-RELEASE-amd64-bootonly.iso (47366144 bytes). 226 File send OK. 47366144 bytes received in 61.47 seconds (752.48 KB/s) [snip] 150 Opening BINARY mode data connection for src/susbin.inf (99 bytes). 226 File send OK. 99 bytes received in 0.20 seconds (0.49 KB/s) ! ln -s . 8.0-RELEASE quit 221 Goodbye. $Id: Ftp_download_FBSD_with_netrc.xml,v 1.6 2010/01/17 03:11:32 j65nko Exp $
$Id: vbul-html.xsl,v 1.15 2010/01/16 00:58:03 j65nko Exp $
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump |
Tags |
.netrc, .netrc macdef, .netrc macro, ftp scripting, mreget, reget |
Thread Tools | |
Display Modes | |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Automating OpenBSD snapshot downloads with a .netrc file | J65nko | Guides | 1 | 7th January 2010 03:09 AM |
FreeBSD 8.0-RELEASE | Nightweaver | FreeBSD Installation and Upgrading | 10 | 10th December 2009 12:12 AM |
FreeBSD 7.2-RELEASE | vermaden | FreeBSD General | 6 | 12th May 2009 08:33 PM |
FreeBSD 6.2-RELEASE > 7.0-RELEASE Upgrade | Marci | FreeBSD Installation and Upgrading | 2 | 23rd July 2008 02:10 PM |
Automating ports update with portmaster | jaymax | FreeBSD Ports and Packages | 3 | 12th June 2008 06:56 AM |