DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides
Reload this Page Tracking OpenBSD snapshots with some simple sh scripts
FAQ Search Today's Posts Mark Forums Read

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 2nd December 2009
J65nko J65nko is offline
Administrator
  
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,055
Default Tracking OpenBSD snapshots with some simple sh scripts
Tracking OpenBSD snapshots with some simple shell scripts

Table of contents
  • 1 Introduction
  • 1.1 Legal stuff
  • 1.2 Dedication
  • 2 How to use the snapshot tracking scripts
  • 2.1 Retrieve ftp mirror status with the 'chk' script
  • 2.2 Seek confirmation from 'differ'
  • 2.2.1 Time stamp of SHA256 file
  • 2.3 Check mirror coherence of the packages with 'pkg-mirror-complete'
  • 2.3.1 A side-by-side comparision with 'pkg-differ'
  • 2.3.2 The Sparc64 packages
  • 2.4 Save ftp mirror data with 'backup'
  • 2.5 'show' time
  • 2.6 Summary of the snapshot tracking scripts
  • 3 Installation and configuration



Tracking OpenBSD snapshots with some simple shell scripts



1 Introduction

For several years I have been installing and testing OpenBSD snapshots. There is no fixed schedule for these snapshots. Usually daily a new snapshot appears on ftp.openbsd.org. Sometimes even two on a single day. Because of this irregularity and frequency some ftp mirrors never really catch up with ftp.openbsd.org.

So it is kind of a problem to find a ftp mirror with an up-to-date copy of the latest snapshot.

For some time I kept track of the snapshots on paper. At that time my main workstation was a 800MHz Pentium III and viewing ftp listings in firefox was slow. And I got tired of the 'eyeball' method to check whether a snapshot was completely mirrored.

The seven small scripts discussed here, help in storing and comparing ftp directory listings of ftp.openbsd.org, and it's mirror sites.

The main script which retrieve and stores information from the ftp sites, uses a '.netrc' file, as described in the ftp(1) manual page. Even when you do not intend to track OpenBSD snapshots, it may be interesting to read about the format and usage of '.netrc' to automate ftp access.



1.1 Legal stuff

Code:
#  Copyright (c) 2006,2007,2008,2009 J65nko <Administrator daemonforums.org>
# 
#  Permission to use, copy, modify, and distribute this software for any
#  purpose with or without fee is hereby granted, provided that the above
#  copyright notice and this permission notice appear in all copies.
# 
#  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
#  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
#  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
#  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
#  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
#  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
#  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#



1.2 Dedication

I would like to dedicate this guide to Joost, friend of my daughter, who also became one of my best friends. Although still being a student, he completely surprised me with a AMD64 motherboard, CPU and memory as birthday gift in 2007.

Joost, thanks for allowing me to step into the 64 bits world sooner than expected. Your gift prompted in several changes in my snapshot infrastructure, which was i386-only. This guide partially documents the result of these adaptations.



2 How to use the snapshot tracking scripts

First we show extensive examples how to use these scripts and how to analyze their output.



2.1 Retrieve ftp mirror status with the 'chk' script

Code:
$ ./chk
Connected to openbsd.sunsite.ualberta.ca.
220 openbsd.srv.ualberta.ca FTP server ready.
331 Guest login ok, send your email address as password.
230-   Welcome to ftp.openbsd.org at the University of Alberta 
230-   in Edmonton, Alberta, Canada.
230-   For other mirror sites visit http://www.openbsd.org/ftp.html
230- 

[snip]

230 Guest login ok, access restrictions apply.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256  SHA256
local: SHA256 remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (129,128,5,191,227,203)
150 Opening BINARY mode data connection for '/pub/OpenBSD/snapshots/i386/SHA256' (2162 bytes).
226 Transfer complete.
2162 bytes received in 0.34 seconds (6.13 KB/s)
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest"
227 Entering Passive Mode (129,128,5,191,196,78)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
2162 bytes received in 0.34 seconds (6.13 KB/s)
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest"
227 Entering Passive Mode (129,128,5,191,196,78)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
ls  /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice  >latest_pkg"
227 Entering Passive Mode (129,128,5,191,231,101)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
quit
We see here that an ftp connection is being made to ftp.openbsd.org in Canada. The commands issued are in blue. An explanation of the ones that save information:
  • get /pub/OpenBSD/snapshots/i386/SHA256 SHA256 retrieves the SHA256 checksum file.
  • The command ls /pub/OpenBSD/snapshots/i386/ "| ./fnice >latest" saves a listing of the installation file sets in the file 'latest'. The listing produced by the ftp server gets formatted nicely with a small awk script called 'fnice'.

    Code:
           97834 Nov 30 10:27 INSTALL.i386
       22356 Nov 30 10:28 INSTALL.linux
        2162 Nov 30 10:28 SHA256
    49042080 Nov 30 10:28 base46.tgz
     7490499 Nov 30 10:28 bsd
     7509983 Nov 30 10:28 bsd.mp
     6234592 Nov 30 10:28 bsd.rd
     6352896 Nov 30 10:28 cd46.iso
       44980 Nov 30 10:28 cdboot
        2048 Nov 30 10:28 cdbr
     3012608 Nov 30 10:28 cdemu46.iso
    91072033 Nov 30 10:28 comp46.tgz
      516752 Nov 30 10:28 etc46.tgz
     1474560 Nov 30 10:28 floppy46.fs
     1474560 Nov 30 10:28 floppyB46.fs
     1474560 Nov 30 10:28 floppyC46.fs
     2618096 Nov 30 10:28 game46.tgz
         269 Nov 30 10:28 index.txt
   249755648 Nov 30 10:28 install46.iso
     8469803 Nov 30 10:27 man46.tgz
     2936282 Nov 30 10:27 misc46.tgz
       53532 Nov 30 10:28 pxeboot
    15527631 Nov 29 08:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz
  • Likewise ls /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice >latest-pkg" stores a listing of the binary packages in the 'latest_pkg' file.

    A snippet from this 5902 lines long file:

    Code:
         1454425 Nov 29 12:19 postgresql-client-8.4.1p1.tgz
      396185 Nov 29 12:19 postgresql-contrib-8.4.1p1.tgz
     2176501 Nov 29 12:19 postgresql-docs-8.4.1p1.tgz
      405864 Nov 29 12:19 postgresql-jdbc-8.2.505p2.tgz
       55822 Nov 29 12:19 postgresql-jdbc-docs-8.2.505.tgz
      468689 Nov 29 12:19 postgresql-odbc-08.03.0400.tgz
     4356898 Nov 29 12:19 postgresql-server-8.4.1p1.tgz
       22650 Nov 29 12:19 postgresql_autodoc-1.40.tgz

The next connection is the second level mirror 'ftp.eu.openbsd.org' in Stockholm, Sweden.

Code:
Connected to ftp-prod-srv1.it.su.se.
220 ftp-prod-srv1.it.su.se FTP server ready.
331 Please specify the password.
230 Login successful.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256  SHA256-stockholm
local: SHA256-stockholm remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (130,237,164,232,203,198)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/SHA256 (2162 bytes).
226 File send OK.
2162 bytes received in 0.19 seconds (11.09 KB/s)
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest-stockholm"
227 Entering Passive Mode (130,237,164,232,204,181)
150 Here comes the directory listing.
226 Directory send OK.
ls  /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice  >latest_pkg-stockholm"
227 Entering Passive Mode (130,237,164,232,246,196)
150 Here comes the directory listing.
226 Directory send OK.
quit
221 Goodbye.
Here the 'SHA256' is being saved as 'SHA256-stockholm'. The listing of the installation file sets as 'latest-stockholm'. The binary packages listing is named 'latest_pkg-stockholm'.

After this the same procedure is repeated for the Vienna, Austria, second level mirror 'ftp.wu-wien.ac.at' and the Dutch 'ftp.nluug.nl'.
At the end we see the following summary:

Code:
Current date : 2009-11-30_22.11_UTC
NOW  date    : 2009-11-30_22:10_UTC
PREV date    : 2009-11-29_20:12_UTC
--------------- NOW ------------------------
 2162 Nov 29  16:06:57 2009 SHA256-vienna
 2162 Nov 29  16:06:57 2009 SHA256-stockholm
 2162 Nov 29  16:06:57 2009 SHA256-nluug
 2162 Nov 30  18:28:41 2009 SHA256
--------------- PREV -------------------
 2162 Nov 28  06:43:08 2009 SHA256-vienna
 2162 Nov 28  06:43:08 2009 SHA256-nluug
 2162 Nov 29  16:06:57 2009 SHA256-stockholm
 2162 Nov 29  16:06:57 2009 SHA256
--------------- X NOW ----------------------
    15527631 Nov 29 08:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz
--------------- X PREV ----------------------
    15527631 Nov 29 08:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz
--------------- PKG NOW --------------------
      121808 Nov 29 12:12 9libs-1.0p5.tgz
        9981 Nov 29 12:12 9menu-1.7p0.tgz
              .... [snip] ...............
      104955 Nov 29 12:22 zziplib-0.13.52.tgz
      125760 Nov 29 12:22 zzuf-0.12.tgz
--------------- PKG PREV --------------------
      121824 Nov 21 11:34 9libs-1.0p5.tgz
        9981 Nov 21 11:34 9menu-1.7p0.tgz
              .... [snip] ...............
      104999 Nov 21 11:43 zziplib-0.13.52.tgz
      125751 Nov 21 11:43 zzuf-0.12.tgz
The 'NOW' section shows the directory listings of the SHA266 files we just retrieved. The 'vienna', 'stockholm' and 'nluug' mirrors have a SHA256 file with the same "Nov 29 16:06:57" time stamp. The 'ftp.openbsd.org' SHA256 is more recent: Nov 30 18:28:41.

The section 'PREV' shows the results from the previous run of the 'chk' script.

While the 'NOW' and 'PREV' sections compare SHA256 files from multiple mirrors, the information of the sections X NOW/PREV and PKG NOW/PREV has a single source, the master site 'ftp.openbsd.org'.

The X NOW and X PREV parts show the dates of the X installation file sets of 'ftp.openbsd.org'. Both have the same "Nov 29 08:02" date and time.

Comparing the PKG NOW and PKG PREV information we notice a new set of binary packages from "Nov 29 12:12" has replaced the previous "Nov 21 11:34" ones.



2.2 Seek confirmation from 'differ'

To verify our eyeball check of the SHA256 files, we run the script 'differ', to compare the contents of all SHA256 files with the SHA256 of the main ftp site.

Code:
$ differ
  1 SHA256-nluug     : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : Files SHA256 and SHA256-stockholm differ
  3 SHA256-vienna    : Files SHA256 and SHA256-vienna differ
This confirms that none of the mirrors has the most recent snapshot yet.

An example where the Stockholm mirror has an identical SHA256 file:

Code:
  1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : =====Match====
  3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
For a more detailed view use differ -v

Code:
$ differ -v
  1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org  & stockholm ---
       97834 Nov 29 08:06 INSTALL.i386      |        97834 Nov 29 15:06 INSTALL.i386
       22356 Nov 29 08:06 INSTALL.linux     |        22356 Nov 29 15:06 INSTALL.linux
        2162 Nov 29 08:06 SHA256            |         2162 Nov 29 15:06 SHA256
    49042398 Nov 29 08:06 base46.tgz        |     49042398 Nov 29 15:06 base46.tgz
     7490469 Nov 29 08:06 bsd               |      7490469 Nov 29 15:06 bsd
     7509916 Nov 29 08:06 bsd.mp            |      7509916 Nov 29 15:06 bsd.mp
     6234592 Nov 29 08:06 bsd.rd            |      6234592 Nov 29 15:06 bsd.rd
     6352896 Nov 29 08:06 cd46.iso          |      6352896 Nov 29 15:06 cd46.iso
       44980 Nov 29 08:06 cdboot            |        44980 Nov 29 15:06 cdboot
        2048 Nov 29 08:06 cdbr              |         2048 Nov 29 15:06 cdbr
     3012608 Nov 29 08:06 cdemu46.iso       |      3012608 Nov 29 15:06 cdemu46.iso
    91072515 Nov 29 08:06 comp46.tgz        |     91072515 Nov 29 15:06 comp46.tgz
      516738 Nov 29 08:06 etc46.tgz         |       516738 Nov 29 15:06 etc46.tgz
     1474560 Nov 29 08:06 floppy46.fs       |      1474560 Nov 29 15:06 floppy46.fs
     1474560 Nov 29 08:06 floppyB46.fs      |      1474560 Nov 29 15:06 floppyB46.fs
     1474560 Nov 29 08:06 floppyC46.fs      |      1474560 Nov 29 15:06 floppyC46.fs
     2618027 Nov 29 08:06 game46.tgz        |      2618027 Nov 29 15:06 game46.tgz
         269 Nov 29 08:06 index.txt         |          269 Nov 29 15:06 index.txt
   249757696 Nov 29 08:06 install46.iso     |    249757696 Nov 29 15:06 install46.iso
     8469749 Nov 29 08:06 man46.tgz         |      8469749 Nov 29 15:06 man46.tgz
     2936230 Nov 29 08:06 misc46.tgz        |      2936230 Nov 29 15:06 misc46.tgz
       53532 Nov 29 08:06 pxeboot           |        53532 Nov 29 15:06 pxeboot
    15527631 Nov 29 08:02 xbase46.tgz       |     15527631 Nov 29 15:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz        |        68103 Nov 29 15:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz       |     35215028 Nov 29 15:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz       |     19874775 Nov 29 15:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz      |      2950471 Nov 29 15:02 xshare46.tgz
  3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
All Files dated Nov 29 08:06 and 08:02 have a corresponding time stamp of Nov 29 15:06 and 15:02. But note that this is not always the case!

Code:
$ differ -v
  1 SHA256-nluug     : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org  & stockholm ---
       97834 Nov 30 10:27 INSTALL.i386      |        97834 Nov 30 17:27 INSTALL.i386
       22356 Nov 30 10:28 INSTALL.linux     |        22356 Nov 30 17:28 INSTALL.linux
        2162 Nov 30 10:28 SHA256            |         2162 Nov 30 17:28 SHA256
    49042080 Nov 30 10:28 base46.tgz        |     49042080 Nov 30 17:28 base46.tgz
     7490499 Nov 30 10:28 bsd               |      7490499 Nov 30 17:28 bsd
     7509983 Nov 30 10:28 bsd.mp            |      7509983 Nov 30 17:28 bsd.mp
     6234592 Nov 30 10:28 bsd.rd            |      6234592 Nov 30 17:28 bsd.rd
     6352896 Nov 30 10:28 cd46.iso          |      6352896 Nov 30 17:28 cd46.iso
       44980 Nov 30 10:28 cdboot            |        44980 Nov 30 17:28 cdboot
        2048 Nov 30 10:28 cdbr              |         2048 Nov 30 17:28 cdbr
     3012608 Nov 30 10:28 cdemu46.iso       |      3012608 Nov 30 17:28 cdemu46.iso
    91072033 Nov 30 10:28 comp46.tgz        |     91072033 Nov 30 17:28 comp46.tgz
      516752 Nov 30 10:28 etc46.tgz         |       516752 Nov 30 17:28 etc46.tgz
     1474560 Nov 30 10:28 floppy46.fs       |      1474560 Nov 30 17:28 floppy46.fs
     1474560 Nov 30 10:28 floppyB46.fs      |      1474560 Nov 30 17:28 floppyB46.fs
     1474560 Nov 30 10:28 floppyC46.fs      |      1474560 Nov 29 15:06 floppyC46.fs
     2618096 Nov 30 10:28 game46.tgz        |      2618027 Nov 29 15:06 game46.tgz
         269 Nov 30 10:28 index.txt         |         1710 Nov 30 11:05 index.txt
   249755648 Nov 30 10:28 install46.iso     |    249757696 Nov 29 15:06 install46.iso
     8469803 Nov 30 10:27 man46.tgz         |      8469749 Nov 29 15:06 man46.tgz
     2936282 Nov 30 10:27 misc46.tgz        |      2936230 Nov 29 15:06 misc46.tgz
       53532 Nov 30 10:28 pxeboot           |        53532 Nov 29 15:06 pxeboot
    15527631 Nov 29 08:02 xbase46.tgz       |     15527631 Nov 29 15:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz        |        68103 Nov 29 15:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz       |     35215028 Nov 29 15:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz       |     19874775 Nov 29 15:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz      |      2950471 Nov 29 15:02 xshare46.tgz
  3 SHA256-vienna    : Files SHA256 and SHA256-vienna differ
Here the SHA256 has been synced, but but files like 'floppyC46.fs' and 'man46.tgz' have not.

A more obvious case where the 'Erlangen' mirror in Germany was still busy updating the X .tgz files. Note that this example dates from Dec 2008, when there were still MD5 checksums files instead of SHA256.

Code:
1 SHA256-calyx : Files SHA256 and SHA256-calyx differ
  2 SHA256-erlangen : =====Match====
--- sdiff of ftp.openbsd.org  & erlangen ---
       94576 Dec  9 18:53 INSTALL.amd64   |   10223616 Dec 11 04:25 .xserv44.tgz.G22368
         807 Dec  9 18:54 MD5             |      94576 Dec 10 02:53 INSTALL.amd64
    52334144 Dec  9 18:54 base44.tgz      |        807 Dec 10 02:54 MD5
     7223978 Dec  9 18:54 bsd             |   52334144 Dec 10 02:54 base44.tgz
     7342003 Dec  9 18:54 bsd.mp          |    7223978 Dec 10 02:54 bsd
     6315592 Dec  9 18:54 bsd.rd          |    7342003 Dec 10 02:54 bsd.mp
     6430720 Dec  9 18:54 cd44.iso        |    6315592 Dec 10 02:54 bsd.rd
       42940 Dec  9 18:54 cdboot          |    6430720 Dec 10 02:54 cd44.iso
        2048 Dec  9 18:54 cdbr            |      42940 Dec 10 02:54 cdboot
   102090065 Dec  9 18:54 comp44.tgz      |       2048 Dec 10 02:54 cdbr
      511724 Dec  9 18:54 etc44.tgz       |  102090065 Dec 10 02:54 comp44.tgz
     1474560 Dec  9 18:54 floppy44.fs     |     511724 Dec 10 02:54 etc44.tgz
     2706370 Dec  9 18:54 game44.tgz      |    1474560 Dec 10 02:54 floppy44.fs
         215 Jul  2 04:06 index.txt       |    2706370 Dec 10 02:54 game44.tgz
   260915200 Dec  9 18:54 install44.iso   |        215 Jul  2 12:06 index.txt
     8214814 Dec  9 18:53 man44.tgz       |  260915200 Dec 10 02:54 install44.iso
     2935078 Dec  9 18:53 misc44.tgz      |    8214814 Dec 10 02:53 man44.tgz
       53268 Dec  9 18:54 pxeboot         |    2935078 Dec 10 02:53 misc44.tgz
    11110781 Dec  9 18:44 xbase44.tgz     |      53268 Dec 10 02:54 pxeboot
       78441 Dec  9 18:44 xetc44.tgz      |   11110781 Dec 10 02:44 xbase44.tgz
    35579685 Dec  9 18:44 xfont44.tgz     |      78441 Dec 10 02:44 xetc44.tgz
    21467266 Dec  9 18:44 xserv44.tgz     |   35579685 Dec 10 02:44 xfont44.tgz
     2778752 Dec  9 18:44 xshare44.tgz    |   21467273 Dec  9 00:17 xserv44.tgz
                                          >    2778563 Dec  9 00:17 xshare44.tgz
The 'sdiff' output shows a '.xserv44.tgz.G22368' file, while there still is a 'xserv44.tgz'. Clearly a temporary file to store the newest version, before removing the old one.

After using 'chk' and 'differ' a couple of times, your non-artificial neural network becomes trained in spotting these differences. Of course this will be easier for American continentals, with only a couple of hours time difference, then the poor Europeans and Asians who have to deal with much larger time offsets.



2.2.1 Time stamp of SHA256 file

The X installation file sets 'xbase??.tgz', 'xetc??tgz', 'xfont??.tgz', 'xserv??.tgz' and 'xshare46.tgz' are usually created separately from the base install sets. Because the SHA256 file contains the checksums of both X and the base install sets this means that that SHA256 can have either the time stamp of the base install sets or the X sets.



2.3 Check mirror coherence of the packages with 'pkg-mirror-complete'

Without any tools, forming an opinion about the mirror status of the nearly 6000 pre-compiled packages, is very difficult. The small 'pkg-mirror-complete' script attempts to give an assessment of the coherence.

To check whether 'ftp.openbsd.org' has a coherent set of packages:

Code:
$ ./pkg-mirror-complete
  Unique dates from
./NOW/latest_pkg:
===============================
     1  11-21 11:34
     2  11-21 11:35
     3  11-21 11:36
     4  11-21 11:37
     5  11-21 11:38
     6  11-21 11:39
     7  11-21 11:40
     8  11-21 11:41
     9  11-21 11:42
    10  11-21 11:43
    11  11-29 04:05
===============================
Last entry:
 11-29 04:05 index.txt 456441

All files with last date:
 "11-29 04:05"
===============================
11-29 04:05 index.txt 456441
===============================
We see a sorted listing of the unique dates retrieved from 'latest_pkg'. The time stamps starting at 21st of Nov 11:34 up to 11:43 follow a regular pattern of 1 minute increments. Only dissonant is the last 11-29 04:05 time stamp. The only file with this time stamp is a file called 'index.txt', which usually is generated hours after a change of directory contents.

The same check but now for the mirror in Stockholm, Sweden

Code:
$ pkg-mirror-complete stockholm
  Unique dates from
./NOW/latest_pkg-stockholm:
===============================
     1  11-21 18:34
     2  11-21 18:35
     3  11-21 18:36
     4  11-21 18:37
     5  11-21 18:38
     6  11-21 18:39
     7  11-21 18:40
     8  11-21 18:41
     9  11-21 18:42
    10  11-21 18:43
    11  11-29 11:05
===============================
Last entry:
 11-29 11:05 index.txt 456441

All files with last date:
 "11-29 11:05"
===============================
11-29 11:05 index.txt 456441
===============================
The 'stockholm' parameter selects the 'latest_pkg-stockholm' as data source for 'pkg-mirror-complete'.

We see the same pattern, although the times show a time zone induced difference of 7 hours. We can rather safely assume that the packages of the Stockholm mirror are completely mirrored

The Dutch nluug mirror caught while updating the packages to the 24th of Dec ones:

Code:
$ pkg-mirror-complete nluug
  Unique dates from
./NOW/latest_pkg-nluug:
===============================
     1  11-15 16:07
     2  11-21 18:34
     3  11-21 18:35
     4  11-21 18:36
     5  11-21 18:37
     6  11-21 18:38
     7  11-21 18:39
     8  11-21 18:40
     9  11-21 18:41
    10  11-21 18:42
    11  11-21 18:43
    12  11-22 11:05
===============================
Last entry:
 11-22 11:05 index.txt 456376

All files with last date:
 "11-22 11:05"
===============================
11-22 11:05 index.txt 456376
===============================
In such cases we better wait before installing of upgrading the new packages.



2.3.1 A side-by-side comparision with 'pkg-differ'

Code:
$ pkg-differ nluug
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | ./NOW/latest_pkg-nluug:
===============================        ===============================
     1  11-21 11:34                  |      1   11-21 18:34
     2  11-21 11:35                  |      2   11-21 18:35
     3  11-21 11:36                  |      3   11-21 18:36
     4  11-21 11:37                  |      4   11-21 18:37
     5  11-21 11:38                  |      5   11-21 18:38
     6  11-21 11:39                  |      6   11-21 18:39
     7  11-21 11:40                  |      7   11-21 18:40
     8  11-21 11:41                  |      8   11-21 18:41
     9  11-21 11:42                  |      9   11-21 18:42
    10  11-21 11:43                  |     10   11-21 18:43
    11  11-28 04:05                  |     11   11-27 11:05
===============================        ===============================
Last entry:                            Last entry:
 11-28 04:05 index.txt 456441        |  11-27 11:05 index.txt 456441
                                       
All files with last date:              All files with last date:
 "11-28 04:05"                       |  "11-27 11:05"
===============================        ===============================
11-28 04:05 index.txt 456441         | 11-27 11:05 index.txt 456441
===============================        ===============================
As can be seen, this simply runs pkg-differ-complete for ftp.openbsd.org and a mirror.

The following shows the Vienna mirror not having completed syncing for both the Nov 15th and 21st package build.

Code:
$ pkg-differ vienna
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | ./NOW/latest_pkg-vienna:
===============================        ===============================
     1  11-21 11:34                  |      1   11-02 20:31
     2  11-21 11:35                  |      2   11-02 20:32
     3  11-21 11:36                  |      3   11-02 20:33
     4  11-21 11:37                  |      4   11-02 20:34
     5  11-21 11:38                  |      5   11-02 20:35
     6  11-21 11:39                  |      6   11-02 20:36
     7  11-21 11:40                  |      7   11-02 20:37
     8  11-21 11:41                  |      8   11-02 20:38
     9  11-21 11:42                  |      9   11-02 20:39
    10  11-21 11:43                  |     10   11-02 20:40
    11  11-28 04:05                  |     11   11-15 15:58
                                     >     12   11-15 15:59
                                                [snip]
                                     >     18   11-15 16:06
                                     >     19   11-15 16:07
                                     >     20   11-21 18:34
                                     >     21   11-21 18:35
                                                [snip]
                                     >     28   11-21 18:42
                                     >     29   11-21 18:43
                                     >     30   11-28 11:05
===============================        ===============================
Last entry:                            Last entry:
 11-28 04:05 index.txt 456441        |  11-28 11:05 index.txt 456441
                                       
All files with last date:              All files with last date:
 "11-28 04:05"                       |  "11-28 11:05"
===============================        ===============================
11-28 04:05 index.txt 456441         | 11-28 11:05 index.txt 456441
===============================        ===============================
Another example where the Vienna mirror seems to synchronize to a fresh directory.

Code:
$ pkg-differ vienna    
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | .//NOW/latest_pkg-vienna
===============================        ===============================
     1  10-26 10:43                  |      1   10-26 16:43
     2  10-26 10:44                  |      2   10-26 16:44
     3  10-26 10:45                  |      3   10-26 16:45
     4  10-26 10:46                  |      4   10-26 16:46
     5  10-26 10:47                  |      5   10-26 16:47
     6  10-26 10:48                  |      6   10-26 16:48
     7  10-26 10:49                  |      7   10-26 16:49
     8  10-26 10:50                  |      8   10-26 16:50
     9  10-26 10:51                  |      9   10-26 16:51
    10  10-26 10:52                  <
    11  10-28 04:05                  <
===============================        ===============================
Last entry:                            Last entry:
 10-28 04:05 index.txt 450746        |  10-26 16:51 tutris-1.0.1p1.tgz 5294
                                       
All files with last date:              All files with last date:
 "10-28 04:05"                       |  "10-26 16:51"
===============================        ===============================
10-28 04:05 index.txt 450746         | 10-26 16:51 tomboy-1.0.0.tgz 8839239
                                     > 10-26 16:51 tomcat-4.1.40.tgz 474034
                                     > 10-26 16:51 tomcat-5.5.27p1.tgz 4928
                                     > 10-26 16:51 tomcat-6.0.20.tgz 498773
                                     > 10-26 16:51 tomcat-admin-4.1.40p0.tg
                                     > 10-26 16:51 tomcat-admin-5.5.27p0.tg
                                     > 10-26 16:51 tt-rss-1.3.3.tgz 802688
                                       [snip]
                                     > 10-26 16:51 ttcp-1.12.tgz 9450
                                     > 10-26 16:51 ttyrec-1.0.8p0.tgz 11201
                                     > 10-26 16:51 turba-h3-2.3.2.tgz 23986
                                     > 10-26 16:51 tutris-1.0.1p1.tgz 52947
===============================        ===============================



2.3.2 The Sparc64 packages

The approach from the pkg-mirror-complete and pkg-differ scripts work well for most architectures. A major pain, however are the packages for the sparc64. Instead of the 10 to 15 different time stamps there are more than 2645!.

Code:
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | ./NOW/latest_pkg-stockholm:
===============================        ===============================
     1  10-25 06:58                  |      1   10-25 12:58
     2  10-25 06:59                  |      2   10-25 12:59
     3  10-25 07:00                  |      3   10-25 13:00
     4  10-25 07:01                  |      4   10-25 13:01
     5  10-25 07:02                  |      5   10-25 13:02
  [snip]
  2644  10-31 22:01                  |   2644   11-01 04:01
  2645  10-31 23:05                  |   2645   11-01 05:05
  2646  10-31 23:16                  |   2646   11-01 05:16
  2647  11-29 04:05                  |   2647   11-29 11:05
===============================        ===============================
Last entry:                            Last entry:
 11-29 04:05 index.txt 434648        |  11-29 11:05 index.txt 434648
                                       
All files with last date:              All files with last date:
 "11-29 04:05"                       |  "11-29 11:05"
===============================        ===============================
11-29 04:05 index.txt 434648         | 11-29 11:05 index.txt 434648
===============================        ===============================
The author, not familiar with the sparc64, just wonders if it would be feasible to touch the compiled packages to a single time stamp before uploading.



2.4 Save ftp mirror data with 'backup'

If a few hours later, we want to check for whether the Stockholm mirror has finished mirroring, we first save the current retrieved listings and SHA256 files with the 'backup' script.

Code:
$ backup -v

./backup: creating backup in S2009-11-30_22:21_UTC ...
drwxr-xr-x  2 j65  j65  512 Dec  1 18:55 S2009-11-30_22:21_UTC

./backup: linking PREV to S2009-11-30_22:21_UTC ...
lrwxr-xr-x  1 j65  j65  21 Dec  1 18:55 PREV -> S2009-11-30_22:21_UTC

total 1500
-rw-r--r--  1 j65  j65      21 Nov 30 23:21 DATE
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256-esat
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256-funkfeuer
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256-stockholm
-rw-r--r--  1 j65  j65     908 Nov 30 23:21 latest
-rw-r--r--  1 j65  j65     908 Nov 30 23:22 latest-esat
-rw-r--r--  1 j65  j65     908 Nov 30 23:22 latest-funkfeuer
-rw-r--r--  1 j65  j65     908 Nov 30 23:21 latest-stockholm
-rw-r--r--  1 j65  j65  166644 Nov 30 23:21 latest_pkg
-rw-r--r--  1 j65  j65  166644 Nov 30 23:22 latest_pkg-esat
-rw-r--r--  1 j65  j65  216589 Nov 30 23:22 latest_pkg-funkfeuer
-rw-r--r--  1 j65  j65  166644 Nov 30 23:21 latest_pkg-stockholm
This script copies the SHA256 files and ftp directory listings into a specially named new directory. The directory name consists of 'S' with the UTC time stamp appended. In this case 'S2009-11-30_22:21_UTC'.

After the files have been copied, the symbolic link PREV is pointed to 'S2009-11-30_22:21_UTC'. The directory NOW still has the original files. These will only be deleted during the next 'chk'.

'backup' will warn you if there already is a backup of the NOW directory.

Code:
$ ./backup
./backup : Backup already exists! Exiting ....



2.5 'show' time

In the explanation of the 'chk' output, we discussed a summary like this:

Code:
Current date : 2009-12-01_18.32_UTC
NOW  date    : 2009-11-29_20:20_UTC
PREV date    : 2009-11-27_20:49_UTC
--------------- NOW ------------------------
 2084 Nov 28  06:43:25 2009 SHA256-vienna
 2084 Nov 28  06:43:25 2009 SHA256-stockholm
 2084 Nov 28  06:43:25 2009 SHA256
--------------- PREV -------------------
 2084 Nov 22  18:13:09 2009 SHA256-vienna
 2084 Nov 27  01:11:32 2009 SHA256-stockholm
 2084 Nov 27  01:11:32 2009 SHA256
--------------- X NOW ----------------------
    17248633 Nov 27 22:43 xbase46.tgz
       69479 Nov 27 22:43 xetc46.tgz
    35217656 Nov 27 22:43 xfont46.tgz
    11389029 Nov 27 22:43 xserv46.tgz
     2952171 Nov 27 22:43 xshare46.tgz
--------------- X PREV ----------------------
    17247839 Nov 22 10:13 xbase46.tgz
       69495 Nov 22 10:13 xetc46.tgz
    35217765 Nov 22 10:13 xfont46.tgz
    11408354 Nov 22 10:13 xserv46.tgz
     2951722 Nov 22 10:13 xshare46.tgz
--------------- PKG NOW --------------------
      145282 Oct 28 09:48 9libs-1.0p5.tgz
       11950 Oct 29 12:46 9menu-1.7p0.tgz
              .... [snip] ...............
      111247 Oct 27 12:48 zziplib-0.13.52.tgz
      129514 Oct 27 09:45 zzuf-0.12.tgz
--------------- PKG PREV --------------------
      145282 Oct 28 09:48 9libs-1.0p5.tgz
       11950 Oct 29 12:46 9menu-1.7p0.tgz
              .... [snip] ...............
      111247 Oct 27 12:48 zziplib-0.13.52.tgz
      129514 Oct 27 09:45 zzuf-0.12.tgz
--------------------------------------
Actually the 'chk' script calls the 'show' script to produce this summary. You also can run it any time.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #2   (View Single Post)  
Old 2nd December 2009
J65nko J65nko is offline
Administrator
  
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,055
Default
2.6 Summary of the snapshot tracking scripts

As extensively shown in the previous sections, you will run the following scripts for the normal day-to-day operations.
  • 'chk' retrieves the SHA256 files and listings of the installation file sets and packages.
  • 'differ' shows which mirror has a copy of the latest snapshot. If a match is reported, use 'differ -v' or 'differ -v | less' to get a detailed view.
  • 'show' produces a comparative summary of the ftp mirror status.
  • 'pkg-mirror-complete' diagnoses the coherence of the packages directory.
  • 'pkg-differ' compares packages directory coherence of a mirror with the master ftp site.
  • 'backup' creates a backup of the previously retrieved data in the PREV directory.

The retrieved data from the ftp mirrors is in normal human readable text format. and is stored in the directories NOW, PREV and the S<time stamp>* backup directories. Feel free to use a browser like 'lynx' or plain old 'less' to investigate them.

Code:
$ ls -l
-rw-r--r--  1 j65  j65  7415 Nov 29 21:21 Logfile
drwxr-xr-x  2 j65  j65   512 Nov 29 21:21 NOW
lrwxr-xr-x  1 j65  j65    21 Nov 29 21:20 PREV -> S2009-11-27_20:49_UTC
drwxr-xr-x  2 j65  j65   512 Nov 21 21:49 S2009-11-05_21:53_UTC
drwxr-xr-x  2 j65  j65   512 Nov 27 21:49 S2009-11-21_20:49_UTC
drwxr-xr-x  2 j65  j65   512 Nov 29 21:20 S2009-11-27_20:49_UTC
lrwxr-xr-x  1 j65  j65     9 Jan  8  2009 backup -> ../backup
lrwxr-xr-x  1 j65  j65     6 Jan  8  2009 chk -> ../chk
lrwxr-xr-x  1 j65  j65    13 Jan  8  2009 chk-latest -> ../chk-latest
lrwxr-xr-x  1 j65  j65     9 Jan  8  2009 differ -> ../differ
lrwxr-xr-x  1 j65  j65     8 Jan  8  2009 fnice -> ../fnice
lrwxr-xr-x  1 j65  j65    11 Jan  8  2009 mk-netrc -> ../mk-netrc
lrwxr-xr-x  1 j65  j65    13 Jan  8  2009 pkg-differ -> ../pkg-differ
lrwxr-xr-x  1 j65  j65    22 Jan  8  2009 pkg-mirror-complete -> ../pkg-mirror-complete
-rw-r--r--  1 j65  j65  2150 Oct 22 03:26 rc.machines
lrwxr-xr-x  1 j65  j65     7 Jan  8  2009 show -> ../show
The NOW directory, containing the latest retrieved checksum files and listings

Code:
total 2356
-rw-r--r--  1 j65  j65      21 Nov 29 21:20 DATE
-rw-r--r--  1 j65  j65    2084 Nov 28 06:43 SHA256
-rw-r--r--  1 j65  j65    2084 Nov 28 06:43 SHA256-stockholm
-rw-r--r--  1 j65  j65    2084 Nov 28 06:43 SHA256-vienna
-rw-r--r--  1 j65  j65     953 Nov 29 21:20 latest
-rw-r--r--  1 j65  j65     953 Nov 29 21:20 latest-stockholm
-rw-r--r--  1 j65  j65     953 Nov 29 21:21 latest-vienna
-rw-r--r--  1 j65  j65  272857 Nov 29 21:20 latest_pkg
-rw-r--r--  1 j65  j65  272857 Nov 29 21:20 latest_pkg-stockholm
-rw-r--r--  1 j65  j65  272857 Nov 29 21:21 latest_pkg-vienna
In the next section we will give detailed steps how to install the scripts, and how to generate the '.netrc' file.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #3   (View Single Post)  
Old 2nd December 2009
J65nko J65nko is offline
Administrator
  
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,055
Default
3 Installation and configuration

Installing the OpenBSD snapshot tracking utilities

For the installation 'root' privileges are not needed. To prevent a possible name clash with other utilities, installation will be done in a subdirectory of your home directory, and not in any of the '*bin*' file hierarchies.
  1. Download the attached tarball 'OBSDsnapshot_tracking-1.05.tgz' and place it in your home directory.
  2. Create a directory, for instance SNAP.

    Code:
    $ mkdir SNAP
  3. Unpack the 'tgz' file in the newly created directory.

    Code:
    $ tar xvzf OBSDsnapshot_tracking-1.05.tgz -C SNAP
backup
chk
chk-latest
differ
fnice
mk-netrc
pkg-differ
pkg-mirror-complete
show
Makefile
rc.machines
  4. Change directory to SNAP.

    Code:
    $ cd SNAP ; ls -l
  1131 Dec  2 01:54 Makefile
   732 Oct 25 00:06 backup
   392 Dec 28  2008 chk
   710 Oct 29 01:33 chk-latest
  1409 Oct 23 03:20 differ
   174 Dec 29  2008 fnice
  2350 Oct 19 03:59 mk-netrc
  1214 Oct 19 04:00 pkg-differ
  1889 Oct 19 04:05 pkg-mirror-complete
  2645 Nov 29 04:01 rc.machines
  1822 Oct 19 04:20 show
  5. Run 'make' to setup a directory for the architecture(s), you are interested in.

    You specify subdirectories for each architecture you are interested in. There is no need to have these exactly match the official OpenBSD names, so for 'i386' you could use INTEL or I386. I prefer the upper-case version of the official names, which makes it easy to use the shell directory name completion feature.

    For the 'i386' architecture:

    Code:
    $ make ARCH=I386 install
    For 'i386', and 'amd64':

    Code:
    $ make ARCH='I386 AMD64' install

mkdir -p I386 I386/NOW
ln -sf NOW I386/PREV
cp rc.machines I386
ln -s  ../backup I386/backup
ln -s  ../chk I386/chk
ln -s  ../chk-latest I386/chk-latest
ln -s  ../differ I386/differ
ln -s  ../fnice I386/fnice
ln -s  ../mk-netrc I386/mk-netrc
ln -s  ../pkg-differ I386/pkg-differ
ln -s  ../pkg-mirror-complete I386/pkg-mirror-complete
ln -s  ../show I386/show
mkdir -p AMD64 AMD64/NOW
ln -sf NOW AMD64/PREV
cp rc.machines AMD64
ln -s  ../backup AMD64/backup
ln -s  ../chk AMD64/chk
ln -s  ../chk-latest AMD64/chk-latest
ln -s  ../differ AMD64/differ
ln -s  ../fnice AMD64/fnice
ln -s  ../mk-netrc AMD64/mk-netrc
ln -s  ../pkg-differ AMD64/pkg-differ
ln -s  ../pkg-mirror-complete AMD64/pkg-mirror-complete
ln -s  ../show AMD64/show
  6. Change directory to the architecture, that you wish to configure.

    After changing to the directory I386, you will see the following files and symbolic links:

    Code:
    $ cd I386 ; ls -l                                                                                                      
drwxr-xr-x  2 j65  j65   512 Dec  2 02:03 NOW
lrwxr-xr-x  1 j65  j65     3 Dec  2 02:03 PREV -> NOW
lrwxr-xr-x  1 j65  j65     9 Dec  2 02:03 backup -> ../backup
lrwxr-xr-x  1 j65  j65     6 Dec  2 02:03 chk -> ../chk
lrwxr-xr-x  1 j65  j65    13 Dec  2 02:03 chk-latest -> ../chk-latest
lrwxr-xr-x  1 j65  j65     9 Dec  2 02:03 differ -> ../differ
lrwxr-xr-x  1 j65  j65     8 Dec  2 02:03 fnice -> ../fnice
lrwxr-xr-x  1 j65  j65    11 Dec  2 02:03 mk-netrc -> ../mk-netrc
lrwxr-xr-x  1 j65  j65    13 Dec  2 02:03 pkg-differ -> ../pkg-differ
lrwxr-xr-x  1 j65  j65    22 Dec  2 02:03 pkg-mirror-complete -> ../pkg-mirror-complete
-rw-r--r--  1 j65  j65  2645 Dec  2 02:03 rc.machines
lrwxr-xr-x  1 j65  j65     7 Dec  2 02:03 show -> ../show
    • NOW : directory for storing most recent ftp mirror listings
    • PREV : symbolic link to the previous results. Because the first time there is no data from an earlier listings retrieval, it is simply linked to the NOW directory.
    • Symbolic links to the architecture independent scripts in the parent directory : 'backup', 'chk', 'chk-latest', 'differ', 'fnice', 'mk-netrc', 'pkg-differ', 'pkg-mirror-complete', 'show'.
    • The configuration file 'rc.machines'.
  7. Edit 'rc.machines'

    This file contains configuration settings for 'mk-netrc', the '.netrc' generator.

    Code:
    USER=anonymous
PASSWORD='assimilated@borg.net'
VERSION=snapshots

DIR=/pub/OpenBSD        # standard directory shortcut

MACHINES="
ftp.openbsd.org::${DIR}
ftp.eu.openbsd.org:stockholm:${DIR}
ftp.wu-wien.ac.at:vienna:${DIR}
"
    The 'USER' and 'PASSWORD' variables are used to create the 'login' and 'password' tokens. 'VERSION' and 'DIR' assist in setting up ftp directory paths like '/pub/OpenBSD/snapshots/i386/'.

    Variable 'MACHINES' defines records with three fields. The first is the ftp site name, needed for the 'machine' token of '.netrc'. Field number two is an identifier to name the listings retrieved from that particular site. The third and last field specifies the directory path leading to the OpenBSD section.

    The comments in 'rc.machines' also describe and explain the format.

    Code:
    # Separate fields with ':'
#
# - field 1: name of ftp site (without 'ftp://')
#
# - field 2: name of identifier to be appended to files.
#            e.g. id 'vienna' will create: SHA256-vienna, latest-vienna AND latest_pkg-vienna
#            EXCEPTION: 'ftp.openbsd.org' has no identifier!
#
#            Some examples:
#
#                   2162 Nov 28 06:43 SHA256
#                   2162 Nov 27 00:33 SHA256-nluug
#                   2162 Nov 28 06:43 SHA256-stockholm
#                   2162 Nov 28 06:43 SHA256-vienna
#                    981 Nov 29 00:58 latest
#                    981 Nov 29 00:58 latest-nluug
#                    981 Nov 29 00:58 latest-stockholm
#                    981 Nov 29 00:58 latest-vienna
#                 286617 Nov 29 00:58 latest_pkg
#                 286617 Nov 29 00:58 latest_pkg-nluug
#                 286617 Nov 29 00:58 latest_pkg-stockholm
#                 300676 Nov 29 00:58 latest_pkg-vienna
#
# - field 3: path to OpenBSD directory on ftp site (normally /pub/OpenBSD) 

# Some exceptions to the standard ftp directory layout: 
#
# ftp.freenet.de:freenet:/pub/ftp.openbsd.org/pub/OpenBSD
# ftp.belnet.be:belnet:/packages/openbsd
    Edit the 'MACHINES' variable to select mirrors that are geographically or connection wise close to you. Use at least a couple of the second-level mirrors. For some architecture more then one snapshot a day is been created. Some 3rd level ftp mirrors are continuously lagging behind one or snapshots for the installation files.

    It is still useful to track these lagging mirrors for the packages, as these are not generated daily.
  8. Run 'mk-netrc' to create a '.netrc' file.

    After you have edited 'rc.machines' it is time to generate the '.netrc' file. 'ftp(1)' describes how this file can automate the ftp login and the retrieval of directory listings and files.

    In invoking 'mk-netrc' you have to use the official OpenBSD architecture names, using the proper case, as these are used as directory names in the ftp file lay-out.

    Code:
    $ ./make-netrc i386
./mk-netrc : sourcing rc.machines...
machine ftp.openbsd.org login anonymous password assimilated@borg.net

macdef init
prompt off
epsv4 off
preserve on
get /pub/OpenBSD/snapshots/i386/SHA256  SHA256
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest"
ls  /pub/OpenBSD/snapshots/packages/386/ "| ./fnice  >latest_pkg"
quit

machine ftp.eu.openbsd.org login anonymous password assimilated@borg.net

macdef init
prompt off
epsv4 off
preserve on
get /pub/OpenBSD/snapshots/i386/SHA256  SHA256-stockholm
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest-stockholm"
ls  /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice  >latest_pkg-stockholm"
quit
    Compare the paths of the ftp get and ls commands, with the ones you see in your browser or ftp program. If it looks goods, redirect the output to '.netrc'. You do not have to worry about the 'sourcing rc.machines...' message. Because it is printed on standard error and not on standard out it will not redirected to '.netrc'.

    Code:
    $  ./make-netrc i386 >.netrc
./mk-netrc : sourcing rc.machines...
    We now pause for a security warning from the ftp man page:

    Code:
    Note that if this token (the password) is present in the .netrc
file for any user other than anonymous, ftp will abort the auto-login
process if the .netrc is readable by anyone besides the user.
    In our case the newly generated '.netrc' file only contains passwords for anonymous users, so ftp will not mind if '.netrc' is group and world readable. But as it is always good to acquire good habits:

    Code:
    $ chmod g=,o= .netrc
    Or:

    Code:
    $ chmod 600 .netrc ; ls -l .netrc
-rw-------  1 j65  j65  1248 Dec  2 02:55 .netrc

At this point we have concluded our configuration of the snapshot tracking utilities for the 'i386' architecture. If do not intend to track snapshots for other architectures you are now ready to run 'chk'.

If you use other architectures, you still have to configure these. Each processor architecture needs it own '.netrc'. Assuming you used make ARCH="I386 AMD64 SPARC64" install, you will have to repeat steps 6 to 8, but this time in the directories AMD64 and SPARC64.



$Id: OBSDsnapshot-tracking.xml,v 1.14 2009/12/02 04:26:05 j65nko Exp $
$Id: book-vbul-html.xsl,v 1.3 2008/12/24 02:59:45 j65nko Exp $
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #4   (View Single Post)  
Old 2nd December 2009
J65nko J65nko is offline
Administrator
  
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 4,055
Default
Download the guide (one single file in HTML format) and scripts
Attached Files
File Type: html OBSDsnapshot-tracking.html (55.3 KB, 144 views)
File Type: tgz OBSDsnapshot_tracking-1.05.tgz (5.5 KB, 104 views)
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Reply

Tags
.netrc, ftp, openbsd snapshot

« Previous Thread | Next Thread »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to update NetBSD using snapshots s0xxx Guides 5 23rd November 2009 04:30 PM
DJB daemontools: compile and install scripts (OpenBSD) J65nko Guides 2 13th November 2009 10:57 AM
Executing web scripts : can't! erehwon OpenBSD General 3 3rd November 2009 10:02 PM
SHA256 replaces MD5 in OpenBSD snapshots J65nko OpenBSD General 3 6th May 2009 04:36 PM
Periodic scripts for mail scripts stukov FreeBSD General 2 8th March 2009 07:51 PM


All times are GMT. The time now is 10:20 PM.

DaemonForums.org RSS Feeds - Contact Us - DaemonForums.org - Archive - Privacy Statement - Top

Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick