DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #1   (View Single Post)  
Old 2nd December 2009
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,142
Thanked 182 Times in 149 Posts
Default Tracking OpenBSD snapshots with some simple sh scripts

Tracking OpenBSD snapshots with some simple shell scripts

Table of contents
  • 1 Introduction
  • 1.1 Legal stuff
  • 1.2 Dedication
  • 2 How to use the snapshot tracking scripts
  • 2.1 Retrieve ftp mirror status with the 'chk' script
  • 2.2 Seek confirmation from 'differ'
  • 2.2.1 Time stamp of SHA256 file
  • 2.3 Check mirror coherence of the packages with 'pkg-mirror-complete'
  • 2.3.1 A side-by-side comparision with 'pkg-differ'
  • 2.3.2 The Sparc64 packages
  • 2.4 Save ftp mirror data with 'backup'
  • 2.5 'show' time
  • 2.6 Summary of the snapshot tracking scripts
  • 3 Installation and configuration



Tracking OpenBSD snapshots with some simple shell scripts



1 Introduction

For several years I have been installing and testing OpenBSD snapshots. There is no fixed schedule for these snapshots. Usually daily a new snapshot appears on ftp.openbsd.org. Sometimes even two on a single day. Because of this irregularity and frequency some ftp mirrors never really catch up with ftp.openbsd.org.

So it is kind of a problem to find a ftp mirror with an up-to-date copy of the latest snapshot.

For some time I kept track of the snapshots on paper. At that time my main workstation was a 800MHz Pentium III and viewing ftp listings in firefox was slow. And I got tired of the 'eyeball' method to check whether a snapshot was completely mirrored.

The seven small scripts discussed here, help in storing and comparing ftp directory listings of ftp.openbsd.org, and it's mirror sites.

The main script which retrieve and stores information from the ftp sites, uses a '.netrc' file, as described in the ftp(1) manual page. Even when you do not intend to track OpenBSD snapshots, it may be interesting to read about the format and usage of '.netrc' to automate ftp access.



1.1 Legal stuff

Code:
#  Copyright (c) 2006,2007,2008,2009 J65nko <Administrator daemonforums.org>
# 
#  Permission to use, copy, modify, and distribute this software for any
#  purpose with or without fee is hereby granted, provided that the above
#  copyright notice and this permission notice appear in all copies.
# 
#  THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
#  WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
#  MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
#  ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
#  WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
#  ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
#  OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#



1.2 Dedication

I would like to dedicate this guide to Joost, friend of my daughter, who also became one of my best friends. Although still being a student, he completely surprised me with a AMD64 motherboard, CPU and memory as birthday gift in 2007.

Joost, thanks for allowing me to step into the 64 bits world sooner than expected. Your gift prompted in several changes in my snapshot infrastructure, which was i386-only. This guide partially documents the result of these adaptations.



2 How to use the snapshot tracking scripts

First we show extensive examples how to use these scripts and how to analyze their output.



2.1 Retrieve ftp mirror status with the 'chk' script

Code:
$ ./chk
Connected to openbsd.sunsite.ualberta.ca.
220 openbsd.srv.ualberta.ca FTP server ready.
331 Guest login ok, send your email address as password.
230-   Welcome to ftp.openbsd.org at the University of Alberta 
230-   in Edmonton, Alberta, Canada.
230-   For other mirror sites visit http://www.openbsd.org/ftp.html
230- 

[snip]

230 Guest login ok, access restrictions apply.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256  SHA256
local: SHA256 remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (129,128,5,191,227,203)
150 Opening BINARY mode data connection for '/pub/OpenBSD/snapshots/i386/SHA256' (2162 bytes).
226 Transfer complete.
2162 bytes received in 0.34 seconds (6.13 KB/s)
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest"
227 Entering Passive Mode (129,128,5,191,196,78)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
2162 bytes received in 0.34 seconds (6.13 KB/s)
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest"
227 Entering Passive Mode (129,128,5,191,196,78)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
ls  /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice  >latest_pkg"
227 Entering Passive Mode (129,128,5,191,231,101)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
quit
We see here that an ftp connection is being made to ftp.openbsd.org in Canada. The commands issued are in blue. An explanation of the ones that save information:
  • get /pub/OpenBSD/snapshots/i386/SHA256 SHA256 retrieves the SHA256 checksum file.
  • The command ls /pub/OpenBSD/snapshots/i386/ "| ./fnice >latest" saves a listing of the installation file sets in the file 'latest'. The listing produced by the ftp server gets formatted nicely with a small awk script called 'fnice'.

    Code:
           97834 Nov 30 10:27 INSTALL.i386
           22356 Nov 30 10:28 INSTALL.linux
            2162 Nov 30 10:28 SHA256
        49042080 Nov 30 10:28 base46.tgz
         7490499 Nov 30 10:28 bsd
         7509983 Nov 30 10:28 bsd.mp
         6234592 Nov 30 10:28 bsd.rd
         6352896 Nov 30 10:28 cd46.iso
           44980 Nov 30 10:28 cdboot
            2048 Nov 30 10:28 cdbr
         3012608 Nov 30 10:28 cdemu46.iso
        91072033 Nov 30 10:28 comp46.tgz
          516752 Nov 30 10:28 etc46.tgz
         1474560 Nov 30 10:28 floppy46.fs
         1474560 Nov 30 10:28 floppyB46.fs
         1474560 Nov 30 10:28 floppyC46.fs
         2618096 Nov 30 10:28 game46.tgz
             269 Nov 30 10:28 index.txt
       249755648 Nov 30 10:28 install46.iso
         8469803 Nov 30 10:27 man46.tgz
         2936282 Nov 30 10:27 misc46.tgz
           53532 Nov 30 10:28 pxeboot
        15527631 Nov 29 08:02 xbase46.tgz
           68103 Nov 29 08:02 xetc46.tgz
        35215028 Nov 29 08:02 xfont46.tgz
        19874775 Nov 29 08:02 xserv46.tgz
         2950471 Nov 29 08:02 xshare46.tgz
  • Likewise ls /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice >latest-pkg" stores a listing of the binary packages in the 'latest_pkg' file.

    A snippet from this 5902 lines long file:

    Code:
         1454425 Nov 29 12:19 postgresql-client-8.4.1p1.tgz
          396185 Nov 29 12:19 postgresql-contrib-8.4.1p1.tgz
         2176501 Nov 29 12:19 postgresql-docs-8.4.1p1.tgz
          405864 Nov 29 12:19 postgresql-jdbc-8.2.505p2.tgz
           55822 Nov 29 12:19 postgresql-jdbc-docs-8.2.505.tgz
          468689 Nov 29 12:19 postgresql-odbc-08.03.0400.tgz
         4356898 Nov 29 12:19 postgresql-server-8.4.1p1.tgz
           22650 Nov 29 12:19 postgresql_autodoc-1.40.tgz

The next connection is the second level mirror 'ftp.eu.openbsd.org' in Stockholm, Sweden.

Code:
Connected to ftp-prod-srv1.it.su.se.
220 ftp-prod-srv1.it.su.se FTP server ready.
331 Please specify the password.
230 Login successful.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256  SHA256-stockholm
local: SHA256-stockholm remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (130,237,164,232,203,198)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/SHA256 (2162 bytes).
226 File send OK.
2162 bytes received in 0.19 seconds (11.09 KB/s)
ls  /pub/OpenBSD/snapshots/i386/ "| ./fnice  >latest-stockholm"
227 Entering Passive Mode (130,237,164,232,204,181)
150 Here comes the directory listing.
226 Directory send OK.
ls  /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice  >latest_pkg-stockholm"
227 Entering Passive Mode (130,237,164,232,246,196)
150 Here comes the directory listing.
226 Directory send OK.
quit
221 Goodbye.
Here the 'SHA256' is being saved as 'SHA256-stockholm'. The listing of the installation file sets as 'latest-stockholm'. The binary packages listing is named 'latest_pkg-stockholm'.

After this the same procedure is repeated for the Vienna, Austria, second level mirror 'ftp.wu-wien.ac.at' and the Dutch 'ftp.nluug.nl'.
At the end we see the following summary:

Code:
Current date : 2009-11-30_22.11_UTC
NOW  date    : 2009-11-30_22:10_UTC
PREV date    : 2009-11-29_20:12_UTC
--------------- NOW ------------------------
 2162 Nov 29  16:06:57 2009 SHA256-vienna
 2162 Nov 29  16:06:57 2009 SHA256-stockholm
 2162 Nov 29  16:06:57 2009 SHA256-nluug
 2162 Nov 30  18:28:41 2009 SHA256
--------------- PREV -------------------
 2162 Nov 28  06:43:08 2009 SHA256-vienna
 2162 Nov 28  06:43:08 2009 SHA256-nluug
 2162 Nov 29  16:06:57 2009 SHA256-stockholm
 2162 Nov 29  16:06:57 2009 SHA256
--------------- X NOW ----------------------
    15527631 Nov 29 08:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz
--------------- X PREV ----------------------
    15527631 Nov 29 08:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz
--------------- PKG NOW --------------------
      121808 Nov 29 12:12 9libs-1.0p5.tgz
        9981 Nov 29 12:12 9menu-1.7p0.tgz
              .... [snip] ...............
      104955 Nov 29 12:22 zziplib-0.13.52.tgz
      125760 Nov 29 12:22 zzuf-0.12.tgz
--------------- PKG PREV --------------------
      121824 Nov 21 11:34 9libs-1.0p5.tgz
        9981 Nov 21 11:34 9menu-1.7p0.tgz
              .... [snip] ...............
      104999 Nov 21 11:43 zziplib-0.13.52.tgz
      125751 Nov 21 11:43 zzuf-0.12.tgz
The 'NOW' section shows the directory listings of the SHA266 files we just retrieved. The 'vienna', 'stockholm' and 'nluug' mirrors have a SHA256 file with the same "Nov 29 16:06:57" time stamp. The 'ftp.openbsd.org' SHA256 is more recent: Nov 30 18:28:41.

The section 'PREV' shows the results from the previous run of the 'chk' script.

While the 'NOW' and 'PREV' sections compare SHA256 files from multiple mirrors, the information of the sections X NOW/PREV and PKG NOW/PREV has a single source, the master site 'ftp.openbsd.org'.

The X NOW and X PREV parts show the dates of the X installation file sets of 'ftp.openbsd.org'. Both have the same "Nov 29 08:02" date and time.

Comparing the PKG NOW and PKG PREV information we notice a new set of binary packages from "Nov 29 12:12" has replaced the previous "Nov 21 11:34" ones.



2.2 Seek confirmation from 'differ'

To verify our eyeball check of the SHA256 files, we run the script 'differ', to compare the contents of all SHA256 files with the SHA256 of the main ftp site.

Code:
$ differ
  1 SHA256-nluug     : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : Files SHA256 and SHA256-stockholm differ
  3 SHA256-vienna    : Files SHA256 and SHA256-vienna differ
This confirms that none of the mirrors has the most recent snapshot yet.

An example where the Stockholm mirror has an identical SHA256 file:

Code:
  1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : =====Match====
  3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
For a more detailed view use differ -v

Code:
$ differ -v
  1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org  & stockholm ---
       97834 Nov 29 08:06 INSTALL.i386      |        97834 Nov 29 15:06 INSTALL.i386
       22356 Nov 29 08:06 INSTALL.linux     |        22356 Nov 29 15:06 INSTALL.linux
        2162 Nov 29 08:06 SHA256            |         2162 Nov 29 15:06 SHA256
    49042398 Nov 29 08:06 base46.tgz        |     49042398 Nov 29 15:06 base46.tgz
     7490469 Nov 29 08:06 bsd               |      7490469 Nov 29 15:06 bsd
     7509916 Nov 29 08:06 bsd.mp            |      7509916 Nov 29 15:06 bsd.mp
     6234592 Nov 29 08:06 bsd.rd            |      6234592 Nov 29 15:06 bsd.rd
     6352896 Nov 29 08:06 cd46.iso          |      6352896 Nov 29 15:06 cd46.iso
       44980 Nov 29 08:06 cdboot            |        44980 Nov 29 15:06 cdboot
        2048 Nov 29 08:06 cdbr              |         2048 Nov 29 15:06 cdbr
     3012608 Nov 29 08:06 cdemu46.iso       |      3012608 Nov 29 15:06 cdemu46.iso
    91072515 Nov 29 08:06 comp46.tgz        |     91072515 Nov 29 15:06 comp46.tgz
      516738 Nov 29 08:06 etc46.tgz         |       516738 Nov 29 15:06 etc46.tgz
     1474560 Nov 29 08:06 floppy46.fs       |      1474560 Nov 29 15:06 floppy46.fs
     1474560 Nov 29 08:06 floppyB46.fs      |      1474560 Nov 29 15:06 floppyB46.fs
     1474560 Nov 29 08:06 floppyC46.fs      |      1474560 Nov 29 15:06 floppyC46.fs
     2618027 Nov 29 08:06 game46.tgz        |      2618027 Nov 29 15:06 game46.tgz
         269 Nov 29 08:06 index.txt         |          269 Nov 29 15:06 index.txt
   249757696 Nov 29 08:06 install46.iso     |    249757696 Nov 29 15:06 install46.iso
     8469749 Nov 29 08:06 man46.tgz         |      8469749 Nov 29 15:06 man46.tgz
     2936230 Nov 29 08:06 misc46.tgz        |      2936230 Nov 29 15:06 misc46.tgz
       53532 Nov 29 08:06 pxeboot           |        53532 Nov 29 15:06 pxeboot
    15527631 Nov 29 08:02 xbase46.tgz       |     15527631 Nov 29 15:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz        |        68103 Nov 29 15:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz       |     35215028 Nov 29 15:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz       |     19874775 Nov 29 15:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz      |      2950471 Nov 29 15:02 xshare46.tgz
  3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
All Files dated Nov 29 08:06 and 08:02 have a corresponding time stamp of Nov 29 15:06 and 15:02. But note that this is not always the case!

Code:
$ differ -v
  1 SHA256-nluug     : Files SHA256 and SHA256-nluug differ
  2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org  & stockholm ---
       97834 Nov 30 10:27 INSTALL.i386      |        97834 Nov 30 17:27 INSTALL.i386
       22356 Nov 30 10:28 INSTALL.linux     |        22356 Nov 30 17:28 INSTALL.linux
        2162 Nov 30 10:28 SHA256            |         2162 Nov 30 17:28 SHA256
    49042080 Nov 30 10:28 base46.tgz        |     49042080 Nov 30 17:28 base46.tgz
     7490499 Nov 30 10:28 bsd               |      7490499 Nov 30 17:28 bsd
     7509983 Nov 30 10:28 bsd.mp            |      7509983 Nov 30 17:28 bsd.mp
     6234592 Nov 30 10:28 bsd.rd            |      6234592 Nov 30 17:28 bsd.rd
     6352896 Nov 30 10:28 cd46.iso          |      6352896 Nov 30 17:28 cd46.iso
       44980 Nov 30 10:28 cdboot            |        44980 Nov 30 17:28 cdboot
        2048 Nov 30 10:28 cdbr              |         2048 Nov 30 17:28 cdbr
     3012608 Nov 30 10:28 cdemu46.iso       |      3012608 Nov 30 17:28 cdemu46.iso
    91072033 Nov 30 10:28 comp46.tgz        |     91072033 Nov 30 17:28 comp46.tgz
      516752 Nov 30 10:28 etc46.tgz         |       516752 Nov 30 17:28 etc46.tgz
     1474560 Nov 30 10:28 floppy46.fs       |      1474560 Nov 30 17:28 floppy46.fs
     1474560 Nov 30 10:28 floppyB46.fs      |      1474560 Nov 30 17:28 floppyB46.fs
     1474560 Nov 30 10:28 floppyC46.fs      |      1474560 Nov 29 15:06 floppyC46.fs
     2618096 Nov 30 10:28 game46.tgz        |      2618027 Nov 29 15:06 game46.tgz
         269 Nov 30 10:28 index.txt         |         1710 Nov 30 11:05 index.txt
   249755648 Nov 30 10:28 install46.iso     |    249757696 Nov 29 15:06 install46.iso
     8469803 Nov 30 10:27 man46.tgz         |      8469749 Nov 29 15:06 man46.tgz
     2936282 Nov 30 10:27 misc46.tgz        |      2936230 Nov 29 15:06 misc46.tgz
       53532 Nov 30 10:28 pxeboot           |        53532 Nov 29 15:06 pxeboot
    15527631 Nov 29 08:02 xbase46.tgz       |     15527631 Nov 29 15:02 xbase46.tgz
       68103 Nov 29 08:02 xetc46.tgz        |        68103 Nov 29 15:02 xetc46.tgz
    35215028 Nov 29 08:02 xfont46.tgz       |     35215028 Nov 29 15:02 xfont46.tgz
    19874775 Nov 29 08:02 xserv46.tgz       |     19874775 Nov 29 15:02 xserv46.tgz
     2950471 Nov 29 08:02 xshare46.tgz      |      2950471 Nov 29 15:02 xshare46.tgz
  3 SHA256-vienna    : Files SHA256 and SHA256-vienna differ
Here the SHA256 has been synced, but but files like 'floppyC46.fs' and 'man46.tgz' have not.

A more obvious case where the 'Erlangen' mirror in Germany was still busy updating the X .tgz files. Note that this example dates from Dec 2008, when there were still MD5 checksums files instead of SHA256.

Code:
1 SHA256-calyx : Files SHA256 and SHA256-calyx differ
  2 SHA256-erlangen : =====Match====
--- sdiff of ftp.openbsd.org  & erlangen ---
       94576 Dec  9 18:53 INSTALL.amd64   |   10223616 Dec 11 04:25 .xserv44.tgz.G22368
         807 Dec  9 18:54 MD5             |      94576 Dec 10 02:53 INSTALL.amd64
    52334144 Dec  9 18:54 base44.tgz      |        807 Dec 10 02:54 MD5
     7223978 Dec  9 18:54 bsd             |   52334144 Dec 10 02:54 base44.tgz
     7342003 Dec  9 18:54 bsd.mp          |    7223978 Dec 10 02:54 bsd
     6315592 Dec  9 18:54 bsd.rd          |    7342003 Dec 10 02:54 bsd.mp
     6430720 Dec  9 18:54 cd44.iso        |    6315592 Dec 10 02:54 bsd.rd
       42940 Dec  9 18:54 cdboot          |    6430720 Dec 10 02:54 cd44.iso
        2048 Dec  9 18:54 cdbr            |      42940 Dec 10 02:54 cdboot
   102090065 Dec  9 18:54 comp44.tgz      |       2048 Dec 10 02:54 cdbr
      511724 Dec  9 18:54 etc44.tgz       |  102090065 Dec 10 02:54 comp44.tgz
     1474560 Dec  9 18:54 floppy44.fs     |     511724 Dec 10 02:54 etc44.tgz
     2706370 Dec  9 18:54 game44.tgz      |    1474560 Dec 10 02:54 floppy44.fs
         215 Jul  2 04:06 index.txt       |    2706370 Dec 10 02:54 game44.tgz
   260915200 Dec  9 18:54 install44.iso   |        215 Jul  2 12:06 index.txt
     8214814 Dec  9 18:53 man44.tgz       |  260915200 Dec 10 02:54 install44.iso
     2935078 Dec  9 18:53 misc44.tgz      |    8214814 Dec 10 02:53 man44.tgz
       53268 Dec  9 18:54 pxeboot         |    2935078 Dec 10 02:53 misc44.tgz
    11110781 Dec  9 18:44 xbase44.tgz     |      53268 Dec 10 02:54 pxeboot
       78441 Dec  9 18:44 xetc44.tgz      |   11110781 Dec 10 02:44 xbase44.tgz
    35579685 Dec  9 18:44 xfont44.tgz     |      78441 Dec 10 02:44 xetc44.tgz
    21467266 Dec  9 18:44 xserv44.tgz     |   35579685 Dec 10 02:44 xfont44.tgz
     2778752 Dec  9 18:44 xshare44.tgz    |   21467273 Dec  9 00:17 xserv44.tgz
                                          >    2778563 Dec  9 00:17 xshare44.tgz
The 'sdiff' output shows a '.xserv44.tgz.G22368' file, while there still is a 'xserv44.tgz'. Clearly a temporary file to store the newest version, before removing the old one.

After using 'chk' and 'differ' a couple of times, your non-artificial neural network becomes trained in spotting these differences. Of course this will be easier for American continentals, with only a couple of hours time difference, then the poor Europeans and Asians who have to deal with much larger time offsets.



2.2.1 Time stamp of SHA256 file

The X installation file sets 'xbase??.tgz', 'xetc??tgz', 'xfont??.tgz', 'xserv??.tgz' and 'xshare46.tgz' are usually created separately from the base install sets. Because the SHA256 file contains the checksums of both X and the base install sets this means that that SHA256 can have either the time stamp of the base install sets or the X sets.



2.3 Check mirror coherence of the packages with 'pkg-mirror-complete'

Without any tools, forming an opinion about the mirror status of the nearly 6000 pre-compiled packages, is very difficult. The small 'pkg-mirror-complete' script attempts to give an assessment of the coherence.

To check whether 'ftp.openbsd.org' has a coherent set of packages:

Code:
$ ./pkg-mirror-complete
  Unique dates from
./NOW/latest_pkg:
===============================
     1  11-21 11:34
     2  11-21 11:35
     3  11-21 11:36
     4  11-21 11:37
     5  11-21 11:38
     6  11-21 11:39
     7  11-21 11:40
     8  11-21 11:41
     9  11-21 11:42
    10  11-21 11:43
    11  11-29 04:05
===============================
Last entry:
 11-29 04:05 index.txt 456441

All files with last date:
 "11-29 04:05"
===============================
11-29 04:05 index.txt 456441
===============================
We see a sorted listing of the unique dates retrieved from 'latest_pkg'. The time stamps starting at 21st of Nov 11:34 up to 11:43 follow a regular pattern of 1 minute increments. Only dissonant is the last 11-29 04:05 time stamp. The only file with this time stamp is a file called 'index.txt', which usually is generated hours after a change of directory contents.

The same check but now for the mirror in Stockholm, Sweden

Code:
$ pkg-mirror-complete stockholm
  Unique dates from
./NOW/latest_pkg-stockholm:
===============================
     1  11-21 18:34
     2  11-21 18:35
     3  11-21 18:36
     4  11-21 18:37
     5  11-21 18:38
     6  11-21 18:39
     7  11-21 18:40
     8  11-21 18:41
     9  11-21 18:42
    10  11-21 18:43
    11  11-29 11:05
===============================
Last entry:
 11-29 11:05 index.txt 456441

All files with last date:
 "11-29 11:05"
===============================
11-29 11:05 index.txt 456441
===============================
The 'stockholm' parameter selects the 'latest_pkg-stockholm' as data source for 'pkg-mirror-complete'.

We see the same pattern, although the times show a time zone induced difference of 7 hours. We can rather safely assume that the packages of the Stockholm mirror are completely mirrored

The Dutch nluug mirror caught while updating the packages to the 24th of Dec ones:

Code:
$ pkg-mirror-complete nluug
  Unique dates from
./NOW/latest_pkg-nluug:
===============================
     1  11-15 16:07
     2  11-21 18:34
     3  11-21 18:35
     4  11-21 18:36
     5  11-21 18:37
     6  11-21 18:38
     7  11-21 18:39
     8  11-21 18:40
     9  11-21 18:41
    10  11-21 18:42
    11  11-21 18:43
    12  11-22 11:05
===============================
Last entry:
 11-22 11:05 index.txt 456376

All files with last date:
 "11-22 11:05"
===============================
11-22 11:05 index.txt 456376
===============================
In such cases we better wait before installing of upgrading the new packages.



2.3.1 A side-by-side comparision with 'pkg-differ'

Code:
$ pkg-differ nluug
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | ./NOW/latest_pkg-nluug:
===============================        ===============================
     1  11-21 11:34                  |      1   11-21 18:34
     2  11-21 11:35                  |      2   11-21 18:35
     3  11-21 11:36                  |      3   11-21 18:36
     4  11-21 11:37                  |      4   11-21 18:37
     5  11-21 11:38                  |      5   11-21 18:38
     6  11-21 11:39                  |      6   11-21 18:39
     7  11-21 11:40                  |      7   11-21 18:40
     8  11-21 11:41                  |      8   11-21 18:41
     9  11-21 11:42                  |      9   11-21 18:42
    10  11-21 11:43                  |     10   11-21 18:43
    11  11-28 04:05                  |     11   11-27 11:05
===============================        ===============================
Last entry:                            Last entry:
 11-28 04:05 index.txt 456441        |  11-27 11:05 index.txt 456441
                                       
All files with last date:              All files with last date:
 "11-28 04:05"                       |  "11-27 11:05"
===============================        ===============================
11-28 04:05 index.txt 456441         | 11-27 11:05 index.txt 456441
===============================        ===============================
As can be seen, this simply runs pkg-differ-complete for ftp.openbsd.org and a mirror.

The following shows the Vienna mirror not having completed syncing for both the Nov 15th and 21st package build.

Code:
$ pkg-differ vienna
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | ./NOW/latest_pkg-vienna:
===============================        ===============================
     1  11-21 11:34                  |      1   11-02 20:31
     2  11-21 11:35                  |      2   11-02 20:32
     3  11-21 11:36                  |      3   11-02 20:33
     4  11-21 11:37                  |      4   11-02 20:34
     5  11-21 11:38                  |      5   11-02 20:35
     6  11-21 11:39                  |      6   11-02 20:36
     7  11-21 11:40                  |      7   11-02 20:37
     8  11-21 11:41                  |      8   11-02 20:38
     9  11-21 11:42                  |      9   11-02 20:39
    10  11-21 11:43                  |     10   11-02 20:40
    11  11-28 04:05                  |     11   11-15 15:58
                                     >     12   11-15 15:59
                                                [snip]
                                     >     18   11-15 16:06
                                     >     19   11-15 16:07
                                     >     20   11-21 18:34
                                     >     21   11-21 18:35
                                                [snip]
                                     >     28   11-21 18:42
                                     >     29   11-21 18:43
                                     >     30   11-28 11:05
===============================        ===============================
Last entry:                            Last entry:
 11-28 04:05 index.txt 456441        |  11-28 11:05 index.txt 456441
                                       
All files with last date:              All files with last date:
 "11-28 04:05"                       |  "11-28 11:05"
===============================        ===============================
11-28 04:05 index.txt 456441         | 11-28 11:05 index.txt 456441
===============================        ===============================
Another example where the Vienna mirror seems to synchronize to a fresh directory.

Code:
$ pkg-differ vienna    
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | .//NOW/latest_pkg-vienna
===============================        ===============================
     1  10-26 10:43                  |      1   10-26 16:43
     2  10-26 10:44                  |      2   10-26 16:44
     3  10-26 10:45                  |      3   10-26 16:45
     4  10-26 10:46                  |      4   10-26 16:46
     5  10-26 10:47                  |      5   10-26 16:47
     6  10-26 10:48                  |      6   10-26 16:48
     7  10-26 10:49                  |      7   10-26 16:49
     8  10-26 10:50                  |      8   10-26 16:50
     9  10-26 10:51                  |      9   10-26 16:51
    10  10-26 10:52                  <
    11  10-28 04:05                  <
===============================        ===============================
Last entry:                            Last entry:
 10-28 04:05 index.txt 450746        |  10-26 16:51 tutris-1.0.1p1.tgz 5294
                                       
All files with last date:              All files with last date:
 "10-28 04:05"                       |  "10-26 16:51"
===============================        ===============================
10-28 04:05 index.txt 450746         | 10-26 16:51 tomboy-1.0.0.tgz 8839239
                                     > 10-26 16:51 tomcat-4.1.40.tgz 474034
                                     > 10-26 16:51 tomcat-5.5.27p1.tgz 4928
                                     > 10-26 16:51 tomcat-6.0.20.tgz 498773
                                     > 10-26 16:51 tomcat-admin-4.1.40p0.tg
                                     > 10-26 16:51 tomcat-admin-5.5.27p0.tg
                                     > 10-26 16:51 tt-rss-1.3.3.tgz 802688
                                       [snip]
                                     > 10-26 16:51 ttcp-1.12.tgz 9450
                                     > 10-26 16:51 ttyrec-1.0.8p0.tgz 11201
                                     > 10-26 16:51 turba-h3-2.3.2.tgz 23986
                                     > 10-26 16:51 tutris-1.0.1p1.tgz 52947
===============================        ===============================



2.3.2 The Sparc64 packages

The approach from the pkg-mirror-complete and pkg-differ scripts work well for most architectures. A major pain, however are the packages for the sparc64. Instead of the 10 to 15 different time stamps there are more than 2645!.

Code:
  Unique dates from                      Unique dates from
./NOW/latest_pkg:                    | ./NOW/latest_pkg-stockholm:
===============================        ===============================
     1  10-25 06:58                  |      1   10-25 12:58
     2  10-25 06:59                  |      2   10-25 12:59
     3  10-25 07:00                  |      3   10-25 13:00
     4  10-25 07:01                  |      4   10-25 13:01
     5  10-25 07:02                  |      5   10-25 13:02
  [snip]
  2644  10-31 22:01                  |   2644   11-01 04:01
  2645  10-31 23:05                  |   2645   11-01 05:05
  2646  10-31 23:16                  |   2646   11-01 05:16
  2647  11-29 04:05                  |   2647   11-29 11:05
===============================        ===============================
Last entry:                            Last entry:
 11-29 04:05 index.txt 434648        |  11-29 11:05 index.txt 434648
                                       
All files with last date:              All files with last date:
 "11-29 04:05"                       |  "11-29 11:05"
===============================        ===============================
11-29 04:05 index.txt 434648         | 11-29 11:05 index.txt 434648
===============================        ===============================
The author, not familiar with the sparc64, just wonders if it would be feasible to touch the compiled packages to a single time stamp before uploading.



2.4 Save ftp mirror data with 'backup'

If a few hours later, we want to check for whether the Stockholm mirror has finished mirroring, we first save the current retrieved listings and SHA256 files with the 'backup' script.

Code:
$ backup -v

./backup: creating backup in S2009-11-30_22:21_UTC ...
drwxr-xr-x  2 j65  j65  512 Dec  1 18:55 S2009-11-30_22:21_UTC

./backup: linking PREV to S2009-11-30_22:21_UTC ...
lrwxr-xr-x  1 j65  j65  21 Dec  1 18:55 PREV -> S2009-11-30_22:21_UTC

total 1500
-rw-r--r--  1 j65  j65      21 Nov 30 23:21 DATE
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256-esat
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256-funkfeuer
-rw-r--r--  1 j65  j65    1726 Nov 23 16:37 SHA256-stockholm
-rw-r--r--  1 j65  j65     908 Nov 30 23:21 latest
-rw-r--r--  1 j65  j65     908 Nov 30 23:22 latest-esat
-rw-r--r--  1 j65  j65     908 Nov 30 23:22 latest-funkfeuer
-rw-r--r--  1 j65  j65     908 Nov 30 23:21 latest-stockholm
-rw-r--r--  1 j65  j65  166644 Nov 30 23:21 latest_pkg
-rw-r--r--  1 j65  j65  166644 Nov 30 23:22 latest_pkg-esat
-rw-r--r--  1 j65  j65  216589 Nov 30 23:22 latest_pkg-funkfeuer
-rw-r--r--  1 j65  j65  166644 Nov 30 23:21 latest_pkg-stockholm
This script copies the SHA256 files and ftp directory listings into a specially named new directory. The directory name consists of 'S' with the UTC time stamp appended. In this case 'S2009-11-30_22:21_UTC'.

After the files have been copied, the symbolic link PREV is pointed to 'S2009-11-30_22:21_UTC'. The directory NOW still has the original files. These will only be deleted during the next 'chk'.

'backup' will warn you if there already is a backup of the NOW directory.

Code:
$ ./backup
./backup : Backup already exists! Exiting ....



2.5 'show' time

In the explanation of the 'chk' output, we discussed a summary like this:

Code:
Current date : 2009-12-01_18.32_UTC
NOW  date    : 2009-11-29_20:20_UTC
PREV date    : 2009-11-27_20:49_UTC
--------------- NOW ------------------------
 2084 Nov 28  06:43:25 2009 SHA256-vienna
 2084 Nov 28  06:43:25 2009 SHA256-stockholm
 2084 Nov 28  06:43:25 2009 SHA256
--------------- PREV -------------------
 2084 Nov 22  18:13:09 2009 SHA256-vienna
 2084 Nov 27  01:11:32 2009 SHA256-stockholm
 2084 Nov 27  01:11:32 2009 SHA256
--------------- X NOW ----------------------
    17248633 Nov 27 22:43 xbase46.tgz
       69479 Nov 27 22:43 xetc46.tgz
    35217656 Nov 27 22:43 xfont46.tgz
    11389029 Nov 27 22:43 xserv46.tgz
     2952171 Nov 27 22:43 xshare46.tgz
--------------- X PREV ----------------------
    17247839 Nov 22 10:13 xbase46.tgz
       69495 Nov 22 10:13 xetc46.tgz
    35217765 Nov 22 10:13 xfont46.tgz
    11408354 Nov 22 10:13 xserv46.tgz
     2951722 Nov 22 10:13 xshare46.tgz
--------------- PKG NOW --------------------
      145282 Oct 28 09:48 9libs-1.0p5.tgz
       11950 Oct 29 12:46 9menu-1.7p0.tgz
              .... [snip] ...............
      111247 Oct 27 12:48 zziplib-0.13.52.tgz
      129514 Oct 27 09:45 zzuf-0.12.tgz
--------------- PKG PREV --------------------
      145282 Oct 28 09:48 9libs-1.0p5.tgz
       11950 Oct 29 12:46 9menu-1.7p0.tgz
              .... [snip] ...............
      111247 Oct 27 12:48 zziplib-0.13.52.tgz
      129514 Oct 27 09:45 zzuf-0.12.tgz
--------------------------------------
Actually the 'chk' script calls the 'show' script to produce this summary. You also can run it any time.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
 

Tags
.netrc, ftp, openbsd snapshot

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Script to update NetBSD using snapshots s0xxx Guides 5 23rd November 2009 04:30 PM
DJB daemontools: compile and install scripts (OpenBSD) J65nko Guides 2 13th November 2009 10:57 AM
Executing web scripts : can't! erehwon OpenBSD General 3 3rd November 2009 10:02 PM
SHA256 replaces MD5 in OpenBSD snapshots J65nko OpenBSD General 3 6th May 2009 04:36 PM
Periodic scripts for mail scripts stukov FreeBSD General 2 8th March 2009 07:51 PM


All times are GMT. The time now is 09:59 AM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick