Tracking OpenBSD snapshots with some simple shell scripts
Table of contents
- 1 Introduction
- 1.1 Legal stuff
- 1.2 Dedication
- 2 How to use the snapshot tracking scripts
- 2.1 Retrieve ftp mirror status with the 'chk' script
- 2.2 Seek confirmation from 'differ'
- 2.2.1 Time stamp of SHA256 file
- 2.3 Check mirror coherence of the packages with 'pkg-mirror-complete'
- 2.3.1 A side-by-side comparision with 'pkg-differ'
- 2.3.2 The Sparc64 packages
- 2.4 Save ftp mirror data with 'backup'
- 2.5 'show' time
- 2.6 Summary of the snapshot tracking scripts
- 3 Installation and configuration
Tracking OpenBSD snapshots with some simple shell scripts
1 Introduction
For several years I have been installing and testing OpenBSD snapshots. There is no fixed schedule for these snapshots. Usually daily a new snapshot appears on ftp.openbsd.org. Sometimes even two on a single day. Because of this irregularity and frequency some ftp mirrors never really catch up with ftp.openbsd.org.
So it is kind of a problem to find a ftp mirror with an up-to-date copy of the latest snapshot.
For some time I kept track of the snapshots on paper. At that time my main workstation was a 800MHz Pentium III and viewing ftp listings in firefox was slow. And I got tired of the 'eyeball' method to check whether a snapshot was completely mirrored.
The seven small scripts discussed here, help in storing and comparing ftp directory listings of ftp.openbsd.org, and it's mirror sites.
The main script which retrieve and stores information from the ftp sites, uses a
'.netrc' file, as described in the
ftp(1) manual page. Even when you do not intend to track OpenBSD snapshots, it may be interesting to read about the format and usage of
'.netrc' to automate ftp access.
1.1 Legal stuff
Code:
# Copyright (c) 2006,2007,2008,2009 J65nko <Administrator daemonforums.org>
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#
1.2 Dedication
I would like to dedicate this guide to Joost, friend of my daughter, who also became one of my best friends. Although still being a student, he completely surprised me with a AMD64 motherboard, CPU and memory as birthday gift in 2007.
Joost, thanks for allowing me to step into the 64 bits world sooner than expected. Your gift prompted in several changes in my snapshot infrastructure, which was i386-only. This guide partially documents the result of these adaptations.
2 How to use the snapshot tracking scripts
First we show extensive examples how to use these scripts and how to analyze their output.
2.1 Retrieve ftp mirror status with the 'chk' script
Code:
$ ./chk
Connected to openbsd.sunsite.ualberta.ca.
220 openbsd.srv.ualberta.ca FTP server ready.
331 Guest login ok, send your email address as password.
230- Welcome to ftp.openbsd.org at the University of Alberta
230- in Edmonton, Alberta, Canada.
230- For other mirror sites visit http://www.openbsd.org/ftp.html
230-
[snip]
230 Guest login ok, access restrictions apply.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256 SHA256
local: SHA256 remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (129,128,5,191,227,203)
150 Opening BINARY mode data connection for '/pub/OpenBSD/snapshots/i386/SHA256' (2162 bytes).
226 Transfer complete.
2162 bytes received in 0.34 seconds (6.13 KB/s)
ls /pub/OpenBSD/snapshots/i386/ "| ./fnice >latest"
227 Entering Passive Mode (129,128,5,191,196,78)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
2162 bytes received in 0.34 seconds (6.13 KB/s)
ls /pub/OpenBSD/snapshots/i386/ "| ./fnice >latest"
227 Entering Passive Mode (129,128,5,191,196,78)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
ls /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice >latest_pkg"
227 Entering Passive Mode (129,128,5,191,231,101)
150 Opening ASCII mode data connection for '/bin/ls'.
226 Transfer complete.
quit
We see here that an ftp connection is being made to ftp.openbsd.org in Canada. The commands issued are in blue. An explanation of the ones that save information:
- get /pub/OpenBSD/snapshots/i386/SHA256 SHA256 retrieves the SHA256 checksum file.
- The command ls /pub/OpenBSD/snapshots/i386/ "| ./fnice >latest" saves a listing of the installation file sets in the file 'latest'. The listing produced by the ftp server gets formatted nicely with a small awk script called 'fnice'.
Code:
97834 Nov 30 10:27 INSTALL.i386
22356 Nov 30 10:28 INSTALL.linux
2162 Nov 30 10:28 SHA256
49042080 Nov 30 10:28 base46.tgz
7490499 Nov 30 10:28 bsd
7509983 Nov 30 10:28 bsd.mp
6234592 Nov 30 10:28 bsd.rd
6352896 Nov 30 10:28 cd46.iso
44980 Nov 30 10:28 cdboot
2048 Nov 30 10:28 cdbr
3012608 Nov 30 10:28 cdemu46.iso
91072033 Nov 30 10:28 comp46.tgz
516752 Nov 30 10:28 etc46.tgz
1474560 Nov 30 10:28 floppy46.fs
1474560 Nov 30 10:28 floppyB46.fs
1474560 Nov 30 10:28 floppyC46.fs
2618096 Nov 30 10:28 game46.tgz
269 Nov 30 10:28 index.txt
249755648 Nov 30 10:28 install46.iso
8469803 Nov 30 10:27 man46.tgz
2936282 Nov 30 10:27 misc46.tgz
53532 Nov 30 10:28 pxeboot
15527631 Nov 29 08:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz
- Likewise ls /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice >latest-pkg" stores a listing of the binary packages in the 'latest_pkg' file.
A snippet from this 5902 lines long file:
Code:
1454425 Nov 29 12:19 postgresql-client-8.4.1p1.tgz
396185 Nov 29 12:19 postgresql-contrib-8.4.1p1.tgz
2176501 Nov 29 12:19 postgresql-docs-8.4.1p1.tgz
405864 Nov 29 12:19 postgresql-jdbc-8.2.505p2.tgz
55822 Nov 29 12:19 postgresql-jdbc-docs-8.2.505.tgz
468689 Nov 29 12:19 postgresql-odbc-08.03.0400.tgz
4356898 Nov 29 12:19 postgresql-server-8.4.1p1.tgz
22650 Nov 29 12:19 postgresql_autodoc-1.40.tgz
The next connection is the second level mirror '
ftp.eu.openbsd.org' in Stockholm, Sweden.
Code:
Connected to ftp-prod-srv1.it.su.se.
220 ftp-prod-srv1.it.su.se FTP server ready.
331 Please specify the password.
230 Login successful.
prompt off
Interactive mode off.
epsv4 off
EPSV/EPRT on IPv4 off.
preserve on
Preserve modification times on.
get /pub/OpenBSD/snapshots/i386/SHA256 SHA256-stockholm
local: SHA256-stockholm remote: /pub/OpenBSD/snapshots/i386/SHA256
227 Entering Passive Mode (130,237,164,232,203,198)
150 Opening BINARY mode data connection for /pub/OpenBSD/snapshots/i386/SHA256 (2162 bytes).
226 File send OK.
2162 bytes received in 0.19 seconds (11.09 KB/s)
ls /pub/OpenBSD/snapshots/i386/ "| ./fnice >latest-stockholm"
227 Entering Passive Mode (130,237,164,232,204,181)
150 Here comes the directory listing.
226 Directory send OK.
ls /pub/OpenBSD/snapshots/packages/i386/ "| ./fnice >latest_pkg-stockholm"
227 Entering Passive Mode (130,237,164,232,246,196)
150 Here comes the directory listing.
226 Directory send OK.
quit
221 Goodbye.
Here the
'SHA256' is being saved as
'SHA256-stockholm'. The listing of the installation file sets as
'latest-stockholm'. The binary packages listing is named
'latest_pkg-stockholm'.
After this the same procedure is repeated for the Vienna, Austria, second level mirror '
ftp.wu-wien.ac.at' and the Dutch '
ftp.nluug.nl'.
At the end we see the following summary:
Code:
Current date : 2009-11-30_22.11_UTC
NOW date : 2009-11-30_22:10_UTC
PREV date : 2009-11-29_20:12_UTC
--------------- NOW ------------------------
2162 Nov 29 16:06:57 2009 SHA256-vienna
2162 Nov 29 16:06:57 2009 SHA256-stockholm
2162 Nov 29 16:06:57 2009 SHA256-nluug
2162 Nov 30 18:28:41 2009 SHA256
--------------- PREV -------------------
2162 Nov 28 06:43:08 2009 SHA256-vienna
2162 Nov 28 06:43:08 2009 SHA256-nluug
2162 Nov 29 16:06:57 2009 SHA256-stockholm
2162 Nov 29 16:06:57 2009 SHA256
--------------- X NOW ----------------------
15527631 Nov 29 08:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz
--------------- X PREV ----------------------
15527631 Nov 29 08:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz
--------------- PKG NOW --------------------
121808 Nov 29 12:12 9libs-1.0p5.tgz
9981 Nov 29 12:12 9menu-1.7p0.tgz
.... [snip] ...............
104955 Nov 29 12:22 zziplib-0.13.52.tgz
125760 Nov 29 12:22 zzuf-0.12.tgz
--------------- PKG PREV --------------------
121824 Nov 21 11:34 9libs-1.0p5.tgz
9981 Nov 21 11:34 9menu-1.7p0.tgz
.... [snip] ...............
104999 Nov 21 11:43 zziplib-0.13.52.tgz
125751 Nov 21 11:43 zzuf-0.12.tgz
The
'NOW' section shows the directory listings of the SHA266 files we just retrieved. The '
vienna', '
stockholm' and '
nluug' mirrors have a SHA256 file with the same "Nov 29 16:06:57" time stamp. The '
ftp.openbsd.org' SHA256 is more recent: Nov 30 18:28:41.
The section
'PREV' shows the results from the previous run of the
'chk' script.
While the
'NOW' and
'PREV' sections compare SHA256 files from multiple mirrors, the information of the sections X NOW/PREV and PKG NOW/PREV has a single source, the master site '
ftp.openbsd.org'.
The X NOW and X PREV parts show the dates of the X installation file sets of '
ftp.openbsd.org'. Both have the same "Nov 29 08:02" date and time.
Comparing the PKG NOW and PKG PREV information we notice a new set of binary packages from "Nov 29 12:12" has replaced the previous "Nov 21 11:34" ones.
2.2 Seek confirmation from 'differ'
To verify our eyeball check of the SHA256 files, we run the script
'differ', to compare the contents of all SHA256 files with the SHA256 of the main ftp site.
Code:
$ differ
1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
2 SHA256-stockholm : Files SHA256 and SHA256-stockholm differ
3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
This confirms that none of the mirrors has the most recent snapshot yet.
An example where the Stockholm mirror has an identical SHA256 file:
Code:
1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
2 SHA256-stockholm : =====Match====
3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
For a more detailed view use
differ -v
Code:
$ differ -v
1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org & stockholm ---
97834 Nov 29 08:06 INSTALL.i386 | 97834 Nov 29 15:06 INSTALL.i386
22356 Nov 29 08:06 INSTALL.linux | 22356 Nov 29 15:06 INSTALL.linux
2162 Nov 29 08:06 SHA256 | 2162 Nov 29 15:06 SHA256
49042398 Nov 29 08:06 base46.tgz | 49042398 Nov 29 15:06 base46.tgz
7490469 Nov 29 08:06 bsd | 7490469 Nov 29 15:06 bsd
7509916 Nov 29 08:06 bsd.mp | 7509916 Nov 29 15:06 bsd.mp
6234592 Nov 29 08:06 bsd.rd | 6234592 Nov 29 15:06 bsd.rd
6352896 Nov 29 08:06 cd46.iso | 6352896 Nov 29 15:06 cd46.iso
44980 Nov 29 08:06 cdboot | 44980 Nov 29 15:06 cdboot
2048 Nov 29 08:06 cdbr | 2048 Nov 29 15:06 cdbr
3012608 Nov 29 08:06 cdemu46.iso | 3012608 Nov 29 15:06 cdemu46.iso
91072515 Nov 29 08:06 comp46.tgz | 91072515 Nov 29 15:06 comp46.tgz
516738 Nov 29 08:06 etc46.tgz | 516738 Nov 29 15:06 etc46.tgz
1474560 Nov 29 08:06 floppy46.fs | 1474560 Nov 29 15:06 floppy46.fs
1474560 Nov 29 08:06 floppyB46.fs | 1474560 Nov 29 15:06 floppyB46.fs
1474560 Nov 29 08:06 floppyC46.fs | 1474560 Nov 29 15:06 floppyC46.fs
2618027 Nov 29 08:06 game46.tgz | 2618027 Nov 29 15:06 game46.tgz
269 Nov 29 08:06 index.txt | 269 Nov 29 15:06 index.txt
249757696 Nov 29 08:06 install46.iso | 249757696 Nov 29 15:06 install46.iso
8469749 Nov 29 08:06 man46.tgz | 8469749 Nov 29 15:06 man46.tgz
2936230 Nov 29 08:06 misc46.tgz | 2936230 Nov 29 15:06 misc46.tgz
53532 Nov 29 08:06 pxeboot | 53532 Nov 29 15:06 pxeboot
15527631 Nov 29 08:02 xbase46.tgz | 15527631 Nov 29 15:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz | 68103 Nov 29 15:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz | 35215028 Nov 29 15:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz | 19874775 Nov 29 15:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz | 2950471 Nov 29 15:02 xshare46.tgz
3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
All Files dated Nov 29 08:06 and 08:02 have a corresponding time stamp of Nov 29 15:06 and 15:02. But note that this is not always the case!
Code:
$ differ -v
1 SHA256-nluug : Files SHA256 and SHA256-nluug differ
2 SHA256-stockholm : =====Match====
--- sdiff of ftp.openbsd.org & stockholm ---
97834 Nov 30 10:27 INSTALL.i386 | 97834 Nov 30 17:27 INSTALL.i386
22356 Nov 30 10:28 INSTALL.linux | 22356 Nov 30 17:28 INSTALL.linux
2162 Nov 30 10:28 SHA256 | 2162 Nov 30 17:28 SHA256
49042080 Nov 30 10:28 base46.tgz | 49042080 Nov 30 17:28 base46.tgz
7490499 Nov 30 10:28 bsd | 7490499 Nov 30 17:28 bsd
7509983 Nov 30 10:28 bsd.mp | 7509983 Nov 30 17:28 bsd.mp
6234592 Nov 30 10:28 bsd.rd | 6234592 Nov 30 17:28 bsd.rd
6352896 Nov 30 10:28 cd46.iso | 6352896 Nov 30 17:28 cd46.iso
44980 Nov 30 10:28 cdboot | 44980 Nov 30 17:28 cdboot
2048 Nov 30 10:28 cdbr | 2048 Nov 30 17:28 cdbr
3012608 Nov 30 10:28 cdemu46.iso | 3012608 Nov 30 17:28 cdemu46.iso
91072033 Nov 30 10:28 comp46.tgz | 91072033 Nov 30 17:28 comp46.tgz
516752 Nov 30 10:28 etc46.tgz | 516752 Nov 30 17:28 etc46.tgz
1474560 Nov 30 10:28 floppy46.fs | 1474560 Nov 30 17:28 floppy46.fs
1474560 Nov 30 10:28 floppyB46.fs | 1474560 Nov 30 17:28 floppyB46.fs
1474560 Nov 30 10:28 floppyC46.fs | 1474560 Nov 29 15:06 floppyC46.fs
2618096 Nov 30 10:28 game46.tgz | 2618027 Nov 29 15:06 game46.tgz
269 Nov 30 10:28 index.txt | 1710 Nov 30 11:05 index.txt
249755648 Nov 30 10:28 install46.iso | 249757696 Nov 29 15:06 install46.iso
8469803 Nov 30 10:27 man46.tgz | 8469749 Nov 29 15:06 man46.tgz
2936282 Nov 30 10:27 misc46.tgz | 2936230 Nov 29 15:06 misc46.tgz
53532 Nov 30 10:28 pxeboot | 53532 Nov 29 15:06 pxeboot
15527631 Nov 29 08:02 xbase46.tgz | 15527631 Nov 29 15:02 xbase46.tgz
68103 Nov 29 08:02 xetc46.tgz | 68103 Nov 29 15:02 xetc46.tgz
35215028 Nov 29 08:02 xfont46.tgz | 35215028 Nov 29 15:02 xfont46.tgz
19874775 Nov 29 08:02 xserv46.tgz | 19874775 Nov 29 15:02 xserv46.tgz
2950471 Nov 29 08:02 xshare46.tgz | 2950471 Nov 29 15:02 xshare46.tgz
3 SHA256-vienna : Files SHA256 and SHA256-vienna differ
Here the SHA256 has been synced, but but files like
'floppyC46.fs' and
'man46.tgz' have not.
A more obvious case where the '
Erlangen' mirror in Germany was still busy updating the X .tgz files. Note that this example dates from Dec 2008, when there were still MD5 checksums files instead of SHA256.
Code:
1 SHA256-calyx : Files SHA256 and SHA256-calyx differ
2 SHA256-erlangen : =====Match====
--- sdiff of ftp.openbsd.org & erlangen ---
94576 Dec 9 18:53 INSTALL.amd64 | 10223616 Dec 11 04:25 .xserv44.tgz.G22368
807 Dec 9 18:54 MD5 | 94576 Dec 10 02:53 INSTALL.amd64
52334144 Dec 9 18:54 base44.tgz | 807 Dec 10 02:54 MD5
7223978 Dec 9 18:54 bsd | 52334144 Dec 10 02:54 base44.tgz
7342003 Dec 9 18:54 bsd.mp | 7223978 Dec 10 02:54 bsd
6315592 Dec 9 18:54 bsd.rd | 7342003 Dec 10 02:54 bsd.mp
6430720 Dec 9 18:54 cd44.iso | 6315592 Dec 10 02:54 bsd.rd
42940 Dec 9 18:54 cdboot | 6430720 Dec 10 02:54 cd44.iso
2048 Dec 9 18:54 cdbr | 42940 Dec 10 02:54 cdboot
102090065 Dec 9 18:54 comp44.tgz | 2048 Dec 10 02:54 cdbr
511724 Dec 9 18:54 etc44.tgz | 102090065 Dec 10 02:54 comp44.tgz
1474560 Dec 9 18:54 floppy44.fs | 511724 Dec 10 02:54 etc44.tgz
2706370 Dec 9 18:54 game44.tgz | 1474560 Dec 10 02:54 floppy44.fs
215 Jul 2 04:06 index.txt | 2706370 Dec 10 02:54 game44.tgz
260915200 Dec 9 18:54 install44.iso | 215 Jul 2 12:06 index.txt
8214814 Dec 9 18:53 man44.tgz | 260915200 Dec 10 02:54 install44.iso
2935078 Dec 9 18:53 misc44.tgz | 8214814 Dec 10 02:53 man44.tgz
53268 Dec 9 18:54 pxeboot | 2935078 Dec 10 02:53 misc44.tgz
11110781 Dec 9 18:44 xbase44.tgz | 53268 Dec 10 02:54 pxeboot
78441 Dec 9 18:44 xetc44.tgz | 11110781 Dec 10 02:44 xbase44.tgz
35579685 Dec 9 18:44 xfont44.tgz | 78441 Dec 10 02:44 xetc44.tgz
21467266 Dec 9 18:44 xserv44.tgz | 35579685 Dec 10 02:44 xfont44.tgz
2778752 Dec 9 18:44 xshare44.tgz | 21467273 Dec 9 00:17 xserv44.tgz
> 2778563 Dec 9 00:17 xshare44.tgz
The
'sdiff' output shows a '
.xserv44.tgz.G22368' file, while there still is a '
xserv44.tgz'. Clearly a temporary file to store the newest version, before removing the old one.
After using
'chk' and
'differ' a couple of times, your non-artificial neural network becomes trained in spotting these differences. Of course this will be easier for American continentals, with only a couple of hours time difference, then the poor Europeans and Asians who have to deal with much larger time offsets.
2.2.1 Time stamp of SHA256 file
The X installation file sets
'xbase??.tgz',
'xetc??tgz',
'xfont??.tgz',
'xserv??.tgz' and
'xshare46.tgz' are usually created separately from the base install sets. Because the SHA256 file contains the checksums of both X and the base install sets this means that that SHA256 can have either the time stamp of the base install sets or the X sets.
2.3 Check mirror coherence of the packages with 'pkg-mirror-complete'
Without any tools, forming an opinion about the mirror status of the nearly 6000 pre-compiled packages, is very difficult. The small
'pkg-mirror-complete' script attempts to give an assessment of the coherence.
To check whether '
ftp.openbsd.org' has a coherent set of packages:
Code:
$ ./pkg-mirror-complete
Unique dates from
./NOW/latest_pkg:
===============================
1 11-21 11:34
2 11-21 11:35
3 11-21 11:36
4 11-21 11:37
5 11-21 11:38
6 11-21 11:39
7 11-21 11:40
8 11-21 11:41
9 11-21 11:42
10 11-21 11:43
11 11-29 04:05
===============================
Last entry:
11-29 04:05 index.txt 456441
All files with last date:
"11-29 04:05"
===============================
11-29 04:05 index.txt 456441
===============================
We see a sorted listing of the unique dates retrieved from
'latest_pkg'. The time stamps starting at 21st of Nov 11:34 up to 11:43 follow a regular pattern of 1 minute increments. Only dissonant is the last 11-29 04:05 time stamp. The only file with this time stamp is a file called
'index.txt', which usually is generated hours after a change of directory contents.
The same check but now for the mirror in Stockholm, Sweden
Code:
$ pkg-mirror-complete stockholm
Unique dates from
./NOW/latest_pkg-stockholm:
===============================
1 11-21 18:34
2 11-21 18:35
3 11-21 18:36
4 11-21 18:37
5 11-21 18:38
6 11-21 18:39
7 11-21 18:40
8 11-21 18:41
9 11-21 18:42
10 11-21 18:43
11 11-29 11:05
===============================
Last entry:
11-29 11:05 index.txt 456441
All files with last date:
"11-29 11:05"
===============================
11-29 11:05 index.txt 456441
===============================
The 'stockholm' parameter selects the
'latest_pkg-stockholm' as data source for
'pkg-mirror-complete'.
We see the same pattern, although the times show a time zone induced difference of 7 hours. We can rather safely assume that the packages of the Stockholm mirror are completely mirrored
The Dutch nluug mirror caught while updating the packages to the 24th of Dec ones:
Code:
$ pkg-mirror-complete nluug
Unique dates from
./NOW/latest_pkg-nluug:
===============================
1 11-15 16:07
2 11-21 18:34
3 11-21 18:35
4 11-21 18:36
5 11-21 18:37
6 11-21 18:38
7 11-21 18:39
8 11-21 18:40
9 11-21 18:41
10 11-21 18:42
11 11-21 18:43
12 11-22 11:05
===============================
Last entry:
11-22 11:05 index.txt 456376
All files with last date:
"11-22 11:05"
===============================
11-22 11:05 index.txt 456376
===============================
In such cases we better wait before installing of upgrading the new packages.
2.3.1 A side-by-side comparision with 'pkg-differ'
Code:
$ pkg-differ nluug
Unique dates from Unique dates from
./NOW/latest_pkg: | ./NOW/latest_pkg-nluug:
=============================== ===============================
1 11-21 11:34 | 1 11-21 18:34
2 11-21 11:35 | 2 11-21 18:35
3 11-21 11:36 | 3 11-21 18:36
4 11-21 11:37 | 4 11-21 18:37
5 11-21 11:38 | 5 11-21 18:38
6 11-21 11:39 | 6 11-21 18:39
7 11-21 11:40 | 7 11-21 18:40
8 11-21 11:41 | 8 11-21 18:41
9 11-21 11:42 | 9 11-21 18:42
10 11-21 11:43 | 10 11-21 18:43
11 11-28 04:05 | 11 11-27 11:05
=============================== ===============================
Last entry: Last entry:
11-28 04:05 index.txt 456441 | 11-27 11:05 index.txt 456441
All files with last date: All files with last date:
"11-28 04:05" | "11-27 11:05"
=============================== ===============================
11-28 04:05 index.txt 456441 | 11-27 11:05 index.txt 456441
=============================== ===============================
As can be seen, this simply runs
pkg-differ-complete for ftp.openbsd.org and a mirror.
The following shows the Vienna mirror not having completed syncing for both the Nov 15th and 21st package build.
Code:
$ pkg-differ vienna
Unique dates from Unique dates from
./NOW/latest_pkg: | ./NOW/latest_pkg-vienna:
=============================== ===============================
1 11-21 11:34 | 1 11-02 20:31
2 11-21 11:35 | 2 11-02 20:32
3 11-21 11:36 | 3 11-02 20:33
4 11-21 11:37 | 4 11-02 20:34
5 11-21 11:38 | 5 11-02 20:35
6 11-21 11:39 | 6 11-02 20:36
7 11-21 11:40 | 7 11-02 20:37
8 11-21 11:41 | 8 11-02 20:38
9 11-21 11:42 | 9 11-02 20:39
10 11-21 11:43 | 10 11-02 20:40
11 11-28 04:05 | 11 11-15 15:58
> 12 11-15 15:59
[snip]
> 18 11-15 16:06
> 19 11-15 16:07
> 20 11-21 18:34
> 21 11-21 18:35
[snip]
> 28 11-21 18:42
> 29 11-21 18:43
> 30 11-28 11:05
=============================== ===============================
Last entry: Last entry:
11-28 04:05 index.txt 456441 | 11-28 11:05 index.txt 456441
All files with last date: All files with last date:
"11-28 04:05" | "11-28 11:05"
=============================== ===============================
11-28 04:05 index.txt 456441 | 11-28 11:05 index.txt 456441
=============================== ===============================
Another example where the Vienna mirror seems to synchronize to a fresh directory.
Code:
$ pkg-differ vienna
Unique dates from Unique dates from
./NOW/latest_pkg: | .//NOW/latest_pkg-vienna
=============================== ===============================
1 10-26 10:43 | 1 10-26 16:43
2 10-26 10:44 | 2 10-26 16:44
3 10-26 10:45 | 3 10-26 16:45
4 10-26 10:46 | 4 10-26 16:46
5 10-26 10:47 | 5 10-26 16:47
6 10-26 10:48 | 6 10-26 16:48
7 10-26 10:49 | 7 10-26 16:49
8 10-26 10:50 | 8 10-26 16:50
9 10-26 10:51 | 9 10-26 16:51
10 10-26 10:52 <
11 10-28 04:05 <
=============================== ===============================
Last entry: Last entry:
10-28 04:05 index.txt 450746 | 10-26 16:51 tutris-1.0.1p1.tgz 5294
All files with last date: All files with last date:
"10-28 04:05" | "10-26 16:51"
=============================== ===============================
10-28 04:05 index.txt 450746 | 10-26 16:51 tomboy-1.0.0.tgz 8839239
> 10-26 16:51 tomcat-4.1.40.tgz 474034
> 10-26 16:51 tomcat-5.5.27p1.tgz 4928
> 10-26 16:51 tomcat-6.0.20.tgz 498773
> 10-26 16:51 tomcat-admin-4.1.40p0.tg
> 10-26 16:51 tomcat-admin-5.5.27p0.tg
> 10-26 16:51 tt-rss-1.3.3.tgz 802688
[snip]
> 10-26 16:51 ttcp-1.12.tgz 9450
> 10-26 16:51 ttyrec-1.0.8p0.tgz 11201
> 10-26 16:51 turba-h3-2.3.2.tgz 23986
> 10-26 16:51 tutris-1.0.1p1.tgz 52947
=============================== ===============================
2.3.2 The Sparc64 packages
The approach from the
pkg-mirror-complete and
pkg-differ scripts work well for most architectures. A major pain, however are the packages for the sparc64. Instead of the 10 to 15 different time stamps there are more than 2645!.
Code:
Unique dates from Unique dates from
./NOW/latest_pkg: | ./NOW/latest_pkg-stockholm:
=============================== ===============================
1 10-25 06:58 | 1 10-25 12:58
2 10-25 06:59 | 2 10-25 12:59
3 10-25 07:00 | 3 10-25 13:00
4 10-25 07:01 | 4 10-25 13:01
5 10-25 07:02 | 5 10-25 13:02
[snip]
2644 10-31 22:01 | 2644 11-01 04:01
2645 10-31 23:05 | 2645 11-01 05:05
2646 10-31 23:16 | 2646 11-01 05:16
2647 11-29 04:05 | 2647 11-29 11:05
=============================== ===============================
Last entry: Last entry:
11-29 04:05 index.txt 434648 | 11-29 11:05 index.txt 434648
All files with last date: All files with last date:
"11-29 04:05" | "11-29 11:05"
=============================== ===============================
11-29 04:05 index.txt 434648 | 11-29 11:05 index.txt 434648
=============================== ===============================
The author, not familiar with the sparc64, just wonders if it would be feasible to
touch the compiled packages to a single time stamp before uploading.
2.4 Save ftp mirror data with 'backup'
If a few hours later, we want to check for whether the Stockholm mirror has finished mirroring, we first save the current retrieved listings and SHA256 files with the
'backup' script.
Code:
$ backup -v
./backup: creating backup in S2009-11-30_22:21_UTC ...
drwxr-xr-x 2 j65 j65 512 Dec 1 18:55 S2009-11-30_22:21_UTC
./backup: linking PREV to S2009-11-30_22:21_UTC ...
lrwxr-xr-x 1 j65 j65 21 Dec 1 18:55 PREV -> S2009-11-30_22:21_UTC
total 1500
-rw-r--r-- 1 j65 j65 21 Nov 30 23:21 DATE
-rw-r--r-- 1 j65 j65 1726 Nov 23 16:37 SHA256
-rw-r--r-- 1 j65 j65 1726 Nov 23 16:37 SHA256-esat
-rw-r--r-- 1 j65 j65 1726 Nov 23 16:37 SHA256-funkfeuer
-rw-r--r-- 1 j65 j65 1726 Nov 23 16:37 SHA256-stockholm
-rw-r--r-- 1 j65 j65 908 Nov 30 23:21 latest
-rw-r--r-- 1 j65 j65 908 Nov 30 23:22 latest-esat
-rw-r--r-- 1 j65 j65 908 Nov 30 23:22 latest-funkfeuer
-rw-r--r-- 1 j65 j65 908 Nov 30 23:21 latest-stockholm
-rw-r--r-- 1 j65 j65 166644 Nov 30 23:21 latest_pkg
-rw-r--r-- 1 j65 j65 166644 Nov 30 23:22 latest_pkg-esat
-rw-r--r-- 1 j65 j65 216589 Nov 30 23:22 latest_pkg-funkfeuer
-rw-r--r-- 1 j65 j65 166644 Nov 30 23:21 latest_pkg-stockholm
This script copies the SHA256 files and ftp directory listings into a specially named new directory. The directory name consists of 'S' with the UTC time stamp appended. In this case
'S2009-11-30_22:21_UTC'.
After the files have been copied, the symbolic link PREV is pointed to
'S2009-11-30_22:21_UTC'. The directory NOW still has the original files. These will only be deleted during the next
'chk'.
'backup' will warn you if there already is a backup of the NOW directory.
Code:
$ ./backup
./backup : Backup already exists! Exiting ....
2.5 'show' time
In the explanation of the
'chk' output, we discussed a summary like this:
Code:
Current date : 2009-12-01_18.32_UTC
NOW date : 2009-11-29_20:20_UTC
PREV date : 2009-11-27_20:49_UTC
--------------- NOW ------------------------
2084 Nov 28 06:43:25 2009 SHA256-vienna
2084 Nov 28 06:43:25 2009 SHA256-stockholm
2084 Nov 28 06:43:25 2009 SHA256
--------------- PREV -------------------
2084 Nov 22 18:13:09 2009 SHA256-vienna
2084 Nov 27 01:11:32 2009 SHA256-stockholm
2084 Nov 27 01:11:32 2009 SHA256
--------------- X NOW ----------------------
17248633 Nov 27 22:43 xbase46.tgz
69479 Nov 27 22:43 xetc46.tgz
35217656 Nov 27 22:43 xfont46.tgz
11389029 Nov 27 22:43 xserv46.tgz
2952171 Nov 27 22:43 xshare46.tgz
--------------- X PREV ----------------------
17247839 Nov 22 10:13 xbase46.tgz
69495 Nov 22 10:13 xetc46.tgz
35217765 Nov 22 10:13 xfont46.tgz
11408354 Nov 22 10:13 xserv46.tgz
2951722 Nov 22 10:13 xshare46.tgz
--------------- PKG NOW --------------------
145282 Oct 28 09:48 9libs-1.0p5.tgz
11950 Oct 29 12:46 9menu-1.7p0.tgz
.... [snip] ...............
111247 Oct 27 12:48 zziplib-0.13.52.tgz
129514 Oct 27 09:45 zzuf-0.12.tgz
--------------- PKG PREV --------------------
145282 Oct 28 09:48 9libs-1.0p5.tgz
11950 Oct 29 12:46 9menu-1.7p0.tgz
.... [snip] ...............
111247 Oct 27 12:48 zziplib-0.13.52.tgz
129514 Oct 27 09:45 zzuf-0.12.tgz
--------------------------------------
Actually the
'chk' script calls the
'show' script to produce this summary. You also can run it any time.