Keeping OpenBSD -stable packages up to date
Per FAQ 15.3.10, security flaws and serious bugs may occur with third party packages. The Project will develop -stable ports, leaving it to the user community to build -stable packages from them. Or, users may choose to use M:Tier's -stable services, for packages and for systems. M:Tier is a third party provider of OpenBSD services with several Project developers on staff.
This guide is currently applicable to OpenBSD 5.7, and contains comments regarding OpenBSD 5.8.
This guide is intended for the admin who needs to build -stable package collections from ports.
Ports building can be daunting. When a problem occurs, the admin needs to be able to read logs, determine a root cause, and effect repairs. If that is not an appropriate use of your time, then M:Tier's -stable services may meet needs admirably.
The reader is expected to be an OpenBSD user who has already built and installed packages from ports. The reader should also be familiar with the basic phases of port building, and understands the concepts of the key targets: fetch, extract, patch, config, build, fake, package, and install.
Obtaining -stable sources, building -stable systems, and building -stable releases are are covered in FAQ 5, the "Following -stable" FAQ, and in the release(8) man page. I expect the reader has already used the cvs(1) command to check out the ports tree. There are ports which require the xenocara or the src trees; the reader should already have checked out these trees or can recognize a build failure caused by not having one of these trees available, and acquire the necessary tree.
This guide begins at the point where the admin has a -stable ports tree checked out, and asks, “Now, what?”
Step 0. Level set
Mixing and matching flavors
Because there are no changes in OS APIs and supporting libraries between -release and -stable, we can freely mix and match -release and -stable ports. This guide recommends two package repositories: -release packages, located at the nearest Project mirror, and -stable packages, stored on a local server. This allows the target -stable system being maintained to use -stable packages if available, and Project provided -release packages otherwise.
Depending on dependencies
When you install an application with pkg_add(1), often additional packages will be installed at the same time. These are application libraries or subordinate applications which your application depends on to run properly. These are run dependencies.
If a run dependency gets a -stable update, it will need to be rebuilt, and its updated package installed. And, the application that depends upon it will need to be rebuilt and reinstalled, even if it does not have a -stable update of its own.
When a port is built, there may be additional dependencies, needed only to build the application package from source. These are build dependencies. They are not needed to run the application, but they must be installed to build it.
Both run and build dependencies create dependency chains. And these can get quite long. You may wonder, if you are only interested in six ports A through F with -stable updates, why you are building dozens of ports. It is the dependencies: Port A may have a build dependency on port U, which must be built and installed, and it has a run dependency on port R, which has a run dependency on port Y, which has a build dependency on port G.
The tools you will use manage these dependency chains for you.
Manual and automatic packages
When you install with pkg_add(1), any named package will be recorded as having been manually installed, while unnamed, run dependencies will be recorded as having been automatically installed.
This makes it easy to remove old dependencies which are no longer needed. The -a option of pkg_delete(1) removes any automatically installed packages which are no longer required by any other installed package.
Step 1: Update your ports tree
From time to time, ports which have -stable updates will be committed. Use the cvs(1) update command whenever you want to apply recently committed changes to your ports tree.
To make it easier to see any updates, use the -q (quiet) option with the cvs update command ("cvs -q up") so that any port updates since the last update -- or from the initial checkout if this is your first update -- can be more easily seen.
Assuming your build machine has a -stable ports tree in /usr/ports, and a $CVSROOT is either established in an environment variable or had been previously provisioned for the tree, an update step looks like this:
$ cd /usr/ports $ cvs -q up -Pd
Any files patched will be noted with:
Any files replaced will be noted with:
If there are local modifications, these will be shown with:
If there are any conflicts that cvs(1) cannot resolve, they will be noted with:
Files noted with M or C will not be updated. If you do not need the local modifications (M files), you may overwrite these files by re-running the update with the -C (clear) option, and the locally modified files will be moved to “dotfiles” in the same directory, should you want to have them for historical purposes.
$ cd /usr/ports $ cvs -q up -PdC
Step 2. Check for any installed packages which have been revised in the ports tree
2.a. Remove any unneeded dependencies
On any -stable system you will be checking, first remove any extra packages which are no longer needed. These may be packages that were installed as run dependencies, but the application that needed them was later removed. Or, if this is a system where ports are built, they may be build dependencies for a previously built port.
2.b. Run out-of-date(1), saving a copy of its report to a file
The out-of-date(1) program inspects the installed package database in /var/db/pkg against the ports tree, and will produce a report for any package which has been updated, or, any package with a run dependency that has been updated. The report is sent to standard output, which can be directed to a file, and if tee(1) is used, will be copied to the console.
For any out of date packages, the report will contain two fields: the first field is the name of the port in pkgpath(7) format, the second field a comment as to the reason it is flagged as out of date.
Step 3. Prepare to build ports
3.a. Ensure you have sufficient free space for building
If ports are needed to be built, you will build them in $WRKOBJDIR, which defaults to /usr/ports/pobj. While the build process will automatically clean this directory as packages are built, some larger ports require significant storage – 10 to 20 GB or more.You may need to restructure filesystems, set this environment variable to an alternate location, or mount a filesystem at /usr/ports/pobj as needed.
3.b. Combine out-of-date reports for multiple -stable systems
If you have multiple -stable systems, combine out-of-date reports from systems of the same architecture into individual files, for building. It is fine to have the same pkgpath(7) mentioned multiple times, from systems that have the same package installed. They need to be a homogenous architecture: your sparc64 reports can be combined together, your i386 reports may be combined together, but do not combine your sparc64 and i386 reports. Each architecture must be kept separate.
Step 4. Build ports with dpb(1)
dpb(1) is the distributed port builder. It can build ports on a single system, or spread the workload across an array of build machines.
4.a. Check your privilege levels
Prior to OpenBSD 5.8, dpb builds with the authority of the building user, and will raise authority via the $SUDO environment variable for actions such as package installation. At OpenBSD 5.8, dpb starts as root and reduces privileges for fetch and build operations.
To raise authority (for package installations) at OpenBSD 5.7, set the $SUDO variable. Here is an example of a mk.conf(5) entry defining the $SUDO variable, using the -E option to retain environment variables:
Where your my.report file contains a list of pkgpath(7) specifications determined by out-of-date(1). These are applications and run dependencies that require building. The -R option requests an inspection of package signatures, so that dpb will walk the complete dependency chain, and build any package that is determines is out of date and needs rebuilding
Looks pretty easy, doesn't it? It is, as long as there are no build problems.
Before doing your first dpb build – read the dpb(1) man page. Then, read it again. Then, be sure it is available while you run dpb(1). A very small report is displayed on a console or terminal window while dpb(1) is running. That report will tell you a lot, but only if you have the man page.
This tool does a great deal, documented in its logs in far more detail than the small amount shown on screen. It does distribution file fetching, port building, dependency installing, deinstalling unneeded dependencies, cleaning unneeded files after package builds complete ... all with very little output to the console or terminal window. If you've run it before, it knows big ports from small ones and uses heuristics to determine the most efficient package build order.
If there are any problems, dpb(1) will tell you which package paths have failed on its console report. The details will be stored in detailed logs, located in /usr/ports/logs/<architecture>. Port building logs are found in this directory structure two ways: by package name, or by pkgpath(7). These are located with /usr/ports/logs/<architecture>/packages or /usr/ports/logs/<architecture>/paths, respectively.
If you are able to determine the cause of any problem, and can correct it, you can delete the failed port's lockfile in /usr/ports/logs/<architecture>/locks, and if dpb(1) is still running it will automatically restart the build of that port when a build slot opens.
5. Clean the build machine of unneeded packages.
You may have hundreds of excess packages installed at the end of a successful dpb(1) build. Remove them, as they are no longer needed.
6. Make your -stable package repository primary, the -release repository secondary.
Whether you use $PKG_PATH or you use installpath in pkg.conf(5), you should place your -stable repository ahead of the Project's -release mirror. On the build machine, the files are stored in /usr/ports/packages/<arch>/all – and you can make these available to other -stable systems via webserver, NFS, or FTP server.
Here is an example, using pkg.conf(5), where the -stable packages are on the same system:
installpath = /usr/ports/packages/i386/all/ installpath += http://<my.nearest.Project.mirror>/pub/OpenBSD/5.7/packages/i386/
installpath = http://<my.local.server>/path/to/sparc64/stable/packages/ installpath += http://<my.nearest.Project.mirror>/pub/OpenBSD/5.7/packages/sparc64/
7. Update your installed packages
On every -stable system, use
Last edited by jggimi; 3rd November 2015 at 11:59 AM. Reason: moved the code in step 2.b for clarity - thanks to a lurker's Email!!
Excellent guide. I had not used dpb(1) as yet as I was trying to get to grips with understanding how to rebuild stable ports manually and had not had the time to sit down and look into it. Now all updated and installed.
I'd converted this guide into a presentation I gave last month at a BSD user group meeting. PDF available here.
Luckily, neither video nor audio was recorded, so you can't hear my inane babble.
|Thread||Thread Starter||Forum||Replies||Last Post|
|Clean install of OpenBSD but keeping existing partition table||daemonbak||OpenBSD Installation and Upgrading||8||27th May 2015 10:39 PM|
|HOWTO: keeping FreeBSD's base system and packages up-to-date||vermaden||Guides||6||13th May 2013 08:51 AM|
|-stable snapshots "openbsd-stable.org" (?)||dnix||OpenBSD Installation and Upgrading||9||18th December 2011 12:48 PM|
|Running -stable and using packages||ai-danno||OpenBSD Installation and Upgrading||4||4th June 2008 03:18 AM|
|Keeping ports in multiple jails up to date||cajunman4life||FreeBSD Installation and Upgrading||1||9th May 2008 11:51 PM|