DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 9th September 2015
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default Keeping OpenBSD -stable packages up to date

Per FAQ 15.3.10, security flaws and serious bugs may occur with third party packages. The Project will develop -stable ports, leaving it to the user community to build -stable packages from them. Or, users may choose to use M:Tier's -stable services, for packages and for systems. M:Tier is a third party provider of OpenBSD services with several Project developers on staff.

This guide is currently applicable to OpenBSD 5.7, and contains comments regarding OpenBSD 5.8.

Audience

This guide is intended for the admin who needs to build -stable package collections from ports.

Ports building can be daunting. When a problem occurs, the admin needs to be able to read logs, determine a root cause, and effect repairs. If that is not an appropriate use of your time, then M:Tier's -stable services may meet needs admirably.

The reader is expected to be an OpenBSD user who has already built and installed packages from ports. The reader should also be familiar with the basic phases of port building, and understands the concepts of the key targets: fetch, extract, patch, config, build, fake, package, and install.

Obtaining -stable sources, building -stable systems, and building -stable releases are are covered in FAQ 5, the "Following -stable" FAQ, and in the release(8) man page. I expect the reader has already used the cvs(1) command to check out the ports tree. There are ports which require the xenocara or the src trees; the reader should already have checked out these trees or can recognize a build failure caused by not having one of these trees available, and acquire the necessary tree.

This guide begins at the point where the admin has a -stable ports tree checked out, and asks, “Now, what?

----------

Step 0. Level set

Mixing and matching flavors

Because there are no changes in OS APIs and supporting libraries between -release and -stable, we can freely mix and match -release and -stable ports. This guide recommends two package repositories: -release packages, located at the nearest Project mirror, and -stable packages, stored on a local server. This allows the target -stable system being maintained to use -stable packages if available, and Project provided -release packages otherwise.

Depending on dependencies

When you install an application with pkg_add(1), often additional packages will be installed at the same time. These are application libraries or subordinate applications which your application depends on to run properly. These are run dependencies.

If a run dependency gets a -stable update, it will need to be rebuilt, and its updated package installed. And, the application that depends upon it will need to be rebuilt and reinstalled, even if it does not have a -stable update of its own.

When a port is built, there may be additional dependencies, needed only to build the application package from source. These are build dependencies. They are not needed to run the application, but they must be installed to build it.

Both run and build dependencies create dependency chains. And these can get quite long. You may wonder, if you are only interested in six ports A through F with -stable updates, why you are building dozens of ports. It is the dependencies: Port A may have a build dependency on port U, which must be built and installed, and it has a run dependency on port R, which has a run dependency on port Y, which has a build dependency on port G.

The tools you will use manage these dependency chains for you.

Manual and automatic packages

When you install with pkg_add(1), any named package will be recorded as having been manually installed, while unnamed, run dependencies will be recorded as having been automatically installed.

This makes it easy to remove old dependencies which are no longer needed. The -a option of pkg_delete(1) removes any automatically installed packages which are no longer required by any other installed package.

Step 1: Update your ports tree

From time to time, ports which have -stable updates will be committed. Use the cvs(1) update command whenever you want to apply recently committed changes to your ports tree.

To make it easier to see any updates, use the -q (quiet) option with the cvs update command ("cvs -q up") so that any port updates since the last update -- or from the initial checkout if this is your first update -- can be more easily seen.

Assuming your build machine has a -stable ports tree in /usr/ports, and a $CVSROOT is either established in an environment variable or had been previously provisioned for the tree, an update step looks like this:
Code:
  $ cd /usr/ports
  $ cvs -q up -Pd
If there are any -stable changes to the tree since the last update, or since you checked out the tree if no update has been run, they will be sent to standard output. This can displayed in the terminal window or console, be directed to a file, or if tee(1) is used, be directed to both a file and window/console.

Any files patched will be noted with:

P <file>

Any files replaced will be noted with:

U <file>

If there are local modifications, these will be shown with:

M <file>

If there are any conflicts that cvs(1) cannot resolve, they will be noted with:

C <file>

Files noted with M or C will not be updated. If you do not need the local modifications (M files), you may overwrite these files by re-running the update with the -C (clear) option, and the locally modified files will be moved to “dotfiles” in the same directory, should you want to have them for historical purposes.
Code:
  $ cd /usr/ports
  $ cvs -q up -PdC
Any file conflicts noted with C are unlikely. But if they occur, they will need to be manually resolved.

Step 2. Check for any installed packages which have been revised in the ports tree

2.a. Remove any unneeded dependencies

On any -stable system you will be checking, first remove any extra packages which are no longer needed. These may be packages that were installed as run dependencies, but the application that needed them was later removed. Or, if this is a system where ports are built, they may be build dependencies for a previously built port.

# pkg_delete -a

2.b. Run out-of-date(1), saving a copy of its report to a file

The out-of-date(1) program inspects the installed package database in /var/db/pkg against the ports tree, and will produce a report for any package which has been updated, or, any package with a run dependency that has been updated. The report is sent to standard output, which can be directed to a file, and if tee(1) is used, will be copied to the console.

$ /usr/ports/infrastructure/bin/out-of-date | tee my.report

For any out of date packages, the report will contain two fields: the first field is the name of the port in pkgpath(7) format, the second field a comment as to the reason it is flagged as out of date.

Step 3. Prepare to build ports

3.a. Ensure you have sufficient free space for building

If ports are needed to be built, you will build them in $WRKOBJDIR, which defaults to /usr/ports/pobj. While the build process will automatically clean this directory as packages are built, some larger ports require significant storage – 10 to 20 GB or more.You may need to restructure filesystems, set this environment variable to an alternate location, or mount a filesystem at /usr/ports/pobj as needed.

3.b. Combine out-of-date reports for multiple -stable systems

If you have multiple -stable systems, combine out-of-date reports from systems of the same architecture into individual files, for building. It is fine to have the same pkgpath(7) mentioned multiple times, from systems that have the same package installed. They need to be a homogenous architecture: your sparc64 reports can be combined together, your i386 reports may be combined together, but do not combine your sparc64 and i386 reports. Each architecture must be kept separate.

Step 4. Build ports with dpb(1)

dpb(1) is the distributed port builder. It can build ports on a single system, or spread the workload across an array of build machines.

4.a. Check your privilege levels

Prior to OpenBSD 5.8, dpb builds with the authority of the building user, and will raise authority via the $SUDO environment variable for actions such as package installation. At OpenBSD 5.8, dpb starts as root and reduces privileges for fetch and build operations.

To raise authority (for package installations) at OpenBSD 5.7, set the $SUDO variable. Here is an example of a mk.conf(5) entry defining the $SUDO variable, using the -E option to retain environment variables:
Code:
SUDO=/usr/bin/sudo -E
4.b Run dpb, requesting an inspection of installed package signatures

$ /usr/ports/infrastructure/bin/dpb -R my.report

Where your my.report file contains a list of pkgpath(7) specifications determined by out-of-date(1). These are applications and run dependencies that require building. The -R option requests an inspection of package signatures, so that dpb will walk the complete dependency chain, and build any package that is determines is out of date and needs rebuilding

Looks pretty easy, doesn't it? It is, as long as there are no build problems.

Before doing your first dpb build – read the dpb(1) man page. Then, read it again. Then, be sure it is available while you run dpb(1). A very small report is displayed on a console or terminal window while dpb(1) is running. That report will tell you a lot, but only if you have the man page.

This tool does a great deal, documented in its logs in far more detail than the small amount shown on screen. It does distribution file fetching, port building, dependency installing, deinstalling unneeded dependencies, cleaning unneeded files after package builds complete ... all with very little output to the console or terminal window. If you've run it before, it knows big ports from small ones and uses heuristics to determine the most efficient package build order.

If there are any problems, dpb(1) will tell you which package paths have failed on its console report. The details will be stored in detailed logs, located in /usr/ports/logs/<architecture>. Port building logs are found in this directory structure two ways: by package name, or by pkgpath(7). These are located with /usr/ports/logs/<architecture>/packages or /usr/ports/logs/<architecture>/paths, respectively.

If you are able to determine the cause of any problem, and can correct it, you can delete the failed port's lockfile in /usr/ports/logs/<architecture>/locks, and if dpb(1) is still running it will automatically restart the build of that port when a build slot opens.

5. Clean the build machine of unneeded packages.

You may have hundreds of excess packages installed at the end of a successful dpb(1) build. Remove them, as they are no longer needed.

# pkg_delete -a

6. Make your -stable package repository primary, the -release repository secondary.

Whether you use $PKG_PATH or you use installpath in pkg.conf(5), you should place your -stable repository ahead of the Project's -release mirror. On the build machine, the files are stored in /usr/ports/packages/<arch>/all – and you can make these available to other -stable systems via webserver, NFS, or FTP server.

Here is an example, using pkg.conf(5), where the -stable packages are on the same system:
Code:
installpath = /usr/ports/packages/i386/all/
installpath += http://<my.nearest.Project.mirror>/pub/OpenBSD/5.7/packages/i386/
Here is a similar pkg.conf(5), where the -stable packages are on a local web server:
Code:
installpath = http://<my.local.server>/path/to/sparc64/stable/packages/
installpath += http://<my.nearest.Project.mirror>/pub/OpenBSD/5.7/packages/sparc64/
For $PKG_PATH, the repositories need to be separated by a colon, as described in pkg_add(1):
Code:
  export PKG_PATH=/usr/ports/packages/powerpc/all/:http://<my.nearest.Project.mirror>/pub/OpenBSD/5.7/packages/powerpc/
With 5.8, there are shortcuts for architecture and release for both $PKG_PATH and pkg.conf(5).

7. Update your installed packages

On every -stable system, use # pkg_add -u. As these were unsigned packages you built yourself, pkg_add(1) will prompt to confirm that unsigned packages should be installed. If you are updating many -stable systems without interaction, the -D unsigned option of pkg_add(1) will permit the installation of unsigned packages without prompting.

Last edited by jggimi; 3rd November 2015 at 11:59 AM. Reason: moved the code in step 2.b for clarity - thanks to a lurker's Email!!
Reply With Quote
  #2   (View Single Post)  
Old 26th October 2015
blackhole's Avatar
blackhole blackhole is offline
Spam Deminer
 
Join Date: Mar 2014
Posts: 316
Default

Excellent guide. I had not used dpb(1) as yet as I was trying to get to grips with understanding how to rebuild stable ports manually and had not had the time to sit down and look into it. Now all updated and installed.

Thanks
Reply With Quote
  #3   (View Single Post)  
Old 12th January 2016
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

I'd converted this guide into a presentation I gave last month at a BSD user group meeting. PDF available here.

Luckily, neither video nor audio was recorded, so you can't hear my inane babble.
Reply With Quote
  #4   (View Single Post)  
Old 26th August 2017
blackhole's Avatar
blackhole blackhole is offline
Spam Deminer
 
Join Date: Mar 2014
Posts: 316
Default

It's ages since I built ports using dpb and now I find:

Code:
$ man 1 dpb         
man: No entry for dpb(1) in the manual.
Quote:
At OpenBSD 5.8, dpb starts as root and reduces privileges for fetch and build operations.
I don't think I've built ports with dpb(1) since 5.7 in fact...

Now I'm confronted with build failures - from a clean 6.1-stable source tree fetched via cvs.

So what happened to the manual? Considering dpb(1) is part of the ports tree, maybe this is normal, but how do I find/install it's man pages...?

Thanks
Reply With Quote
  #5   (View Single Post)  
Old 27th August 2017
hanzer's Avatar
hanzer hanzer is offline
Real Name: Adam Jensen
just passing through
 
Join Date: Oct 2013
Location: EST USA
Posts: 314
Default

I found the man page here: $ man -m /usr/ports/infrastructure/man dpb
Reply With Quote
  #6   (View Single Post)  
Old 27th August 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

And you'll find the binary in the /usr/ports/infrastructure branch as well.
Reply With Quote
  #7   (View Single Post)  
Old 27th August 2017
blackhole's Avatar
blackhole blackhole is offline
Spam Deminer
 
Join Date: Mar 2014
Posts: 316
Default

Thanks, found it but still not able to build anything, despite specifying the build and fetch users. I'll keep at it.
Reply With Quote
  #8   (View Single Post)  
Old 28th August 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

The security model has changed since you last used dpb, and since I'd written that tutorial. You must start dpb as root. See "THE SECURITY MODEL OF DPB" in the man page.
Reply With Quote
  #9   (View Single Post)  
Old 29th August 2017
blackhole's Avatar
blackhole blackhole is offline
Spam Deminer
 
Join Date: Mar 2014
Posts: 316
Default

Yes, I have been running as root, but every package fetch/build errors.

I've checked out the -stable ports tree. As I understand it (and from reading the mailing list thread when this was first introduced) dpb drops privileges for build and fetch. So it's possible I have the wrong permissions on the ports tree or am missing something obvious?!

I set the permissions/group on the ports tree as per "Avoiding root privileges":

https://www.openbsd.org/faq/faq5.html#Bld

From dpb(1), the build_user defaults to whoami(1), which will be root. I have tried setting this to my own user (which is in the wsrc group) and still get all errors.

Out of ideas.

Thanks
Reply With Quote
Old 29th August 2017
jggimi's Avatar
jggimi jggimi is offline
More noise than signal
 
Join Date: May 2008
Location: USA
Posts: 7,977
Default

Let's start with fetch. FETCH_USER, if not set by you, will default to _pfetch. This user requires write access to DISTDIR. (Default: /usr/ports/distfiles) You can either set FETCH_USER or provide write access to _pfetch. FETCH_USER is set with -D.

You may set build_user within a -h hosts file, or you may set it with -D.
Per the security model section, this user must be able to write to WRKOBJDIR and have read access to the rest of the ports tree. If you don't set it, it defaults to _pbuild.

The whoami(1) model is only used on remote hosts.

I'm going to guess that you must explicitly set FETCH_USER and build_user, or you must change the access/ownership model of your ports tree to support using the default userids. I've set the two userids, rather than mucking about with ports tree ownerships.
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Clean install of OpenBSD but keeping existing partition table daemonbak OpenBSD Installation and Upgrading 8 27th May 2015 10:39 PM
HOWTO: keeping FreeBSD's base system and packages up-to-date vermaden Guides 6 13th May 2013 08:51 AM
-stable snapshots "openbsd-stable.org" (?) dnix OpenBSD Installation and Upgrading 9 18th December 2011 12:48 PM
Running -stable and using packages ai-danno OpenBSD Installation and Upgrading 4 4th June 2008 03:18 AM
Keeping ports in multiple jails up to date cajunman4life FreeBSD Installation and Upgrading 1 9th May 2008 11:51 PM


All times are GMT. The time now is 12:23 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick