Automating patch generation and application for configuration files

[J65nko]

1. Automating patch generation and application for configuration files

• 1.1 Introduction
• 1.2 Patching '/etc/mail/aliases'manually
• 1.3 Automating patch creation
• 1.4 Using 'patchcreate' to create a 'sshd_config' patch
• 1.5 The 'patchcreate' script
• 1.6 Comparison of 'sshd' before and after patching

1.1 Introduction

Anyone following mailing lists will have seem messages where a developer asks an user to apply a source code patch for solving a problem or enabling support for a new piece of hardware. This is only one of the multiple uses of patches, they also can be used to customize configuration files.

What does the man pages for patch(1) tell us?

NAME
     patch - apply a diff file to an original
       
DESCRIPTION
     patch will take a patch file containing any of the four forms of differ-
     ence listing produced by the diff(1) program and apply those differences
     to an original file, producing a patched version.  If patchfile is omit-
     ted, or is a hyphen, the patch will be read from the standard input.

Because 'diff' is the program used to create a patch, an excerpt of it's man page

NAME
     diff - differential file and directory comparator
       
DESCRIPTION
     The diff utility compares the contents of file1 and file2 and writes to
     the standard output the list of changes necessary to convert one file in-
     to the other.  No output is produced if the files are identical.

[J65nko]

1.2 Patching '/etc/mail/aliases'manually

The OpenBSD afterboot(8) man page lists a configuration task we will automate with a patch.

Edit /etc/mail/aliases and set the three standard aliases to go to either
a mailing list, or the system administrator.
      
      # Well-known aliases -- these should be filled in!
      root:           sysadm
      manager:        root
      dumper:         root
       
Run newaliases(8) after changes.

We will use directory ORIG to save a copy of the original '/etc/mail/aliases' file. The edited version will be in directory NEW.

$ mkdir WORK ; cd WORK
$ mkdir NEW ORIG
$ cp /etc/mail/aliases . 
$ cp aliases ORIG
$ cp aliases NEW
$ vi NEW/aliases

After editing we create the patch with diff

$ diff -u ORIG/aliases NEW/aliases

--- ORIG/aliases        Sat Jan 23 03:00:16 2010
+++ NEW/aliases Sat Jan 23 03:03:54 2010
@@ -64,9 +64,9 @@
 sshd:   /dev/null
 
 # Well-known aliases -- these should be filled in!
-# root:
-# manager:
-# dumper:
+root: j65nko
+manager: j65nko
+dumper: j65nko
 
 # RFC 2142: NETWORK OPERATIONS MAILBOX NAMES
 abuse:         root
@@ -74,11 +74,11 @@
 security:      root
 
 # RFC 2142: SUPPORT MAILBOX NAMES FOR SPECIFIC INTERNET SERVICES
-# hostmaster:  root
-# usenet:      root
-# news:                usenet
-# webmaster:   root
-# ftp:         root
+hostmaster:    root
+usenet:        root
+news:          usenet
+webmaster:     root
+ftp:           root
 
 # uncomment this for msgs:
 # msgs: "|/usr/bin/msgs -s"

As you can see it is very easy to figure out the changes. The deleted lines from ORIG/aliases are prefixed with "---", the added lines in NEW/aliases with "+++".

A simple redirect will save the patch to file:

$ diff -u ORIG/aliases NEW/aliases >patch

Applying the patch is simple. We will use the following options:

     -b, --backup
             Save a backup copy of the file before it is modified.  By default
             the original file is saved with a backup extension of ".orig" un-
             less the file already has a numbered backup, in which case a num-
             bered backup is made.  This is equivalent to specifying "-V
             existing".  This option is currently the default, unless --posix
             is specified.

     -p strip-count, --strip strip-count
             Sets the pathname strip count, which controls how pathnames found
             in the patch file are treated, in case you keep your files in a
             different directory than the person who sent out the patch.  The
             strip count specifies how many slashes are to be stripped from
             the front of the pathname.  (Any intervening directory names also
             go away.)  For example, supposing the file name in the patch file
             was /u/howard/src/blurfl/blurfl.c:

             Setting -p0 gives the entire pathname unmodified.

$ patch -b -p0 aliases <patch

Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- ORIG/aliases       Sat Jan 23 03:00:16 2010
|+++ NEW/aliases        Sat Jan 23 03:03:54 2010
--------------------------
Patching file aliases using Plan A...
Hunk #1 succeeded at 64.
Hunk #2 succeeded at 74.
done

[J65nko]

1.3 Automating patch creation

In this section we will discuss the script 'patchcreate'. It will generate a patch, which subsequently is embedded in a shell script using a shell here document. Running the shell script will apply the patch.

The unmodified script patches the file in the current directory, just like we did manually earlier on. After a test confirms the proper working, only a simple edit is needed to have the patch script work on the real actual file. In our case "/etc/mail/aliases".

For the impatient we will look at the generated patch script first.

$ cat -n _patcher
     1  # ----------------------------------------------------------
     2  echo
     3  echo --- patch script for: aliases --- BEGIN
     4  
     5  # ---  edit the following line if needed
     6  FILE=./aliases
     7  
     8  patch -b -p0 ${FILE} <<END_OF_PATCH
     9  --- ORIG/aliases        Sat Jan 23 02:16:03 2010
    10  +++ NEW/aliases Sat Jan 23 02:17:39 2010
    11  @@ -64,9 +64,9 @@
    12   sshd:   /dev/null
    13   
    14   # Well-known aliases -- these should be filled in!
    15  -# root:
    16  -# manager:
    17  -# dumper:
    18  +root: j65nko
    19  +manager: j65nko
    20  +dumper: j65nko
    21   
    22   # RFC 2142: NETWORK OPERATIONS MAILBOX NAMES
    23   abuse:         root
    24  @@ -74,11 +74,11 @@
    25   security:      root
    26   
    27   # RFC 2142: SUPPORT MAILBOX NAMES FOR SPECIFIC INTERNET SERVICES
    28  -# hostmaster:  root
    29  -# usenet:      root
    30  -# news:                usenet
    31  -# webmaster:   root
    32  -# ftp:         root
    33  +hostmaster:    root
    34  +usenet:        root
    35  +news:          usenet
    36  +webmaster:     root
    37  +ftp:           root
    38   
    39   # uncomment this for msgs:
    40   # msgs: "|/usr/bin/msgs -s"
    41  END_OF_PATCH
    42  
    43  echo  --- patch script for: aliases --- END

The FILE variable in line 6 defines the target of the patch. By changing it to '/etc/mail/aliases' this file will be patched.

The patch command is on line 8, the end marker END_OF_PATCH of the here document at line 41.

An attentive reader may wonder why the typic shell shebang "#!/bin/sh" line is missing. The patch script is meant as component or a building block of a larger script. I use these kind of self-contained patch scripts in 'install.site' scripts used to customize an OpenBSD install. For details see Customizing the install process.

Of course you can run it stand-alone with a simple sh _patcher, e.g. as a post-install script for a FreeBSD or NetBSD system.


1.4 Using 'patchcreate' to create a 'sshd_config' patch

We will now see how the script automates the creation of a patch for the '/etc/sshd_config' file. First we will extract this ssh configuration file from a 'etc46.tgz' file.

$ tar tvzf /home/j65nko/Snapshots/etc46.tgz | grep ssh
 
drwx------  2 root     wheel            0 Jan 19 23:44 ./etc/skel/.ssh
-rw-------  1 root     wheel            0 Jan 19 23:44 ./etc/skel/.ssh/authorized_keys
drwxr-xr-x  2 root     wheel            0 Jan 19 23:44 ./etc/ssh
-rw-r--r--  1 root     wheel         1555 Jan 19 23:44 ./etc/ssh/ssh_config
-rw-r--r--  1 root     wheel         2524 Jan 19 23:44 ./etc/ssh/sshd_config

$ tar xvzf /home/j65nko/Snapshots.old/etc46.tgz ./etc/ssh/sshd_config 
./etc/ssh/sshd_config

$ ls -l etc/ssh/sshd_config
-rw-r--r--  1 j65nko  j65nko  2524 Jan 19 23:44 etc/ssh/sshd_config

$ cp etc/ssh/sshd_config .
$ ls -l sshd_config
-rw-r--r--  1 j65nko  wheel  2524 Jan 24 05:15 sshd_config

Having the file in our current directory, we can run 'patchcreate'

$ patchcreate sshd_config

Ok, found file sshd_config
Saving copy of sshd_config in directory ORIG
-rw-r--r--  1 j65nko  j65nko  2524 Jan 24 05:16 ORIG/sshd_config
Copy original sshd_config to directory NEW for editing
-rw-r--r--  1 j65nko  j65nko  2524 Jan 24 05:16 NEW/sshd_config

Press Enter to edit sshd_config 

[snip vi session]

Copying original ORIG/sshd_config back to current dir to test <_patcher>
-rw-r--r--  1 j65nko  j65nko   630 Jan 24 05:20 _patcher
-rw-r--r--  1 j65nko  j65nko  2524 Jan 24 05:16 sshd_config

The generated patch script:

$ cat -n _patcher
     1  # ----------------------------------------------------------
     2  echo
     3  echo --- patch script for: sshd_config --- BEGIN
     4  
     5  # ---  edit the following line if needed
     6  FILE=./sshd_config
     7  
     8  patch -b -p0 ${FILE} <<END_OF_PATCH
     9  --- ORIG/sshd_config    Sun Jan 24 05:16:30 2010
    10  +++ NEW/sshd_config     Sun Jan 24 05:20:26 2010
    11  @@ -9,9 +9,11 @@
    12   # default value.
    13   
    14   #Port 22
    15  -#AddressFamily any
    16  -#ListenAddress 0.0.0.0
    17  +AddressFamily inet 
    18  +ListenAddress 192.168.222.244 
    19   #ListenAddress ::
    20  +
    21  +AllowUsers j65nko robert
    22   
    23   # The default requires explicit activation of protocol 1
    24   #Protocol 2
    25  END_OF_PATCH
    26  
    27  echo  --- patch script for: sshd_config --- END

Executing the script:

$ sh _patcher
--- patch script for: sshd_config --- BEGIN
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|--- ORIG/sshd_config   Sun Jan 24 05:16:30 2010
|+++ NEW/sshd_config    Sun Jan 24 05:20:26 2010
--------------------------
Patching file ./sshd_config using Plan A...
Hunk #1 succeeded at 9.
done
--- patch script for: sshd_config --- END

$ cat -n sshd_config
[snip]
     9  # default value
    10  
    11  #Port 22
    12  AddressFamily inet 
    13  ListenAddress 192.168.222.244 
    14  #ListenAddress ::
    15  
    16  AllowUsers j65nko robert
[snip]  
    17

[J65nko]

1.5 The 'patchcreate' script

     1  #!/bin/sh
     2  # $Id: Patchcreate.xml,v 1.6 2010/01/24 14:11:53 j65nko Exp $
     3  
     4  EDIT=/usr/bin/vi
     5  PATCHSCRIPT=_patcher
     6  
     7  mkdir -p ORIG 
     8  mkdir -p NEW
     9  
    10  # -- file name specified?
    11  if [ $# -ne 1 ] ; then
    12     echo $0 ERROR: No file name specified!
    13     exit 1
    14  fi 
    15  
    16  # -- specified file exists?
    17  if [ -f $1 -o -f ORIG/$1 ] ; then 
    18      echo Ok, found file $1
    19  else
    20      echo $0 ERROR file "$1" does not exist!
    21      exit 1
    22  fi    
    23  
    24  if [ -f ORIG/$1 ]; then
    25      echo Good, found original $1 in directory ORIG
    26  else
    27       echo Saving copy of $1 in directory ORIG 
    28       cp -p $1 ORIG/$1
    29  fi
    30  
    31  ls -l ORIG/$1
    32  
    33  if [ -f NEW/$1 ]; then
    34      echo Good, found  $1 in directory NEW
    35      ls -l NEW/$1
    36  else
    37      echo Copy original $1 to directory NEW for editing 
    38      cp -p ORIG/$1 NEW/$1
    39      ls -l NEW/$1
    40  fi
    41  
    42  # exit
    43  
    44  printf "\nPress Enter to edit $1" ; read X
    45  
    46  $EDIT NEW/$1
    47  
    48  # ------- create patch script with the patch in-line
    49  
    50  cat <<END > ${PATCHSCRIPT}
    51  # ----------------------------------------------------------
    52  echo
    53  echo --- patch script for: $1 --- BEGIN
    54  
    55  # ---  edit the following line if needed
    56  FILE=./$1
    57  
    58  patch -b -p0 \${FILE} <<END_OF_PATCH
    59  $(diff -u ORIG/${1} NEW/${1})
    60  $(echo 'END_OF_PATCH')
    61  
    62  echo  --- patch script for: $1 --- END
    63  END
    64  
    65  echo "Copying original ORIG/$1 back to current dir to test <${PATCHSCRIPT}>" 
    66  cp -p ORIG/$1 .
    67  ls -l $1 ${PATCHSCRIPT}
    68

Keep the unmodified file in 'ORIG', the file to be changed, or already modified in 'NEW'. At the end (lines 65-67), copy the original to the current directory for testing.

The actual work, creating a here document to be used by 'patch', is done in 58-60. The complete script generation, starts on line 50 and ends on line 63 and is done with another here document.

As dessert, the next section shows the effect of this 'sshd_config' patch.


1.6 Comparison of 'sshd' before and after patching

Before the patch:

$ netstat -a -f inet
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp          0     48  vintrax.ssh            hercules.47094         ESTABLISHED
tcp          0      0  localhost.submissi     *.*                    LISTEN
tcp          0      0  localhost.smtp         *.*                    LISTEN
tcp          0      0  *.ssh                  *.*                    LISTEN
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
udp          0      0  vintrax.4599           virtueledoos.nl.ntp   
udp          0      0  vintrax.13835          ntp2.hro.nl.ntp       
udp          0      0  vintrax.43599          ntp.mediamatic.n.ntp  
udp          0      0  *.syslog               *.*                   

$ netstat -a -f inet6
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp6         0      0  localhost.submissi     *.*                    LISTEN
tcp6         0      0  localhost.smtp         *.*                    LISTEN
tcp6         0      0  *.ssh                  *.*                    LISTEN

After the patch and stopping and restarting 'sshd':

$ netstat -af inet
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp          0     48  vintrax.ssh            hercules.33562         ESTABLISHED
tcp          0      0  vintrax.ssh            *.*                    LISTEN
tcp          0      0  localhost.submissi     *.*                    LISTEN
tcp          0      0  localhost.smtp         *.*                    LISTEN

$ netstat -af inet6
Active Internet connections (including servers)
Proto   Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp6         0      0  localhost.submissi     *.*                    LISTEN
tcp6         0      0  localhost.smtp         *.*                    LISTEN

Instead of LISTENing on all IP4 addresses (*.ssh), 'sshd' will now restrict itself to the 192.168.222.244 'vintrax' IP address. It also stopped LISTENing on all IPv6 addresses (*.ssh).

$Id: Patchcreate.xml,v 1.6 2010/01/24 14:11:53 j65nko Exp $
$Id: vbul-html.xsl,v 1.15 2010/01/16 00:58:03 j65nko Exp $

[J65nko]

Download of the patchcreate script

[J65nko]

In the following Makefile _aliases is a patch script for the /etc/mail/aliases file created and discussed in the previous post.

The file _newaliases:

# -----------------
cat <<END
Running NEWALIASES to rebuild the aliases database ....
$(newaliases)

END
# --

The _existing-root-mail installs or moves the existing /var/mail/root to /var/mail/${USER}. If done as first thing after an install, this will usually do no harm, because the user's /var/mail file will not exist yet or will be empty.

If you don't want this , read the install(1) man page for the correct option to create a backup of an existing /var/mail/${USER} first.

# -------------------------------------------
USER=j65nko
echo --- move existing root email to $USER
install -o ${USER} -g ${USER} -m u=rw /var/mail/root /var/mail/${USER}

The Makefile itself:

# Makefile to divert root mail to user account

FILE    = _rootmail-to-j65nko

${FILE}:    _aliases _newaliases _existing-root-mail
        cat ${.ALLSRC} >${.TARGET}

clean:
        rm -f ${FILE}

You run the Makefile with the make command:

$ make
cat _aliases _newaliases _existing-root-mail >_rootmail-to-j65nko

All components are concatenated to a single file. The resulting file name can be changed easily by editing the FILE variable.
After editing one of the components run make again to create an updated version.

A commonly encountered issue with Makefiles:

"Makefile", line 6: Need an operator
Fatal errors encountered -- cannot continue

You probably got this error by copying and pasting the Makefile instead of downloading and unpacking the attached tgz file.

The Makefile requires a tab before the commands, which (re)create the target. Here the cat command in line 6. By pasting, you lost this tab and it was replaced by a couple of spaces.

You can diagnose this easily with:

 cat -ten Makefile                                 
     1  # Makefile to divert root mail to user account$
     2  $
     3  FILE     = _rootmail-to-j65nko$
     4  $
     5  ${FILE}:    _aliases _newaliases _existing-root-mail$
     6          cat ${.ALLSRC} >${.TARGET}$

The "operator" line 6, has no tab, a ^I, but a series of spaces.

The correct version

$ cat -ten Makefile

     1  # Makefile to divert root mail to user account$
     2  $
     3  FILE^I= _rootmail-to-j65nko$
     4  $
     5  ${FILE}:    _aliases _newaliases _existing-root-mail$
     6  ^Icat ${.ALLSRC} >${.TARGET}$

Here line 6 has a tab character

[persistent_ignoramus]

This thread should be made sticky. I've spent quite a while searching for it.