DaemonForums  

Go Back   DaemonForums > Miscellaneous > Guides

Guides All Guides and HOWTO's.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 2nd February 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default Taking your sed(1) skills beyond the easily swallowed 's/x/y/' porridge

1. Taking your sed(1) skills beyond the easily swallowed 's/x/y/' porridge
  • 1.1 Introduction
  • 1.2 Progressively cooking a nice 'al dente' 'sed' command file
  • 1.3 Modifying the '/etc/ttys' file


1.1 Introduction

The DESCRIPTION from the sed(1) man page:

Code:
The sed utility reads the specified files, or the standard input if no
files are specified, modifying the input as specified by a list of com-
mands.  The input is then written to the standard output.
Please note that 'sed' does not overwrite or change files. Some 'sed' versions, e.g. the GNU sed, allow you to overwrite the original.

A simple example. If we have a file with the following contents:

Code:
Dear Jennifer,

I miss your beautiful brown eyes. 
....
We can adapt this letter for our new love Sally, who has deep blue eyes.

Code:
$ sed -e 's/Jennifer/Sally/' -e 's/brown/blue/' letter.jennifer
Dear Sally,
        
I miss your beautiful blue eyes.
....
As stated sed(1) has modified the input and written it to standard output. For mailing this letter we have to redirect the outcome to file with something like $ sed -e 's/Jennifer/Sally/' -e 's/brown/blue/' letter.jennifer >letter.sally
Reply With Quote
  #2   (View Single Post)  
Old 2nd February 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default

1.2 Progressively cooking a nice 'al dente' 'sed' command file

The example from the previous section is not too difficult, an easily digested porridge. As an exercise to move beyond this stage and to become a 'real (wo)man', we will modify an '/etc/tttys' so we can connect to a computer with a serial connection. No more watery gui/desktop soup with hieroglyphs for us!

According to the OpenBSD FAQ we have to change

Code:
tty00   "/usr/libexec/getty std.9600"   unknown off
into

Code:
tty00   "/usr/libexec/getty std.9600"   vt220   on secure
To make it more challenging, we will save the original line. By prefixing the original line with a "#" comment, we can easily backout a modification.
  1. Select the line with 'tty00' and just print it.

    Code:
    $ sed -ne '/^tty00/p' /etc/ttys
    tty00   "/usr/libexec/getty std.9600"   vt220   off secure
    The -n option

    Code:
    -n     By default, each line of input is echoed to the standard output
           after all of the commands have been applied to it.  The -n option
           suppresses this behavior.
    By using this option we prevent the complete file from being echoed, both modified and unmodified lines. To force 'sed' to show us the changed line, we have to specify the p modifier in the /^tty00/ pattern.

    The same results, but now using a command file called 'sercon':

    Code:
    $ sed -nf sercon /etc/ttys
    tty00   "/usr/libexec/getty std.9600"   vt220   off secure
    The 'sercon' file:

    Code:
    /^tty00/ {
    p
    }
    For all lines containing the 'tty00' sequence at the beginning of the line, perform all commands between the starting "{" and closing "}".
  2. Saving a copy with a '#" in front.

    Code:
    /^tty00/ {
    h
    s/^/# /
    p
    g
    p
    }
    Explanation:

    Code:
    h	: save the current line (pattern space) into hold space
    	  do not delete pattern space
    
    s/^/# /	: replace the null string at begin of line ('^') with a
    	  '#' followed by a space
    p	: print the modified line from pattern space
    
    g	: get the line from hold space into pattern
    	  space, and overwrite whatever is in pattern space
     
    p	: print the line from pattern space
    The result:

    Code:
    $ sed -nf sercon /etc/ttys
    # tty00 "/usr/libexec/getty std.9600"   vt220   off secure
    tty00   "/usr/libexec/getty std.9600"   vt220   off secure
  3. Replace 'vt220' with 'xterm' and 'off' with 'on'

    Code:
    /^tty00/ {
    h
    s/^/# /
    p
    g
    s/vt220/xterm/
    s/off/on /
    p
    }
    The result:

    Code:
    $ sed -nf sercon /etc/ttys
    # tty00 "/usr/libexec/getty std.9600"   vt220   off secure
    tty00   "/usr/libexec/getty std.9600"   xterm   on  secure
  4. The command file, to insert "# --- original ---" and "# --- modified ---".

    Code:
    /^tty00/ {
    h
    i\
    # --- original ---
    s/^/# /
    p
    i\
    # --- modified ---
    g
    s/vt220/xterm/
    s/off/on /
    p
    }
    Code:
    $ sed -nf sercon /etc/ttys
    # --- original ---
    # tty00 "/usr/libexec/getty std.9600"   vt220   off secure
    # --- modified ---
    tty00   "/usr/libexec/getty std.9600"   xterm   on  secure
  5. For the final version we omit the -n option, but we see the tty00 line appear twice.

    Code:
    $ sed -f sercon /etc/ttys | less
    ttyCa   "/usr/libexec/getty Pc"         vt220   off secure
    ttyCb   "/usr/libexec/getty Pc"         vt220   off secure
    # --- original ---
    # tty00 "/usr/libexec/getty std.9600"   vt220   off secure
    # --- modified ---
    tty00   "/usr/libexec/getty std.9600"   xterm   on  secure
    tty00   "/usr/libexec/getty std.9600"   xterm   on  secure
    tty01   "/usr/libexec/getty std.9600"   vt220   off secure
    tty02   "/usr/libexec/getty std.9600"   unknown off
    But that is easily fixed by commenting out the last p command.

    Code:
    /^tty00/ {
    h
    i\
    # --- original ---
    s/^/# /
    p
    i\
    # --- modified ---
    g
    s/vt220/xterm/
    s/off/on /
    #p
    }
    The 'sed' output is now:

    Code:
    ttyCb   "/usr/libexec/getty Pc"         vt220   off secure
    # --- original ---
    # tty00 "/usr/libexec/getty std.9600"   vt220   off secure
    # --- modified ---
    tty00   "/usr/libexec/getty std.9600"   xterm   on  secure
    tty01   "/usr/libexec/getty std.9600"   vt220   off secure
  6. A last simple modification to enable both tty00 and tty01

    Code:
    /^tty00/,/^tty01/ {
    h
    i\
    # --- original ---
    s/^/# /
    p
    i\
    # --- modified ---
    g
    s/vt220/xterm/
    s/off/on /
    #p
    }
    The /^tty00/,/^tty01/ is what sed(1) calls an [2addr] An alternative would be to use /^tty0[01]/.

    Code:
    ttyCb   "/usr/libexec/getty Pc"         vt220   off secure
    # --- original ---
    # tty00 "/usr/libexec/getty std.9600"   vt220   off secure
    # --- modified ---
    tty00   "/usr/libexec/getty std.9600"   xterm   on  secure
    # --- original ---
    # tty01 "/usr/libexec/getty std.9600"   vt220   off secure
    # --- modified ---
    tty01   "/usr/libexec/getty std.9600"   xterm   on  secure
    tty02   "/usr/libexec/getty std.9600"   unknown off
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump

Last edited by J65nko; 2nd February 2010 at 03:01 AM. Reason: Small typo
Reply With Quote
  #3   (View Single Post)  
Old 2nd February 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default

1.3 Modifying the '/etc/ttys' file

Before modifying this important configuration file, we will first perform a test.

Code:
$ mkdir temp ; cd temp
$ cp /etc/ttys .
$ ls -l
total 48
-rw-r--r--  1 j65nko  j65nko  22944 Feb  2 02:32 ttys
Having a copy in our 'temp' directory, we do the following:

Code:
$ sed -f ../sercon ttys >ttys
$ cat ttys
$ ls -l ttys
-rw-r--r--  1 j65nko  j65nko  0 Feb  2 02:38 ttys
Oops, that is clearly the wrong way. The file was truncated to 0.

Please do not blame 'sed' for this behaviour, because this is regular 'sh' behaviour, as explained by the OpenBSD sh(1) man page:

Code:
 ....the command cmd < foo > foo will
open foo for reading and then truncate it when it opens it for
writing, before cmd gets a chance to actually read foo.
The proper way is to make a backup of the original, use the backup file as input for 'sed', and redirect the 'sed' output to the original file.

Code:
$ cp /etc/ttys ttys.orig
$ sed -f ../sercon ttys.orig >ttys
An unified 'diff' shows the differences:

Code:
$ diff -u ttys.orig ttys

--- ttys.orig   Tue Feb  2 02:58:32 2010
+++ ttys        Tue Feb  2 02:58:52 2010
@@ -16,8 +16,14 @@
 ttyC9  "/usr/libexec/getty Pc"         vt220   off secure
 ttyCa  "/usr/libexec/getty Pc"         vt220   off secure
 ttyCb  "/usr/libexec/getty Pc"         vt220   off secure
-tty00  "/usr/libexec/getty std.9600"   vt220   off secure
-tty01  "/usr/libexec/getty std.9600"   vt220   off secure
+# --- original ---
+# tty00        "/usr/libexec/getty std.9600"   vt220   off secure
+# --- modified ---
+tty00  "/usr/libexec/getty std.9600"   xterm   on  secure
+# --- original ---
+# tty01        "/usr/libexec/getty std.9600"   vt220   off secure
+# --- modified ---
+tty01  "/usr/libexec/getty std.9600"   xterm   on  secure
 tty02  "/usr/libexec/getty std.9600"   unknown off
 tty03  "/usr/libexec/getty std.9600"   unknown off
 tty04  "/usr/libexec/getty std.9600"   unknown off
$Id: Advanced-sed.xml,v 1.4 2010/02/02 02:28:13 j65nko Exp $
$Id: vbul-html.xsl,v 1.15 2010/01/16 00:58:03 j65nko Exp $
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #4   (View Single Post)  
Old 2nd February 2010
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is online now
Real Name: Martin
Old man from scene 24
 
Join Date: Apr 2008
Location: Eindhoven, Netherlands
Posts: 2,069
Thanked 198 Times in 156 Posts
Default

A quick comment on "1.3 Modifying the '/etc/ttys' file", FreeBSD sed has the -i option, which makes editing files a bit easier:

Code:
     -i extension
	     Edit files in-place similarly to -I, but treat each file indepen-
	     dently from other files.  In particular, line numbers in each
	     file start at 1, the ``$'' address matches the last line of the
	     current file, and address ranges are limited to the current file.
	     (See Sed Addresses.)  The net result is as though each file were
	     edited by a separate sed instance.
And

Code:
     -I extension
	     Edit files in-place, saving backups with the specified extension.
	     If a zero-length extension is given, no backup will be saved.  It
	     is not recommended to give a zero-length extension when in-place
	     editing files, as you risk corruption or partial content in situ-
	     ations where disk space is exhausted, etc.

	     Note that in-place editing with -I still takes place in a single
	     continuous line address space covering all files, although each
	     file preserves its individuality instead of forming one output
	     stream.  The line counter is never reset between files, address
	     ranges can span file boundaries, and the ``$'' address matches
	     only the last line of the last file.  (See Sed Addresses.)  That
	     can lead to unexpected results in many cases of in-place editing,
	     where using -i is desired.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #5   (View Single Post)  
Old 2nd February 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default

An example using the '$' address, mentioned in the quote by Carpetsmoker from the FreeBSD sed(1) man page:
Code:
$ cat animal.sed

1 {
   i\
A list of animals in our zoo\
----------------------------
}


$ {
   a\
==== END of LIST ====
}
A sample file
Code:
$ cat animal.txt
giraffe
lion
elephant
The result:
Code:
$ sed -f animal.sed animal.txt

 A list of animals in our zoo
----------------------------
giraffe
lion
elephant
==== END of LIST ====
Now we will use a second file
Code:
$ cat animal2.txt
ostrich
hyena
gorilla
Telling sed(1) to process both files
Code:
$ sed -f animal.sed animal*.txt
A list of animals in our zoo
----------------------------
giraffe
lion
elephant
ostrich
hyena
gorilla
==== END of LIST ====
So this a simulation of the behaviour of the GNU/FreeBSD -I option by OpenBSD's sed.

The -i flag equivalent would be
Code:
$ sed -f animal.sed animal.txt ; sed -f animal.sed animal2.txt

A list of animals in our zoo
----------------------------
giraffe
lion
elephant
==== END of LIST ====

A list of animals in our zoo
----------------------------
ostrich
hyena
gorilla
==== END of LIST ====
Note: for readability some whitespace added.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #6   (View Single Post)  
Old 3rd February 2010
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,148
Thanked 182 Times in 149 Posts
Default sed(1) command file to collect title tags from an XML document

Because of the heavy commenting, no further explanation here.

Code:
$ cat titlecollect
                                                    
# -- sed command file to collect all title tags from an XML document
#
# example lines:
# <title>Automating patch generation and application for configuration files</title>
# <section><title>Introduction</title>
# <section><title>Automating patch creation</title
#

/<title>..*<\/title>/ {
# replace  <section> and </section> tags with 'nothing'
s/<section>// 
s/<\/section>//
# insert four blanks as indentation at begin of line 
s/^/    /
# append a "\n" to 'hold space', then append 'pattern space' 
# to 'hold space' 
# in other words,  collect all these line to 'hold space'
H
}

# stuff to do at last line
$ {
# insert XML header and 'toc' root element
i\
<?xml version="1.0" ?>\
<?xml-stylesheet href="XSL" type="text/xsl" ?>\
\
<toc>
# get all collected stuff from hold space to pattern space 
g
# append a closing 'toc" XML element and a comment
a\
</toc>\
\
<!-- end of table of contents ->
# print 'pattern space'
p
}
# -- end of sed command file
An example run:
Code:
$ sed -nf titlecollect Patchcreate.xml  

<?xml version="1.0" ?>
<?xml-stylesheet href="XSL" type="text/xsl" ?>

<toc>
    <title>Automating patch generation and application for configuration files</title>
    <title>Introduction</title>
    <title>Patching <file>/etc/mail/aliases</file>manually</title>
    <title>Automating patch creation</title> 
    <title>Using <file>patchcreate</file> to create a <file>sshd_config</file> patch</title>
    <title>The <file>patchcreate</file> script</title>
    <title>Comparison of  <prog>sshd</prog> before and after patching</title>
</toc>
The pure sed command file without comments
Code:
 $ grep -v '^#' titlecollect    

/<title>..*<\/title>/ {
s/<section>// 
s/<\/section>//
s/^/    /
H
}

$ {
i\
<?xml version="1.0" ?>\
<?xml-stylesheet href="XSL" type="text/xsl" ?>\
\
<toc>
g
a\
</toc>\
\
<!-- end of table of contents ->
p
}
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
Reply

Tags
sed, sed command file

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Xorg taking the fun out of BSD Carpetsmoker General software and network 122 6th December 2010 01:21 PM


All times are GMT. The time now is 03:10 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick