Regular expressions: renaming files with 'sed'

J65nko · #1 **(View Single Post)** 12th December 2008

I hade a bunch of directories with files like this:

Code:

$ ls NOW/latest-pkg*
NOW/latest-pkg            NOW/latest-pkg-erlangen   NOW/latest-pkg-plig
NOW/latest-pkg-calyx      NOW/latest-pkg-esat       NOW/latest-pkg-stacken

The '-' between 'latest' and 'pkg' had to be replaced with an underscore '_'.
The plan was to save the file names in a temporay file and use sed to create sh 'mv' commands to rename the files.

First save the file names in a file called 'tmp'.

Code:

$ ls NOW/latest-pkg* >tmp
$ cat tmp

NOW/latest-pkg
NOW/latest-pkg-calyx
NOW/latest-pkg-erlangen
NOW/latest-pkg-esat
NOW/latest-pkg-plig
NOW/latest-pkg-stacken

Now enter a sed(1) one-liner to munge each file name into a 'mv' command like this:

Code:

mv NOW/latest-pkg NOW/latest_pkg

Plan of attack:

Save the text up to the first '-' and save the trailing text.
prepend a 'mv ' followed by a blank
reconstruct the original file name
insert a blank
retrieve the text leading to the first '-'
insert the desired underscore character '_'
retrieve the text after the original '-'

Code:

$ sed -e 's/^\([^-]*\)-\(pkg.*\)/mv \1-\2 \1_\2/' tmp
mv NOW/latest-pkg NOW/latest_pkg
mv NOW/latest-pkg-calyx NOW/latest_pkg-calyx
mv NOW/latest-pkg-erlangen NOW/latest_pkg-erlangen
mv NOW/latest-pkg-esat NOW/latest_pkg-esat
mv NOW/latest-pkg-plig NOW/latest_pkg-plig
mv NOW/latest-pkg-stacken NOW/latest_pkg-stacken

That looks good. (I have to admit this was my second try. I had to break out of the first attempt with CNTRL-C. )

We now have two options to execute this:

Save the sed output to file and feed it to the shell for execution:

Code:

$ sed -e 's/^\([^-]*\)-\(pkg.*\)/mv \1-\2 \1_\2/' tmp >tmp.sh
$ sh tmp.sh

Feed the sed output to the shell directly through a pipe line:
Code:
```
$ sed -e 's/^$[^-]*$-$pkg.*$/mv \1-\2 \1_\2/' tmp | sh
```

I went for the last option. The result:

Code:

 $ ls -l NOW/latest_*
-rw-r--r--  1 j65nko  j65nko  253269 Dec 12 02:41 NOW/latest_pkg
-rw-r--r--  1 j65nko  j65nko  293597 Dec 12 02:41 NOW/latest_pkg-calyx
-rw-r--r--  1 j65nko  j65nko  253269 Dec 12 02:41 NOW/latest_pkg-erlangen
-rw-r--r--  1 j65nko  j65nko  253269 Dec 12 02:42 NOW/latest_pkg-esat
-rw-r--r--  1 j65nko  j65nko  252528 Dec 12 02:41 NOW/latest_pkg-plig
-rw-r--r--  1 j65nko  j65nko  253269 Dec 12 02:41 NOW/latest_pkg-stacken

An explanation of the sed search and replace command:

Code:

s/^\([^-]*\)-\(pkg.*\)/mv \1-\2 \1_\2/

The search pattern:

Code:

s/^\([^-]*\)-\(pkg.*\)/

s		: search
/		: delimiter to mark start of search pattern
^		: beginning of line

\(		: start saving text for replay in containter \1
[^-]		: whatever character as long it is not a '-'
*		: zero or more of the preceding
\)		: stop saving in container \1 
		  we now have saved or stored the text 'NOW/latest'

-		: a '-', which we didn't save because we need to replace
		  it with a underscore '_'

\(		: start saving text for replay in container \1
pkg		: a 'p', followed by a 'k', followed by a 'g'
.		: followed by whatever character
*		: zero or more of instances of the preceding atom '.'
\)		: stop saving in container \2

The replacement:

Code:

/ mv \1-\2 \1_\2/

/		: end of seach pattern, start of replacement
mv 		: a 'm' and a 'v' followed by a space

		: now we reconstruct our original file name:

\1		: replay or the text 'NOW/latest' from container \1
-		: a '-'
\2		: the text from container \2

		: a space to separate the original name from the new one

\1		: fetch text 'latest' from container \1
_		: the underscore we want
\2		: the remainder of the file name from container \2

sverreh · #2 **(View Single Post)** 12th December 2008

WOW, that was an impressing demo of mastering regexp

I think I would have used hours to do that!
As usual your post is very clear, easy to follow (even for amateurs like me) and straight to the point.

To rename a lot of files, I usually utilize vils.

Code:

sverre % ls
latest-pkg              latest-pkg-erlangen     latest-pkg-plig
latest-pkg-calyx        latest-pkg-esat         latest-pkg-stacken
sverre %  vils
      1 0001 latest-pkg
      2 0002 latest-pkg-calyx
      3 0003 latest-pkg-erlangen
      4 0004 latest-pkg-esat
      5 0005 latest-pkg-plig
      6 0006 latest-pkg-stacken
~

Then, in vils give the command

Code:

:%s/-/_/

which gives

Code:

       1 0001 latest_pkg
       2 0002 latest_pkg-calyx
       3 0003 latest_pkg-erlangen
       4 0004 latest_pkg-esat
       5 0005 latest_pkg-plig
       6 0006 latest_pkg-stacken
~

Then exit vils in the standard vi way:

Code:

:wq 
sverre % ls
latest_pkg              latest_pkg-erlangen     latest_pkg-plig
latest_pkg-calyx        latest_pkg-esat         latest_pkg-stacken
sverre %

This is probably faster if you have the files in only a few directories, but I in your case with many directories I would think your sed-script is more suitable. Not to mention how fun it is to experiment with regular expressions!

J65nko · #3 **(View Single Post)** 12th December 2008

Never heard of vils before. Using that utility simplifies things a lot because you don't have to generate the "mv <original_file>" part of the renaming commands.

How does vils handle spaces in file names?

sverreh · #4 **(View Single Post)** 12th December 2008

Quote:

Originally Posted by J65nko

How does vils handle spaces in file names?

Never tried that before, so let's test.

Code:

 % vils

  1 0001 last_pkg
  2 0002 last_pkg-calyx
  3 0003 latest_pkg-erlangen
  4 0004 latest_pkg-esat
  5 0005 latest_pkg-plig
  6 0006 latest_pkg-stacken

:%s/_/ /  1 0001 last pkg
  2 0002 last pkg-calyx
  3 0003 latest pkg-erlangen
  4 0004 latest pkg-esat
  5 0005 latest pkg-plig
  6 0006 latest pkg-stacken
:wq
sverre % ls
last pkg                latest pkg-erlangen     latest pkg-plig
last pkg-calyx          latest pkg-esat         latest pkg-stacken
sverre %

So creating file names with blanks seems to work. Now let's see what happens when we try to reinsert the underscore:

Code:

vils
  1 0001 last pkg
  2 0002 last pkg-calyx
  3 0003 latest pkg-erlangen
  4 0004 latest pkg-esat
  5 0005 latest pkg-plig
  6 0006 latest pkg-stacken
:%s/ /_/ 
  1 0001_last pkg
  2 0002_last pkg-calyx
  3 0003_latest pkg-erlangen
  4 0004_latest pkg-esat
  5 0005_latest pkg-plig
  6 0006_latest pkg-stacken

Ooops! Not exactly what we wanted

We undo the change by typing u (we are actually working in vi here) and continue:

Code:

 
  1 0001 last pkg
  2 0002 last pkg-calyx
  3 0003 latest pkg-erlangen
  4 0004 latest pkg-esat
  5 0005 latest pkg-plig
  6 0006 latest pkg-stacken
:%s/\([A-z]\) /\1_/
  1 0001 last_pkg
  2 0002 last_pkg-calyx
  3 0003 latest_pkg-erlangen
  4 0004 latest_pkg-esat
  5 0005 latest_pkg-plig
  6 0006 latest_pkg-stacken
:wq
sverre % ls
last_pkg                latest_pkg-erlangen     latest_pkg-plig
last_pkg-calyx          latest_pkg-esat         latest_pkg-stacken
sverre %
~

Yes, it can be done, but requires some use of "leaning toothpicks". However, I don't think that scares you!

Of course, since we are actually editing the file names in vi, we could also replace each blank with an underscore manually, or by using the command:

Code:

:%s/ /_/gc

but that is not so interesting.

(But I could have done it faster that way.) That's life

J65nko · #5 **(View Single Post)** 13th December 2008

You also can restore the blanks into underscores with:

Code:

:%s/ pkg/_pkg/

In vi as well in sed and perl, you can avoid the 'leaning toothpick syndrome' by choosing another delimiter e.g.

Code:

:%! pkg!_pkg!

Code:

s!^\([^-]*\)-\(pkg.*\)!mv \1-\2 \1_\2!

or

Code:

s#^\([^-]*\)-\(pkg.*\)#mv \1-\2 \1_\2#

sverreh · #6 **(View Single Post)** 13th December 2008

Quote:

Originally Posted by J65nko

You also can restore the blanks into underscores with:

Code:

:%s/ pkg/_pkg/

Right! But only when all the file names that you want to modify have "pkg" after the blank. I was trying to be a little more general here.

Quote:

In vi as well in sed and perl, you can avoid the 'leaning toothpick syndrome' by choosing another delimiter

Good idea! Looks better, and may even be easier to debug.

robbak · #7 **(View Single Post)** 13th December 2008

There is also some bash tricks that should be able to do that in one step. Let's see if I can work it out.

Code:

robbak@robbak-PC:~/testdir$ touch file-one
robbak@robbak-PC:~/testdir$ touch file-two
robbak@robbak-PC:~/testdir$ touch file-three
robbak@robbak-PC:~/testdir$ touch "with space-four"
robbak@robbak-PC:~/testdir$ for filename in *; do mv "$filename" "${filename/-/_}"; done
robbak@robbak-PC:~/testdir$ ls
file_one  file_three  file_two  with space_four
robbak@robbak-PC:~/testdir$

Of course this may be a bashisim.

sverreh · #8 **(View Single Post)** 13th December 2008

Nice little script, robbak!
What if I want to change all "-" to "_" in the filenames? Is there a command for that too in bash?

Quote:

Originally Posted by robbak

Of course this may be a bashisim.

I guess it is. I tried it in Bourne shell:

Code:

$ for filename in *
> do
> mv "$filename" "${filename/-/_}"
> done
${filename/...}: Bad substitution

robbak · #9 **(View Single Post)** 13th December 2008

Quote:

Originally Posted by man bash

${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in pathname
expansion. Parameter is expanded and the longest match of pat‐
tern against its value is replaced with string. If Ipattern
begins with /, all matches of pattern are replaced with string.
Normally only the first match is replaced. If pattern begins
with #, it must match at the beginning of the expanded value of
parameter. If pattern begins with %, it must match at the end
of the expanded value of parameter. If string is null, matches
of pattern are deleted and the / following pattern may be omit‐
ted. If parameter is @ or *, the substitution operation is
applied to each positional parameter in turn, and the expansion
is the resultant list. If parameter is an array variable sub‐
scripted with @ or *, the substitution operation is applied to
each member of the array in turn, and the expansion is the
resultant list.

So, to replace all, not just the first, the substitution becomes ${filename//-/_}

Code:

robbak@robbak-PC:~/testdir$ ls
file-one  file-three  file-two  with-space-five-mess  with space-four
robbak@robbak-PC:~/testdir$ for filename in *; do mv "$filename" "${filename//-/_}"; done
robbak@robbak-PC:~/testdir$ ls
file_one  file_three  file_two  with_space_five_mess  with space_four
robbak@robbak-PC:~/testdir$

sverreh · #10 **(View Single Post)** 13th December 2008

Great! So now we have three good ways of doing this renaming:

j65nko's sed script
vils from the ports
The bash name substitution method from robbak

Let's go out in the world and look for files to rename!

I will start renaming the files in /bin/ and /usr/bin.

Just kidding!

robbak · #11 **(View Single Post)** 13th December 2008

Don't stop at three! No one has provided the standard incomprehensible perl script yet!

J65nko · #12 **(View Single Post)** 14th December 2008

The perl version is less incomprehensible as the sed one.

Code:

$ perl -n -e 's/^([^-]*)-(pkg.*)/mv $1-$2 $1_$2/; print;' tmp 
mv NOW/latest-pkg NOW/latest_pkg
mv NOW/latest-pkg-calyx NOW/latest_pkg-calyx
mv NOW/latest-pkg-erlangen NOW/latest_pkg-erlangen
mv NOW/latest-pkg-esat NOW/latest_pkg-esat
mv NOW/latest-pkg-plig NOW/latest_pkg-plig
mv NOW/latest-pkg-stacken NOW/latest_pkg-stacken

It is nearly identical to the incomprehensible regex for sed.

The differences

Code:

sed     perl
\(      (
\)      )
\1      $1
\2      $2

Less danger for the leaning toothpick syndrome

sverreh · #13 **(View Single Post)** 14th December 2008

Quote:

Originally Posted by J65nko

The perl version is less incomprehensible as the sed one.

Less danger for the leaning toothpick syndrome

I was sure you couldn't resist the challenge from robbak!

Well done!

So when will we have versions in C, C++, C#, Python, HTML, PHP, Cobol, Fortran, Algol, Simula, Basic and Haskell? This project is taking off!

TerryP · #14 **(View Single Post)** 14th December 2008

You can't rename a file in [pure] HTML, and it lacks Regex.

sverreh · #15 **(View Single Post)** 14th December 2008

Too bad!

That was quite a setback.

J65nko · #16 **(View Single Post)** 14th December 2008

You can generate those 'mv' commands with XSLT though.

Source XML file "files,xml":

Code:

<?xml version="1.0" ?>
<?xml-stylesheet href="rename.xsl" type="text/xsl" ?>

<collection>
    <file>latest-pkg</file>
    <file>latest-pkg-calyx</file>
    <file>latest-pkg-erlangen</file>
    <file>latest-pkg-esat</file>
    <file>latest-pkg-plig</file>
    <file>latest-pkg-stacken</file>
</collection>

The XSLT file "rename.xsl":

Code:

<?xml version="1.0" encoding='UTF-8' ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"> 


<xsl:output
    method='html'
    omit-xml-declaration='no'
    indent='yes' 
    standalone='yes'
    doctype-public='-//W3C//DTD XHTML 1.0 Strict//EN'
    doctype-system='http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd'
/>


<xsl:template match="/collection">
  <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"  >
    <head>
      <title>
        Renaming a file with XSL
     </title>
    </head>
    <body>
        <xsl:call-template name='show_original' />
        <xsl:call-template name='show_rename_cmd' />
    </body>
  </html>
</xsl:template>


<xsl:template name="show_original">
  <h1>Original file names</h1>
    <ul>
      <xsl:for-each select='file' >
        <li><xsl:value-of select="." /> </li>
      </xsl:for-each>
    </ul>
</xsl:template>


<xsl:template name="show_rename_cmd">
  <h1>Rename command</h1>
      <ul>
  <xsl:for-each select='file' >
         <li>
            <xsl:value-of select='concat("mv ", current(), " ")' />
            <xsl:value-of select="substring-before( current(),'-')" />
            <xsl:text>_</xsl:text>
            <xsl:value-of select="substring-after( current(),'-')" />
         </li>
  </xsl:for-each>
      </ul>
</xsl:template>

</xsl:stylesheet>

Pointing a modern browser to "files.xml" will show the following:

Code:

Original file names

    * latest-pkg
    * latest-pkg-calyx
    * latest-pkg-erlangen
    * latest-pkg-esat
    * latest-pkg-plig
    * latest-pkg-stacken

Rename command

    * mv latest-pkg latest_pkg
    * mv latest-pkg-calyx latest_pkg-calyx
    * mv latest-pkg-erlangen latest_pkg-erlangen
    * mv latest-pkg-esat latest_pkg-esat
    * mv latest-pkg-plig latest_pkg-plig
    * mv latest-pkg-stacken latest_pkg-stacken

If you don't believe me, point your browser at files.xml
Feel free to view the source code. In firefox: View -> Page Source, or just press CNTRL+U

J65nko · #17 **(View Single Post)** 15th December 2008

Using OpenBSD make:

Code:

$ cat Makefile                                                    

FILES = latest-pkg latest-pkg-calyx latest-pkg-erlangen \
        latest-pkg-esat latest-pkg-plig latest-pkg-stacken

main:
.for THIS in ${FILES}
        @echo mv ${THIS} ${THIS:S/-pkg/_pkg/g}  
.endfor

$ make

mv latest-pkg latest_pkg
mv latest-pkg-calyx latest_pkg-calyx
mv latest-pkg-erlangen latest_pkg-erlangen
mv latest-pkg-esat latest_pkg-esat
mv latest-pkg-plig latest_pkg-plig
mv latest-pkg-stacken latest_pkg-stacken

The :S variable modifier is described in make(1).

Carpetsmoker · #18 **(View Single Post)** 15th December 2008

FreeBSD supports ${var:S/} too, and some other variants, all documented in make(1)

s0xxx · #19 **(View Single Post)** 3rd January 2009

Sorry to kick-in late, great guide j65nko! But was there a need for using regexp? Could have a simple sed 's/-/_/' file do the job? I see the pattern is repeating and you needed to change only the first occurrence of "-" in string (note there is no g at the end).

Also noone offered an awk solution to the problem:

Code:

awk '{ sub(/-/,"_"); print }' file

(also not using gsub for global change)

J65nko · #20 **(View Single Post)** 3rd January 2009

The input was a list of files, the output should be mv commands to rename the files. The sed solution is a little bit complicated, because there is no easy way to save the original file name.

Your one-liner does not create these 'mv' commands files, as was the requirement

This modification of your script does produce them:

Code:

{ 
  orig = $1
  sub(/-/,"_") 
  printf("mv %s %s\n", orig, $1) ;  
}

Saved as 'ren_awk' you use it this way:

Code:

$ awk -f ren_awk tmp 
mv NOW/latest-pkg NOW/latest_pkg
mv NOW/latest-pkg-calyx NOW/latest_pkg-calyx
mv NOW/latest-pkg-erlangen NOW/latest_pkg-erlangen
mv NOW/latest-pkg-esat NOW/latest_pkg-esat
mv NOW/latest-pkg-plig NOW/latest_pkg-plig
mv NOW/latest-pkg-stacken NOW/latest_pkg-stacken

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Cleaning Portsnap files in /var/db/portsnap/files	bram85	FreeBSD Ports and Packages	2	5th October 2009 09:54 AM
PHP regular expression help	cajunman4life	Programming	2	16th August 2008 05:17 PM
How to sync files over ftp	graudeejs	FreeBSD General	4	4th August 2008 10:18 PM
Mount filesystem with a regular user	ivanatora	FreeBSD General	15	30th July 2008 08:51 AM
are you an former bsdforum regular?	ephemera	Off-Topic	18	28th July 2008 12:57 PM