DaemonForums  

Go Back   DaemonForums > Miscellaneous > Programming

Programming C, bash, Python, Perl, PHP, Java, you name it.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 24th August 2010
guitarscn guitarscn is offline
Package Pilot
 
Join Date: Oct 2008
Posts: 166
Default Writing a simple script to edit text files and execute commands

Okay this will probably have multiple parts to it but I don't really want to trouble you guys with more help because I'm a total noob so I can just do the first part by hand (it's just editing a few hundred lines of text in a file; I have to do the same thing on each line and I'm sure there's a scripted solution for it but I can spare a couple hours).

So basically I have a file with several hundred lines, and each line is only 2-3 words.

I need the script to do these things in order:

1) kill process1
2) put the 1st line of this file list into already existing text file #1 (requires root to write): the line it needs to write to is like: "option LINE1" where the 1st line has to be written where 'LINE1' is, somewhere in this text file
3) save this text file
4) put the 1st line of this file list into existing text file #2: the line is like "set = { opt1 = LINE1; opt2 = LINE1; opt3 = LINE1; }" and again I want the 1st line to be filled in all where 'LINE1' says
5) save text file
6) start process1
7) run a certain command in shell which will execute a CLI program and leave it running (I don't think it can be run in the background because I tried starting it with the '&' sign at the end and it ran but it didn't connect the way it properly was supposed to)
8) repeat but use the next line in the file list

What programming language would be the quickest and easiest way to write something like this? It doesn't have to be fancy or anything, just work even if it's dirty
Reply With Quote
  #2   (View Single Post)  
Old 24th August 2010
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by guitarscn View Post
I need the script to do these things in order...
I would highly recommend you implement this as a shell script, & I would further suggest it be done as a Bourne shell script. Most shells (Korn, Z, Bash, etc.) at a bare minimum support Bourne shell conventions, so in many ways it is the least common denominator solution. Bourne shell scripts will minimal changes should run on most shells.
  • A reasonable introduction to Bourne shell programming is the following:

    http://steve-parker.org/sh/sh.shtml

    Study it (or any other tutorials you might find) carefully.
  • Think of what standard commands can be used to accomplish each portion of the problem described.
  • Write it down & test. Confirm to yourself that the code does what you think it does.
  • When you get stuck, ask questions here, but help us understand what you have already done by describing the problem, describe what you think the solution might be, & describe what code you believe implements a working solution.

Last edited by ocicat; 24th August 2010 at 04:39 AM. Reason: updated tutorial link
Reply With Quote
  #3   (View Single Post)  
Old 24th August 2010
guitarscn guitarscn is offline
Package Pilot
 
Join Date: Oct 2008
Posts: 166
Default

That page says the guide is obsolete and is only preserved for historical purposes. Should I use the recommended guide linked on that page?

I have a general idea of how to stop/run processes because I do that every day, so I guess the main chunk of this is editing the text files and putting in the words where they belong. Would sed/awk work? Or will this not achieve the desired results because they are only stream editors and do one pass and are not state machines? (I remember the last time I worked with sed/awk to make mass changes to a text file, I was very unhappy with using such complicated syntaxes and stuff)

Last edited by guitarscn; 24th August 2010 at 04:34 AM.
Reply With Quote
  #4   (View Single Post)  
Old 24th August 2010
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by guitarscn View Post
That page says the guide is obsolete and is only preserved for historical purposes.
Updated. Thanks for pointing this out.
Quote:
Would sed/awk work?
Yes, but the overall solution may or may not be implemented solely in terms of one utility or the other. This is where you might need to write one portion in sed(1), another in awk(1), & glue everything together with sh(1).
Reply With Quote
  #5   (View Single Post)  
Old 24th August 2010
guitarscn guitarscn is offline
Package Pilot
 
Join Date: Oct 2008
Posts: 166
Default

How are your personal experiences with sed/awk? Generally between my friends and I, we all hate it for whatever reasons. I'm just not comfortable with it. I wonder what makes it so annoying to me.
Reply With Quote
  #6   (View Single Post)  
Old 24th August 2010
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by guitarscn View Post
How are your personal experiences with sed/awk?
When it comes to shell programming, sed(1) & awk(1) are indispensible.

For example, here's code I use within a shell script configuring a USB drive as a install medium (so I don't have to continually burn CD's...). Note that I have numerous subshells exploiting sed(1) & awk(1):
Code:
# ensure checksums match                                                                                      
verify_checksums()
{
    typeset FILE LOCAL SERVER ERROR=0

    for FILE in * ; do
        if [ $FILE != SHA256 ] ; then
            echo $FILE | awk '{ printf "%-20s", $1 }'
            SERVER=$(egrep "^SHA256 \($FILE\)" SHA256 | awk '{ print $4 }')
            LOCAL=$(cksum -a sha256 $FILE | sed 's!^.* = \(.*\)$!\1!')
            if [ $LOCAL = $SERVER ] ; then
                echo 'OK'
            else
                echo 'not OK'
                ERROR=1
            fi
        fi
    done
        
    [ $ERROR -ne 0 ] && panic 'SHA256 mismatch encountered'
}
This is fairly typical of the code I write. Perhaps it might be possible to write it with other utilities, but sed(1) & awk(1) are both powerful & malleable, so I continue to use them.
Reply With Quote
  #7   (View Single Post)  
Old 24th August 2010
Carpetsmoker's Avatar
Carpetsmoker Carpetsmoker is offline
Real Name: Martin
Tcpdump Spy
 
Join Date: Apr 2008
Location: Netherlands
Posts: 2,243
Default

There's also cut(1), which is something of a poor man's awk. I've actually never used awk in any shell script.

For example this line:
Code:
SERVER=$(egrep "^SHA256 \($FILE\)" SHA256 | awk '{ print $4 }')
Might be "rewritten" as:
Code:
SERVER=$(egrep "^SHA256 \($FILE\)" SHA256 | cut -f 4 -d ' ')
I find the later to be much more sensible, but that is my personal preference.

It should be pointed out that awk is much more powerful and can do much more than just simple selecting of columns.
__________________
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
Reply With Quote
  #8   (View Single Post)  
Old 24th August 2010
s0xxx's Avatar
s0xxx s0xxx is offline
Package Pilot
 
Join Date: May 2008
Posts: 192
Default

Apology for slight OT, maybe (untested):
Code:
SERVER=$(egrep "^SHA256 \($FILE\)" SHA256 | awk '{ print $4 }')
Code:
SERVER=$(awk -v f="($FILE)" '$2==f{print $4}' SHA256)
__________________
The best way to learn UNIX is to play with it, and the harder you play, the more you learn.
If you play hard enough, you'll break something for sure, and having to fix a badly broken system is arguably the fastest way of all to learn. -Michael Lucas, AbsoluteBSD
Reply With Quote
  #9   (View Single Post)  
Old 24th August 2010
rocket357's Avatar
rocket357 rocket357 is offline
Real Name: Jonathon
Wannabe OpenBSD porter
 
Join Date: Jun 2010
Location: 127.0.0.1
Posts: 429
Default

Quote:
Originally Posted by Carpetsmoker View Post
I've actually never used awk in any shell script.
Very similar here...I've only used awk *very* sparingly because I never really took the time to learn it.

sed is a different story. I've studied it and learned quite a bit of what it can do, and I find it absolutely indispensable. I have used sed to save jobs before, and that's no joke (corrupted plain-text database dump that was "cleaned up" with a sed one-liner so it could be properly restored).
Reply With Quote
Old 24th August 2010
guitarscn guitarscn is offline
Package Pilot
 
Join Date: Oct 2008
Posts: 166
Default

Okay, I decided to make it a bit simpler, I have this so far:

Code:
#!/bin/bash
killall process1
#editing configs
process1 &
sleep 5
process2
The #editing configs part is the only part left. So this is what I need to get this script to do now:

The large file is in this format:
Code:
apple
banana
candy
...
[several hundred lines of this]
yoyo
zebra
1st config file:
Code:
option WORD1
otheroption blaha
moreoption blahb
2nd config:
Code:
name = { john }
colors = { red,blue }
options = { opt1 = WORD1; opt2 = WORD1; opt3 = WORD1; }
I need the instances where it says 'WORD1' in both config files to be edited and replaced with the line I choose from the large file list. So say I want the word 'candy' to be placed where 'WORD1' is, I would run the script with the number '3' because 'candy' is the 3rd word in the list. But if I run the script again though (in a new terminal window), I have to make sure that 'candy' is replaced by the next word/line I choose because I can't have it hanging around from my previous use of the script.
Reply With Quote
Old 24th August 2010
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 3,318
Default

Quote:
Originally Posted by guitarscn View Post
But if I run the script again though (in a new terminal window), I have to make sure that 'candy' is replaced by the next word/line I choose because I can't have it hanging around from my previous use of the script.
It sounds like you are concerned about losing data integrity if there are multiple instances of the script running simultaneously. If this is the case, look at one of Heiner's tricks posted on Shelldorado:

http://www.shelldorado.com/shelltips...ml#oneinstance
Reply With Quote
Old 24th August 2010
s0xxx's Avatar
s0xxx s0xxx is offline
Package Pilot
 
Join Date: May 2008
Posts: 192
Default

Quote:
The #editing configs part is the only part left. So this is what I need to get this script to do now:

... snipped ...

I need the instances where it says 'WORD1' in both config files to be edited and replaced with the line I choose from the large file list. So say I want the word 'candy' to be placed where 'WORD1' is, I would run the script with the number '3' because 'candy' is the 3rd word in the list. But if I run the script again though (in a new terminal window), I have to make sure that 'candy' is replaced by the next word/line I choose because I can't have it hanging around from my previous use of the script.
If I understood correctly, this is what you're trying to achieve:
Code:
$ cat 1stconfig
option WORD1
otheroption blaha
moreoption blahb

$ cat 2ndconfig
name = { john }
colors = { red,blue }
options = { opt1 = WORD1; opt2 = WORD1; opt3 = WORD1; }

$ cat large_file
apple
banana
candy
randy
mandy
yoyo
zebra

$ awk -f rep.awk 3
1stconfig: option candy
1stconfig: otheroption blaha
1stconfig: moreoption blahb
2ndconfig: name = { john }
2ndconfig: colors = { red,blue }
2ndconfig: options = { opt1 = candy; opt2 = candy; opt3 = candy; }

$ awk -f rep.awk 4
1stconfig: option randy
1stconfig: otheroption blaha
1stconfig: moreoption blahb
2ndconfig: name = { john }
2ndconfig: colors = { red,blue }
2ndconfig: options = { opt1 = randy; opt2 = randy; opt3 = randy; }
And here is the script to do it. Note it lacks error checks, and it doesn't edit the files, but rather print the output to stdout. Enough to get you started.
Code:
$ cat rep.awk
BEGIN{
        argument=ARGV[1]
        ARGC--
        ARGV[ARGC++]="large_file"
        ARGV[ARGC++]="1stconfig"
        ARGV[ARGC++]="2ndconfig"
}
FILENAME=="large_file"{
        if(NR==argument)
                word=$1
}
FILENAME=="1stconfig"{
        sub(/WORD1/,word,$2)
        print FILENAME": "$0
}
FILENAME=="2ndconfig"{
        gsub(/WORD1/,word,$0)
        print FILENAME": "$0
}
Script also doesn't delete the word from large_list (as you requested), but it should be fairly ease to do it.
__________________
The best way to learn UNIX is to play with it, and the harder you play, the more you learn.
If you play hard enough, you'll break something for sure, and having to fix a badly broken system is arguably the fastest way of all to learn. -Michael Lucas, AbsoluteBSD
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Request for sh script for compressing files bgobs Programming 5 16th January 2010 07:20 PM
execute commands from xinitrc rex FreeBSD General 3 22nd October 2008 10:24 PM
ruby execute commands remotely Dr_Death_UAE Programming 0 18th August 2008 11:23 AM
Cron won't execute a perl script ivanatora FreeBSD General 4 17th August 2008 07:53 AM
Create a script to rmove oldest files disco Programming 5 14th July 2008 09:25 PM


All times are GMT. The time now is 06:00 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick