DaemonForums  

Go Back   DaemonForums > Miscellaneous > Programming

Programming C, bash, Python, Perl, PHP, Java, you name it.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 20th May 2013
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 566
Thanked 14 Times in 13 Posts
Default Unexpected / Inconsistent shell behaviour

A while ago I was working with a system shell script and encountered a really unexpected result when I tried to do something a certain way. Below I've distilled it into a simple example. Odd shell behaviour of some sort has been discussed on this forum before, but I couldn't find it, so I hope this isn't a duplicate discussion.

Consider the following example script for a Bourne-like shell:

Code:
#!/path/to/shell

VAR=Old

echo A | while read REPLY; do

    echo Here

    VAR=New

done

echo VAR = $VAR
Note that the first line of the script (in red) shouldn't be used literally. Replace it with paths to various Bourne-like shells to be tested.

At this point, to get the most out of this, you shouldn't read past the end of this paragraph. Just study the script above and decide what you think the output should be. Then come back and read the rest of the post. No cheating now, look at the script first.

Unexpected Behaviour

When I ran the script on "most" shells, the output was totally unexpected. It gave this:

Here
VAR = Old

Note that since it prints "Here", the script does go into the do loop where VAR gets set to New. Yet upon exiting the loop, VAR is back to the "Old" value! It's as if what happens in the loop is somehow quite local, and is lost after leaving the loop. I got this weird behaviour with the following shells:

bash
sh NetBSD
ksh NetBSD
ksh OpenBSD
sh SunOS

Expected Behaviour

The output I'd expected was this:

Here
VAR = New

After checking more shells, this output was produced by:

ksh93 CentOS
ksh SunOS

One conclusion is that Bourne-like shells are not consistent about this.

Question: Is the "locality" of the unexpected output intended and is it documented somewhere? I looked at some man pages a bit and didn't find anything, but maybe it's subtle or I plain missed it.

Thanks for reading, and for any comments.
Reply With Quote
  #2   (View Single Post)  
Old 20th May 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,154
Thanked 182 Times in 149 Posts
Default

The following is tested on OpenBSD

A new shell is spawned to handle the pipeline and while ... done loop:

Code:
#!/bin/sh

VAR=Old

echo A | while read REPLY; do
    echo Here
    VAR=New
    echo "Inside while ... done loop: VAR = $VAR"
    pgrep sh | wc -l
done

echo VAR = $VAR
pgrep sh | wc -l
Running this produces:
Code:
Here
Inside while ... done loop: VAR = New
      10
VAR = Old
       9
So there is one more shell running inside that loop. The variable is set to within that sub-shell and this sub-shell cannot modify the variable of the parent shell/process.

If you mess around with shell commands in Makefiles you encounter similar issues
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #3   (View Single Post)  
Old 20th May 2013
J65nko J65nko is offline
Administrator
 
Join Date: May 2008
Location: Budel - the Netherlands
Posts: 3,154
Thanked 182 Times in 149 Posts
Default

It is the pipeline that is the cause for spawning the sub-shell:

Code:
#!/bin/sh

VAR=Old

pgrep sh | wc -l

echo Entering the loop
# echo A | while read REPLY; do
while read REPLY; do
    echo Here
    VAR=New
    echo "Inside while ... done loop: VAR = $VAR"
    pgrep sh | wc -l
    if [ "X$REPLY" = X"Q" ] ; then 
       break
    fi
done

echo Back to main program: VAR = $VAR
pgrep sh | wc -l
Running it:
Code:
$  ./test2.sh 
       9
Entering the loop
a
Here
Inside while ... done loop: VAR = New
       9
a
Here
Inside while ... done loop: VAR = New
       9
Q
Here
Inside while ... done loop: VAR = New
       9
Back to main program: VAR = New
       9
So no change in the number of shells here and the variable now has been modified as expected.
__________________
You don't need to be a genius to debug a pf.conf firewall ruleset, you just need the guts to run tcpdump
Reply With Quote
  #4   (View Single Post)  
Old 20th May 2013
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 566
Thanked 14 Times in 13 Posts
Default

Thank you J65nko, that is great practical detective work!

Tomorrow I'll have to re-look at some man pages and see what they say about spawning a sub-shell in such circumstances (or don't say, as the case may be).
Reply With Quote
  #5   (View Single Post)  
Old 20th May 2013
IdOp's Avatar
IdOp IdOp is offline
Too dumb for a smartphone
 
Join Date: May 2008
Location: twisting on the daemon's fork(2)
Posts: 566
Thanked 14 Times in 13 Posts
Default

Here are a few selected quotes from man pages, which confirm what J65nko observed.

Quote:
Originally Posted by bash
Each command in a pipeline is executed as a separate process (i.e., in a subshell).
Quote:
Originally Posted by sh (NetBSD)
Note that unlike some other shells, each process in the pipeline is a child of the invoking shell (unless it is a shell built-in, in which case it executes in the current shell -- but any effect it has on the environment is wiped).
For ksh(1), I found nothing so explicit, but some implicit indications:

Quote:
Originally Posted by ksh
Job control refers to the shell's ability to monitor and control jobs, which are processes or groups of processes created for commands or pipelines.
... suggesting a pipeline is a group of processes (one for each command).

The above are examples of what I called "unexpected" behaviour. So far so good.

Now an oddity. The SunOS ksh has "expected" behaviour, but its man page says:

Quote:
Originally Posted by ksh SunOS
A pipeline is a sequence of one or more commands separated by |. The standard output of each command but the last is connected by a pipe(2) to the standard input of the next command. Each command is run as a separate process; ...
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Identify rules to behaviour Droid OpenBSD Security 5 25th March 2012 05:58 PM
DragonflyBSD: unexpected pkg_admin running. colonelmooch Other BSD and UNIX/UNIX-like 0 4th August 2011 04:26 PM
Am I blind? syntax error: `(' unexpected guitarscn Programming 1 10th November 2010 08:53 PM
strange behaviour after improper shutdown karri FreeBSD General 1 15th October 2008 03:08 PM
Funny network behaviour :) PatrickBaer General software and network 5 9th October 2008 09:47 AM


All times are GMT. The time now is 09:37 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick