By Eric Lilja


2013-05-31 09:38:24 8 Comments

In the following program, if I set the variable $foo to the value 1 inside the first if statement, it works in the sense that its value is remembered after the if statement. However, when I set the same variable to the value 2 inside an if which is inside a while statement, it's forgotten after the while loop. It's behaving like I'm using some sort of copy of the variable $foo inside the while loop and I am modifying only that particular copy. Here's a complete test program:

#!/bin/bash

set -e
set -u 
foo=0
bar="hello"  
if [[ "$bar" == "hello" ]]
then
    foo=1
    echo "Setting \$foo to 1: $foo"
fi

echo "Variable \$foo after if statement: $foo"   
lines="first line\nsecond line\nthird line" 
echo -e $lines | while read line
do
    if [[ "$line" == "second line" ]]
    then
    foo=2
    echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done

echo "Variable \$foo after while loop: $foo"

# Output:
# $ ./testbash.sh
# Setting $foo to 1: 1
# Variable $foo after if statement: 1
# Value of $foo in while loop body: 1
# Variable $foo updated to 2 inside if inside while loop
# Value of $foo in while loop body: 2
# Value of $foo in while loop body: 2
# Variable $foo after while loop: 1

# bash --version
# GNU bash, version 4.1.10(4)-release (i686-pc-cygwin)

6 comments

@Kemin Zhou 2018-03-07 04:02:28

This is an interesting question and touches on a very basic concept in Bourne shell and subshell. Here I provide a solution that is different from the previous solutions by doing some kind of filtering. I will give an example that may be useful in real life. This is a fragment for checking that downloaded files conform to a known checksum. The checksum file look like the following (Showing just 3 lines):

49174 36326 dna_align_feature.txt.gz
54757     1 dna.txt.gz
55409  9971 exon_transcript.txt.gz

The shell script:

#!/bin/sh

.....

failcnt=0 # this variable is only valid in the parent shell
#variable xx captures all the outputs from the while loop
xx=$(cat ${checkfile} | while read -r line; do
    num1=$(echo $line | awk '{print $1}')
    num2=$(echo $line | awk '{print $2}')
    fname=$(echo $line | awk '{print $3}')
    if [ -f "$fname" ]; then
        res=$(sum $fname)
        filegood=$(sum $fname | awk -v na=$num1 -v nb=$num2 -v fn=$fname '{ if (na == $1 && nb == $2) { print "TRUE"; } else { print "FALSE"; }}')
        if [ "$filegood" = "FALSE" ]; then
            failcnt=$(expr $failcnt + 1) # only in subshell
            echo "$fname BAD $failcnt"
        fi
    fi
done | tail -1) # I am only interested in the final result
# you can capture a whole bunch of texts and do further filtering
failcnt=${xx#* BAD } # I am only interested in the number
# this variable is in the parent shell
echo failcnt $failcnt
if [ $failcnt -gt 0 ]; then
    echo $failcnt files failed
else
    echo download successful
fi

The parent and subshell communicate through the echo command. You can pick some easy to parse text for the parent shell. This method does not break your normal way of thinking, just that you have to do some post processing. You can use grep, sed, awk, and more for doing so.

@P.P. 2013-05-31 09:40:30

echo -e $lines | while read line 
    ...
done

The while loop is executed in a subshell. So any changes you do to the variable will not be available once the subshell exits.

Instead you can use a here string to re-write the while loop to be in the main shell process; only echo -e $lines will run in a subshell:

while read line
do
    if [[ "$line" == "second line" ]]
    then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done <<< "$(echo -e "$lines")"

You can get rid of the rather ugly echo in the here-string above by expanding the backslash sequences immediately when assigning lines. The $'...' form of quoting can be used there:

lines=$'first line\nsecond line\nthird line'
while read line; do
    ...
done <<< "$lines"

@beliy 2017-05-18 14:43:03

better change <<< "$(echo -e "$lines")" to simple <<< "$lines"

@mt eee 2018-11-09 07:57:31

what if the source was from tail -f instead of fixed text?

@P.P. 2018-11-10 16:42:31

@mteee You can use while read -r line; do echo "LINE: $line"; done < <(tail -f file) (obviously the loop won't terminate as it continues to wait for input from tail).

@Jens 2013-05-31 16:09:22

You are the 742342nd user to ask this bash FAQ. The answer also describes the general case of variables set in subshells created by pipes:

E4) If I pipe the output of a command into read variable, why doesn't the output show up in $variable when the read command finishes?

This has to do with the parent-child relationship between Unix processes. It affects all commands run in pipelines, not just simple calls to read. For example, piping a command's output into a while loop that repeatedly calls read will result in the same behavior.

Each element of a pipeline, even a builtin or shell function, runs in a separate process, a child of the shell running the pipeline. A subprocess cannot affect its parent's environment. When the read command sets the variable to the input, that variable is set only in the subshell, not the parent shell. When the subshell exits, the value of the variable is lost.

Many pipelines that end with read variable can be converted into command substitutions, which will capture the output of a specified command. The output can then be assigned to a variable:

grep ^gnu /usr/lib/news/active | wc -l | read ngroup

can be converted into

ngroup=$(grep ^gnu /usr/lib/news/active | wc -l)

This does not, unfortunately, work to split the text among multiple variables, as read does when given multiple variable arguments. If you need to do this, you can either use the command substitution above to read the output into a variable and chop up the variable using the bash pattern removal expansion operators or use some variant of the following approach.

Say /usr/local/bin/ipaddr is the following shell script:

#! /bin/sh
host `hostname` | awk '/address/ {print $NF}'

Instead of using

/usr/local/bin/ipaddr | read A B C D

to break the local machine's IP address into separate octets, use

OIFS="$IFS"
IFS=.
set -- $(/usr/local/bin/ipaddr)
IFS="$OIFS"
A="$1" B="$2" C="$3" D="$4"

Beware, however, that this will change the shell's positional parameters. If you need them, you should save them before doing this.

This is the general approach -- in most cases you will not need to set $IFS to a different value.

Some other user-supplied alternatives include:

read A B C D << HERE
    $(IFS=.; echo $(/usr/local/bin/ipaddr))
HERE

and, where process substitution is available,

read A B C D < <(IFS=.; echo $(/usr/local/bin/ipaddr))

@Evi1M4chine 2016-01-29 19:20:41

You forgot the Family Feud problem. Sometimes it’s very hard to find the same combination of words as whoever wrote the answer, so that you’re neither flooded in wrong results, nor filter out said answer.

@TrueY 2013-05-31 10:28:38

UPDATED#2

Explanation is in Blue Moons's answer.

Alternative solutions:

Eliminate echo

while read line; do
...
done <<EOT
first line
second line
third line
EOT

Add the echo inside the here-is-the-document

while read line; do
...
done <<EOT
$(echo -e $lines)
EOT

Run echo in background:

coproc echo -e $lines
while read -u ${COPROC[0]} line; do 
...
done

Redirect to a file handle explicitly (Mind the space in < <!):

exec 3< <(echo -e  $lines)
while read -u 3 line; do
...
done

Or just redirect to the stdin:

while read line; do
...
done < <(echo -e  $lines)

And one for chepner (eliminating echo):

arr=("first line" "second line" "third line");
for((i=0;i<${#arr[*]};++i)) { line=${arr[i]}; 
...
}

Variable $lines can be converted to an array without starting a new sub-shell. The characters \ and n has to be converted to some character (e.g. a real new line character) and use the IFS (Internal Field Separator) variable to split the string into array elements. This can be done like:

lines="first line\nsecond line\nthird line"
echo "$lines"
OIFS="$IFS"
IFS=$'\n' arr=(${lines//\\n/$'\n'}) # Conversion
IFS="$OIFS"
echo "${arr[@]}", Length: ${#arr[*]}
set|grep ^arr

Result is

first line\nsecond line\nthird line
first line second line third line, Length: 3
arr=([0]="first line" [1]="second line" [2]="third line")

@chepner 2013-05-31 12:44:27

+1 for the here-doc, since the lines variable's only purpose seems to be to feed the while loop.

@TrueY 2013-05-31 12:57:25

@chepner: Thx! I added another one, dedicated to You!

@dma_k 2016-01-19 20:20:27

There is yet another solution given here: for line in $(echo -e $lines); do ... done

@TrueY 2016-01-20 08:59:23

@dma_k Thanks for your comment! This solution would result 6 lines containing a single word. OP's request was different...

@Hamy 2016-11-26 05:03:26

upvoted. running echo in a subshell inside here-is, was one of the few solutions that worked in ash

@Jonathan Quist 2014-10-20 18:56:41

Hmmm... I would almost swear that this worked for the original Bourne shell, but don't have access to a running copy just now to check.

There is, however, a very trivial workaround to the problem.

Change the first line of the script from:

#!/bin/bash

to

#!/bin/ksh

Et voila! A read at the end of a pipeline works just fine, assuming you have the Korn shell installed.

@Marcin 2014-08-30 03:40:26

How about a very simple method

    +call your while loop in a function 
     - set your value inside (nonsense, but shows the example)
     - return your value inside 
    +capture your value outside
    +set outside
    +display outside


    #!/bin/bash
    # set -e
    # set -u
    # No idea why you need this, not using here

    foo=0
    bar="hello"

    if [[ "$bar" == "hello" ]]
    then
        foo=1
        echo "Setting  \$foo to $foo"
    fi

    echo "Variable \$foo after if statement: $foo"

    lines="first line\nsecond line\nthird line"

    function my_while_loop
    {

    echo -e $lines | while read line
    do
        if [[ "$line" == "second line" ]]
        then
        foo=2; return 2;
        echo "Variable \$foo updated to $foo inside if inside while loop"
        fi

        echo -e $lines | while read line
do
    if [[ "$line" == "second line" ]]
    then
    foo=2;          
    echo "Variable \$foo updated to $foo inside if inside while loop"
    return 2;
    fi

    # Code below won't be executed since we returned from function in 'if' statement
    # We aready reported the $foo var beint set to 2 anyway
    echo "Value of \$foo in while loop body: $foo"

done
}

    my_while_loop; foo="$?"

    echo "Variable \$foo after while loop: $foo"


    Output:
    Setting  $foo 1
    Variable $foo after if statement: 1
    Value of $foo in while loop body: 1
    Variable $foo after while loop: 2

    bash --version

    GNU bash, version 3.2.51(1)-release (x86_64-apple-darwin13)
    Copyright (C) 2007 Free Software Foundation, Inc.

@Mark Amery 2016-09-10 19:42:42

Maybe there's a decent answer under the surface here, but you've butchered the formatting to the point that it's unpleasant to try and read.

@Marcin 2017-08-11 14:20:52

You mean the original code is pleasant to read? (I've just followed :p)

Related Questions

Sponsored Content

19 Answered Questions

[SOLVED] How can I declare and use Boolean variables in a shell script?

24 Answered Questions

[SOLVED] What is the scope of variables in JavaScript?

19 Answered Questions

[SOLVED] Loop through an array of strings in Bash?

  • 2012-01-16 13:21:16
  • Mo.
  • 1093493 View
  • 1346 Score
  • 19 Answer
  • Tags:   arrays bash shell

30 Answered Questions

[SOLVED] How to concatenate string variables in Bash

18 Answered Questions

[SOLVED] Using global variables in a function

5 Answered Questions

32 Answered Questions

[SOLVED] How to check if a variable is set in Bash?

  • 2010-08-30 14:54:38
  • prosseek
  • 1164092 View
  • 1385 Score
  • 32 Answer
  • Tags:   bash shell variables

12 Answered Questions

[SOLVED] Syntax for a single-line Bash infinite while loop

  • 2009-08-17 16:31:47
  • Brian Deacon
  • 603043 View
  • 587 Score
  • 12 Answer
  • Tags:   bash loops while-loop

13 Answered Questions

[SOLVED] Emulate a do-while loop in Python?

5 Answered Questions

[SOLVED] Is there a reason for C#'s reuse of the variable in a foreach?

Sponsored Content