How to substitute shell variables in complex text files


I have several text files in which I have introduced shell variables ($VAR1 or $VAR2 for instance).

I would like to take those files (one by one) and save them in new files where all variables would have been replaced.

To do this, I used the following shell script (found on StackOverflow):

while read line
    eval echo "$line" >> destination.txt
done < "source.txt"

This works very well on very basic files.

But on more complex files, the "eval" command does too much:

  • Lines starting with "#" are skipped

  • XML files parsing results in tons of errors

Is there a better way to do it? (in shell script... I know this is easily done with Ant for instance)

Kind regards

This question is tagged with linux shell string-interpolation

~ Asked on 2013-01-04 10:43:48

The Best Answer is


Looking, it turns out on my system there is an envsubst command which is part of the gettext-base package.

So, this makes it easy:

envsubst < "source.txt" > "destination.txt"

Note if you want to use the same file for both, you'll have to use something like moreutil's sponge, as suggested by Johnny Utahh: envsubst < "source.txt" | sponge "source.txt". (Because the shell redirect will otherwise empty the file before its read.)

~ Answered on 2013-01-04 12:54:10


In reference to answer 2, when discussing envsubst, you asked:

How can I make it work with the variables that are declared in my .sh script?

The answer is you simply need to export your variables before calling envsubst.

You can also limit the variable strings you want to replace in the input using the envsubst SHELL_FORMAT argument (avoiding the unintended replacement of a string in the input with a common shell variable value - e.g. $HOME).

For instance:

export VAR1='somevalue' VAR2='someothervalue'

envsubst "$MYVARS" <source.txt >destination.txt

Will replace all instances of $VAR1 and $VAR2 (and only VAR1 and VAR2) in source.txt with 'somevalue' and 'someothervalue' respectively.

~ Answered on 2014-01-21 17:31:18

Most Viewed Questions: