I'm writing a script to automate creating configuration files for Apache and PHP for my own webserver. I don't want to use any GUIs like CPanel or ISPConfig.
I have some templates of Apache and PHP configuration files. Bash script needs to read templates, make variable substitution and output parsed templates into some folder. What is the best way to do that? I can think of several ways. Which one is the best or may be there are some better ways to do that? I want to do that in pure Bash (it's easy in PHP for example)
1) How to replace ${} placeholders in a text file?
template.txt:
the number is ${i}
the word is ${word}
script.sh:
#!/bin/sh
#set variables
i=1
word="dog"
#read in template one line at the time, and replace variables
#(more natural (and efficient) way, thanks to Jonathan Leffler)
while read line
do
eval echo "$line"
done < "./template.txt"
BTW, how do I redirect output to external file here? Do I need to escape something if variables contain, say, quotes?
2) Using cat & sed for replacing each variable with its value:
Given template.txt:
The number is ${i}
The word is ${word}
Command:
cat template.txt | sed -e "s/\${i}/1/" | sed -e "s/\${word}/dog/"
Seems bad to me because of the need to escape many different symbols and with many variables the line will be tooooo long.
Can you think of some other elegant and safe solution?
This question is related to
bash
templates
templating
Here's a modified perl script based on a few of the other answers:
perl -pe 's/([^\\]|^)\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}/$1.$ENV{$2}/eg' -i template
Features (based on my needs, but should be easy to modify):
I agree with using sed: it is the best tool for search/replace. Here is my approach:
$ cat template.txt
the number is ${i}
the dog's name is ${name}
$ cat replace.sed
s/${i}/5/
s/${name}/Fido/
$ sed -f replace.sed template.txt > out.txt
$ cat out.txt
the number is 5
the dog's name is Fido
I needed to keep double quotes in my configuration file so double escaping double quotes with sed helps:
render_template() {
eval "echo \"$(sed 's/\"/\\\\"/g' $1)\""
}
I can't think of keeping trailing new lines, but empty lines in between are kept.
Although it is an old topic, IMO I found out more elegant solution here: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/
#!/bin/sh
# render a template configuration file
# expand variables + preserve formatting
render_template() {
eval "echo \"$(cat $1)\""
}
user="Gregory"
render_template /path/to/template.txt > path/to/configuration_file
All credits to Grégory Pakosz.
To follow up on plockc's answer on this page, here is a dash-suitable version, for those of you looking to avoid bashisms.
eval "cat <<EOF >outputfile
$( cat template.in )
EOF
" 2> /dev/null
Taking the answer from ZyX using pure bash but with new style regex matching and indirect parameter substitution it becomes:
#!/bin/bash
regex='\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}'
while read line; do
while [[ "$line" =~ $regex ]]; do
param="${BASH_REMATCH[1]}"
line=${line//${BASH_REMATCH[0]}/${!param}}
done
echo $line
done
Look at simple variables substitution python script here: https://github.com/jeckep/vsubst
It is very simple to use:
python subst.py --props secure.properties --src_path ./templates --dst_path ./dist
If using Perl is an option and you're content with basing expansions on environment variables only (as opposed to all shell variables), consider Stuart P. Bentley's robust answer.
This answer aims to provide a bash-only solution that - despite use of eval - should be safe to use.
The goals are:
${name} and $name variable references.$(...) and legacy syntax `...`) $((...)) and legacy syntax $[...]).\ (\${name})." and \ instances.Function expandVars():
expandVars() {
local txtToEval=$* txtToEvalEscaped
# If no arguments were passed, process stdin input.
(( $# == 0 )) && IFS= read -r -d '' txtToEval
# Disable command substitutions and arithmetic expansions to prevent execution
# of arbitrary commands.
# Note that selectively allowing $((...)) or $[...] to enable arithmetic
# expressions is NOT safe, because command substitutions could be embedded in them.
# If you fully trust or control the input, you can remove the `tr` calls below
IFS= read -r -d '' txtToEvalEscaped < <(printf %s "$txtToEval" | tr '`([' '\1\2\3')
# Pass the string to `eval`, escaping embedded double quotes first.
# `printf %s` ensures that the string is printed without interpretation
# (after processing by by bash).
# The `tr` command reconverts the previously escaped chars. back to their
# literal original.
eval printf %s "\"${txtToEvalEscaped//\"/\\\"}\"" | tr '\1\2\3' '`(['
}
Examples:
$ expandVars '\$HOME="$HOME"; `date` and $(ls)'
$HOME="/home/jdoe"; `date` and $(ls) # only $HOME was expanded
$ printf '\$SHELL=${SHELL}, but "$(( 1 \ 2 ))" will not expand' | expandVars
$SHELL=/bin/bash, but "$(( 1 \ 2 ))" will not expand # only ${SHELL} was expanded
${HOME:0:10}, as long as they contain no embedded command or arithmetic substitutions, such as ${HOME:0:$(echo 10)}
$( and ` instances are blindly escaped).${HOME (missing closing }) BREAK the function.\$name prevents expansion.\ not followed by $ is preserved as is.\ instances, you must double them; e.g.:
\\ -> \ - the same as just \\\\\ -> \\0x1, 0x2, 0x3.eval.If you're looking for a more restrictive solution that only supports ${name} expansions - i.e., with mandatory curly braces, ignoring $name references - see this answer of mine.
Here is an improved version of the bash-only, eval-free solution from the accepted answer:
The improvements are:
${name} and $name variable references.\-escaping variable references that shouldn't be expanded.eval-based solution above,
IFS= read -d '' -r lines # read all input from stdin at once
end_offset=${#lines}
while [[ "${lines:0:end_offset}" =~ (.*)\$(\{([a-zA-Z_][a-zA-Z_0-9]*)\}|([a-zA-Z_][a-zA-Z_0-9]*))(.*) ]] ; do
pre=${BASH_REMATCH[1]} # everything before the var. reference
post=${BASH_REMATCH[5]}${lines:end_offset} # everything after
# extract the var. name; it's in the 3rd capture group, if the name is enclosed in {...}, and the 4th otherwise
[[ -n ${BASH_REMATCH[3]} ]] && varName=${BASH_REMATCH[3]} || varName=${BASH_REMATCH[4]}
# Is the var ref. escaped, i.e., prefixed with an odd number of backslashes?
if [[ $pre =~ \\+$ ]] && (( ${#BASH_REMATCH} % 2 )); then
: # no change to $lines, leave escaped var. ref. untouched
else # replace the variable reference with the variable's value using indirect expansion
lines=${pre}${!varName}${post}
fi
end_offset=${#pre}
done
printf %s "$lines"
Here is another solution: generate a bash script with all the variables and the contents of the template file, that script would look like this:
word=dog
i=1
cat << EOF
the number is ${i}
the word is ${word}
EOF
If we feed this script into bash it would produce the desired output:
the number is 1
the word is dog
Here is how to generate that script and feed that script into bash:
(
# Variables
echo word=dog
echo i=1
# add the template
echo "cat << EOF"
cat template.txt
echo EOF
) | bash
cat command with HEREDOCIf you want to redirect this output into a file, replace the last line with:
) | bash > output.txt
A longer but more robust version of the accepted answer:
perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})?;substr($1,0,int(length($1)/2)).($2&&length($1)%2?$2:$ENV{$3||$4});eg' template.txt
This expands all instances of $VAR or ${VAR} to their environment values (or, if they're undefined, the empty string).
It properly escapes backslashes, and accepts a backslash-escaped $ to inhibit substitution (unlike envsubst, which, it turns out, doesn't do this).
So, if your environment is:
FOO=bar
BAZ=kenny
TARGET=backslashes
NOPE=engi
and your template is:
Two ${TARGET} walk into a \\$FOO. \\\\
\\\$FOO says, "Delete C:\\Windows\\System32, it's a virus."
$BAZ replies, "\${NOPE}s."
the result would be:
Two backslashes walk into a \bar. \\
\$FOO says, "Delete C:\Windows\System32, it's a virus."
kenny replies, "${NOPE}s."
If you only want to escape backslashes before $ (you could write "C:\Windows\System32" in a template unchanged), use this slightly-modified version:
perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\});substr($1,0,int(length($1)/2)).(length($1)%2?$2:$ENV{$3||$4});eg' template.txt
Try envsubst
FOO=foo
BAR=bar
export FOO BAR
envsubst <<EOF
FOO is $FOO
BAR is $BAR
EOF
I'd have done it this way, probably less efficient, but easier to read/maintain.
TEMPLATE='/path/to/template.file'
OUTPUT='/path/to/output.file'
while read LINE; do
echo $LINE |
sed 's/VARONE/NEWVALA/g' |
sed 's/VARTWO/NEWVALB/g' |
sed 's/VARTHR/NEWVALC/g' >> $OUTPUT
done < $TEMPLATE
I think eval works really well. It handles templates with linebreaks, whitespace, and all sorts of bash stuff. If you have full control over the templates themselves of course:
$ cat template.txt
variable1 = ${variable1}
variable2 = $variable2
my-ip = \"$(curl -s ifconfig.me)\"
$ echo $variable1
AAA
$ echo $variable2
BBB
$ eval "echo \"$(<template.txt)\"" 2> /dev/null
variable1 = AAA
variable2 = BBB
my-ip = "11.22.33.44"
This method should be used with care, of course, since eval can execute arbitrary code. Running this as root is pretty much out of the question. Quotes in the template need to be escaped, otherwise they will be eaten by eval.
You can also use here documents if you prefer cat to echo
$ eval "cat <<< \"$(<template.txt)\"" 2> /dev/null
@plockc provoded a solution that avoids the bash quote escaping issue:
$ eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
Edit: Removed part about running this as root using sudo...
Edit: Added comment about how quotes need to be escaped, added plockc's solution to the mix!
This page describes an answer with awk
awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < input.txt > output.txt
Perfect case for shtpl. (project of mine, so it is not widely in use and lacks in documentation. But here is the solution it offers anyhow. May you want to test it.)
Just execute:
$ i=1 word=dog sh -c "$( shtpl template.txt )"
Result is:
the number is 1
the word is dog
Have fun.
envsubst was new to me. Fantastic.
For the record, using a heredoc is a great way to template a conf file.
STATUS_URI="/hows-it-goin"; MONITOR_IP="10.10.2.15";
cat >/etc/apache2/conf.d/mod_status.conf <<EOF
<Location ${STATUS_URI}>
SetHandler server-status
Order deny,allow
Deny from all
Allow from ${MONITOR_IP}
</Location>
EOF
# Usage: template your_file.conf.template > your_file.conf
template() {
local IFS line
while IFS=$'\n\r' read -r line ; do
line=${line//\\/\\\\} # escape backslashes
line=${line//\"/\\\"} # escape "
line=${line//\`/\\\`} # escape `
line=${line//\$/\\\$} # escape $
line=${line//\\\${/\${} # de-escape ${ - allows variable substitution: ${var} ${var:-default_value} etc
# to allow arithmetic expansion or command substitution uncomment one of following lines:
# line=${line//\\\$\(/\$\(} # de-escape $( and $(( - allows $(( 1 + 2 )) or $( command ) - UNSECURE
# line=${line//\\\$\(\(/\$\(\(} # de-escape $(( - allows $(( 1 + 2 ))
eval "echo \"${line}\"";
done < "$1"
}
This is the pure bash function adjustable to your liking, used in production and should not break on any input. If it breaks - let me know.
Instead of reinventing the wheel go with envsubst Can be used in almost any scenario, for instance building configuration files from environment variables in docker containers.
If on mac make sure you have homebrew then link it from gettext:
brew install gettext
brew link --force gettext
./template.cfg
# We put env variables into placeholders here
this_variable_1 = ${SOME_VARIABLE_1}
this_variable_2 = ${SOME_VARIABLE_2}
./.env:
SOME_VARIABLE_1=value_1
SOME_VARIABLE_2=value_2
./configure.sh
#!/bin/bash
cat template.cfg | envsubst > whatever.cfg
Now just use it:
# make script executable
chmod +x ./configure.sh
# source your variables
. .env
# export your variables
# In practice you may not have to manually export variables
# if your solution depends on tools that utilise .env file
# automatically like pipenv etc.
export SOME_VARIABLE_1 SOME_VARIABLE_2
# Create your config file
./configure.sh
Here's another pure bash solution:
$ cat code
#!/bin/bash
LISTING=$( ls )
cat_template() {
echo "cat << EOT"
cat "$1"
echo EOT
}
cat_template template | LISTING="$LISTING" bash
$ cat template (with trailing newlines and double quotes)
<html>
<head>
</head>
<body>
<p>"directory listing"
<pre>
$( echo "$LISTING" | sed 's/^/ /' )
<pre>
</p>
</body>
</html>
output
<html>
<head>
</head>
<body>
<p>"directory listing"
<pre>
code
template
<pre>
</p>
</body>
</html>
I have a bash solution like mogsie but with heredoc instead of herestring to allow you to avoid escaping double quotes
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
Here's a bash function that preserves whitespace:
# Render a file in bash, i.e. expand environment variables. Preserves whitespace.
function render_file () {
while IFS='' read line; do
eval echo \""${line}"\"
done < "${1}"
}
You can also use bashible (which internally uses the evaluating approach described above/below).
There is an example, how to generate a HTML from multiple parts:
https://github.com/mig1984/bashible/tree/master/examples/templates
Source: Stackoverflow.com