Bash Templating How to build configuration files from templates with Bash

Question

I m writing a script to automate creating configuration files for Apache and PHP for my own webserver  I don t want to use any GUIs like CPanel or ISPConfig   I have some templates of Apache and PHP configuration files  Bash script needs to read templates  make variable substitution and output parsed templates into some folder  What is the best way to do that  I can think of several ways  Which one is the best or may be there are some better ways to do that  I want to do that in pure Bash  it s easy in PHP for example   1  How to replace     placeholders in a text file   template txt   the number is   i  the word is   word    script sh      bin sh   set variables i 1 word  dog   read in template one line at the time  and replace variables   more natural  and efficient  way  thanks to Jonathan Leffler  while read line do     eval echo   line  done  lt     template txt    BTW  how do I redirect output to external file here  Do I need to escape something if variables contain  say  quotes   2  Using cat  amp  sed for replacing each variable with its value   Given template txt   The number is   i  The word is   word    Command   cat template txt   sed -e  s    i  1     sed -e  s    word  dog     Seems bad to me because of the need to escape many different symbols and with many variables the line will be tooooo long   Can you think of some other elegant and safe solution

User · Answer

Instead of reinventing the wheel go with envsubst Can be used in almost any scenario, for instance building configuration files from environment variables in docker containers.

If on mac make sure you have homebrew then link it from gettext:

brew install gettext
brew link --force gettext

./template.cfg

# We put env variables into placeholders here
this_variable_1 = ${SOME_VARIABLE_1}
this_variable_2 = ${SOME_VARIABLE_2}

./.env:

SOME_VARIABLE_1=value_1
SOME_VARIABLE_2=value_2

./configure.sh

#!/bin/bash
cat template.cfg | envsubst > whatever.cfg

Now just use it:

# make script executable
chmod +x ./configure.sh
# source your variables
. .env
# export your variables
# In practice you may not have to manually export variables 
# if your solution depends on tools that utilise .env file 
# automatically like pipenv etc. 
export SOME_VARIABLE_1 SOME_VARIABLE_2
# Create your config file
./configure.sh

User · Answer

Here is another solution  generate a bash script with all the variables and the contents of the template file  that script would look like this   word dog            i 1                 cat  lt  lt  EOF          the number is   i   the word is   word   EOF                   If we feed this script into bash it would produce the desired output   the number is 1 the word is dog   Here is how to generate that script and feed that script into bash           Variables     echo word dog     echo i 1        add the template     echo  cat  lt  lt  EOF      cat template txt     echo EOF     bash   Discussion   The parentheses opens a sub shell  its purpose is to group together all the output generated Within the sub shell  we generate all the variable declarations Also in the sub shell  we generate the cat command with HEREDOC Finally  we feed the sub shell output to bash and produce the desired output If you want to redirect this output into a file  replace the last line with       bash  gt  output txt

User · Answer

You can use this   perl -p -i -e  s               defined  ENV  1     ENV  1      amp  eg   lt  template txt   to replace all        strings with corresponding enviroment variables  do not forget to export them before running this script    For pure bash this should work  assuming that variables do not contain        strings       bin bash while read -r line   do     while      line           a-zA-Z   a-zA-Z 0-9           do         LHS   BASH REMATCH 1           RHS    eval echo     LHS              line   line   LHS  RHS      done     echo   line  done     Solution that does not hang if RHS references some variable that references itself      bin bash line    cat  echo -n a   end offset    line  while       line 0  end offset                 a-zA-Z   a-zA-Z 0-9                do     PRE    BASH REMATCH 1        POST    BASH REMATCH 4    line  end offset    line        VARNAME    BASH REMATCH 3        eval  VARVAL     VARNAME        line   PRE VARVAL POST      end offset    PRE  done echo -n    line 0 -1     WARNING  I do not know a way to correctly handle input with NULs in bash or preserve the amount of trailing newlines  Last variant is presented as it is because shells    love    binary input    read will interpret backslashes  read -r will not interpret backslashes  but still will drop the last line if it does not end with a newline           will strip as many trailing newlines as there are present  so I end     with   echo -n a and use echo -n    line 0 -1    this drops the last character  which is a  and preserves as many trailing newlines as there was in the input  including no

User · Answer

You can also use bashible  which internally uses the evaluating approach described above below    There is an example  how to generate a HTML from multiple parts   https   github com mig1984 bashible tree master examples templates

User · Answer

I think eval works really well   It handles templates with linebreaks  whitespace  and all sorts of bash stuff   If you have full control over the templates themselves of course     cat template txt variable1     variable1  variable2    variable2 my-ip       curl -s ifconfig me       echo  variable1 AAA   echo  variable2 BBB   eval  echo      lt template txt     2 gt   dev null variable1   AAA variable2   BBB my-ip    11 22 33 44    This method should be used with care  of course  since eval can execute arbitrary code  Running this as root is pretty much out of the question  Quotes in the template need to be escaped  otherwise they will be eaten by eval   You can also use here documents if you prefer cat to echo    eval  cat  lt  lt  lt       lt template txt     2 gt   dev null    plockc provoded a solution that avoids the bash quote escaping issue     eval  cat  lt  lt EOF    lt template txt  EOF   2 gt   dev null   Edit  Removed part about running this as root using sudo      Edit  Added comment about how quotes need to be escaped  added plockc s solution to the mix

User · Answer

Perfect case for shtpl   project of mine  so it is not widely in use and lacks in documentation  But here is the solution it offers anyhow  May you want to test it    Just execute      i 1 word dog sh -c     shtpl template txt      Result is   the number is 1 the word is dog   Have fun

User · Answer

A longer but more robust version of the accepted answer   perl -pe  s           a-zA-Z   a-zA-Z 0-9          a-zA-Z   a-zA-Z 0-9        substr  1 0 int length  1  2     2 amp  amp length  1  2  2  ENV  3   4   eg  template txt   This expands all instances of  VAR or   VAR  to their environment values  or  if they re undefined  the empty string    It properly escapes backslashes  and accepts a backslash-escaped   to inhibit substitution  unlike envsubst  which  it turns out  doesn t do this    So  if your environment is   FOO bar BAZ kenny TARGET backslashes NOPE engi   and your template is   Two   TARGET  walk into a    FOO           FOO says   Delete C   Windows  System32  it s a virus    BAZ replies      NOPE s     the result would be   Two backslashes walk into a  bar       FOO says   Delete C  Windows System32  it s a virus   kenny replies     NOPE s     If you only want to escape backslashes before    you could write  C  Windows System32  in a template unchanged   use this slightly-modified version   perl -pe  s           a-zA-Z   a-zA-Z 0-9          a-zA-Z   a-zA-Z 0-9       substr  1 0 int length  1  2    length  1  2  2  ENV  3   4   eg  template txt

User · Answer

If you want to use Jinja2 templates  see this project  j2cli   It supports    Templates from JSON  INI  YAML files and input streams Templating from environment variables

User · Answer

Here s a modified perl script based on a few of the other answers   perl -pe  s                a-zA-Z   a-zA-Z 0-9       1  ENV  2  eg  -i template   Features  based on my needs  but should be easy to modify     Skips escaped parameter expansions  e g     VAR    Supports parameter expansions of the form   VAR   but not  VAR  Replaces   VAR  with a blank string if there is no VAR envar  Only supports a-z  A-Z  0-9 and underscore characters in the name  excluding digits in the first position

User · Answer

I d have done it this way  probably less efficient  but easier to read maintain   TEMPLATE   path to template file  OUTPUT   path to output file   while read LINE  do   echo  LINE     sed  s VARONE NEWVALA g      sed  s VARTWO NEWVALB g      sed  s VARTHR NEWVALC g   gt  gt   OUTPUT done  lt   TEMPLATE

User · Answer

Here s another pure bash solution    it s using heredoc  so    complexity doesn t increase because of additionaly required syntax template can include bash code   that also allows you to indent stuff properly  See below    it doesn t use eval  so    no problems with the rendering of trailing empty lines no problems with quotes in the template      cat code     bin bash LISTING    ls    cat template       echo  cat  lt  lt  EOT    cat   1    echo EOT    cat template template   LISTING   LISTING  bash     cat template  with trailing newlines and double quotes    lt html gt     lt head gt     lt  head gt     lt body gt        lt p gt  directory listing         lt pre gt     echo   LISTING    sed  s                       lt pre gt       lt  p gt     lt  body gt   lt  html gt    output   lt html gt     lt head gt     lt  head gt     lt body gt        lt p gt  directory listing         lt pre gt          code         template        lt pre gt       lt  p gt     lt  body gt   lt  html gt

User · Answer

I agree with using sed  it is the best tool for search replace  Here is my approach     cat template txt the number is   i  the dog s name is   name     cat replace sed s   i  5  s   name  Fido     sed -f replace sed template txt  gt  out txt    cat out txt the number is 5 the dog s name is Fido

User · Answer

Taking the answer from ZyX using pure bash but with new style regex matching and indirect parameter substitution it becomes      bin bash regex        a-zA-Z   a-zA-Z 0-9       while read line  do     while      line      regex     do         param    BASH REMATCH 1            line   line    BASH REMATCH 0      param       done     echo  line done

User · Answer

This page describes an answer with awk  awk   while match  0                 var substr  0 RSTART 2 RLENGTH -3  gsub       var    ENVIRON var    1   lt  input txt  gt  output txt

User · Answer

I have a bash solution like mogsie but with heredoc instead of herestring to allow you to avoid escaping double quotes  eval  cat  lt  lt EOF    lt template txt  EOF   2 gt   dev null

User · Answer

Usage  template your file conf template  gt  your file conf template             local IFS line         while IFS    n r  read -r line   do                 line   line                     escape backslashes                 line   line                     escape                   line   line                     escape                   line   line                     escape                   line   line                     de-escape            - allows variable substitution    var    var -default value  etc                   to allow arithmetic expansion or command substitution uncomment one of following lines                  line   line                     de-escape    and     - allows     1   2    or    command   - UNSECURE                 line   line                     de-escape            - allows     1   2                    eval  echo     line              done  lt    1      This is the pure bash function adjustable to your liking  used in production and should not break on any input  If it breaks - let me know

User · Answer

Here s a bash function that preserves whitespace     Render a file in bash  i e  expand environment variables  Preserves whitespace  function render file          while IFS    read line  do         eval echo      line         done  lt     1

User · Answer

Edit Jan 6  2017  I needed to keep double quotes in my configuration file so double escaping double quotes with sed helps   render template       eval  echo     sed  s          g   1         I can t think of keeping trailing new lines  but empty lines in between are kept     Although it is an old topic  IMO I found out more elegant solution here  http   pempek net articles 2013 07 08 bash-sh-as-template-engine      bin sh    render a template configuration file   expand variables   preserve formatting render template       eval  echo     cat  1        user  Gregory  render template  path to template txt  gt  path to configuration file   All credits to Gr  gory Pakosz

User · Answer

Try envsubst  FOO foo BAR bar export FOO BAR  envsubst  lt  lt EOF FOO is  FOO BAR is  BAR EOF

User · Answer

Look at simple variables substitution python script here  https   github com jeckep vsubst  It is very simple to use   python subst py --props secure properties --src path   templates --dst path   dist

User · Answer

envsubst was new to me   Fantastic   For the record  using a heredoc is a great way to template a conf file   STATUS URI   hows-it-goin    MONITOR IP  10 10 2 15    cat  gt  etc apache2 conf d mod status conf  lt  lt EOF  lt Location   STATUS URI  gt      SetHandler server-status     Order deny allow     Deny from all     Allow from   MONITOR IP   lt  Location gt  EOF

User · Answer

To follow up on plockc s answer on this page  here is a dash-suitable version  for those of you looking to avoid bashisms   eval  cat  lt  lt EOF  gt outputfile    cat template in   EOF   2 gt   dev null

User · Answer

If using Perl is an option and you re content with basing expansions on environment variables only  as opposed to all shell variables   consider Stuart P  Bentley s robust answer   This answer aims to provide a bash-only solution that - despite use of eval - should be safe to use   The goals are    Support expansion of both   name  and  name variable references  Prevent all other expansions    command substitutions         and legacy syntax         arithmetic substitutions           and legacy syntax           Allow selective suppression of variable expansion by prefixing with       name    Preserve special chars  in the input  notably   and   instances  Allow input either via arguments or via stdin    Function expandVars     expandVars       local txtToEval    txtToEvalEscaped     If no arguments were passed  process stdin input             0     amp  amp  IFS  read -r -d    txtToEval     Disable command substitutions and arithmetic expansions to prevent execution     of arbitrary commands      Note that selectively allowing          or        to enable arithmetic     expressions is NOT safe  because command substitutions could be embedded in them      If you fully trust or control the input  you can remove the  tr  calls below   IFS  read -r -d    txtToEvalEscaped  lt   lt  printf  s   txtToEval    tr         1 2 3       Pass the string to  eval   escaping embedded double quotes first       printf  s  ensures that the string is printed without interpretation      after processing by by bash       The  tr  command reconverts the previously escaped chars  back to their     literal original    eval printf  s      txtToEvalEscaped                tr   1 2 3            Examples     expandVars    HOME   HOME    date  and   ls    HOME   home jdoe    date  and   ls     only  HOME was expanded    printf    SHELL   SHELL   but      1   2     will not expand    expandVars  SHELL  bin bash  but      1   2     will not expand   only   SHELL  was expanded    For performance reasons  the function reads stdin input all at once into memory  but it s easy to adapt the function to a line-by-line approach  Also supports non-basic variable expansions such as   HOME 0 10   as long as they contain no embedded command or arithmetic substitutions  such as   HOME 0   echo 10     Such embedded substitutions actually BREAK the function  because all    and   instances are blindly escaped   Similarly  malformed variable references such as   HOME  missing closing    BREAK the function   Due to bash s handling of double-quoted strings  backslashes are handled as follows      name prevents expansion  A single   not followed  by   is preserved as is  If you want to represent multiple adjacent   instances  you must double them  e g        -    - the same as just        -      The input mustn t contain the following  rarely used  characters  which are used for internal purposes  0x1  0x2  0x3   There s a largely hypothetical concern that if bash should introduce new expansion syntax  this function might not prevent such expansions - see below for a solution that doesn t use eval      If you re looking for a more restrictive solution that only supports   name  expansions - i e   with mandatory curly braces  ignoring  name references - see this answer of mine     Here is an improved version of the bash-only  eval-free solution from the accepted answer   The improvements are    Support for expansion of both   name  and  name variable references  Support for  -escaping variable references that shouldn t be expanded  Unlike the eval-based solution above    non-basic expansions are ignored malformed variable references are ignored  they don t break the script      IFS  read -d    -r lines   read all input from stdin at once  end offset    lines   while       lines 0 end offset                 a-zA-Z   a-zA-Z 0-9        a-zA-Z   a-zA-Z 0-9              do       pre   BASH REMATCH 1     everything before the var  reference       post   BASH REMATCH 5    lines end offset    everything after         extract the var  name  it s in the 3rd capture group  if the name is enclosed in        and the 4th otherwise          -n   BASH REMATCH 3       amp  amp  varName   BASH REMATCH 3      varName   BASH REMATCH 4           Is the var ref  escaped  i e   prefixed with an odd number of backslashes        if     pre             amp  amp        BASH REMATCH    2     then                no change to  lines  leave escaped var  ref  untouched       else   replace the variable reference with the variable s value using indirect expansion            lines   pre    varName   post        fi       end offset    pre   done  printf  s   lines

[bash] Bash Templating: How to build configuration files from templates with Bash?

Examples related to bash

Examples related to templates

Examples related to templating