How to replace placeholders in a text file

Question

I want to pipe the output of a  template  file into MySQL  the file having variables like   dbName  interspersed  What is the command line utility to replace these instances and dump the output to standard output

User · Answer

Here s a robust Bash function that - despite using eval - should be safe to use   All   varName  variable references in the input text are expanded based on the calling shell s variables     Nothing else is expanded  neither variable references whose names are not enclosed in        such as  varName   nor command substitutions         and legacy syntax         nor arithmetic substitutions           and legacy syntax           To treat a   as a literal   -escape it  e g     HOME   Note that input is only accepted via stdin   Example     expandVarsStrict  lt  lt  lt   HOME is    HOME     date  and    ls     only   HOME  is expanded  HOME is   Users jdoe    date  and   ls    Function source code   expandVarsStrict      local line lineEscaped   while IFS  read -r line       -n  line     do    the      clause ensures that the last line is read even if it doesn t end with  n       Escape ALL chars  that could trigger an expansion       IFS  read -r -d    lineEscaped  lt   lt  printf  s   line    tr          1 2 3 4             then selectively reenable    references     lineEscaped   lineEscaped     4              Finally  escape embedded double quotes to preserve them      lineEscaped   lineEscaped               eval  printf   s n     lineEscaped      tr   1 2 3 4           done     The function assumes that no 0x1  0x2  0x3  and 0x4 control characters are present in the input  because those chars  are used internally - since the function processes text  that should be a safe assumption

User · Answer

I was thinking about this again  given the recent interest  and I think that the tool that I was originally thinking of was m4  the macro processor for autotools  So instead of the variable I originally specified  you d use    echo  I am a DBNAME    m4 -DDBNAME  database name

User · Answer

If you are open to using Perl  that would be my suggestion  Although there are probably some sed and or AWK experts that probably know how to do this much easier  If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily  but you might just as well put it into a standard Perl script at that point   perl -p -e  s     dbName   testdb s  yourfile   mysql   A short Perl script to do something slightly more complicated  handle multiple keys       usr bin env perl my  replace      dbName    gt   testdb    somethingElse    gt   fooBar     undef     my  buf    lt STDIN gt    buf    s           replace     g for keys  replace  print  buf    If you name the above script as replace-script  it could then be used as follows   replace-script  lt  yourfile   mysql

User · Answer

If you are open to using Perl  that would be my suggestion  Although there are probably some sed and or AWK experts that probably know how to do this much easier  If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily  but you might just as well put it into a standard Perl script at that point   perl -p -e  s     dbName   testdb s  yourfile   mysql   A short Perl script to do something slightly more complicated  handle multiple keys       usr bin env perl my  replace      dbName    gt   testdb    somethingElse    gt   fooBar     undef     my  buf    lt STDIN gt    buf    s           replace     g for keys  replace  print  buf    If you name the above script as replace-script  it could then be used as follows   replace-script  lt  yourfile   mysql

User · Answer

If you are open to using Perl  that would be my suggestion  Although there are probably some sed and or AWK experts that probably know how to do this much easier  If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily  but you might just as well put it into a standard Perl script at that point   perl -p -e  s     dbName   testdb s  yourfile   mysql   A short Perl script to do something slightly more complicated  handle multiple keys       usr bin env perl my  replace      dbName    gt   testdb    somethingElse    gt   fooBar     undef     my  buf    lt STDIN gt    buf    s           replace     g for keys  replace  print  buf    If you name the above script as replace-script  it could then be used as follows   replace-script  lt  yourfile   mysql

User · Answer

Here s a robust Bash function that - despite using eval - should be safe to use   All   varName  variable references in the input text are expanded based on the calling shell s variables     Nothing else is expanded  neither variable references whose names are not enclosed in        such as  varName   nor command substitutions         and legacy syntax         nor arithmetic substitutions           and legacy syntax           To treat a   as a literal   -escape it  e g     HOME   Note that input is only accepted via stdin   Example     expandVarsStrict  lt  lt  lt   HOME is    HOME     date  and    ls     only   HOME  is expanded  HOME is   Users jdoe    date  and   ls    Function source code   expandVarsStrict      local line lineEscaped   while IFS  read -r line       -n  line     do    the      clause ensures that the last line is read even if it doesn t end with  n       Escape ALL chars  that could trigger an expansion       IFS  read -r -d    lineEscaped  lt   lt  printf  s   line    tr          1 2 3 4             then selectively reenable    references     lineEscaped   lineEscaped     4              Finally  escape embedded double quotes to preserve them      lineEscaped   lineEscaped               eval  printf   s n     lineEscaped      tr   1 2 3 4           done     The function assumes that no 0x1  0x2  0x3  and 0x4 control characters are present in the input  because those chars  are used internally - since the function processes text  that should be a safe assumption

User · Answer

here s my solution with perl based on former answer  replaces environment variables   perl -p -e  s       w      exists  ENV  1   ENV  1   missing variable  1   eg   lt  infile  gt  outfile

User · Answer

Sed   Given template txt    The number is   i  The word is   word    we just have to say   sed -e  s    i  1   -e  s    word  dog   template txt   Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation

User · Answer

It can be done in bash itself if you have control of the configuration file format  You just need to source       the configuration file rather than subshell it  That ensures the variables are created in the context of the current shell  and continue to exist  rather than the subshell  where the variable disappear when the subshell exits      cat config data     export parm jdbc jdbc db2   box7 co uk 5000 INSTA     export parm user pax     export parm pwd never you mind    cat go bash       config data     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   If your config file cannot be a shell script  you can just  compile  it before executing thus  the compilation depends on your input format      cat config data     parm jdbc jdbc db2   box7 co uk 5000 INSTA   JDBC URL     parm user pax                                user name     parm pwd never you mind                      password    cat go bash     cat config data           sed  s                   sed  s    t                 sed  s     t                grep -v                sed  s   export            gt config data-compiled       config data-compiled     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   In your specific case  you could use something like     cat config data     export p p1 val1     export p p2 val2   cat go bash         config data     echo  select   from dbtable where p1     p p1  and p2 like   p p2   order by p1    bash go bash     select   from dbtable where p1    val1  and p2 like  val2   order by p1   Then pipe the output of go bash into MySQL and voila  hopefully you won t destroy your database  -

User · Answer

Lots of choices here  but figured I d toss mine on the heap   It is perl based  only targets variables of the form         takes the file to process as an argument and outputs the converted file on stdout   use Env  Env  import     while  lt  gt           s      w     1 eeg   text           print   text     Of course I m not really a perl person  so there could easily be a fatal flaw  works for me though

User · Answer

I found this thread while wondering the same thing   It inspired me to this  careful with the backticks     echo  MYTEST pass    cat FILE hello  MYTEST world   eval echo  cat FILE  hello pass  world

User · Answer

If you are open to using Perl  that would be my suggestion  Although there are probably some sed and or AWK experts that probably know how to do this much easier  If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily  but you might just as well put it into a standard Perl script at that point   perl -p -e  s     dbName   testdb s  yourfile   mysql   A short Perl script to do something slightly more complicated  handle multiple keys       usr bin env perl my  replace      dbName    gt   testdb    somethingElse    gt   fooBar     undef     my  buf    lt STDIN gt    buf    s           replace     g for keys  replace  print  buf    If you name the above script as replace-script  it could then be used as follows   replace-script  lt  yourfile   mysql

User · Answer

Use  bin sh  Create a small shell script that sets the variables  and then parse the template using the shell itself  Like so  edit to handle newlines correctly    File template txt   the number is   i  the word is   word    File script sh      bin sh   Set variables i 1 word  dog    Read in template one line at the time  and replace variables  more  natural  and efficient  way  thanks to Jonathan Leffler   while read line do     eval echo   line  done  lt     template txt    Output    sh script sh the number is 1 the word is dog

User · Answer

It can be done in bash itself if you have control of the configuration file format  You just need to source       the configuration file rather than subshell it  That ensures the variables are created in the context of the current shell  and continue to exist  rather than the subshell  where the variable disappear when the subshell exits      cat config data     export parm jdbc jdbc db2   box7 co uk 5000 INSTA     export parm user pax     export parm pwd never you mind    cat go bash       config data     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   If your config file cannot be a shell script  you can just  compile  it before executing thus  the compilation depends on your input format      cat config data     parm jdbc jdbc db2   box7 co uk 5000 INSTA   JDBC URL     parm user pax                                user name     parm pwd never you mind                      password    cat go bash     cat config data           sed  s                   sed  s    t                 sed  s     t                grep -v                sed  s   export            gt config data-compiled       config data-compiled     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   In your specific case  you could use something like     cat config data     export p p1 val1     export p p2 val2   cat go bash         config data     echo  select   from dbtable where p1     p p1  and p2 like   p p2   order by p1    bash go bash     select   from dbtable where p1    val1  and p2 like  val2   order by p1   Then pipe the output of go bash into MySQL and voila  hopefully you won t destroy your database  -

User · Answer

In place perl editing of potentially multiple files  with backups     perl -e  s               defined  ENV  1     ENV  1       eg        -i orig       -p config test

User · Answer

In place perl editing of potentially multiple files  with backups     perl -e  s               defined  ENV  1     ENV  1       eg        -i orig       -p config test

User · Answer

Use  bin sh  Create a small shell script that sets the variables  and then parse the template using the shell itself  Like so  edit to handle newlines correctly    File template txt   the number is   i  the word is   word    File script sh      bin sh   Set variables i 1 word  dog    Read in template one line at the time  and replace variables  more  natural  and efficient  way  thanks to Jonathan Leffler   while read line do     eval echo   line  done  lt     template txt    Output    sh script sh the number is 1 the word is dog

User · Answer

file tpl   The following bash function should only replace   var1  syntax and ignore  other shell special chars such as  backticks  or  var2 or  double quotes    If I have missed anything - let me know    script sh   template          usage  template file tpl     while read -r line   do             line   line                       line   line                       line   line                       line   line                         eval  echo    line          done  lt    1     var1   replaced   var2   not replaced    template file tpl  gt  result txt

User · Answer

Create rendertemplate sh      usr bin env bash  eval  echo     cat  1       And template tmpl   Hello    WORLD  Goodbye    CHEESE    Render the template     export WORLD Foo   CHEESE Bar   rendertemplate sh template tmpl  Hello  Foo Goodbye  Bar

User · Answer

Update  Here is a solution from yottatsa on a similar question that only does replacement for variables like  VAR or   VAR   and is a brief one-liner  i 32 word foo envsubst  lt  template txt   Of course if i and word are in your environment  then it is just  envsubst  lt  template txt   On my Mac it looks like it was installed as part of gettext and from MacGPG2  Old Answer  Here is an improvement to the solution from mogsie on a similar question  my solution does not require you to escale double quotes  mogsie s does  but his is a one liner   eval  cat  lt  lt EOF    lt template txt  EOF   2 gt   dev null   The power on these two solutions is that you only get a few types of shell expansions that don t occur normally                  and         though backslash is an escape character here  but you don t have to worry that the parsing has a bug  and it does multiple lines just fine

User · Answer

I created a shell templating script named shtpl  My shtpl uses a jinja-like syntax which  now that I use ansible a lot  I m pretty familiar with    cat  tmp test    aux 4       myarray   a b c d         A RANDOM  RANDOM     A RANDOM    if      A RANDOM 2       0     A RANDOM is even    else     A RANDOM is odd    endif       if      A RANDOM 2       0       for n in 1 2 3  aux      myarray    n-1       myarray    n-1      etc passwd field   n    grep  USER  etc passwd   cut -d  -f n     endfor       else       for n in  1  4       myarray    n-1       myarray    n-1      etc group field   n    grep   USER  etc group   cut -d  -f n     endfor       endif          shtpl  lt   tmp test 6535 6535 is odd  myarray 0   a  etc group field  1  myusername  myarray 1   b  etc group field  2  x  myarray 2   c  etc group field  3  1001  myarray 3   d  etc group field  4     More info on my github

User · Answer

Use  bin sh  Create a small shell script that sets the variables  and then parse the template using the shell itself  Like so  edit to handle newlines correctly    File template txt   the number is   i  the word is   word    File script sh      bin sh   Set variables i 1 word  dog    Read in template one line at the time  and replace variables  more  natural  and efficient  way  thanks to Jonathan Leffler   while read line do     eval echo   line  done  lt     template txt    Output    sh script sh the number is 1 the word is dog

User · Answer

I found this thread while wondering the same thing   It inspired me to this  careful with the backticks     echo  MYTEST pass    cat FILE hello  MYTEST world   eval echo  cat FILE  hello pass  world

User · Answer

It can be done in bash itself if you have control of the configuration file format  You just need to source       the configuration file rather than subshell it  That ensures the variables are created in the context of the current shell  and continue to exist  rather than the subshell  where the variable disappear when the subshell exits      cat config data     export parm jdbc jdbc db2   box7 co uk 5000 INSTA     export parm user pax     export parm pwd never you mind    cat go bash       config data     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   If your config file cannot be a shell script  you can just  compile  it before executing thus  the compilation depends on your input format      cat config data     parm jdbc jdbc db2   box7 co uk 5000 INSTA   JDBC URL     parm user pax                                user name     parm pwd never you mind                      password    cat go bash     cat config data           sed  s                   sed  s    t                 sed  s     t                grep -v                sed  s   export            gt config data-compiled       config data-compiled     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   In your specific case  you could use something like     cat config data     export p p1 val1     export p p2 val2   cat go bash         config data     echo  select   from dbtable where p1     p p1  and p2 like   p p2   order by p1    bash go bash     select   from dbtable where p1    val1  and p2 like  val2   order by p1   Then pipe the output of go bash into MySQL and voila  hopefully you won t destroy your database  -

User · Answer

Sed   Given template txt    The number is   i  The word is   word    we just have to say   sed -e  s    i  1   -e  s    word  dog   template txt   Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation

User · Answer

Create rendertemplate sh      usr bin env bash  eval  echo     cat  1       And template tmpl   Hello    WORLD  Goodbye    CHEESE    Render the template     export WORLD Foo   CHEESE Bar   rendertemplate sh template tmpl  Hello  Foo Goodbye  Bar

User · Answer

I would suggest using something like Sigil      https   github com gliderlabs sigil  It is compiled to a single binary  so it s extremely easy to install on systems   Then you can do a simple one-liner like the following     cat my-file conf template   sigil -p   env   gt  my-file conf   This is much safer than eval and easier then using regex or sed

User · Answer

template txt  Variable 1 value    var1  Variable 2 value    var2    data sh     usr bin env bash declare var1  value 1  declare var2  value 2    parser sh     usr bin env bash    args declare file data  1 declare file input  2 declare file output  3  source  file data eval  echo      lt   file input      gt   file output        parser sh data sh template txt parsed file txt   parsed file txt  Variable 1 value  value 1 Variable 2 value  value 2

User · Answer

It can be done in bash itself if you have control of the configuration file format  You just need to source       the configuration file rather than subshell it  That ensures the variables are created in the context of the current shell  and continue to exist  rather than the subshell  where the variable disappear when the subshell exits      cat config data     export parm jdbc jdbc db2   box7 co uk 5000 INSTA     export parm user pax     export parm pwd never you mind    cat go bash       config data     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   If your config file cannot be a shell script  you can just  compile  it before executing thus  the compilation depends on your input format      cat config data     parm jdbc jdbc db2   box7 co uk 5000 INSTA   JDBC URL     parm user pax                                user name     parm pwd never you mind                      password    cat go bash     cat config data           sed  s                   sed  s    t                 sed  s     t                grep -v                sed  s   export            gt config data-compiled       config data-compiled     echo  JDBC string is    parm jdbc     echo  Username is       parm user     echo  Password is       parm pwd    bash go bash     JDBC string is  jdbc db2   box7 co uk 5000 INSTA     Username is     pax     Password is     never you mind   In your specific case  you could use something like     cat config data     export p p1 val1     export p p2 val2   cat go bash         config data     echo  select   from dbtable where p1     p p1  and p2 like   p p2   order by p1    bash go bash     select   from dbtable where p1    val1  and p2 like  val2   order by p1   Then pipe the output of go bash into MySQL and voila  hopefully you won t destroy your database  -

User · Answer

I was thinking about this again  given the recent interest  and I think that the tool that I was originally thinking of was m4  the macro processor for autotools  So instead of the variable I originally specified  you d use    echo  I am a DBNAME    m4 -DDBNAME  database name

User · Answer

I created a shell templating script named shtpl  My shtpl uses a jinja-like syntax which  now that I use ansible a lot  I m pretty familiar with    cat  tmp test    aux 4       myarray   a b c d         A RANDOM  RANDOM     A RANDOM    if      A RANDOM 2       0     A RANDOM is even    else     A RANDOM is odd    endif       if      A RANDOM 2       0       for n in 1 2 3  aux      myarray    n-1       myarray    n-1      etc passwd field   n    grep  USER  etc passwd   cut -d  -f n     endfor       else       for n in  1  4       myarray    n-1       myarray    n-1      etc group field   n    grep   USER  etc group   cut -d  -f n     endfor       endif          shtpl  lt   tmp test 6535 6535 is odd  myarray 0   a  etc group field  1  myusername  myarray 1   b  etc group field  2  x  myarray 2   c  etc group field  3  1001  myarray 3   d  etc group field  4     More info on my github

User · Answer

file tpl   The following bash function should only replace   var1  syntax and ignore  other shell special chars such as  backticks  or  var2 or  double quotes    If I have missed anything - let me know    script sh   template          usage  template file tpl     while read -r line   do             line   line                       line   line                       line   line                       line   line                         eval  echo    line          done  lt    1     var1   replaced   var2   not replaced    template file tpl  gt  result txt

User · Answer

Lots of choices here  but figured I d toss mine on the heap   It is perl based  only targets variables of the form         takes the file to process as an argument and outputs the converted file on stdout   use Env  Env  import     while  lt  gt           s      w     1 eeg   text           print   text     Of course I m not really a perl person  so there could easily be a fatal flaw  works for me though

User · Answer

Sed   Given template txt    The number is   i  The word is   word    we just have to say   sed -e  s    i  1   -e  s    word  dog   template txt   Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation

User · Answer

here s my solution with perl based on former answer  replaces environment variables   perl -p -e  s       w      exists  ENV  1   ENV  1   missing variable  1   eg   lt  infile  gt  outfile

User · Answer

Use  bin sh  Create a small shell script that sets the variables  and then parse the template using the shell itself  Like so  edit to handle newlines correctly    File template txt   the number is   i  the word is   word    File script sh      bin sh   Set variables i 1 word  dog    Read in template one line at the time  and replace variables  more  natural  and efficient  way  thanks to Jonathan Leffler   while read line do     eval echo   line  done  lt     template txt    Output    sh script sh the number is 1 the word is dog

User · Answer

Here is a way to get the shell to do the substitution for you  as if the contents of the file were instead typed between double quotes   Using the example of template txt with contents   The number is   i  The word is   word    The following line will cause the shell to interpolate the contents of template txt and write the result to standard out   i  1  word  dog  sh -c  echo      cat template txt        Explanation    i and word are passed as environment variables scopped to the execution of sh  sh executes the contents of the string it is passed  Strings written next to one another become one string  that string is     echo         cat template txt          Since the substitution is between       cat template txt   becomes the output of cat template txt  So the command executed by sh -c becomes    echo  The number is   i  nThe word is   word    where i and word are the specified environment variables

User · Answer

Sed   Given template txt    The number is   i  The word is   word    we just have to say   sed -e  s    i  1   -e  s    word  dog   template txt   Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation

User · Answer

I would suggest using something like Sigil      https   github com gliderlabs sigil  It is compiled to a single binary  so it s extremely easy to install on systems   Then you can do a simple one-liner like the following     cat my-file conf template   sigil -p   env   gt  my-file conf   This is much safer than eval and easier then using regex or sed

User · Answer

template txt  Variable 1 value    var1  Variable 2 value    var2    data sh     usr bin env bash declare var1  value 1  declare var2  value 2    parser sh     usr bin env bash    args declare file data  1 declare file input  2 declare file output  3  source  file data eval  echo      lt   file input      gt   file output        parser sh data sh template txt parsed file txt   parsed file txt  Variable 1 value  value 1 Variable 2 value  value 2

User · Answer

Update  Here is a solution from yottatsa on a similar question that only does replacement for variables like  VAR or   VAR   and is a brief one-liner  i 32 word foo envsubst  lt  template txt   Of course if i and word are in your environment  then it is just  envsubst  lt  template txt   On my Mac it looks like it was installed as part of gettext and from MacGPG2  Old Answer  Here is an improvement to the solution from mogsie on a similar question  my solution does not require you to escale double quotes  mogsie s does  but his is a one liner   eval  cat  lt  lt EOF    lt template txt  EOF   2 gt   dev null   The power on these two solutions is that you only get a few types of shell expansions that don t occur normally                  and         though backslash is an escape character here  but you don t have to worry that the parsing has a bug  and it does multiple lines just fine

User · Answer

Here is a way to get the shell to do the substitution for you  as if the contents of the file were instead typed between double quotes   Using the example of template txt with contents   The number is   i  The word is   word    The following line will cause the shell to interpolate the contents of template txt and write the result to standard out   i  1  word  dog  sh -c  echo      cat template txt        Explanation    i and word are passed as environment variables scopped to the execution of sh  sh executes the contents of the string it is passed  Strings written next to one another become one string  that string is     echo         cat template txt          Since the substitution is between       cat template txt   becomes the output of cat template txt  So the command executed by sh -c becomes    echo  The number is   i  nThe word is   word    where i and word are the specified environment variables

[bash] How to replace ${} placeholders in a text file?

Examples related to bash

Examples related to command-line

Examples related to text-processing

Examples related to templating