Escape a string for a sed replace pattern

Question

In my bash script I have an external  received from user  string  which I should use in sed pattern   REPLACE   lt funny characters here gt   sed  s KEYWORD  REPLACE g    How can I escape the  REPLACE string so it would be safely accepted by sed as a literal replacement   NOTE  The KEYWORD is a dumb substring with no matches etc  It is not supplied by user

User · Answer

I have an improvement over the sedeasy function, which WILL break with special characters like tab.

function sedeasy_improved {
    sed -i "s/$(
        echo "$1" | sed -e 's/\([[\/.*]\|\]\)/\\&/g' 
            | sed -e 's:\t:\\t:g'
    )/$(
        echo "$2" | sed -e 's/[\/&]/\\&/g' 
            | sed -e 's:\t:\\t:g'
    )/g" "$3"
}

So, whats different? $1 and $2 wrapped in quotes to avoid shell expansions and preserve tabs or double spaces.

Additional piping | sed -e 's:\t:\\t:g' (I like : as token) which transforms a tab in \t.

User · Answer

These are the escape codes that I ve found        x2a      x28      x29       x22      x2f      x5c       x27      x3f      x25      x5e

User · Answer

Based on Pianosaurus s regular expressions  I made a bash function that escapes both keyword and replacement   function sedeasy     sed -i  s   echo  1   sed -e  s                    amp  g     echo  2   sed -e  s     amp      amp  g   g   3     Here s how you use it   sedeasy  include  etc nginx conf d     include  apps   conf nginx conf   etc nginx nginx conf

User · Answer

Based on Pianosaurus s regular expressions  I made a bash function that escapes both keyword and replacement   function sedeasy     sed -i  s   echo  1   sed -e  s                    amp  g     echo  2   sed -e  s     amp      amp  g   g   3     Here s how you use it   sedeasy  include  etc nginx conf d     include  apps   conf nginx conf   etc nginx nginx conf

User · Answer

I have an improvement over the sedeasy function, which WILL break with special characters like tab.

function sedeasy_improved {
    sed -i "s/$(
        echo "$1" | sed -e 's/\([[\/.*]\|\]\)/\\&/g' 
            | sed -e 's:\t:\\t:g'
    )/$(
        echo "$2" | sed -e 's/[\/&]/\\&/g' 
            | sed -e 's:\t:\\t:g'
    )/g" "$3"
}

So, whats different? $1 and $2 wrapped in quotes to avoid shell expansions and preserve tabs or double spaces.

Additional piping | sed -e 's:\t:\\t:g' (I like : as token) which transforms a tab in \t.

User · Answer

These are the escape codes that I ve found        x2a      x28      x29       x22      x2f      x5c       x27      x3f      x25      x5e

User · Answer

The only three literal characters which are treated specially in the replace clause are    to close the clause      to escape characters  backreference   amp c    and  amp   to include the match in the replacement    Therefore  all you need to do is escape those three characters   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g    Example     export REPLACE          gt  lt  amp      echo fooKEYWORDbar   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g  foo      gt  lt  amp  bar

User · Answer

Here is an example of an AWK I used a while ago   It is an AWK that prints new AWKS   AWK and SED being similar it may be a good template   ls   awk    print  awk               print  1  2  3                  1   old ext  gt     1   new ext       gt  for the birds   It looks excessive  but somehow that combination of quotes works to keep the   printed as literals   Then if I remember correctly the vaiables are just surrounded with quotes like this    1    Try it  let me know how it works with SED

User · Answer

If you are just looking to replace Variable value in sed command then just remove  Example   sed -i  s dev- dev- ENV g  test to sed -i s dev- dev- ENV g test

User · Answer

echo  1 2 3  4  5    sed -r  s                    amp  g s        g

User · Answer

The only three literal characters which are treated specially in the replace clause are    to close the clause      to escape characters  backreference   amp c    and  amp   to include the match in the replacement    Therefore  all you need to do is escape those three characters   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g    Example     export REPLACE          gt  lt  amp      echo fooKEYWORDbar   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g  foo      gt  lt  amp  bar

User · Answer

An easier way to do this is simply building the string before hand and using it as a parameter for sed  rpstring  s KEYWORD  REPLACE g  sed -i  rpstring  test txt

User · Answer

don t forget all the pleasure that occur with the shell limitation around   and    so  in ksh   Var   gt New version of   content  here  lt   printf   s     Var     sed  s   amp                 amp  g    read -r EscVar  echo  Here is your   text   to change    sed  s text   EscVar  g

User · Answer

The sed command allows you to use other characters instead of   as separator   sed  s  http   www  fubar  com  URL FUBAR g    The double quotes are not a problem

User · Answer

Warning  This does not consider newlines  For a more in-depth answer  see this SO-question instead   Thanks  Ed Morton  amp  Niklas Peter   Note that escaping everything is a bad idea  Sed needs many characters to be escaped to get their special meaning  For example  if you escape a digit in the replacement string  it will turn in to a backreference   As Ben Blank said  there are only three characters that need to be escaped in the replacement string  escapes themselves  forward slash for end of statement and  amp  for replace all    ESCAPED REPLACE   printf   s n    REPLACE    sed -e  s     amp      amp  g     Now you can use ESCAPED REPLACE in the original sed statement sed  s KEYWORD  ESCAPED REPLACE g    If you ever need to escape the KEYWORD string  the following is the one you need   sed -e  s                g   And can be used by   KEYWORD  The Keyword You Need   ESCAPED KEYWORD   printf   s n    KEYWORD    sed -e  s               amp  g       Now you can use it inside the original sed statement to replace text sed  s  ESCAPED KEYWORD  ESCAPED REPLACE g    Remember  if you use a character other than   as delimiter  you need replace the slash in the expressions above wih the character you are using  See PeterJCLaw s comment for explanation   Edited  Due to some corner cases previously not accounted for  the commands above have changed several times  Check the edit history for details

User · Answer

It turns out you re asking the wrong question  I also asked the wrong question  The reason it s wrong is the beginning of the first sentence   In my bash script       I had the same question  amp  made the same mistake  If you re using bash  you don t need to use sed to do string replacements  and it s much cleaner to use the replace feature built into bash    Instead of something like  for example   function escape-all-funny-characters     UNKNOWN CODE THAT ANSWERS THE QUESTION YOU ASKED    INPUT  some long string with KEYWORD that need replacing KEYWORD   A    escape-all-funny-characters  KEYWORD    B    escape-all-funny-characters   lt funny characters here gt     OUTPUT    sed  s  A  B g   lt  lt  lt   INPUT      you can use bash features exclusively   INPUT  some long string with KEYWORD that need replacing KEYWORD   A  KEYWORD  B   lt funny characters here gt   OUTPUT    INPUT    A    B

User · Answer

Here is an example of an AWK I used a while ago   It is an AWK that prints new AWKS   AWK and SED being similar it may be a good template   ls   awk    print  awk               print  1  2  3                  1   old ext  gt     1   new ext       gt  for the birds   It looks excessive  but somehow that combination of quotes works to keep the   printed as literals   Then if I remember correctly the vaiables are just surrounded with quotes like this    1    Try it  let me know how it works with SED

User · Answer

It s a bit late to respond    but there IS a much simpler way to do this   Just change the delimiter  i e   the character that separates fields    So  instead of s foo bar  you write s bar foo   And  here s the easy way to do this   sed  s     50017 DEFINER  snafu   localhost      g    The resulting output is devoid of that nasty DEFINER clause

User · Answer

The only three literal characters which are treated specially in the replace clause are    to close the clause      to escape characters  backreference   amp c    and  amp   to include the match in the replacement    Therefore  all you need to do is escape those three characters   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g    Example     export REPLACE          gt  lt  amp      echo fooKEYWORDbar   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g  foo      gt  lt  amp  bar

User · Answer

Use awk - it is cleaner     awk -v R    addr   file     sub  THIS   R   0   print  0     lt  lt  lt   http   file   THIS  path to a file  is   a   nightmare  http   file     addr  file  path to a file  is   a   nightmare

User · Answer

echo  1 2 3  4  5    sed -r  s                    amp  g s        g

User · Answer

It s a bit late to respond    but there IS a much simpler way to do this   Just change the delimiter  i e   the character that separates fields    So  instead of s foo bar  you write s bar foo   And  here s the easy way to do this   sed  s     50017 DEFINER  snafu   localhost      g    The resulting output is devoid of that nasty DEFINER clause

User · Answer

Here is an example of an AWK I used a while ago   It is an AWK that prints new AWKS   AWK and SED being similar it may be a good template   ls   awk    print  awk               print  1  2  3                  1   old ext  gt     1   new ext       gt  for the birds   It looks excessive  but somehow that combination of quotes works to keep the   printed as literals   Then if I remember correctly the vaiables are just surrounded with quotes like this    1    Try it  let me know how it works with SED

User · Answer

don t forget all the pleasure that occur with the shell limitation around   and    so  in ksh   Var   gt New version of   content  here  lt   printf   s     Var     sed  s   amp                 amp  g    read -r EscVar  echo  Here is your   text   to change    sed  s text   EscVar  g

User · Answer

If you are just looking to replace Variable value in sed command then just remove  Example   sed -i  s dev- dev- ENV g  test to sed -i s dev- dev- ENV g test

User · Answer

Warning  This does not consider newlines  For a more in-depth answer  see this SO-question instead   Thanks  Ed Morton  amp  Niklas Peter   Note that escaping everything is a bad idea  Sed needs many characters to be escaped to get their special meaning  For example  if you escape a digit in the replacement string  it will turn in to a backreference   As Ben Blank said  there are only three characters that need to be escaped in the replacement string  escapes themselves  forward slash for end of statement and  amp  for replace all    ESCAPED REPLACE   printf   s n    REPLACE    sed -e  s     amp      amp  g     Now you can use ESCAPED REPLACE in the original sed statement sed  s KEYWORD  ESCAPED REPLACE g    If you ever need to escape the KEYWORD string  the following is the one you need   sed -e  s                g   And can be used by   KEYWORD  The Keyword You Need   ESCAPED KEYWORD   printf   s n    KEYWORD    sed -e  s               amp  g       Now you can use it inside the original sed statement to replace text sed  s  ESCAPED KEYWORD  ESCAPED REPLACE g    Remember  if you use a character other than   as delimiter  you need replace the slash in the expressions above wih the character you are using  See PeterJCLaw s comment for explanation   Edited  Due to some corner cases previously not accounted for  the commands above have changed several times  Check the edit history for details

User · Answer

An easier way to do this is simply building the string before hand and using it as a parameter for sed  rpstring  s KEYWORD  REPLACE g  sed -i  rpstring  test txt

User · Answer

Here is an example of an AWK I used a while ago   It is an AWK that prints new AWKS   AWK and SED being similar it may be a good template   ls   awk    print  awk               print  1  2  3                  1   old ext  gt     1   new ext       gt  for the birds   It looks excessive  but somehow that combination of quotes works to keep the   printed as literals   Then if I remember correctly the vaiables are just surrounded with quotes like this    1    Try it  let me know how it works with SED

User · Answer

If the case happens to be that you are generating a random password to pass to sed replace pattern  then you choose to be careful about which set of characters in the random string  If you choose a password made by encoding a value as base64  then there is is only character that is both possible in base64 and is also a special character in sed replace pattern  That character is      and is easily removed from the password you are generating     password 32 characters log  minus any copies of the     character  pass  openssl rand -base64 32   sed -e  s     g

User · Answer

It turns out you re asking the wrong question  I also asked the wrong question  The reason it s wrong is the beginning of the first sentence   In my bash script       I had the same question  amp  made the same mistake  If you re using bash  you don t need to use sed to do string replacements  and it s much cleaner to use the replace feature built into bash    Instead of something like  for example   function escape-all-funny-characters     UNKNOWN CODE THAT ANSWERS THE QUESTION YOU ASKED    INPUT  some long string with KEYWORD that need replacing KEYWORD   A    escape-all-funny-characters  KEYWORD    B    escape-all-funny-characters   lt funny characters here gt     OUTPUT    sed  s  A  B g   lt  lt  lt   INPUT      you can use bash features exclusively   INPUT  some long string with KEYWORD that need replacing KEYWORD   A  KEYWORD  B   lt funny characters here gt   OUTPUT    INPUT    A    B

User · Answer

Use awk - it is cleaner     awk -v R    addr   file     sub  THIS   R   0   print  0     lt  lt  lt   http   file   THIS  path to a file  is   a   nightmare  http   file     addr  file  path to a file  is   a   nightmare

User · Answer

The only three literal characters which are treated specially in the replace clause are    to close the clause      to escape characters  backreference   amp c    and  amp   to include the match in the replacement    Therefore  all you need to do is escape those three characters   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g    Example     export REPLACE          gt  lt  amp      echo fooKEYWORDbar   sed  s KEYWORD   echo  REPLACE   sed -e  s         g  s         g  s  amp      amp  g   g  foo      gt  lt  amp  bar

User · Answer

The sed command allows you to use other characters instead of   as separator   sed  s  http   www  fubar  com  URL FUBAR g    The double quotes are not a problem

User · Answer

If the case happens to be that you are generating a random password to pass to sed replace pattern  then you choose to be careful about which set of characters in the random string  If you choose a password made by encoding a value as base64  then there is is only character that is both possible in base64 and is also a special character in sed replace pattern  That character is      and is easily removed from the password you are generating     password 32 characters log  minus any copies of the     character  pass  openssl rand -base64 32   sed -e  s     g

[string] Escape a string for a sed replace pattern

Examples related to string

Examples related to sed

Examples related to escaping