Which characters need to be escaped when using Bash

Question

Is there any comprehensive list of characters that need to be escaped in Bash  Can it be checked just with sed   In particular  I was checking whether   needs to be escaped or not  I tried  echo  h h    sed  s   i g    and worked fine  without escaping    Does it mean   does not need to be escaped  Was this a good way to check the necessity   And more general  are they the same characters to escape in shell and bash

User · Answer

I presume that you re talking about bash strings   There are different types of strings which have a different set of requirements for escaping  eg  Single quotes strings are different from double quoted strings   The best reference is the Quoting section of the bash manual   It explains which characters needs escaping   Note that some characters may need escaping depending on which options are enabled such as history expansion

User · Answer

Using the print   q  technique  we can run a loop to find out which characters are special      bin bash special           amp    -                  lt  gt      for   i 0  i  lt     special   i      do     char    special i 1       printf -v q char   q    char      if      char       q char      then         printf  Yes - character  s needs to be escaped n    char      else         printf  No - character  s does not need to be escaped n    char      fi done   sort   It gives this output   No  character   does not need to be escaped No  character   does not need to be escaped No  character - does not need to be escaped No  character   does not need to be escaped No  character   does not need to be escaped No  character   does not need to be escaped No  character   does not need to be escaped No  character   does not need to be escaped No  character   does not need to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character  amp  needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character  lt  needs to be escaped Yes  character  gt  needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped Yes  character   needs to be escaped   Some of the results  like   look a little suspicious   Would be interesting to get  CharlesDuffy s inputs on this

User · Answer

format that can be reused as shell input Edit february 2021  bash   var Q  Under bash  you could store your variable content with Parameter Expansion s   command for Parameter transformation     parameter operator         Parameter transformation   The expansion is either a transforma-        tion of the value of parameter or  information  about  parameter        itself   depending on the value of operator   Each operator is a        single letter          Q      The expansion is a string that is the value of  parameter               quoted in a format that can be reused as input             A      The  expansion  is  a string in the form of an assignment               statement or declare command  that   if  evaluated   will               recreate parameter with its attributes and value    Sample    var   Hello nGood world  n    echo  quot  var quot  Hello Good world     echo  quot   var Q  quot    Hello nGood world  n     echo  quot   var A  quot  var   Hello nGood world  n   Old answer There is a special printf format directive   q  built for this kind of request   printf  -v var  format  arguments    q     causes printf to output the corresponding argument         in a format that can be reused as shell input    Some samples  read foo Hello world printf  quot  q n quot   quot  foo quot  Hello  world  printf  quot  q n quot    Hello world  n    Hello world  n   This could be used through variables too  printf -v var  quot  q quot   quot  foo  quot  echo  quot  var quot    Hello world n   Quick check with all  128  ascii bytes  Note that all bytes from 128 to 255 have to be escaped  for i in  0  127   do     printf -v var    o  i     printf -v var  var     printf -v res  quot  q quot   quot  var quot      esc E        quot  var quot     quot  res quot     amp  amp  esc -     printf  quot  02X  s  -7s n quot   i  esc  quot  res quot  done       column  This must render something like  00 E            1A E    032     34 - 4          4E - N          68 - h       01 E    001     1B E    E       35 - 5          4F - O          69 - i       02 E    002     1C E    034     36 - 6          50 - P          6A - j       03 E    003     1D E    035     37 - 7          51 - Q          6B - k       04 E    004     1E E    036     38 - 8          52 - R          6C - l       05 E    005     1F E    037     39 - 9          53 - S          6D - m       06 E    006     20 E            3A -            54 - T          6E - n       07 E    a       21 E            3B E            55 - U          6F - o       08 E    b       22 E   quot          3C E   lt          56 - V          70 - p       09 E    t       23 E            3D -            57 - W          71 - q       0A E    n       24 E            3E E   gt          58 - X          72 - r       0B E    v       25 -            3F E            59 - Y          73 - s       0C E    f       26 E   amp          40 -            5A - Z          74 - t       0D E    r       27 E            41 - A          5B E            75 - u       0E E    016     28 E            42 - B          5C E            76 - v       0F E    017     29 E            43 - C          5D E            77 - w       10 E    020     2A E            44 - D          5E E            78 - x       11 E    021     2B -            45 - E          5F -            79 - y       12 E    022     2C E            46 - F          60 E            7A - z       13 E    023     2D - -          47 - G          61 - a          7B E         14 E    024     2E -            48 - H          62 - b          7C E         15 E    025     2F -            49 - I          63 - c          7D E         16 E    026     30 - 0          4A - J          64 - d          7E E         17 E    027     31 - 1          4B - K          65 - e          7F E    177  18 E    030     32 - 2          4C - L          66 - f       19 E    031     33 - 3          4D - M          67 - g        Where first field is hexa value of byte  second contain E if character need to be escaped and third field show escaped presentation of character  Why    You could see some characters that don t always need to be escaped  like      and    So not always but sometime  echo test 1  2  3 and 4 5  test 1  2  3 and 4 5   or echo test   1  2  3   test   1  2  3    but care  echo test 1 2 3  test1 test2 test3  echo test   1 2 3  test 1 test 2 test 3  echo test     1   2   3    test  1 test  2 test  3  echo test     1    2   3    test  1  2 test  3

User · Answer

I noticed that bash automatically escapes some characters when using auto-complete   For example  if you have a directory named dir A  bash will auto-complete to dir  A  Using this  I runned some experiments using characters of the ASCII table and derived the following lists   Characters that bash escapes on auto-complete   includes space        amp         lt   gt              Characters that bash does not escape      - 0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ abcdefghijklmnopqrstuvwxyz     I excluded    as it cannot be used in directory names

User · Answer

Characters that need escaping are different in Bourne or POSIX shell than Bash   Generally  very  Bash is a superset of those shells  so anything you escape in shell should be escaped in Bash   A nice general rule would be  if in doubt  escape it    But escaping some characters gives them a special meaning  like  n   These are listed in the man bash pages under Quoting and echo   Other than that  escape any character that is not alphanumeric  it is safer   I don t know of a single definitive list     The man pages list them all somewhere  but not in one place   Learn the language  that is the way to be sure   One that has caught me out is     This is a special character  history expansion  in Bash  and csh  but not in Korn shell   Even echo  Hello world   gives problems   Using single-quotes  as usual  removes the special meaning

User · Answer

To save someone else from having to RTFM    in bash      Enclosing characters in double quotes preserves the literal value of all characters within the quotes  with the exception of      x60      and  when history expansion is enabled          so if you escape those  and the quote itself  of course  you re probably okay   If you take a more conservative  when in doubt  escape it  approach  it should be possible to avoid getting instead characters with special meaning by not escaping identifier characters  i e  ASCII letters  numbers  or       It s very unlikely these would ever  i e  in some weird POSIX-ish shell  have special meaning and thus need to be escaped

User · Answer

There are two easy and safe rules which work not only in sh but also bash  1  Put the whole string in single quotes This works for all chars except single quote itself  To escape the single quote  close the quoting before it  insert the single quote  and re-open the quoting   I    m a s fe  tring which ends in newline    sed command  sed -e  quot s           g  1s         s       quot  2  Escape every char with a backslash This works for all characters except newline  For newline characters use single or double quotes  Empty strings must still be handled - replace with  quot  quot   I   m   a   s   f e     t r i n g   w h i c h   e n d s   i n   n e w l i n e quot   quot   sed command  sed -e  s      amp  g  1  s     quot  quot     1 s    quot      s    quot     2b  More readable version of 2 There s an easy safe set of characters  like  a-zA-Z0-9        -   which can be left unescaped to keep it more readable I  m  a  s fe    tring  which  ends  in  newline quot   quot   sed command  LC ALL C sed -e  s   a-zA-Z0-9       -     amp  g  1  s     quot  quot     1 s    quot      s    quot      Note that in a sed program  one can t know whether the last line of input ends with a newline byte  except when it s empty   That s why both above sed commands assume it does not  You can add a quoted newline manually  Note that shell variables are only defined for text in the POSIX sense  Processing binary data is not defined  For the implementations that matter  binary works with the exception of NUL bytes  because variables are implemented with C strings  and meant to be used as C strings  namely program arguments   but you should switch to a  quot binary quot  locale such as latin1    You can easily validate the rules by reading the POSIX spec for sh  For bash  check the reference manual linked by  AustinPhillips

[bash] Which characters need to be escaped when using Bash?

Examples related to bash

Examples related to shell

Examples related to unix

Examples related to escaping

Examples related to special-characters