RE error illegal byte sequence on Mac OS X

Question

I m trying to replace a string in a Makefile on Mac OS X for cross-compiling to iOS  The string has embedded double quotes  The command is   sed -i     s  iphoneos-cross   llvm-gcc -O3  iphoneos-cross   clang -Os g  Configure   And the error is   sed  RE error  illegal byte sequence   I ve tried escaping the double quotes  commas  dashes  and colons with no joy  For example   sed -i     s   iphoneos-cross      llvm-gcc   -O3   iphoneos-cross      clang   -Os g  Configure   I m having a heck of a time debugging the issue  Does anyone know how to get sed to print the position of the illegal byte sequence  Or does anyone know what the illegal byte sequence is

User · Answer

My workaround had been using Perl   find   -type f -print0   xargs -0 perl -pi -e  s was now g

User · Answer

You simply have to pipe an iconv command before the sed command  Ex with file txt input       iconv -f ISO-8859-1 -t UTF8-MAC file txt   sed  s something                g            -f option is the  from  codeset and -t option is the  to  codeset conversion   Take care of case   web pages usually show lowercase like that  lt  charset iso-8859-1    and iconv uses uppercase  You have list of iconv supported codesets in you system with command iconv -l  UTF8-MAC is modern OS Mac codeset for conversion

User · Answer

My workaround had been using gnu sed  Worked fine for my purposes

User · Answer

Add the following lines to your    bash profile or    zshrc file s    export LC CTYPE C  export LANG C

User · Answer

mklement0 s answer is great  but I have some small tweaks   It seems like a good idea to explicitly specify bash s encoding when using iconv  Also  we should prepend a byte-order mark  even though the unicode standard doesn t recommend it  because there can be legitimate confusions between UTF-8 and ASCII without a byte-order mark  Unfortunately  iconv doesn t prepend a byte-order mark when you explicitly specify an endianness  UTF-16BE or UTF-16LE   so we need to use UTF-16  which uses platform-specific endianness  and then use file --mime-encoding to discover the true endianness iconv used    I uppercase all my encodings because when you list all of iconv s supported encodings with iconv -l they are all uppercase      Find out MY FILE s encoding   We ll convert back to this at the end FILE ENCODING     file --brief --mime-encoding MY FILE      Find out bash s encoding  with which we should encode   MY FILE so sed doesn t fail with    sed  RE error  illegal byte sequence BASH ENCODING     locale charmap   tr   lower     upper        Convert to UTF-16  unknown endianness  so iconv ensures   we have a byte-order mark iconv -f   FILE ENCODING  -t UTF-16 MY FILE  gt  MY FILE utf16 encoding   Whether we re using UTF-16BE or UTF-16LE UTF16 ENCODING     file --brief --mime-encoding MY FILE utf16 encoding      Now we can use MY FILE bash encoding with sed iconv -f   UTF16 ENCODING  -t   BASH ENCODING  MY FILE utf16 encoding  gt  MY FILE bash encoding   sed  sed  s     amp    MY FILE bash encoding  gt  MY FILE SEDDED bash encoding   now convert MY FILE SEDDED bash encoding back to its original encoding iconv -f   BASH ENCODING  -t   FILE ENCODING  MY FILE SEDDED bash encoding  gt  MY FILE SEDDED   Now MY FILE SEDDED has been processed by sed  and is in the same encoding as MY FILE

User · Answer

A sample command that exhibits the symptom  sed  s        lt  lt  lt    xfc  fails  because byte 0xfc is not a valid UTF-8 char  Note that  by contrast  GNU sed  Linux  but also installable on macOS  simply passes the invalid byte through  without reporting an error   Using the formerly accepted answer is an option if you don t mind losing support for your true locale  if you re on a US system and you never need to deal with foreign characters  that may be fine    However  the same effect can be had ad-hoc for a single command only   LC ALL C sed -i     s  iphoneos-cross   llvm-gcc -O3  iphoneos-cross   clang -Os g  Configure   Note  What matters is an effective LC CTYPE setting of C  so LC CTYPE C sed     would normally also work  but if LC ALL happens to be set  to something other than C   it will override individual LC  -category variables such as LC CTYPE  Thus  the most robust approach is to set LC ALL   However   effectively  setting LC CTYPE to C treats strings as if each byte were its own character  no interpretation based on encoding rules is performed   with no regard for the - multibyte-on-demand - UTF-8 encoding that OS X employs by default  where foreign characters have multibyte encodings   In a nutshell  setting LC CTYPE to C causes the shell and utilities to only recognize basic English letters as letters  the ones in the 7-bit ASCII range   so that foreign chars  will not be treated as letters  causing  for instance  upper- lowercase conversions to fail   Again  this may be fine if you needn t match multibyte-encoded characters such as     and simply want to pass such characters through   If this is insufficient and or you want to understand the cause of the original error  including determining what input bytes caused the problem  and perform encoding conversions on demand  read on below     The problem is that the input file s encoding does not match the shell s  More specifically  the input file contains characters encoded in a way that is not valid in UTF-8  as  Klas Lindb  ck stated in a comment  - that s what the sed error message is trying to say by invalid byte sequence   Most likely  your input file uses a single-byte 8-bit encoding such as ISO-8859-1  frequently used to encode  Western European  languages   Example    The accented letter    has Unicode codepoint 0xE0  224  - the same as in ISO-8859-1  However  due to the nature of UTF-8 encoding  this single codepoint is represented as 2 bytes - 0xC3 0xA0  whereas trying to pass the single byte 0xE0 is invalid under UTF-8   Here s a demonstration of the problem using the string voil   encoded as ISO-8859-1  with the    represented as one byte  via an ANSI-C-quoted bash string          that uses  x e0  to create the byte    Note that the sed command is effectively a no-op that simply passes the input through  but we need it to provoke the error       - gt   illegal byte sequence   byte 0xE0 is not a valid char  sed  s     amp     lt  lt  lt   voil x e0     To simply ignore the problem  the above LCTYPE C approach can be used       No error  bytes are passed through       will render as      though   LC CTYPE C sed  s     amp     lt  lt  lt   voil x e0     If you want to determine which parts of the input cause the problem  try the following       Convert bytes in the 8-bit range  high bit set  to hex  representation      - gt   voil x e0   iconv -f ASCII --byte-subst   x  02x    lt  lt  lt   voil x e0     The output will show you all bytes that have the high bit set  bytes that exceed the 7-bit ASCII range  in hexadecimal form   Note  however  that that also includes correctly encoded UTF-8 multibyte sequences - a more sophisticated approach would be needed to specifically identify invalid-in-UTF-8 bytes      Performing encoding conversions on demand   Standard utility iconv can be used to convert to  -t  and or from  -f  encodings  iconv -l lists all supported ones   Examples   Convert FROM ISO-8859-1 to the encoding in effect in the shell  based on LC CTYPE  which is UTF-8-based by default   building on the above example       Converts to UTF-8  output renders correctly as  voil    sed  s     amp     lt  lt  lt    iconv -f ISO-8859-1  lt  lt  lt   voil x e0       Note that this conversion allows you to properly match foreign characters       Correctly matches      and replaces it with       - gt   voil    sed  s          lt  lt  lt    iconv -f ISO-8859-1  lt  lt  lt   voil x e0       To convert the input BACK to ISO-8859-1 after processing  simply pipe the result to another iconv command   sed  s          lt  lt  lt    iconv -f ISO-8859-1  lt  lt  lt   voil x e0       iconv -t ISO-8859-1

User · Answer

Does anyone know how to get sed to print the position of the illegal byte sequence? Or does anyone know what the illegal byte sequence is?

$ uname -a
Darwin Adams-iMac 18.7.0 Darwin Kernel Version 18.7.0: Tue Aug 20 16:57:14 PDT 2019; root:xnu-4903.271.2~2/RELEASE_X86_64 x86_64

I got part of the way to answering the above just by using tr.

I have a .csv file that is a credit card statement and I am trying to import it into Gnucash. I am based in Switzerland so I have to deal with words like Zürich. Suspecting Gnucash does not like " " in numeric fields, I decide to simply replace all

; ;

with

;;

Here goes:

$ head -3 Auswertungen.csv | tail -1 | sed -e 's/; ;/;;/g'
sed: RE error: illegal byte sequence

I used od to shed some light: Note the 374 halfway down this od -c output

$ head -3 Auswertungen.csv | tail -1 | od -c
0000000    1   6   8   7       9   6   1   9       7   1   2   2   ;   5
0000020    4   6   8       8   7   X   X       X   X   X   X       2   6
0000040    6   0   ;   M   Y       N   A   M   E       I   S   X   ;   1
0000060    4   .   0   2   .   2   0   1   9   ;   9   5   5   2       -
0000100        M   i   t   a   r   b   e   i   t   e   r   r   e   s   t
0000120                Z 374   r   i   c   h                            
0000140    C   H   E   ;   R   e   s   t   a   u   r   a   n   t   s   ,
0000160        B   a   r   s   ;   6   .   2   0   ;   C   H   F   ;    
0000200    ;   C   H   F   ;   6   .   2   0   ;       ;   1   5   .   0
0000220    2   .   2   0   1   9  \n                                    
0000227

Then I thought I might try to persuade tr to substitute 374 for whatever the correct byte code is. So first I tried something simple, which didn't work, but had the side effect of showing me where the troublesome byte was:

$ head -3 Auswertungen.csv | tail -1 | tr . .  ; echo
tr: Illegal byte sequence
1687 9619 7122;5468 87XX XXXX 2660;MY NAME ISX;14.02.2019;9552 - Mitarbeiterrest   Z

You can see tr bails at the 374 character.

Using perl seems to avoid this problem

$ head -3 Auswertungen.csv | tail -1 | perl -pne 's/; ;/;;/g'
1687 9619 7122;5468 87XX XXXX 2660;ADAM NEALIS;14.02.2019;9552 - Mitarbeiterrest   Z?rich       CHE;Restaurants, Bars;6.20;CHF;;CHF;6.20;;15.02.2019

[regex] RE error: illegal byte sequence on Mac OS X

The answer is

Examples related to regex

Examples related to macos

Examples related to bash

Examples related to sed

Tags