UnicodeDecodeError utf8 codec can t decode byte 0x9c

Question

I have a socket server that is supposed to receive UTF-8 valid characters from clients  The problem is some clients  mainly hackers  are sending all the wrong kind of data over it  I can easily distinguish the genuine client  but I am logging to files all the data sent so I can analyze it later  Sometimes I get characters like this    that cause the UnicodeDecodeError error  I need to be able to make the string UTF-8 with or without those characters   Update  For my particular case the socket service was an MTA and thus I only expect to receive ASCII commands such as  EHLO example com MAIL FROM   lt john doe example com gt       I was logging all of this in JSON  Then some folks out there without good intentions decided to send all kind of junk  That is why for my specific case it is perfectly OK to strip the non ASCII characters

User · Answer

gt  gt  gt    x9c  decode  cp1252   u  u0153   gt  gt  gt  print   x9c  decode  cp1252

User · Answer

I have resolved this problem using this code df   pd read csv gdp path  engine  python

User · Answer

the first Using get encoding type to get the files type of encode   import os     from chardet import detect    get file encoding type def get encoding type file       with open file   rb   as f          rawdata   f read       return detect rawdata   encoding     the second  opening the files with the type   open current file   r   encoding   get encoding type  errors  ignore

User · Answer

http   docs python org howto unicode html the-unicode-type  str   unicode str  errors  replace     or  str   unicode str  errors  ignore     Note  This will strip out  ignore  the characters in question returning the string without them   For me this is ideal case since I m using it as protection against non-ASCII input which is not allowed by my application   Alternatively  Use the open method from the codecs module to read in the file   import codecs with codecs open file name   r   encoding  utf-8                    errors  ignore   as fdata

User · Answer

This solution works nice when using Latin American accents  such as       I have solved this problem just by adding df   pd read csv fileName encoding  latin1

User · Answer

This type of issue crops up for me now that I ve moved to Python 3  I had no idea Python 2 was simply steam rolling any issues with file encoding    I found this nice explanation of the differences and how to find a solution after none of the above worked for me    http   python-notes curiousefficiency org en latest python3 text file processing html  In short  to make Python 3 behave as similarly as possible to Python 2 use   with open filename  encoding  latin-1   as datafile        work on datafile here   However  read the article  there is no one size fits all solution

User · Answer

Just in case of someone has the same problem  I am using vim with YouCompleteMe  failed to start ycmd with this error message  what I did is  export LC CTYPE  en US UTF-8   the problem is gone

User · Answer

I had same problem with UnicodeDecodeError and i solved it with this line  Don t know if is the best way but it worked for me   str   str decode  unicode escape   encode  utf-8

User · Answer

Changing the engine from C to Python did the trick for me   Engine is C   pd read csv gdp path  sep   t   engine  c         utf-8  codec can t decode byte 0x92 in position 18  invalid start byte   Engine is Python   pd read csv gdp path  sep   t   engine  python     No errors for me

User · Answer

What can you do if you need to make a change to a file  but don   t know the file   s encoding  If you know the encoding is ASCII-compatible and only want to examine or modify the ASCII parts  you can open the file with the surrogateescape error handler   with open fname   r   encoding  ascii   errors  surrogateescape   as f      data   f read

[python] UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c

Examples related to python

Examples related to linux

Examples related to python-unicode