UnicodeDecodeError ascii codec can t decode byte 0xd1 in position 2 ordinal not in range 128

Question

I am attempting to work with a very large dataset that has some non-standard characters in it  I need to use unicode  as per the job specs  but I am baffled   And quite possibly doing it all wrong    I open the CSV using    15     ncesReader   csv reader open  geocoded output csv    rb    delimiter   t   quotechar        Then  I attempt to encode it with   name school name encode  utf-8    street row 9  encode  utf-8    city row 10  encode  utf-8    state row 11  encode  utf-8    zip5 row 12   zip4 row 13  county row 25  encode  utf-8    lat row 22   lng row 23     I m encoding everything except the lat and lng because those need to be sent out to an API  When I run the program to parse the dataset into what I can use  I get the following Traceback   Traceback  most recent call last     File  push into db py   line 80  in  lt module gt      main     File  push into db py   line 74  in main     district map   buildDistrictSchoolMap     File  push into db py   line 32  in buildDistrictSchoolMap     county row 25  encode  utf-8    lat row 22   lng row 23   UnicodeDecodeError   ascii  codec can t decode byte 0xd1 in position 2  ordinal not in range 128    I think I should tell you that I m using python 2 7 2  and this is part of an app build on django 1 4  I ve read several posts on this topic  but none of them seem to directly apply  Any help will be greatly appreciated   You might also want to know that some of the non-standard characters causing the issue are    and possibly

User · Answer

My computer had the wrong locale set   I first did   gt  gt  gt  import locale  gt  gt  gt  locale getpreferredencoding False   ANSI X3 4-1968    locale getpreferredencoding False  is the function called by open   when you don t provide an encoding  The output should be  UTF-8   but in this case it s some variant of ASCII   Then I ran the bash command locale and got this output    locale LANG  LANGUAGE  LC CTYPE  POSIX  LC NUMERIC  POSIX  LC TIME  POSIX  LC COLLATE  POSIX  LC MONETARY  POSIX  LC MESSAGES  POSIX  LC PAPER  POSIX  LC NAME  POSIX  LC ADDRESS  POSIX  LC TELEPHONE  POSIX  LC MEASUREMENT  POSIX  LC IDENTIFICATION  POSIX  LC ALL    So  I was using the default Ubuntu locale  which causes Python to open files as ASCII instead of UTF-8  I had to set my locale to en US UTF-8  sudo apt install locales  sudo locale-gen en US en US UTF-8     sudo dpkg-reconfigure locales   If you can t change the locale system wide  you can invoke all your Python code like this   PYTHONIOENCODING  UTF-8  python3   path to your script py   or do   export PYTHONIOENCODING  UTF-8    to set it in the shell you run that in

User · Answer

Or when you deal with text in Python if it is a Unicode text  make a note it is Unicode   Set text u unicode text  instead just text  unicode text    This worked in my case

User · Answer

For Python 3 users   changing the encoding from  ascii  to  latin1  works   Also  you can try finding the encoding automatically by reading the top 10000 bytes using the below snippet   import chardet   with open  dataset path    rb   as rawdata                result   chardet detect rawdata read 10000     print result

User · Answer

open with encoding UTF 16 because of lat and long    with open csv name here   r   encoding  utf-16   as f

User · Answer

The main reason for the error is that the default encoding assumed by python is ASCII   Hence  if the string data to be encoded by encode  utf8   contains character that is outside of ASCII range e g  for a string like  hgvcj   387   python would throw error because the string is not in the expected encoding format    If you are using python version earlier than version 3 5  a reliable fix would be to set the default encoding assumed by python to utf8   import sys reload sys  sys setdefaultencoding  utf8   name   school name encode  utf8     This way python would be able to anticipate characters within a string that fall outside of ASCII range   However  if you are using python version 3 5 or above  reload   function is not available  so you would have to fix it using decode e g   name   school name decode  utf8   encode  utf8

User · Answer

for Python 3 users  you can do  with open csv name here   r   encoding  utf-8   as f       some codes   it works  with flask too

User · Answer

Unicode is not equal to UTF-8  The latter is just an encoding for the former   You are doing it the wrong way around  You are reading UTF-8-encoded data  so you have to decode the UTF-8-encoded String into a unicode string   So just replace  encode with  decode  and it should work  if your  csv is UTF-8-encoded    Nothing to be ashamed of  though  I bet 3 in 5 programmers had trouble at first understanding this  if not more     Update  If your input data is not UTF-8 encoded  then you have to  decode   with the appropriate encoding  of course  If nothing is given  python assumes ASCII  which obviously fails on non-ASCII-characters

User · Answer

It does work by just taking the argument  rb  read binary instead of  r  read

User · Answer

if you get this issue while running certbot while creating or renewing certificate  Please use the following method  grep -r -P     x00- x7f    etc apache2  etc letsencrypt  etc nginx  That command found the offending character      in one  conf file in the comment  After removing it  you can edit comments as you wish  and reloading nginx  everything worked again   Source  https   github com certbot certbot issues 5236

User · Answer

Just add this lines to your codes    import sys reload sys  sys setdefaultencoding  utf-8

[python] UnicodeDecodeError: 'ascii' codec can't decode byte 0xd1 in position 2: ordinal not in range(128)

Examples related to python

Examples related to django

Examples related to utf-8