[python] UnicodeDecodeError: 'utf8' codec can't decode byte 0xa5 in position 0: invalid start byte

I am using Python-2.6 CGI scripts but found this error in server log while doing json.dumps(),

Traceback (most recent call last):
  File "/etc/mongodb/server/cgi-bin/getstats.py", line 135, in <module>
    print json.dumps(??__get?data())
  File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa5 in position 0: invalid start byte

?Here ,

?__get?data() function returns dictionary {} .

Before posting this question I have referred this of question os SO.

UPDATES

Following line is hurting JSON encoder,

now = datetime.datetime.now()
now = datetime.datetime.strftime(now, '%Y-%m-%dT%H:%M:%S.%fZ')
print json.dumps({'current_time': now}) # this is the culprit

I got a temporary fix for it

print json.dumps( {'old_time': now.encode('ISO-8859-1').strip() })

But I am not sure is it correct way to do it.

The following snippet worked for me.

import pandas as pd
df = pd.read_csv(filename, sep = ';', encoding = 'latin1', error_bad_lines=False) #error_bad_lines is avoid single line error

On read csv, I added an encoding method:

import pandas as pd
dataset = pd.read_csv('sample_data.csv', header= 0,
                        encoding= 'unicode_escape')

As of 2018-05 this is handled directly with decode, at least for Python 3.

I'm using the below snippet for invalid start byte and invalid continuation byte type errors. Adding errors='ignore' fixed it for me.

with open(out_file, 'rb') as f:
    for line in f:
        print(line.decode(errors='ignore'))

After trying all the aforementioned workarounds, if it still throws the same error, you can try exporting the file as CSV (a second time if you already have). Especially if you're using scikit learn, it is best to import the dataset as a CSV file.

I spent hours together, whereas the solution was this simple. Export the file as a CSV to the directory where Anaconda or your classifier tools are installed and try.

Instead of looking for ways to decode a5 (Yen ¥) or 96 (en-dash –), tell MySQL that your client is encoded "latin1", but you want "utf8" in the database.

See details in Trouble with UTF-8 characters; what I see is not what I stored

Simple Solution:

import pandas as pd
df = pd.read_csv('file_name.csv', engine='python')

Set default encoder at the top of your code

import sys
reload(sys)
sys.setdefaultencoding("ISO-8859-1")

Following line is hurting JSON encoder,

now = datetime.datetime.now()
now = datetime.datetime.strftime(now, '%Y-%m-%dT%H:%M:%S.%fZ')
print json.dumps({'current_time': now}) // this is the culprit

I got a temporary fix for it

print json.dumps( {'old_time': now.encode('ISO-8859-1').strip() })

Marking this as correct as a temporary fix (Not sure so).

Your string has a non ascii character encoded in it.

Not being able to decode with utf-8 may happen if you've needed to use other encodings in your code. For example:

>>> 'my weird character \x96'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x96 in position 19: invalid start byte

In this case, the encoding is windows-1252 so you have to do:

>>> 'my weird character \x96'.decode('windows-1252')
u'my weird character \u2013'

Now that you have Unicode, you can safely encode into utf-8.

This solution worked for me:

import pandas as pd
data = pd.read_csv("training.csv", encoding = 'unicode_escape')

In my case, i had to save the file as UTF8 with BOM not just as UTF8 utf8 then this error was gone.

If the above methods are not working for you, you may want to look into changing the encoding of the csv file itself.

Using Excel:

1. Open csv file using Excel
2. Navigate to File menu option and click Save As
3. Click Browse to select a location to save the file
4. Enter intended filename
5. Select CSV (Comma delimited) (*.csv) option
6. Click Tools drop-down box and click Web Options
7. Under Encoding tab, select the option Unicode (UTF-8) from Save this document as drop-down list
8. Save the file

Using Notepad:

1. Open csv file using notepad
2. Navigate to File > Save As option
3. Next, select the location to the file
4. Select the Save as type option as All Files(.)
5. Specify the file name with .csv extension
6. From Encoding drop-down list, select UTF-8 option.
7. Click Save to save the file

By doing this, you should be able to import csv files without encountering the UnicodeCodeError.

Inspired by @aaronpenne and @Soumyaansh

f = open("file.txt", "rb")
text = f.read().decode(errors='replace')

You may use any standard encoding of your specific usage and input.

utf-8 is the default.

iso8859-1 is also popular for Western Europe.

e.g: bytes_obj.decode('iso8859-1')

see: docs

I switched this simply by defining a different codec package in the read_csv() command:

encoding = 'unicode_escape'

Eg:

import pandas as pd
data = pd.read_csv(filename, encoding= 'unicode_escape')

from io import BytesIO

df = pd.read_excel(BytesIO(bytes_content), engine='openpyxl')

worked for me

Try the below code snippet:

with open(path, 'rb') as f:
  text = f.read()