The .encode
method gets applied to a Unicode string to make a byte-string; but you're calling it on a byte-string instead... the wrong way 'round! Look at the codecs
module in the standard library and codecs.open
in particular for better general solutions for reading UTF-8 encoded text files. However, for the csv
module in particular, you need to pass in utf-8 data, and that's what you're already getting, so your code can be much simpler:
import csv
def unicode_csv_reader(utf8_data, dialect=csv.excel, **kwargs):
csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
for row in csv_reader:
yield [unicode(cell, 'utf-8') for cell in row]
filename = 'da.csv'
reader = unicode_csv_reader(open(filename))
for field1, field2, field3 in reader:
print field1, field2, field3
PS: if it turns out that your input data is NOT in utf-8, but e.g. in ISO-8859-1, then you do need a "transcoding" (if you're keen on using utf-8 at the csv
module level), of the form line.decode('whateverweirdcodec').encode('utf-8')
-- but probably you can just use the name of your existing encoding in the yield
line in my code above, instead of 'utf-8'
, as csv
is actually going to be just fine with ISO-8859-* encoded bytestrings.