Reading a UTF8 CSV file with Python

Question

I am trying to read a CSV file with accented characters with Python  only French and or Spanish characters   Based on the Python 2 5 documentation for the csvreader  http   docs python org library csv html   I came up with the following code to read the CSV file since the csvreader supports only ASCII   def unicode csv reader unicode csv data  dialect csv excel    kwargs         csv py doesn t do Unicode  encode temporarily as UTF-8      csv reader   csv reader utf 8 encoder unicode csv data                               dialect dialect    kwargs      for row in csv reader            decode UTF-8 back to Unicode  cell by cell          yield  unicode cell   utf-8   for cell in row   def utf 8 encoder unicode csv data       for line in unicode csv data          yield line encode  utf-8    filename    output csv  reader   unicode csv reader open filename   try      products          for field1  field2  field3 in reader                Below is an extract of the CSV file I am trying to read   0665000FS10120684 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Bleu 0665000FS10120689 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Gris 0665000FS10120687 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Vert       Even though I try to encode decode to UTF-8  I am still getting the following exception   Traceback  most recent call last     File    Test py   line 53  in  lt module gt      for field1  field2  field3 in reader    File    Test py   line 40  in unicode csv reader     for row in csv reader    File    Test py   line 46  in utf 8 encoder     yield line encode  utf-8    ignore   UnicodeDecodeError   ascii  codec can t decode byte 0xc3 in position 68  ordinal not in range 128    How do I fix this

User · Answer

Had the same problem on another server  but realized that locales are messed   export LC ALL  en US UTF-8    fixed the problem

User · Answer

Worth noting that if nothing worked for you  you may have forgotten to escape your path  For example  this code   f   open  C  Some Path To file csv     Would result in an error      SyntaxError   unicode error   unicodeescape  codec can t decode bytes   in position 2-3  truncated  UXXXXXXXX escape   To fix  simply do   f   open  C   Some  Path  To  file csv

User · Answer

If you want to read a CSV File with encoding utf-8  a minimalistic approach that I recommend you is to use something like this   with open file name  encoding  utf8   as csv file    With that statement  you can use later a CSV reader to work with

User · Answer

The  encode method gets applied to a Unicode string to make a byte-string  but you re calling it on a byte-string instead    the wrong way  round   Look at the codecs module in the standard library and codecs open in particular for better general solutions for reading UTF-8 encoded text files  However  for the csv module in particular  you need to pass in utf-8 data  and that s what you re already getting  so your code can be much simpler   import csv  def unicode csv reader utf8 data  dialect csv excel    kwargs       csv reader   csv reader utf8 data  dialect dialect    kwargs      for row in csv reader          yield  unicode cell   utf-8   for cell in row   filename    da csv  reader   unicode csv reader open filename   for field1  field2  field3 in reader    print field1  field2  field3    PS  if it turns out that your input data is NOT in utf-8  but e g  in ISO-8859-1  then you do need a  transcoding   if you re keen on using utf-8 at the csv module level   of the form line decode  whateverweirdcodec   encode  utf-8   -- but probably you can just use the name of your existing encoding in the yield line in my code above  instead of  utf-8   as csv is actually going to be just fine with ISO-8859-  encoded bytestrings

User · Answer

Python 2 X  There is a unicode-csv library which should solve your problems  with added benefit of not naving to write any new csv-related code   Here is a example from their readme    gt  gt  gt  import unicodecsv  gt  gt  gt  from cStringIO import StringIO  gt  gt  gt  f   StringIO    gt  gt  gt  w   unicodecsv writer f  encoding  utf-8    gt  gt  gt  w writerow  u      u        gt  gt  gt  f seek 0   gt  gt  gt  r   unicodecsv reader f  encoding  utf-8    gt  gt  gt  row   r next    gt  gt  gt  print row 0   row 1          Python 3 X  In python 3 this is supported out of the box by the build-in csv module  See this example    import csv with open  some csv   newline     encoding  utf-8   as f      reader   csv reader f      for row in reader          print row

User · Answer

Also checkout the answer in this post  https   stackoverflow com a 9347871 1338557  It suggests use of library called ucsv py  Short and simple replacement for CSV written to address the encoding problem utf-8  for Python 2 7  Also provides support for csv DictReader  Edit  Adding sample code that I used   import ucsv as csv   Read CSV file containing the right tags to produce fileObj   open  awol title strings csv    rb   dictReader   csv DictReader fileObj  fieldnames     titles    tags    delimiter        quotechar         Build a dictionary from the CSV file- gt    lt string gt   lt tags to produce gt   titleStringsDict   dict   for row in dictReader      titleStringsDict update  unicode row  titles    unicode row  tags

User · Answer

Looking at the Latin-1 unicode table  I see the character code 00E9  LATIN SMALL LETTER E WITH ACUTE   This is the accented character in your sample data  A simple test in Python shows that UTF-8 encoding for this character is different from the unicode  almost UTF-16  encoding    gt  gt  gt  u  u00e9  u  xe9   gt  gt  gt  u  u00e9  encode  utf-8     xc3 xa9   gt  gt  gt     I suggest you try to encode  UTF-8   the unicode data before calling the special unicode csv reader    Simply reading the data from a file might hide the encoding  so check the actual character values

User · Answer

The link to the help page is the same for python 2 6 and as far as I know there was no change in the csv module since 2 5  besides bug fixes   Here is the code that just works without any encoding decoding  file da csv contains the same data as the variable data   I assume that your file should be read correctly without any conversions   test py      - - coding  utf-8 - -     NOTE  this first line is important for the version b  read from a string unicode  variable    import csv  data        0665000FS10120684 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Bleu 0665000FS10120689 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Gris 0665000FS10120687 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Vert       a  read from a file print  reading from a file   for  f1  f2  f3  in csv reader open  da csv    dialect csv excel       print  f1  f2  f3     b  read from a string unicode  variable print  reading from a list of strings   reader   csv reader data split   n    dialect csv excel  for  f1  f2  f3  in reader      print  f1  f2  f3    da csv   0665000FS10120684 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Bleu 0665000FS10120689 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Gris 0665000FS10120687 SD1200IS Appareil photo num  rique PowerShot de 10 Mpx de Canon avec tr  pied  SD1200IS  - Vert

User · Answer

Using codecs open as Alex Martelli suggested proved to be useful to me   import codecs  delimiter       reader   codecs open  your filename csv    r   encoding  utf-8   for line in reader      row   line split delimiter        do something with your row

[python] Reading a UTF8 CSV file with Python

Examples related to python

Examples related to utf-8

Examples related to csv

Examples related to character-encoding