Pandas df to csv file csv encode utf-8 still gives trash characters for minus sign

Question

I ve read something about a Python 2 limitation with respect to Pandas  to csv      etc        Have I hit it  I m on Python 2 7 3  This turns out trash characters for   and - when they appear in strings  Aside from that the export is perfect   df to csv  file csv   encoding  utf-8      Is there any workaround   df head   is this   demography  Adults  49 yrs  Adults 18-49 yrs at high risk      state                                                            Alabama                 32 7                             38 6    Alaska                  31 2                             33 2    Arizona                 22 9                             38 8    Arkansas                31 2                             34 0    California              29 8                             38 8     csv output is this  state   Adults        49 yrs    Adults 18       49 yrs at high risk   0   Alabama     32 7    38 6 1   Alaska  31 2    33 2 2   Arizona     22 9    38 8 3   Arkansas 31 2   34 4   California 29 8  38 8   the whole code is this     import pandas import xlrd import csv import json  df   pandas DataFrame   dy   pandas DataFrame     first merge all this xls together   workbook   xlrd open workbook  csv merger vaccoverage xls   worksheets   workbook sheet names     for i in range 3 len worksheets        dy   pandas io excel read excel workbook  i  engine  xlrd   index None      i   i 1     df   df append dy   df index name    index   df columns     demography    area   state    month    rate    moe     Then just grab month    May   may mask   df  month       May  may df    df may mask     then delete some columns we dont need  may df   may df drop  area   1  may df   may df drop  month   1  may df   may df drop  moe   1    print may df dtypes  uh oh  it sees  rate  as type  object   not  float    Better change that   may df   may df convert objects  rate   convert numeric True   print may df dtypes  that s better  res   may df pivot table  rate    state    demography   print res head      and this is going to spit out an array of Objects  each Object a state containing its demographics res reset index   to json  thejson json   orient  records    and a  csv for good measure res reset index   to csv  thecsv csv   orient  records   encoding  utf-8

User · Answer

Your  bad  output is UTF-8 displayed as CP1252   On Windows  many editors assume the default ANSI encoding  CP1252 on US Windows  instead of UTF-8 if there is no byte order mark  BOM  character at the start of the file   While a BOM is meaningless to the UTF-8 encoding  its UTF-8-encoded presence serves as a signature for some programs   For example  Microsoft Office s Excel requires it even on non-Windows OSes   Try   df to csv  file csv  encoding  utf-8-sig     That encoder will add the BOM

[python] Pandas df.to_csv("file.csv" encode="utf-8") still gives trash characters for minus sign

Examples related to python

Examples related to csv

Examples related to utf-8

Examples related to pandas