UnicodeEncodeError ascii codec can t encode character u xef in position 0 ordinal not in range 128

Question

I want to parse my XML document  So I have stored my XML document as below  class XMLdocs db Expando        id   db IntegerProperty          name db StringProperty        content db BlobProperty       Now my below is my code  parser   make parser        curHandler   BasketBallHandler     parser setContentHandler curHandler    for q in XMLdocs all              parser parse StringIO StringIO q content     I am getting below error   ascii  codec can t encode character u  xef  in position 0  ordinal not in range 128  Traceback  most recent call last       File   base python runtime python lib versions 1 google appengine ext webapp   init   py   line 517  in   call       handler post  groups       File   base data home apps parsepython 1 348669006354245654 mapreduce base handler py   line 59  in post     self handle        File   base data home apps parsepython 1 348669006354245654 mapreduce handlers py   line 168  in handle     scan aborted   not self process entity entity  ctx       File   base data home apps parsepython 1 348669006354245654 mapreduce handlers py   line 233  in process entity     handler entity       File   base data home apps parsepython 1 348669006354245654 parseXML py   line 71  in process     parser parse StringIO StringIO q content        File   base python runtime python dist lib python2 5 xml sax expatreader py   line 107  in parse     xmlreader IncrementalParser parse self  source       File   base python runtime python dist lib python2 5 xml sax xmlreader py   line 123  in parse     self feed buffer      File   base python runtime python dist lib python2 5 xml sax expatreader py   line 207  in feed     self  parser Parse data  isFinal       File   base data home apps parsepython 1 348669006354245654 parseXML py   line 136  in characters        print ch    UnicodeEncodeError   ascii  codec can t encode character u  xef  in position 0  ordinal not in range 128

User · Answer

An easy solution to overcome this problem is to set your default encoding to utf8  Follow is an example  import sys  reload sys  sys setdefaultencoding  utf8

User · Answer

The actual best answer for this problem depends on your environment  specifically what encoding your terminal expects     The quickest one-line solution is to encode everything you print to ASCII  which your terminal is almost certain to accept  while discarding characters that you cannot print   print ch  fails print ch encode  ascii    ignore     The better solution is to change your terminal s encoding to utf-8  and encode everything as utf-8 before printing  You should get in the habit of thinking about your unicode encoding EVERY time you print or read a string

User · Answer

Just putting  encode  utf-8   at the end of object will do the job in recent versions of Python

User · Answer

The problem is that you re trying to print an unicode character to a possibly non-unicode terminal  You need to encode it with the  replace option before printing it  e g  print ch encode sys stdout encoding   replace

User · Answer

The problem according to your traceback is the print statement on line 136 of parseXML py  Unfortunately you didn t see fit to post that part of your code  but I m going to guess it is just there for debugging  If you change it to   print repr ch    then you should at least see what you are trying to print

User · Answer

It seems you are hitting a UTF-8 byte order mark  BOM   Try using this unicode string with BOM extracted out   import codecs  content   unicode q content strip codecs BOM UTF8    utf-8   parser parse StringIO StringIO content     I used strip instead of lstrip because in your case you had multiple occurences of BOM  possibly due to concatenated file contents

User · Answer

This worked for me   from django utils encoding import smart str content   smart str content

[python] UnicodeEncodeError: 'ascii' codec can't encode character u'\xef' in position 0: ordinal not in range(128)

Examples related to python

Examples related to google-app-engine

Examples related to xml-parsing