python 3 2 UnicodeEncodeError charmap codec can t encode character u2013 in position 9629 character maps to undefined

Question

I m trying to make a script that gets data out from an sqlite3 database  but I have run in to a problem   The field in the database is of type text and the contains a html formated text  see the text below   lt html gt   lt head gt   lt title gt Yahoo  lt  title gt   lt  head gt   lt body gt   lt style type  text css  gt  html     yshortcuts  border-bottom none  important    ReadMsgBody  width 100     ExternalClass width 100     lt  style gt   lt table cellpadding  0  cellspacing  0  bgcolor   ffffff  gt       lt tr gt   lt td width  550  valign  top  align  left  gt        lt table cellpadding  0  cellspacing  0  width  500  gt           lt tr gt               lt td colspan  3  gt  lt img        src  http   mail yimg com nq assets sharedmessages v1 us logo gif  width  292  height  51  style  display block   border  0  alt  Yahoo  Mail  gt  lt  td gt           lt  tr gt           lt tr gt               lt td rowspan  3  width  1  bgcolor   c7c4ca  gt  lt  td gt               lt td width  498  height  1  bgcolor   c7c4ca  gt  lt  td gt               lt td rowspan  3  width  1  bgcolor   c7c4ca  gt  lt  td gt           lt  tr gt           lt tr gt               lt td width  498  valign  top  align  left  gt               lt table cellpadding  0  cellspacing  0  gt                   lt tr gt                       lt td width  498  bgcolor   61399d  align  left  valign  top  gt                       lt table cellspacing  0  cellpadding  0  gt  lt tr gt  lt td height  24  gt  lt  td gt  lt  tr gt  lt  table gt                       lt div style  font-family Arial  Helvetica  sans-serif font-size 23px line-height 27px margin-bottom 10px color  ffffff margin-left 15px   gt  lt span style  color  ffffff text-decoration none font-weight bold line-height 27px   gt V  lkommen till Yahoo  Mail  lt  span gt  lt  div gt                       lt div style  font-family Arial  Helvetica  sans-serif font-size 22px line-height 26px margin-bottom 1px color  ffffff margin-left 15px margin-bottom 7px margin-right 15px   gt Ansluta och dela g  r snabbt och enkelt och   r tillg  ngligt   verallt  lt  div gt                       lt  td gt                   lt  tr gt                   lt tr gt                       lt td gt  lt img src  http   mail yimg com nq assets sharedmessages v1 all b1 gif  width  498  height  18  style  display block   border  0  gt  lt  td gt                   lt  tr gt               lt  table gt               lt table cellpadding  0  cellspacing  0  width  498  gt                   lt tr gt                       lt td width  292  valign  top  gt                       lt table cellpadding  0  cellspacing  0  gt                           lt tr gt                               lt td gt  lt img src  http   mail yimg com nq assets sharedmessages v1 all grad gif  width  292  height  9  style  display block   gt  lt  td gt                           lt  tr gt                           lt tr gt                               lt td width  292  bgcolor   ffffff  align  left  valign  top  gt                               lt table cellspacing  0  cellpadding  0  gt  lt tr gt  lt td height  11  gt  lt  td gt  lt  tr gt  lt  table gt                               lt div style  margin-left 15px   gt                                                     lt div style  font-family Arial  Helvetica  sans-serif font-size 14px line-height 18px color  333333 margin-bottom 11px font-weight bold   gt Det   r l  tt som en pl  tt att komma ig  ng  lt  div gt                                   lt table cellpadding  0  cellspacing  0  width  267  gt                                       lt tr gt                                           lt td width  16  align  left  valign  top  gt  lt div style  font-family Arial  Helvetica  sans-serif font-size 14px line-height 16px color  61399d margin-bottom 9px font-weight bold   gt 1   lt  div gt  lt  td gt                                           lt td align  left  valign  top  gt  lt div style  font-family Arial  Helvetica  sans-serif font-size 13px line-height 16px color  61399d margin-bottom 9px   gt  lt a rel  nofollow  target   blank  href  http   us-mg999 mail yahoo com neo launch action contacts  style  text-decoration underline color  61399d   gt  lt span gt L  gg till alla dina kontakter p   en plats lt  span gt  lt  a gt   lt  div gt  lt  td gt                                       lt  tr gt                                       lt tr gt                                           lt td align  left  valign  top  gt  lt div style  font-family Arial  Helvetica  sans-serif font-size 14px line-height 16px color  61399d margin-bottom 9px font-weight bold   gt 2   lt  div gt  lt  td gt                                           lt td align  left  valign  top  gt  lt div style  font-family Arial  Helvetica  sans-serif font-size 13px line-height 16px color  61399d margin-bottom 9px   gt  lt a rel  nofollow  target   blank  href  http   mrd mail yahoo com themes  style  text-decoration underline color  61399d   gt  lt span gt Anpassa din nya inkorg lt  span gt  lt  a gt   lt  div gt  lt  td gt                                       lt  tr gt                                       lt tr gt                                           lt td align  left  valign  top  gt  lt div style  font-family Arial  Helvetica  sans-serif font-size 14px line-height 16px color  61399d margin-bottom 9px font-weight bold   gt 3   lt  div gt  lt  td gt                                           lt td align  left  valign  top  gt  lt div style  font-family Arial  Helvetica  sans-serif font-size 13px line-height 16px color  61399d   gt  lt a rel  nofollow  target   blank  href  http   se overview mail yahoo com mobile  style  text-decoration underline color  61399d   gt  lt span gt Anslut   verallt p   dina mobila enheter lt  span gt  lt  a gt   lt  div gt  lt  td gt                                       lt  tr gt                                   lt  table gt                                lt  div gt                               lt  td gt                           lt  tr gt                           lt tr gt  lt td height  13  gt  lt  td gt  lt  tr gt                       lt  table gt                       lt  td gt                       lt td width  196  valign  top  gt                       lt table cellpadding  0  cellspacing  0  gt                           lt tr gt                               lt td width  1  bgcolor   fbfbfd  valign  top  gt  lt img src  http   mail yimg com nq assets sharedmessages v1 all g1 gif  width  1  height  21  style  display block   gt  lt  td gt                               lt td width  1  bgcolor   f5f6fa  valign  top  gt  lt img src  http   mail yimg com nq assets sharedmessages v1 all g2 gif  width  1  height  21  style  display block   gt  lt  td gt                               lt td width  1  bgcolor   e8eaf1  valign  top  gt  lt img src  http   mail yimg com nq assets sharedmessages v1 all g3 gif  width  1  height  21  style  display block   gt  lt  td gt                               lt td width  1  bgcolor   d4d4d4  gt  lt  td gt                               lt td width  186  bgcolor   f0f0f0  align  left  valign  top  gt                                 lt table cellspacing  0  cellpadding  0  gt  lt tr gt  lt td height  3  gt     lt  td gt  lt  tr gt  lt  table gt                               lt div style  margin-left 11px   gt                               lt div style  font-family Arial  Helvetica  sans-serif font-size 13px line-height 16px color  333333 margin-bottom 9px   gt  lt b gt Info f  r dig  lt  b gt  lt  div gt                               lt div style  font-family Arial  Helvetica  sans-serif font-size 12px color  43494e line-height 18px margin-bottom 10px   gt                              Yahoo -ID och e-postadress  lt br   gt                               lt div style  font-family Arial  Helvetica  sans-serif font-size 12px color  43494e line-height 18px   gt                              H  ll ditt konto och inst  llningar aktuella   lt br gt  lt a rel  nofollow  target   blank  href  https   edit yahoo com config eval profile  style  text-decoration underline color  61399d   gt  lt span gt Mitt konto lt  span gt  lt  a gt                                lt  div gt                               lt  div gt                               lt table cellspacing  0  cellpadding  0  gt  lt tr gt  lt td height  20  gt  lt  td gt  lt  tr gt  lt  table gt                               lt  td gt                               lt td width  1  bgcolor   dbdbdb  gt  lt  td gt                               lt td width  1  bgcolor   ced2de  gt  lt  td gt                               lt td width  1  bgcolor   dbdfed  gt  lt  td gt                               lt td width  1  bgcolor   e8ebf3  gt  lt  td gt                               lt td width  1  bgcolor   f3f4f9  gt  lt  td gt                               lt td width  1  bgcolor   fafbfc  gt  lt  td gt                           lt  tr gt                           lt tr gt                               lt td colspan  11  gt  lt img src  http   mail yimg com nq assets sharedmessages v1 all b2 gif  width  196  height  8  style  display block   border  0  gt  lt  td gt                           lt  tr gt                           lt tr gt  lt td height  13  gt  lt  td gt  lt  tr gt                       lt  table gt                       lt  td gt                       lt td width  10  gt  lt  td gt                   lt  tr gt               lt  table gt               lt  td gt           lt  tr gt           lt tr gt               lt td width  498  height  1  bgcolor   c7c4ca  gt  lt  td gt           lt  tr gt       lt  table gt       lt table cellpadding  0  cellspacing  0  width  500  gt           lt tr gt               lt td align  center  valign  top  gt               lt table cellspacing  0  cellpadding  0  gt  lt tr gt  lt td height  10  gt  lt  td gt  lt  tr gt  lt  table gt                   lt div style  font-family Arial  Helvetica  sans-serif font-size 11px line-height 18px margin-bottom 10px   gt                   lt a rel  nofollow  target   blank  href  http   info yahoo com legal se yahoo utos html  style  text-decoration underline color  61399d   gt Yahoo  Villkor f  r anv  ndning lt  a gt  amp nbsp  amp nbsp  amp nbsp   amp nbsp  amp nbsp  amp nbsp  lt a rel  nofollow  target   blank  href  http   info yahoo com legal se yahoo mail atos html  style  text-decoration underline color  61399d   gt Yahoo  Mail    Villkor f  r anv  ndning lt  a gt  amp nbsp  amp nbsp  amp nbsp   amp nbsp  amp nbsp  amp nbsp  lt a rel  nofollow  target   blank  href  http   info yahoo com privacy se yahoo details html  style  text-decoration underline color  61399d   gt Yahoo  Sekretesspolicy lt  a gt                   lt  div gt               lt  td gt           lt  tr gt           lt tr gt               lt td align  left  valign  top  gt                   lt div style  font-family Arial  Helvetica  sans-serif font-size 11px line-height 14px color  545454 margin-left 16px margin-right 14px   gt Var god svara inte p   detta meddelande  Detta   r ett servicemeddelande som r  r din anv  ndning av Yahoo  Mail  Om du vill veta mer om Yahoo s anv  ndning av personlig information  inklusive anv  ndning av webb-beacons i HTML-baserad e-post  kan du l  sa v  r Yahoo  Sekretesspolicy  Yahoo s adress   r 701 First Avenue  Sunnyvale  CA 94089  USA  lt br   gt  lt br   gt RefID  lp-1037111 lt  div gt               lt  td gt           lt  tr gt       lt  table gt            lt  td gt   lt  tr gt   lt  table gt   lt img width  1  height  1  src  http   pclick internal yahoo com p s 2143684696  gt   lt  body gt   lt  html gt     and the python code that try to extract the data is as follows    gt  gt  gt  import sqlite3  gt  gt  gt  conn   sqlite3 connect  C  temp Mobils export com yahoo mobile client android mail databases mail db    gt  gt  gt  c   conn cursor    gt  gt  gt  conn row factory sqlite3 Row  gt  gt  gt  c execute  select body from messages 1 where  id 7    lt sqlite3 Cursor object at 0x0000000001FB78F0 gt   gt  gt  gt  r   c fetchone    gt  gt  gt  r keys     body    gt  gt  gt  print r  body    Traceback  most recent call last     File   lt stdin gt    line 1  in  lt module gt    File  C  Python32 lib encodings cp850 py   line 19  in encode     return codecs charmap encode input self errors encoding map  0  UnicodeEncodeError   charmap  codec can t encode character   u2013  in position 9629  character maps to  lt undefined gt   gt  gt  gt    Does anybody have any idea of how to print write this to a file  Yes I know that this is printed to stdout  but I get the same UnicodeEncodeError when I try to write to a file  I tried both write method of a file object and print r  body    file f

User · Answer

for me   using export PYTHONIOENCODING UTF-8 before executing python command worked

User · Answer

Maybe a little late to reply  I happen to run into the same problem today  I find that on Windows you can change the console encoder to utf-8 or other encoder that can represent your data  Then you can print it to sys stdout   First  run following code in the console   chcp 65001 set PYTHONIOENCODING utf-8   Then  start python do anything you want

User · Answer

While Python 3 deals in Unicode  the Windows console or POSIX tty that you re running inside does not  So  whenever you print  or otherwise send Unicode strings to stdout  and it s attached to a console tty  Python has to encode it   The error message indirectly tells you what character set Python was trying to use     File  C  Python32 lib encodings cp850 py   line 19  in encode   This means the charset is cp850   You can test or yourself that this charset doesn t have the appropriate character just by doing   u2013  encode  cp850    Or you can look up cp850 online  e g   at Wikipedia    It s possible that Python is guessing wrong  and your console is really set for  say UTF-8   In that case  just manually set sys stdout encoding  utf-8    It s also possible that you intended your console to be set for UTF-8 but did something wrong   In that case  you probably want to follow up at superuser com    But if nothing is wrong  you just can t print that character  You will have to manually encode it with one of the non-strict error-handlers  For example    gt  gt  gt    u2013  encode  cp850   UnicodeEncodeError   charmap  codec can t encode character   u2013  in position 0  character maps to  lt undefined gt   gt  gt  gt    u2013  encode  cp850   errors  replace   b        So  how do you print a string that won t print on your console   You can replace every print function with something like this    gt  gt  gt  print r  body   encode  cp850   errors  replace   decode  cp850            but that s going to get pretty tedious pretty fast   The simple thing to do is to just set the error handler on sys stdout    gt  gt  gt  sys stdout errors    replace   gt  gt  gt  print r  body          For printing to a file  things are pretty much the same  except that you don t have to set f errors after the fact  you can set it at construction time  Instead of this   with open  path    w   encoding  cp850   as f    Do this   with open  path    w   encoding  cp850   errors  replace   as f        Or  of course  if you can use UTF-8 files  just do that  as Mark Ransom s answer shows   with open  path    w   encoding  utf-8   as f

User · Answer

When you open the file you want to write to  open it with a specific encoding that can handle all the characters   with open  filename    w   encoding  utf-8   as f      print r  body    file f

[python] python 3.2 UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position 9629: character maps to <undefined>

Examples related to python

Examples related to python-3.x

Examples related to sqlite