UnicodeEncodeError latin-1 codec can t encode character

Question

What could be causing this error when I try to insert a foreign character into the database    gt  gt UnicodeEncodeError   latin-1  codec can t encode character u  u201c  in position 0  ordinal not in range 256    And how do I resolve it   Thanks

User · Answer

You are trying to store a Unicode codepoint  u201c using an encoding ISO-8859-1   Latin-1 that can t describe that codepoint  Either you might need to alter the database to use utf-8  and store the string data using an appropriate encoding  or you might want to sanitise your inputs prior to storing the content  i e  using something like Sam Ruby s excellent i18n guide  That talks about the issues that windows-1252 can cause  and suggests how to process it  plus links to sample code

User · Answer

Use the below snippet to convert the text from Latin to English  import unicodedata def strip accents text       return    join char for char in                    unicodedata normalize  NFKD   text                     if unicodedata category char      Mn    strip accents                     output        aeinouu

User · Answer

I ran into this same issue when using the Python MySQLdb module   Since MySQL will let you store just about any binary data you want in a text field regardless of character set  I found my solution here   Using UTF8 with Python MySQLdb  Edit  Quote from the above URL to satisfy the request in the first comment         UnicodeEncodeError  latin-1  codec can t encode character           This is because MySQLdb normally tries to encode everythin to latin-1    This can be fixed by executing the following commands right after   you ve etablished the connection    db set character set  utf8   dbc execute  SET NAMES utf8    dbc execute  SET CHARACTER SET utf8    dbc execute  SET character set connection utf8          db  is the result of MySQLdb connect    and  dbc  is the result of   db cursor

User · Answer

The best solution is    set mysql s charset to  utf-8   do like this comment add use unicode True and charset  utf8       db   MySQLdb connect host  localhost   user    root   passwd       db    testdb   use unicode True  charset  utf8       KyungHoon Kim Mar   13  14 at 17 04    detail see    class Connection  mysql connection           MySQL Database Connection Object         default cursor   cursors Cursor      def   init   self   args    kwargs                        Create a connection to the database  It is strongly recommended         that you only use keyword parameters  Consult the MySQL C API         documentation for more information           host           string  host to connect          user           string  user to connect as          passwd           string  password to use          db           string  database to use          port           integer  TCP IP port to connect to          unix socket           string  location of unix socket to use          conv           conversion dictionary  see MySQLdb converters          connect timeout           number of seconds to wait before the connection attempt           fails           compress           if set  compression is enabled          named pipe           if set  a named pipe is used to connect  Windows only           init command           command which is run once the connection is created          read default file           file from which default client values are read          read default group           configuration group to use from the default file          cursorclass           class object  used to create cursors  keyword only           use unicode           If True  text-like columns are returned as unicode objects           using the connection s character set   Otherwise  text-like           columns are returned as strings   columns are returned as           normal strings  Unicode objects will always be encoded to           the connection s character set regardless of this setting           charset           If supplied  the connection character set will be changed           to this character set  MySQL-4 1 and newer   This implies           use unicode True           sql mode           If supplied  the session SQL mode will be changed to this           setting  MySQL-4 1 and newer   For more details and legal           values  see the MySQL documentation           client flag           integer  flags to use or 0            see MySQL docs or constants CLIENTS py           ssl           dictionary or mapping  contains SSL connection parameters            see the MySQL documentation for more details            mysql ssl set      If this is set  and the client does not           support SSL  NotSupportedError will be raised           local infile           integer  non-zero enables LOAD LOCAL INFILE  zero disables          autocommit           If False  default   autocommit is disabled            If True  autocommit is enabled            If None  autocommit isn t set and server default is used           There are a number of undocumented  non-standard methods  See the         documentation for the MySQL C API for some hints on what they do

User · Answer

I hope your database is at least UTF-8  Then you will need to run yourstring encode  utf-8   before you try putting it into the database

User · Answer

SQLAlchemy users can simply specify their field as convert unicode True   Example  sqlalchemy String 1000  convert unicode True   SQLAlchemy will simply accept unicode objects and return them back  handling the encoding itself   Docs

User · Answer

Character U 201C Left Double Quotation Mark is not present in the Latin-1  ISO-8859-1  encoding   It is present in code page 1252  Western European   This is a Windows-specific encoding that is based on ISO-8859-1 but which puts extra characters into the range 0x80-0x9F  Code page 1252 is often confused with ISO-8859-1  and it s an annoying but now-standard web browser behaviour that if you serve your pages as ISO-8859-1  the browser will treat them as cp1252 instead  However  they really are two distinct encodings    gt  gt  gt  u He said  u201CHello u201D  encode  iso-8859-1   UnicodeEncodeError  gt  gt  gt  u He said  u201CHello u201D  encode  cp1252    He said  x93Hello x94    If you are using your database only as a byte store  you can use cp1252 to encode     and other characters present in the Windows Western code page  But still other Unicode characters which are not present in cp1252 will cause errors   You can use encode       ignore   to suppress the errors by getting rid of the characters  but really in this century you should be using UTF-8 in both your database and your pages  This encoding allows any character to be used  You should also ideally tell MySQL you are using UTF-8 strings  by setting the database connection and the collation on string columns   so it can get case-insensitive comparison and sorting right

User · Answer

Python   You will need to add    -   - coding  UTF-8 -   -  remove the spaces around      to the first line of the python file  and then add the following to the text to encode   encode  ascii    xmlcharrefreplace    This will replace all the unicode characters with it s ASCII equivalent

User · Answer

Latin-1  aka ISO 8859-1  is a single octet character encoding scheme  and you can t fit  u201c       into a byte   Did you mean to use UTF-8 encoding

[python] UnicodeEncodeError: 'latin-1' codec can't encode character

Examples related to python

Examples related to mysql

Examples related to unicode

Examples related to pylons