Why should we NOT use sys setdefaultencoding utf-8 in a py script

Question

I have seen few py scripts which use this at the top of the script  In what cases one should use it   import sys reload sys  sys setdefaultencoding  utf-8

User · Answer

tl dr  The answer is NEVER   unless you really know what you re doing   9 10 times the solution can be resolved with a proper understanding of encoding decoding   1 10 people have an incorrectly defined locale or environment and need to set   PYTHONIOENCODING  UTF-8      in their environment to fix console printing problems   What does it do   sys setdefaultencoding  utf-8    struck through to avoid re-use  changes the default encoding decoding used whenever Python 2 x needs to convert a Unicode   to a str    and vice-versa  and the encoding is not given  I e   str u  u20AC   unicode             format u  u20AC      In Python 2 x  the default encoding is set to ASCII and the above examples will fail with   UnicodeDecodeError   ascii  codec can t decode byte 0xe2 in position 0  ordinal not in range 128     My console is configured as UTF-8  so           xe2 x82 xac   hence exception on  xe2   or  UnicodeEncodeError   ascii  codec can t encode character u  u20ac  in position 0  ordinal not in range 128    sys setdefaultencoding  utf-8   will allow these to work for me  but won t necessarily work for people who don t use UTF-8  The default of ASCII ensures that assumptions of encoding are not baked into code  Console  sys setdefaultencoding  utf-8   also has a side effect of appearing to fix sys stdout encoding  used when printing characters to the console  Python uses the user s locale  Linux OS X Un x  or codepage  Windows  to set this  Occasionally  a user s locale is broken and just requires PYTHONIOENCODING to fix the console encoding   Example     export LANG en GB gibberish   python  gt  gt  gt  import sys  gt  gt  gt  sys stdout encoding  ANSI X3 4-1968   gt  gt  gt  print u  u20AC  Traceback  most recent call last     File   lt stdin gt    line 1  in  lt module gt  UnicodeEncodeError   ascii  codec can t encode character u  u20ac  in position 0  ordinal not in range 128   gt  gt  gt  exit      PYTHONIOENCODING UTF-8 python  gt  gt  gt  import sys  gt  gt  gt  sys stdout encoding  UTF-8   gt  gt  gt  print u  u20AC        What s so bad with sys setdefaultencoding  utf-8     People have been developing against Python 2 x for 16 years on the understanding that the default encoding is ASCII  UnicodeError exception handling methods have been written to handle string to Unicode conversions on strings that are found to contain non-ASCII   From https   anonbadger wordpress com 2015 06 16 why-sys-setdefaultencoding-will-break-code   def welcome message byte string       try          return u  s runs your business    byte string     except UnicodeError          return u  s runs your business    unicode byte string              encoding detect encoding byte string    print welcome message u Angstrom         encode  latin-1         Previous to setting defaultencoding this code would be unable to decode the          in the ascii encoding and then would enter the exception handler to guess the encoding and properly turn it into unicode  Printing  Angstrom        runs your business  Once you   ve set the defaultencoding to utf-8 the code will find that the byte string can be interpreted as utf-8 and so it will mangle the data and return this instead  Angstrom  U  runs your business    Changing what should be a constant will have dramatic effects on modules you depend upon  It s better to just fix the data coming in and out of your code   Example problem  While the setting of defaultencoding to UTF-8 isn t the root cause in the following example  it shows how problems are masked and how  when the input encoding changes  the code breaks in an unobvious way   UnicodeDecodeError    39 utf8  39  codec can  39 t decode byte 0x80 in position 3131  invalid start byte

User · Answer

The first danger lies in reload sys    When you reload a module  you actually get two copies of the module in your runtime  The old module is a Python object like everything else  and stays alive as long as there are references to it  So  half of the objects will be pointing to the old module  and half to the new one  When you make some change  you will never see it coming when some random object doesn t see the change    This is IPython shell   In  1   import sys  In  2   sys stdout Out 2    lt colorama ansitowin32 StreamWrapper at 0x3a2aac8 gt   In  3   reload sys   lt module  sys   built-in  gt   In  4   sys stdout Out 4    lt open file   lt stdout gt    mode  w  at 0x00000000022E20C0 gt   In  11   import IPython terminal  In  14   IPython terminal interactiveshell sys stdout Out 14    lt colorama ansitowin32 StreamWrapper at 0x3a9aac8 gt   Now  sys setdefaultencoding   proper  All that it affects is implicit conversion str lt - gt unicode  Now  utf-8 is the sanest encoding on the planet  backward-compatible with ASCII and all   the conversion now  just works   what could possibly go wrong   Well  anything  And that is the danger    There may be some code that relies on the UnicodeError being thrown for non-ASCII input  or does the transcoding with an error handler  which now produces an unexpected result  And since all code is tested with the default setting  you re strictly on  unsupported  territory here  and no-one gives you guarantees about how their code will behave  The transcoding may produce unexpected or unusable results if not everything on the system uses UTF-8 because Python 2 actually has multiple independent  default string encodings    Remember  a program must work for the customer  on the customer s equipment     Again  the worst thing is you will never know that because the conversion is implicit -- you don t really know when and where it happens   Python Zen  koan 2 ahoy   You will never know why  and if  your code works on one system and breaks on another   Or better yet  works in IDE and breaks in console

User · Answer

usr bin env python  - - coding  utf-8 - - u   u mo  ambique  print u encode  utf-8   print u  chmod  x test py   test py mo  ambique mo  ambique    test py  gt  output txt Traceback  most recent call last     File    test py   line 5  in  lt module gt      print u UnicodeEncodeError   ascii  codec can t encode character  u  xe7  in position 2  ordinal not in range 128    on shell works   sending to sdtout not   so that is one workaround  to write to stdout    I made other approach  which is not run if sys stdout encoding  is not define  or in others words   need export PYTHONIOENCODING UTF-8 first to write to stdout                                                           import sys if  sys stdout encoding is None                   print  gt  gt  sys stderr   please set python env PYTHONIOENCODING UTF-8  example  export PYTHONIOENCODING UTF-8  when write to stdout        exit 1      so  using same example   export PYTHONIOENCODING UTF-8   test py  gt  output txt   will work

User · Answer

As per the documentation  This allows you to switch from the default ASCII to other encodings such as UTF-8  which the Python runtime will use whenever it has to decode a string buffer to unicode     This function is only available at Python start-up time  when Python scans the environment  It has to be called in a system-wide module  sitecustomize py  After this module has been evaluated  the setdefaultencoding   function is removed from the sys module    The only way to actually use it is with a reload hack that brings the attribute back    Also  the use of sys setdefaultencoding   has always been discouraged  and it has become a no-op in py3k  The encoding of py3k is hard-wired to  utf-8  and changing it raises an error   I suggest some pointers for reading    http   blog ianbicking org illusive-setdefaultencoding html http   nedbatchelder com blog 200401 printing unicode from python html http   www diveintopython3 net strings html one-ring-to-rule-them-all http   boodebr org main python all-about-python-and-unicode http   blog notdot net 2010 07 Getting-unicode-right-in-Python

[python] Why should we NOT use sys.setdefaultencoding("utf-8") in a py script?

tl;dr

What does it do?

Console

What's so bad with sys.setdefaultencoding("utf-8")?

Example problem

Examples related to python

Examples related to encoding

Examples related to utf-8

Examples related to python-2.x

Examples related to sys