Sample code:
>>> import json
>>> json_string = json.dumps("??? ????")
>>> print(json_string)
"\u05d1\u05e8\u05d9 \u05e6\u05e7\u05dc\u05d4"
The problem: it's not human readable. My (smart) users want to verify or even edit text files with JSON dumps (and I’d rather not use XML).
Is there a way to serialize objects into UTF-8 JSON strings (instead of \uXXXX
)?
Here's my solution using json.dump():
def jsonWrite(p, pyobj, ensure_ascii=False, encoding=SYSTEM_ENCODING, **kwargs):
with codecs.open(p, 'wb', 'utf_8') as fileobj:
json.dump(pyobj, fileobj, ensure_ascii=ensure_ascii,encoding=encoding, **kwargs)
where SYSTEM_ENCODING is set to:
locale.setlocale(locale.LC_ALL, '')
SYSTEM_ENCODING = locale.getlocale()[1]
If you are loading JSON string from a file & file contents arabic texts. Then this will work.
{
"key1" : "?????????",
"key2" : "????? ??????"
}
with open(arabic.json, encoding='utf-8') as f:
# deserialises it
json_data = json.load(f)
f.close()
# json formatted string
json_data2 = json.dumps(json_data, ensure_ascii = False)
# If have to get the JSON index in Django Template file, then simply decode the encoded string.
json.JSONDecoder().decode(json_data2)
done! Now we can get the results as JSON index with arabic value.
>>>import json
>>>json_string = json.dumps("??? ????")
>>>json_string.encode('ascii').decode('unicode-escape')
'"??? ????"'
>>>s = '? ?a? ???'
>>>print('unicode: ' + s.encode('unicode-escape').decode('utf-8'))
unicode: \u6f22 \u03c7\u03b1\u03bd \u0445\u0430\u043d
>>>u = s.encode('unicode-escape').decode('utf-8')
>>>print('original: ' + u.encode("utf-8").decode('unicode-escape'))
original: ? ?a? ???
original resource:https://blog.csdn.net/chuatony/article/details/72628868
To write to a file
import codecs
import json
with codecs.open('your_file.txt', 'w', encoding='utf-8') as f:
json.dump({"message":"xin chào vi?t nam"}, f, ensure_ascii=False)
To print to stdout
import json
print(json.dumps({"message":"xin chào vi?t nam"}, ensure_ascii=False))
Using ensure_ascii=False in json.dumps is the right direction to solve this problem, as pointed out by Martijn. However, this may raise an exception:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe7 in position 1: ordinal not in range(128)
You need extra settings in either site.py or sitecustomize.py to set your sys.getdefaultencoding() correct. site.py is under lib/python2.7/ and sitecustomize.py is under lib/python2.7/site-packages.
If you want to use site.py, under def setencoding(): change the first if 0: to if 1: so that python will use your operation system's locale.
If you prefer to use sitecustomize.py, which may not exist if you haven't created it. simply put these lines:
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
Then you can do some Chinese json output in utf-8 format, such as:
name = {"last_name": u"?"}
json.dumps(name, ensure_ascii=False)
You will get an utf-8 encoded string, rather than \u escaped json string.
To verify your default encoding:
print sys.getdefaultencoding()
You should get "utf-8" or "UTF-8" to verify your site.py or sitecustomize.py settings.
Please note that you could not do sys.setdefaultencoding("utf-8") at interactive python console.
Peters' python 2 workaround fails on an edge case:
d = {u'keyword': u'bad credit \xe7redit cards'}
with io.open('filename', 'w', encoding='utf8') as json_file:
data = json.dumps(d, ensure_ascii=False).decode('utf8')
try:
json_file.write(data)
except TypeError:
# Decode data to Unicode first
json_file.write(data.decode('utf8'))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 25: ordinal not in range(128)
It was crashing on the .decode('utf8') part of line 3. I fixed the problem by making the program much simpler by avoiding that step as well as the special casing of ascii:
with io.open('filename', 'w', encoding='utf8') as json_file:
data = json.dumps(d, ensure_ascii=False, encoding='utf8')
json_file.write(unicode(data))
cat filename
{"keyword": "bad credit çredit cards"}
Use codecs if possible,
with codecs.open('file_path', 'a+', 'utf-8') as fp:
fp.write(json.dumps(res, ensure_ascii=False))
UPDATE: This is wrong answer, but it's still useful to understand why it's wrong. See comments.
How about unicode-escape
?
>>> d = {1: "??? ????", 2: u"??? ????"}
>>> json_str = json.dumps(d).decode('unicode-escape').encode('utf8')
>>> print json_str
{"1": "??? ????", "2": "??? ????"}
As of Python 3.7 the following code works fine:
from json import dumps
result = {"symbol": "ƒ"}
json_string = dumps(result, sort_keys=True, indent=2, ensure_ascii=False)
print(json_string)
Output:
{"symbol": "ƒ"}
The following is my understanding var reading answer above and google.
# coding:utf-8
r"""
@update: 2017-01-09 14:44:39
@explain: str, unicode, bytes in python2to3
#python2 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 7: ordinal not in range(128)
#1.reload
#importlib,sys
#importlib.reload(sys)
#sys.setdefaultencoding('utf-8') #python3 don't have this attribute.
#not suggest even in python2 #see:http://stackoverflow.com/questions/3828723/why-should-we-not-use-sys-setdefaultencodingutf-8-in-a-py-script
#2.overwrite /usr/lib/python2.7/sitecustomize.py or (sitecustomize.py and PYTHONPATH=".:$PYTHONPATH" python)
#too complex
#3.control by your own (best)
#==> all string must be unicode like python3 (u'xx'|b'xx'.encode('utf-8')) (unicode 's disappeared in python3)
#see: http://blog.ernest.me/post/python-setdefaultencoding-unicode-bytes
#how to Saving utf-8 texts in json.dumps as UTF8, not as \u escape sequence
#http://stackoverflow.com/questions/18337407/saving-utf-8-texts-in-json-dumps-as-utf8-not-as-u-escape-sequence
"""
from __future__ import print_function
import json
a = {"b": u"??"} # add u for python2 compatibility
print('%r' % a)
print('%r' % json.dumps(a))
print('%r' % (json.dumps(a).encode('utf8')))
a = {"b": u"??"}
print('%r' % json.dumps(a, ensure_ascii=False))
print('%r' % (json.dumps(a, ensure_ascii=False).encode('utf8')))
# print(a.encode('utf8')) #AttributeError: 'dict' object has no attribute 'encode'
print('')
# python2:bytes=str; python3:bytes
b = a['b'].encode('utf-8')
print('%r' % b)
print('%r' % b.decode("utf-8"))
print('')
# python2:unicode; python3:str=unicode
c = b.decode('utf-8')
print('%r' % c)
print('%r' % c.encode('utf-8'))
"""
#python2
{'b': u'\u4e2d\u6587'}
'{"b": "\\u4e2d\\u6587"}'
'{"b": "\\u4e2d\\u6587"}'
u'{"b": "\u4e2d\u6587"}'
'{"b": "\xe4\xb8\xad\xe6\x96\x87"}'
'\xe4\xb8\xad\xe6\x96\x87'
u'\u4e2d\u6587'
u'\u4e2d\u6587'
'\xe4\xb8\xad\xe6\x96\x87'
#python3
{'b': '??'}
'{"b": "\\u4e2d\\u6587"}'
b'{"b": "\\u4e2d\\u6587"}'
'{"b": "??"}'
b'{"b": "\xe4\xb8\xad\xe6\x96\x87"}'
b'\xe4\xb8\xad\xe6\x96\x87'
'??'
'??'
b'\xe4\xb8\xad\xe6\x96\x87'
"""
Thanks for the original answer here. With python 3 the following line of code:
print(json.dumps(result_dict,ensure_ascii=False))
was ok. Consider trying not writing too much text in the code if it's not imperative.
This might be good enough for the python console. However, to satisfy a server you might need to set the locale as explained here (if it is on apache2) http://blog.dscpl.com.au/2014/09/setting-lang-and-lcall-when-using.html
basically install he_IL or whatever language locale on ubuntu check it is not installed
locale -a
install it where XX is your language
sudo apt-get install language-pack-XX
For example:
sudo apt-get install language-pack-he
add the following text to /etc/apache2/envvrs
export LANG='he_IL.UTF-8'
export LC_ALL='he_IL.UTF-8'
Than you would hopefully not get python errors on from apache like:
print (js) UnicodeEncodeError: 'ascii' codec can't encode characters in position 41-45: ordinal not in range(128)
Also in apache try to make utf the default encoding as explained here:
How to change the default encoding to UTF-8 for Apache?
Do it early because apache errors can be pain to debug and you can mistakenly think it's from python which possibly isn't the case in that situation
Source: Stackoverflow.com