I'm used to bringing data in and out of Python using CSV files, but there are obvious challenges to this. Are there simple ways to store a dictionary (or sets of dictionaries) in a JSON or pickle file?
For example:
data = {}
data ['key1'] = "keyinfo"
data ['key2'] = "keyinfo2"
I would like to know both how to save this, and then how to load it back in.
This question is related to
python
json
dictionary
save
pickle
Pickle save:
try:
import cPickle as pickle
except ImportError: # Python 3.x
import pickle
with open('data.p', 'wb') as fp:
pickle.dump(data, fp, protocol=pickle.HIGHEST_PROTOCOL)
See the pickle module documentation for additional information regarding the protocol
argument.
Pickle load:
with open('data.p', 'rb') as fp:
data = pickle.load(fp)
JSON save:
import json
with open('data.json', 'w') as fp:
json.dump(data, fp)
Supply extra arguments, like sort_keys
or indent
, to get a pretty result. The argument sort_keys will sort the keys alphabetically and indent will indent your data structure with indent=N
spaces.
json.dump(data, fp, sort_keys=True, indent=4)
JSON load:
with open('data.json', 'r') as fp:
data = json.load(fp)
Minimal example, writing directly to a file:
import json
json.dump(data, open(filename, 'wb'))
data = json.load(open(filename))
or safely opening / closing:
import json
with open(filename, 'wb') as outfile:
json.dump(data, outfile)
with open(filename) as infile:
data = json.load(infile)
If you want to save it in a string instead of a file:
import json
json_str = json.dumps(data)
data = json.loads(json_str)
My use case was to save multiple JSON objects to a file and marty's answer helped me somewhat. But to serve my use case, the answer was not complete as it would overwrite the old data every time a new entry was saved.
To save multiple entries in a file, one must check for the old content (i.e., read before write). A typical file holding JSON data will either have a list
or an object
as root. So I considered that my JSON file always has a list of objects
and every time I add data to it, I simply load the list first, append my new data in it, and dump it back to a writable-only instance of file (w
):
def saveJson(url,sc): # This function writes the two values to the file
newdata = {'url':url,'sc':sc}
json_path = "db/file.json"
old_list= []
with open(json_path) as myfile: # Read the contents first
old_list = json.load(myfile)
old_list.append(newdata)
with open(json_path,"w") as myfile: # Overwrite the whole content
json.dump(old_list, myfile, sort_keys=True, indent=4)
return "success"
The new JSON file will look something like this:
[
{
"sc": "a11",
"url": "www.google.com"
},
{
"sc": "a12",
"url": "www.google.com"
},
{
"sc": "a13",
"url": "www.google.com"
}
]
NOTE: It is essential to have a file named file.json
with []
as initial data for this approach to work
PS: not related to original question, but this approach could also be further improved by first checking if our entry already exists (based on one or multiple keys) and only then append and save the data.
For completeness, we should include ConfigParser and configparser which are part of the standard library in Python 2 and 3, respectively. This module reads and writes to a config/ini file and (at least in Python 3) behaves in a lot of ways like a dictionary. It has the added benefit that you can store multiple dictionaries into separate sections of your config/ini file and recall them. Sweet!
Python 2.7.x example.
import ConfigParser
config = ConfigParser.ConfigParser()
dict1 = {'key1':'keyinfo', 'key2':'keyinfo2'}
dict2 = {'k1':'hot', 'k2':'cross', 'k3':'buns'}
dict3 = {'x':1, 'y':2, 'z':3}
# Make each dictionary a separate section in the configuration
config.add_section('dict1')
for key in dict1.keys():
config.set('dict1', key, dict1[key])
config.add_section('dict2')
for key in dict2.keys():
config.set('dict2', key, dict2[key])
config.add_section('dict3')
for key in dict3.keys():
config.set('dict3', key, dict3[key])
# Save the configuration to a file
f = open('config.ini', 'w')
config.write(f)
f.close()
# Read the configuration from a file
config2 = ConfigParser.ConfigParser()
config2.read('config.ini')
dictA = {}
for item in config2.items('dict1'):
dictA[item[0]] = item[1]
dictB = {}
for item in config2.items('dict2'):
dictB[item[0]] = item[1]
dictC = {}
for item in config2.items('dict3'):
dictC[item[0]] = item[1]
print(dictA)
print(dictB)
print(dictC)
Python 3.X example.
import configparser
config = configparser.ConfigParser()
dict1 = {'key1':'keyinfo', 'key2':'keyinfo2'}
dict2 = {'k1':'hot', 'k2':'cross', 'k3':'buns'}
dict3 = {'x':1, 'y':2, 'z':3}
# Make each dictionary a separate section in the configuration
config['dict1'] = dict1
config['dict2'] = dict2
config['dict3'] = dict3
# Save the configuration to a file
f = open('config.ini', 'w')
config.write(f)
f.close()
# Read the configuration from a file
config2 = configparser.ConfigParser()
config2.read('config.ini')
# ConfigParser objects are a lot like dictionaries, but if you really
# want a dictionary you can ask it to convert a section to a dictionary
dictA = dict(config2['dict1'] )
dictB = dict(config2['dict2'] )
dictC = dict(config2['dict3'])
print(dictA)
print(dictB)
print(dictC)
{'key2': 'keyinfo2', 'key1': 'keyinfo'}
{'k1': 'hot', 'k2': 'cross', 'k3': 'buns'}
{'z': '3', 'y': '2', 'x': '1'}
[dict1]
key2 = keyinfo2
key1 = keyinfo
[dict2]
k1 = hot
k2 = cross
k3 = buns
[dict3]
z = 3
y = 2
x = 1
To write to a file:
import json
myfile.write(json.dumps(mydict))
To read from a file:
import json
mydict = json.loads(myfile.read())
myfile
is the file object for the file that you stored the dict in.
Also see the speeded-up package ujson:
import ujson
with open('data.json', 'wb') as fp:
ujson.dump(data, fp)
If you're after serialization, but won't need the data in other programs, I strongly recommend the shelve
module. Think of it as a persistent dictionary.
myData = shelve.open('/path/to/file')
# Check for values.
keyVar in myData
# Set values
myData[anotherKey] = someValue
# Save the data for future use.
myData.close()
If you want an alternative to pickle
or json
, you can use klepto
.
>>> init = {'y': 2, 'x': 1, 'z': 3}
>>> import klepto
>>> cache = klepto.archives.file_archive('memo', init, serialized=False)
>>> cache
{'y': 2, 'x': 1, 'z': 3}
>>>
>>> # dump dictionary to the file 'memo.py'
>>> cache.dump()
>>>
>>> # import from 'memo.py'
>>> from memo import memo
>>> print memo
{'y': 2, 'x': 1, 'z': 3}
With klepto
, if you had used serialized=True
, the dictionary would have been written to memo.pkl
as a pickled dictionary instead of with clear text.
You can get klepto
here: https://github.com/uqfoundation/klepto
dill
is probably a better choice for pickling then pickle
itself, as dill
can serialize almost anything in python. klepto
also can use dill
.
You can get dill
here: https://github.com/uqfoundation/dill
The additional mumbo-jumbo on the first few lines are because klepto
can be configured to store dictionaries to a file, to a directory context, or to a SQL database. The API is the same for whatever you choose as the backend archive. It gives you an "archivable" dictionary with which you can use load
and dump
to interact with the archive.
If save to a JSON file, the best and easiest way of doing this is:
import json
with open("file.json", "wb") as f:
f.write(json.dumps(dict).encode("utf-8"))
Source: Stackoverflow.com