Storing Python dictionaries

Question

I m used to bringing data in and out of Python using CSV files  but there are obvious challenges to this  Are there simple ways to store a dictionary  or sets of dictionaries  in a JSON or pickle file  For example  data      data   key1      quot keyinfo quot  data   key2      quot keyinfo2 quot   I would like to know both how to save this  and then how to load it back in

User · Answer

Minimal example  writing directly to a file   import json json dump data  open filename   wb    data   json load open filename     or safely opening   closing   import json with open filename   wb   as outfile      json dump data  outfile  with open filename  as infile      data   json load infile    If you want to save it in a string instead of a file   import json json str   json dumps data  data   json loads json str

User · Answer

If you re after serialization  but won t need the data in other programs  I strongly recommend the shelve module  Think of it as a persistent dictionary  myData   shelve open   path to file      Check for values  keyVar in myData    Set values myData anotherKey    someValue    Save the data for future use  myData close

User · Answer

If save to a JSON file  the best and easiest way of doing this is  import json with open  quot file json quot    quot wb quot   as f      f write json dumps dict  encode  quot utf-8 quot

User · Answer

If you want an alternative to pickle or json  you can use klepto    gt  gt  gt  init     y   2   x   1   z   3   gt  gt  gt  import klepto  gt  gt  gt  cache   klepto archives file archive  memo   init  serialized False   gt  gt  gt  cache           y   2   x   1   z   3   gt  gt  gt   gt  gt  gt    dump dictionary to the file  memo py   gt  gt  gt  cache dump     gt  gt  gt    gt  gt  gt    import from  memo py   gt  gt  gt  from memo import memo  gt  gt  gt  print memo   y   2   x   1   z   3    With klepto  if you had used serialized True  the dictionary would have been written to memo pkl as a pickled dictionary instead of with clear text   You can get klepto here  https   github com uqfoundation klepto  dill is probably a better choice for pickling then pickle itself  as dill can serialize almost anything in python   klepto also can use dill   You can get dill here  https   github com uqfoundation dill  The additional mumbo-jumbo on the first few lines are because klepto can be configured to store dictionaries to a file  to a directory context  or to a SQL database   The API is the same for whatever you choose as the backend archive   It gives you an  archivable  dictionary with which you can use load and dump to interact with the archive

User · Answer

For completeness  we should include ConfigParser and configparser which are part of the standard library in Python 2 and 3  respectively  This module reads and writes to a config ini file and  at least in Python 3  behaves in a lot of ways like a dictionary  It has the added benefit that you can store multiple dictionaries into separate sections of your config ini file and recall them  Sweet  Python 2 7 x example  import ConfigParser  config   ConfigParser ConfigParser    dict1     key1   keyinfo    key2   keyinfo2   dict2     k1   hot    k2   cross    k3   buns   dict3     x  1   y  2   z  3     Make each dictionary a separate section in the configuration config add section  dict1   for key in dict1 keys        config set  dict1   key  dict1 key       config add section  dict2   for key in dict2 keys        config set  dict2   key  dict2 key    config add section  dict3   for key in dict3 keys        config set  dict3   key  dict3 key      Save the configuration to a file f   open  config ini    w   config write f  f close      Read the configuration from a file config2   ConfigParser ConfigParser   config2 read  config ini    dictA      for item in config2 items  dict1        dictA item 0     item 1   dictB      for item in config2 items  dict2        dictB item 0     item 1   dictC      for item in config2 items  dict3        dictC item 0     item 1   print dictA  print dictB  print dictC   Python 3 X example  import configparser  config   configparser ConfigParser    dict1     key1   keyinfo    key2   keyinfo2   dict2     k1   hot    k2   cross    k3   buns   dict3     x  1   y  2   z  3     Make each dictionary a separate section in the configuration config  dict1     dict1 config  dict2     dict2 config  dict3     dict3    Save the configuration to a file f   open  config ini    w   config write f  f close      Read the configuration from a file config2   configparser ConfigParser   config2 read  config ini      ConfigParser objects are a lot like dictionaries  but if you really   want a dictionary you can ask it to convert a section to a dictionary dictA   dict config2  dict1     dictB   dict config2  dict2     dictC   dict config2  dict3     print dictA  print dictB  print dictC   Console output   key2    keyinfo2    key1    keyinfo     k1    hot    k2    cross    k3    buns     z    3    y    2    x    1    Contents of config ini  dict1  key2   keyinfo2 key1   keyinfo   dict2  k1   hot k2   cross k3   buns   dict3  z   3 y   2 x   1

User · Answer

Also see the speeded-up package ujson  import ujson  with open  data json    wb   as fp      ujson dump data  fp

User · Answer

My use case was to save multiple JSON objects to a file and marty s answer helped me somewhat  But to serve my use case  the answer was not complete as it would overwrite the old data every time a new entry was saved  To save multiple entries in a file  one must check for the old content  i e   read before write   A typical file holding JSON data will either have a list or an object as root  So I considered that my JSON file always has a list of objects and every time I add data to it  I simply load the list first  append my new data in it  and dump it back to a writable-only instance of file  w   def saveJson url sc     This function writes the two values to the file     newdata     url  url  sc  sc      json path    quot db file json quot       old list         with open json path  as myfile     Read the contents first         old list   json load myfile      old list append newdata       with open json path  quot w quot   as myfile     Overwrite the whole content         json dump old list  myfile  sort keys True  indent 4       return  quot success quot    The new JSON file will look something like this                   quot sc quot    quot a11 quot            quot url quot    quot www google com quot                        quot sc quot    quot a12 quot            quot url quot    quot www google com quot                        quot sc quot    quot a13 quot            quot url quot    quot www google com quot            NOTE  It is essential to have a file named file json with    as initial data for this approach to work PS  not related to original question  but this approach could also be further improved by first checking if our entry already exists  based on one or multiple keys  and only then append and save the data

User · Answer

Pickle save  try      import cPickle as pickle except ImportError     Python 3 x     import pickle  with open  data p    wb   as fp      pickle dump data  fp  protocol pickle HIGHEST PROTOCOL   See the pickle module documentation for additional information regarding the protocol argument  Pickle load  with open  data p    rb   as fp      data   pickle load fp    JSON save  import json  with open  data json    w   as fp      json dump data  fp   Supply extra arguments  like sort keys or indent  to get a pretty result  The argument sort keys will sort the keys alphabetically and indent will indent your data structure with indent N spaces  json dump data  fp  sort keys True  indent 4   JSON load  with open  data json    r   as fp      data   json load fp

User · Answer

To write to a file   import json myfile write json dumps mydict     To read from a file   import json mydict   json loads myfile read      myfile is the file object for the file that you stored the dict in

[python] Storing Python dictionaries

Examples related to python

Examples related to json

Examples related to dictionary

Examples related to save

Examples related to pickle