Update value of a nested dictionary of varying depth

Question

I m looking for a way to update dict dictionary1 with the contents of dict update wihout overwriting levelA  dictionary1   level1    level2    levelA  0  levelB  1    update   level1    level2    levelB  10    dictionary1 update update  print dictionary1   level1     level2     levelB   10      I know that update deletes the values in level2 because it s updating the lowest key level1   How could I tackle this  given that dictionary1 and update can have any length

User · Answer

Same solution as the accepted one  but clearer variable naming  docstring  and fixed a bug where    as a value would not override   import collections   def deep update source  overrides               Update a nested dictionary or similar mapping      Modify   source   in place              for key  value in overrides iteritems            if isinstance value  collections Mapping  and value              returned   deep update source get key       value              source key    returned         else              source key    overrides key      return source   Here are a few test cases   def test deep update        source     hello1   1      overrides     hello2   2      deep update source  overrides      assert source      hello1   1   hello2   2       source     hello    to override       overrides     hello    over       deep update source  overrides      assert source      hello    over        source     hello     value    to override    no change   1       overrides     hello     value    over        deep update source  overrides      assert source      hello     value    over    no change   1        source     hello     value    to override    no change   1       overrides     hello     value            deep update source  overrides      assert source      hello     value        no change   1        source     hello     value        no change   1       overrides     hello     value   2       deep update source  overrides      assert source      hello     value   2   no change   1     This functions is available in the charlatan package  in charlatan utils

User · Answer

FM s answer has the right general idea  i e  a recursive solution  but somewhat peculiar coding and at least one bug   I d recommend  instead   Python 2   import collections  def update d  u       for k  v in u iteritems            if isinstance v  collections Mapping               d k    update d get k       v          else              d k    v     return d   Python 3   import collections abc  def update d  u       for k  v in u items            if isinstance v  collections abc Mapping               d k    update d get k       v          else              d k    v     return d   The bug shows up when the  update  has a k  v item where v is a dict and k is not originally a key in the dictionary being updated --  FM s code  skips  this part of the update  because it performs it on an empty new dict which isn t saved or returned anywhere  just lost when the recursive call returns    My other changes are minor  there is no reason for the if else construct when  get does the same job faster and cleaner  and isinstance is best applied to abstract base classes  not concrete ones  for generality

User · Answer

Minor improvements to  Alex s answer that enables updating of dictionaries of differing depths as well as limiting the depth that the update dives into the original nested dictionary  but the updating dictionary depth is not limited   Only a few cases have been tested   def update d  u  depth -1               Recursively merge or update dict-like objects        gt  gt  gt  update   k1     k2   2      k1     k2     k3   3     k4   4         k1     k2     k3   3     k4   4               for k  v in u iteritems            if isinstance v  Mapping  and not depth    0              r   update d get k       v  depth max depth - 1  -1               d k    r         elif isinstance d  Mapping               d k    u k          else              d    k  u k       return d

User · Answer

The code below should solve the update   k1   1     k1     k2   2    issue in  Alex Martelli s answer the right way   def deepupdate original  update          Recursively update a dict       Subdict s won t be overwritten but also updated              if not isinstance original  abc Mapping           return update     for key  value in update items            if isinstance value  abc Mapping               original key    deepupdate original get key       value          else              original key    value     return original

User · Answer

you could try this  it works with lists and is pure   def update keys newd  dic  mapping     def upsingle d k v       if k in mapping        d mapping k     v     else        d k    v   for ekey  evalue in dic items        upsingle newd  ekey  evalue      if type evalue  is dict        update keys newd  evalue  mapping      if type evalue  is list        upsingle newd  ekey   update keys     i  mapping  for i in evalue     return newd

User · Answer

That s a bit to the side but do you really need nested dictionaries  Depending on the problem  sometimes flat dictionary may suffice    and look good at it    gt  gt  gt  dict1      level1   level2   levelA    0   gt  gt  gt  dict1  level1   level2   levelB     1  gt  gt  gt  update      level1   level2   levelB    10   gt  gt  gt  dict1 update update   gt  gt  gt  print dict1    level1    level2    levelB    10    level1    level2    levelA    0

User · Answer

I used the solution  Alex Martelli suggests  but it fails  TypeError  bool  object does not support item assignment  when the two dictionaries differ in data type at some level    In case at the same level the element of dictionary d is just a scalar  ie  Bool  while the element of dictionary u is still dictionary the reassignment fails as no dictionary assignment is possible into scalar  like True k     One added condition fixes that   from collections import Mapping  def update deep d  u       for k  v in u items              this condition handles the problem         if not isinstance d  Mapping               d   u         elif isinstance v  Mapping               r   update deep d get k       v              d k    r         else              d k    u k       return d

User · Answer

Alex s answer is good  but doesn t work when replacing an element such as an integer with a dictionary  such as update   foo  0    foo    bar  1     This update addresses it   import collections def update d  u       for k  v in u iteritems            if isinstance d  collections Mapping               if isinstance v  collections Mapping                   r   update d get k       v                  d k    r             else                  d k    u k          else              d    k  u k       return d  update   k1   1     k1     k2     k3   3

User · Answer

I recommend to replace    by type v    in order to propagate object type of any dict subclass stored in u but absent from d   For example  this would preserve types such as collections OrderedDict   Python 2   import collections  def update d  u       for k  v in u iteritems            if isinstance v  collections Mapping               d k    update d get k  type v      v          else              d k    v     return d   Python 3   import collections abc  def update d  u       for k  v in u items            if isinstance v  collections abc Mapping               d k    update d get k  type v      v          else              d k    v     return d

User · Answer

It could be that you stumble over a non-standard-dictionary  like me today  which has no iteritems-Attribute   In this case it s easy to interpret this type of dictionary as a standard-dictionary  E g   Python 2 7       import collections     def update orig dict  new dict           for key  val in dict new dict  iteritems                if isinstance val  collections Mapping                   tmp   update orig dict get key        val                  orig dict key    tmp             elif isinstance val  list                   orig dict key     orig dict key    val              else                  orig dict key    new dict key          return orig dict      import multiprocessing     d multiprocessing Manager   dict   sample   data        u   other   1234       x update d  u      x items     Python 3 8       def update orig dict  new dict           orig dict dict orig dict          for key  val in dict new dict  items                if isinstance val  collections abc Mapping                   tmp   update orig dict get key        val                  orig dict key    tmp             elif isinstance val  list                   orig dict key     orig dict key    val              else                  orig dict key    new dict key          return orig dict      import collections     import multiprocessing     d multiprocessing Manager   dict   sample   data        u   other   1234   deeper     very    deep         x update d  u      x items

User · Answer

I know this question is pretty old  but still posting what I do when I have to update a nested dictionary  We can use the fact that dicts are passed by reference in python Assuming that the path of the key is known and is dot separated  Forex if we have a dict named data      log config worker          version   1        root              handlers                  queue                       level    DEBUG               disable existing loggers   true        handlers              queue                  queue   null                class    myclass1 QueueHandler                       number of archived logs   15    log max size    300M     cron job dir     etc cron hourly      logs dir     var log patternex      log rotate dir     etc logrotate d       And we want to update the queue class  the path of the key would be - log config worker handlers queue class  We can use the following function to update the value   def get updated dict obj  path  value       key list   path split           for k in key list  -1           obj   obj k       obj key list -1     value  get updated dict data   log config worker handlers queue class    myclass2 QueueHandler     This would update the dictionary correctly

User · Answer

Another way of using recursion   def updateDict dict1 dict2       keys1   list dict1 keys        keys2  list dict2 keys        keys2    x for x in keys2 if x in keys1      for x in keys2          if  x in keys1   amp   type dict1 x   is dict   amp   type dict2 x   is dict               updateDict dict1 x  dict2 x           else              dict1 update  x dict2 x        return dict1

User · Answer

Yes  And another solution  My solution differs in the keys that are being checked  In all other solutions we only look at the keys in dict b  But here we look in the union of both dictionaries   Do with it as you please  def update nested dict a  dict b       set keys   set dict a keys    union set dict b keys         for k in set keys          v   dict a get k          if isinstance v  dict               new dict   dict b get k  None              if new dict                  update nested v  new dict          else              new value   dict b get k  None              if new value                  dict a k    new value

User · Answer

Just use python-benedict  I did it   it has a merge  deepupdate  utility method and many others  It works with python 2   python 3 and it is well tested  from benedict import benedict  dictionary1 benedict   level1    level2    levelA  0  levelB  1     update   level1    level2    levelB  10    dictionary1 merge update  print dictionary1     gt  gt    level1    level2    levelA  0  levelB  10     Installation  pip install python-benedict Documentation  https   github com fabiocaccamo python-benedict Note  I am the author of this project

User · Answer

This question is old  but I landed here when searching for a  quot deep merge quot  solution  The answers above inspired what follows  I ended up writing my own because there were bugs in all the versions I tested  The critical point missed was  at some arbitrary depth of the two input dicts  for some key  k  the decision tree when d k  or u k  is not a dict was faulty  Also  this solution does not require recursion  which is more symmetric with how dict update   works  and returns None  import collections def deep merge d  u       quot  quot  quot Do a deep merge of one dict into another      This will update d with values in u  but will not delete keys in d    not found in u at some arbitrary depth of d  That is  u is deeply    merged into d      Args -      d  u  dicts     Note  this is destructive to d  but not u      Returns  None     quot  quot  quot     stack     d u      while stack        d u   stack pop 0        for k v in u items             if not isinstance v  collections Mapping                 u k  is not a dict  nothing to merge  so just set it                regardless if d k   was  a dict             d k    v          else                note  u k  is a dict             if k not in d                    add new key into d                 d k    v             elif not isinstance d k   collections Mapping                     d k  is not a dict  so just set it to u k                     overriding whatever it was                 d k    v             else                    both d k  and u k  are dicts  push them on the stack                   to merge                 stack append  d k   v

User · Answer

If you want to replace a  full nested dictionary with arrays  you can use this snippet     It will replace any  old value  by  new value   It s roughly doing a depth-first rebuilding of the dictionary  It can even work with List or Str int given as input parameter of first level   def update values dict original dict  future dict  old value  new value         Recursively updates values of a nested dict by performing recursive calls      if isinstance original dict  Dict             It s a dict         tmp dict              for key  value in original dict items                tmp dict key    update values dict value  future dict  old value  new value          return tmp dict     elif isinstance original dict  List             It s a List         tmp list              for i in original dict              tmp list append update values dict i  future dict  old value  new value           return tmp list     else            It s not a dict  maybe a int  a string  etc          return original dict if original dict    old value else new value

User · Answer

Took me a little bit on this one  but thanks to  Alex s post  he filled in the gap I was missing  However  I came across an issue if a value within the recursive dict happens to be a list  so I thought I d share  and extend his answer   import collections  def update orig dict  new dict       for key  val in new dict iteritems            if isinstance val  collections Mapping               tmp   update orig dict get key        val              orig dict key    tmp         elif isinstance val  list               orig dict key     orig dict get key        val          else              orig dict key    new dict key      return orig dict

User · Answer

In neither of these answers the authors seem to understand the concept of updating an object stored in a dictionary nor even of iterating over dictionary items  as opposed to keys   So I had to write one which doesn t make pointless tautological dictionary stores and retrievals  The dicts are assumed to store other dicts or simple types   def update nested dict d  other       for k  v in other items            if isinstance v  collections Mapping               d v   d get k              if isinstance d v  collections Mapping                   update nested dict d v  v              else                  d k    v copy           else              d k    v   Or even simpler one working with any type   def update nested dict d  other       for k  v in other items            d v   d get k          if isinstance v  collections Mapping  and isinstance d v  collections Mapping               update nested dict d v  v          else              d k    deepcopy v    or d k    v if you know what you re doing

User · Answer

Here s an Immutable version of recursive dictionary merge in case anybody needs it  Based upon  Alex Martelli s answer  Python 3 x  import collections from copy import deepcopy   def merge dict1  dict2           Return a new dictionary by merging two dictionaries recursively           result   deepcopy dict1       for key  value in dict2 items            if isinstance value  collections Mapping               result key    merge result get key       value          else              result key    deepcopy dict2 key        return result  Python 2 x  import collections from copy import deepcopy   def merge dict1  dict2           Return a new dictionary by merging two dictionaries recursively           result   deepcopy dict1       for key  value in dict2 iteritems            if isinstance value  collections Mapping               result key    merge result get key       value          else              result key    deepcopy dict2 key        return result

User · Answer

def update value  nvalue       if not isinstance value  dict  or not isinstance nvalue  dict           return nvalue     for k  v in nvalue items            value setdefault k  dict            if isinstance v  dict               v   update value k   v          value k    v     return value   use dict or collections Mapping

User · Answer

If you want a one-liner      dictionary1      level1     dictionary1  level1        level2     dictionary1  level1    level2        levelB  10

User · Answer

I made a simple function  in which you give the key  the new value and the dictionary as input  and it recursively updates it with the value  def update key value dictionary       if key in dictionary keys            dictionary key    value         return     dic aux          for val aux in dictionary values            if isinstance val aux dict               dic aux append val aux      for i in dic aux          update key value i      for  key2 val aux2  in dictionary items            if isinstance val aux2 dict               dictionary key2    val aux2  dictionary1   level1    level2    levelA  0  levelB  1    update  levelB  10 dictionary1  print dictionary1    output    level1     level2     levelA   0   levelB   10     Hope it answers

User · Answer

Update to  Alex Martelli s answer to fix a bug in his code to make the solution more robust   def update dict d  u       for k  v in u items            if isinstance v  collections Mapping               default   v copy               default clear               r   update dict d get k  default   v              d k    r         else              d k    v     return d   The key is that we often want to create the same type at recursion  so here we use v copy   clear   but not     And this is especially useful if the dict here is of type collections defaultdict which can have different kinds of default factorys   Also notice that the u iteritems   has been changed to u items   in Python3

User · Answer

a new Q how to By a keys chain  dictionary1   level1    level2    levelA  0  levelB  1    anotherLevel1    anotherLevel2    anotherLevelA  0  anotherLevelB  1    update   anotherLevel1    anotherLevel2  1014   dictionary1 update update  print dictionary1   level1    level2    levelA  0  levelB  1    anotherLevel1    anotherLevel2  1014

User · Answer

Thanks to hobs for his comment on Alex s answer  Indeed update   k1   1     k1     k2   2    will cause TypeError   int  object does not support item assignment  We should check the types of the input values at the beginning of the function  So  I suggest the following function  which should solve this  and other  problem  Python 3  from collections abc import Mapping   def deep update d1  d2       if all  isinstance d  Mapping  for d in  d1  d2             for k  v in d2 items                d1 k    deep update d1 get k   v          return d1     return d2

[python] Update value of a nested dictionary of varying depth

Examples related to python