Access nested dictionary items via a list of keys

Question

I have a complex dictionary structure which I would like to access via a list of keys to address the correct item   dataDict          a             r   1           s   2           t   3                 b             u   1           v                  x   1               y   2               z   3                     w   3                  maplist     a    r     or      maplist     b    v    y     I have made the following code which works but I m sure there is a better and more efficient way to do this if anyone has an idea     Get a given data from a dictionary with position provided as a list def getFromDict dataDict  mapList           for k in mapList  dataDict   dataDict k      return dataDict    Set a given data in a dictionary with position provided as a list def setInDict dataDict  mapList  value        for k in mapList  -1   dataDict   dataDict k      dataDict mapList -1     value

User · Accepted Answer

Use reduce   to traverse the dictionary  from functools import reduce    forward compatibility for Python 3 import operator  def getFromDict dataDict  mapList       return reduce operator getitem  mapList  dataDict   and reuse getFromDict to find the location to store the value for setInDict    def setInDict dataDict  mapList  value       getFromDict dataDict  mapList  -1   mapList -1     value  All but the last element in mapList is needed to find the  parent  dictionary to add the value to  then use the last element to set the value to the right key  Demo   gt  gt  gt  getFromDict dataDict    quot a quot    quot r quot    1  gt  gt  gt  getFromDict dataDict    quot b quot    quot v quot    quot y quot    2  gt  gt  gt  setInDict dataDict    quot b quot    quot v quot    quot w quot    4   gt  gt  gt  import pprint  gt  gt  gt  pprint pprint dataDict    a     r   1   s   2   t   3     b     u   1   v     w   4   x   1   y   2   z   3    w   3    Note that the Python PEP8 style guide prescribes snake case names for functions  The above works equally well for lists or a mix of dictionaries and lists  so the names should really be get by path   and set by path    from functools import reduce    forward compatibility for Python 3 import operator  def get by path root  items        quot  quot  quot Access a nested object in root by item sequence  quot  quot  quot      return reduce operator getitem  items  root   def set by path root  items  value        quot  quot  quot Set a value in a nested object in root by item sequence  quot  quot  quot      get by path root  items  -1   items -1     value  And for completion s sake  a function to delete a key  def del by path root  items        quot  quot  quot Delete a key-value in a nested object in root by item sequence  quot  quot  quot      del get by path root  items  -1   items -1

User · Answer

Instead of taking a performance hit each time you want to look up a value  how about you flatten the dictionary once then simply look up the key like b v y  def flatten mydict     new dict        for key value in mydict items        if type value     dict         dict        join  key   key    value for  key   value in flatten value  items          new dict update  dict      else        new dict key  value   return new dict  dataDict      a         r   1       s   2       t   3         b         u   1       v              x   1           y   2           z   3             w   3              flat dict   flatten dataDict  print flat dict   b w   3   b u   1   b v y   2   b v x   1   b v z   3   a r   1   a s   2   a t   3    This way you can simply look up items using flat dict  b v y   which will give you 1   And instead of traversing the dictionary on each lookup  you may be able to speed this up by flattening the dictionary and saving the output so that a lookup from cold start would mean loading up the flattened dictionary and simply performing a key value lookup with no traversal

User · Answer

If you also want the ability to work with arbitrary json including nested lists and dicts  and nicely handle invalid lookup paths  here s my solution   from functools import reduce   def get furthest s  path               Gets the furthest value along a given key path in a subscriptable structure       subscriptable  list - gt  any      param s  the subscriptable structure to examine      param path  the lookup path to follow      return  a tuple of the value at the furthest valid key  and whether the full path is valid              def step key acc  key           s   acc 0          if isinstance s  str               return  s  False          try              return  s key   acc 1           except LookupError              return  s  False       return reduce step key  path   s  True     def get val s  path       val  successful   get furthest s  path      if successful          return val     else          raise LookupError  Invalid lookup path      format path     def set val s  path  value       get val s  path  -1   path -1     value

User · Answer

Using reduce is clever  but the OP s set method may have issues if the parent keys do not pre-exist in the nested dictionary  Since this is the first SO post I saw for this subject in my google search  I would like to make it slightly better   The set method in   Setting a value in a nested python dictionary given a list of indices and value   seems more robust to missing parental keys  To copy it over   def nested set dic  keys  value       for key in keys  -1           dic   dic setdefault key          dic keys -1     value   Also  it can be convenient to have a method that traverses the key tree and get all the absolute key paths  for which I have created   def keysInDict dataDict  parent          if not isinstance dataDict  dict           return  tuple parent       else          return reduce list   add                  keysInDict v parent  k   for k v in dataDict items           One use of it is to convert the nested tree to a pandas DataFrame  using the following code  assuming that all leafs in the nested dictionary have the same depth    def dict to df dataDict       ret          for k in keysInDict dataDict           v   np array  getFromDict dataDict  k             v   pd DataFrame v          v columns   pd MultiIndex from product list k     v columns           ret append v      return reduce pd DataFrame join  ret

User · Answer

It seems more pythonic to use a for loop  See the quote from What   s New In Python 3 0   Removed reduce    Use functools reduce   if you really need it  however  99 percent of the time an explicit for loop is more readable   def nested get dic  keys           for key in keys          dic   dic key      return dic  Note that the accepted solution doesn t set non-existing nested keys  it raises KeyError   Using the approach below will create non-existing nodes instead  def nested set dic  keys  value       for key in keys  -1           dic   dic setdefault key          dic keys -1     value  The code works in both Python 2 and 3

User · Answer

Very late to the party  but posting in case this may help someone in the future  For my use case  the following function worked the best  Works to pull any data type out of dictionary  dict is the dictionary containing our value  list is a list of  steps  towards our value  def getnestedvalue dict  list        length   len list      try          for depth  key in enumerate list               if depth    length - 1                  output   dict key                  return output             dict   dict key      except  KeyError  TypeError           return None      return None

User · Answer

How about using recursive functions   To get a value   def getFromDict dataDict  maplist       first  rest   maplist 0   maplist 1        if rest             if  rest  is not empty  run the function recursively         return getFromDict dataDict first   rest      else          return dataDict first    And to set a value   def setInDict dataDict  maplist  value       first  rest   maplist 0   maplist 1        if rest          try              if not isinstance dataDict first   dict                     if the key is not a dict  then make it a dict                 dataDict first               except KeyError                if key doesn t exist  create one             dataDict first                setInDict dataDict first   rest  value      else          dataDict first    value

User · Answer

Extending  DomTomCat and others  approach  these functional  ie  return modified data via deepcopy without affecting the input  setter and mapper works for nested dict and list     setter   def set at path data0  keys  value       data   deepcopy data0      if len keys  gt 1          if isinstance data dict               return  k  set by path v keys 1   value  if k  keys 0  else v  for k v in data items            if isinstance data list               return  set by path x 1  keys 1   value  if x 0   keys 0  else x 1  for x in enumerate data       else          data keys -1   value         return data   mapper   def map at path data0  keys  f       data   deepcopy data0      if len keys  gt 1          if isinstance data dict               return  k  map at path v keys 1   f  if k  keys 0  else v  for k v in data items            if isinstance data list               return  map at path x 1  keys 1   f  if x 0   keys 0  else x 1  for x in enumerate data       else          data keys -1   f data keys -1            return data

User · Answer

You can make use of the eval function in python   def nested parse nest  map list       nestq    nest             join map list             return eval nestq      builtins    None     nest  nest     Explanation  For your example query  maplist     b    v    y    nestq will be  nest  b    v    y    where nest is the nested dictionary   The eval builtin function executes the given string  However  it is important to be careful about possible vulnerabilities that arise from use of eval function  Discussion can be found here    https   nedbatchelder com blog 201206 eval really is dangerous html https   www journaldev com 22504 python-eval-function   In the nested parse   function  I have made sure that no   builtins   globals are available and only local variable that is available is the nest dictionary

User · Answer

a method for concatenating strings   def get sub object from path dict name  map list       for i in map list           string       s      i         dict name     string     value   eval dict name      return value  Sample   dict     new    person    time     for    one    map list     time    for   print get sub object from path   dict  map list   Output   one

User · Answer

This library may be helpful  https   github com akesterson dpath-python     A python library for accessing and searching dictionaries via    slashed paths ala xpath      Basically it lets you glob over a dictionary as if it were a   filesystem

User · Answer

An alternative way if you don t want to raise errors if one of the keys is absent  so that your main code can run without interruption    def get value self your dict  keys       curr dict    your dict     for k in keys          v   curr dict get k None          if v is None              break         if isinstance v dict               curr dict   v     return v   In this case  if any of the input keys is not present  None is returned  which can be used as a check in your main code to perform an alternative task

User · Answer

You can use pydash   import pydash as      get dataDict    b    v    y    default  Default     https   pydash readthedocs io en latest api html

User · Answer

How about check and then set dict element without processing all indexes twice   Solution   def nested yield nested  keys list               Get current nested data by send None  method  Allows change it to Value by calling send Value  next time      param nested  list or dict of lists or dicts      param keys list  list of indexes keys             if not len keys list      assign to 1st level list         if isinstance nested  list               while True                  nested      yield nested         else              raise IndexError  Only lists can take element without key         last key   keys list pop       for key in keys list          nested   nested key       while True          try              nested last key    yield nested last key          except IndexError as e              print  no index    in     format last key  nested               yield None   Example workflow   ny   nested yield nested dict  nested address  data element   ny send None  if data element        process element         else        extend update nested data     ny send new data element          ny close     Test   gt  gt  gt  cfg    Options     1  0    2  4  8 16     3  9         ny   nested yield cfg    Options  1 1 1       ny send None   8  16   gt  gt  gt  ny send  Hello     Hello    gt  gt  gt  cfg   Options     1   0     2   4   Hello       3   9      gt  gt  gt  ny close

User · Answer

It s satisfying to see these answers for having two static methods for setting  amp  getting nested attributes  These solutions are way better than using nested trees https   gist github com hrldcpr 2012250  Here s my implementation   Usage   To set nested attribute call sattr my dict  1  2  3  5  is equal to my dict 1  2  3  4  5  To get a nested attribute call gattr my dict  1  2   def gattr d   attrs               This method receives a dict and list of attributes to return the innermost value of the give dict                    try          for at in attrs              d   d at          return d     except KeyError  TypeError           return None   def sattr d   attrs               Adds  val  to dict in the hierarchy mentioned via  attrs     For ex      sattr animals   cat    leg   fingers   4  is equivalent to animals  cat    leg    fingers   4     This method creates necessary objects until it reaches the final depth     This behaviour is also known as autovivification and plenty of implementation are around     This implementation addresses the corner case of replacing existing primitives     https   gist github com hrldcpr 2012250 gistcomment-1779319             for attr in attrs  -2           if type d get attr   is not dict              d attr               d   d attr      d attrs -2     attrs -1

User · Answer

Pure Python style  without any import   def nested set element  value   keys       if type element  is not dict          raise AttributeError  nested set   expects dict as first argument        if len keys   lt  2          raise AttributeError  nested set   expects at least three arguments  not enough given          keys   keys  -1       element   element     for key in  keys           element    element key       element keys -1     value  example     foo      bar      baz    ok        keys     foo    bar   nested set example   yay    keys  print example    Output    foo     bar    yay

User · Answer

Solved this with recursion   def get d l       if len l   1  return d l 0       return get d l 0   l 1      Using your example   dataDict          a             r   1           s   2           t   3                 b             u   1           v                  x   1               y   2               z   3                     w   3             maplist1     a    r   maplist2     b    v    y   print get dataDict  maplist1     1 print get dataDict  maplist2     2

[python] Access nested dictionary items via a list of keys?

Examples related to python

Examples related to list

Examples related to dictionary