How to convert an xml string to a dictionary

Question

I have a program that reads an xml document from a socket  I have the xml document stored in a string which I would like to convert directly to a Python dictionary  the same way it is done in Django s simplejson library   Take as an example   str    lt  xml version  1 0    gt  lt person gt  lt name gt john lt  name gt  lt age gt 20 lt  age gt  lt  person  dic xml   convert to dic str    Then dic xml would look like   person       name     john    age    20

User · Answer

The following XML-to-Python-dict snippet parses entities as well as attributes following this XML-to-JSON "specification". It is the most general solution handling all cases of XML.

from collections import defaultdict

def etree_to_dict(t):
    d = {t.tag: {} if t.attrib else None}
    children = list(t)
    if children:
        dd = defaultdict(list)
        for dc in map(etree_to_dict, children):
            for k, v in dc.items():
                dd[k].append(v)
        d = {t.tag: {k:v[0] if len(v) == 1 else v for k, v in dd.items()}}
    if t.attrib:
        d[t.tag].update(('@' + k, v) for k, v in t.attrib.items())
    if t.text:
        text = t.text.strip()
        if children or t.attrib:
            if text:
              d[t.tag]['#text'] = text
        else:
            d[t.tag] = text
    return d

It is used:

from xml.etree import cElementTree as ET
e = ET.XML('''
<root>
  <e />
  <e>text</e>
  <e name="value" />
  <e name="value">text</e>
  <e> <a>text</a> <b>text</b> </e>
  <e> <a>text</a> <a>text</a> </e>
  <e> text <a>text</a> </e>
</root>
''')

from pprint import pprint
pprint(etree_to_dict(e))

The output of this example (as per above-linked "specification") should be:

{'root': {'e': [None,
                'text',
                {'@name': 'value'},
                {'#text': 'text', '@name': 'value'},
                {'a': 'text', 'b': 'text'},
                {'a': ['text', 'text']},
                {'#text': 'text', 'a': 'text'}]}}

Not necessarily pretty, but it is unambiguous, and simpler XML inputs result in simpler JSON. :)

Update

If you want to do the reverse, emit an XML string from a JSON/dict, you can use:

try:
  basestring
except NameError:  # python3
  basestring = str

def dict_to_etree(d):
    def _to_etree(d, root):
        if not d:
            pass
        elif isinstance(d, basestring):
            root.text = d
        elif isinstance(d, dict):
            for k,v in d.items():
                assert isinstance(k, basestring)
                if k.startswith('#'):
                    assert k == '#text' and isinstance(v, basestring)
                    root.text = v
                elif k.startswith('@'):
                    assert isinstance(v, basestring)
                    root.set(k[1:], v)
                elif isinstance(v, list):
                    for e in v:
                        _to_etree(e, ET.SubElement(root, k))
                else:
                    _to_etree(v, ET.SubElement(root, k))
        else:
            raise TypeError('invalid type: ' + str(type(d)))
    assert isinstance(d, dict) and len(d) == 1
    tag, body = next(iter(d.items()))
    node = ET.Element(tag)
    _to_etree(body, node)
    return ET.tostring(node)

pprint(dict_to_etree(d))

User · Answer

The most recent versions of the PicklingTools libraries  1 3 0 and 1 3 1  support tools for converting from XML to a Python dict   The download is available here   PicklingTools 1 3 1  There is quite a bit of documentation for the converters here  the documentation describes in detail all of the decisions and issues that will arise when converting between XML and Python dictionaries  there are a number of edge cases  attributes  lists  anonymous lists  anonymous dicts  eval  etc  that most converters don t handle    In general  though  the converters are easy to use   If an  example xml  contains    lt top gt     lt a gt 1 lt  a gt     lt b gt 2 2 lt  b gt     lt c gt three lt  c gt   lt  top gt    Then to convert it to a dictionary    gt  gt  gt  from xmlloader import    gt  gt  gt  example   file  example xml    r       A document containing XML  gt  gt  gt  xl   StreamXMLLoader example  0        0   all defaults on operation  gt  gt  gt  result   xl expect XML    gt  gt  gt  print result   top     a    1    c    three    b    2 2      There are tools for converting in both C   and Python  the C   and Python do indentical conversion  but the C   is about 60x faster

User · Answer

xmltodict  full disclosure  I wrote it  does exactly that   xmltodict parse      lt  xml version  1 0    gt   lt person gt     lt name gt john lt  name gt     lt age gt 20 lt  age gt   lt  person gt         u person    u age   u 20   u name   u john

User · Answer

The easiest to use XML parser for Python is ElementTree  as of 2 5x and above it is in the standard library xml etree ElementTree   I don t think there is anything that does exactly what you want out of the box  It would be pretty trivial to write something to do what you want using ElementTree  but why convert to a dictionary  and why not just use ElementTree directly

User · Answer

def xml to dict node       u          param node lxml node      return  dict               return   tag   node tag   text   node text   attrib   node attrib   children    child tag  xml to dict child  for child in node

User · Answer

I have modified one of the answers to my taste and to work with multiple values with the same tag for example consider the following xml code saved in XML xml file        lt A gt           lt B gt               lt BB gt inAB lt  BB gt               lt C gt                   lt D gt                       lt E gt                          inABCDE                      lt  E gt                       lt E gt value2 lt  E gt                       lt E gt value3 lt  E gt                   lt  D gt                   lt inCout-ofD gt 123 lt  inCout-ofD gt               lt  C gt           lt  B gt           lt B gt abc lt  B gt           lt F gt F lt  F gt       lt  A gt    and in python   import xml etree ElementTree as ET     class XMLToDictionary dict       def   init   self  parentElement           self parentElement   parentElement         for child in list parentElement               child text   child text if  child text    None  else                  if len child     0                  self update self  addToDict key  child tag  value   child text strip    dict   self               else                  innerChild   XMLToDictionary parentElement child                  self update self  addToDict key innerChild parentElement tag  value innerChild  dict self        def getDict self           return  self parentElement tag  self       class  addToDict dict           def   init   self  key  value  dict               if not key in dict                  self update  key  value               else                  identical   dict key  if type dict key      list else  dict key                   self update  key  identical    value      tree   ET parse    XML xml   root   tree getroot   parseredDict   XMLToDictionary root  getDict   print parseredDict    the output is     A     B      BB    inAB    C     D     E     inABCDE    value2    value3      inCout-ofD    123      abc     F    F

User · Answer

I have a recursive method to get a dictionary from a lxml element      def recursive dict element           return  element tag split      1                   dict map recursive dict  element getchildren                            element attrib

User · Answer

I wrote a simple recursive function to do the job  from xml etree import ElementTree root   ElementTree XML xml to convert   def xml to dict recursive root        if len root getchildren       0          return  root tag root text      else          return  root tag list map xml to dict recursive  root getchildren

User · Answer

From  K3---rnc response  the best for me  I ve added a small modifications to get an OrderedDict from an XML text  some times order matters    def etree to ordereddict t   d   OrderedDict   d t tag    OrderedDict   if t attrib else None children   list t  if children      dd   OrderedDict       for dc in map etree to ordereddict  children           for k  v in dc iteritems                if k not in dd                  dd k    list               dd k  append v      d   OrderedDict       d t tag    OrderedDict       for k  v in dd iteritems            if len v     1              d t tag  k    v 0          else              d t tag  k    v if t attrib      d t tag  update        k  v  for k  v in t attrib iteritems    if t text      text   t text strip       if children or t attrib          if text              d t tag    text     text     else          d t tag    text return d   Following  K3---rnc example  you can use it   from xml etree import cElementTree as ET e   ET XML      lt root gt     lt e   gt     lt e gt text lt  e gt     lt e name  value    gt     lt e name  value  gt text lt  e gt     lt e gt   lt a gt text lt  a gt   lt b gt text lt  b gt   lt  e gt     lt e gt   lt a gt text lt  a gt   lt a gt text lt  a gt   lt  e gt     lt e gt  text  lt a gt text lt  a gt   lt  e gt   lt  root gt        from pprint import pprint pprint etree to ordereddict e     Hope it helps

User · Answer

dibrovsd  Solution will not work if the xml have more than one tag with same name  On your line of thought  I have modified the code a bit and written it for general node instead of root   from collections import defaultdict def xml2dict node       d  count   defaultdict list   1     for i in node          d i tag         str count    text     i findtext      0          d i tag         str count    attrib     i attrib   attrib gives the list         d i tag         str count    children     xml2dict i    it gives dict      return d

User · Answer

This is a great module that someone created  I ve used it several times   http   code activestate com recipes 410469-xml-as-dictionary   Here is the code from the website just in case the link goes bad    from xml etree import cElementTree as ElementTree  class XmlListConfig list       def   init   self  aList           for element in aList              if element                    treat like dict                 if len element     1 or element 0  tag    element 1  tag                      self append XmlDictConfig element                     treat like list                 elif element 0  tag    element 1  tag                      self append XmlListConfig element               elif element text                  text   element text strip                   if text                      self append text    class XmlDictConfig dict               Example usage        gt  gt  gt  tree   ElementTree parse  your file xml        gt  gt  gt  root   tree getroot        gt  gt  gt  xmldict   XmlDictConfig root       Or  if you want to use an XML string        gt  gt  gt  root   ElementTree XML xml string       gt  gt  gt  xmldict   XmlDictConfig root       And then use xmldict for what it is    a dict              def   init   self  parent element           if parent element items                self update dict parent element items             for element in parent element              if element                    treat like dict - we assume that if the first two tags                   in a series are different  then they are all different                  if len element     1 or element 0  tag    element 1  tag                      aDict   XmlDictConfig element                    treat like list - we assume that if the first two tags                   in a series are the same  then the rest are the same                  else                        here  we put the list in dictionary  the key is the                       tag name the list elements all share in common  and                       the value is the list itself                      aDict    element 0  tag  XmlListConfig element                     if the tag has attributes  add those to the dict                 if element items                        aDict update dict element items                     self update  element tag  aDict                 this assumes that if you ve got an attribute in a tag                you won t be having any text  This may or may not be a                good idea -- time will tell  It works for the way we are               currently doing XML configuration files                elif element items                    self update  element tag  dict element items                    finally  if there are no child tags and no attributes  extract               the text             else                  self update  element tag  element text     Example usage   tree   ElementTree parse  your file xml   root   tree getroot   xmldict   XmlDictConfig root      Or  if you want to use an XML string   root   ElementTree XML xml string  xmldict   XmlDictConfig root

User · Answer

This lightweight version  while not configurable  is pretty easy to tailor as needed  and works in old pythons   Also it is rigid - meaning the results are the same regardless of the existence of attributes   import xml etree ElementTree as ET  from copy import copy  def dictify r root True       if root          return  r tag   dictify r  False       d copy r attrib      if r text          d   text   r text     for x in r findall                 if x tag not in d              d x tag             d x tag  append dictify x False       return d   So   root   ET fromstring   lt erik gt  lt a x  1  gt v lt  a gt  lt a y  2  gt w lt  a gt  lt  erik gt     dictify root    Results in     erik     a      x    1     text    v      y    2     text    w

User · Answer

The code from http   code activestate com recipes 410469-xml-as-dictionary  works well  but if there are multiple elements that are the same at a given place in the hierarchy it just overrides them   I added a shim between that looks to see if the element already exists before self update     If so  pops the existing entry and creates a lists out of the existing and the new   Any subsequent duplicates are added to the list   Not sure if this can be handled more gracefully  but it works   import xml etree ElementTree as ElementTree  class XmlDictConfig dict       def   init   self  parent element           if parent element items                self updateShim dict parent element items             for element in parent element              if len element                   aDict   XmlDictConfig element                  if element items                        aDict updateShim dict element items                     self updateShim  element tag  aDict               elif element items                    self updateShim  element tag  dict element items                  else                  self updateShim  element tag  element text strip          def updateShim  self  aDict            for key in aDict keys                if key in self                  value   self pop key                  if type value  is not list                      listOfDicts                          listOfDicts append value                      listOfDicts append aDict key                       self update  key  listOfDicts                    else                      value append aDict key                       self update  key  value               else                  self update aDict

User · Answer

You can do this quite easily with lxml  First install it    sudo  pip install lxml   Here is a recursive function I wrote that does the heavy lifting for you   from lxml import objectify as xml objectify   def xml to dict xml str           Convert xml to dict  using lxml v3 4 2 xml processing library         def xml to dict recursion xml object           dict object   xml object   dict           if not dict object              return xml object         for key  value in dict object items                dict object key    xml to dict recursion value          return dict object     return xml to dict recursion xml objectify fromstring xml str    xml string       lt  xml version  1 0  encoding  UTF-8   gt  lt Response gt  lt NewOrderResp gt   lt IndustryType gt Test lt  IndustryType gt  lt SomeData gt  lt SomeNestedData1 gt 1234 lt  SomeNestedData1 gt   lt SomeNestedData2 gt 3455 lt  SomeNestedData2 gt  lt  SomeData gt  lt  NewOrderResp gt  lt  Response gt      print xml to dict xml string    The below variant preserves the parent key   element   def xml to dict xml str           Convert xml to dict  using lxml v3 4 2 xml processing library  see http   lxml de          def xml to dict recursion xml object           dict object   xml object   dict           if not dict object     if empty dict returned             return xml object         for key  value in dict object items                dict object key    xml to dict recursion value          return dict object     xml obj   objectify fromstring xml str      return  xml obj tag  xml to dict recursion xml obj     If you want to only return a subtree and convert it to dict  you can use Element find   to get the subtree and then convert it    xml obj find           lxml objectify ObjectifiedElement instance   See the lxml docs here  I hope this helps

User · Answer

Here s a link to an ActiveState solution - and the code in case it disappears again                                                      xmlreader py                                                     from xml dom minidom import parse   class NotTextNodeError      pass   def getTextFromNode node               scans through all children of node and gathers the     text  if node has non-text child-nodes  then     NotTextNodeError is raised              t          for n in node childNodes      if n nodeType    n TEXT NODE          t    n nodeValue     else          raise NotTextNodeError     return t   def nodeToDic node               nodeToDic   scans through the children of node and makes a     dictionary from the content      three cases are differentiated      - if the node contains no other nodes  it is a text-node     and  nodeName text  is merged into the dictionary      - if the node has the attribute  method  set to  true       then it s children will be appended to a list and this     list is merged to the dictionary in the form   nodeName list       - else  nodeToDic   will call itself recursively on     the nodes children  merging  nodeName nodeToDic    to     the dictionary               dic           for n in node childNodes      if n nodeType    n ELEMENT NODE          continue     if n getAttribute  multiple       true             node with multiple children            put them in a list         l              for c in n childNodes              if c nodeType    n ELEMENT NODE              continue         l append nodeToDic c               dic update  n nodeName l           continue      try          text   getTextFromNode n      except NotTextNodeError                 normal  node             dic update  n nodeName nodeToDic n                continue            text node         dic update  n nodeName text       continue     return dic   def readConfig filename       dom   parse filename      return nodeToDic dom       def test        dic   readConfig  sample xml        print dic  Config    Name       print     for item in dic  Config    Items        print  Item s Name    item  Name       print  Item s Value    item  Value    test                                                         sample xml                                                      lt  xml version  1 0  encoding  UTF-8   gt    lt Config gt       lt Name gt My Config File lt  Name gt        lt Items multiple  true  gt       lt Item gt           lt Name gt First Item lt  Name gt           lt Value gt Value 1 lt  Value gt       lt  Item gt       lt Item gt           lt Name gt Second Item lt  Name gt           lt Value gt Value 2 lt  Value gt       lt  Item gt       lt  Items gt    lt  Config gt                                                        output                                                     My Config File  Item s Name  First Item Item s Value  Value 1 Item s Name  Second Item Item s Value  Value 2

User · Answer

At one point I had to parse and write XML that only consisted of elements without attributes so a 1 1 mapping from XML to dict was possible easily  This is what I came up with in case someone else also doesnt need attributes   def xmltodict element       if not isinstance element  ElementTree Element           raise ValueError  must pass xml etree ElementTree Element object        def xmltodict handler parent element           result   dict           for element in parent element              if len element                   obj   xmltodict handler element              else                  obj   element text              if result get element tag                   if hasattr result element tag    append                        result element tag  append obj                  else                      result element tag     result element tag   obj              else                  result element tag    obj         return result      return  element tag  xmltodict handler element     def dicttoxml element       if not isinstance element  dict           raise ValueError  must pass dict type       if len element     1          raise ValueError  dict must have exactly one root key        def dicttoxml handler result  key  value           if isinstance value  list               for e in value                  dicttoxml handler result  key  e          elif isinstance value  basestring               elem   ElementTree Element key              elem text   value             result append elem          elif isinstance value  int  or isinstance value  float               elem   ElementTree Element key              elem text   str value              result append elem          elif value is None              result append ElementTree Element key           else              res   ElementTree Element key              for k  v in value items                    dicttoxml handler res  k  v              result append res       result   ElementTree Element element keys   0       for key  value in element element keys   0   items            dicttoxml handler result  key  value      return result  def xmlfiletodict filename       return xmltodict ElementTree parse filename  getroot     def dicttoxmlfile element  filename       ElementTree ElementTree dicttoxml element   write filename   def xmlstringtodict xmlstring       return xmltodict ElementTree fromstring xmlstring  getroot     def dicttoxmlstring element       return ElementTree tostring dicttoxml element

User · Answer

Disclaimer  This modified XML parser was inspired by Adam Clark  The original XML parser works for most of simple cases  However  it didn t work for some complicated XML files  I debugged the code line by line and finally fixed some issues  If you find some bugs  please let me know  I am glad to fix it   class XmlDictConfig dict                    Note  need to add a root into if no exising         Example usage       gt  gt  gt  tree   ElementTree parse  your file xml        gt  gt  gt  root   tree getroot        gt  gt  gt  xmldict   XmlDictConfig root      Or  if you want to use an XML string       gt  gt  gt  root   ElementTree XML xml string       gt  gt  gt  xmldict   XmlDictConfig root      And then use xmldict for what it is    a dict              def   init   self  parent element           if parent element items                self updateShim  dict parent element items              for element in parent element              if len element                   aDict   XmlDictConfig element                  if element items                    aDict updateShim dict element items                     self updateShim  element tag  aDict               elif element items         items   is specialy for attribtes                 elementattrib  element items                   if element text                                 elementattrib append  element tag element text          add tag text if there exist                 self updateShim  element tag  dict elementattrib                else                  self updateShim  element tag  element text        def updateShim  self  aDict            for key in aDict keys        keys   includes tag and attributes             if key in self                  value   self pop key                  if type value  is not list                      listOfDicts                          listOfDicts append value                      listOfDicts append aDict key                       self update  key  listOfDicts                   else                      value append aDict key                       self update  key  value               else                  self update  key aDict key       it was self update aDict

[python] How to convert an xml string to a dictionary?

Update

Examples related to python

Examples related to xml

Examples related to json

Examples related to dictionary

Examples related to xml-deserialization