Efficient way to remove keys with empty strings from a dict

Question

I have a dict and would like to remove all the keys for which there are empty value strings   metadata    u Composite PreviewImage   u  Binary data 101973 bytes                u EXIF CFAPattern2   u      What is the best way to do this

User · Answer

Some benchmarking  1  List comprehension recreate dict In  7     timeit dic    str i  i for i in xrange 10    dic  10     None  dic  5     None         dic    k  v for k  v in dic items   if v is not None      1000000 loops  best of 7  375 ns per loop  2  List comprehension recreate dict using dict   In  8     timeit dic    str i  i for i in xrange 10    dic  10     None  dic  5     None         dic   dict  k  v  for k  v in dic items   if v is not None  1000000 loops  best of 7  681 ns per loop  3  Loop and delete key if v is None In  10     timeit dic    str i  i for i in xrange 10    dic  10     None  dic  5     None          for k  v in dic items               if v is None               del dic k            10000000 loops  best of 7  160 ns per loop  so loop and delete is the fastest at 160ns  list comprehension is half as slow at  375ns and with a call to dict   is half as slow again  680ns  Wrapping 3 into a function brings it back down again to about 275ns  Also for me PyPy was about twice as fast as neet python

User · Answer

Based on Ryan s solution  if you also have lists and nested dictionaries   For Python 2   def remove empty from dict d       if type d  is dict          return dict  k  remove empty from dict v   for k  v in d iteritems   if v and remove empty from dict v       elif type d  is list          return  remove empty from dict v  for v in d if v and remove empty from dict v       else          return d   For Python 3   def remove empty from dict d       if type d  is dict          return dict  k  remove empty from dict v   for k  v in d items   if v and remove empty from dict v       elif type d  is list          return  remove empty from dict v  for v in d if v and remove empty from dict v       else          return d

User · Answer

For python 3  dict  k  v  for k  v in metadata items   if v

User · Answer

BrenBarn s solution is ideal  and pythonic  I might add   Here is another  fp  solution  however   from operator import itemgetter dict filter itemgetter 1   metadata items

User · Answer

Some of Methods mentioned above ignores if there are any integers and float with values 0  amp  0 0  If someone wants to avoid the above can use below code removes empty strings and None values from nested dictionary and nested list    def remove empty from dict d       if type d  is dict           temp              for k v in d items                if v    None or v                        pass             elif type v  is int or type v  is float                   temp k    remove empty from dict v              elif  v or remove empty from dict v                     temp k    remove empty from dict v          return  temp     elif type d  is list          return  remove empty from dict v  for v in d if   str v  strip   or str remove empty from dict v   strip    and  v    None or remove empty from dict v     None        else          return d

User · Answer

If you want a full-featured  yet succinct approach to handling real-world data structures which are often nested  and can even contain cycles  I recommend looking at the remap utility from the boltons utility package    After pip install boltons or copying iterutils py into your project  just do     from boltons iterutils import remap  drop falsey   lambda path  key  value  bool value  clean   remap metadata  visit drop falsey    This page has many more examples  including ones working with much larger objects from Github s API   It s pure-Python  so it works everywhere  and is fully tested in Python 2 7 and 3 3   Best of all  I wrote it for exactly cases like this  so if you find a case it doesn t handle  you can bug me to fix it right here

User · Answer

If you really need to modify the original dictionary   empty keys    k for k v in metadata iteritems   if not v  for k in empty keys      del metadata k    Note that we have to make a list of the empty keys because we can t modify a dictionary while iterating through it  as you may have noticed   This is less expensive  memory-wise  than creating a brand-new dictionary  though  unless there are a lot of entries with empty values

User · Answer

If you have a nested dictionary  and you want this to work even for empty sub-elements  you can use a recursive variant of BrenBarn s suggestion   def scrub dict d       if type d  is dict          return dict  k  scrub dict v   for k  v in d iteritems   if v and scrub dict v       else          return d

User · Answer

I read all replies in this thread and some referred also to this thread  Remove empty dicts in nested dictionary with recursive function  I originally used solution here and it worked great   Attempt 1  Too Hot  not performant or future-proof    def scrub dict d       if type d  is dict          return dict  k  scrub dict v   for k  v in d iteritems   if v and scrub dict v       else          return d   But some performance and compatibility concerns were raised in Python 2 7 world    use isinstance instead of type unroll the list comp into for loop for efficiency use python3 safe items instead of iteritems   Attempt 2  Too Cold  Lacks Memoization    def scrub dict d       new dict          for k  v in d items            if isinstance v dict               v   scrub dict v          if not v in  u    None                   new dict k    v     return new dict   DOH  This is not recursive and not at all memoizant   Attempt 3  Just Right  so far    def scrub dict d       new dict          for k  v in d items            if isinstance v dict               v   scrub dict v          if not v in  u    None                   new dict k    v     return new dict

User · Answer

quot As I also currently write a desktop application for my work with Python  I found in data-entry application when there is lots of entry and which some are not mandatory thus user can left it blank  for validation purpose  it is easy to grab all entries and then discard empty key or value of a dictionary  So my code above a show how we can easy take them out  using dictionary comprehension and keep dictionary value element which is not blank  I use Python 3 8 3 data            20       50       100   1 1    200   1 2    dic    key value for key value in data items   if value         print dic     100    1 1    200    1 2

User · Answer

An alternative way you can do this  is using dictionary comprehension  This should be compatible with 2 7   result         key  value for key  value in       foo    bar    lorem   None  items       if value

User · Answer

It can get even shorter than BrenBarn s solution  and more readable I think    k  v for k  v in metadata items   if v    Tested with Python 2 7 3

User · Answer

Quick Answer  TL DR   Example01      example01 -------------------  mydict         alpha  0                 bravo   0                  charlie   three                  delta                     echo  False                 foxy   False                  golf                     hotel                                               newdict     dict   vkey  vdata  for vkey  vdata in mydict iteritems   if vdata     print newdict      result01 ------------------- result01        foxy    False    charlie    three    bravo    0         Detailed Answer  Problem   Context   Python 2 x Scenario  Developer wishes modify a dictionary to exclude blank values   aka remove empty values from a dictionary aka delete keys with blank values aka filter dictionary for non-blank values over each key-value pair    Solution   example01 use python list-comprehension syntax with simple conditional to remove  empty  values   Pitfalls   example01 only operates on a copy of the original dictionary  does not modify in place  example01 may produce unexpected results depending on what developer means by  empty    Does developer mean to keep values that are falsy  If the values in the dictionary are not gauranteed to be strings  developer may have unexpected data loss  result01 shows that only three key-value pairs were preserved from the original set    Alternate example   example02 helps deal with potential pitfalls The approach is to use a more precise definition of  empty  by changing the conditional  Here we only want to filter out values that evaluate to blank strings  Here we also use  strip   to filter out values that consist of only whitespace          Example02      example02 -------------------  mydict         alpha  0                 bravo   0                  charlie   three                  delta                     echo  False                 foxy   False                  golf                     hotel                       newdict     dict   vkey  vdata  for vkey  vdata in mydict iteritems   if str vdata  strip       print newdict      result02 ------------------- result02        alpha   0     bravo    0       charlie    three       delta          echo   False     foxy    False            See also   list-comprehension falsy checking for empty string modifying original dictionary in place dictionary comprehensions pitfalls of checking for empty string

User · Answer

Dicts mixed with Arrays   The answer at Attempt 3  Just Right  so far  from BlissRage s answer does not properly handle arrays elements  I m including a patch in case anyone needs it  The method is handles list with the statement block of if isinstance v  list    which scrubs the list using the original scrub dict d  implementation         staticmethod     def scrub dict d           new dict              for k  v in d items                if isinstance v  dict                   v   scrub dict v              if isinstance v  list                   v   scrub list v              if not v in  u    None                       new dict k    v         return new dict       staticmethod     def scrub list d           scrubbed list              for i in d              if isinstance i  dict                   i   scrub dict i              scrubbed list append i          return scrubbed list

User · Answer

Here is an option if you are using pandas   import pandas as pd  d   dict fromkeys   a    b    c    d    d  b      not null  d  c           empty string  print d     convert  dict  to  Series  and replace any blank strings with  None     use the   dropna    method and   then convert back to a  dict  d    pd Series d  replace     None  dropna   to dict    print d

User · Answer

Python 2 X  dict  k  v  for k  v in metadata iteritems   if v      Python 2 7 - 3 X   k  v for k  v in metadata items   if v is not None    Note that all of your keys have values   It s just that some of those values are the empty string   There s no such thing as a key in a dict without a value  if it didn t have a value  it wouldn t be in the dict

User · Answer

Building on the answers from patriciasz and nneonneo  and accounting for the possibility that you might want to delete keys that have only certain falsy things  e g      but not others  e g  0   or perhaps you even want to include some truthy things  e g   SPAM    then you could make a highly specific hitlist   unwanted        u    None  False       SPAM     Unfortunately  this doesn t quite work  because for example 0 in unwanted evaluates to True  We need to discriminate between 0 and other falsy things  so we have to use is   any  0 is i for i in unwanted        evaluates to False   Now use it to del the unwanted things   unwanted keys    k for k  v in metadata items   if any  v is i for i in unwanted    for k in unwanted keys  del metadata k    If you want a new dictionary  instead of modifying metadata in place   newdict    k  v for k  v in metadata items   if not any  v is i for i in unwanted

[python] Efficient way to remove keys with empty strings from a dict

Some benchmarking:

1. List comprehension recreate dict

2. List comprehension recreate dict using dict()

3. Loop and delete key if v is None

Examples related to python

Examples related to dictionary