How to perfectly override a dict

Question

How can I make as  perfect  a subclass of dict as possible  The end goal is to have a simple dict in which the keys are lowercase   It would seem that there should be some tiny set of primitives I can override to make this work  but according to all my research and attempts it seem like this isn t the case    If I override   getitem     setitem    then get set don t work     How can I make them work   Surely I don t need to implement them individually  Am I preventing pickling from working  and do I need to implement   setstate   etc  Do I need repr  update and   init    Should I just use mutablemapping  it seems one shouldn t use UserDict or DictMixin    If so  how  The docs aren t exactly enlightening    Here is my first go at it  get   doesn t work and no doubt there are many other minor problems   class arbitrary dict dict          A dictionary that applies an arbitrary key-altering function        before accessing the keys          def   keytransform   self  key           return key        Overridden methods  List from        https   stackoverflow com questions 2390827 how-to-properly-subclass-dict      def   init   self   args    kwargs           self update  args    kwargs         Note  I m using dict directly  since super dict  self  doesn t work        I m not sure why  perhaps dict is not a new-style class       def   getitem   self  key           return dict   getitem   self  self   keytransform   key        def   setitem   self  key  value           return dict   setitem   self  self   keytransform   key   value       def   delitem   self  key           return dict   delitem   self  self   keytransform   key        def   contains   self  key           return dict   contains   self  self   keytransform   key     class lcdict arbitrary dict       def   keytransform   self  key           return str key  lower

User · Answer

After trying out both of the top two suggestions, I've settled on a shady-looking middle route for Python 2.7. Maybe 3 is saner, but for me:

class MyDict(MutableMapping):
   # ... the few __methods__ that mutablemapping requires
   # and then this monstrosity
   @property
   def __class__(self):
       return dict

which I really hate, but seems to fit my needs, which are:

can override **my_dict
- if you inherit from dict, this bypasses your code. try it out.
- this makes #2 unacceptable for me at all times, as this is quite common in python code
masquerades as isinstance(my_dict, dict)
- rules out MutableMapping alone, so #1 is not enough
- I heartily recommend #1 if you don't need this, it's simple and predictable
fully controllable behavior
- so I cannot inherit from dict

If you need to tell yourself apart from others, personally I use something like this (though I'd recommend better names):

def __am_i_me(self):
  return True

@classmethod
def __is_it_me(cls, other):
  try:
    return other.__am_i_me()
  except Exception:
    return False

As long as you only need to recognize yourself internally, this way it's harder to accidentally call __am_i_me due to python's name-munging (this is renamed to _MyDict__am_i_me from anything calling outside this class). Slightly more private than _methods, both in practice and culturally.

So far I have no complaints, aside from the seriously-shady-looking __class__ override. I'd be thrilled to hear of any problems that others encounter with this though, I don't fully understand the consequences. But so far I've had no problems whatsoever, and this allowed me to migrate a lot of middling-quality code in lots of locations without needing any changes.

As evidence: https://repl.it/repls/TraumaticToughCockatoo

Basically: copy the current #2 option, add print 'method_name' lines to every method, and then try this and watch the output:

d = LowerDict()  # prints "init", or whatever your print statement said
print '------'
splatted = dict(**d)  # note that there are no prints here

You'll see similar behavior for other scenarios. Say your fake-dict is a wrapper around some other datatype, so there's no reasonable way to store the data in the backing-dict; **your_dict will be empty, regardless of what every other method does.

This works correctly for MutableMapping, but as soon as you inherit from dict it becomes uncontrollable.

Edit: as an update, this has been running without a single issue for almost two years now, on several hundred thousand (eh, might be a couple million) lines of complicated, legacy-ridden python. So I'm pretty happy with it :)

Edit 2: apparently I mis-copied this or something long ago. @classmethod __class__ does not work for isinstance checks - @property __class__ does: https://repl.it/repls/UnitedScientificSequence

User · Answer

How can I make as  perfect  a subclass of dict as possible       The end goal is to have a simple dict in which the keys are lowercase          If I override   getitem     setitem    then get set don t work  How   do I make them work  Surely I don t need to implement them   individually    Am I preventing pickling from working  and do I need to implement     setstate   etc    Do I need repr  update and   init      Should I just use mutablemapping  it seems one shouldn t use UserDict   or DictMixin   If so  how  The docs aren t exactly enlightening       The accepted answer would be my first approach  but since it has some issues  and since no one has addressed the alternative  actually subclassing a dict  I m going to do that here   What s wrong with the accepted answer   This seems like a rather simple request to me      How can I make as  perfect  a subclass of dict as possible     The end goal is to have a simple dict in which the keys are lowercase    The accepted answer doesn t actually subclass dict  and a test for this fails    gt  gt  gt  isinstance MyTransformedDict    Test    test      dict  False   Ideally  any type-checking code would be testing for the interface we expect  or an abstract base class  but if our data objects are being passed into functions that are testing for dict - and we can t  fix  those functions  this code will fail   Other quibbles one might make    The accepted answer is also missing the classmethod  fromkeys   The accepted answer also has a redundant   dict   - therefore taking up more space in memory    gt  gt  gt  s foo    bar   gt  gt  gt  s   dict     foo    bar    store     test    test         Actually subclassing dict  We can reuse the dict methods through inheritance  All we need to do is create an interface layer that ensures keys are passed into the dict in lowercase form if they are strings      If I override   getitem     setitem    then get set don t work  How do I make them work  Surely I don t need to implement them individually    Well  implementing them each individually is the downside to this approach and the upside to using MutableMapping  see the accepted answer   but it s really not that much more work   First  let s factor out the difference between Python 2 and 3  create a singleton   RaiseKeyError  to make sure we know if we actually get an argument to dict pop  and create a function to ensure our string keys are lowercase   from itertools import chain try                 Python 2     str base   basestring     items    iteritems  except NameError    Python 3     str base   str  bytes  bytearray     items    items    RaiseKeyError   object     singleton for no-default behavior  def ensure lower maybe str          dict keys can be any hashable object - only call lower if str        return maybe str lower   if isinstance maybe str  str base  else maybe str   Now we implement - I m using super with the full arguments so that this code works for Python 2 and 3   class LowerDict dict      dicts take a mapping or iterable as their optional first argument       slots          no   dict   - that would be redundant      staticmethod   because this doesn t make sense as a global function      def  process args mapping       kwargs           if hasattr mapping  items               mapping   getattr mapping  items            return   ensure lower k   v  for k  v in chain mapping  getattr kwargs  items          def   init   self  mapping       kwargs           super LowerDict  self    init   self  process args mapping    kwargs       def   getitem   self  k           return super LowerDict  self    getitem   ensure lower k       def   setitem   self  k  v           return super LowerDict  self    setitem   ensure lower k   v      def   delitem   self  k           return super LowerDict  self    delitem   ensure lower k       def get self  k  default None           return super LowerDict  self  get ensure lower k   default      def setdefault self  k  default None           return super LowerDict  self  setdefault ensure lower k   default      def pop self  k  v  RaiseKeyError           if v is  RaiseKeyError              return super LowerDict  self  pop ensure lower k           return super LowerDict  self  pop ensure lower k   v      def update self  mapping       kwargs           super LowerDict  self  update self  process args mapping    kwargs       def   contains   self  k           return super LowerDict  self    contains   ensure lower k       def copy self     don t delegate w  super - dict copy   - gt  dict            return type self  self       classmethod     def fromkeys cls  keys  v None           return super LowerDict  cls  fromkeys  ensure lower k  for k in keys   v      def   repr   self           return   0   1    format type self    name    super LowerDict  self    repr        We use an almost boiler-plate approach for any method or special method that references a key  but otherwise  by inheritance  we get methods  len  clear  items  keys  popitem  and values for free  While this required some careful thought to get right  it is trivial to see that this works    Note that haskey was deprecated in Python 2  removed in Python 3    Here s some usage    gt  gt  gt  ld   LowerDict dict foo  bar     gt  gt  gt  ld  FOO    bar   gt  gt  gt  ld  foo    bar   gt  gt  gt  ld pop  FoO    bar   gt  gt  gt  ld setdefault  Foo    gt  gt  gt  ld   foo   None   gt  gt  gt  ld get  Bar    gt  gt  gt  ld setdefault  Bar    gt  gt  gt  ld   bar   None   foo   None   gt  gt  gt  ld popitem     bar   None       Am I preventing pickling from working  and do I need to implement     setstate   etc    pickling  And the dict subclass pickles just fine    gt  gt  gt  import pickle  gt  gt  gt  pickle dumps ld  b  x80 x03c  main   nLowerDict nq x00  x81q x01X x03 x00 x00 x00fooq x02Ns    gt  gt  gt  pickle loads pickle dumps ld     foo   None   gt  gt  gt  type pickle loads pickle dumps ld     lt class    main   LowerDict  gt      repr       Do I need repr  update and   init      We defined update and   init    but you have a beautiful   repr   by default    gt  gt  gt  ld   without   repr   defined for the class  we get this   foo   None    However  it s good to write a   repr   to improve the debugability of your code  The ideal test is eval repr obj      obj  If it s easy to do for your code  I strongly recommend it    gt  gt  gt  ld   LowerDict      gt  gt  gt  eval repr ld      ld True  gt  gt  gt  ld   LowerDict dict a 1  b 2  c 3    gt  gt  gt  eval repr ld      ld True   You see  it s exactly what we need to recreate an equivalent object - this is something that might show up in our logs or in backtraces    gt  gt  gt  ld LowerDict   a   1   c   3   b   2     Conclusion     Should I just use mutablemapping  it seems one shouldn t use UserDict   or DictMixin   If so  how  The docs aren t exactly enlightening    Yeah  these are a few more lines of code  but they re intended to be comprehensive  My first inclination would be to use the accepted answer   and if there were issues with it  I d then look at my answer - as it s a little more complicated  and there s no ABC to help me get my interface right   Premature optimization is going for greater complexity in search of performance  MutableMapping is simpler - so it gets an immediate edge  all else being equal  Nevertheless  to lay out all the differences  let s compare and contrast   I should add that there was a push to put a similar dictionary into the collections module  but it was rejected  You should probably just do this instead   my dict transform key     It should be far more easily debugable   Compare and contrast  There are 6 interface functions implemented with the MutableMapping  which is missing fromkeys  and 11 with the dict subclass  I don t need to implement   iter   or   len    but instead I have to implement get  setdefault  pop  update  copy    contains    and fromkeys - but these are fairly trivial  since I can use inheritance for most of those implementations   The MutableMapping implements some things in Python that dict implements in C - so I would expect a dict subclass to be more performant in some cases   We get a free   eq   in both approaches - both of which assume equality only if another dict is all lowercase - but again  I think the dict subclass will compare more quickly   Summary    subclassing MutableMapping is simpler with fewer opportunities for bugs  but slower  takes more memory  see redundant dict   and fails isinstance x  dict  subclassing dict is faster  uses less memory  and passes isinstance x  dict   but it has greater complexity to implement    Which is more perfect  That depends on your definition of perfect

User · Answer

All you will have to do is   class BatchCollection dict       def   init   self   args    kwargs           dict   init    args    kwargs    OR  class BatchCollection dict       def   init   self  inpt              super BatchCollection  self    init   inpt    A sample usage for my personal use      EXAMPLE class BatchCollection dict       def   init   self  inpt              dict   init    args    kwargs       def   setitem   self  key  item           if  isinstance key  tuple  and len key     2                 and isinstance item  collections Iterable                  self   dict   key    item             super BatchCollection  self    setitem   key  item          else              raise Exception                   Valid key should be a tuple  database name  table name                     and value should be iterable     Note  tested only in python3

User · Answer

My requirements were a bit stricter    I had to retain case info  the strings are paths to files displayed to the user  but it s a windows app so internally all operations must be case insensitive  I needed keys to be as small as possible  it did make a difference in memory performance  chopped off 110 mb out of 370   This meant that caching lowercase version of keys is not an option  I needed creation of the data structures to be as fast as possible  again made a difference in performance  speed this time   I had to go with a builtin   My initial thought was to substitute our clunky Path class for a case insensitive unicode subclass - but    proved hard to get that right - see  A case insensitive string class in python turns out that explicit dict keys handling makes code verbose and messy - and error prone  structures are passed hither and thither  and it is not clear if they have CIStr instances as keys elements  easy to forget plus some dict CIstr path   is ugly    So I had finally to write down that case insensitive dict  Thanks to code by  AaronHall that was made 10 times easier   class CIstr unicode          See https   stackoverflow com a 43122305 281545  especially for inlines          slots          does make a difference in memory performance       --Hash Compare     def   hash   self           return hash self lower        def   eq   self  other           if isinstance other  CIstr               return self lower      other lower           return NotImplemented     def   ne   self  other           if isinstance other  CIstr               return self lower      other lower           return NotImplemented     def   lt   self  other           if isinstance other  CIstr               return self lower    lt  other lower           return NotImplemented     def   ge   self  other           if isinstance other  CIstr               return self lower    gt   other lower           return NotImplemented     def   gt   self  other           if isinstance other  CIstr               return self lower    gt  other lower           return NotImplemented     def   le   self  other           if isinstance other  CIstr               return self lower    lt   other lower           return NotImplemented      --repr     def   repr   self           return   0   1    format type self    name                                     super CIstr  self    repr       def  ci str maybe str          dict keys can be any hashable object - only call CIstr if str        return CIstr maybe str  if isinstance maybe str  basestring  else maybe str  class LowerDict dict          Dictionary that transforms its keys to CIstr instances      Adapted from  https   stackoverflow com a 39375731 281545               slots          no   dict   - that would be redundant       staticmethod   because this doesn t make sense as a global function      def  process args mapping       kwargs           if hasattr mapping   iteritems                mapping   getattr mapping   iteritems             return    ci str k   v  for k  v in                 chain mapping  getattr kwargs   iteritems           def   init   self  mapping       kwargs             dicts take a mapping or iterable as their optional first argument         super LowerDict  self    init   self  process args mapping    kwargs       def   getitem   self  k           return super LowerDict  self    getitem    ci str k       def   setitem   self  k  v           return super LowerDict  self    setitem    ci str k   v      def   delitem   self  k           return super LowerDict  self    delitem    ci str k       def copy self     don t delegate w  super - dict copy   - gt  dict            return type self  self      def get self  k  default None           return super LowerDict  self  get  ci str k   default      def setdefault self  k  default None           return super LowerDict  self  setdefault  ci str k   default        no default   object       def pop self  k  v   no default           if v is LowerDict   no default                super will raise KeyError if no default and key does not exist             return super LowerDict  self  pop  ci str k           return super LowerDict  self  pop  ci str k   v      def update self  mapping       kwargs           super LowerDict  self  update self  process args mapping    kwargs       def   contains   self  k           return super LowerDict  self    contains    ci str k        classmethod     def fromkeys cls  keys  v None           return super LowerDict  cls  fromkeys   ci str k  for k in keys   v      def   repr   self           return   0   1    format type self    name                                     super LowerDict  self    repr        Implicit vs explicit is still a problem  but once dust settles  renaming of attributes variables to start with ci  and a big fat doc comment explaining that ci stands for case insensitive  I think is a perfect solution - as readers of the code must be fully aware that we are dealing with case insensitive underlying data structures  This will hopefully fix some hard to reproduce bugs  which I suspect boil down to case sensitivity   Comments corrections welcome

User · Answer

You can write an object that behaves like a dict quite easily with ABCs  Abstract Base Classes  from the collections abc module   It even tells you if you missed a method  so below is the minimal version that shuts the ABC up  from collections abc import MutableMapping   class TransformedDict MutableMapping        quot  quot  quot A dictionary that applies an arbitrary key-altering        function before accessing the keys quot  quot  quot       def   init   self   args    kwargs           self store   dict           self update dict  args    kwargs      use the free update to set keys      def   getitem   self  key           return self store self  keytransform key        def   setitem   self  key  value           self store self  keytransform key     value      def   delitem   self  key           del self store self  keytransform key        def   iter   self           return iter self store           def   len   self           return len self store       def  keytransform self  key           return key  You get a few free methods from the ABC  class MyTransformedDict TransformedDict        def  keytransform self  key           return key lower     s   MyTransformedDict    Test    test      assert s get  TEST   is s  test       free get assert  TeSt  in s                    free   contains                                         free setdefault    eq    and so on  import pickle   works too since we just use a normal dict assert pickle loads pickle dumps s      s  I wouldn t subclass dict  or other builtins  directly  It often makes no sense  because what you actually want to do is implement the interface of a dict  And that is exactly what ABCs are for

[python] How to "perfectly" override a dict?

Examples related to python

Examples related to inheritance

Examples related to dictionary

Examples related to get

Examples related to set