[python] How do I implement __getattribute__ without an infinite recursion error?

I want to override access to one variable in a class, but return all others normally. How do I accomplish this with __getattribute__?

I tried the following (which should also illustrate what I'm trying to do) but I get a recursion error:

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:
            return self.__dict__[name]

>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp

Are you sure you want to use __getattribute__? What are you actually trying to achieve?

The easiest way to do what you ask is:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    test = 0

or:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    @property
    def test(self):
        return 0

Edit: Note that an instance of D would have different values of test in each case. In the first case d.test would be 20, in the second it would be 0. I'll leave it to you to work out why.

Edit2: Greg pointed out that example 2 will fail because the property is read only and the __init__ method tried to set it to 20. A more complete example for that would be:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    _test = 0

    def get_test(self):
        return self._test

    def set_test(self, value):
        self._test = value

    test = property(get_test, set_test)

Obviously, as a class this is almost entirely useless, but it gives you an idea to move on from.

Actually, I believe you want to use the __getattr__ special method instead.

Quote from the Python docs:

__getattr__( self, name)

Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.

Note: for this to work, the instance should not have a test attribute, so the line self.test=20 should be removed.

Python language reference:

In order to avoid infinite recursion in this method, its implementation should always call the base class method with the same name to access any attributes it needs, for example, object.__getattribute__(self, name).

Meaning:

def __getattribute__(self,name):
    ...
        return self.__dict__[name]

You're calling for an attribute called __dict__. Because it's an attribute, __getattribute__ gets called in search for __dict__ which calls __getattribute__ which calls ... yada yada yada

return  object.__getattribute__(self, name)

Using the base classes __getattribute__ helps finding the real attribute.

Actually, I believe you want to use the __getattr__ special method instead.

Quote from the Python docs:

__getattr__( self, name)

Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.

Note: for this to work, the instance should not have a test attribute, so the line self.test=20 should be removed.

Python language reference:

In order to avoid infinite recursion in this method, its implementation should always call the base class method with the same name to access any attributes it needs, for example, object.__getattribute__(self, name).

Meaning:

def __getattribute__(self,name):
    ...
        return self.__dict__[name]

You're calling for an attribute called __dict__. Because it's an attribute, __getattribute__ gets called in search for __dict__ which calls __getattribute__ which calls ... yada yada yada

return  object.__getattribute__(self, name)

Using the base classes __getattribute__ helps finding the real attribute.

Here is a more reliable version:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21
    def __getattribute__(self, name):
        if name == 'test':
            return 0.
        else:
            return super(D, self).__getattribute__(name)

It calls __getattribute__ method from parent class, eventually falling back to object.__getattribute__ method if other ancestors don't override it.

How is the __getattribute__ method used?

It is called before the normal dotted lookup. If it raises AttributeError, then we call __getattr__.

Use of this method is rather rare. There are only two definitions in the standard library:

$ grep -Erl  "def __getattribute__\(self" cpython/Lib | grep -v "/test/"
cpython/Lib/_threading_local.py
cpython/Lib/importlib/util.py

Best Practice

The proper way to programmatically control access to a single attribute is with property. Class D should be written as follows (with the setter and deleter optionally to replicate apparent intended behavior):

class D(object):
    def __init__(self):
        self.test2=21

    @property
    def test(self):
        return 0.

    @test.setter
    def test(self, value):
        '''dummy function to avoid AttributeError on setting property'''

    @test.deleter
    def test(self):
        '''dummy function to avoid AttributeError on deleting property'''

And usage:

>>> o = D()
>>> o.test
0.0
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0

A property is a data descriptor, thus it is the first thing looked for in the normal dotted lookup algorithm.

Options for __getattribute__

You several options if you absolutely need to implement lookup for every attribute via __getattribute__.

• raise AttributeError, causing __getattr__ to be called (if implemented)
• return something from it by
  • using super to call the parent (probably object's) implementation
  • calling __getattr__
  • implementing your own dotted lookup algorithm somehow

For example:

class NoisyAttributes(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self, name):
        print('getting: ' + name)
        try:
            return super(NoisyAttributes, self).__getattribute__(name)
        except AttributeError:
            print('oh no, AttributeError caught and reraising')
            raise
    def __getattr__(self, name):
        """Called if __getattribute__ raises AttributeError"""
        return 'close but no ' + name    


>>> n = NoisyAttributes()
>>> nfoo = n.foo
getting: foo
oh no, AttributeError caught and reraising
>>> nfoo
'close but no foo'
>>> n.test
getting: test
20

What you originally wanted.

And this example shows how you might do what you originally wanted:

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:
            return super(D, self).__getattribute__(name)

And will behave like this:

>>> o = D()
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
>>> del o.test

Traceback (most recent call last):
  File "<pyshell#216>", line 1, in <module>
    del o.test
AttributeError: test

Code review

Your code with comments. You have a dotted lookup on self in __getattribute__. This is why you get a recursion error. You could check if name is "__dict__" and use super to workaround, but that doesn't cover __slots__. I'll leave that as an exercise to the reader.

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:      #   v--- Dotted lookup on self in __getattribute__
            return self.__dict__[name]

>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp

Actually, I believe you want to use the __getattr__ special method instead.

Quote from the Python docs:

__getattr__( self, name)

Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.

Note: for this to work, the instance should not have a test attribute, so the line self.test=20 should be removed.

Are you sure you want to use __getattribute__? What are you actually trying to achieve?

The easiest way to do what you ask is:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    test = 0

or:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    @property
    def test(self):
        return 0

Edit: Note that an instance of D would have different values of test in each case. In the first case d.test would be 20, in the second it would be 0. I'll leave it to you to work out why.

Edit2: Greg pointed out that example 2 will fail because the property is read only and the __init__ method tried to set it to 20. A more complete example for that would be:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    _test = 0

    def get_test(self):
        return self._test

    def set_test(self, value):
        self._test = value

    test = property(get_test, set_test)

Obviously, as a class this is almost entirely useless, but it gives you an idea to move on from.

Are you sure you want to use __getattribute__? What are you actually trying to achieve?

The easiest way to do what you ask is:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    test = 0

or:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    @property
    def test(self):
        return 0

Edit: Note that an instance of D would have different values of test in each case. In the first case d.test would be 20, in the second it would be 0. I'll leave it to you to work out why.

Edit2: Greg pointed out that example 2 will fail because the property is read only and the __init__ method tried to set it to 20. A more complete example for that would be:

class D(object):
    def __init__(self):
        self.test = 20
        self.test2 = 21

    _test = 0

    def get_test(self):
        return self._test

    def set_test(self, value):
        self._test = value

    test = property(get_test, set_test)

Obviously, as a class this is almost entirely useless, but it gives you an idea to move on from.

How is the __getattribute__ method used?

It is called before the normal dotted lookup. If it raises AttributeError, then we call __getattr__.

Use of this method is rather rare. There are only two definitions in the standard library:

$ grep -Erl  "def __getattribute__\(self" cpython/Lib | grep -v "/test/"
cpython/Lib/_threading_local.py
cpython/Lib/importlib/util.py

Best Practice

The proper way to programmatically control access to a single attribute is with property. Class D should be written as follows (with the setter and deleter optionally to replicate apparent intended behavior):

class D(object):
    def __init__(self):
        self.test2=21

    @property
    def test(self):
        return 0.

    @test.setter
    def test(self, value):
        '''dummy function to avoid AttributeError on setting property'''

    @test.deleter
    def test(self):
        '''dummy function to avoid AttributeError on deleting property'''

And usage:

>>> o = D()
>>> o.test
0.0
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0

A property is a data descriptor, thus it is the first thing looked for in the normal dotted lookup algorithm.

Options for __getattribute__

You several options if you absolutely need to implement lookup for every attribute via __getattribute__.

• raise AttributeError, causing __getattr__ to be called (if implemented)
• return something from it by
  • using super to call the parent (probably object's) implementation
  • calling __getattr__
  • implementing your own dotted lookup algorithm somehow

For example:

class NoisyAttributes(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self, name):
        print('getting: ' + name)
        try:
            return super(NoisyAttributes, self).__getattribute__(name)
        except AttributeError:
            print('oh no, AttributeError caught and reraising')
            raise
    def __getattr__(self, name):
        """Called if __getattribute__ raises AttributeError"""
        return 'close but no ' + name    


>>> n = NoisyAttributes()
>>> nfoo = n.foo
getting: foo
oh no, AttributeError caught and reraising
>>> nfoo
'close but no foo'
>>> n.test
getting: test
20

What you originally wanted.

And this example shows how you might do what you originally wanted:

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:
            return super(D, self).__getattribute__(name)

And will behave like this:

>>> o = D()
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
>>> del o.test

Traceback (most recent call last):
  File "<pyshell#216>", line 1, in <module>
    del o.test
AttributeError: test

Code review

Your code with comments. You have a dotted lookup on self in __getattribute__. This is why you get a recursion error. You could check if name is "__dict__" and use super to workaround, but that doesn't cover __slots__. I'll leave that as an exercise to the reader.

class D(object):
    def __init__(self):
        self.test=20
        self.test2=21
    def __getattribute__(self,name):
        if name=='test':
            return 0.
        else:      #   v--- Dotted lookup on self in __getattribute__
            return self.__dict__[name]

>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp