We Keep Coding

how to make classes with __getattr__ pickable

How can I modify the classes below to make them pickeable?
This question: How to make a class which has __getattr__ properly pickable? is similar but refer to wrong exception in the use of getattr.
This other question seems to provide meaningful insight Why does pickle.dumps call __getattr__?, however it fails to provide an example, and I honestly cannot understand what I am suppose to implement.
import pickle
class Foo(object):
def __init__(self, dct):
for key in dct:
setattr(self, key, dct[key])
class Bar(object):
def __init__(self, dct):
for key in dct:
setattr(self, key, dct[key])
def __getattr__(self, attr):
"""If attr is not in channel, look in timing_data
"""
return getattr(self.foo, attr)
if __name__=='__main__':
dct={'a':1,'b':2,'c':3}
foo=Foo(dct)
dct2={'d':1,'e':2,'f':3,'foo':foo}
bar=Bar(dct2)
pickle.dump(bar,open('test.pkl','w'))
bar=pickle.load(open('test.pkl','r'))
Results:
14 """If attr is not in channel, look in timing_data
15 """
---> 16 return getattr(self.foo, attr)
17
18 if __name__=='__main__':
RuntimeError: maximum recursion depth exceeded while calling a Python object

The problem here is that your __getattr__ method is poorly implemented. It assumes that self.foo exists. If self.foo doesn't exist, trying to access it ends up calling __getattr__ - which results in infinite recursion:
>>> bar = Bar({}) # no `foo` attribute
>>> bar.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "untitled.py", line 19, in __getattr__
return getattr(self.foo, attr)
File "untitled.py", line 19, in __getattr__
return getattr(self.foo, attr)
File "untitled.py", line 19, in __getattr__
return getattr(self.foo, attr)
[Previous line repeated 329 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
To fix this, you have to throw an AttributeError if no foo attribute exists:
def __getattr__(self, attr):
"""If attr is not in channel, look in timing_data
"""
if 'foo' not in vars(self):
raise AttributeError
return getattr(self.foo, attr)
(I used the vars function to get the object's dict, because it looks nicer than self.__dict__.)
Now everything works as expected:
dct={'a':1,'b':2,'c':3}
foo=Foo(dct)
dct2={'d':1,'e':2,'f':3,'foo':foo}
bar=Bar(dct2)
data = pickle.dumps(bar)
bar = pickle.loads(data)
print(vars(bar))
# output:
# {'d': 1, 'e': 2, 'f': 3, 'foo': <__main__.Foo object at 0x7f040fc7e7f0>}

override __getattr__ for methods and not variables

i want the next code to work
class A(object):
def __getattr__(self, item):
print item
return self.item
def x(self):
print 4
a = A()
a.x()
and the output will ber
x
4
i know its not working becuase x is like a static variable and not an instance variable.
I saw this __getattr__ for static/class variables in python and it doesn't seem to work in my case
how can it be done?
thx

There are a couple of obvious problems with your code:
class A(object):
def __getattr__(self, item): # 1
print item
return self.item # 2
def x(self): # 1 again
print 4
__getattr__ will only be invoked if item cannot be found the normal way. For item == 'x', therefore, it is never invoked.
Which is probably just as well, since self.item looks for the attribute item, not the attribute corresponding to whatever is assigned to item. This doesn't exist, so would invoke __getattr__. If you try A().y() you'll get RuntimeError: maximum recursion depth exceeded while calling a Python object.
Instead, I think you want to use __getattribute__, which is always invoked. You need to be careful not to get the same runtime error, though; here I avoid it by calling the superclass implementation of __getattribute__, the naïve way of calling getattr(self, item) would fail:
class A(object):
def __getattribute__(self, item):
print item
return super(A, self).__getattribute__(item)
def x(self):
print 4
Which gives:
>>> A().x()
x
4
>>> A().y()
y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __getattribute__
AttributeError: 'A' object has no attribute 'y'
Note that both __getattr__ and __getattribute__ apply equally to attributes and methods (which are, more or less, just callable attributes).

How to automate the delegation of __special_methods__ in Python?

Let spam be an instance of some class Spam, and suppose that spam.ham is an object of some built-in type, say dict. Even though Spam is not a subclass of dict, I would like its instances to have the same API as a regular dict (i.e. the same methods with the same signatures), but I want to avoid typing out a bazillion boilerplate methods of the form:
def apimethod(self, this, that):
return self.ham.apimethod(this, that)
I tried the following:
class Spam(object):
def __init__(self):
self.ham = dict()
def __getattr__(self, attr):
return getattr(self.ham, attr)
...but it works for "regular" methods, like keys and items, but not for special methods, like __setitem__, __getitem__, and __len__:
>>> spam = Spam()
>>> spam.keys()
[]
>>> spam['eggs'] = 42
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'Spam' object does not support item assignment
>>> spam.ham['eggs'] = 42
>>> foo.items()
[('eggs', 42)]
>>> spam['eggs']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'Spam' object is not subscritable
>>> len(spam)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'Spam' object has no len()
All the special methods I tried produced similar errors.
How can I automate the definition of special methods (so that they get referred to the delegate)?
Clarification: I'm not necessarily looking for solutions that leverage the standard method lookup sequence. My goal here is to minimize boilerplate code.
Thanks!

This may not be helpful if you need a solution that prohibits metaclasses as well, but here is the solution I came up with:
def _wrapper(func):
def _wrapped(self, *args, **kwargs):
return getattr(self.ham, func)(*args, **kwargs)
return _wrapped
class DictMeta(type):
def __new__(cls, name, bases, dct):
default_attrs = dir(object)
for attr in dir(dict):
if attr not in default_attrs:
dct[attr] = _wrapper(attr)
return type.__new__(cls, name, bases, dct)
class Spam(object):
__metaclass__ = DictMeta
def __init__(self):
self.ham = dict()
Seems to do what you're looking for:
>>> spam = Spam()
>>> spam['eggs'] = 42
>>> spam.items()
[('eggs', 42)]
>>> len(spam)
1
>>> spam.ham
{'eggs': 42}
If on Python 3.x use class Spam(object, metaclass=DictMeta) and remove the __metaclass__ line from the body of Spam.

This looks like a job for ... a metaclass!
def make_method(p, m):
def method(self, *a, **k):
return getattr(getattr(self, p),m)(*a, **k)
return method
class Proxier(type):
def __new__(cls, name, bases, dict):
objs = dict.get('proxyobjs', [])
if objs:
old_init = dict.get('__init__', lambda self: None)
def new_init(self, *a, **k):
for (n,v) in objs.iteritems():
setattr(self, n, v())
old_init(self, *a, **k)
dict['__init__'] = new_init
meths = dict.get('proxymethods', {})
for (proxyname, methnames) in meths.iteritems():
for methname in methnames:
dict[methname] = make_method(proxyname, methname)
return super(Proxier, cls).__new__(cls, name, bases, dict)
class Spam(object):
__metaclass__ = Proxier
proxyobjs = {'ham': dict,
'eggs': list,
}
proxymethods = {'ham': ('__setitem__', '__getitem__', '__delitem__'),
'eggs': ('__contains__', 'append')
}
It works!
In [28]: s = Spam()
In [29]: s[4] = 'hi'
In [30]: s.append(3)
In [31]: 3 in s
Out[31]: True
In [32]: 4 in s
Out[32]: False
In [33]: s[4]
Out[33]: 'hi'
Note that you have to specify what parts of the interface you're using (otherwise, why not just inherit?). So we have __contains__ from list, and __getitem__ from dict, and the __iter__ from neither. (And only one way to mutate the underlying list, using append but not extend or __delitem__.) So (like Martian) I'm not sure how useful this will be.

Attribute access for special methods doesn't obey normal attribute access rules, basically those methods MUST exist at class level, read http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes
So you need to add all those methods either manually or you can add them to class programmatically and best way to do that is thru metaclass. Also note that I am not adding all methods in dict but only special methods because rest can be easily redirected thru __getattr__
def redirect(methodname):
def _redirect(self, *args, **kwargs):
print "redirecting",methodname
method = getattr(self.ham, methodname)
return method(*args, **kwargs)
return _redirect
class DictRedirect(object):
def __new__(cls, name, bases, attrs):
# re-create all special methods from dict
dict_attr_names = set(dir(dict))
common_names = set(dir(cls))
for methodname in dict_attr_names-common_names:
if not methodname.startswith('__'):
continue
attrs[methodname] = redirect(methodname)
return type(name, bases, attrs)
class Spam(object):
__metaclass__ = DictRedirect
def __init__(self):
self.ham = dict()
def __getattr__(self, name):
return getattr(self.ham, name)
spam = Spam()
spam['eggs'] = 'yolk'
print 'keys =',spam.keys()
print spam['eggs']
output:
redirecting __setitem__
keys = ['eggs']
redirecting __getitem__
yolk
Disclaimer: IMO this is too much magic and should be avoided except for having fun :)

Not sure __getattribute__ will help, but the reason is the special methods are looked up in the class not in the instance: http://docs.python.org/reference/datamodel.html#special-method-lookup-for-new-style-classes , as for example the special methods like __getattr__ and __getattribute__ themselves have to be looked up somewhere.
Proxying like this seems asking for trouble to me without careful thinking, for example how should things like __dict__ and __class__ behave and about possible method conflicts if your wrapper happens to have any methods, and sure there are other problems.
Re: is-a vs. has-a:
If you just duplicate whole interface of contained member, it seems like anti-pattern to me, as that's what inheritance is for. What if you have a two has-a relations to two dict objects?
In has-a relation, one usually picks useful methods often exporting them under different names to make sensible API. So instead Spam.append(item) you would have Spam.addBot(bot).

Raising an exception on updating a 'constant' attribute in python

As python does not have concept of constants, would it be possible to raise an exception if an 'constant' attribute is updated? How?
class MyClass():
CLASS_CONSTANT = 'This is a constant'
var = 'This is a not a constant, can be updated'
#this should raise an exception
MyClass.CLASS_CONSTANT = 'No, this cannot be updated, will raise an exception'
#this should not raise an exception
MyClass.var = 'updating this is fine'
#this also should raise an exception
MyClass().CLASS_CONSTANT = 'No, this cannot be updated, will raise an exception'
#this should not raise an exception
MyClass().var = 'updating this is fine'
Any attempt to change CLASS_CONSTANT as a class attribute or as an instance attribute should raise an exception.
Changing var as a class attribute or as an instance attribute should not raise an exception.

Customizing __setattr__ in every class (e.g. as exemplified in my old recipe that #ainab's answer is pointing to, and other answers), only works to stop assignment to INSTANCE attributes and not to CLASS attributes. So, none of the existing answers would actually satisfy your requirement as stated.
If what you asked for IS actually exactly what you want, you could resort to some mix of custom metaclasses and descriptors, such as:
class const(object):
def __init__(self, val): self.val = val
def __get__(self, *_): return self.val
def __set__(self, *_): raise TypeError("Can't reset const!")
class mcl(type):
def __init__(cls, *a, **k):
mkl = cls.__class__
class spec(mkl): pass
for n, v in vars(cls).items():
if isinstance(v, const):
setattr(spec, n, v)
spec.__name__ = mkl.__name__
cls.__class__ = spec
class with_const:
__metaclass__ = mcl
class foo(with_const):
CLASS_CONSTANT = const('this is a constant')
print foo().CLASS_CONSTANT
print foo.CLASS_CONSTANT
foo.CLASS_CONSTANT = 'Oops!'
print foo.CLASS_CONSTANT
This is pretty advanced stuff, so you might prefer the simpler __setattr__ approach suggested in other answers, despite it NOT meeting your requirements as stated (i.e., you might reasonably choose to weaken your requirements in order to gain simplicity;-). But the techniques here might still be interesting: the custom descriptor type const is another way (IMHO far nicer than overriding __setattr__ in each and every class that needs some constants AND making all attributes constants rather than picking and choosing...) to block assignment to an instance attribute; the rest of the code is about a custom metaclass creating unique per-class sub-metaclasses of itself, in order to exploit said custom descriptor to the fullest and achieving the exact functionality you specifically asked for.

You could do something like this:
(from http://www.siafoo.net/snippet/108)
class Constants:
# A constant variable
foo = 1337
def __setattr__(self, attr, value):
if hasattr(self, attr):
raise ValueError, 'Attribute %s already has a value and so cannot be written to' % attr
self.__dict__[attr] = value
Then use it like this:
>>> const = Constants()
>>> const.test1 = 42
>>> const.test1
42
>>> const.test1 = 43
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __setattr__
ValueError: Attribute test1 already has a value and so cannot be written to
>>> const.test1
42

You can use a metaclass to achieve this:
class ImmutableConstants(type):
def __init__(cls, name, bases, dct):
type.__init__(cls, name, bases, dct)
old_setattr = cls.__setattr__
def __setattr__(self, key, value):
cls.assert_attribute_mutable(key)
old_setattr(self, key, value)
cls.__setattr__ = __setattr__
def __setattr__(self, key, value):
self.assert_attribute_mutable(key)
type.__setattr__(self, key, value)
def assert_attribute_mutable(self, name):
if name.isupper():
raise AttributeError('Attribute %s is constant' % name)
class Foo(object):
__metaclass__ = ImmutableConstants
CONST = 5
class_var = 'foobar'
Foo.class_var = 'new value'
Foo.CONST = 42 # raises
But are you sure this is a real issue? Are you really accidentally setting constants all over the place? You can find most of these pretty easily with a grep -r '\.[A-Z][A-Z0-9_]*\s*=' src/.

If you really want to have constant that can't be changed then look at this: http://code.activestate.com/recipes/65207/

Start reading this:
http://docs.python.org/reference/datamodel.html#customizing-attribute-access
You basically write your own version of __setattr__ that throws exceptions for some attributes, but not others.

How do I implement __getattribute__ without an infinite recursion error?

I want to override access to one variable in a class, but return all others normally. How do I accomplish this with __getattribute__?
I tried the following (which should also illustrate what I'm trying to do) but I get a recursion error:
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else:
return self.__dict__[name]
>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp

You get a recursion error because your attempt to access the self.__dict__ attribute inside __getattribute__ invokes your __getattribute__ again. If you use object's __getattribute__ instead, it works:
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else:
return object.__getattribute__(self, name)
This works because object (in this example) is the base class. By calling the base version of __getattribute__ you avoid the recursive hell you were in before.
Ipython output with code in foo.py:
In [1]: from foo import *
In [2]: d = D()
In [3]: d.test
Out[3]: 0.0
In [4]: d.test2
Out[4]: 21
Update:
There's something in the section titled More attribute access for new-style classes in the current documentation, where they recommend doing exactly this to avoid the infinite recursion.

Actually, I believe you want to use the __getattr__ special method instead.
Quote from the Python docs:
__getattr__( self, name)
Called when an attribute lookup has not found the attribute in the usual places (i.e. it is not an instance attribute nor is it found in the class tree for self). name is the attribute name. This method should return the (computed) attribute value or raise an AttributeError exception.
Note that if the attribute is found through the normal mechanism, __getattr__() is not called. (This is an intentional asymmetry between __getattr__() and __setattr__().) This is done both for efficiency reasons and because otherwise __setattr__() would have no way to access other attributes of the instance. Note that at least for instance variables, you can fake total control by not inserting any values in the instance attribute dictionary (but instead inserting them in another object). See the __getattribute__() method below for a way to actually get total control in new-style classes.
Note: for this to work, the instance should not have a test attribute, so the line self.test=20 should be removed.

Python language reference:
In order to avoid infinite recursion
in this method, its implementation
should always call the base class
method with the same name to access
any attributes it needs, for example,
object.__getattribute__(self, name).
Meaning:
def __getattribute__(self,name):
...
return self.__dict__[name]
You're calling for an attribute called __dict__. Because it's an attribute, __getattribute__ gets called in search for __dict__ which calls __getattribute__ which calls ... yada yada yada
return object.__getattribute__(self, name)
Using the base classes __getattribute__ helps finding the real attribute.

How is the __getattribute__ method used?
It is called before the normal dotted lookup. If it raises AttributeError, then we call __getattr__.
Use of this method is rather rare. There are only two definitions in the standard library:
$ grep -Erl "def __getattribute__\(self" cpython/Lib | grep -v "/test/"
cpython/Lib/_threading_local.py
cpython/Lib/importlib/util.py
Best Practice
The proper way to programmatically control access to a single attribute is with property. Class D should be written as follows (with the setter and deleter optionally to replicate apparent intended behavior):
class D(object):
def __init__(self):
self.test2=21
#property
def test(self):
return 0.
#test.setter
def test(self, value):
'''dummy function to avoid AttributeError on setting property'''
#test.deleter
def test(self):
'''dummy function to avoid AttributeError on deleting property'''
And usage:
>>> o = D()
>>> o.test
0.0
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
A property is a data descriptor, thus it is the first thing looked for in the normal dotted lookup algorithm.
Options for __getattribute__
You several options if you absolutely need to implement lookup for every attribute via __getattribute__.
raise AttributeError, causing __getattr__ to be called (if implemented)
return something from it by
using super to call the parent (probably object's) implementation
calling __getattr__
implementing your own dotted lookup algorithm somehow
For example:
class NoisyAttributes(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self, name):
print('getting: ' + name)
try:
return super(NoisyAttributes, self).__getattribute__(name)
except AttributeError:
print('oh no, AttributeError caught and reraising')
raise
def __getattr__(self, name):
"""Called if __getattribute__ raises AttributeError"""
return 'close but no ' + name
>>> n = NoisyAttributes()
>>> nfoo = n.foo
getting: foo
oh no, AttributeError caught and reraising
>>> nfoo
'close but no foo'
>>> n.test
getting: test
20
What you originally wanted.
And this example shows how you might do what you originally wanted:
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else:
return super(D, self).__getattribute__(name)
And will behave like this:
>>> o = D()
>>> o.test = 'foo'
>>> o.test
0.0
>>> del o.test
>>> o.test
0.0
>>> del o.test
Traceback (most recent call last):
File "<pyshell#216>", line 1, in <module>
del o.test
AttributeError: test
Code review
Your code with comments. You have a dotted lookup on self in __getattribute__.
This is why you get a recursion error. You could check if name is "__dict__" and use super to workaround, but that doesn't cover __slots__. I'll leave that as an exercise to the reader.
class D(object):
def __init__(self):
self.test=20
self.test2=21
def __getattribute__(self,name):
if name=='test':
return 0.
else: # v--- Dotted lookup on self in __getattribute__
return self.__dict__[name]
>>> print D().test
0.0
>>> print D().test2
...
RuntimeError: maximum recursion depth exceeded in cmp

Are you sure you want to use __getattribute__? What are you actually trying to achieve?
The easiest way to do what you ask is:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
test = 0
or:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
#property
def test(self):
return 0
Edit:
Note that an instance of D would have different values of test in each case. In the first case d.test would be 20, in the second it would be 0. I'll leave it to you to work out why.
Edit2:
Greg pointed out that example 2 will fail because the property is read only and the __init__ method tried to set it to 20. A more complete example for that would be:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
_test = 0
def get_test(self):
return self._test
def set_test(self, value):
self._test = value
test = property(get_test, set_test)
Obviously, as a class this is almost entirely useless, but it gives you an idea to move on from.

Here is a more reliable version:
class D(object):
def __init__(self):
self.test = 20
self.test2 = 21
def __getattribute__(self, name):
if name == 'test':
return 0.
else:
return super(D, self).__getattribute__(name)
It calls __getattribute__ method from parent class, eventually falling back to object.__getattribute__ method if other ancestors don't override it.