How can I modify the classes below to make them pickeable?
This question: How to make a class which has __getattr__ properly pickable? is similar but refer to wrong exception in the use of getattr.
This other question seems to provide meaningful insight Why does pickle.dumps call __getattr__?, however it fails to provide an example, and I honestly cannot understand what I am suppose to implement.
import pickle
class Foo(object):
def __init__(self, dct):
for key in dct:
setattr(self, key, dct[key])
class Bar(object):
def __init__(self, dct):
for key in dct:
setattr(self, key, dct[key])
def __getattr__(self, attr):
"""If attr is not in channel, look in timing_data
"""
return getattr(self.foo, attr)
if __name__=='__main__':
dct={'a':1,'b':2,'c':3}
foo=Foo(dct)
dct2={'d':1,'e':2,'f':3,'foo':foo}
bar=Bar(dct2)
pickle.dump(bar,open('test.pkl','w'))
bar=pickle.load(open('test.pkl','r'))
Results:
14 """If attr is not in channel, look in timing_data
15 """
---> 16 return getattr(self.foo, attr)
17
18 if __name__=='__main__':
RuntimeError: maximum recursion depth exceeded while calling a Python object
The problem here is that your __getattr__ method is poorly implemented. It assumes that self.foo exists. If self.foo doesn't exist, trying to access it ends up calling __getattr__ - which results in infinite recursion:
>>> bar = Bar({}) # no `foo` attribute
>>> bar.x
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "untitled.py", line 19, in __getattr__
return getattr(self.foo, attr)
File "untitled.py", line 19, in __getattr__
return getattr(self.foo, attr)
File "untitled.py", line 19, in __getattr__
return getattr(self.foo, attr)
[Previous line repeated 329 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
To fix this, you have to throw an AttributeError if no foo attribute exists:
def __getattr__(self, attr):
"""If attr is not in channel, look in timing_data
"""
if 'foo' not in vars(self):
raise AttributeError
return getattr(self.foo, attr)
(I used the vars function to get the object's dict, because it looks nicer than self.__dict__.)
Now everything works as expected:
dct={'a':1,'b':2,'c':3}
foo=Foo(dct)
dct2={'d':1,'e':2,'f':3,'foo':foo}
bar=Bar(dct2)
data = pickle.dumps(bar)
bar = pickle.loads(data)
print(vars(bar))
# output:
# {'d': 1, 'e': 2, 'f': 3, 'foo': <__main__.Foo object at 0x7f040fc7e7f0>}
Related
Refer to the second top answer to an existing question: Difference between __getattr__ vs __getattribute__, which including code suggested by someone:
class Count(object):
def __init__(self, mymin, mymax):
self.mymin = mymin
self.mymax = mymax
self.current = None
def __getattr__(self, item):
self.__dict__[item] = 0
return 0
def __getattribute__(self, item):
if item.startswith('cur'):
raise AttributeError
return super(Count, self).__getattribute__(item)
obj1 = Count(1, 10)
print(obj1.mymin)
print(obj1.mymax)
print(obj1.current)
My question is:
When I run the code, it did not run into an infinite recursion deep (by ending with maximum recursion depth exceeded). Why? And, if I change the code super(Count, self).__getattribute__(item) to super(object, self).__getattribute__(item), it did run into an infinite loop. Why again?
Please explain the reason with a detailed calling process.
I will try to make it simpler by breaking the self.__dict__[item] into 2 parts:
class Count(object):
def __getattr__(self, item):
print('__getattr__:', item)
d = self.__dict__
print('resolved __dict__')
d[item] = 0
return 0
def __getattribute__(self, item):
print('__getattribute__:', item)
if item.startswith('cur'):
raise AttributeError
return super(Count, self).__getattribute__(item)
obj1 = Count()
print(obj1.current)
The output is
__getattribute__: current
__getattr__: current
__getattribute__: __dict__
resolved __dict__
0
Now, if we replace super(Count, self) with the incorrect construct super(object, self) the message is not printed. It is because __getattribute__ will also mask the access to __dict__. However the super object will point to the base class of object which does not exist and hence our __getattribute__ function will always throw AttributeError.
Now, after __getattribute__ fails, __getattr__ is being tried for it ... well, instead of just resolving __dict__ to some value, it tries to get it as an attribute - and ends up calling__getattribute__ again. Hence we get.
....
__getattribute__: __dict__
__getattr__: __dict__
__getattribute__: __dict__
__getattr__: __dict__
__getattribute__: __dict__
__getattr__: __dict__
__getattribute__: __dict__
__getattr__: __dict__
__getattribute__: __dict__
__getattr__: __dict__
Traceback (most recent call last):
File "getattribute.py", line 15, in <module>
print(obj1.current)
File "getattribute.py", line 4, in __getattr__
d = self.__dict__
File "getattribute.py", line 4, in __getattr__
d = self.__dict__
File "getattribute.py", line 4, in __getattr__
d = self.__dict__
[Previous line repeated 328 more times]
File "getattribute.py", line 8, in __getattribute__
print('__getattribute__: ', item)
RecursionError: maximum recursion depth exceeded while calling a Python object
Had you used setattr(self, item, 0) instead of looking up self.__dict__ this could have been "avoided":
class Count(object):
def __getattr__(self, item):
setattr(self, item, 0)
return 0
def __getattribute__(self, item):
if item.startswith('cur'):
raise AttributeError
return super(object, self).__getattribute__(item)
obj1 = Count()
print(obj1.current)
of course such code would not have been correct - trying to access any other attribute would have failed nevertheless.
i want the next code to work
class A(object):
def __getattr__(self, item):
print item
return self.item
def x(self):
print 4
a = A()
a.x()
and the output will ber
x
4
i know its not working becuase x is like a static variable and not an instance variable.
I saw this __getattr__ for static/class variables in python and it doesn't seem to work in my case
how can it be done?
thx
There are a couple of obvious problems with your code:
class A(object):
def __getattr__(self, item): # 1
print item
return self.item # 2
def x(self): # 1 again
print 4
__getattr__ will only be invoked if item cannot be found the normal way. For item == 'x', therefore, it is never invoked.
Which is probably just as well, since self.item looks for the attribute item, not the attribute corresponding to whatever is assigned to item. This doesn't exist, so would invoke __getattr__. If you try A().y() you'll get RuntimeError: maximum recursion depth exceeded while calling a Python object.
Instead, I think you want to use __getattribute__, which is always invoked. You need to be careful not to get the same runtime error, though; here I avoid it by calling the superclass implementation of __getattribute__, the naïve way of calling getattr(self, item) would fail:
class A(object):
def __getattribute__(self, item):
print item
return super(A, self).__getattribute__(item)
def x(self):
print 4
Which gives:
>>> A().x()
x
4
>>> A().y()
y
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __getattribute__
AttributeError: 'A' object has no attribute 'y'
Note that both __getattr__ and __getattribute__ apply equally to attributes and methods (which are, more or less, just callable attributes).
Is it possible to assign a numeric value to a variable in such a way that it is limited to a certain range? More specifically I want a variable that can never go below zero, because if that was about to happen an exception would be raised.
Imaginary example:
>>> var = AlwaysPositive(0)
>>> print var
0
>>> var += 3
>>> print var
3
>>> var -= 4
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AlwaysPositiveError: dropping AlwaysPositive integer below zero
The reason I ask is because I am debugging a game I am writing. Where humans understand implicitly you can never have -1 cards in your hand, a computer does not. I can make functions that check all values used in the game and call those functions at multiple positions throughout the script and see if any weird values appear. But I was wondering if there perhaps was an easier way to do this?
Sub-classing int is probably the best way to do this if you really need to, but the implementations shown so far are naive. I would do:
class NegativeValueError(ValueError):
pass
class PositiveInteger(int):
def __new__(cls, value, base=10):
if isinstance(value, basestring):
inst = int.__new__(cls, value, base)
else:
inst = int.__new__(cls, value)
if inst < 0:
raise NegativeValueError()
return inst
def __repr__(self):
return "PositiveInteger({})".format(int.__repr__(self))
def __add__(self, other):
return PositiveInteger(int.__add__(self, other))
# ... implement other numeric type methods (__sub__, __mul__, etc.)
This allows you to construct a PositiveInteger just like a regular int:
>>> PositiveInteger("FFF", 16)
PositiveInteger(4095)
>>> PositiveInteger(5)
PositiveInteger(5)
>>> PositiveInteger(-5)
Traceback (most recent call last):
File "<pyshell#24>", line 1, in <module>
PositiveInteger(-5)
File "<pyshell#17>", line 8, in __new__
raise NegativeValueError()
NegativeValueError
See e.g. the datamodel docs on numeric type emulation for details of the methods you will need to implement. Note that you don't need to explicitly check for negative numbers in most of those methods, as when you return PositiveInteger(...) the __new__ will do it for you. In use:
>>> i = PositiveInteger(5)
>>> i + 3
PositiveInteger(8)
Alternatively, if these non-negative integers will be attributes of a class, you could enforce positive values using the descriptor protocol, e.g.:
class PositiveIntegerAttribute(object):
def __init__(self, name):
self.name = name
def __get__(self, obj, typ=None):
return getattr(obj, self.name)
def __set__(self, obj, val):
if not isinstance(val, (int, long)):
raise TypeError()
if val < 0:
raise NegativeValueError()
setattr(obj, self.name, val)
def __delete__(self, obj):
delattr(obj, self.name)
You can then use this as follows:
>>> class Test(object):
foo = PositiveIntegerAttribute('_foo')
>>> t = Test()
>>> t.foo = 1
>>> t.foo = -1
Traceback (most recent call last):
File "<pyshell#34>", line 1, in <module>
t.foo = -1
File "<pyshell#28>", line 13, in __set__
raise NegativeValueError()
NegativeValueError
>>> t.foo += 3
>>> t.foo
4
>>> t.foo -= 5
Traceback (most recent call last):
File "<pyshell#37>", line 1, in <module>
t.foo -= 5
File "<pyshell#28>", line 13, in __set__
raise NegativeValueError()
NegativeValueError
You can subclass your own data type from int and provide it with a bunch of magic methods overloading the operators you need.
class Alwayspositive(int):
def __init__(self, *args, **kwargs):
super(Alwayspositive, self).__init__(*args, **kwargs)
def __neg__(self):
raise AlwayspositiveError()
def __sub__(self, other):
result = super(Alwayspositive, self).__sub__(other)
if result < 0:
raise AlwayspositiveError()
return result
And so on. This is quite a lot of work and debug to make such a class safe, but it will allow you to debug your code with a very little changes between debug and release mode.
Looking to find a solution(not sure if it exists!) to the following situation:
Starting point is a dictionary dict = {k1:v1, k2:v2,...,kn:vn} where n is not fixed.
Is there a way to write a generic class that will have n methods generated dynamically that can be called as in the following example:
class example(dict):
example.k1()
example.k2()
.
.
.
example.kn()
Eachexample.ki()where 1<=i<=n, should return the corresponding vi.
Instead of creating so many method dynamically better override __getattr__ method of your class and return a callable from there:
class Example(dict):
def __getattr__(self, k):
if k in self:
return lambda: self[k]
raise TypeError('Example object has not attribute {!r}'.format(k))
Note that for keys like keys(), items(), etc __getattr__ won't be called as they are found in the class by __getattribute__ itself. And better don't name any of your keys after them.
Demo:
>>> d = Example(a=1, b=2, c=3)
>>> d.a()
1
>>> d.b()
2
>>> d.foo()
Traceback (most recent call last):
File "<pyshell#14>", line 1, in <module>
d.foo()
File "/home/ashwini/py/so.py", line 7, in __getattr__
raise TypeError('Example object has not attribute {!r}'.format(k))
TypeError: Example object has not attribute 'foo'
What you want is to override the __getattr__ function described here.
To take your example:
class example(dict):
def __getattr__(self, name):
return lambda: self[name]
This allows you to do:
e = example()
e["foo"] = 1
print e.foo()
==> 1
I think adding a method to class dynamically can help u.
class example(object) :
dict={'k1':'v1','k2':'v2','k3':'v3','kn':'vn'}
def getvalue(self,key) :
return self.dict[key]
if __name__=="__main__" :
e = example()
e.method1=e.getvalue # this is adding a method to example class dynamically.
print e.method1('k1')
e.method2=e.getvalue
print e.method2('k2')
e.method3=e.getvalue
print e.method3('k3')
e.methodn=e.getvalue
print e.methodn('kn')
this outputs
v1
v2
v3
vn
How do I override the __getattr__ method of a class without breaking the default behavior?
Overriding __getattr__ should be fine -- __getattr__ is only called as a last resort i.e. if there are no attributes in the instance that match the name. For instance, if you access foo.bar, then __getattr__ will only be called if foo has no attribute called bar. If the attribute is one you don't want to handle, raise AttributeError:
class Foo(object):
def __getattr__(self, name):
if some_predicate(name):
# ...
else:
# Default behaviour
raise AttributeError
However, unlike __getattr__, __getattribute__ will be called first (only works for new style classes i.e. those that inherit from object). In this case, you can preserve default behaviour like so:
class Foo(object):
def __getattribute__(self, name):
if some_predicate(name):
# ...
else:
# Default behaviour
return object.__getattribute__(self, name)
See the Python docs for more.
class A(object):
def __init__(self):
self.a = 42
def __getattr__(self, attr):
if attr in ["b", "c"]:
return 42
raise AttributeError("%r object has no attribute %r" %
(self.__class__.__name__, attr))
>>> a = A()
>>> a.a
42
>>> a.b
42
>>> a.missing
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in __getattr__
AttributeError: 'A' object has no attribute 'missing'
>>> hasattr(a, "b")
True
>>> hasattr(a, "missing")
False
To extend Michael answer, if you want to maintain the default behavior using __getattr__, you can do it like so:
class Foo(object):
def __getattr__(self, name):
if name == 'something':
return 42
# Default behaviour
return self.__getattribute__(name)
Now the exception message is more descriptive:
>>> foo.something
42
>>> foo.error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 5, in __getattr__
AttributeError: 'Foo' object has no attribute 'error'