I'm struggling with __getattr__. I have a complex recursive codebase, where it is important to let exceptions propagate.
class A(object):
#property
def a(self):
raise AttributeError('lala')
def __getattr__(self, name):
print('attr: ', name)
return 1
print(A().a)
Results in:
('attr: ', 'a')
1
Why this behaviour? Why is no exception thrown? This behaviour is not documented (__getattr__ documentation). getattr() could just use A.__dict__. Any thoughts?
I just changed the code to
class A(object):
#property
def a(self):
print "trying property..."
raise AttributeError('lala')
def __getattr__(self, name):
print('attr: ', name)
return 1
print(A().a)
and, as we see, indeed the property is tried first. But as it claims not to be there (by raising AttributeError), __getattr__() is called as "last resort".
It is not documented clearly, but can maybe be counted under "Called when an attribute lookup has not found the attribute in the usual places".
Using __getattr__ and properties in the same class is dangerous, because it can lead to errors that are very difficult to debug.
If the getter of a property throws AttributeError, then the AttributeError is silently caught, and __getattr__ is called. Usually, this causes __getattr__ to fail with an exception, but if you are extremely unlucky, it doesn't, and you won't even be able to easily trace the problem back to __getattr__.
EDIT: Example code for this problem can be found in this answer.
Unless your property getter is trivial, you can never be 100% sure it won't throw AttributeError. The exception may be thrown several levels deep.
Here is what you could do:
Avoid using properties and __getattr__ in the same class.
Add a try ... except block to all property getters that are not trivial
Keep property getters simple, so you know they won't throw AttributeError
Write your own version of the #property decorator, which catches AttributeError and re-throws it as RuntimeError.
See also http://blog.devork.be/2011/06/using-getattr-and-property_17.html
EDIT: In case anyone is considering solution 4 (which I don't recommend), it can be done like this:
def property_(f):
def getter(*args, **kwargs):
try:
return f(*args, **kwargs)
except AttributeError as e:
raise RuntimeError, "Wrapped AttributeError: " + str(e), sys.exc_info()[2]
return property(getter)
Then use #property_ instead of #property in classes that override __getattr__.
__getattribute__ documentation says:
If the class also defines __getattr__(), the latter will not be called unless __getattribute__() either calls it explicitly or raises an AttributeError.
I read this (by inclusio unius est exclusio alterius) as saying that attribute access will call __getattr__ if object.__getattribute__ (which is "called unconditionally to implement attribute accesses") happens to raise AttributeError - whether directly or inside a descriptor __get__ (e.g. a property fget); note that __get__ should "return the (computed) attribute value or raise an AttributeError exception".
As an analogy, operator special methods can raise NotImplementedError whereupon the other operator methods (e.g. __radd__ for __add__) will be tried.
__getattr__ is called when an attribute access fails with an AttributeError. Maybe this is why you think it 'catches' the errors. However, it doesn't, it's Python's attribute access functionality that catches them, and then calls __getattr__.
But __getattr__ itself doesn't catch any errors. If you raise an AttributeError in __getattr__ you get infinite recursion.
regularly run into this problem because I implement __getattr__ a lot and have lots of #property methods. Here's a decorator I came up with to get a more useful error message:
def replace_attribute_error_with_runtime_error(f):
#functools.wraps(f)
def wrapped(*args, **kwargs):
try:
return f(*args, **kwargs)
except AttributeError as e:
# logging.exception(e)
raise RuntimeError(
'{} failed with an AttributeError: {}'.format(f.__name__, e)
)
return wrapped
And use it like this:
class C(object):
def __getattr__(self, name):
...
#property
#replace_attribute_error_with_runtime_error
def complicated_property(self):
...
...
The error message of the underlying exception will include name of the class whose instance raised the underlying AttributeError.
You can also log it if you want to.
You're doomed anyways when you combine #property with __getattr__:
class Paradise:
pass
class Earth:
#property
def life(self):
print('Checking for paradise (just for fun)')
return Paradise.breasts
def __getattr__(self, item):
print("sorry! {} does not exist in Earth".format(item))
earth = Earth()
try:
print('Life in earth: ' + str(earth.life))
except AttributeError as e:
print('Exception found!: ' + str(e))
Gives the following output:
Checking for paradise (just for fun)
sorry! life does not exist in Earth
Life in earth: None
When your real problem was with calling Paradise.breasts.
__getattr__ is always called when an AtributeError is risen. The content of the exception is ignored.
The sad thing is that there's no solution to this problem given hasattr(earth, 'life') will return True (just because __getattr__ is defined), but will still be reached by the attribute 'life' as it didn't exist, whereas the real underlying problem is with Paradise.breasts.
My partial solution involves using a try-except in #property blocks which are known to hit upon AttributeError exceptions.
Related
I'm learning about python's inheritance, and have come across a behaviour I don't quite understand. Here is a minimal working example:
class Test():
def meth1(self):
print('accessing meth1')
return super().a #calling random nonexisting attribute; error (as expected)
#property
def prop1(self):
print('accessing prop1')
return super().a #calling random nonexisting attribute; no error?
def __getattr__(self, name):
print('getattr ' + name)
test = Test()
Calling .meth1() fails as expected...
In [1]: test.meth1()
accessing meth1
Traceback (most recent call last):
File "<ipython-input-160-4a0675c95211>", line 1, in <module>
test.meth1()
File "<ipython-input-159-1401fb9a0e13>", line 5, in meth1
return super().a #calling random nonexisting attribute; error (as expected)
AttributeError: 'super' object has no attribute 'a'
...as super() is object which does indeed not have this attribute.
But .prop1 does not...
In [2]: test.prop1
accessing prop1
getattr prop1
...which I don't understand. It seems the property is called twice, once 'normally' and once via __getattr__.
Some observations:
I assume it's got something to do with the property decorator.
The attribute .a seems to never be accessed.
If I replace the return super().a line in prop1 with something like return 5, the __getattr__ method is never called.
If I actually make Test inherit from a class having an attribute a, its value is returned from test.meth1(), but not from test.prop1.
Could someone explain what's going on here? I've not been able to find any useful information addressing the combination of attribute decorators and super().
Many thanks,
TLDR: meth1 raises AttributeError after lookup, when __getattr__ is not involved. prop1 raises AttributeError during lookup, triggering a fallback to __getattr__ which succeeds to return None.
>>> test.prop1 # AttributeError happens here during lookup
accessing prop1
getattr prop1
>>> meth = test.meth1 # AttributeError happens *not* here during lookup
>>> meth() # AttributeError happens here *after* lookup
...
AttributeError: 'super' object has no attribute 'a'
The __getattr__ method is only called when an "attribute is not found" – in other words that AttributeError is raised on access. The same behaviour occurs when the property raises the error directly:
class Test():
#property
def prop1(self):
print('accessing prop1')
raise AttributeError # replaces `super().a`
def __getattr__(self, name):
print('getattr ' + name)
test = Test()
test.prop1 # < explicitly raises AttributeError
# accessing prop1
# getattr prop1
test.prop2 # < implicitly raises AttributeError
# getattr prop2
The AttributeError does not reveal whether it comes from a missing prop1 attribute or some nested internal attribute (say, super().a). Thus, both trigger the fallback to __getattr__.
This is intended behaviour of __getattr__.
object.__getattr__(self, name)
Called when the default attribute access fails with an AttributeError (either __getattribute__() raises an AttributeError because name is not an instance attribute or an attribute in the class tree for self; or __get__() of a name property raises AttributeError).
It allows properties to fallback to the regular lookup mechanism when they cannot produce a value.
I have a python class in which I open files and read out data. If some creteria are not met, I raise an error, but before that I specify the error by giving the object an attribute: self.Error = specification. But since the error raising undos everything in the try block I can't access it. This happens in the __init__ function, so the created object doesn't even exist..
Here's the necessary code:
class MyClass:
def __init__(self):
#do something
if this_or_that:
self.Error = specification
raise MyCostumError
try:
object = MyClass()
except MyCostumError:
print(object.Error)
I get: NameError: name 'object' is not defined
Just for clarification:
I have defined MyCostumError, the variable names are just for better understanding: I use good ones and they are defined and I need the clarification, because an Error can be raised in different lines.
So here's my question:
Is there something like try/except, but when an error is raised it does NOT undo everything. Or am I just stupid and there is a much easier method for a achieving this?
If you are raising an exception in the initializer, you should not rely on the object to be created to get some error information to the caller. This is where you should use the exception to pass that information:
class MyCustomError(Exception):
pass
class MyClass:
def __init__(self):
#do something
if this_or_that:
raise MyCustomError(specification) # put the spec in the exception itself
try:
object = MyClass()
except MyCustomError as e:
print(e) # the spec is in the exception object
You are trying to reference to an object that cannot exist. Let me explain:
If an error occurs when you try to initialise an object, that object will not be initialised. So if you try to acced to it when it is not initialised, you will get an error.
try:
object = MyClass() #initialising object successful, object existing.
except: #initialising failed, object does not exist.
print(object.Error) #nameError, since object was never created.
Try/except doesn't undo anything, just stops doing something if an error occurs.
Error raising doesn't undo anything. Have a look at the docs.
As your output states, the object is not defined, this is because when you raise an error in the __init__, it is seen as the initialosor of your class failing, and this does not return an object.
I think this is what you're looking for:
class MyClass:
def __init__(self):
# do initialisation stuff
def other_method(self):
# do something
if this_or_that:
self.Error = specification
raise MyCustomError(specification)
object = MyClass()
try:
object.other_method()
except MyCustomError as e:
print(e)
print(object.Error)
It's not a beautiful solution but it should work:
errorcode = None
class MyClass:
def __init__(self):
global errorcode
#do something
if this_or_that:
errorcode = specification
raise MyCostumError
try:
object = MyClass()
except MyCostumError:
print(errorcode)
Given your question I think the following should fit your use case well.
class MyClass:
def __init__(self):
# Do something
try:
if this_or_that:
self.Error = specification
raise MyCostumError
except MyCustomError as e:
# Handle your custom error however you like
object = MyClass()
In the above case you should be able to mitigate the risk of instantiation failing due to custom exception/error raising failing by handling this behaviour within MyClass.__init__ itself.
This is also a much cleaner solution in terms of keeping logic relating to instantiation of MyClass objects contained within the __init__ function of the class - i.e. you won't have to worry about wrapping instantiations of this class in try/except blocks each time they are present in your code.
The IPython source code includes a getattr check that tests for the existence of '_ipython_canary_method_should_not_exist_' at the beginning of the get_real_method function:
def get_real_method(obj, name):
"""Like getattr, but with a few extra sanity checks:
- If obj is a class, ignore everything except class methods
- Check if obj is a proxy that claims to have all attributes
- Catch attribute access failing with any exception
- Check that the attribute is a callable object
Returns the method or None.
"""
try:
canary = getattr(obj, '_ipython_canary_method_should_not_exist_', None)
except Exception:
return None
if canary is not None:
# It claimed to have an attribute it should never have
return None
And although it's easy enough to find other coders special-casing this name, it's a harder to find any meaningful explanation of why.
Given these two classes:
from __future__ import print_function
class Parrot(object):
def __getattr__(self, attr):
print(attr)
return lambda *a, **kw: print(attr, a, kw)
class DeadParrot(object):
def __getattr__(self, attr):
print(attr)
if attr == '_ipython_canary_method_should_not_exist_':
raise AttributeError(attr)
return lambda *a, **kw: print(attr, a, kw)
It seems that IPython is using the existence or lack of this method to decide whether to use repr or one of its rich display methods. Intentionally thwarting the test in DeadParrot causes IPython to look up and invoke _repr_mimebundle_.
I'm writing an object that pretends all attrs exist. How do I decide whether to special-case this?
I just encountered an unexpected behavior. This is a simple class with a __getattr__ method and a property attribute with a typo inside:
class A(object):
def __getattr__(self, attr):
if not attr.startswith("ignore_"):
raise AttributeError(attr)
#property
def prop(self):
return self.some_typo
a = A() # Instantiating
a.ignore_this # This is ignored
a.prop # This raises an Attribute Error
This is the expected outcome (the one I get if __getattr__ is commented):
AttributeError: 'A' object has no attribute 'some_typo'
And this is what I get:
AttributeError: prop
I know this has to do with__getattr__ catching the AttributeError but is there a nice and clean workaround for this issue? Because I can assure you, this is a debug nightmare...
You can just raise a better exception message:
class A(object):
def __getattr__(self, attr):
if not attr.startswith("ignore_"):
raise AttributeError("%r object has not attribute %r" % (self.__class__.__name__, attr))
#property
def prop(self):
return self.some_typo
a=A()
a.ignore_this
a.prop
EDIT: calling __getattribute__ from object base class solves the problem
class A(object):
def __getattr__(self, attr):
if not attr.startswith("ignore_"):
return self.__getattribute__(attr)
#property
def prop(self):
return self.some_typo
As mentioned by #asmeurer, the solution by #mguijarr calls prop twice. When prop first runs, it raises an AttributeError which triggers __getattr__. Then self.__getattribute__(attr) triggers prop again, finally resulting in the desired exception.
BETTER ANSWER:
Here we are better off replacing __getattribute__ instead of __getattr__. It gives us more control since __getattribute__ is invoked on all attribute access. In contrast, __getattr__ is only called when there has already been an AttributeError, and it doesn't give us access to that original error.
class A(object):
def __getattribute__(self, attr):
try:
return super().__getattribute__(attr)
except AttributeError as e:
if not attr.startswith("ignore_"):
raise e
#property
def prop(self):
print("hi")
return self.some_typo
To explain, since A subclasses object in this case, super().__getattribute__(attr) is equivalent to object.__getattribute__(self, attr). That reads a's underlying object attribute, avoiding the infinite recursion had we instead used self.__getattribute__(attr).
In case of AttributeError, we have full control to either fail or reraise, and reraising gives a sensible error message.
How would you implement lazy load of object attributes, i.e. if attributes are accessed but don't exist yet, some object method is called which is supposed to load these?
My first attempt is
def lazyload(cls):
def _getattr(obj, attr):
if "_loaded" not in obj.__dict__:
obj._loaded=True
try:
obj.load()
except Exception as e:
raise Exception("Load method failed when trying to access attribute '{}' of object\n{}".format(attr, e))
if attr not in obj.__dict__:
AttributeError("No attribute '{}' in '{}' (after loading)".format(attr, type(obj))) # TODO: infinite recursion if obj fails
return getattr(obj, attr)
else:
raise AttributeError("No attribute '{}' in '{}' (already loaded)".format(attr, type(obj)))
cls.__getattr__=_getattr
return cls
#lazyload
class Test:
def load(self):
self.x=1
t=Test() # not loaded yet
print(t.x) # will load as x isnt known yet
I will make lazyload specific to certain attribute names only.
As I havent done much meta-classing yet, I'm not sure if that is the right approach.
What would you suggest?
Seems like a simple property would do the trick better:
#property
def my_attribute():
if not hasattr(self, '_my_attribute'):
do_expensive_operation_to_get_attribute()
return self._my_attribute
Look at lazy from django/utils/functionals.py
https://docs.djangoproject.com/en/2.1/_modules/django/utils/functional