The IPython source code includes a getattr check that tests for the existence of '_ipython_canary_method_should_not_exist_' at the beginning of the get_real_method function:
def get_real_method(obj, name):
"""Like getattr, but with a few extra sanity checks:
- If obj is a class, ignore everything except class methods
- Check if obj is a proxy that claims to have all attributes
- Catch attribute access failing with any exception
- Check that the attribute is a callable object
Returns the method or None.
"""
try:
canary = getattr(obj, '_ipython_canary_method_should_not_exist_', None)
except Exception:
return None
if canary is not None:
# It claimed to have an attribute it should never have
return None
And although it's easy enough to find other coders special-casing this name, it's a harder to find any meaningful explanation of why.
Given these two classes:
from __future__ import print_function
class Parrot(object):
def __getattr__(self, attr):
print(attr)
return lambda *a, **kw: print(attr, a, kw)
class DeadParrot(object):
def __getattr__(self, attr):
print(attr)
if attr == '_ipython_canary_method_should_not_exist_':
raise AttributeError(attr)
return lambda *a, **kw: print(attr, a, kw)
It seems that IPython is using the existence or lack of this method to decide whether to use repr or one of its rich display methods. Intentionally thwarting the test in DeadParrot causes IPython to look up and invoke _repr_mimebundle_.
I'm writing an object that pretends all attrs exist. How do I decide whether to special-case this?
Related
Ive been on a tear of writing some decorators recently.
One of the ones I just wrote allows you to put the decorator just before a class definition, and it will cause every method of the class to print some logigng info when its run (more for debugging/initial super basic speed tests during a build)
def class_logit(cls):
class NCls(object):
def __init__(self, *args, **kwargs):
self.instance = cls(*args, **kwargs)
#staticmethod
def _class_logit(original_function):
def arg_catch(*args, **kwargs):
start = time.time()
result = original_function(*args, **kwargs)
print('Called: {0} | From: {1} | Args: {2} | Kwargs: {3} | Run Time: {4}'
''.format(original_function.__name__, str(inspect.getmodule(original_function)),
args, kwargs, time.time() - start))
return result
return arg_catch
def __getattribute__(self, s):
try:
x = super(NCls, self).__getattribute__(s)
except AttributeError:
pass
else:
return x
x = self.instance.__getattribute__(s)
if type(x) == type(self.__init__):
return self._class_logit(x)
else:
return x
return NCls
This works great when applied to a very basic class i create.
Where I start to encounter issues is when I apply it to a class that is inheriting another - for instance, using QT:
#scld.class_logit
class TestWindow(QtGui.QDialog):
def __init__(self):
print self
super(TestWindow, self).__init__()
a = TestWindow()
Im getting the following error... and im not entirely sure what to do about it!
self.instance = cls(*args, **kwargs)
File "<string>", line 15, in __init__
TypeError: super(type, obj): obj must be an instance or subtype of type
Any help would be appreciated!
(Apologies in advance, no matter WHAT i do SO is breaking the formatting on my first bit of code... Im even manually spending 10 minutes adding spaces but its coming out incorrectly... sorry!)
You are being a bit too intrusive with your decorator.
While if you want to profile methods defined on the Qt framework itself, a somewhat aggressive approach is needed, your decorator replaces the entire class by a proxy.
Qt bindings are somewhat complicated indeed, and it is hard to tell why it is erroring when being instantiated in this case.
So - first things first - if your intent would be to apply the decorator to a class hierarchy defined by yourself, or at least one defined in pure Python, a good approach there could be using metaclasses: with a metaclass you could decorate each method when a class is created, and do not mess anymore at runtime, when methods are retrieved from each class.
but Qt, as some other libraries, have its methods and classes defined in native code, and that will prevent you from wrapping existing methods in a new class. So, wrapping the methods on attribute retrieval on __getattribute__ could work.
Here is a simpler approach that instead of using a Proxy, just plug-in a foreign __getattribute__ that does the wrap-with-logger thing you want.
Your mileage may vary with it. Specially, it won't be triggered if one method of the class is called by other method in native code - as this won't go through Python's attribute retrieval mechanism (instead, it will use C++ method retrieval directly).
from PyQt5 import QtWidgets, QtGui
def log_dec(func):
def wraper(*args, **kwargs):
print(func.__name__, args, kwargs)
return func(*args, **kwargs)
return wraper
def decorate(cls):
def __getattribute__(self, attr):
attr = super(cls, self).__getattribute__(attr)
if callable(attr):
return log_dec(attr)
return attr
cls.__getattribute__ = __getattribute__
return cls
#decorate
class Example(QtGui.QWindow):
pass
app = QtWidgets.QApplication([])
w = Example()
w.show()
(Of course, just replace the basic logger by your fancy logger above)
I just encountered an unexpected behavior. This is a simple class with a __getattr__ method and a property attribute with a typo inside:
class A(object):
def __getattr__(self, attr):
if not attr.startswith("ignore_"):
raise AttributeError(attr)
#property
def prop(self):
return self.some_typo
a = A() # Instantiating
a.ignore_this # This is ignored
a.prop # This raises an Attribute Error
This is the expected outcome (the one I get if __getattr__ is commented):
AttributeError: 'A' object has no attribute 'some_typo'
And this is what I get:
AttributeError: prop
I know this has to do with__getattr__ catching the AttributeError but is there a nice and clean workaround for this issue? Because I can assure you, this is a debug nightmare...
You can just raise a better exception message:
class A(object):
def __getattr__(self, attr):
if not attr.startswith("ignore_"):
raise AttributeError("%r object has not attribute %r" % (self.__class__.__name__, attr))
#property
def prop(self):
return self.some_typo
a=A()
a.ignore_this
a.prop
EDIT: calling __getattribute__ from object base class solves the problem
class A(object):
def __getattr__(self, attr):
if not attr.startswith("ignore_"):
return self.__getattribute__(attr)
#property
def prop(self):
return self.some_typo
As mentioned by #asmeurer, the solution by #mguijarr calls prop twice. When prop first runs, it raises an AttributeError which triggers __getattr__. Then self.__getattribute__(attr) triggers prop again, finally resulting in the desired exception.
BETTER ANSWER:
Here we are better off replacing __getattribute__ instead of __getattr__. It gives us more control since __getattribute__ is invoked on all attribute access. In contrast, __getattr__ is only called when there has already been an AttributeError, and it doesn't give us access to that original error.
class A(object):
def __getattribute__(self, attr):
try:
return super().__getattribute__(attr)
except AttributeError as e:
if not attr.startswith("ignore_"):
raise e
#property
def prop(self):
print("hi")
return self.some_typo
To explain, since A subclasses object in this case, super().__getattribute__(attr) is equivalent to object.__getattribute__(self, attr). That reads a's underlying object attribute, avoiding the infinite recursion had we instead used self.__getattribute__(attr).
In case of AttributeError, we have full control to either fail or reraise, and reraising gives a sensible error message.
I have a xmlrpc server running looking like the following
from SimpleXMLRPCServer import SimpleXMLRPCServer
def add(x,y):
return x+y
server = SimpleXMLRPCServer(("localhost", 8000))
server.register_function(add, 'add.numbers')
server.serve_forever()
which is called used within the following code:
import xmlrpclib
class DeviceProxy(object):
def __init__(self, uri):
self.rpc = xmlrpclib.ServerProxy(uri)
def __getattr__(self, attr):
return getattr(self.rpc, attr)
original = DeviceProxy.__getattr__
def mygetattr(device, attr):
def wrapper(*args, **kw):
print('called with %r and %r' % (args, kw))
return original(device, attr)(*args, **kw)
return wrapper
DeviceProxy.__getattr__ = mygetattr
dev = DeviceProxy("http://localhost:8000/RPC2")
print dev.add.numbers(4,6)
As you can see, the Proxy class wraps the xmlrpc proxy for reasons outside the scope of this question, forwarding arbitrary calls via the __getattr__ method . For further reasons outside the scope for this question, I need to wrap/replace this __getattr__ method by a different method to e.g. print out the name of the function called, the arguments etc. (see related question here).
But this approach does not work, it gives the following error:
AttributeError: 'function' object has no attribute 'numbers'
The example works as expected when I
do not replace DeviceProxy.__getattr__ with something else
replace DeviceProxy.__getattr__ with the function
def dummy(instance, attr):
return original(device,attr)
replace the name of the xmlrpc function by a zero-dotted name (e.g. just sum instead of sum.numbers)
You can verify yourself that the following, direct call via the xmlrpc proxy will work as expected:
dev = xmlrpclib.ServerProxy("http://localhost:8000/RPC2")
print dev.add.numbers(4,6)
My question: How to solve my problem, i.e. how to be able to wrap/overwrite the DeviceProxy.__getattr__ correctly to be able to see the function called, all arguments etc WITHOUT making changes in the xmlrpc server or the DeviceProxy class?
I can see two problems here:
Are all attributes of a DeviceProxy functions? If they're not, then you're sometimes returning a function when an object is expected
When you wrap the function, you're not copying across members - use functools.wraps to achieve that.
This ought to work
from functools import wraps
#wraps(original) # probably not needed, but sensible
def mygetattr(device, key):
attr = original(device, key)
if callable(attr):
#wraps(attr) # copy across __name__, __dict__ etc
def wrapper(*args, **kw):
print('called with %r and %r' % (args, kw))
return attr(*args, **kw)
return wrapper
else: # handle (or rather, don't) non-callable attributes
return attr
I'm struggling with __getattr__. I have a complex recursive codebase, where it is important to let exceptions propagate.
class A(object):
#property
def a(self):
raise AttributeError('lala')
def __getattr__(self, name):
print('attr: ', name)
return 1
print(A().a)
Results in:
('attr: ', 'a')
1
Why this behaviour? Why is no exception thrown? This behaviour is not documented (__getattr__ documentation). getattr() could just use A.__dict__. Any thoughts?
I just changed the code to
class A(object):
#property
def a(self):
print "trying property..."
raise AttributeError('lala')
def __getattr__(self, name):
print('attr: ', name)
return 1
print(A().a)
and, as we see, indeed the property is tried first. But as it claims not to be there (by raising AttributeError), __getattr__() is called as "last resort".
It is not documented clearly, but can maybe be counted under "Called when an attribute lookup has not found the attribute in the usual places".
Using __getattr__ and properties in the same class is dangerous, because it can lead to errors that are very difficult to debug.
If the getter of a property throws AttributeError, then the AttributeError is silently caught, and __getattr__ is called. Usually, this causes __getattr__ to fail with an exception, but if you are extremely unlucky, it doesn't, and you won't even be able to easily trace the problem back to __getattr__.
EDIT: Example code for this problem can be found in this answer.
Unless your property getter is trivial, you can never be 100% sure it won't throw AttributeError. The exception may be thrown several levels deep.
Here is what you could do:
Avoid using properties and __getattr__ in the same class.
Add a try ... except block to all property getters that are not trivial
Keep property getters simple, so you know they won't throw AttributeError
Write your own version of the #property decorator, which catches AttributeError and re-throws it as RuntimeError.
See also http://blog.devork.be/2011/06/using-getattr-and-property_17.html
EDIT: In case anyone is considering solution 4 (which I don't recommend), it can be done like this:
def property_(f):
def getter(*args, **kwargs):
try:
return f(*args, **kwargs)
except AttributeError as e:
raise RuntimeError, "Wrapped AttributeError: " + str(e), sys.exc_info()[2]
return property(getter)
Then use #property_ instead of #property in classes that override __getattr__.
__getattribute__ documentation says:
If the class also defines __getattr__(), the latter will not be called unless __getattribute__() either calls it explicitly or raises an AttributeError.
I read this (by inclusio unius est exclusio alterius) as saying that attribute access will call __getattr__ if object.__getattribute__ (which is "called unconditionally to implement attribute accesses") happens to raise AttributeError - whether directly or inside a descriptor __get__ (e.g. a property fget); note that __get__ should "return the (computed) attribute value or raise an AttributeError exception".
As an analogy, operator special methods can raise NotImplementedError whereupon the other operator methods (e.g. __radd__ for __add__) will be tried.
__getattr__ is called when an attribute access fails with an AttributeError. Maybe this is why you think it 'catches' the errors. However, it doesn't, it's Python's attribute access functionality that catches them, and then calls __getattr__.
But __getattr__ itself doesn't catch any errors. If you raise an AttributeError in __getattr__ you get infinite recursion.
regularly run into this problem because I implement __getattr__ a lot and have lots of #property methods. Here's a decorator I came up with to get a more useful error message:
def replace_attribute_error_with_runtime_error(f):
#functools.wraps(f)
def wrapped(*args, **kwargs):
try:
return f(*args, **kwargs)
except AttributeError as e:
# logging.exception(e)
raise RuntimeError(
'{} failed with an AttributeError: {}'.format(f.__name__, e)
)
return wrapped
And use it like this:
class C(object):
def __getattr__(self, name):
...
#property
#replace_attribute_error_with_runtime_error
def complicated_property(self):
...
...
The error message of the underlying exception will include name of the class whose instance raised the underlying AttributeError.
You can also log it if you want to.
You're doomed anyways when you combine #property with __getattr__:
class Paradise:
pass
class Earth:
#property
def life(self):
print('Checking for paradise (just for fun)')
return Paradise.breasts
def __getattr__(self, item):
print("sorry! {} does not exist in Earth".format(item))
earth = Earth()
try:
print('Life in earth: ' + str(earth.life))
except AttributeError as e:
print('Exception found!: ' + str(e))
Gives the following output:
Checking for paradise (just for fun)
sorry! life does not exist in Earth
Life in earth: None
When your real problem was with calling Paradise.breasts.
__getattr__ is always called when an AtributeError is risen. The content of the exception is ignored.
The sad thing is that there's no solution to this problem given hasattr(earth, 'life') will return True (just because __getattr__ is defined), but will still be reached by the attribute 'life' as it didn't exist, whereas the real underlying problem is with Paradise.breasts.
My partial solution involves using a try-except in #property blocks which are known to hit upon AttributeError exceptions.
I've started to use the python descriptor protocol more extensively in the code I've been writing. Typically, the default python lookup magic is what I want to happen, but sometimes I'm finding I want to get the descriptor object itself instead the results of its __get__ method. Wanting to know the type of the descriptor, or access state stored in the descriptor, or somesuch thing.
I wrote the code below to walk the namespaces in what I believe is the correct ordering, and return the attribute raw regardless of whether it is a descriptor or not. I'm surprised though that I can't find a built-in function or something in the standard library to do this -- I figure it has to be there and I just haven't noticed it or googled for the right search term.
Is there functionality somewhere in the python distribution that already does this (or something similar)?
Thanks!
from inspect import isdatadescriptor
def namespaces(obj):
obj_dict = None
if hasattr(obj, '__dict__'):
obj_dict = object.__getattribute__(obj, '__dict__')
obj_class = type(obj)
return obj_dict, [t.__dict__ for t in obj_class.__mro__]
def getattr_raw(obj, name):
# get an attribute in the same resolution order one would normally,
# but do not call __get__ on the attribute even if it has one
obj_dict, class_dicts = namespaces(obj)
# look for a data descriptor in class hierarchy; it takes priority over
# the obj's dict if it exists
for d in class_dicts:
if name in d and isdatadescriptor(d[name]):
return d[name]
# look for the attribute in the object's dictionary
if obj_dict and name in obj_dict:
return obj_dict[name]
# look for the attribute anywhere in the class hierarchy
for d in class_dicts:
if name in d:
return d[name]
raise AttributeError
Edit Wed, Oct 28, 2009.
Denis's answer gave me a convention to use in my descriptor classes to get the descriptor objects themselves. But, I had an entire class hierarchy of descriptor classes, and I didn't want to begin every __get__ function with a boilerplate
def __get__(self, instance, instance_type):
if instance is None:
return self
...
To avoid this, I made the root of the descriptor class tree inherit from the following:
def decorate_get(original_get):
def decorated_get(self, instance, instance_type):
if instance is None:
return self
return original_get(self, instance, instance_type)
return decorated_get
class InstanceOnlyDescriptor(object):
"""All __get__ functions are automatically wrapped with a decorator which
causes them to only be applied to instances. If __get__ is called on a
class, the decorator returns the descriptor itself, and the decorated
__get__ is not called.
"""
class __metaclass__(type):
def __new__(cls, name, bases, attrs):
if '__get__' in attrs:
attrs['__get__'] = decorate_get(attrs['__get__'])
return type.__new__(cls, name, bases, attrs)
Most descriptors do their job when accessed as instance attribute only. So it's convenient to return itself when it's accessed for class:
class FixedValueProperty(object):
def __init__(self, value):
self.value = value
def __get__(self, inst, cls):
if inst is None:
return self
return self.value
This allows you to get descriptor itself:
>>> class C(object):
... prop = FixedValueProperty('abc')
...
>>> o = C()
>>> o.prop
'abc'
>>> C.prop
<__main__.FixedValueProperty object at 0xb7eb290c>
>>> C.prop.value
'abc'
>>> type(o).prop.value
'abc'
Note, that this works for (most?) built-in descriptors too:
>>> class C(object):
... #property
... def prop(self):
... return 'abc'
...
>>> C.prop
<property object at 0xb7eb0b6c>
>>> C.prop.fget
<function prop at 0xb7ea36f4>
Accessing descriptor could be useful when you need to extent it in subclass, but there is a better way to do this.
The inspect library provides a function to retrieve an attribute without any descriptor magic: inspect.getattr_static.
Documentation: https://docs.python.org/3/library/inspect.html#fetching-attributes-statically
(This is an old question, but I keep coming across it when trying to remember how to do this, so I'm posting this answer so I can find it again!)
The above method
class FixedValueProperty(object):
def __init__(self, value):
self.value = value
def __get__(self, inst, cls):
if inst is None:
return self
return self.value
Is a great method whenever you control the code of the property, but there are some cases, such as when the property is part of a library controlled by someone else, where another approach is useful. This alternative approach can also be useful in other situations such as implementing object mapping, walking a name-space as described in the question, or other specialised libraries.
Consider a class with a simple property:
class ClassWithProp:
#property
def value(self):
return 3
>>>test=ClassWithProp()
>>>test.value
3
>>>test.__class__.__dict__.['value']
<property object at 0x00000216A39D0778>
When accessed from the container objects class dict, the 'descriptor magic' is bypassed. Note also that if we assign the property to a new class variable, it behaves just like the original with 'descriptor magic', but if assigned to an instance variable, the property behaves as any normal object and also bypasses 'descriptor magic'.
>>> test.__class__.classvar = test.__class__.__dict__['value']
>>> test.classvar
3
>>> test.instvar = test.__class__.__dict__['value']
>>> test.instvar
<property object at 0x00000216A39D0778>
Let's say we want to get the descriptor for obj.prop where type(obj) is C.
C.prop usually works because the descriptor usually returns itself when accessed via C (i.e., bound to C). But C.prop may trigger a descriptor in its metaclass. If prop were not present in obj, obj.prop would raise AttributeError while C.prop might not. So it's better to use inspect.getattr_static(obj, 'prop').
If you are not satisfied with that, here's a CPython-specific method (from _PyObject_GenericGetAttrWithDict in Objects/object.c):
import ctypes, _ctypes
_PyType_Lookup = ctypes.pythonapi._PyType_Lookup
_PyType_Lookup.argtypes = (ctypes.py_object, ctypes.py_object)
_PyType_Lookup.restype = ctypes.c_void_p
def type_lookup(ty, name):
"""look for a name through the MRO of a type."""
if not isinstance(ty, type):
raise TypeError('ty must be a type')
result = _PyType_Lookup(ty, name)
if result is None:
raise AttributeError(name)
return _ctypes.PyObj_FromPtr(result)
type_lookup(type(obj), 'prop') returns the descriptor in the same way when CPython uses it at obj.prop if obj is a usual object (not class, for example).