We Keep Coding

Alternatives to decorators for saving metadata about classes

I'm writing a GUI library, and I'd like to let the programmer provide meta-information about their program which I can use to fine-tune the GUI. I was planning to use function decorators for this purpose, for example like this:
class App:
#Useraction(description='close the program', hotkey='ctrl+q')
def quit(self):
sys.exit()
The problem is that this information needs to be bound to the respective class. For example, if the program is an image editor, it might have an Image class which provides some more Useractions:
class Image:
#Useraction(description='invert the colors')
def invert_colors(self):
...
However, since the concept of unbound methods has been removed in python 3, there doesn't seem to be a way to find a function's defining class. (I found this old answer, but that doesn't work in a decorator.)
So, since it looks like decorators aren't going to work, what would be the best way to do this? I'd like to avoid having code like
class App:
def quit(self):
sys.exit()
Useraction(App.quit, description='close the program', hotkey='ctrl+q')
if at all possible.
For completeness' sake, the #Useraction decorator would look somewhat like this:
class_metadata= defaultdict(dict)
def Useraction(**meta):
def wrap(f):
cls= get_defining_class(f)
class_metadata[cls][f]= meta
return f
return wrap

You are using decorators to add meta data to methods. That is fine. It can be done e.g. this way:
def user_action(description):
def decorate(func):
func.user_action = {'description': description}
return func
return decorate
Now, you want to collect that data and store it in a global dictionary in form class_metadata[cls][f]= meta. For that, you need to find all decorated methods and their classes.
The simplest way to do that is probably using metaclasses. In metaclass, you can define what happens when a class is created. In this case, go through all methods of the class, find decorated methods and store them in the dictionary:
class UserActionMeta(type):
user_action_meta_data = collections.defaultdict(dict)
def __new__(cls, name, bases, attrs):
rtn = type.__new__(cls, name, bases, attrs)
for attr in attrs.values():
if hasattr(attr, 'user_action'):
UserActionMeta.user_action_meta_data[rtn][attr] = attr.user_action
return rtn
I have put the global dictionary user_action_meta_data in the meta class just because it felt logical. It can be anywhere.
Now, just use that in any class:
class X(metaclass=UserActionMeta):
#user_action('Exit the application')
def exit(self):
pass
Static UserActionMeta.user_action_meta_data now contains the data you want:
defaultdict(<class 'dict'>, {<class '__main__.X'>: {<function exit at 0x00000000029F36C8>: {'description': 'Exit the application'}}})

I've found a way to make decorators work with the inspect module, but it's not a great solution, so I'm still open to better suggestions.
Basically what I'm doing is to traverse the interpreter stack until I find the current class. Since no class object exists at this time, I extract the class's qualname and module instead.
import inspect
def get_current_class():
"""
Returns the name of the current module and the name of the class that is currently being created.
Has to be called in class-level code, for example:
def deco(f):
print(get_current_class())
return f
def deco2(arg):
def wrap(f):
print(get_current_class())
return f
return wrap
class Foo:
print(get_current_class())
#deco
def f(self):
pass
#deco2('foobar')
def f2(self):
pass
"""
frame= inspect.currentframe()
while True:
frame= frame.f_back
if '__module__' in frame.f_locals:
break
dict_= frame.f_locals
cls= (dict_['__module__'], dict_['__qualname__'])
return cls
Then in a sort of post-processing step, I use the module and class names to find the actual class object.
def postprocess():
global class_metadata
def findclass(module, qualname):
scope= sys.modules[module]
for name in qualname.split('.'):
scope= getattr(scope, name)
return scope
class_metadata= {findclass(cls[0], cls[1]):meta for cls,meta in class_metadata.items()}
The problem with this solution is the delayed class lookup. If classes are overwritten or deleted, the post-processing step will find the wrong class or fail altogether. Example:
class C:
#Useraction(hotkey='ctrl+f')
def f(self):
print('f')
class C:
pass
postprocess()

keep track of member variable access through instance

I am trying to write some module which keep track of member variable access
through instance.
1. is it possible to know member variable has access using instance at run time?
2. if yes, any design/pointer or idea
Purpose: I would like to write simple script which will read sample file(module) and member variable accessed by instance. So we can develop this as a part of debuging framework.
For example, if I write in main time.initial_time than my script able to detect that initial_time has been accessed by time Instance. it will be run at the run time. I mean, it will be part of existing flow
Real Purpose
The object contain 1000 value but some of them used by each module. if it's become debug framework so we can easily identify and print information of member variable access by instance. Yes each module create instance of data class.
Sample file
"""testing pylint code"""
#!/usr/bin/env py
class Sample(object):
"""create sample class"""
def __init__(self):
"""seting variable"""
self.intial_time = 0
def main():
"""main functionality"""
time = Sample()
print time.initial_time
if __name__ == " __main__":
main()

You can do it using descriptors.
Properties is a special case of descriptors but I believe they will not help you as much in this case.
Here is a descriptor that does exactly what you want:
from collections import defaultdict
class TrackedAttribute:
def __init__(self, default_value):
self.default = default_value
# Dict mapping an instance to it's value
self.instance_dict = defaultdict(lambda: default_value)
def __get__(self, inst, owner):
if inst is None:
print("Accessed from class %r" % (owner,))
return self.default
print("Accessed from instance %r" % (inst,))
return self.instance_dict[inst]
def __set__(self, inst, value):
print("Setting from instance %r" % (inst,))
self.instance_dict[inst] = value
class Simple:
time = TrackedAttribute(0)

There may be a better answer more suitable to your specific needs (trying to identify unused variables), but Python has a property decorator that you could use:
class Sample(object):
def __init__(self):
self._initial_time = 0
#property
def initial_time(self):
print('self.initial_time has been read')
return self._initial_time
>>> print(Sample().initial_time)
self.initial_time has been read
0
>>>

Normal attributes in a class implementing getattr / setattr [duplicate]

This question already has answers here:
How to use __setattr__ correctly, avoiding infinite recursion
(5 answers)
Closed 7 years ago.
I have this:
class MySession:
def __init__(self, session):
session['my-data'] = {} # my data is here
self._session = session
def __getattr__(self, name):
return self._session['my-data'][name]
def __setattr__(self, name, value):
my_data = self._session['my-data']
my_data[name] = value
self._session['my-data'] = my_data
obj = MySession({})
obj.x = 3
Basically I want to encapsulate access to the session (sub-)dictionary with an object attribute access. But I can not do it, since this causes infinite recursion, I guess because doing this:
self._session = session
calls setattr, which in turn calls getattr, which in turn calls getattr, etc
How can I pre-initialize some (normal) attributes in a class implementing getattr / setattr?

The __getattr__ method is only called for attributes that don't exist in the normal attribute dictionary. __setattr__ however is called unconditionally (it's mirror is really __getattribute__ rather than __getattr__). If you can get your _session attribute set up properly in __init__, you won't need to worry about anything in the other methods.
To add an attribute without running into any recursion, use super(MySession, self).__setattr__ to call the version of the method you inherited from object (you should always inherit from object in Python 2, to make your class a new-style class, in Python 3, it's the default). You could also call object.__setattr__ directly, but using super is better if you ever end up using multiple inheritance.
class MySession(object):
def __init__(self, session):
session['my-data'] = {}
super(MySession, self).__setattr__("_session", session) # avoid our __setattr__
def __getattr__(self, name):
return self._session['my-data'][name] # this doesn't recurse if _session exists
def __setattr__(self, name, value):
my_data = self._session['my-data']
my_data[name] = value
self._session['my-data'] = my_data

You could initialize first and change the setters getters later on:
def __init__(self, session):
session['my-data'] = {}
self._session = session
self.__setattr__ = self._setattr
self.__getattr__ = self._getattr
assuming self._setattr and self._getattr are implemented of course :)

Python - Exists a function that is called when an object does not implement a function?

In Smalltalk there is a message DoesNotUnderstand that is called when an object does not understand a message (this is, when the object does not have the message sent implemented).
So, I like to know if in python there is a function that does the same thing.
In this example:
class MyObject:
def __init__(self):
print "MyObject created"
anObject = MyObject() # prints: MyObject created
anObject.DoSomething() # raise an Exception
So, can I add a method to MyObject so I can know when DoSomething is intented to be called?
PS: Sorry for my poor English.

Here is a proposition for what you want to do:
class callee:
def __init__(self, name):
self.name = name
def __call__(self):
print self.name, "has been called"
class A:
def __getattr__(self, attr):
return callee(attr)
a = A()
a.DoSomething()
>>> DoSomething has been called

You are looking for the __getattr__ method. Have a look here.
If you want "total control" of a class, look at the __getattribute__ special method then (here).

Have you looked at object.__getattr__(self, name), or object.__getattribute__(self, name) for new-style classes? (see Special method names, Python language reference)

I don't know why luc had the two separate classes. You can do it all with one class, if you use a closure. Like so:
class A(object):
__ignored_attributes__ = set(["__str__"])
def __getattr__(self, name):
if __name__ in self.__ignored_attributes__:
return None
def fn():
print name, "has been called with self =", self
return fn
a = A()
a.DoSomething()
I added the bit about __ignored_attributes__ because Python was looking up __str__ in the class and that got a little messy.

python attribute lookup without any descriptor magic?

I've started to use the python descriptor protocol more extensively in the code I've been writing. Typically, the default python lookup magic is what I want to happen, but sometimes I'm finding I want to get the descriptor object itself instead the results of its __get__ method. Wanting to know the type of the descriptor, or access state stored in the descriptor, or somesuch thing.
I wrote the code below to walk the namespaces in what I believe is the correct ordering, and return the attribute raw regardless of whether it is a descriptor or not. I'm surprised though that I can't find a built-in function or something in the standard library to do this -- I figure it has to be there and I just haven't noticed it or googled for the right search term.
Is there functionality somewhere in the python distribution that already does this (or something similar)?
Thanks!
from inspect import isdatadescriptor
def namespaces(obj):
obj_dict = None
if hasattr(obj, '__dict__'):
obj_dict = object.__getattribute__(obj, '__dict__')
obj_class = type(obj)
return obj_dict, [t.__dict__ for t in obj_class.__mro__]
def getattr_raw(obj, name):
# get an attribute in the same resolution order one would normally,
# but do not call __get__ on the attribute even if it has one
obj_dict, class_dicts = namespaces(obj)
# look for a data descriptor in class hierarchy; it takes priority over
# the obj's dict if it exists
for d in class_dicts:
if name in d and isdatadescriptor(d[name]):
return d[name]
# look for the attribute in the object's dictionary
if obj_dict and name in obj_dict:
return obj_dict[name]
# look for the attribute anywhere in the class hierarchy
for d in class_dicts:
if name in d:
return d[name]
raise AttributeError
Edit Wed, Oct 28, 2009.
Denis's answer gave me a convention to use in my descriptor classes to get the descriptor objects themselves. But, I had an entire class hierarchy of descriptor classes, and I didn't want to begin every __get__ function with a boilerplate
def __get__(self, instance, instance_type):
if instance is None:
return self
...
To avoid this, I made the root of the descriptor class tree inherit from the following:
def decorate_get(original_get):
def decorated_get(self, instance, instance_type):
if instance is None:
return self
return original_get(self, instance, instance_type)
return decorated_get
class InstanceOnlyDescriptor(object):
"""All __get__ functions are automatically wrapped with a decorator which
causes them to only be applied to instances. If __get__ is called on a
class, the decorator returns the descriptor itself, and the decorated
__get__ is not called.
"""
class __metaclass__(type):
def __new__(cls, name, bases, attrs):
if '__get__' in attrs:
attrs['__get__'] = decorate_get(attrs['__get__'])
return type.__new__(cls, name, bases, attrs)

Most descriptors do their job when accessed as instance attribute only. So it's convenient to return itself when it's accessed for class:
class FixedValueProperty(object):
def __init__(self, value):
self.value = value
def __get__(self, inst, cls):
if inst is None:
return self
return self.value
This allows you to get descriptor itself:
>>> class C(object):
... prop = FixedValueProperty('abc')
...
>>> o = C()
>>> o.prop
'abc'
>>> C.prop
<__main__.FixedValueProperty object at 0xb7eb290c>
>>> C.prop.value
'abc'
>>> type(o).prop.value
'abc'
Note, that this works for (most?) built-in descriptors too:
>>> class C(object):
... #property
... def prop(self):
... return 'abc'
...
>>> C.prop
<property object at 0xb7eb0b6c>
>>> C.prop.fget
<function prop at 0xb7ea36f4>
Accessing descriptor could be useful when you need to extent it in subclass, but there is a better way to do this.

The inspect library provides a function to retrieve an attribute without any descriptor magic: inspect.getattr_static.
Documentation: https://docs.python.org/3/library/inspect.html#fetching-attributes-statically
(This is an old question, but I keep coming across it when trying to remember how to do this, so I'm posting this answer so I can find it again!)

The above method
class FixedValueProperty(object):
def __init__(self, value):
self.value = value
def __get__(self, inst, cls):
if inst is None:
return self
return self.value
Is a great method whenever you control the code of the property, but there are some cases, such as when the property is part of a library controlled by someone else, where another approach is useful. This alternative approach can also be useful in other situations such as implementing object mapping, walking a name-space as described in the question, or other specialised libraries.
Consider a class with a simple property:
class ClassWithProp:
#property
def value(self):
return 3
>>>test=ClassWithProp()
>>>test.value
3
>>>test.__class__.__dict__.['value']
<property object at 0x00000216A39D0778>
When accessed from the container objects class dict, the 'descriptor magic' is bypassed. Note also that if we assign the property to a new class variable, it behaves just like the original with 'descriptor magic', but if assigned to an instance variable, the property behaves as any normal object and also bypasses 'descriptor magic'.
>>> test.__class__.classvar = test.__class__.__dict__['value']
>>> test.classvar
3
>>> test.instvar = test.__class__.__dict__['value']
>>> test.instvar
<property object at 0x00000216A39D0778>

Let's say we want to get the descriptor for obj.prop where type(obj) is C.
C.prop usually works because the descriptor usually returns itself when accessed via C (i.e., bound to C). But C.prop may trigger a descriptor in its metaclass. If prop were not present in obj, obj.prop would raise AttributeError while C.prop might not. So it's better to use inspect.getattr_static(obj, 'prop').
If you are not satisfied with that, here's a CPython-specific method (from _PyObject_GenericGetAttrWithDict in Objects/object.c):
import ctypes, _ctypes
_PyType_Lookup = ctypes.pythonapi._PyType_Lookup
_PyType_Lookup.argtypes = (ctypes.py_object, ctypes.py_object)
_PyType_Lookup.restype = ctypes.c_void_p
def type_lookup(ty, name):
"""look for a name through the MRO of a type."""
if not isinstance(ty, type):
raise TypeError('ty must be a type')
result = _PyType_Lookup(ty, name)
if result is None:
raise AttributeError(name)
return _ctypes.PyObj_FromPtr(result)
type_lookup(type(obj), 'prop') returns the descriptor in the same way when CPython uses it at obj.prop if obj is a usual object (not class, for example).