This question already has answers here:
Decorating class methods - how to pass the instance to the decorator?
(3 answers)
Closed 3 years ago.
I am new to Python decorators (wow, great feature!), and I have trouble getting the following to work because the self argument gets sort of mixed up.
#this is the decorator
class cacher(object):
def __init__(self, f):
self.f = f
self.cache = {}
def __call__(self, *args):
fname = self.f.__name__
if (fname not in self.cache):
self.cache[fname] = self.f(self,*args)
else:
print "using cache"
return self.cache[fname]
class Session(p.Session):
def __init__(self, user, passw):
self.pl = p.Session(user, passw)
#cacher
def get_something(self):
print "get_something called with self = %s "% self
return self.pl.get_something()
s = Session(u,p)
s.get_something()
When I run this, I get:
get_something called with self = <__main__.cacher object at 0x020870F0>
Traceback:
...
AttributeError: 'cacher' object has no attribute 'pl'
for the line where I do self.cache[fname] = self.f(self,*args)
The problem - Obviously, the problem is that self is the cacher object instead of a Session instance, which indeed doesn't have a pl attribute. However I can't find how to solve this.
Solutions I've considered, but can't use - I thought of making the decorator class return a function instead of a value (like in section 2.1 of this article) so that self is evaluated in the right context, but that isn't possible since my decorator is implemented as a class and uses the build-in __call__ method. Then I thought to not use a class for my decorator, so that I don't need the __call__method, but I can't do that because I need to keep state between decorator calls (i.e. for keeping track of what is in the self.cache attribute).
Question - So, apart from using a global cache dictionary variable (which I didn't try, but assume will work), is there any other way to make this decorator work?
Edit: this SO question seems similar Decorating python class methods, how do I pass the instance to the decorator?
Use the descriptor protocol like this:
import functools
class cacher(object):
def __init__(self, f):
self.f = f
self.cache = {}
def __call__(self, *args):
fname = self.f.__name__
if (fname not in self.cache):
self.cache[fname] = self.f(self,*args)
else:
print "using cache"
return self.cache[fname]
def __get__(self, instance, instancetype):
"""Implement the descriptor protocol to make decorating instance
method possible.
"""
# Return a partial function with the first argument is the instance
# of the class decorated.
return functools.partial(self.__call__, instance)
Edit :
How it's work ?
Using the descriptor protocol in the decorator will allow us to access the method decorated with the correct instance as self, maybe some code can help better:
Now when we will do:
class Session(p.Session):
...
#cacher
def get_something(self):
print "get_something called with self = %s "% self
return self.pl.get_something()
equivalent to:
class Session(p.Session):
...
def get_something(self):
print "get_something called with self = %s "% self
return self.pl.get_something()
get_something = cacher(get_something)
So now get_something is an instance of cacher . so when we will call the method get_something it will be translated to this (because of the descriptor protocol):
session = Session()
session.get_something
# <==>
session.get_something.__get__(get_something, session, <type ..>)
# N.B: get_something is an instance of cacher class.
and because :
session.get_something.__get__(get_something, session, <type ..>)
# return
get_something.__call__(session, ...) # the partial function.
so
session.get_something(*args)
# <==>
get_something.__call__(session, *args)
Hopefully this will explain how it work :)
Closures are often a better way to go, since you don't have to muck about with the descriptor protocol. Saving mutable state across calls is even easier than with a class, since you just stick the mutable object in the containing scope (references to immutable objects can be handled either via the nonlocal keyword, or by stashing them in a mutable object like a single-entry list).
#this is the decorator
from functools import wraps
def cacher(f):
# No point using a dict, since we only ever cache one value
# If you meant to create cache entries for different arguments
# check the memoise decorator linked in other answers
print("cacher called")
cache = []
#wraps(f)
def wrapped(*args, **kwds):
print ("wrapped called")
if not cache:
print("calculating and caching result")
cache.append(f(*args, **kwds))
return cache[0]
return wrapped
class C:
#cacher
def get_something(self):
print "get_something called with self = %s "% self
C().get_something()
C().get_something()
If you aren't completely familiar with the way closures work, adding more print statements (as I have above) can be illustrative. You will see that cacher is only called as the function is defined, but wrapped is called each time the method is called.
This does highlight how you need to be careful with memoisation techniques and instance methods though - if you aren't careful to account for changes in the value of self, you will end up sharing cached answers across instances, which may not be what you want.
First, you explicitly pass cacher object as first argument in the following line:
self.cache[fname] = self.f(self,*args)
Python automatically adds self to the list of arguments for methods only. It converts functions (but not other callables as your cacher object!) defined in class namespace to methods. To get such behavior I see two ways:
Change your decorator to return function by using closures.
Implement descriptor protocol to pass self argument yourself as it's done in memoize decorator recipe.
Related
I'm trying to implement a decorator which accepts some arguments. Usually decorators with arguments are implemented as double-nested closures, like this:
def mydecorator(param1, param2):
# do something with params
def wrapper(fn):
def actual_decorator(actual_func_arg1, actual_func_arg2):
print("I'm decorated!")
return fn(actual_func_arg1, actual_func_arg2)
return actual_decorator
return wrapper
But personally I don't like such approach because it is very unreadable and difficult to understand.
So I ended up with this:
class jsonschema_validate(object):
def __init__(self, schema):
self._schema = schema
def __call__(self, fn):
self._fn = fn
return self._decorator
def _decorator(self, req, resp, *args, **kwargs):
try:
jsonschema.validate(req.media, self._schema, format_checker=jsonschema.FormatChecker())
except jsonschema.ValidationError as e:
_log.exception('Validation failed: %r', e)
raise errors.HTTPBadRequest('Bad request')
return self._fn(req, resp, *args, **kwargs)
The idea is very simple: at instantiation time we just captures decorator args, and at call time we capture decorated function and return decorator instance's method, which is bound. It is important it to be bound because at decorator's invocation time we want to access self with all information stored in it.
Then we use it on some class:
class MyResource(object):
#jsonschema_validate(my_resource_schema)
def on_post(self, req, resp):
pass
Unfortunately, this approach doesn't work. The problem is that at decorator invocation time we looses context of decorated instance because at decoration time (when defining class) decorated method is not bound. Binding occurs later at attribute access time. But at this moment we already have decorator's bound method (jsonschema_validate._decorator) and self is passed implicitly, and it's value isn't MyResource instance, rather jsonschema_validate instance. And we don't want to loose this self value because we want to access it's attributes at decorator invocation time. In the end it results in TypeError when calling self._fn(req, resp, *args, **kwargs) with complains that "required positional argument 'resp' is missing" because passed in req arg becomes MyResource.on_post "self" and all arguments effectively "shifts".
So, is there a way implement decorator as a class rather than as a bunch of nested functions?
Note
As my first attempt of implementing decorator as simple class was failed rather quickly, I immediately reverted to nested functions. It seems like properly implemented class approach is even more unreadable and tangled, but I want to find solution anyway for the fun of the thing.
UPDATE
Finally found solution, see my own answer.
This is fun! Thanks for posting this question.
Writing a simple decorator that doesn't take arguments is pretty easy, but extending that to a class that then gets called three times is a bit more challenging. I opted to use a functools.partial to solve this problem.
from functools import partial, update_wrapper
from unittest import TestCase, main
class SimpleDecorator(object):
def __new__(cls, func, **params):
self = super(SimpleDecorator, cls).__new__(cls)
self.func = func
self.params = params
return update_wrapper(self, func)
def __call__(self, *args, **kwargs):
args, kwargs = self.before(*args, **kwargs)
return self.after(self.func(*args, **kwargs))
def after(self, value):
return value
def before(self, *args, **kwargs):
return args, kwargs
class ParamsDecorator(SimpleDecorator):
def __new__(cls, **params):
return partial(super(ParamsDecorator, cls).__new__, cls, **params)
class DecoratorTestCase(TestCase):
def test_simple_decorator(self):
class TestSimpleDecorator(SimpleDecorator):
def after(self, value):
value *= 2
return super().after(value)
#TestSimpleDecorator
def _test_simple_decorator(value):
"""Test simple decorator"""
return value + 1
self.assertEqual(_test_simple_decorator.__name__, '_test_simple_decorator')
self.assertEqual(_test_simple_decorator.__doc__, 'Test simple decorator')
self.assertEqual(_test_simple_decorator(1), 4)
def test_params_decorator(self):
class TestParamsDecorator(ParamsDecorator):
def before(self, value, **kwargs):
value *= self.params['factor']
return super().before(value, **kwargs)
#TestParamsDecorator(factor=3)
def _test_params_decorator(value):
"""Test params decorator"""
return value + 1
self.assertEqual(_test_params_decorator.__name__, '_test_params_decorator')
self.assertEqual(_test_params_decorator.__doc__, 'Test params decorator')
self.assertEqual(_test_params_decorator(2), 7)
As you can see I've opted for a design with hooks for modifying the arguments and responses in methods. Hopefully, most of the time this would prevent needing to touch __call__ or __new__.
I couldn't think of a way to attach params to ParamsDecorator after returning the partial, so I had to opt for putting it into the SimpleDecorator but not using it.
I think that this does a good job of keeping the content flat instead of nested. I also like that this can take care of functools.wraps for you so you shouldn't need to worry about including that on these decorators. The downside to writing a decorator this way is you're now introducing a new module that you would need to install or maintain and then import every time you write a decorator.
Finally got it!
As I wrote, the problem that a method can't have two self, so we need to capture both values in some way. Descriptors and closures to the rescue!
Here is complete example:
class decorator_with_args(object):
def __init__(self, arg):
self._arg = arg
def __call__(self, fn):
self._fn = fn
return self
def __get__(self, instance, owner):
if instance is None:
return self
def _decorator(self_, *args, **kwargs):
print(f'decorated! arg: {self._arg}')
return self._fn(self_, *args, **kwargs)
return _decorator.__get__(instance, owner)
Let's break it down to pieces!
It starts exactly as my previous attempt. In __init__ we just capture decorator arguments to it's private attribute(s).
Things get more interested in next part: a __call__ method.
def __call__(self, fn):
self._fn = fn
return self
As before, we capture decorated method to decorator's private attribute. But then, instead of returning actual decorator method (def _decorator in previous example), we return self. So decorated method becomes instance of decorator. This is required to allow it to act as descriptor. According to docs:
a descriptor is an object attribute with "binding behavior"
Confusing, uh? Actually, it is easier than it looks. Descriptor is just an object with "magic" (dunder) methods which is assigned to another's object attribute. When you try to access this attribute, those dunder methods will be invoked with some calling convention. And we'll return to "binding behavior" a little bit later.
Let's look at the details.
def __get__(self, instance, owner):
Descriptor must implement at least __get__ dunder (and __set__ & __delete__ optionally). This is called "descriptor protocol" (similar to "context manager protocol", "collection protocol" an so on).
if instance is None:
return self
This is by convention. When descriptor accessed on class rather than instance, it should return itself.
Next part is most interesting.
def _decorator(self_, *args, **kwargs):
print(f'decorated! arg: {self._arg}')
return self._fn(self_, *args, **kwargs)
return _decorator.__get__(instance, owner)
We need to capture decorator's self as well as decorated instance's self in some way. As we can't define function with two self (even if we can, Python couldn't understand us), so we enclose decorator's self with closure - an inner function. In this closure, we actually do alter behavior of decorated method (print('decorated! arg: {self._arg}')) and then call original one. Again, as there is already argument named self, we need to choose another name for instance's self - in this example I named it self_, but actually it is self' - "self prime" (kinda math humor).
return _decorator.__get__(instance, owner)
And finally, usually, when we define closures, we just return it: def inner(): pass; return inner. But here we can't do that. Because of "binding behavior". What we need is returned closure to be bound to decorated instance in order it to work properly. Let me explain with an example.
class Foo(object):
def foo(self):
print(self)
Foo.foo
# <function Foo.foo at 0x7f5b1f56dcb0>
Foo().foo
# <bound method Foo.foo of <__main__.Foo object at 0x7f5b1f586910>>
When you access method on class, it is just a plain Python function. What makes it a method instead is binding. Binding is an act of linking object's methods with instance which is passed implicitly as firs argument. By convention, it is called self, but roughly speaking this is not required. You can even store method in other variable and call it, and will still have reference to instance:
f = Foo()
f.foo()
# <__main__.Foo object at 0x7f5b1f5868d0>
other_foo = f.foo
other_foo()
# <__main__.Foo object at 0x7f5b1f5868d0>
So, we need to bind our returned closure to decorated instance. How to do that? Remember when we was looking at method? It could be something like that:
# <bound method Foo.foo of <__main__.Foo object at 0x7f5b1f586910>>
Let's look at it's type:
type(f.foo)
# <class 'method'>
Wow! It actually even a class! Let's create it!
method()
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# NameError: name 'method' is not defined
Unfortunately, we can't do that directly. But, there is types.MethodType:
types.MethodType
# <class 'method'>
Seems like we finally found what we wanted! But, actually, we don't need to create method manually!. All we need to do is to delegate to standard mechanics of method creation. And this is how actually methods work in Python - they are just descriptors which bind themselves to an instances when accessed as instance's attribute!
To support method calls, functions include the __get__() method for binding methods during attribute access.
So, we need to just delegate binding machinery to function itself:
_decorator.__get__(instance, owner)
and get method with right binding!
I have subclassed the built-in property class (call it SpecialProperty) in order to add more fields to it:
class SpecialProperty(property):
extra_field_1 = None
extra_field_2 = None
def __init__(self, fget=None, fset=None, fdel=None, doc=None):
super().__init__(fget, fset, fdel, doc)
def make_special_property(func):
prop = SerialisableProperty(fget=func)
return prop
and I am able to use it in the same fashion as the built-in property() decorator:
#my_module.make_special_property
def get_my_property(self): return self._my_property
I now want to further specialise my SpecialProperty instances populating one of the extra fields I have added to the class with an arbitrary value.
Is it possible, in Python, to write a decorator that will return a property with also accepting extra parameters?
I'd like to do it via the decorator because this is where and when the information is most relevant, however I'm finding myself stuck. I suspect this falls under the domain of decorators with arguments that have been well documented (Decorators with arguments? (Stack Overflow), or Python Decorators II: Decorator Arguments (artima.com) to only cite a couple sources), however I find myself unable to apply the same pattern to my case.
Here's how I'm trying to write it:
#my_module.make_special_property("example string")
def get_my_property(self): return self._my_property
And on the class declaring get_my_property:
>>> DeclaringClass.my_property
<SpecialProperty object at 0x...>
>>> DeclaringClass.my_property.extra_field_1
'example string'
Since I am making properties, the decorated class member should be swapped with an instance of SpecialProperty, and hence should not be a callable anymore -- thus, I am unable to apply the "nested wrapper" pattern for allowing a decorator with arguments.
Non working example:
def make_special_property(custom_arg_1):
def wrapper(func):
prop = SerialisableProperty(fget=func)
prop.extra_field_1 = custom_arg_1
return prop
return wrapper # this returns a callable (function)
I shouldn't have a callable be returned here, if I want a property I should have a SpecialProperty instance be returned, but I can't call return wrapper(func) for obvious reasons.
Your decorator doesn't return a callable. Your decorator factory returns a decorator, which returns a property. You might understand better if you rename the functions:
def make_decorator(custom_arg_1):
def decorator(func):
prop = SerialisableProperty(fget=func)
prop.extra_field_1 = custom_arg_1
return prop
return decorator
When you decorate with make_decorator, it is called with an argument, and decorator is returned and called on the decorated function.
I am trying to write decorator that will do different job depending on that whether it will be decorating method or a function.
I tried something like this:
def dec(f):
def wrapper(*v, **kv):
if '__class__' in dir(f):
text = "Its a method"
else:
text = "Its a function"
print(text)
return f(*v, **kv)
return wrapper
class A:
#dec
def f(self):
pass
#dec
def f():
pass
a = A()
a.f()
f()
It returns:
Its a method
Its a method
But this won't work since all functions also have also __class__ attribute :) I tried also to do inspect.ismethod on A.f but A.f is treaten just like function but in A namespace and it return of course false. It would return True if we have done inspect.ismethod(a.f) where a is instance of A but I can't wait to have instance.
As Martijn already explained in his comments, being defined in a class statement's body doesn't make a function a method - the method object is created at lookup time, as explained here - so what you are decorating is and will always (well almost cf below) be a function. FWIW, Python's "methods" are just thin wrappers around the function, class and (eventually) instance.
The only case where you might get something else than a function are classmethods and staticmethods (in which case you'll get resp. a classmethod or staticmethod object instead), and even then it depends on the decorators order.
If you want to get at the method's current instance (or class) in your decorator, you know that it will always be the first positional argument - but this might not be of great help... To make a long story short, you have to either explicitely tell your decorator if it should treat the function as plain function or method, or decorate it with a custom descriptor in which case you'll know if the function has - or not - been looked up as a method... A naïve implementation migh look like this:
class MyDecorator(object):
def __init__(self, func, instance=None):
self.func = func
self.instance = instance
def __call__(self, *args, **kw):
if self.instance:
print "is method - instance : %s" % self.instance
args = (self.instance,) + args
else:
print "is_function"
return self.func(*args, **kw)
def __get__(self, instance, cls):
return type(self)(self.func, instance)
I'm trying to write a class method decorator that modifies its class' state. I'm having troubles implementing it at the moment.
Side question: When does a decorator get called? Does it load when the class is instantiated or on during read time when the class read?
What I'm trying to do is this:
class ObjMeta(object):
methods = []
# This should be a decorator that magically updates the 'methods'
# attribute (or list) of this class that's being read by the proxying
# class below.
def method_wrapper(method):
#functools.wraps(method)
def wrapper(*args, **kwargs):
ObjMeta.methods.append(method.__name__)
return method(*args, **kwargs)
return wrapper
# Our methods
#method_wrapper
def method1(self, *args):
return args
#method_wrapper
def method2(self, *args):
return args
class Obj(object):
klass = None
def __init__(self, object_class=ObjMeta):
self.klass = object_class
self._set_methods(object_class)
# We dynamically load the method proxies that calls to our meta class
# that actually contains the methods. It's actually dependent to the
# meta class' methods attribute that contains a list of names of its
# existing methods. This is where I wanted it to be done automagically with
# the help of decorators
def _set_methods(self, object_class):
for method_name in object_class:
setattr(self, method_name, self._proxy_method(method_name))
# Proxies the method that's being called to our meta class
def _proxy_method(self, method_name):
def wrapper(*fargs, **fkwargs):
return getattr(self.klass(*fargs, **fkwargs), method_name)
return wrapper()
I think it's ugly to write a list of methods manually in the class so perhaps a decorator would fix this.
It's for an open-source project I'm working that ports underscore.js to python. I understand that it says I should just use itertools or something. I'm just doing this just for the love of programming and learning. BTW, project is hosted here
Thanks!
There are a few things wrong here.
Anything inside the inner wrapper is called when the method itself is called. Basically, you're replacing the method with that function, which wraps the original. So, your code as it stands would add the method name to the list each time it is called, which probably isn't what you want. Instead, that append should be at the method_wrapper level, ie outside of the inner wrapper. This is called when the method is defined, which happens the first time the module containing the class is imported.
The second thing wrong is that you never actually call the method - you simply return it. Instead of return method you should be returning the value of calling the method with the supplied args - return method(*args, **kwargs).
I have the following decorator, which saves a configuration file after a method decorated with #saveconfig is called:
class saveconfig(object):
def __init__(self, f):
self.f = f
def __call__(self, *args):
self.f(object, *args)
# Here i want to access "cfg" defined in pbtools
print "Saving configuration"
I'm using this decorator inside the following class. After the method createkvm is called, the configuration object self.cfg should be saved inside the decorator:
class pbtools()
def __init__(self):
self.configfile = open("pbt.properties", 'r+')
# This variable should be available inside my decorator
self.cfg = ConfigObj(infile = self.configfile)
#saveconfig
def createkvm(self):
print "creating kvm"
My problem is that i need to access the object variable self.cfg inside the decorator saveconfig. A first naive approach was to add a parameter to the decorator which holds the object, like #saveconfig(self), but this doesn't work.
How can I access object variables of the method host inside the decorator? Do i have to define the decorator inside the same class to get access?
You have to make your decorator class behave as a descriptor to be able to access the instance:
class saveconfig(object):
def __init__(self, f):
self.f = f
def __get__(self, instance, owner):
def wrapper(*args):
print "Saving configuration"
print instance.cfg
return self.f(instance, *args)
return wrapper
Your code passes object as first parameter to self.f(), where it should pass the pbtools instance.
You are passing object as self to the decorated method. The thing is, you can't easily get self because Python sees the decorated method, which is now an objects, and doesn't consider it a method (that should be passed self when called - or, more generally, that should work as a property returning a bound method). You can work around this, as pointed out by #Sven Marnach.
However, you could easily rewrite this decorator without a class, using a closure (is a bit shorter and also solves the above problem):
def saveconfig(f):
#functools.wraps(f) # to preserve name, docstring, etc.
def wrapper(*args, **kwargs): # **kwargs for compability with functions that use them
f(*args, **kwargs)
# save config
return wrapper
Other notes:
Terminology mixup: There is no class variable in the example. A class variable would be x = ... indented as far as the method definitions and be shared between all instances (specifically, it would be an attribute of the object that is pbtools) - everything on self is an instance attribute.
At class definition time (when you define methods, apply decorators, etc.) there is no self!
You can also use a simple function for what you want:
def saveconfig(f):
# this method replaces the decorated, so `self` will be the pbtools instance
def wrapped(self, *args):
f(self, *args)
# Here i want to access "cfg" defined in pbtools
print "Saving configuration", self.cfg
return wrapped
If saveconfig must be a class then you need Sven's solution.