I am learning to use decorators and I can't figure out how to pass already defined attributes to the wrapper without making a function specific decorator.
Let's say I have a decorator :
def decorator(func):
def wrapper():
print("Before the function")
func()
print("After the function")
return wrapper
With this I am only able to use it with functions with only defined attributes or without any attribute like :
#decorator
def foo1(attribute1=10, attribute2=20):
print(attribute1, attribute2)
return
foo1()
But it makes me unable to run :
foo1(1, 2)
With this problem, I also can't use this decorator on different functions that don't have the same amount of attributes to set.
So, it there a way to fix this problem without the use of *args and **kwargs or at least without having to call a function that would look like this : foo((arg1, arg2, argn))? Because it would make me unable to define any attribute. This is my only restrain.
Thanks.
The wrapper has to accept arguments (because it replaces the original function bound to the decorated name), and those arguments have to be passed to func.
def decorator(func):
def wrapper(*args, **kwargs):
print("Before the function")
func(*args, **kwargs)
print("After the function")
return wrapper
I have a series of functions in a module, for example
def some_func1(param1, param2): pass
def another_func(param1, param2): pass
I would like the easiest way to add them to the list of called functions and then call some of them at a certain point by filter, with parameters. I tried to add to a list of functions using a decorator, something like this:
all_func = {}
def decorator(filter):
def function_decorator(func):
def wrapped(*args, **kwargs):
return func(*args, **kwargs)
return wrapped
all_func[filter] = function_decorator
return function_decorator
#decorator(filter='1')
def some_func1(param1, param2): pass
#decorator(filter='2')
def another_func(param1, param2): pass
#decorator(filter='3')
def another_big_func(param1, *arg): pass
and then when using a loop using a filter, call the functions that match the filter:
arg1 = some_obj
arg2 = another_obj
for i in range(3, 4):
func = all_func.get(i)
func(arg1, arg2)
That is, when adding a function, I simply prescribe a new filter value, not caring that I need to register something else somewhere, and then I call them on this filter
But something went wrong, at the time of all_func.get (i) I get a function_decorator, not a function, and I don’t understand how to call it with the given parameters.
I haven’t worked with decorators before, maybe I somehow misunderstood the concept.
You registered the decorator, not the wrapper. Note that what you named decorator() is really a decorator factory, calling decorator(...) produces the decorator function function_decorator(). Python then calls the function_decorator() function to do the actual decorating task, and its return value is used to replace the decorated function. You want to register the result of that process, not function_decorator itself.
Register the wrapper, wrapped(), instead:
def decorator(filter):
def function_decorator(func):
def wrapped(*args, **kwargs):
return func(*args, **kwargs)
all_func[filter] = wrapped
return wrapped
return function_decorator
Now all_func.get(i) will return one of the wrapped() functions. Calling wrapped() then calls the decorated function, via the func reference.
Your wrapper() is really a no-op, so can be omitted. Decorators don't have to return a replacement function, the original function can be returned unchanged. Use the func reference to add to your registry:
def decorator(filter):
def function_decorator(func):
all_func[filter] = func
return func
return function_decorator
Note that you don't have a list, you have a dictionary, a mapping from key to value. Not that matters to the problem.
Slightly unrelated (cf Martijn's answer for the core issue), but:
1/ all_funcs is a dict, not a list
2/ if you're using strings as keys when populating your dict (#decorator(filter='1'), you have to use strings as keys too to get your functions back (range() yields ints, not strings), so you want
for key in map(str, range(3, 4)):
func = all_funcs.get(key)
I was wondering if there was a way for a decorated function to refer to an object created by the wrapper of a decorator. My question arose when I was thinking to use a decorator to :
make a wrapper that creates a figure with subplots
inside the wrapper execute the decorated function which would add some plots
finally save the figure in the wrapper
However, the decorated function would need to refer the figure created by the wrapper. How can the decorated function refer to that object ? Do we necessarily have to resort to global variables ?
Here is a short example where I reference in the decorated function a variable created in the wrapper (but I did not manage to do this without tweaking with globals):
def my_decorator(func):
def my_decorator_wrapper(*args, **kwargs):
global x
x = 0
print("x in wrapper:", x)
return func(*args, **kwargs)
return my_decorator_wrapper
#my_decorator
def decorated_func():
global x
x += 1
print("x in decorated_func:", x)
decorated_func()
# prints:
# x in wrapper: 0
# x in decorated_func: 1
I know this would be easily done in a class, but I am asking this question out of curiosity.
Try to avoid using global variables.
Use arguments to pass objects to functions
There is one canonical way to pass a value to a function: arguments.
Pass the object as argument to the decorated function when the wrapper is called.
from functools import wraps
def decorator(f):
obj = 1
#wraps(f)
def wrapper(*args):
return f(obj, *args)
return wrapper
#decorator
def func(x)
print(x)
func() # prints 1
Use a default argument for passing the same object
If you need to pass the same object to all functions, storing it as default argument of your decorator is an alternative.
from functools import wraps
def decorator(f, obj={}):
#wraps(f)
def wrapper(*args):
return f(obj, *args)
return wrapper
#decorator
def func(params)
params['foo'] = True
#decorator
def gunc(params)
print(params)
func()
# proof that gunc receives the same object
gunc() # prints {'foo': True}
The above creates a common private dict which can only be accessed by decorated functions. Since a dict is mutable, changes will be reflected across function calls.
Yes, the function can refer to it by looking at itself.
the decorator end. it just takes attributes and sets them on the function
if it looks a bit complicated, that's because decorators that take parameters need this particular structure to work. see Decorators with parameters?
def declare_view(**kwds):
"""declaratively assocatiate a Django View function with resources
"""
def actual_decorator(func):
for k, v in kwds.items():
setattr(func, k, v)
return func
return actual_decorator
calling the decorator
#declare_view(
x=2
)
def decorated_func():
#the function can look at its own name, because the function exists
#by the time it gets called.
print("x in decorated_func:", decorated_func.x)
decorated_func()
output
x in decorated_func: 2
In practice I've used this quite a bit. The idea for me is to associate Django view functions with particular backend data classes and templates they have to collaborate with. Because it is declarative, I can introspect through all the Django views and track their associated URLs as well as custom data objects and templates. Works very well, but yes, the function does expect certain attributes to be existing on itself. It doesn't know that a decorator set them.
Oh, and there's no good reason, in my case, for these to be passed as parameters in my use cases, these variables hold basically hardcoded values which never change, from the POV of the function.
Odd at first, but quite powerful and no runtime or maintenance drawbacks.
Here's some live example that puts that in context.
#declare_view(
viewmanager_cls=backend.VueManagerDetailPSCLASSDEFN,
template_name="pssecurity/detail.html",
objecttype=constants.OBJECTTYPE_PERMISSION_LIST[0],
bundle_name="pssecurity/detail.psclassdefn",
)
def psclassdefn_detail(request, CLASSID, dbr=None, PORTAL_NAME="EMPLOYEE"):
"""
"""
f_view = psclassdefn_detail
viewmanager = f_view.viewmanager_cls(request, mdb, f_view=f_view)
...do things based on the parameters...
return viewmanager.HttpResponse(f_view.template_name)
Classes as Decorators
This article points to classes as decorators, which seems a more elegant way to point to the state defined in the decorator. It relies on function attributes and uses the special .__call__() method in the decorating class.
Here is my example revisited using a class instead of a function as a decorator:
class my_class_decorator:
def __init__(self, func):
self.func = func
self.x = 0
def __call__(self, *args, **kwargs):
print("x in wrapper:", self.x)
return self.func(*args, **kwargs)
#my_class_decorator
def decorated_func():
decorated_func.x += 1
print("x in decorated_func:", decorated_func.x)
decorated_func()
# prints:
# x in wrapper: 0
# x in decorated_func: 1
I have created a python decorator as shown below. I want this decorator to accept an unknown list of arguments. But the code below doesn't work.
#!/usr/bin/env python
from functools import wraps
def my_decorator(decorated_function, **kwargs):
print "kwargs = {}".format(kwargs)
#wraps(decorated_function)
def inner_function(*args, **kwargs):
print "Hello world"
return decorated_function(*args, **kwargs)
return inner_function
#my_decorator(arg_1="Yolo", arg2="Bolo")
def my_func():
print "Woo Hoo!"
my_func()
When I run it, I get this error:
File "./decorator_test.py", line 14, in <module>
#my_decorator(arg_1="Yolo", arg2="Bolo")
TypeError: my_decorator() takes exactly 1 argument (0 given)
Why is the **kwargs not accepting the multiple arguments I'm sending into the decorator? How can I fix it?
Essentially, when you have a decorator that accepts arguments, you're kind of calling it twice, once with the keyword arguments, and then again with just the function to be decorated.
Before the #deco syntax, you would decorate a function just by passing it into the decorator function
func = deco(func)
And if you wanted to include other keyword arguments, you could just pass them in at the same time
func = deco(func, arg=True, arg2=5)
But the #deco syntax doesn't allow that, it works more like this, so you're calling a function returned by the decorator function.
func = deco(arg=True, arg2=5)(func)
To do this with the #deco syntax, you would create a decorator that tests to see whether it was called with just a function (the decoration part), or whether it was called with keyword arguments (which sets up the decorator to be passed a function).
def deco_with_kwargs(func=None, **kwargs):
def deco(func_):
def wrapped_func(*fargs, **fkwargs):
print kwargs
return func_(*fargs, **fkwargs)
return wrapped_func
if func:
return deco(func)
else:
return deco
You could use it like this without any keyword arguments.
#deco_with_args
def my_func():
return 1
Or you could call it like this with keyword arguments. In this case, you're actually calling the decorator function, which returns another decorator function, which is then called to actually decorate the function.
#deco_with_args(test=True)
def my_func():
return 1
How to get all methods of a given class A that are decorated with the #decorator2?
class A():
def method_a(self):
pass
#decorator1
def method_b(self, b):
pass
#decorator2
def method_c(self, t=5):
pass
Method 1: Basic registering decorator
I already answered this question here: Calling functions by array index in Python =)
Method 2: Sourcecode parsing
If you do not have control over the class definition, which is one interpretation of what you'd like to suppose, this is impossible (without code-reading-reflection), since for example the decorator could be a no-op decorator (like in my linked example) that merely returns the function unmodified. (Nevertheless if you allow yourself to wrap/redefine the decorators, see Method 3: Converting decorators to be "self-aware", then you will find an elegant solution)
It is a terrible terrible hack, but you could use the inspect module to read the sourcecode itself, and parse it. This will not work in an interactive interpreter, because the inspect module will refuse to give sourcecode in interactive mode. However, below is a proof of concept.
#!/usr/bin/python3
import inspect
def deco(func):
return func
def deco2():
def wrapper(func):
pass
return wrapper
class Test(object):
#deco
def method(self):
pass
#deco2()
def method2(self):
pass
def methodsWithDecorator(cls, decoratorName):
sourcelines = inspect.getsourcelines(cls)[0]
for i,line in enumerate(sourcelines):
line = line.strip()
if line.split('(')[0].strip() == '#'+decoratorName: # leaving a bit out
nextLine = sourcelines[i+1]
name = nextLine.split('def')[1].split('(')[0].strip()
yield(name)
It works!:
>>> print(list( methodsWithDecorator(Test, 'deco') ))
['method']
Note that one has to pay attention to parsing and the python syntax, e.g. #deco and #deco(... are valid results, but #deco2 should not be returned if we merely ask for 'deco'. We notice that according to the official python syntax at http://docs.python.org/reference/compound_stmts.html decorators are as follows:
decorator ::= "#" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
We breathe a sigh of relief at not having to deal with cases like #(deco). But note that this still doesn't really help you if you have really really complicated decorators, such as #getDecorator(...), e.g.
def getDecorator():
return deco
Thus, this best-that-you-can-do strategy of parsing code cannot detect cases like this. Though if you are using this method, what you're really after is what is written on top of the method in the definition, which in this case is getDecorator.
According to the spec, it is also valid to have #foo1.bar2.baz3(...) as a decorator. You can extend this method to work with that. You might also be able to extend this method to return a <function object ...> rather than the function's name, with lots of effort. This method however is hackish and terrible.
Method 3: Converting decorators to be "self-aware"
If you do not have control over the decorator definition (which is another interpretation of what you'd like), then all these issues go away because you have control over how the decorator is applied. Thus, you can modify the decorator by wrapping it, to create your own decorator, and use that to decorate your functions. Let me say that yet again: you can make a decorator that decorates the decorator you have no control over, "enlightening" it, which in our case makes it do what it was doing before but also append a .decorator metadata property to the callable it returns, allowing you to keep track of "was this function decorated or not? let's check function.decorator!". And then you can iterate over the methods of the class, and just check to see if the decorator has the appropriate .decorator property! =) As demonstrated here:
def makeRegisteringDecorator(foreignDecorator):
"""
Returns a copy of foreignDecorator, which is identical in every
way(*), except also appends a .decorator property to the callable it
spits out.
"""
def newDecorator(func):
# Call to newDecorator(method)
# Exactly like old decorator, but output keeps track of what decorated it
R = foreignDecorator(func) # apply foreignDecorator, like call to foreignDecorator(method) would have done
R.decorator = newDecorator # keep track of decorator
#R.original = func # might as well keep track of everything!
return R
newDecorator.__name__ = foreignDecorator.__name__
newDecorator.__doc__ = foreignDecorator.__doc__
# (*)We can be somewhat "hygienic", but newDecorator still isn't signature-preserving, i.e. you will not be able to get a runtime list of parameters. For that, you need hackish libraries...but in this case, the only argument is func, so it's not a big issue
return newDecorator
Demonstration for #decorator:
deco = makeRegisteringDecorator(deco)
class Test2(object):
#deco
def method(self):
pass
#deco2()
def method2(self):
pass
def methodsWithDecorator(cls, decorator):
"""
Returns all methods in CLS with DECORATOR as the
outermost decorator.
DECORATOR must be a "registering decorator"; one
can make any decorator "registering" via the
makeRegisteringDecorator function.
"""
for maybeDecorated in cls.__dict__.values():
if hasattr(maybeDecorated, 'decorator'):
if maybeDecorated.decorator == decorator:
print(maybeDecorated)
yield maybeDecorated
It works!:
>>> print(list( methodsWithDecorator(Test2, deco) ))
[<function method at 0x7d62f8>]
However, a "registered decorator" must be the outermost decorator, otherwise the .decorator attribute annotation will be lost. For example in a train of
#decoOutermost
#deco
#decoInnermost
def func(): ...
you can only see metadata that decoOutermost exposes, unless we keep references to "more-inner" wrappers.
sidenote: the above method can also build up a .decorator that keeps track of the entire stack of applied decorators and input functions and decorator-factory arguments. =) For example if you consider the commented-out line R.original = func, it is feasible to use a method like this to keep track of all wrapper layers. This is personally what I'd do if I wrote a decorator library, because it allows for deep introspection.
There is also a difference between #foo and #bar(...). While they are both "decorator expressons" as defined in the spec, note that foo is a decorator, while bar(...) returns a dynamically-created decorator, which is then applied. Thus you'd need a separate function makeRegisteringDecoratorFactory, that is somewhat like makeRegisteringDecorator but even MORE META:
def makeRegisteringDecoratorFactory(foreignDecoratorFactory):
def newDecoratorFactory(*args, **kw):
oldGeneratedDecorator = foreignDecoratorFactory(*args, **kw)
def newGeneratedDecorator(func):
modifiedFunc = oldGeneratedDecorator(func)
modifiedFunc.decorator = newDecoratorFactory # keep track of decorator
return modifiedFunc
return newGeneratedDecorator
newDecoratorFactory.__name__ = foreignDecoratorFactory.__name__
newDecoratorFactory.__doc__ = foreignDecoratorFactory.__doc__
return newDecoratorFactory
Demonstration for #decorator(...):
def deco2():
def simpleDeco(func):
return func
return simpleDeco
deco2 = makeRegisteringDecoratorFactory(deco2)
print(deco2.__name__)
# RESULT: 'deco2'
#deco2()
def f():
pass
This generator-factory wrapper also works:
>>> print(f.decorator)
<function deco2 at 0x6a6408>
bonus Let's even try the following with Method #3:
def getDecorator(): # let's do some dispatching!
return deco
class Test3(object):
#getDecorator()
def method(self):
pass
#deco2()
def method2(self):
pass
Result:
>>> print(list( methodsWithDecorator(Test3, deco) ))
[<function method at 0x7d62f8>]
As you can see, unlike method2, #deco is correctly recognized even though it was never explicitly written in the class. Unlike method2, this will also work if the method is added at runtime (manually, via a metaclass, etc.) or inherited.
Be aware that you can also decorate a class, so if you "enlighten" a decorator that is used to both decorate methods and classes, and then write a class within the body of the class you want to analyze, then methodsWithDecorator will return decorated classes as well as decorated methods. One could consider this a feature, but you can easily write logic to ignore those by examining the argument to the decorator, i.e. .original, to achieve the desired semantics.
To expand upon #ninjagecko's excellent answer in Method 2: Source code parsing, you can use the ast module introduced in Python 2.6 to perform self-inspection as long as the inspect module has access to the source code.
def findDecorators(target):
import ast, inspect
res = {}
def visit_FunctionDef(node):
res[node.name] = [ast.dump(e) for e in node.decorator_list]
V = ast.NodeVisitor()
V.visit_FunctionDef = visit_FunctionDef
V.visit(compile(inspect.getsource(target), '?', 'exec', ast.PyCF_ONLY_AST))
return res
I added a slightly more complicated decorated method:
#x.y.decorator2
def method_d(self, t=5): pass
Results:
> findDecorators(A)
{'method_a': [],
'method_b': ["Name(id='decorator1', ctx=Load())"],
'method_c': ["Name(id='decorator2', ctx=Load())"],
'method_d': ["Attribute(value=Attribute(value=Name(id='x', ctx=Load()), attr='y', ctx=Load()), attr='decorator2', ctx=Load())"]}
If you do have control over the decorators, you can use decorator classes rather than functions:
class awesome(object):
def __init__(self, method):
self._method = method
def __call__(self, obj, *args, **kwargs):
return self._method(obj, *args, **kwargs)
#classmethod
def methods(cls, subject):
def g():
for name in dir(subject):
method = getattr(subject, name)
if isinstance(method, awesome):
yield name, method
return {name: method for name,method in g()}
class Robot(object):
#awesome
def think(self):
return 0
#awesome
def walk(self):
return 0
def irritate(self, other):
return 0
and if I call awesome.methods(Robot) it returns
{'think': <mymodule.awesome object at 0x000000000782EAC8>, 'walk': <mymodulel.awesome object at 0x000000000782EB00>}
For those of us who just want the absolute simplest possible case - namely, a single-file solution where we have total control over both the class we're working with and the decorator we're trying to track, I've got an answer. ninjagecko linked to a solution for when you have control over the decorator you want to track, but I personally found it to be complicated and really hard to understand, possibly because I've never worked with decorators until now. So, I've created the following example, with the goal of being as straightforward and simple as possible. It's a decorator, a class with several decorated methods, and code to retrieve+run all methods that have a specific decorator applied to them.
# our decorator
def cool(func, *args, **kwargs):
def decorated_func(*args, **kwargs):
print("cool pre-function decorator tasks here.")
return_value = func(*args, **kwargs)
print("cool post-function decorator tasks here.")
return return_value
# add is_cool property to function so that we can check for its existence later
decorated_func.is_cool = True
return decorated_func
# our class, in which we will use the decorator
class MyClass:
def __init__(self, name):
self.name = name
# this method isn't decorated with the cool decorator, so it won't show up
# when we retrieve all the cool methods
def do_something_boring(self, task):
print(f"{self.name} does {task}")
#cool
# thanks to *args and **kwargs, the decorator properly passes method parameters
def say_catchphrase(self, *args, catchphrase="I'm so cool you could cook an egg on me.", **kwargs):
print(f"{self.name} says \"{catchphrase}\"")
#cool
# the decorator also properly handles methods with return values
def explode(self, *args, **kwargs):
print(f"{self.name} explodes.")
return 4
def get_all_cool_methods(self):
"""Get all methods decorated with the "cool" decorator.
"""
cool_methods = {name: getattr(self, name)
# get all attributes, including methods, properties, and builtins
for name in dir(self)
# but we only want methods
if callable(getattr(self, name))
# and we don't need builtins
and not name.startswith("__")
# and we only want the cool methods
and hasattr(getattr(self, name), "is_cool")
}
return cool_methods
if __name__ == "__main__":
jeff = MyClass(name="Jeff")
cool_methods = jeff.get_all_cool_methods()
for method_name, cool_method in cool_methods.items():
print(f"{method_name}: {cool_method} ...")
# you can call the decorated methods you retrieved, just like normal,
# but you don't need to reference the actual instance to do so
return_value = cool_method()
print(f"return value = {return_value}\n")
Running the above example gives us the following output:
explode: <bound method cool.<locals>.decorated_func of <__main__.MyClass object at 0x00000220B3ACD430>> ...
cool pre-function decorator tasks here.
Jeff explodes.
cool post-function decorator tasks here.
return value = 4
say_catchphrase: <bound method cool.<locals>.decorated_func of <__main__.MyClass object at 0x00000220B3ACD430>> ...
cool pre-function decorator tasks here.
Jeff says "I'm so cool you could cook an egg on me."
cool post-function decorator tasks here.
return value = None
Note that the decorated methods in this example have different types of return values and different signatures, so the practical value of being able to retrieve and run them all is a bit dubious. However, in cases where there are many similar methods, all with the same signature and/or type of return value (like if you're writing a connector to retrieve unnormalized data from one database, normalize it, and insert it into a second, normalized database, and you have a bunch similar methods, e.g. 15 read_and_normalize_table_X methods), being able to retrieve (and run) them all on the fly could be more useful.
Maybe, if the decorators are not too complex (but I don't know if there is a less hacky way).
def decorator1(f):
def new_f():
print "Entering decorator1", f.__name__
f()
new_f.__name__ = f.__name__
return new_f
def decorator2(f):
def new_f():
print "Entering decorator2", f.__name__
f()
new_f.__name__ = f.__name__
return new_f
class A():
def method_a(self):
pass
#decorator1
def method_b(self, b):
pass
#decorator2
def method_c(self, t=5):
pass
print A.method_a.im_func.func_code.co_firstlineno
print A.method_b.im_func.func_code.co_firstlineno
print A.method_c.im_func.func_code.co_firstlineno
I don't want to add much, just a simple variation of ninjagecko's Method 2. It works wonders.
Same code, but using list comprehension instead of a generator, which is what I needed.
def methodsWithDecorator(cls, decoratorName):
sourcelines = inspect.getsourcelines(cls)[0]
return [ sourcelines[i+1].split('def')[1].split('(')[0].strip()
for i, line in enumerate(sourcelines)
if line.split('(')[0].strip() == '#'+decoratorName]
A simple way to solve this problem is to put code in the decorator that adds each function/method, that is passed in, to a data set (for example a list).
e.g.
def deco(foo):
functions.append(foo)
return foo
now every function with the deco decorator will be added to functions.