References
Before writing this question, I have referred some of the following interesting questions and feel that this scenario is not explained/covered:
How to make a chain of function decorators? (especially answer #2)
How to decorate a method inside a class?
Python decorators in classes
Python Class Based Decorator with parameters that can decorate a method or a function (close but not exactly what I am asking)
I have a function as follows:
def simple_function():
print 'this is a simple_function'
And, I have the following class with some methods
class Test:
def Test_method_1(self, func, args=None, kwargs=None):
print 'this is Test::Test_method_1'
<execute some instructions here>
def Test_method_2(self):
print 'this is Test::Test_method_2'
I am using Test_method_1 and simple_function as follows:
t = Test()
t.Test_method_1(simple_function)
Say if the simple_function takes arbitrary arguments, then the Test_method_1 is called as follows:
def simple_function(a, b, c, d=f, g=h):
print 'this is a simple_function'
t = Test()
t.Test_method_1(simple_function, args=[a, b, c], kwargs={d:f, g:h})
Now, I want to use Test_method_2 as a decorator version of Test_method_1. So the simple_function definition can be written as follows:
t = Test()
#t.Test_method_2
def simple_function():
print 'this is a simple_function'
Note: the Test_method_2 will call Test_method_1 with appropriate arguments.
Now the questions:
1 - Is this possible?
2 - If possible, how do the name of the decorated function (here simple_function) will be passed as a argument to Test_method_2 (including the self)?
3 - [Here is the killer question, I believe] I want to pass arbitrary arguments to both Test_method_2 (the decorator) and simple_function (the function being decorated) - *args and **kwargs - How should the Test_method_2 be defined to receive arguments (including the self)?
The usage of Test_method_2 as a decorator for simple_function with arbitrary arguments for both is as follows:
t = Test()
#t.Test_method_2(a, b, c, d, e=f, g=h)
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
Sure it's possible. All the #decorator syntax does is define the function that follows, then call the decorator, passing in the function that followed and replacing the reference to that function with whatever was returned.
So the following:
#foo
def bar():
pass
is translated to:
def bar():
pass
bar = foo(bar)
This means that your t.Test_method_2() method must expect one argument, the to-be-decorated function, and return something callable:
import functools
def Test_method_2(self, func):
#functools.wraps(func)
def wrapper(self, *args, **kw):
print 'Wrapped function name:', func.__name__
return func(*args, **kw)
return wrapper
would be the minimal decorator that returns a wrapper function, and prints the name of the wrapped function when called. It doesn't matter what the argument is called; I used func here but it can be any legal python identifier.
The self is the standard part of the method signature. Because you are referencing Test_method_2 on an instance t, Python automatically takes care of the self parameter for you as it does with all methods.
Anything after # is just an expression. So if you use the syntax:
#t.Test_method_2(a, b, c, d, e=f, g=h)
then Test_method_2() should instead return a decorator function. One extra level of scope nesting should do that:
def Test_method_2(self, *args, **kw):
def decorator(func):
#functools.wraps(func)
def wrapper(*wrapperargs, **wrapperkw):
fargs = args + wrapperargs
fkw = dict(kw)
fkw.update(wrapperkw)
return func(*fargs, **fkw)
return wrapper
return decorator
Deconstructing this:
#t.Test_method_2(5, 6, 7, 8, baz='spam', ham='eggs')
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
The part after the #, t.Test_method_2(5, 6, 7, 8, baz='spam', ham='eggs') returns the nested function decorator:
#decorator
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
which python then turns into:
simple_function = decorator(simple_function)
and decorator(func) returns wrapper(*wrapperargs, **wrapperkw).
Calling simple_function(1, 2, foo='bar') then results in a call to wrapper(1, 2, foo='bar') which calls the original simple_function() with fargs = [5, 6, 7, 8, 1, 2] and fkw = {'baz': 'spam', 'ham': 'eggs', 'foo': 'bar'} passed in as positional and keyword arguments.
The class decorator pattern you see in the linked questions works in a similar way; the expression after the # returns something that is called; instead of a nested function a class instance is being created instead. It's just two different approaches to storing state for the decorator to use.
The nested functions is a little more compact to write, while using a class gives you more introspection options than a nested scope.
Related
I have a function that can receive 3 arguments. Only X is mandatory and I could write this function like
def foo(X, A=None, B=None):
pass
But I want to force the user to call this function passing explicitly the arguments
foo(X=1, A=2, B=3)
foo(1, 2, 3) # Error
Is it possible make a decorator for that?
#explicitarguments
def foo(X, A=None, B=None):
pass
If you can modify the function definition, refer to this question.
A minimal decorator to patch existing functions looks like this:
def explicitarguments(f):
def f_new(**kwargs):
return f(**kwargs)
return f_new
Let's say I have a function foo that gets a few parameters
def foo(width, height, depth=0):
...
I want to write a wrapper function that gets all of foo's parameters and passes them on, e.g.
def goo(width, height, depth=0):
...
foo(width, height, depth)
...
But this is ugly, since I have to repeat the variables and the default values.
What's the idiomatic way to do this in python?
A few options I thought about:
passing to goo a dictionary called foo_params and calling foo(**foo_params) but then is error prone since I don't know
if all the arguments are there
writing another wrapper for foo that checks if the params with default values are None and if so doesn't pass them
Putting the default values as constants so I won't repeat them
You can use *args and **kwargs syntax to pass an unknown amount of arguments and/or keyword arguments:
>>> def dec(func):
def inner(*args, **kwargs):
print('decorated function')
func(*args, **kwargs)
return inner
>>> #dec
def func(a, b):
return a + b
>>> func(1, 2)
decorated function
>>>
One downside to using *args and **kwargs is that you'll lose the orginal function signature of the decorated function. eg:
>>> help(func)
Help on function inner in module __main__:
inner(*args, **kwargs)
>>>
The solution is to use functools.wraps(). It basically copies of the data from the decorated function to the wrapper function:
>>> from functools import wraps
>>>
>>> def dec(func):
#wraps(func)
def inner(*args, **kwargs):
print('decorated function')
func(*args, **kwargs)
return inner
>>> #dec
def func(a, b):
return a + b
>>> func(1, 2)
decorated function
>>>
A you can see below, if you now do help(func) the original signature for func will be displayed:
>>> help(func)
Help on function func in module __main__:
func(a, b)
>>>
I think you are looking for functools's partial function:
from functools import partial
def foo(a,b):
return a + b
goo = partial(foo, b = 1)
goo(5) # returns 6
I am using a single decorator for two separate functions: one with specification of a decorator argument; and another one without it.
When the optional argument is not passed, the return type is a function (specifically, the inner_function in the decorator). However, when the optional argument is passed it works as expected.
Can you explain what is going on here and why it acts differently in these cases?
def cache_disk(cache_folder="./cache"):
def wrapper(f):
def inner_function(*args, **kwargs):
result = f(*args, **kwargs)
return result
return inner_function
return wrapper
#cache_disk
def func1(data):
return [d for d in data]
#cache_disk(cache_folder='./cache/')
def func2(data):
return [d for d in data]
data = [1,2,3]
print(func1(data))
print(func2(data))
Result:
<function inner_function at 0x7f1f283d5c08>
[1, 2, 3]
Note that:
#decorator # no arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(func) # one 'level' of calls
and that:
#decorator(...): # arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(...)(func) # two 'levels' of calls
In the first case, there is a single argument to the decorator, the func itself. In the second case, the arguments to the decorator are the ... from the # line, and it's the function returned by the decorator that is called with func as an argument.
In your example,
#cache_disk
def func1(data):
...
the decorator cache_disk gets a single, callable argument (func, which becomes args[0]) and returns the wrapper. Then when you call:
print(func1(data))
wrapper gets a single argument (data, which becomes f) and returns inner_function.
Therefore, you have three choices:
Decorate func1 with #cache_disk() (note parentheses), passing no arguments to cache_disk itself and func to wrapper;
Alter cache_disk to behave differently depending on whether it's passed a single, callable argument or something else; or
As #o11c pointed out in the comments, use e.g. cache_disk.wrapper = cache_disk() to provide a convenient alias for the parameter-less version, then decorate with #cache_disk.wrapper.
if you want default values, you need to call the function, which returns a decorator:
#cache_disk()
def func1(data):
return [d for d in data]
I try to understand python decorator
def dec(func):
def wrap(*args,**kw):
print args, kw
return func(*args,**kw)
return wrap
#dec
def myfunc(a=1,b=2,c=3):
return a+b+c
>>> myfunc()
() {}
6
>>> myfunc(1,2,3)
(1, 2, 3) {}
6
>>> myfunc(1,2,c=5)
(1, 2) {'c': 5}
8
>>>
When I run myfunc() args and kw are nothing, but when I run myfunc(1,2,3) and myfunc(1,2,c=5), args and kw were passed to dec function.
As I know,
#dec
def myfunc(...)
equals to myfunc = dec(myfunc) <-- no arguments were mentioned here.
How arguments were passed to wrap function in dec? How to understand these?
Not sure if i understand correctly your problem, but the default values for myfunc arguments are known only to myfunc - your decorator has no knowledge of them, so it cannot print them.
That's why:
myfunc()
results in printing:
() {}
Both *args and **kw are empty for the decorator, but the decorated function will use the default values in this case.
In the second and third case you get the parameters printed, as they are explicitly passed to the decorated function:
def wrap(*args,**kw): <- this is what is actually called when you invoke decorated function, no default values here
print args, kw
return func(*args,**kw) <- this is the function you decorate
#if func has default parameter values, then those will be used
#when you don't pass them to the wrapper but only **inside** func
return wrap
Edit:
It looks like you're mistaking calling the decorated function with decorating the function:
myfunc = dec(myfunc)
decorates myfunc using dec and is equivalent to
#dec
def myfunc(...)
On the other hand, after using either of them:
myfunc(whatever)
calls the wrap function defined in your decorator, which will in turn call the original myfunc
Another way to think of it is by saying:
def wrapper(some_function):
def _inner(*args, **kwargs):
#do some work
return func(*args, **kwargs)
return _inner
#wrapper
def foo(a, b, c):
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
...you're getting a result which is roughly similar to the following:
def foo(a, b, c):
#do some work
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
This isn't exactly right because the #do some work is occurring before the actual foo() call, but as an approximation this is what you're getting. For that reason, the wrapper can't really 'see' the default arguments for foo() if any exist. So a better way to think of it might be:
#always execute this code before calling...
def foo(a, b, c):
return "Foo called and I saw %d, %d, %d" % (a, b, c)
So something really basic might look like this.
def wrap(f):
... def _inner(*a, **k):
... new_a = (ar*2 for ar in a)
... new_k = {}
... for k, v in k.iteritems():
... new_k[k] = v*2
... return f(*new_a, **new_k)
... return _inner
...
>>> def foo(a=2, b=4, c=6):
... return "%d, %d, %d" % (a, b, c)
...
>>> foo()
'2, 4, 6'
>>> foo(1, 5, 7)
'1, 5, 7'
>>> foo = wrap(foo) #actually wrapping it here
>>> foo()
'2, 4, 6'
>>> foo(3, 5, 6)
'6, 10, 12'
>>> foo(3, 5, c=7)
'6, 10, 14'
>>>
Decorators are function wrappers. They give back a function that wraps the original one into some pre- and post-processing code, but still need to call the original function (normally with the same argument as you would call it in absence of a decorator).
Python has two types of arguments, positional and keyword arguments (this has nothing to do with decorators, that's generic python basics). * is for positional (internally is a list), ** for keyword (dictionary) arguments. By specifying both you allow your decorator to accept all at all possible types of arguments and pass them through to the underlying function. The contract of the call is, however, still defined by your function. E.g. if it only takes keyword arguments it will fail when the decorator function passes through a positional argument.
In your particular example, you have some pre-processing code (i.e. code that will run before the original function is called). For example in this code you can print out arguments *args that your original function might fail to accept all together because it does not take any position arguments.
You do not necessarily have to pass through *args and **kwargs. In fact you can define a decorator which makes some decisions based on the arguments you pass in about what to provide to the original function as arguments, e.g.
def dec(fun):
def wrap(*args, **kwargs):
if 'a' in kwargs:
return fun(kwargs[a])
else:
return fun(*args)
return wrap
I have a question about decorators. I understand what are decorators and I know how to use it, I have read all this tutorial How to make a chain of function decorators?
I understand that :
>>> def my_decorator(fn):
>>> print 'Do something before'
>>> print fn()
>>> def foo():
>>> return 'Hello World!'
>>> foo = my_decorator(foo)
Is the same at that :
>>> def my_decorator(fn):
>>> print 'Do something before'
>>> print fn()
>>> #my_decorator
>>> def foo():
>>> return 'Hello World!'
I know what are closures and why we use closure in a decorator with parameters (to get the decorator parameters in nested function) but that I don't understand is why we use closure and nested functions to get arguments and the function.
How the closure (or something else) can access parameters and the function outside. I am unable to do the same without the #decorator.
Here for example I can access my foo() and the parameters of the function without passing this function in parameter :
def my_decorator(str):
def wrapper(fn):
def inner_function(*args):
print 'Do something before'
return fn(*args)
return inner_function
return wrapper
#my_decorator('test')
def foo(a, b):
return a + b
print foo(1, 1)
How this is possible ?
I found the solution :
In fact the decorator use the closure functionality :
So here is the solution to do the same thing without the decorator and with parameters (it's just to understand the operation, and to learn)
def wrapper(str):
def decorator_factory(fn):
def inner_function(*args):
print 'Do something before'
return fn(*args)
return inner_function
return decorator_factory
#my_decorator('test')
def foo(a, b):
return a + b
# with decorator
print foo(1, 1)
# without decorator
print wrapper('str')(foo)(1, 1)
Decorator is a function that takes one argument (this argument is function) - this is my definition of decorator. In your case wrapper is decorator. While my_decorator is used to collect arguments for inner_function. inner_function is used to replace original [not decorated] function. Step by step explanation would be:
my_decorator is called to collect options for inner_function
my_decorator returns wrapper
wrapper is responsible for capturing original function or in other words decorate it. In your case original function is foo.
wrapper returns replacement function for original (which is inner_function)
from now foo points to inner_function, therefore inner_function is executed then foo is called
Hope it makes things a bit clearer.