Let's say I have a function foo that gets a few parameters
def foo(width, height, depth=0):
...
I want to write a wrapper function that gets all of foo's parameters and passes them on, e.g.
def goo(width, height, depth=0):
...
foo(width, height, depth)
...
But this is ugly, since I have to repeat the variables and the default values.
What's the idiomatic way to do this in python?
A few options I thought about:
passing to goo a dictionary called foo_params and calling foo(**foo_params) but then is error prone since I don't know
if all the arguments are there
writing another wrapper for foo that checks if the params with default values are None and if so doesn't pass them
Putting the default values as constants so I won't repeat them
You can use *args and **kwargs syntax to pass an unknown amount of arguments and/or keyword arguments:
>>> def dec(func):
def inner(*args, **kwargs):
print('decorated function')
func(*args, **kwargs)
return inner
>>> #dec
def func(a, b):
return a + b
>>> func(1, 2)
decorated function
>>>
One downside to using *args and **kwargs is that you'll lose the orginal function signature of the decorated function. eg:
>>> help(func)
Help on function inner in module __main__:
inner(*args, **kwargs)
>>>
The solution is to use functools.wraps(). It basically copies of the data from the decorated function to the wrapper function:
>>> from functools import wraps
>>>
>>> def dec(func):
#wraps(func)
def inner(*args, **kwargs):
print('decorated function')
func(*args, **kwargs)
return inner
>>> #dec
def func(a, b):
return a + b
>>> func(1, 2)
decorated function
>>>
A you can see below, if you now do help(func) the original signature for func will be displayed:
>>> help(func)
Help on function func in module __main__:
func(a, b)
>>>
I think you are looking for functools's partial function:
from functools import partial
def foo(a,b):
return a + b
goo = partial(foo, b = 1)
goo(5) # returns 6
Related
I have a lot of functions like the following, which recursively call themselves to get one or many returns depending on the type of the argument:
def get_data_sensor(self, sensorname):
if isinstance(sensorname, list):
return [get_data_sensor(self, sensor) for sensor in sensorname]
return get_data(os.path.join(self.get_info("path"), "{0}.npy".format(sensorname)))
I would like to call my function recursively without having to know my current function name, I don't want to have my function name twice in the code to limit copy-paste error.
Determine function name from within that function (without using traceback) shows how to get the actual function name, but I need the function itself to call it.
You can abstract that logic outside the function entirely by using a decorator (see this thorough answer if you're unfamiliar with decorators):
from functools import wraps
def autolist(func):
#wraps(func)
def wrapper(args):
if isinstance(args, list):
return [func(arg) for arg in args]
return func(args)
return wrapper
This decorator can be applied to any function requiring the pattern, which now only needs to implement the scalar case:
>>> #autolist
... def square(x):
... return x ** 2
...
>>> square(1)
1
>>> square([1, 2, 3])
[1, 4, 9]
If you're applying it to a method, as self implies, you'll also need to take that argument into account in the wrapper. For example, if the relevant argument is always the last one you could do:
def autolist(func):
#wraps(func)
def wrapper(*args):
*args, last_arg = args
if isinstance(last_arg, list):
return [func(*args, arg) for arg in last_arg]
return func(*args, last_arg)
return wrapper
This would work on methods, too:
>>> class Squarer:
... #autolist
... def square(self, x):
... return x ** 2
...
>>> Squarer().square(1)
1
>>> Squarer().square([1, 2, 3])
[1, 4, 9]
I have a function -
def add(a, b):
return a+b
Now I want it to add 100 to the result using a decorator such that -
#decorator(100)
def add(a, b):
return (a+b)
class decorator():
def __init__(self, c):
self.c = c
def __call__(self, f):
def _wrapped_f(a, b):
return self.c + f(a, b)
return _wrapped_f
The problem is when I want to keep the extra argument variable and not fixed to 100.
Something like -
#decorator(c)
def add(a, b):
return (a+b)
Can I somehow assign the value of this variable during the function call. (when add function is called)
The reason I want to do this using a decorator is because I don't want to modify my function add.
I cannot afford an extra parameter in function something like -
def(a, b, c=0):
return (a+b+c)
Hence, the need of the decorator.
Sure. All a decorator does is replace the original function with a wrapped version; that wrapper can take as many parameters as you like.
I'm not sure why you are using a class - this would be clearer as a standard decorator.
def decorator(func):
def wrapped(a, b, c):
return c + func(a, b)
return wrapped
#decorator
def add(a, b):
return (a+b)
Not 100% sure what you want to do with this decorator, so this might not be exactly what you want, but even then, others might find it useful.
You can add another parameter to the function, just by adding more parameters to the _wrapped function, or even by passing *args and **kwargs to that function. But how do you handle those parameters? Surely, just adding those parameters to the result only makes sense for the add function.
In the general case, you could use the decorated function itself to process the additional parameters. Then, just have the decorated function reduce all the parameters using the original function:
from functools import reduce # Python 3
def vararg(f):
""" change two-arg-function into vararg-function"""
def _f(*args):
return reduce(f, args)
return _f
#vararg
def add(a, b):
return a + b
print(add(1, 2)) # -> 3
print(add(1, 2, 3)) # -> 6
print(add(1, 2, 3, 4)) # -> 10
print(add(1, 2, 3, 4, 5)) # -> 15
I am using a single decorator for two separate functions: one with specification of a decorator argument; and another one without it.
When the optional argument is not passed, the return type is a function (specifically, the inner_function in the decorator). However, when the optional argument is passed it works as expected.
Can you explain what is going on here and why it acts differently in these cases?
def cache_disk(cache_folder="./cache"):
def wrapper(f):
def inner_function(*args, **kwargs):
result = f(*args, **kwargs)
return result
return inner_function
return wrapper
#cache_disk
def func1(data):
return [d for d in data]
#cache_disk(cache_folder='./cache/')
def func2(data):
return [d for d in data]
data = [1,2,3]
print(func1(data))
print(func2(data))
Result:
<function inner_function at 0x7f1f283d5c08>
[1, 2, 3]
Note that:
#decorator # no arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(func) # one 'level' of calls
and that:
#decorator(...): # arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(...)(func) # two 'levels' of calls
In the first case, there is a single argument to the decorator, the func itself. In the second case, the arguments to the decorator are the ... from the # line, and it's the function returned by the decorator that is called with func as an argument.
In your example,
#cache_disk
def func1(data):
...
the decorator cache_disk gets a single, callable argument (func, which becomes args[0]) and returns the wrapper. Then when you call:
print(func1(data))
wrapper gets a single argument (data, which becomes f) and returns inner_function.
Therefore, you have three choices:
Decorate func1 with #cache_disk() (note parentheses), passing no arguments to cache_disk itself and func to wrapper;
Alter cache_disk to behave differently depending on whether it's passed a single, callable argument or something else; or
As #o11c pointed out in the comments, use e.g. cache_disk.wrapper = cache_disk() to provide a convenient alias for the parameter-less version, then decorate with #cache_disk.wrapper.
if you want default values, you need to call the function, which returns a decorator:
#cache_disk()
def func1(data):
return [d for d in data]
I try to understand python decorator
def dec(func):
def wrap(*args,**kw):
print args, kw
return func(*args,**kw)
return wrap
#dec
def myfunc(a=1,b=2,c=3):
return a+b+c
>>> myfunc()
() {}
6
>>> myfunc(1,2,3)
(1, 2, 3) {}
6
>>> myfunc(1,2,c=5)
(1, 2) {'c': 5}
8
>>>
When I run myfunc() args and kw are nothing, but when I run myfunc(1,2,3) and myfunc(1,2,c=5), args and kw were passed to dec function.
As I know,
#dec
def myfunc(...)
equals to myfunc = dec(myfunc) <-- no arguments were mentioned here.
How arguments were passed to wrap function in dec? How to understand these?
Not sure if i understand correctly your problem, but the default values for myfunc arguments are known only to myfunc - your decorator has no knowledge of them, so it cannot print them.
That's why:
myfunc()
results in printing:
() {}
Both *args and **kw are empty for the decorator, but the decorated function will use the default values in this case.
In the second and third case you get the parameters printed, as they are explicitly passed to the decorated function:
def wrap(*args,**kw): <- this is what is actually called when you invoke decorated function, no default values here
print args, kw
return func(*args,**kw) <- this is the function you decorate
#if func has default parameter values, then those will be used
#when you don't pass them to the wrapper but only **inside** func
return wrap
Edit:
It looks like you're mistaking calling the decorated function with decorating the function:
myfunc = dec(myfunc)
decorates myfunc using dec and is equivalent to
#dec
def myfunc(...)
On the other hand, after using either of them:
myfunc(whatever)
calls the wrap function defined in your decorator, which will in turn call the original myfunc
Another way to think of it is by saying:
def wrapper(some_function):
def _inner(*args, **kwargs):
#do some work
return func(*args, **kwargs)
return _inner
#wrapper
def foo(a, b, c):
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
...you're getting a result which is roughly similar to the following:
def foo(a, b, c):
#do some work
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
This isn't exactly right because the #do some work is occurring before the actual foo() call, but as an approximation this is what you're getting. For that reason, the wrapper can't really 'see' the default arguments for foo() if any exist. So a better way to think of it might be:
#always execute this code before calling...
def foo(a, b, c):
return "Foo called and I saw %d, %d, %d" % (a, b, c)
So something really basic might look like this.
def wrap(f):
... def _inner(*a, **k):
... new_a = (ar*2 for ar in a)
... new_k = {}
... for k, v in k.iteritems():
... new_k[k] = v*2
... return f(*new_a, **new_k)
... return _inner
...
>>> def foo(a=2, b=4, c=6):
... return "%d, %d, %d" % (a, b, c)
...
>>> foo()
'2, 4, 6'
>>> foo(1, 5, 7)
'1, 5, 7'
>>> foo = wrap(foo) #actually wrapping it here
>>> foo()
'2, 4, 6'
>>> foo(3, 5, 6)
'6, 10, 12'
>>> foo(3, 5, c=7)
'6, 10, 14'
>>>
Decorators are function wrappers. They give back a function that wraps the original one into some pre- and post-processing code, but still need to call the original function (normally with the same argument as you would call it in absence of a decorator).
Python has two types of arguments, positional and keyword arguments (this has nothing to do with decorators, that's generic python basics). * is for positional (internally is a list), ** for keyword (dictionary) arguments. By specifying both you allow your decorator to accept all at all possible types of arguments and pass them through to the underlying function. The contract of the call is, however, still defined by your function. E.g. if it only takes keyword arguments it will fail when the decorator function passes through a positional argument.
In your particular example, you have some pre-processing code (i.e. code that will run before the original function is called). For example in this code you can print out arguments *args that your original function might fail to accept all together because it does not take any position arguments.
You do not necessarily have to pass through *args and **kwargs. In fact you can define a decorator which makes some decisions based on the arguments you pass in about what to provide to the original function as arguments, e.g.
def dec(fun):
def wrap(*args, **kwargs):
if 'a' in kwargs:
return fun(kwargs[a])
else:
return fun(*args)
return wrap
References
Before writing this question, I have referred some of the following interesting questions and feel that this scenario is not explained/covered:
How to make a chain of function decorators? (especially answer #2)
How to decorate a method inside a class?
Python decorators in classes
Python Class Based Decorator with parameters that can decorate a method or a function (close but not exactly what I am asking)
I have a function as follows:
def simple_function():
print 'this is a simple_function'
And, I have the following class with some methods
class Test:
def Test_method_1(self, func, args=None, kwargs=None):
print 'this is Test::Test_method_1'
<execute some instructions here>
def Test_method_2(self):
print 'this is Test::Test_method_2'
I am using Test_method_1 and simple_function as follows:
t = Test()
t.Test_method_1(simple_function)
Say if the simple_function takes arbitrary arguments, then the Test_method_1 is called as follows:
def simple_function(a, b, c, d=f, g=h):
print 'this is a simple_function'
t = Test()
t.Test_method_1(simple_function, args=[a, b, c], kwargs={d:f, g:h})
Now, I want to use Test_method_2 as a decorator version of Test_method_1. So the simple_function definition can be written as follows:
t = Test()
#t.Test_method_2
def simple_function():
print 'this is a simple_function'
Note: the Test_method_2 will call Test_method_1 with appropriate arguments.
Now the questions:
1 - Is this possible?
2 - If possible, how do the name of the decorated function (here simple_function) will be passed as a argument to Test_method_2 (including the self)?
3 - [Here is the killer question, I believe] I want to pass arbitrary arguments to both Test_method_2 (the decorator) and simple_function (the function being decorated) - *args and **kwargs - How should the Test_method_2 be defined to receive arguments (including the self)?
The usage of Test_method_2 as a decorator for simple_function with arbitrary arguments for both is as follows:
t = Test()
#t.Test_method_2(a, b, c, d, e=f, g=h)
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
Sure it's possible. All the #decorator syntax does is define the function that follows, then call the decorator, passing in the function that followed and replacing the reference to that function with whatever was returned.
So the following:
#foo
def bar():
pass
is translated to:
def bar():
pass
bar = foo(bar)
This means that your t.Test_method_2() method must expect one argument, the to-be-decorated function, and return something callable:
import functools
def Test_method_2(self, func):
#functools.wraps(func)
def wrapper(self, *args, **kw):
print 'Wrapped function name:', func.__name__
return func(*args, **kw)
return wrapper
would be the minimal decorator that returns a wrapper function, and prints the name of the wrapped function when called. It doesn't matter what the argument is called; I used func here but it can be any legal python identifier.
The self is the standard part of the method signature. Because you are referencing Test_method_2 on an instance t, Python automatically takes care of the self parameter for you as it does with all methods.
Anything after # is just an expression. So if you use the syntax:
#t.Test_method_2(a, b, c, d, e=f, g=h)
then Test_method_2() should instead return a decorator function. One extra level of scope nesting should do that:
def Test_method_2(self, *args, **kw):
def decorator(func):
#functools.wraps(func)
def wrapper(*wrapperargs, **wrapperkw):
fargs = args + wrapperargs
fkw = dict(kw)
fkw.update(wrapperkw)
return func(*fargs, **fkw)
return wrapper
return decorator
Deconstructing this:
#t.Test_method_2(5, 6, 7, 8, baz='spam', ham='eggs')
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
The part after the #, t.Test_method_2(5, 6, 7, 8, baz='spam', ham='eggs') returns the nested function decorator:
#decorator
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
which python then turns into:
simple_function = decorator(simple_function)
and decorator(func) returns wrapper(*wrapperargs, **wrapperkw).
Calling simple_function(1, 2, foo='bar') then results in a call to wrapper(1, 2, foo='bar') which calls the original simple_function() with fargs = [5, 6, 7, 8, 1, 2] and fkw = {'baz': 'spam', 'ham': 'eggs', 'foo': 'bar'} passed in as positional and keyword arguments.
The class decorator pattern you see in the linked questions works in a similar way; the expression after the # returns something that is called; instead of a nested function a class instance is being created instead. It's just two different approaches to storing state for the decorator to use.
The nested functions is a little more compact to write, while using a class gives you more introspection options than a nested scope.