I have a lot of functions like the following, which recursively call themselves to get one or many returns depending on the type of the argument:
def get_data_sensor(self, sensorname):
if isinstance(sensorname, list):
return [get_data_sensor(self, sensor) for sensor in sensorname]
return get_data(os.path.join(self.get_info("path"), "{0}.npy".format(sensorname)))
I would like to call my function recursively without having to know my current function name, I don't want to have my function name twice in the code to limit copy-paste error.
Determine function name from within that function (without using traceback) shows how to get the actual function name, but I need the function itself to call it.
You can abstract that logic outside the function entirely by using a decorator (see this thorough answer if you're unfamiliar with decorators):
from functools import wraps
def autolist(func):
#wraps(func)
def wrapper(args):
if isinstance(args, list):
return [func(arg) for arg in args]
return func(args)
return wrapper
This decorator can be applied to any function requiring the pattern, which now only needs to implement the scalar case:
>>> #autolist
... def square(x):
... return x ** 2
...
>>> square(1)
1
>>> square([1, 2, 3])
[1, 4, 9]
If you're applying it to a method, as self implies, you'll also need to take that argument into account in the wrapper. For example, if the relevant argument is always the last one you could do:
def autolist(func):
#wraps(func)
def wrapper(*args):
*args, last_arg = args
if isinstance(last_arg, list):
return [func(*args, arg) for arg in last_arg]
return func(*args, last_arg)
return wrapper
This would work on methods, too:
>>> class Squarer:
... #autolist
... def square(self, x):
... return x ** 2
...
>>> Squarer().square(1)
1
>>> Squarer().square([1, 2, 3])
[1, 4, 9]
Related
I have the following decorator:
def my_deco(limit=None):
def wrap_outer(func):
def wrap_inner(*args, **kwargs):
return func(*args, **kwargs)
return wrap_inner
return wrap_outer
It call the actual decorated function by
func(*args, **kwargs)
If a function return multiple values, this will return a single tuple that contains 3, and 4. How do I keep the multiple value returning behavior, instead of a tuple?
#my_deco()
def f():
return 3, 4
As noted in the comments, the mechanism that Python uses to return multiple values is to return a tuple.
This is pretty easy to see in a stripped down example:
def f():
return 3, 4
print(type(f()))
<class 'tuple'>
And, it should also be noted that this isn't something that happens behind the scenes through some magic; 3, 4 alone creates a tuple before the function even returns.
So to answer your question, this is a non-problem. The behavior is the same in both cases.
Let's say I have a function foo that gets a few parameters
def foo(width, height, depth=0):
...
I want to write a wrapper function that gets all of foo's parameters and passes them on, e.g.
def goo(width, height, depth=0):
...
foo(width, height, depth)
...
But this is ugly, since I have to repeat the variables and the default values.
What's the idiomatic way to do this in python?
A few options I thought about:
passing to goo a dictionary called foo_params and calling foo(**foo_params) but then is error prone since I don't know
if all the arguments are there
writing another wrapper for foo that checks if the params with default values are None and if so doesn't pass them
Putting the default values as constants so I won't repeat them
You can use *args and **kwargs syntax to pass an unknown amount of arguments and/or keyword arguments:
>>> def dec(func):
def inner(*args, **kwargs):
print('decorated function')
func(*args, **kwargs)
return inner
>>> #dec
def func(a, b):
return a + b
>>> func(1, 2)
decorated function
>>>
One downside to using *args and **kwargs is that you'll lose the orginal function signature of the decorated function. eg:
>>> help(func)
Help on function inner in module __main__:
inner(*args, **kwargs)
>>>
The solution is to use functools.wraps(). It basically copies of the data from the decorated function to the wrapper function:
>>> from functools import wraps
>>>
>>> def dec(func):
#wraps(func)
def inner(*args, **kwargs):
print('decorated function')
func(*args, **kwargs)
return inner
>>> #dec
def func(a, b):
return a + b
>>> func(1, 2)
decorated function
>>>
A you can see below, if you now do help(func) the original signature for func will be displayed:
>>> help(func)
Help on function func in module __main__:
func(a, b)
>>>
I think you are looking for functools's partial function:
from functools import partial
def foo(a,b):
return a + b
goo = partial(foo, b = 1)
goo(5) # returns 6
I am using a single decorator for two separate functions: one with specification of a decorator argument; and another one without it.
When the optional argument is not passed, the return type is a function (specifically, the inner_function in the decorator). However, when the optional argument is passed it works as expected.
Can you explain what is going on here and why it acts differently in these cases?
def cache_disk(cache_folder="./cache"):
def wrapper(f):
def inner_function(*args, **kwargs):
result = f(*args, **kwargs)
return result
return inner_function
return wrapper
#cache_disk
def func1(data):
return [d for d in data]
#cache_disk(cache_folder='./cache/')
def func2(data):
return [d for d in data]
data = [1,2,3]
print(func1(data))
print(func2(data))
Result:
<function inner_function at 0x7f1f283d5c08>
[1, 2, 3]
Note that:
#decorator # no arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(func) # one 'level' of calls
and that:
#decorator(...): # arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(...)(func) # two 'levels' of calls
In the first case, there is a single argument to the decorator, the func itself. In the second case, the arguments to the decorator are the ... from the # line, and it's the function returned by the decorator that is called with func as an argument.
In your example,
#cache_disk
def func1(data):
...
the decorator cache_disk gets a single, callable argument (func, which becomes args[0]) and returns the wrapper. Then when you call:
print(func1(data))
wrapper gets a single argument (data, which becomes f) and returns inner_function.
Therefore, you have three choices:
Decorate func1 with #cache_disk() (note parentheses), passing no arguments to cache_disk itself and func to wrapper;
Alter cache_disk to behave differently depending on whether it's passed a single, callable argument or something else; or
As #o11c pointed out in the comments, use e.g. cache_disk.wrapper = cache_disk() to provide a convenient alias for the parameter-less version, then decorate with #cache_disk.wrapper.
if you want default values, you need to call the function, which returns a decorator:
#cache_disk()
def func1(data):
return [d for d in data]
I try to understand python decorator
def dec(func):
def wrap(*args,**kw):
print args, kw
return func(*args,**kw)
return wrap
#dec
def myfunc(a=1,b=2,c=3):
return a+b+c
>>> myfunc()
() {}
6
>>> myfunc(1,2,3)
(1, 2, 3) {}
6
>>> myfunc(1,2,c=5)
(1, 2) {'c': 5}
8
>>>
When I run myfunc() args and kw are nothing, but when I run myfunc(1,2,3) and myfunc(1,2,c=5), args and kw were passed to dec function.
As I know,
#dec
def myfunc(...)
equals to myfunc = dec(myfunc) <-- no arguments were mentioned here.
How arguments were passed to wrap function in dec? How to understand these?
Not sure if i understand correctly your problem, but the default values for myfunc arguments are known only to myfunc - your decorator has no knowledge of them, so it cannot print them.
That's why:
myfunc()
results in printing:
() {}
Both *args and **kw are empty for the decorator, but the decorated function will use the default values in this case.
In the second and third case you get the parameters printed, as they are explicitly passed to the decorated function:
def wrap(*args,**kw): <- this is what is actually called when you invoke decorated function, no default values here
print args, kw
return func(*args,**kw) <- this is the function you decorate
#if func has default parameter values, then those will be used
#when you don't pass them to the wrapper but only **inside** func
return wrap
Edit:
It looks like you're mistaking calling the decorated function with decorating the function:
myfunc = dec(myfunc)
decorates myfunc using dec and is equivalent to
#dec
def myfunc(...)
On the other hand, after using either of them:
myfunc(whatever)
calls the wrap function defined in your decorator, which will in turn call the original myfunc
Another way to think of it is by saying:
def wrapper(some_function):
def _inner(*args, **kwargs):
#do some work
return func(*args, **kwargs)
return _inner
#wrapper
def foo(a, b, c):
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
...you're getting a result which is roughly similar to the following:
def foo(a, b, c):
#do some work
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
This isn't exactly right because the #do some work is occurring before the actual foo() call, but as an approximation this is what you're getting. For that reason, the wrapper can't really 'see' the default arguments for foo() if any exist. So a better way to think of it might be:
#always execute this code before calling...
def foo(a, b, c):
return "Foo called and I saw %d, %d, %d" % (a, b, c)
So something really basic might look like this.
def wrap(f):
... def _inner(*a, **k):
... new_a = (ar*2 for ar in a)
... new_k = {}
... for k, v in k.iteritems():
... new_k[k] = v*2
... return f(*new_a, **new_k)
... return _inner
...
>>> def foo(a=2, b=4, c=6):
... return "%d, %d, %d" % (a, b, c)
...
>>> foo()
'2, 4, 6'
>>> foo(1, 5, 7)
'1, 5, 7'
>>> foo = wrap(foo) #actually wrapping it here
>>> foo()
'2, 4, 6'
>>> foo(3, 5, 6)
'6, 10, 12'
>>> foo(3, 5, c=7)
'6, 10, 14'
>>>
Decorators are function wrappers. They give back a function that wraps the original one into some pre- and post-processing code, but still need to call the original function (normally with the same argument as you would call it in absence of a decorator).
Python has two types of arguments, positional and keyword arguments (this has nothing to do with decorators, that's generic python basics). * is for positional (internally is a list), ** for keyword (dictionary) arguments. By specifying both you allow your decorator to accept all at all possible types of arguments and pass them through to the underlying function. The contract of the call is, however, still defined by your function. E.g. if it only takes keyword arguments it will fail when the decorator function passes through a positional argument.
In your particular example, you have some pre-processing code (i.e. code that will run before the original function is called). For example in this code you can print out arguments *args that your original function might fail to accept all together because it does not take any position arguments.
You do not necessarily have to pass through *args and **kwargs. In fact you can define a decorator which makes some decisions based on the arguments you pass in about what to provide to the original function as arguments, e.g.
def dec(fun):
def wrap(*args, **kwargs):
if 'a' in kwargs:
return fun(kwargs[a])
else:
return fun(*args)
return wrap
A number of times I've thought that generator style can be more direct that returning a list, for example,
def foo(input_array):
for x in input_array:
yield processed(x)
vs.
def bar(input_array):
accumulator = []
for x in input_array:
accumulator.append(processed(x))
return accumulator
(okay, if it were really that simple, I'd write map, but you get the point: the generator version is cleaner). However, a return type of a generator is not always desired. Is there a built-in decorator that I can use to change foo into a function returning a list or tuple? The way I'd write it myself is,
import functools
def transform_return_value(transformer):
def inner(f):
#functools.wraps(f)
def new_f(*argv, **kwargs):
return transformer(f(*argv, **kwargs))
return new_f
return inner
#transform_return_value(list)
def foo(input_array):
for x in input_array:
yield processed(x)
To the best of my knowledge (and I've looked, because I've wondered exactly the same thing), no: there is no direct way of doing this with the standard library.
There is a thoroughly tested listify wrapper in unstdlib.py library, though: https://github.com/shazow/unstdlib.py/blob/master/unstdlib/standard/list_.py#L149
def listify(fn=None, wrapper=list):
"""
A decorator which wraps a function's return value in ``list(...)``.
Useful when an algorithm can be expressed more cleanly as a generator but
the function should return an list.
Example::
>>> #listify
... def get_lengths(iterable):
... for i in iterable:
... yield len(i)
>>> get_lengths(["spam", "eggs"])
[4, 4]
>>>
>>> #listify(wrapper=tuple)
... def get_lengths_tuple(iterable):
... for i in iterable:
... yield len(i)
>>> get_lengths_tuple(["foo", "bar"])
(3, 3)
"""
def listify_return(fn):
#wraps(fn)
def listify_helper(*args, **kw):
return wrapper(fn(*args, **kw))
return listify_helper
if fn is None:
return listify_return
return listify_return(fn)
Although #David Wolever's answer is suerly the cleanest way, one thing I often find myself doing (as it doesn't require to define an external decorator) is writing the generator as a local function, like this:
def foo(input_array):
def gen():
for x in input_array:
yield processed(x)
return list(gen())
Here's an alternative, simple decorator without any bells and whistles:
from functools import wraps
from types import GeneratorType
def listify(func):
"""decorator for making generator functions return a list instead"""
#wraps(func)
def new_func(*args, **kwargs):
r = func(*args, **kwargs)
if isinstance(r, GeneratorType):
return list(r)
else:
return r
return new_func
For efficient and concise list definitions try using list comprehension:
def foo(input_array):
return [processed(x) for x in input_array]
If you want a function to return a list, have it return a list. This is much cleaner, easier to understand, read and debug than using decorator.
You may prefer to write this inline, rather than call a function.