I have a question about decorators. I understand what are decorators and I know how to use it, I have read all this tutorial How to make a chain of function decorators?
I understand that :
>>> def my_decorator(fn):
>>> print 'Do something before'
>>> print fn()
>>> def foo():
>>> return 'Hello World!'
>>> foo = my_decorator(foo)
Is the same at that :
>>> def my_decorator(fn):
>>> print 'Do something before'
>>> print fn()
>>> #my_decorator
>>> def foo():
>>> return 'Hello World!'
I know what are closures and why we use closure in a decorator with parameters (to get the decorator parameters in nested function) but that I don't understand is why we use closure and nested functions to get arguments and the function.
How the closure (or something else) can access parameters and the function outside. I am unable to do the same without the #decorator.
Here for example I can access my foo() and the parameters of the function without passing this function in parameter :
def my_decorator(str):
def wrapper(fn):
def inner_function(*args):
print 'Do something before'
return fn(*args)
return inner_function
return wrapper
#my_decorator('test')
def foo(a, b):
return a + b
print foo(1, 1)
How this is possible ?
I found the solution :
In fact the decorator use the closure functionality :
So here is the solution to do the same thing without the decorator and with parameters (it's just to understand the operation, and to learn)
def wrapper(str):
def decorator_factory(fn):
def inner_function(*args):
print 'Do something before'
return fn(*args)
return inner_function
return decorator_factory
#my_decorator('test')
def foo(a, b):
return a + b
# with decorator
print foo(1, 1)
# without decorator
print wrapper('str')(foo)(1, 1)
Decorator is a function that takes one argument (this argument is function) - this is my definition of decorator. In your case wrapper is decorator. While my_decorator is used to collect arguments for inner_function. inner_function is used to replace original [not decorated] function. Step by step explanation would be:
my_decorator is called to collect options for inner_function
my_decorator returns wrapper
wrapper is responsible for capturing original function or in other words decorate it. In your case original function is foo.
wrapper returns replacement function for original (which is inner_function)
from now foo points to inner_function, therefore inner_function is executed then foo is called
Hope it makes things a bit clearer.
Related
I've written a simple decorator:
from functools import wraps
import random
def my_dec(f):
lst = list()
#wraps(f)
def wrapper(*args):
lst.append(random.randint(0, 9))
print(lst)
return f(*args)
return wrapper
#my_dec
def foo():
print("foo called")
Now, if I call foo multiple times lst is not being flushed. Instead, it builds up over time. Thus, multiple calls of foo return an output like this:
foo()
> [4]
> foo called
foo()
> [4, 9]
> foo called
foo()
> [4, 9, 1]
> foo called
...
Why is that? I thought a decorator is just syntactic sugar for my_dec(foo)?! I assumed that each call to my_dec flushes lst.
You're right... The decorator is just syntactic sugar. Specifically:
#decorator
def foo():
pass
is exactly the same thing as:
def foo():
pass
foo = decorator(foo)
Let's be a little more outlandish and rewrite this another way that is mostly equivalent1:
def bar():
pass
foo = decorator(bar)
del bar
Hopefully written out this way, you can see that if I call foo a bunch of times, I'm not calling decorator a bunch of times. decorator only got called once (to help create foo).
Now in your example, your decorator creates a list immediately when it gets called:
def my_dec(f):
lst = list() # list created here!
#wraps(f)
def wrapper(*args):
lst.append(random.randint(0, 9))
print(lst)
return f(*args)
return wrapper
The function returned wrapper gets assigned to your foo, so when you call foo, you're calling wrapper. Note that there is no code in wrapper that would reset lst -- only code that would add more elements to lst so there is nothing here to indicate the lst should get "flushed" between calls.
1(depending on what the decorator does, you might see some differences in the function's __name__ attribute, but otherwise it's the same thing...)
Also note that you'll have one lst for each time the decorator is called. We can go crazy with this one if we like and decorate foo twice:
#my_dec
#my_dec
def foo():
pass
Or we can decorate more than one function:
#my_dec
def foo():
pass
#my_dec
def bar():
pass
And then when we call foo and bar, we'll see that they each accumulate their own (distinct) lists of random numbers. In other words, each time your decorator is applied to something, a new list will be created and each time that "something" is called, the list will grow.
I am using a single decorator for two separate functions: one with specification of a decorator argument; and another one without it.
When the optional argument is not passed, the return type is a function (specifically, the inner_function in the decorator). However, when the optional argument is passed it works as expected.
Can you explain what is going on here and why it acts differently in these cases?
def cache_disk(cache_folder="./cache"):
def wrapper(f):
def inner_function(*args, **kwargs):
result = f(*args, **kwargs)
return result
return inner_function
return wrapper
#cache_disk
def func1(data):
return [d for d in data]
#cache_disk(cache_folder='./cache/')
def func2(data):
return [d for d in data]
data = [1,2,3]
print(func1(data))
print(func2(data))
Result:
<function inner_function at 0x7f1f283d5c08>
[1, 2, 3]
Note that:
#decorator # no arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(func) # one 'level' of calls
and that:
#decorator(...): # arguments
def func(...):
...
is equivalent to:
def func(...):
...
func = decorator(...)(func) # two 'levels' of calls
In the first case, there is a single argument to the decorator, the func itself. In the second case, the arguments to the decorator are the ... from the # line, and it's the function returned by the decorator that is called with func as an argument.
In your example,
#cache_disk
def func1(data):
...
the decorator cache_disk gets a single, callable argument (func, which becomes args[0]) and returns the wrapper. Then when you call:
print(func1(data))
wrapper gets a single argument (data, which becomes f) and returns inner_function.
Therefore, you have three choices:
Decorate func1 with #cache_disk() (note parentheses), passing no arguments to cache_disk itself and func to wrapper;
Alter cache_disk to behave differently depending on whether it's passed a single, callable argument or something else; or
As #o11c pointed out in the comments, use e.g. cache_disk.wrapper = cache_disk() to provide a convenient alias for the parameter-less version, then decorate with #cache_disk.wrapper.
if you want default values, you need to call the function, which returns a decorator:
#cache_disk()
def func1(data):
return [d for d in data]
What is the correct way to wrap mulptiple functions in the loop in Python?
I have universal wrapper and the list of functions. I need to wrap all functions from the list inside one loop but for f in funcs: doesn't work for me.
What will be the correct way to perform this?
def orig_func1(x, y):
print "x=%i y=%i" % (x,y)
def orig_func2(a, b):
print "a=%i b=%i" % (a,b)
def wrapper(func):
f_name = func.__name__
print 'adding hook for function [%s]' % f_name
def inner(*args, **kwargs):
print 'I am before original function'
ret = func(*args, **kwargs)
print 'I am after original function'
return ret
return inner
funcs = [orig_func1, orig_func2]
print funcs
for f in funcs:
f = wrapper(f)
print funcs
and the results showing that functions from the list are not changed:
[<function orig_func1 at 0x022F78F0>, <function orig_func2 at 0x022F7930>]
adding hook for function [orig_func1]
adding hook for function [orig_func2]
[<function orig_func1 at 0x022F78F0>, <function orig_func2 at 0x022F7930>]
x=1 y=2
a=3 b=4
Inside that loop, f is nothing but a local variable. You're not changing anything meaningful unless you modify the list directly. Instead of:
for f in funcs:
f = wrapper(f)
You should do this:
for i, f in enumerate(funcs):
funcs[i] = wrapper(f)
This will change the functions in your list to new, wrapped ones that you can use. But it still won't change the ultimate definition of the function. Nothing will, once it's been defined, short of a complete redefinition or a wrapper used right above the function definition; calling orig_func1 directly will net the same results before and after the for loop. If you want to modify a function at runtime, you'll have to keep referring to this wrapped version of the function that you've just created.
Instead of looping over functions trying to wrap them, you should be using Python Decorators. They are the correct way to modify the behavior of your functions, rather than your current looping method. If the official docs don't make it clear enough, here and here are a couple of tutorials that helped me quite a bit.
Your existing code actually looks a lot like some of the code snippets from my first tutorial link. You should replace your loop with the #decorator syntax instead of the manual wrapping.
That being said, you can accomplish what you originally intended with a comprehension. Replace your loop with this:
funcs = [wrapper(func) for func in funcs]
The other comments and answers were correct that your modification of f in the loop wouldn't work because it had a scope local to that loop and was not modifying your list.
I've implemented required behavior using following approach:
import Module1, Module2
funcs = ['Module1.func1','Module1.func2','Module2.func1','Module2.func2']
hooks = {}
def wrapper(func,f_name):
if not hooks.has_key(f_name):
hooks[f_name] = {'before':[],
'after':[]}
def inner(*args, **kwargs):
for f in hooks[f_name]['before']:
f(*args, **kwargs)
ret = func(*args, **kwargs)
for f in hooks[f_name]['after']:
f(*args, **kwargs)
return ret
return inner
def implementHooks():
for f in funcs:
obj_name, func_name = f.rsplit('.',1)
obj = globals()[obj_name]
func = getattr(obj, func_name)
setattr(obj, func_name, wrapper(func, f))
implementHooks()
def module1_func1_hook():
print 'Before running module1.func1()'
hooks['Module1.func1']['before'] += [module1_func1_hook]
I try to understand python decorator
def dec(func):
def wrap(*args,**kw):
print args, kw
return func(*args,**kw)
return wrap
#dec
def myfunc(a=1,b=2,c=3):
return a+b+c
>>> myfunc()
() {}
6
>>> myfunc(1,2,3)
(1, 2, 3) {}
6
>>> myfunc(1,2,c=5)
(1, 2) {'c': 5}
8
>>>
When I run myfunc() args and kw are nothing, but when I run myfunc(1,2,3) and myfunc(1,2,c=5), args and kw were passed to dec function.
As I know,
#dec
def myfunc(...)
equals to myfunc = dec(myfunc) <-- no arguments were mentioned here.
How arguments were passed to wrap function in dec? How to understand these?
Not sure if i understand correctly your problem, but the default values for myfunc arguments are known only to myfunc - your decorator has no knowledge of them, so it cannot print them.
That's why:
myfunc()
results in printing:
() {}
Both *args and **kw are empty for the decorator, but the decorated function will use the default values in this case.
In the second and third case you get the parameters printed, as they are explicitly passed to the decorated function:
def wrap(*args,**kw): <- this is what is actually called when you invoke decorated function, no default values here
print args, kw
return func(*args,**kw) <- this is the function you decorate
#if func has default parameter values, then those will be used
#when you don't pass them to the wrapper but only **inside** func
return wrap
Edit:
It looks like you're mistaking calling the decorated function with decorating the function:
myfunc = dec(myfunc)
decorates myfunc using dec and is equivalent to
#dec
def myfunc(...)
On the other hand, after using either of them:
myfunc(whatever)
calls the wrap function defined in your decorator, which will in turn call the original myfunc
Another way to think of it is by saying:
def wrapper(some_function):
def _inner(*args, **kwargs):
#do some work
return func(*args, **kwargs)
return _inner
#wrapper
def foo(a, b, c):
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
...you're getting a result which is roughly similar to the following:
def foo(a, b, c):
#do some work
return "Foo called, and I saw %d, %d, %d" % (a, b, c)
This isn't exactly right because the #do some work is occurring before the actual foo() call, but as an approximation this is what you're getting. For that reason, the wrapper can't really 'see' the default arguments for foo() if any exist. So a better way to think of it might be:
#always execute this code before calling...
def foo(a, b, c):
return "Foo called and I saw %d, %d, %d" % (a, b, c)
So something really basic might look like this.
def wrap(f):
... def _inner(*a, **k):
... new_a = (ar*2 for ar in a)
... new_k = {}
... for k, v in k.iteritems():
... new_k[k] = v*2
... return f(*new_a, **new_k)
... return _inner
...
>>> def foo(a=2, b=4, c=6):
... return "%d, %d, %d" % (a, b, c)
...
>>> foo()
'2, 4, 6'
>>> foo(1, 5, 7)
'1, 5, 7'
>>> foo = wrap(foo) #actually wrapping it here
>>> foo()
'2, 4, 6'
>>> foo(3, 5, 6)
'6, 10, 12'
>>> foo(3, 5, c=7)
'6, 10, 14'
>>>
Decorators are function wrappers. They give back a function that wraps the original one into some pre- and post-processing code, but still need to call the original function (normally with the same argument as you would call it in absence of a decorator).
Python has two types of arguments, positional and keyword arguments (this has nothing to do with decorators, that's generic python basics). * is for positional (internally is a list), ** for keyword (dictionary) arguments. By specifying both you allow your decorator to accept all at all possible types of arguments and pass them through to the underlying function. The contract of the call is, however, still defined by your function. E.g. if it only takes keyword arguments it will fail when the decorator function passes through a positional argument.
In your particular example, you have some pre-processing code (i.e. code that will run before the original function is called). For example in this code you can print out arguments *args that your original function might fail to accept all together because it does not take any position arguments.
You do not necessarily have to pass through *args and **kwargs. In fact you can define a decorator which makes some decisions based on the arguments you pass in about what to provide to the original function as arguments, e.g.
def dec(fun):
def wrap(*args, **kwargs):
if 'a' in kwargs:
return fun(kwargs[a])
else:
return fun(*args)
return wrap
References
Before writing this question, I have referred some of the following interesting questions and feel that this scenario is not explained/covered:
How to make a chain of function decorators? (especially answer #2)
How to decorate a method inside a class?
Python decorators in classes
Python Class Based Decorator with parameters that can decorate a method or a function (close but not exactly what I am asking)
I have a function as follows:
def simple_function():
print 'this is a simple_function'
And, I have the following class with some methods
class Test:
def Test_method_1(self, func, args=None, kwargs=None):
print 'this is Test::Test_method_1'
<execute some instructions here>
def Test_method_2(self):
print 'this is Test::Test_method_2'
I am using Test_method_1 and simple_function as follows:
t = Test()
t.Test_method_1(simple_function)
Say if the simple_function takes arbitrary arguments, then the Test_method_1 is called as follows:
def simple_function(a, b, c, d=f, g=h):
print 'this is a simple_function'
t = Test()
t.Test_method_1(simple_function, args=[a, b, c], kwargs={d:f, g:h})
Now, I want to use Test_method_2 as a decorator version of Test_method_1. So the simple_function definition can be written as follows:
t = Test()
#t.Test_method_2
def simple_function():
print 'this is a simple_function'
Note: the Test_method_2 will call Test_method_1 with appropriate arguments.
Now the questions:
1 - Is this possible?
2 - If possible, how do the name of the decorated function (here simple_function) will be passed as a argument to Test_method_2 (including the self)?
3 - [Here is the killer question, I believe] I want to pass arbitrary arguments to both Test_method_2 (the decorator) and simple_function (the function being decorated) - *args and **kwargs - How should the Test_method_2 be defined to receive arguments (including the self)?
The usage of Test_method_2 as a decorator for simple_function with arbitrary arguments for both is as follows:
t = Test()
#t.Test_method_2(a, b, c, d, e=f, g=h)
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
Sure it's possible. All the #decorator syntax does is define the function that follows, then call the decorator, passing in the function that followed and replacing the reference to that function with whatever was returned.
So the following:
#foo
def bar():
pass
is translated to:
def bar():
pass
bar = foo(bar)
This means that your t.Test_method_2() method must expect one argument, the to-be-decorated function, and return something callable:
import functools
def Test_method_2(self, func):
#functools.wraps(func)
def wrapper(self, *args, **kw):
print 'Wrapped function name:', func.__name__
return func(*args, **kw)
return wrapper
would be the minimal decorator that returns a wrapper function, and prints the name of the wrapped function when called. It doesn't matter what the argument is called; I used func here but it can be any legal python identifier.
The self is the standard part of the method signature. Because you are referencing Test_method_2 on an instance t, Python automatically takes care of the self parameter for you as it does with all methods.
Anything after # is just an expression. So if you use the syntax:
#t.Test_method_2(a, b, c, d, e=f, g=h)
then Test_method_2() should instead return a decorator function. One extra level of scope nesting should do that:
def Test_method_2(self, *args, **kw):
def decorator(func):
#functools.wraps(func)
def wrapper(*wrapperargs, **wrapperkw):
fargs = args + wrapperargs
fkw = dict(kw)
fkw.update(wrapperkw)
return func(*fargs, **fkw)
return wrapper
return decorator
Deconstructing this:
#t.Test_method_2(5, 6, 7, 8, baz='spam', ham='eggs')
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
The part after the #, t.Test_method_2(5, 6, 7, 8, baz='spam', ham='eggs') returns the nested function decorator:
#decorator
def simple_function(x, y, z, n=m):
print 'this is a simple_function'
which python then turns into:
simple_function = decorator(simple_function)
and decorator(func) returns wrapper(*wrapperargs, **wrapperkw).
Calling simple_function(1, 2, foo='bar') then results in a call to wrapper(1, 2, foo='bar') which calls the original simple_function() with fargs = [5, 6, 7, 8, 1, 2] and fkw = {'baz': 'spam', 'ham': 'eggs', 'foo': 'bar'} passed in as positional and keyword arguments.
The class decorator pattern you see in the linked questions works in a similar way; the expression after the # returns something that is called; instead of a nested function a class instance is being created instead. It's just two different approaches to storing state for the decorator to use.
The nested functions is a little more compact to write, while using a class gives you more introspection options than a nested scope.