My question starts with how decorators work in python. Let's look at the following code:
def decorator(F):
def wrapper():
print("start")
F()
print("end")
return wrapper
def f1():
print("f1")
decorated_f1 = decorator(f1)
decorated_f1()
It prints,
start
f1
end
First of all, as much as I know, python uses lazy evaluation. Therefore, it does not evaluate F(), until it is required (when actually decorated_f1() is called). By then, the scope of argument is F is over (end of the decorator function). I would like to know, what python stores in memory when a function is created and overall the memory management that happens for decorators.
The second part of my question is about the results that I get after running the following codes,
def decorator(F):
def wrapper():
print("start")
F()
print("end")
return wrapper
def f1():
print("f1")
decorated_f1 = decorator(f1)
def f1():
print("new f1")
decorated_new_f1 = decorator(f1)
decorated_f1()
decorated_new_f1()
it results in,
start
f1
end
start
new f1
end
However, the following code
def f1():
print("f1")
F = f1
def wrapper():
print("start")
F()
print("end")
wrapped_f1 = wrapper
def f1():
print("new f1")
F = f1
def wrapper():
print("start")
F()
print("end")
wrapped_new_f1 = wrapper
wrapped_f1()
wrapped_new_f1()
produces,
start
new f1
end
start
new f1
end
This makes me confused, because I thought these two codes should be very similar in output. That is why I need help for clarifying what and how things are stored in memory when functions or decorators are declared in python.
You are correct that the second and third example are quite close in how they work. However there is one key difference. The scope the functions are evaluated in. The scope in this case can be seen as all the variables some place in the code has access too, and where those come from. You might have heard of the global scope, this is the scope all code is evaluated in and all variables that are global are accessible by any part of the code. There are also function scopes. These are the variables defined inside of the function, and other functions won't have access to these variables.
To further understand the difference there is one more think you should know. Functions are a reference type in Python, this means that when you declare one all later uses reference the value rather then directly accessing the value. This is the same as for lists and dictionaries.
Why does this all matter? Well because a decorator stores the function scope to correctly evaluate later. This includes the current value of F in this function
def decorator(F):
def wrapper():
print("start")
F()
print("end")
return wrapper
However, if you don't pass F as an argument then the decorator will store the function as a reference to the global function. Thus when you chance F in the third example the F() in wrapped_f1 also changes. This explains the output you see.
Lastly, as a tip. There is a specific syntax for using decorators on functions, and it might make using them easier. Usually a decorator would look like this.
def decorator(F):
def wrapper():
print("start decorating")
F()
print("End decorating")
return wrapper
def undecorated_func():
print("Hellor world!")
#decorator
def decorated_func():
print("Hello World!")
undecorated_func()
decorated_func()
Then the output is
Hello world!
Start decorating
Hello World!
End decorating
Related
Playing with decorator, founds something strange: I have something simple as below (a decorator to register the function to a list:
stored_f = []
def register(f):
stored_f.append(f.__name__)
return f
#register
def say_hello(name):
print(f"Hello {name}")
#register
def ur_awesome(name):
return f"{name}, you are awesome!"
I thought the above code will do nothing (it will print nothing out), because I haven't executed anything, but actually a print command "print(stored_f)" will have a output:
['say_hello', 'ur_awesome']
Meaning the line "stored_f.append(f.__name__)" in register() actually got executed, twice. This is weird to me.
I tried to define a wrapper() inside register like normally and then return wrapper, the issue will go away. But still I don't understand why the above code will partially execute the decorator without calling it.
Importing and running a Python file executes it and adding the decorator with #register executes the decorator.
So, even though your script only consists of the variable declaration, the decorator and two function definitions, when you run it, both those definitions will be executed (defining the functions) and decorator functions on them will be executed, so that the functions are correctly decorated.
Added after your comment: consider this:
stored_f = []
def register(f):
stored_f.append(f.__name__)
print('it runs!')
return f
#register
def say_hello(name):
print(f"Hello {name}")
#register
def ur_awesome(name):
return f"{name}, you are awesome!"
You'll find that, if you run this script, the output is:
it runs!
it runs!
All the code in the decorator runs and you just return the same function it received, so f will work just the same before and after.
Also, if you add say_hello("john") to the end of that script, you'll find the decorator does not run again, only the function runs again.
Also, compare this:
stored_f = []
def register(f):
stored_f.append(f.__name__)
def decorated_f(*args):
print('it runs!')
f(*args)
return decorated_f
#register
def say_hello(name):
print(f"Hello {name}")
say_hello("john")
say_hello("pete")
This results in:
it runs!
Hello john
it runs!
Hello pete
Perhaps that's closer to what you were expecting. Note that the decorator doesn't return the original function, it returns the new one.
Based on Grimar's answer, I did the following, may explain it from another angle, at least to myself:
def do_twice(f):
print('This will be executed anyway')
def wrapper():
f()
f()
return wrapper
#do_twice
def say_hello():
print('This is say hello')
output:
This will be executed anyway
if I run:
say_hello()
output:
This is say hello
This is say hello
In Python, is there a nice way to apply a decorator (or something similar) not to a whole function, but to a subsegment of a function body?
What I want might look something like this:
def deco_double(f):
def wrapper():
f()
f()
return wrapper
def foo():
#deco_double:
print("hello")
print("stack")
#deco_double:
print("overflow")
foo()
So the execution result be like:
hello
stack
hello
stack
overflow
overflow
It's okay if the solution is not exactly a decorator, what I need is a method to wrap function subsegments, but in a well abstracted way. Thank you in advance :)
edit:
Just separating into multiple functions is not an option for me. I'm developing a framework for programmers and I want them to write functions in a compact way (for writing foo() in the example. Also, what I'm actually doing is far more complicated than just repetitions)
edit2:
By now it seems I have no choice but to expect framework users to somehow declare named functions...
Function bodies are compiled into a single "code" object that is run as a whole - changing the way this code object is run, inserting things in it, and such are things that can be as complicated as the language code itself (i.e. the code that actually "executes" Python bytecode).
So, any changes in the flow of execution are far easier done by using the statements in the language that already do that.
If you want the equivalent of a generic decorator in parts inside a function body, the easiest thing to do is to subdivide that function itself into inner functions - and then you can apply your transforms, or execute each part more than once, by simply calling those functions (and even decorate them directly).
However, in the case you bring in your question, you could, and probably should, just use a plain old for loop:
def foo():
for _ in (0, 1):
print("hello")
print("stack")
for _ in (0, 1):
print("overflow")
For arbitrary "decorator like behavior", as I wrote above, just use nested functions:
def foo():
#deco_double
def part1()
print("hello")
print("stack")
part1()
#deco_double:
def part2():
print("overflow")
part2()
You will have to extract that partial functionality of foo() to separate functions and then apply the decorator as a function and call it expilictly.
def deco_double(f):
def wrapper():
f()
f()
return wrapper
def my_f1():
print("hello")
print("stack")
def my_f2():
print("overflow")
def foo():
deco_double(my_f1)()
deco_double(my_f2)()
foo()
Simply declare two functions with any name with decorator #deco_double and call them in any other function, in your case foo() and then simply call foo().
def deco_double(f):
def wrapper():
f()
f()
return wrapper
#deco_double
def func1():
print("hello")
print("stack")
#deco_double
def func2():
print("overflow")
def foo():
func1()
func2()
foo()
Output of this code.
hello
stack
hello
stack
overflow
overflow
You'll likely need to pass functions using lambda if you want to achieve this:
def doubler(f):
f()
f()
def foo():
doubler(lambda: print("hello"))
doubler(lambda: print("world"))
foo()
"hello"
"hello"
"world"
"world"
Wrap the code you want to be run twice in the lambda, and pass the function to doubler
I learned that we can equate one function with another in Python like this:
def func_1(x)
print("func_1")
print(x)
def func_2(x)
print("func_2")
print(x)
func_1 =func_2
So here what is happening is every call to func_1 executes func_2.
However, I read about decorators and the following is a simple code illustrating them:
def our_decorator(func):
def function_wrapper(x):
print("Before calling " + func.__name__)
func(x)
print("After calling " + func.__name__)
return function_wrapper
def foo(x):
print("Hi, foo has been called with " + str(x))
print("We call foo before decoration:")
foo("Hi")
print("We now decorate foo with f:")
foo = our_decorator(foo)
print("We call foo after decoration:")
foo(42)
Here as we can see in the following line:
foo = our_decorator(foo)
something like the previous function equation is taking place . I thought this is how decorators might be working, i.e., replacing call to the decoratee by call to decorator .
However, under this impression if I write up a code like the following:
def our_decorator():
def function_wrapper(x):
print("Before calling " )
foo(x)
print("After calling " )
return function_wrapper
def foo(x):
print("Hi, foo has been called with " + str(x))
print("We call foo before decoration:")
foo("Hi")
print("We now decorate foo with f:")
foo = our_decorator()
print("We call foo after decoration:")
foo(42)
The above results in infinite recursion printing infinite number of "Before calling ".
So, I could conclude that a decorator must be something which takes a function as an argument.
So the equating of functions differ in these two cae, namely equating with function which takes another function as an argument, and equating two functions which don't take another function as argument.
How might this two differing in internal implementation?
What you call "equating functions" is really just a variable assignment. A function definition (with def) creates a function and assigns it to a variable name. After you do func_1 = func_2, you have 2 variables referring to the same function.
What happens in your decorator examples is a natural consequence of the previous paragraph. Leave a comment if you need further clarification on that.
I hope your question can be answered by explaining few terms.
You are using the term "equating" for what it is commonly called "assigning". name = expr is an assignment statement. The name name is given (assigned) to the object which is the result of the expression expr.
In Python functions are not treated special. This is sometimes expressed with the sentence "functions are first class objects" and basically it means that function objects can be assigned to variables (names), passed as arguments etc. the same way as numbers, strings and other objects.
A (function) decorator processes another function. It is a function that takes the function to be decorated as its argument and returns a "decorated" (i.e. enhanced or modifed in some way) version of it. Sometimes it only registers the function e.g. as a handler or as a part of an API and returns it unchanged. There is a special syntax for it:
#decorator
def func(...):
pass
which is equivalent to:
func = decorator(func)
and also:
#decorator(args)
def func(...):
pass
which is equivalent to:
_real_decorator = decorator(args)
func = _real_decorator(func)
Because this #decorator syntax is so simple to use and easy to read, you normally don't write:
func = decorator(func)
To summarize:
func1 = some_func is plain assignment, giving other name to some_func.
func2 = create_function() this would be called a function factory in some languages. You wrote that in your question.
func = decorate_function(func) this is a decoration of func
Note: there exist class decorators, they are very similar, but enhance class definitions instead of functions.
A decorator looks like this:
def decorator_with_args(*args, **kwargs):
def wrapper(f: "the function being decorated"):
def wrapped(*args, **kwargs):
# inside here is the code that is actually executed
# when calling the decorated function. This should
# always include...
f(*args, **kwargs)
# and usually return its result
return wrapped
return wrapper
# or
def decorator_without_args(f: "the function being decorated"):
def wrapped(*args, **kwargs):
# as above
return f(*args, **kwargs)
return wrapped
and is used by:
#decorator_with_args("some", "args")
def foo(x):
print("foo:", x) # or whatever
#decorator_without_args
def bar(x):
print("bar:", x)
This is equivalent to defining each function without the #decorator... magic and applying the decorator afterwards
def baz(x):
print("baz:", x)
baz = decorator_with_args("some", "arguments")(baz)
# or
baz = decorator_without_args(baz)
In your example code you call foo inside your decorator, then you decorate foo with that decorator, so you end up recursing infinitely. Every time you call foo, it runs your decorator code which also invokes foo. Every time your decorator invokes foo, it runs your decorator code which also invokes foo. Every time your decorator's decorator invokes foo, is runs your decorator code which also... etc
I have below decorator demonstration code. If I execute it without explicitly calling greet function, it is executing print statement inside decorator function and outputs Inside decorator.
I am unable to understand this behavior of decorator. How the time_decorator is called even if I didn't call greet function?
I am using Python 3.
def time_decorator(original_func):
print('Inside decorator')
def wrapper(*args, **kwargs):
start = time.clock()
result = original_func(*args, **kwargs)
end = time.clock()
print('{0} is executed in {1}'.format(original_func.__name__, end-start))
return result
return wrapper
#time_decorator
def greet(name):
return 'Hello {0}'.format(name)
Decorators are called at start time (when the python interpreter reads the code as the program starts), not at runtime (when the decorated function is actually called).
At runtime, it is the wrapped function wrapper which is called and which itself calls the decorated function and returns its result.
So this is totally normal that the print line gets executed.
If, i.e, you decorate 10 functions, you will see 10 times the print output. No need to even call the decorated functions for this to happen.
Move the print inside wrapper and this won't happen anymore.
Decorators as well as metaclasses are part of what is called meta-programming (modify / create code, from existing code). This is a really fascinating aspect of programming which takes time to understand but offers amazing possibilities.
time_decorator is executed during function decoration. wrapper is not (this is function invoked when decorated greet() is called).
# is just syntactic sugar. Following code snippets are equivalent.
Decorator syntax:
#time_decorator
def greet(name):
return 'Hello {0}'.format(name)
Explicit decoration process - decorator is a function that returns new function based on another one.
def greet(name):
return 'Hello {0}'.format(name)
greet = time_decorator(greet)
Say for example I have a function
def A(): pass
I call this function from two different places
def B():
#some code
A()
#some code
def C():
#some code
A()
#some code
Decorators as we know, in layman language is something like a hook which lets you perform a set of operation before and after the function call. We apply decorators on function definition using #.
My requirement here is like I don't want the decorator to be applied every time the function is called. Like in the above example if I apply decorator on function A. It would be applied in both invokes. Can we use # at the invoke of the function A?
Something like,
def A(): pass
def B():
#some code
A()
#some code
def C():
#some code
#decorator
A()
#some code
This way I don't have to worry about merging my code after every release of particular opensource project.
A decorator is just syntactic sugar, where the outcome of the #expression is called and the return value is used to replace the decorated object.
If you only need to have a decorated function in one location you can do decorator(A)() to have it return a decorated version just to be used in C and call that decorated object on the spot.
Better yet, store that decorated version for C to use so you don't keep decorating it each time:
A_decorated = decorator(A)
def C():
# some code
A_decorated()
# some code