We Keep Coding

Memory management for functions and decorators

My question starts with how decorators work in python. Let's look at the following code:
def decorator(F):
def wrapper():
print("start")
F()
print("end")
return wrapper
def f1():
print("f1")
decorated_f1 = decorator(f1)
decorated_f1()
It prints,
start
f1
end
First of all, as much as I know, python uses lazy evaluation. Therefore, it does not evaluate F(), until it is required (when actually decorated_f1() is called). By then, the scope of argument is F is over (end of the decorator function). I would like to know, what python stores in memory when a function is created and overall the memory management that happens for decorators.
The second part of my question is about the results that I get after running the following codes,
def decorator(F):
def wrapper():
print("start")
F()
print("end")
return wrapper
def f1():
print("f1")
decorated_f1 = decorator(f1)
def f1():
print("new f1")
decorated_new_f1 = decorator(f1)
decorated_f1()
decorated_new_f1()
it results in,
start
f1
end
start
new f1
end
However, the following code
def f1():
print("f1")
F = f1
def wrapper():
print("start")
F()
print("end")
wrapped_f1 = wrapper
def f1():
print("new f1")
F = f1
def wrapper():
print("start")
F()
print("end")
wrapped_new_f1 = wrapper
wrapped_f1()
wrapped_new_f1()
produces,
start
new f1
end
start
new f1
end
This makes me confused, because I thought these two codes should be very similar in output. That is why I need help for clarifying what and how things are stored in memory when functions or decorators are declared in python.

You are correct that the second and third example are quite close in how they work. However there is one key difference. The scope the functions are evaluated in. The scope in this case can be seen as all the variables some place in the code has access too, and where those come from. You might have heard of the global scope, this is the scope all code is evaluated in and all variables that are global are accessible by any part of the code. There are also function scopes. These are the variables defined inside of the function, and other functions won't have access to these variables.
To further understand the difference there is one more think you should know. Functions are a reference type in Python, this means that when you declare one all later uses reference the value rather then directly accessing the value. This is the same as for lists and dictionaries.
Why does this all matter? Well because a decorator stores the function scope to correctly evaluate later. This includes the current value of F in this function
def decorator(F):
def wrapper():
print("start")
F()
print("end")
return wrapper
However, if you don't pass F as an argument then the decorator will store the function as a reference to the global function. Thus when you chance F in the third example the F() in wrapped_f1 also changes. This explains the output you see.
Lastly, as a tip. There is a specific syntax for using decorators on functions, and it might make using them easier. Usually a decorator would look like this.
def decorator(F):
def wrapper():
print("start decorating")
F()
print("End decorating")
return wrapper
def undecorated_func():
print("Hellor world!")
#decorator
def decorated_func():
print("Hello World!")
undecorated_func()
decorated_func()
Then the output is
Hello world!
Start decorating
Hello World!
End decorating

How to print python code after decorators are applied?

I'm trying to understand a complex python decorator and I'd like to see the resulting python code after the decorator has been applied, i.e. the actual python code for the wrapped function once python has applied the decorator. Is this possible?
I thought I could do this with inspect but didn't get the result I expected. I tried putting the code in a module, importing it then using print(inspect.getsource(module_name)), but this prints the original source code with the #decorator, not the wrapped functions after decorators applied.
def my_decorator(func):
def wrapper():
print("Something is happening before the function is called.")
func()
print("Something is happening after the function is called.")
return wrapper
#my_decorator
def say_whee():
print("Whee!")
say_whee()
The above produces a result similar to code below before python runs it. I'd like to see the code for resulting function after the decorator above has been applied
def say_whee2():
print("Whee!")
def say_whee_decorated():
print("Something is happening before the function is called.")
say_whee2()
print("Something is happening after the function is called.")
say_whee_decorated()
I expected to see something like say_whee_decorated() as a result of inspect.getsource() but instead, got:
def my_decorator(func):
def wrapper():
print("Something is happening before the function is called.")
func()
print("Something is happening after the function is called.")
return wrapper
#my_decorator
def say_whee():
print("Whee!")

I can see what you're after but I think you're misunderstanding the nature of decorators (and of inspect.getsource()), they don't create new code to wrap the function, they just pass the decorated function into the decorating one and execute it, there is no intermediate code created so you shouldn't expect to see any.

The mechanism of equating functions in python

I learned that we can equate one function with another in Python like this:
def func_1(x)
print("func_1")
print(x)
def func_2(x)
print("func_2")
print(x)
func_1 =func_2
So here what is happening is every call to func_1 executes func_2.
However, I read about decorators and the following is a simple code illustrating them:
def our_decorator(func):
def function_wrapper(x):
print("Before calling " + func.__name__)
func(x)
print("After calling " + func.__name__)
return function_wrapper
def foo(x):
print("Hi, foo has been called with " + str(x))
print("We call foo before decoration:")
foo("Hi")
print("We now decorate foo with f:")
foo = our_decorator(foo)
print("We call foo after decoration:")
foo(42)
Here as we can see in the following line:
foo = our_decorator(foo)
something like the previous function equation is taking place . I thought this is how decorators might be working, i.e., replacing call to the decoratee by call to decorator .
However, under this impression if I write up a code like the following:
def our_decorator():
def function_wrapper(x):
print("Before calling " )
foo(x)
print("After calling " )
return function_wrapper
def foo(x):
print("Hi, foo has been called with " + str(x))
print("We call foo before decoration:")
foo("Hi")
print("We now decorate foo with f:")
foo = our_decorator()
print("We call foo after decoration:")
foo(42)
The above results in infinite recursion printing infinite number of "Before calling ".
So, I could conclude that a decorator must be something which takes a function as an argument.
So the equating of functions differ in these two cae, namely equating with function which takes another function as an argument, and equating two functions which don't take another function as argument.
How might this two differing in internal implementation?

What you call "equating functions" is really just a variable assignment. A function definition (with def) creates a function and assigns it to a variable name. After you do func_1 = func_2, you have 2 variables referring to the same function.
What happens in your decorator examples is a natural consequence of the previous paragraph. Leave a comment if you need further clarification on that.

I hope your question can be answered by explaining few terms.
You are using the term "equating" for what it is commonly called "assigning". name = expr is an assignment statement. The name name is given (assigned) to the object which is the result of the expression expr.
In Python functions are not treated special. This is sometimes expressed with the sentence "functions are first class objects" and basically it means that function objects can be assigned to variables (names), passed as arguments etc. the same way as numbers, strings and other objects.
A (function) decorator processes another function. It is a function that takes the function to be decorated as its argument and returns a "decorated" (i.e. enhanced or modifed in some way) version of it. Sometimes it only registers the function e.g. as a handler or as a part of an API and returns it unchanged. There is a special syntax for it:
#decorator
def func(...):
pass
which is equivalent to:
func = decorator(func)
and also:
#decorator(args)
def func(...):
pass
which is equivalent to:
_real_decorator = decorator(args)
func = _real_decorator(func)
Because this #decorator syntax is so simple to use and easy to read, you normally don't write:
func = decorator(func)
To summarize:
func1 = some_func is plain assignment, giving other name to some_func.
func2 = create_function() this would be called a function factory in some languages. You wrote that in your question.
func = decorate_function(func) this is a decoration of func
Note: there exist class decorators, they are very similar, but enhance class definitions instead of functions.

A decorator looks like this:
def decorator_with_args(*args, **kwargs):
def wrapper(f: "the function being decorated"):
def wrapped(*args, **kwargs):
# inside here is the code that is actually executed
# when calling the decorated function. This should
# always include...
f(*args, **kwargs)
# and usually return its result
return wrapped
return wrapper
# or
def decorator_without_args(f: "the function being decorated"):
def wrapped(*args, **kwargs):
# as above
return f(*args, **kwargs)
return wrapped
and is used by:
#decorator_with_args("some", "args")
def foo(x):
print("foo:", x) # or whatever
#decorator_without_args
def bar(x):
print("bar:", x)
This is equivalent to defining each function without the #decorator... magic and applying the decorator afterwards
def baz(x):
print("baz:", x)
baz = decorator_with_args("some", "arguments")(baz)
# or
baz = decorator_without_args(baz)
In your example code you call foo inside your decorator, then you decorate foo with that decorator, so you end up recursing infinitely. Every time you call foo, it runs your decorator code which also invokes foo. Every time your decorator invokes foo, it runs your decorator code which also invokes foo. Every time your decorator's decorator invokes foo, is runs your decorator code which also... etc

Python decorator - Trying to understand a simple example

I am trying to understand python decorators.
I devised that simple example where I want the decorator function to be a custom log that just print error if for instance I try to sum_ and int and a str
def log(fun):
try:
return fun(*args)
except:
print('error!')
#log
def sum_(a,b):
return a+b
This returns "error" already simply when I define the function. I suspect there are multiple wrong things in what I did... I tried to look into the other questions about that topic, but I find them all to intricate to understand how such a simple example should be drafted ,esp how to pass the arguments from the original function.
All help and pointers appreciated

That's because you're not forwarding the args from the function to your decorator, and the catch-all exception catches the NameError for args; one of the reasons to always specify the exception class.
Here's a modified version of your code with the try-catch removed and the function arguments correctly forwarded:
def log(fun):
def wrapper(*args):
print('in decorator!')
return fun(*args)
return wrapper
#log
def sum_(a,b):
return a+b
print sum_(1,2)

The reason you're getting an error is simply because args is undefined in your decorator. This isn't anything special about decorators, just a regular NameError. For this reason you probably want to restrict your exception clause to just TypeErrors so that you're not silencing other errors. A full implementation would be
import functools
def log(fun):
#functools.wraps(fun)
def inner(*args):
try:
return fun(*args)
except TypeError:
print('error!')
return inner
#log
def sum_(a, b):
return a + b
It's also a good idea to decorate your inner functions with the functools.wrap decorator, which transfers the name and docstring from your original function to your decorated one.

The log decorator, in this case, does not return a function, but a value. This may point on an assumption that the decorator function replaces the original function, where in fact, it is called to create a replacement function.
A fix which may represent the intention:
def log(fun):
def my_func(*args):
try:
return fun(*args)
except:
print('error!')
return my_func
In this case, my_func is the actual function which is called for sum_(1, 2), and internally, it calls the original function (the original sum_) which the decorator received as an argument.
A trivial example that illustrates the order of the actions:
def my_decorator(fun):
print 'This will be printed first, during module load'
def my_wrapper(*args):
print 'This will be printed during call, before the original func'
return fun(*args)
return my_wrapper()
#my_decorator
def func():
print('This will be printed in the original func')

#decorators in Python: why the inner defined function?

I'm just starting with Python and I have just been exposed to decorators. I wrote the following code, mimicking what I am seeing, and it works:
def decorator_function(passed_function):
def inner_decorator():
print('this happens before')
passed_function()
print('this happens after')
return inner_decorator
#decorator_function
def what_we_call():
print('The actual function we called.')
what_we_call()
But then I wrote this, which throws errors:
def decorator_function(passed_function):
print('this happens before')
passed_function()
print('this happens after')
#decorator_function
def what_we_call():
print('The actual function we called.')
what_we_call()
So, why do we need to have that inner nested function inside the decorator function? what purpose does it serve? Wouldn't it be simpler to just use the syntax of the second? What am I not getting?
The funny thing is that BOTH have the same (correct) output, but the second on has error text as well, saying "TypeError: 'NoneType' object is not callable"
Please use language and examples suitable for someone just starting with Python, his first programming language - and also new to OOP as well! :) Thanks.

The reason is that when you wrap what_we_call in decorator_function by doing:
#decorator_function
def what_we_call():
...
What you're doing is:
what_we_call = decorator_function(what_we_call)
In you first example it works because you don't run the inner_function actually, you only initialise it, and then you return the new inner_function back (that you will call later when call the decorated what_we_call):
def decorator_function(passed_function):
def inner_decorator():
print('this happens before')
passed_function()
print('this happens after')
return inner_decorator
Contrarily, in your second example you're going to run 2 print statements and the passed_function (what_we_call in our case) in the between:
def decorator_function(passed_function):
print('this happens before')
passed_function()
print('this happens after')
In other words, you don't return a function in the example of before:
what_we_call = decorator_function(what_we_call)
You run the code (and you see the output), but then decorator_function returns 'None' to what_we_call (overwriting the original function), and when you call 'None' as if it was a function Python complains.

Python decorators are basically just syntactic sugar. This:
#decorator
def fn(arg1, arg2):
return arg1 + arg2
Becomes this:
def fn(arg1, arg2):
return arg1 + arg2
fn = decorator(fn)
That is, a decorator basically accepts a function as an argument, and returns "something"; that "something" is bound to the name of the decorated function.
In nearly all cases, that "something" should be another function, because it is expected that fn will be a function (and will probably be called as though it is).