I am having trouble in understanding the flow of control in a Python program having a decorator.
def executor(func):
def innerExecutor():
#print("inside executor")
print("------------------------")
func()
print("------------------------")
return innerExecutor
#executor
def multiply():
n1 = 10
n2 = 20
print("multiplication = ", (n1 * n2))
multiply()
What trouble I am having with is: When the executor() function is called, and it returns the reference of innerExecutor(), where is this reference stored? When is the control actually passed to the inner function? I am very new in Python.
The syntax
#some_decorator
def somefunc():
...
is equivalent to:
somefunc = some_decorator(somefunc)
This is possible in Python because functions are first-class objects. They can be passed as arguments to, and returned from, other functions - so-called higher-order functions.
In your example, the final call to multiply() actually runs innerExector(), a function created by the decorated definition of multiply(). To understand what's going on here, take a closer look at the definition of executor:
def executor(func):
def innerExecutor():
#print("inside executor")
print("------------------------")
func()
print("------------------------")
return innerExecutor
When the code:
#executor
def multiply():
...
is run, the function executor is called with the function multiply as its argument. executor defines the function innerExecutor(), and returns it, but does not execute it. This instance of innerExecutor() is called in place of all future references to multiply().
But wait - usually, we don't want to completely replace the function being decorated. innerExecutor() needs to call the decorated function at some point. Recall that a decorated function definition:
#executor
def multiply():
...
is equivalent to:
def multiply():
...
multiply = executor(multiply)
Passing the function to be decorated as an argument to the decorator, allows the definition of innerExecutor() to call it. But wait - innerExecutor() isn't called until much later, after executor() has returned and it's argument func has gone out of scope. So why does it work?
Thanks to another feature of Python called a closure. What this means is the inner function can refer to local variables defined in the enclosing scope - including arguments passed into the outer function executor - even after exiting the enclosing scope. The magical thing about closures is that the interpreter detects the dependency of the inner function definition on a local variable of the outer function, and keeps it available even after executor() returns. This link goes into more detail about closures.
Finally, the call to multiply() in the last line actually calls the instance of innerExecutor() that was created when the decorated definition was run, which in turn calls the original undecorated version of multiply().
Every time a decorator is used to decorate a function definition, it creates a new instance of the inner function, which replaces the undecorated function. So if you decorate 5 different functions, 5 different instances of innerExecutor() are created. Later, when any of those decorated functions are called, it will instead call instead the appropriate instance of innerExecutor(), which in turn calls the corresponding undecorated function.
it returns the reference of innerExecutor(), where is this reference stored?
When is the control actually passed to the inner function?
It is not stored in anywhere, it is called as below:
innerExecutor(multiply)
That's why your decorator method needs to return its own reference, otherwise, it would be equivalent to this:
None(multiply) # TypeError: 'NoneType' object is not callable
Here's the documentation of aboves behavior
Related
The following code is from Matplotlib's __init__ file:
def _logged_cached(fmt, func=None):
"""
Decorator that logs a function's return value, and memoizes that value.
After ::
#_logged_cached(fmt)
def func(): ...
the first call to *func* will log its return value at the DEBUG level using
%-format string *fmt*, and memoize it; later calls to *func* will directly
return that value.
"""
if func is None: # Return the actual decorator.
return functools.partial(_logged_cached, fmt)
called = False
ret = None
#functools.wraps(func)
def wrapper(**kwargs):
nonlocal called, ret
if not called:
ret = func(**kwargs)
called = True
_log.debug(fmt, ret)
return ret
return wrapper
Question: How does the caching in the above function work? Won't the local variables of _logged_cached be reinitialised on each call?
As I understand it, the local variables are deleted after the function returns (right?). If so, then the caching won't work. Yet, the documentation says the decorated function will return the same object on each call -- and it does. How, though?
The reason the caching works is because of a concept called closures. Your understanding is correct that local variables are deleted after a function exits but the variables used here for caching aren't local to the inner function.
If you use the _logged_cached decorator on a function, the function will essentially be replaced by the nested function wrapper. This wrapper function when returned becomes a closure (a function object) which will remember the non-local variables in the enclosing scope. So it will essentially have a mapping that remembers the variables called and ret. Each time you call the function object (the decorated function) the inner scope will have called and ret available to use.
Normally they would only be usable as read-only in the inner function but by using the keyword nonlocal, the inner function is actually able to change/update the values of the variables called and ret in the closure. Thus that mapping we mentioned before saved in the function object gets its values updated and each subsequential call to the decorated function will make use of those same values.
I am a beginner in python and I am trying to wrap my head around function decorators in python. And I cannot figure out how functions return functions.
I mean in what order does interpreter interprets this function:
def decorator(another_func):
def wrapper():
print('before actual function')
return another_func()
print('pos')
return wrapper
And what is the difference between these 2 statements:-
return wrapper
AND
return wrapper()
I am using Head First Python, but this topic I feel is not described very well in there, please suggest any video or a good resource so that I can understand it.
The key to understanding the difference is understanding that everything is an object in python, including functions. When you use the name of the function without parenthesis (return wrapper), you are returning the actual function itself. When you use parenthesis, you're calling the function. Take a look at the following example code:
def foo(arg):
return 2
bar = foo
baz = foo()
qux = bar()
bar()
If you print baz or qux, it will print two. If you print bar, it will give you the memory address to reference the function, not a number. But, if you call the function, you are now printing the results of th
I cannot figure out how functions return functions.
As already explained by LTheriault, in python everything is an object. Not only this, but also everything happens at runtime - the def statement is an executable statement, which creates a function object from the code within the def block and bind this object to the function's name in the current namespace - IOW it's mostly syntactic sugar for some operations you could code manually (a very welcome syntactic sugar though - building a function object "by hand" is quite a lot of work).
Note that having functions as "first-class citizens" is not Python specific - that's the basis of functional programming.
I mean in what order does interpreter interprets this function:
def decorator(another_func):
def wrapper():
print('before actual function')
return another_func()
print('pos')
return wrapper
Assuming the decorator function is declared at the module top-level: the runtime first takes the code block that follows the def statement, compiles it into a code object, creates a function object (instance of type 'function') from this code object and a couple other things (the arguments list etc), and finally binds this function object to the declared name (nb: 'binds' => "assigns to").
The inner def statement is actually only executed when the outer function is called, and it's executed anew each time the outer function is called - IOW, each call to decorator returns a new function instance.
The above explanation is of course quite simplified (hence partially inexact), but it's enough to understand the basic principle.
I was wondering why the following works:
def wrapper():
def wrap(p=10):
def f():
print(p)
f()
return wrap
f2 = wrapper()
f2()
But this doesn't:
def f():
print(p)
def enhance(f):
def wrap(p=10):
f()
return wrap
f2 = enhance(f)
f2() # NameError: name 'p' is not defined
Is there a way I can modify the second scenario so that variable p is defined? I was playing around with function decorators but couldn't figure it out how to expose the variables to the function I'm passing into the decorators.
I think I understand what you are really asking. You're taking about decorators, not variable scope. You say you can't figure out how to "expose the variables to the function I'm passing to the decorators." In your case 2, the function you are passing to enhance doesn't have any variables (arguments). Suppose we give it an argument, like this:
def f(p):
print(p)
def enhance(f):
def wrap(p=10):
f(p) # pass the argument to f
return wrap
f2 = enhance(f)
f2()
Now you have a function, named enhance, which can be used as a decorator. The function to be decorated takes one argument. The decorator will replace this function with a new function, which can be called with one or zero arguments. If called with no arguments it will get the value "10" as a default.
Decorators replace one function with another function. In general it isn't the decorator's job to supply the arguments, except in the case of default arguments as you are trying to do. The arguments come from the code that calls the function.
because in example 2 you’re referencing p that is not defined in one function and used as a parameter in the other function each of which is defined in their own scope.
in example 1 a function defined within the scope of another ie a nested function, has access to the outer functions scope (and therefore its variables)
Here is the code:
def caller(callee):
callee()
def wrapper():
def a():
print v0
for i in range(5):
v0 = i*i
caller(a)
wrapper()
The above code outputs:
0
1
4
9
16
But I don't understand how a resolves the variable v0. I can not find the related python docs regarding this language feature.
The scope of variables defined in a function includes all the nested functions within it. So variables defined in wrapper() are accessible within a(), because this function is nested in it. This is known as lexical scoping, and it's often used in creating closures.
This is explained in the Python Resolution of Names documentation:
A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name.
The function a is defined per invocation of wrapper. When a given instance of a is called, it looks up v0 within the context of its definition, which is the specific call to wrapper that defined it. By the time a has been called, v0 has been defined within wrapper, and a uses that definition of v0.
In this example, the invocation of wrapper that defines a is still active when a is called, but it need not be. In other words, wrapper could return, and a reference to a from that context could still be called. In this case it would be a closure. But that does not occur in this example.
Here's an example that does use closures:
def foo(x):
def bar():
return x
return bar
f1 = foo(123)
f2 = foo(456)
print(f1())
print(f2())
The output is:
123
456
When foo is called, it returns an instance of bar which evaluates x in the context in which foo was called. In the first invocation x is 123, and in the second it is 456. Those bindings persist even after the calls to foo have returned.
Remember that all code in a function is not run until the function is called. By the time a() is called, v0 has been defined in an outer scope, so it can recognize the variable and evaluate it.
I am having trouble understanding why the following happens. I am having a decorator which does nothing except it checks whether a function is a method. I thought I have understood what method in Python is, but obviously, this is not the case:
import inspect
def deco(f):
def g(*args):
print inspect.ismethod(f)
return f(*args)
return g
class Adder:
#deco
def __call__(self, a):
return a + 1
class Adder2:
def __call__(self, a):
return a + 2
Adder2.__call__ = deco(Adder2.__call__)
Now, running the following:
>>> a = Adder()
>>> a(1)
False
2
>>> a2 = Adder2()
>>> a2(1)
True
3
I would expect for this code to print True two times.
So, decorating the function manually as in the Adder2 is not an exact equivalent to decorating via the #deco function?
Can someone be so glad and explain why this happens?
Methods are functions that are associated with a class. Methods are only created when you retrieve them from an already defined class; a method is a wrapper around a function, with a reference to the class as well (and optionally a reference to the instance).
What happens in the first case is: Python compiles your class definition Adder. It finds the decorator definition and a function. The decorator is passed the function, returning a new function. That function is added to the class definition (stored in the class __dict__). All this time you are dealing with a python function, not a method. That happens later.
When you then call a(1), a lookup reveals that the instance doesn't have a __call__ but the Adder class does, so it is retrieved using __getattribute__(). This finds a function (your deco decorator), which is a descriptor so it's __get__() method is called (so Adder.__call__.__get__(a, Adder)), returning a bound method, which is then called and passed in the 1 value. The method is bound because instance is not None when __get__() is called. Your decorator, which wrapped a function at class building time, prints False because it was passed an unwrapped function to start with.
In the second case, however, you retrieve a method (again via __getattribute__() calling __get__() on the undecorated Adder2.__call__ function), this time unbound (as there is no instance, only a class passed to __get__() (the full call is Adder2.__call__.__get__(None, Adder2)), and you then decorate that method. Now ismethod() prints True.
Note that in Python 3, the latter case changes. In Python 3 there no longer is a concept of an unbound method, only functions and bound methods. The term 'bound' is thus dropped altogether. Your second case would also print False as Adder2.__call__.__get__(None, Adder2) returns a function in that case.
Inside a class definition, __call__ is a function, not a method. The act of accessing the function through attribute lookup (e.g. using the dot syntax), such as with Adder2.__call__, returns an unbound method. And a2.__call__ returns a bound method (with self bound to a2).
Note that in Python3 the concept of unbound method has been dropped. There, Adder2.__call__ is a function as well.