This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
Please consider the code below.
a = []
def func1(x):
return x
for i in range(3):
def func2():
return func1(i)
a.append(func2)
for k in range(3):
print(a[k]())
This prints out
2
2
2
From 'The use of aliases' in http://gestaltrevision.be/wiki/python/aliases (last section) and in 'Scope' section in http://gestaltrevision.be/wiki/python/functions_basics, I learnt that function parameters are actually aliases of arguments that are passed.
So according to that, in
def func1(x): return x
for i in range(3):
def func2(): return func1(i)
I reasoned since x would be stored as an alias to i, even though i is reassigned each time the loop is executed, it would not matter to its alias, x.
So I expected the first three lines to output 0, 1, 2 instead of 2, 2, 2.
Can you explain what I did wrong here? Thanks
You create here a closure func2 that uses variable i from the enclosing scope.
The func2's instances are created by DEF statement, at the time of the FOR loop execution.
Then you execute the func2's after the FOR loop is exited.
In python a loop variable doesn't destroyed after the loop exit.
So your closure uses the current value of the i in enclosing scope, at the moment of exiting from the loop.
So in this code func1 changes nothing, the result will be the same without it.
If you want your code to work the way you want, do as follows
def func2(i):
def func1():
return i
return func1
a = [func2(i) for i in range(3)]
for k in range(3):
print(a[k]()) # prints 0 1 2
Now, why didn't your code work? Well it has to do with when objects are bound in to names in a closure, which func1 is. In your code the parameter x to func1 is being bound at runtime. Hence as each function in a has func1(i) and the value of i at printing time is 2, you get all 2. So the solution is to bind it at compile time i.e when func2 returns func1, i is already bound in func1.
When you do this:
for i in range(3):
def func2():
return func1(i)
You're "redefining" your func2 for every i in [0, 1, 2]. The final definition that lives is:
def func2():
return func1(2)
It's as simple as that. Unfortunately it does not behave the way you expect.
Related
Python
Can anyone help me to understand this code, I am new to Python, how does this function work?
def makeInc(x):
def inc(y):
return y + x
return inc
incOne = makeInc(1)
incFive = makeInc(5)
print(incOne(5)) # returns 6
print(incFive(5)) # returns 10
Higher-order functions
Functions like makeInc that in turn, return another function are called higher order functions. Usually, functions are known to accept data as input and return data as output. With higher order functions, functions instead of data, either return code as output or accept code as input. This code is wrapped into a function. In Python, functions are first class citizens which means functions, just like data, can be passed around. For instance:
myvariable = print
Notice, how I have assigned print to myvariable and how I have dropped the parentheses after print Functions without parentheses are called function objects. This means myvariable now is just another name for print:
print("Hello World!")
myvariable("Hello World!")
Both of the above statements do the exact same thing. What can be assigned to variables can also be returned from functions:
def myfunction():
return print
myfunction()("Hello World!");
Now let's look at your example:
def makeInc(x):
def inc(y):
return y + x
return inc
makeInc is a function that accepts a parameter called x. It then defines another nested inner function called inc which takes in a parameter called y. The thing about nested functions is that they have access to the variables of the enclosing function as well. Here, inc is the inner function but it has access to x which is a variable of the enclosing outer scope.
The last statement return inc returns the inner function to the caller of makeInc. What makeInc essentially is doing, is creating a custom function based on the parameter it receives.
For instance:
x = makeInc(10)
makeInc will first accept 10 and then return a function that takes in an argument y and it increments y by 10.
Here, x is a function that takes in any argument y and then increments it by 10:
x(42) # Returns 52
nonlocal
However, there is a caveat when using nested functions:
def outer():
x = 10
def inner():
x = 20
inner()
print(x) # prints 10
Here, you would assume that the last print statement will print 20. But no! When you assign x = 20 in the inner function, it creates a new local variable called x which is initialized to 20. The outer x remains untouched. To modify the outer x, use the nonlocal keyword:
def outer():
x = 10
def inner():
nonlocal x = 20
inner()
print(x) # prints 20
If you are directly reading x inside inner() instead of assigning to it, you do not need nonlocal.
What is happening here is that makeInc() returns a function handle pointing to specific implementation of inc(). So, calling makeInc(5) "replaces" the x in inc(y) to 5 and returns the callable handle of that function. This handle is saved in incFive. You can now call the function as defined (inc(y)). Since you set x=5 before, the result will be y+5.
I'm a fairly experienced C/C++ (and to some degree, Java) programmer. I'm learning python, but I'm baffled at some strange (for my backgroung) behaviors of the language.
I'm learning about nested function and closures (reading "Learning Python", that seems a really good source for me).
I understand that if I nest a def inside a for loop when I call the created function, it looks up the last value of the captured loop variable (as it captures by reference, as a C++ programmer would put it)
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
and running the program the result is
>>> for f in funcs:
f()
3
3
3
3
Now, I was wrapping my head around this when I stumbled upon this (what to me seems) an inconsistency: if I do
for i in range(4):
funcs[i]()
0
1
2
3
more baffling, if I do
>>> i = 2
>>> funcs[i]()
2
and now, all functions in list returns 2:
for f in funcs:
f()
2
2
2
2
there must be some scope related question that I can't grasp
First, this creates a list of four functions.
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
Each of these functions looks up the value of i and then prints it.
This loops through the list of function and calls each one:
>>> for f in funcs:
f()
As stated above, these functions look up i, which is 3 right now due to the for i in range(4) loop that completed earlier, so you get four printouts of 3.
Now you loop again, using i as the loop variable:
for i in range(4):
funcs[i]()
0
1
2
3
The first time through the loop, i is 0, so when the function looks up i, it gets 0, and then prints that. Then it changes to 1, then 2, then 3.
The following code simply changes i in yet another way, and calls a function:
>>> i = 2
>>> funcs[i]()
2
You could've called any of those functions and they still would've printed 2, because that's the value of i now. You're just getting lost because you looped over range(4) to create these functions, then you looped over range(4) to index the list of functions, and you keep reusing i, and then you reassign i and also use it to index the list.
If you want each function's printed value of i to be fixed at what it was when you defined the function, the easiest way to do that is with a default argument, as those are evaluated when the function is defined rather than when it's called:
funcs = []
for i in range(4):
def f(num=i):
print(num)
funcs.append(f)
For the sake of completeness, this is an alternate implementation:
def getfunc(i):
return lambda: i
funcs = []
for i in range(5):
funcs.append(getfunc(i))
for item in funcs:
print(item())
Your functions
def f():
print(i)
print the current value of i.
If you write
for i in range(4):
funcs[i]()
then i is being set to 0,1,2,3 as you go through the loop. That's what for i in range(4) means.
If you write
for f in funcs:
f()
then i continues with whatever value it already had.
take some time for me to understand, actually in this example,
You will find f.__closure__ is None if you print it, ie. nothing related to closure, it's just about procedure the undefined local var i look for its value:
it can't find its value in local scope, finally find it in global scope (like python MRO)
There is no inconsistency here. The value of i in f() depends on the value of i from the parent scope. After you've run the first for i in range(4) i has the value of the last item in the range, which is 3, and thus all subsequent calls to f() will print 3
If you run
for i in range(4):
funcs[i]()
you redefine the value of i at each iteration step, and so you get 0,1,2,3 as the values printed by f. Doing
for x in range(4):
funcs[x]()
will not affect the value of i and so you'll get 3 as the value of i in all function calls
This question already has answers here:
Modify global list inside a function
(5 answers)
Closed 6 years ago.
I was experimenting with this code:
def a():
#global p
p.append(4);
d=9
p=[2,3];
d=8
a();
print p # ----> [2, 3, 4]
print d # ----> 8
The variable d value is not changed as I didn't use the global keyword. But the list p was modified in function even though I didn't use global. Are all lists global by default in functions?
The critical difference is the assignment here. You are fine calling methods on existing global objects, but you can't assign to them without calling them global. In your code, the name d is being reassigned to reference another value. If you changed p with assignment you'd have a similar result
def a():
p = [5, 7] # new local variable, doesn't change global
p.append(9) # doesn't change global p
This makes sense if you think about what happens when python encounters the name for the first time. In the function you've provided, python will see p.append and say "hm, I don't have a local by the name p, let me look in the enclosing scope." It sees the global p and uses that.
In the example I've shown, python will say "there's no explicit global so I assume this is supposed to be a new local variable." and create one.
Names in python are just references. If python followed the behavior you are expecting you'd need a global for every function you called, let me explain:
def a():
p.append(1) # I should need 'global p' to do this
This would mean if you had
def g():
...
def f():
g() # this would also need 'global g', otherwise how does it see g?
def f2():
global g
def g(): # changes the global g function
return 0
First of all, this post does NOT answer my question or give me any guide to answer my question at all.
My question is about mechanism function resolving non-local variables.
Code
# code block 1
def func():
vals = [0, 0, 0]
other_vals = [7, 8, 9]
other = 12
def func1():
vals[1] += 1
print(vals)
def func2():
vals[2] += 2
print vals
return (func1, func2)
f1, f2 = func()
Try to run f1, f2:
>>> f1()
[0, 1, 0]
>>> f2
[0, 1, 2]
This shows that the object previously referred by vals are shared by f1 and f2, and not garbage collected after execution of func.
Will objects referred by other_vals and other be garbage collected? I think so. But how does Python decide not to garbage collect vals?
Assumption 1
Python interpreter will resolve variable names within func1 and func2 to figure out references inside the function, and increase the reference count of [0, 0, 0] by 1 preventing it from garbage collection after the func call.
But if I do
# code block 2
def outerfunc():
def innerfunc():
print(non_existent_variable)
f = outerfunc()
No error reported. Further more
# code block 3
def my_func():
print(yet_to_define)
yet_to_define = "hello"
works.
Assumption 2
Variable names are resolved dynamically at run time. This makes observations in code block 2 and 3 easy to explain, but how did the interpreter know it need to increase reference count of [0, 0, 0] in code block 1?
Which assumption is correct?
Your first example creates a closure; also see Why aren't python nested functions called closures?, Can you explain closures (as they relate to Python)?, and What exactly is contained within a obj.__closure__?.
The closure mechanism ensures that the interpreter stores a reference to vals in the returned function objects func1 and func2. Your Assumption 1 is correct: that reference prevents vals from being garbage collected when func returns.
In your second example, the interpreter cannot see a reference to non_existent_variable in the enclosing scope(s), but that's ok because your Assumption 2 is also correct, so you're free to use names that haven't yet been bound to objects at function declaration time, so long as the name is in scope when you actually call the function.
The answer to "how did the interpreter know it need to increase reference count of [0, 0, 0] in code block 1?" is that the closure mechanism is an explicit thing the interpreter does when it executes a function definition, i.e., when it's creating a function object from the function definition in your script.
Every Python function object (both normal def-style functions and lambdas) has an attribute to store this closure information, with a minor difference between Python 2 and Python 3. See the links at the start of this answer for details, but I will mention here that Python 3 provides the nonlocal keyword, which works a bit like the global keyword: nonlocal allows you to make assignments to closed-over simple variables; J.F. Sebastian's answer has a simple example illustrating the use of nonlocal.
Note that with nested functions the inner function definitions are processed each time you call the outer function, which allows you to do things like:
def func(vals):
def func1():
vals[1] += 1
print(vals)
def func2():
vals[2] += 2
print(vals)
return func1, func2
f1, f2 = func([0, 0, 0])
f1()
f2()
f1, f2 = func([10, 20, 30])
f1()
f2()
output
[0, 1, 0]
[0, 1, 2]
[10, 21, 30]
[10, 21, 32]
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
Below is an example I got from someone's blog about python closure.
I run it in python 2.7 and get a output different from my expect.
flist = []
for i in xrange(3):
def func(x):
return x*i
flist.append(func)
for f in flist:
print f(2)
My expected output is: 0, 2, 4
But the output is: 4, 4, 4
Is there anyone could help to explain it?
Thank you in advance.
Loops do not introduce scope in Python, so all three functions close over the same i variable, and will refer to its final value after the loop finishes, which is 2.
It seems as though nearly everyone I talk to who uses closures in Python has been bitten by this. The corollary is that the outer function can change i but the inner function cannot (since that would make i a local instead of a closure based on Python's syntactic rules).
There are two ways to address this:
# avoid closures and use default args which copy on function definition
for i in xrange(3):
def func(x, i=i):
return x*i
flist.append(func)
# or introduce an extra scope to close the value you want to keep around:
for i in xrange(3):
def makefunc(i):
def func(x):
return x*i
return func
flist.append(makefunc(i))
# the second can be simplified to use a single makefunc():
def makefunc(i):
def func(x):
return x*i
return func
for i in xrange(3):
flist.append(makefunc(i))
# if your inner function is simple enough, lambda works as well for either option:
for i in xrange(3):
flist.append(lambda x, i=i: x*i)
def makefunc(i):
return lambda x: x*i
for i in xrange(3):
flist.append(makefunc(i))
You are not creating closures. You are generating a list of functions which each access the global variable i which is equal to 2 after the first loop. Thus you end up with 2 * 2 for each function call.
Each function accesses the global i.
functools.partial comes to rescue:
from functools import partial
flist = []
for i in xrange(3):
def func(x, multiplier=None):
return x * multiplier
flist.append(partial(func, multiplier=i))