This question already has answers here:
What do lambda function closures capture?
(7 answers)
Closed 6 months ago.
While I was investigating a problem I had with lexical closures in Javascript code, I came along this problem in Python:
flist = []
for i in xrange(3):
def func(x): return x * i
flist.append(func)
for f in flist:
print f(2)
Note that this example mindfully avoids lambda. It prints "4 4 4", which is surprising. I'd expect "0 2 4".
This equivalent Perl code does it right:
my #flist = ();
foreach my $i (0 .. 2)
{
push(#flist, sub {$i * $_[0]});
}
foreach my $f (#flist)
{
print $f->(2), "\n";
}
"0 2 4" is printed.
Can you please explain the difference ?
Update:
The problem is not with i being global. This displays the same behavior:
flist = []
def outer():
for i in xrange(3):
def inner(x): return x * i
flist.append(inner)
outer()
#~ print i # commented because it causes an error
for f in flist:
print f(2)
As the commented line shows, i is unknown at that point. Still, it prints "4 4 4".
Python is actually behaving as defined. Three separate functions are created, but they each have the closure of the environment they're defined in - in this case, the global environment (or the outer function's environment if the loop is placed inside another function). This is exactly the problem, though - in this environment, i is modified, and the closures all refer to the same i.
Here is the best solution I can come up with - create a function creater and invoke that instead. This will force different environments for each of the functions created, with a different i in each one.
flist = []
for i in xrange(3):
def funcC(j):
def func(x): return x * j
return func
flist.append(funcC(i))
for f in flist:
print f(2)
This is what happens when you mix side effects and functional programming.
The functions defined in the loop keep accessing the same variable i while its value changes. At the end of the loop, all the functions point to the same variable, which is holding the last value in the loop: the effect is what reported in the example.
In order to evaluate i and use its value, a common pattern is to set it as a parameter default: parameter defaults are evaluated when the def statement is executed, and thus the value of the loop variable is frozen.
The following works as expected:
flist = []
for i in xrange(3):
def func(x, i=i): # the *value* of i is copied in func() environment
return x * i
flist.append(func)
for f in flist:
print f(2)
Here's how you do it using the functools library (which I'm not sure was available at the time the question was posed).
from functools import partial
flist = []
def func(i, x): return x * i
for i in range(3):
flist.append(partial(func, i))
for f in flist:
print(f(2))
Outputs 0 2 4, as expected.
look at this:
for f in flist:
print f.func_closure
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
It means they all point to the same i variable instance, which will have a value of 2 once the loop is over.
A readable solution:
for i in xrange(3):
def ffunc(i):
def func(x): return x * i
return func
flist.append(ffunc(i))
What is happening is that the variable i is captured, and the functions are returning the value it is bound to at the time it is called. In functional languages this kind of situation never arises, as i wouldn't be rebound. However with python, and also as you've seen with lisp, this is no longer true.
The difference with your scheme example is to do with the semantics of the do loop. Scheme is effectively creating a new i variable each time through the loop, rather than reusing an existing i binding as with the other languages. If you use a different variable created external to the loop and mutate it, you'll see the same behaviour in scheme. Try replacing your loop with:
(let ((ii 1)) (
(do ((i 1 (+ 1 i)))
((>= i 4))
(set! flist
(cons (lambda (x) (* ii x)) flist))
(set! ii i))
))
Take a look here for some further discussion of this.
[Edit] Possibly a better way to describe it is to think of the do loop as a macro which performs the following steps:
Define a lambda taking a single parameter (i), with a body defined by the body of the loop,
An immediate call of that lambda with appropriate values of i as its parameter.
ie. the equivalent to the below python:
flist = []
def loop_body(i): # extract body of the for loop to function
def func(x): return x*i
flist.append(func)
map(loop_body, xrange(3)) # for i in xrange(3): body
The i is no longer the one from the parent scope but a brand new variable in its own scope (ie. the parameter to the lambda) and so you get the behaviour you observe. Python doesn't have this implicit new scope, so the body of the for loop just shares the i variable.
The problem is that all of the local functions bind to the same environment and thus to the same i variable. The solution (workaround) is to create separate environments (stack frames) for each function (or lambda):
t = [ (lambda x: lambda y : x*y)(x) for x in range(5)]
>>> t[1](2)
2
>>> t[2](2)
4
I'm still not entirely convinced why in some languages this works one way, and in some another way. In Common Lisp it's like Python:
(defvar *flist* '())
(dotimes (i 3 t)
(setf *flist*
(cons (lambda (x) (* x i)) *flist*)))
(dolist (f *flist*)
(format t "~a~%" (funcall f 2)))
Prints "6 6 6" (note that here the list is from 1 to 3, and built in reverse").
While in Scheme it works like in Perl:
(define flist '())
(do ((i 1 (+ 1 i)))
((>= i 4))
(set! flist
(cons (lambda (x) (* i x)) flist)))
(map
(lambda (f)
(printf "~a~%" (f 2)))
flist)
Prints "6 4 2"
And as I've mentioned already, Javascript is in the Python/CL camp. It appears there is an implementation decision here, which different languages approach in distinct ways. I would love to understand what is the decision, exactly.
The variable i is a global, whose value is 2 at each time the function f is called.
I would be inclined to implement the behavior you're after as follows:
>>> class f:
... def __init__(self, multiplier): self.multiplier = multiplier
... def __call__(self, multiplicand): return self.multiplier*multiplicand
...
>>> flist = [f(i) for i in range(3)]
>>> [g(2) for g in flist]
[0, 2, 4]
Response to your update: It's not the globalness of i per se which is causing this behavior, it's the fact that it's a variable from an enclosing scope which has a fixed value over the times when f is called. In your second example, the value of i is taken from the scope of the kkk function, and nothing is changing that when you call the functions on flist.
The reasoning behind the behavior has already been explained, and multiple solutions have been posted, but I think this is the most pythonic (remember, everything in Python is an object!):
flist = []
for i in xrange(3):
def func(x): return x * func.i
func.i=i
flist.append(func)
for f in flist:
print f(2)
Claudiu's answer is pretty good, using a function generator, but piro's answer is a hack, to be honest, as it's making i into a "hidden" argument with a default value (it'll work fine, but it's not "pythonic").
I didn't like how solutions above created wrappers in the loop. Note: python 3.xx
flist = []
def func(i):
return lambda x: x * i
for i in range(3):
flist.append(func(i))
for f in flist:
print f(2)
Related
I'm a fairly experienced C/C++ (and to some degree, Java) programmer. I'm learning python, but I'm baffled at some strange (for my backgroung) behaviors of the language.
I'm learning about nested function and closures (reading "Learning Python", that seems a really good source for me).
I understand that if I nest a def inside a for loop when I call the created function, it looks up the last value of the captured loop variable (as it captures by reference, as a C++ programmer would put it)
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
and running the program the result is
>>> for f in funcs:
f()
3
3
3
3
Now, I was wrapping my head around this when I stumbled upon this (what to me seems) an inconsistency: if I do
for i in range(4):
funcs[i]()
0
1
2
3
more baffling, if I do
>>> i = 2
>>> funcs[i]()
2
and now, all functions in list returns 2:
for f in funcs:
f()
2
2
2
2
there must be some scope related question that I can't grasp
First, this creates a list of four functions.
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
Each of these functions looks up the value of i and then prints it.
This loops through the list of function and calls each one:
>>> for f in funcs:
f()
As stated above, these functions look up i, which is 3 right now due to the for i in range(4) loop that completed earlier, so you get four printouts of 3.
Now you loop again, using i as the loop variable:
for i in range(4):
funcs[i]()
0
1
2
3
The first time through the loop, i is 0, so when the function looks up i, it gets 0, and then prints that. Then it changes to 1, then 2, then 3.
The following code simply changes i in yet another way, and calls a function:
>>> i = 2
>>> funcs[i]()
2
You could've called any of those functions and they still would've printed 2, because that's the value of i now. You're just getting lost because you looped over range(4) to create these functions, then you looped over range(4) to index the list of functions, and you keep reusing i, and then you reassign i and also use it to index the list.
If you want each function's printed value of i to be fixed at what it was when you defined the function, the easiest way to do that is with a default argument, as those are evaluated when the function is defined rather than when it's called:
funcs = []
for i in range(4):
def f(num=i):
print(num)
funcs.append(f)
For the sake of completeness, this is an alternate implementation:
def getfunc(i):
return lambda: i
funcs = []
for i in range(5):
funcs.append(getfunc(i))
for item in funcs:
print(item())
Your functions
def f():
print(i)
print the current value of i.
If you write
for i in range(4):
funcs[i]()
then i is being set to 0,1,2,3 as you go through the loop. That's what for i in range(4) means.
If you write
for f in funcs:
f()
then i continues with whatever value it already had.
take some time for me to understand, actually in this example,
You will find f.__closure__ is None if you print it, ie. nothing related to closure, it's just about procedure the undefined local var i look for its value:
it can't find its value in local scope, finally find it in global scope (like python MRO)
There is no inconsistency here. The value of i in f() depends on the value of i from the parent scope. After you've run the first for i in range(4) i has the value of the last item in the range, which is 3, and thus all subsequent calls to f() will print 3
If you run
for i in range(4):
funcs[i]()
you redefine the value of i at each iteration step, and so you get 0,1,2,3 as the values printed by f. Doing
for x in range(4):
funcs[x]()
will not affect the value of i and so you'll get 3 as the value of i in all function calls
I am reading Hackers and Painters and am confused by a problem mentioned by the author to illustrate the power of different programming languages.
The problem is:
We want to write a function that generates accumulators—a function that takes a number n, and returns a function that takes another number i and returns n incremented by i. (That’s incremented by, not plus. An accumulator has to accumulate.)
The author mentions several solutions with different programming languages. For example, Common Lisp:
(defun foo (n)
(lambda (i) (incf n i)))
and JavaScript:
function foo(n) { return function (i) { return n += i } }
However, when it comes to Python, the following codes do not work:
def foo(n):
s = n
def bar(i):
s += i
return s
return bar
f = foo(0)
f(1) # UnboundLocalError: local variable 's' referenced before assignment
A simple modification will make it work:
def foo(n):
s = [n]
def bar(i):
s[0] += i
return s[0]
return bar
I am new to Python. Why doesn the first solution not work while the second one does? The author mentions lexical variables but I still don't get it.
s += i is just sugar for s = s + i.*
This means you assign a new value to the variable s (instead of mutating it in place). When you assign to a variable, Python assumes it is local to the function. However, before assigning it needs to evaluate s + i, but s is local and still unassigned -> Error.
In the second case s[0] += i you never assign to s directly, but only ever access an item from s. So Python can clearly see that it is not a local variable and goes looking for it in the outer scope.
Finally, a nicer alternative (in Python 3) is to explicitly tell it that s is not a local variable:
def foo(n):
s = n
def bar(i):
nonlocal s
s += i
return s
return bar
(There is actually no need for s - you could simply use n instead inside bar.)
*The situation is slightly more complex, but the important issue is that computation and assignment are performed in two separate steps.
An infinite generator is one implementation. You can call __next__ on a generator instance to extract successive results iteratively.
def incrementer(n, i):
while True:
n += i
yield n
g = incrementer(2, 5)
print(g.__next__()) # 7
print(g.__next__()) # 12
print(g.__next__()) # 17
If you need a flexible incrementer, one possibility is an object-oriented approach:
class Inc(object):
def __init__(self, n=0):
self.n = n
def incrementer(self, i):
self.n += i
return self.n
g = Inc(2)
g.incrementer(5) # 7
g.incrementer(3) # 10
g.incrementer(7) # 17
In Python if we use a variable and pass it to a function then it will be Call by Value whatever changes you make to the variable it will not be reflected to the original variable.
But when you use a list instead of a variable then the changes that you make to the list in the functions are reflected in the original List outside the function so this is called call by reference.
And this is the reason for the second option does work and the first option doesn't.
In python what is the equivalent of the quote operator? I am finding the need to delay evaluation. For example, suppose in the following lisp psuedocode I have:
a = '(func, 'g)
g = something
(eval a)
What I am doing is deferring evaluation of g till a later time. This is necessary because I want to define g later. What is the equivalent idea of this psuedocode in python?
a = lambda: func(g)
g = something
a()
This isn't quite the most literal translation - the most literal translation would use a string and eval - but it's probably the best fit. Quoting probably isn't what you wanted in Lisp anyway; you probably wanted to delay something or create a lambda. Note that func and g are closure variables in the lambda function, rather than symbols, so if you call a from an environment with different bindings for func or g, it'll still use the variables from a's environment of definition.
Using eval for delaying evaluation is bad, both in Lisp and Python.
in Python, and in Lisp, you can delay evaluation using a closure:
def print_it(x):
def f():
print g(x)
return f
f = print_it(42)
def g(x):
return x * x
f()
Please note that what is captured in a closure is not the value of a variable, but the variable itself and this is sometimes surprising:
fa = []
for x in range(10):
def g():
print x
fa.append(g)
for f in fa:
f() # all of them will print 9
x = 42
fa[0]() # the output will be 42
to solve this problem (that can also be present in Common Lisp) you may see things like:
for x in range(10):
def g(x = x):
print x
fa.append(g)
or (in CL) things like
(let ((a a))
(lambda () (print a)))
Python also has a lambda special form for anonymous functions, but they are limited to one single expression and cannot contain any statement. A locally def-ined function instead is a regular function without any limitations.
for x in range(10):
# print is a statement in Python 2.x and cannot be in a lambda
fa.append(lambda x=x: sys.stdout.write(str(x) + "\n"))
Finally Python 2.x has a syntax limitation and closed-over variables are read-only because if there is an assignment (or augmented-assignment) in a function there are only two possibilities:
The variable is a global (and has been previously declared so with global x)
The variable is a local
and in particular it's ruled out that a variable being assigned could be a local of an enclosing function scope.
Python 3.x removed this limitation by providing a new possible declaration nonlocal x and now the famous adder example can be implemented as
def adder(x):
def f(y):
nonlocal x
x += y
return x
return f
So today in computer science I asked about using a function as a variable. For example, I can create a function, such as returnMe(i) and make an array that will be used to call it. Like h = [help,returnMe] and then I can say h1 and it would call returnMe("Bob"). Sorry I was a little excited about this. My question is is there a way of calling like h.append(def function) and define a function that only exists in the array?
EDIT:
Here Is some code that I wrote with this!
So I just finished an awesome FizzBuzz with this solution thank you so much again! Here's that code as an example:
funct = []
s = ""
def newFunct(str, num):
return (lambda x: str if(x%num==0) else "")
funct.append(newFunct("Fizz",3))
funct.append(newFunct("Buzz",5))
for x in range(1,101):
for oper in funct:
s += oper(x)
s += ":"+str(x)+"\n"
print s
You can create anonymous functions using the lambda keyword.
def func(x,keyword='bar'):
return (x,keyword)
is roughly equivalent to:
func = lambda x,keyword='bar':(x,keyword)
So, if you want to create a list with functions in it:
my_list = [lambda x:x**2,lambda x:x**3]
print my_list[0](2) #4
print my_list[1](2) #8
Not really in Python. As mgilson shows, you can do this with trivial functions, but they can only contain expressions, not statements, so are very limited (you can't assign to a variable, for example).
This is of course supported in other languages: in Javascript, for example, creating substantial anonymous functions and passing them around is a very idiomatic thing to do.
You can create the functions in the original scope, assign them to the array and then delete them from their original scope. Thus, you can indeed call them from the array but not as a local variable. I am not sure if this meets your requirements.
#! /usr/bin/python3.2
def a (x): print (x * 2)
def b (x): print (x ** 2)
l = [a, b]
del a
del b
l [0] (3) #works
l [1] (3) #works
a (3) #fails epicly
You can create a list of lambda functions to increment by every number from 0 to 9 like so:
increment = [(lambda arg: (lambda x: arg + x))(i) for i in range(10)]
increment[0](1) #returns 1
increment[9](10) #returns 19
Side Note:
I think it's also important to note that this (function pointers not lambdas) is somewhat like how python holds methods in most classes, except instead of a list, it's a dictionary with function names pointing to the functions. In many but not all cases instance.func(args) is equivalent to instance.__dict__['func'](args) or type(class).__dict__['func'](args)
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
Below is an example I got from someone's blog about python closure.
I run it in python 2.7 and get a output different from my expect.
flist = []
for i in xrange(3):
def func(x):
return x*i
flist.append(func)
for f in flist:
print f(2)
My expected output is: 0, 2, 4
But the output is: 4, 4, 4
Is there anyone could help to explain it?
Thank you in advance.
Loops do not introduce scope in Python, so all three functions close over the same i variable, and will refer to its final value after the loop finishes, which is 2.
It seems as though nearly everyone I talk to who uses closures in Python has been bitten by this. The corollary is that the outer function can change i but the inner function cannot (since that would make i a local instead of a closure based on Python's syntactic rules).
There are two ways to address this:
# avoid closures and use default args which copy on function definition
for i in xrange(3):
def func(x, i=i):
return x*i
flist.append(func)
# or introduce an extra scope to close the value you want to keep around:
for i in xrange(3):
def makefunc(i):
def func(x):
return x*i
return func
flist.append(makefunc(i))
# the second can be simplified to use a single makefunc():
def makefunc(i):
def func(x):
return x*i
return func
for i in xrange(3):
flist.append(makefunc(i))
# if your inner function is simple enough, lambda works as well for either option:
for i in xrange(3):
flist.append(lambda x, i=i: x*i)
def makefunc(i):
return lambda x: x*i
for i in xrange(3):
flist.append(makefunc(i))
You are not creating closures. You are generating a list of functions which each access the global variable i which is equal to 2 after the first loop. Thus you end up with 2 * 2 for each function call.
Each function accesses the global i.
functools.partial comes to rescue:
from functools import partial
flist = []
for i in xrange(3):
def func(x, multiplier=None):
return x * multiplier
flist.append(partial(func, multiplier=i))