I'm a fairly experienced C/C++ (and to some degree, Java) programmer. I'm learning python, but I'm baffled at some strange (for my backgroung) behaviors of the language.
I'm learning about nested function and closures (reading "Learning Python", that seems a really good source for me).
I understand that if I nest a def inside a for loop when I call the created function, it looks up the last value of the captured loop variable (as it captures by reference, as a C++ programmer would put it)
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
and running the program the result is
>>> for f in funcs:
f()
3
3
3
3
Now, I was wrapping my head around this when I stumbled upon this (what to me seems) an inconsistency: if I do
for i in range(4):
funcs[i]()
0
1
2
3
more baffling, if I do
>>> i = 2
>>> funcs[i]()
2
and now, all functions in list returns 2:
for f in funcs:
f()
2
2
2
2
there must be some scope related question that I can't grasp
First, this creates a list of four functions.
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
Each of these functions looks up the value of i and then prints it.
This loops through the list of function and calls each one:
>>> for f in funcs:
f()
As stated above, these functions look up i, which is 3 right now due to the for i in range(4) loop that completed earlier, so you get four printouts of 3.
Now you loop again, using i as the loop variable:
for i in range(4):
funcs[i]()
0
1
2
3
The first time through the loop, i is 0, so when the function looks up i, it gets 0, and then prints that. Then it changes to 1, then 2, then 3.
The following code simply changes i in yet another way, and calls a function:
>>> i = 2
>>> funcs[i]()
2
You could've called any of those functions and they still would've printed 2, because that's the value of i now. You're just getting lost because you looped over range(4) to create these functions, then you looped over range(4) to index the list of functions, and you keep reusing i, and then you reassign i and also use it to index the list.
If you want each function's printed value of i to be fixed at what it was when you defined the function, the easiest way to do that is with a default argument, as those are evaluated when the function is defined rather than when it's called:
funcs = []
for i in range(4):
def f(num=i):
print(num)
funcs.append(f)
For the sake of completeness, this is an alternate implementation:
def getfunc(i):
return lambda: i
funcs = []
for i in range(5):
funcs.append(getfunc(i))
for item in funcs:
print(item())
Your functions
def f():
print(i)
print the current value of i.
If you write
for i in range(4):
funcs[i]()
then i is being set to 0,1,2,3 as you go through the loop. That's what for i in range(4) means.
If you write
for f in funcs:
f()
then i continues with whatever value it already had.
take some time for me to understand, actually in this example,
You will find f.__closure__ is None if you print it, ie. nothing related to closure, it's just about procedure the undefined local var i look for its value:
it can't find its value in local scope, finally find it in global scope (like python MRO)
There is no inconsistency here. The value of i in f() depends on the value of i from the parent scope. After you've run the first for i in range(4) i has the value of the last item in the range, which is 3, and thus all subsequent calls to f() will print 3
If you run
for i in range(4):
funcs[i]()
you redefine the value of i at each iteration step, and so you get 0,1,2,3 as the values printed by f. Doing
for x in range(4):
funcs[x]()
will not affect the value of i and so you'll get 3 as the value of i in all function calls
Related
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
What do lambda function closures capture?
(7 answers)
Closed 6 months ago.
This is so weird, if I make a list of lambda functions and with a for, I print the output of each function, everything works, but if I individually print the output of a single function like the first one, it gives me the output of the last function or I don't even know what it gives me, for example:
X = [1,2,3,4]
L = []
for i in range(len(X)):
L.append(lambda x: X[i]**x)
for i in range(len(X)):
print(L[i](2))
This gives me:
1
4
9
16
That is correct, but if i want only the first one:
print(L[0](2))
# -> 16
And if I want the second one does the same, and so on, I checked the lambda functions were all different and they are. I don't know what's going on
The lambda references the global variable i, so after the for loop, i==3, computing X[3]**2:
X = [1,2,3,4]
L = []
for i in range(len(X)):
L.append(lambda x: X[i]**x)
for f in L:
print(f(2))
Output:
16
16
16
16
A way to fix is to capture the current value of global i as a local parameter i when the function is defined:
X = [1,2,3,4]
L = []
for i in range(len(X)):
L.append(lambda x, i=i: X[i]**x) # capture i as a parameter
for f in L:
print(f(2))
Output:
1
4
9
16
You are expecting the value of i to be part of the function, not the name. That's not how function definitions work, either with def statements or lambda expressions.
I'd recommend defining a function maker, so that your expected function can close over the local value of i.
def make_function(i):
return lambda x: X[i]*x*x
Now i refers not to a global variable i, but to a local variable i in the function call. Every call to make_function creates a different local variable initialized with the value of the argument, and that variable is what your function will refer to when it is called.
for i in range(len(X)):
L.append(make_function(i))
Your lambda closes around the variable i. Despite what the for loops make it look like, there's actually only one variable i in your entire code, so when you call L[i](2) or L[0](2), you're using the current value of i. Let's look at your first example.
for i in range(len(X)):
print(L[i](2))
Here, we call L[i] with the enclosing value of i, so this is really basically
for i in range(len(X)):
print(X[i] ** 2)
On the other hand, in your second example
print(L[0](2))
The i value is the i from the final loop iteration above. To get around this, you need to explicitly capture the current value of i.
L.append(lambda x, i=i: X[i]**x)
This exploits one of Python's least intuitive features to explicitly capture a value in a lambda.
Here is a very simplified example of what I am trying to do:
x = 3
def f():
print(x)
x = 5
f() #f prints 5 but I want it to print 3.
Is there a way, when declaring the function, to turn x into a constant that points somewhere other than the global variable x? I can't provide arguments to the function.
This is a pretty common trick (you usually see it in lambda expressions that want to bind a particular value within a loop):
x = 3
def f(x=x):
print(x)
x = 5
f() # prints 3
The trick is that default parameter values are evaluated at the time of function definition, so in the expression x=x, the x on the right hand side is evaluated (producing the value 3) and then stored as the default value of the x parameter in the function (which shadows the x in the outer scope).
You could equivalently write:
x = 3
def f(n=x):
print(n)
x = 5
f() # prints 3
which has the same result, but doesn't shadow the x variable.
From what I understand, you seem to want x to hold two values simultaneously - which is what complex data structures are for. A list would work fine, or a dict:
>>> x = [3]
>>> def f():
... print(x[0]) # always refers to first element. Functionally constant.
...
>>> x.append(5)
>>> f()
3
>>>
However, it sounds like you really have an XY problem, where you're asking about your solution instead of your actual problem. Go back to your code and check if this seems to be the case. If so, we might be able to point you towards a better way of solving your real issue.
I am reading Hackers and Painters and am confused by a problem mentioned by the author to illustrate the power of different programming languages.
The problem is:
We want to write a function that generates accumulators—a function that takes a number n, and returns a function that takes another number i and returns n incremented by i. (That’s incremented by, not plus. An accumulator has to accumulate.)
The author mentions several solutions with different programming languages. For example, Common Lisp:
(defun foo (n)
(lambda (i) (incf n i)))
and JavaScript:
function foo(n) { return function (i) { return n += i } }
However, when it comes to Python, the following codes do not work:
def foo(n):
s = n
def bar(i):
s += i
return s
return bar
f = foo(0)
f(1) # UnboundLocalError: local variable 's' referenced before assignment
A simple modification will make it work:
def foo(n):
s = [n]
def bar(i):
s[0] += i
return s[0]
return bar
I am new to Python. Why doesn the first solution not work while the second one does? The author mentions lexical variables but I still don't get it.
s += i is just sugar for s = s + i.*
This means you assign a new value to the variable s (instead of mutating it in place). When you assign to a variable, Python assumes it is local to the function. However, before assigning it needs to evaluate s + i, but s is local and still unassigned -> Error.
In the second case s[0] += i you never assign to s directly, but only ever access an item from s. So Python can clearly see that it is not a local variable and goes looking for it in the outer scope.
Finally, a nicer alternative (in Python 3) is to explicitly tell it that s is not a local variable:
def foo(n):
s = n
def bar(i):
nonlocal s
s += i
return s
return bar
(There is actually no need for s - you could simply use n instead inside bar.)
*The situation is slightly more complex, but the important issue is that computation and assignment are performed in two separate steps.
An infinite generator is one implementation. You can call __next__ on a generator instance to extract successive results iteratively.
def incrementer(n, i):
while True:
n += i
yield n
g = incrementer(2, 5)
print(g.__next__()) # 7
print(g.__next__()) # 12
print(g.__next__()) # 17
If you need a flexible incrementer, one possibility is an object-oriented approach:
class Inc(object):
def __init__(self, n=0):
self.n = n
def incrementer(self, i):
self.n += i
return self.n
g = Inc(2)
g.incrementer(5) # 7
g.incrementer(3) # 10
g.incrementer(7) # 17
In Python if we use a variable and pass it to a function then it will be Call by Value whatever changes you make to the variable it will not be reflected to the original variable.
But when you use a list instead of a variable then the changes that you make to the list in the functions are reflected in the original List outside the function so this is called call by reference.
And this is the reason for the second option does work and the first option doesn't.
So today in computer science I asked about using a function as a variable. For example, I can create a function, such as returnMe(i) and make an array that will be used to call it. Like h = [help,returnMe] and then I can say h1 and it would call returnMe("Bob"). Sorry I was a little excited about this. My question is is there a way of calling like h.append(def function) and define a function that only exists in the array?
EDIT:
Here Is some code that I wrote with this!
So I just finished an awesome FizzBuzz with this solution thank you so much again! Here's that code as an example:
funct = []
s = ""
def newFunct(str, num):
return (lambda x: str if(x%num==0) else "")
funct.append(newFunct("Fizz",3))
funct.append(newFunct("Buzz",5))
for x in range(1,101):
for oper in funct:
s += oper(x)
s += ":"+str(x)+"\n"
print s
You can create anonymous functions using the lambda keyword.
def func(x,keyword='bar'):
return (x,keyword)
is roughly equivalent to:
func = lambda x,keyword='bar':(x,keyword)
So, if you want to create a list with functions in it:
my_list = [lambda x:x**2,lambda x:x**3]
print my_list[0](2) #4
print my_list[1](2) #8
Not really in Python. As mgilson shows, you can do this with trivial functions, but they can only contain expressions, not statements, so are very limited (you can't assign to a variable, for example).
This is of course supported in other languages: in Javascript, for example, creating substantial anonymous functions and passing them around is a very idiomatic thing to do.
You can create the functions in the original scope, assign them to the array and then delete them from their original scope. Thus, you can indeed call them from the array but not as a local variable. I am not sure if this meets your requirements.
#! /usr/bin/python3.2
def a (x): print (x * 2)
def b (x): print (x ** 2)
l = [a, b]
del a
del b
l [0] (3) #works
l [1] (3) #works
a (3) #fails epicly
You can create a list of lambda functions to increment by every number from 0 to 9 like so:
increment = [(lambda arg: (lambda x: arg + x))(i) for i in range(10)]
increment[0](1) #returns 1
increment[9](10) #returns 19
Side Note:
I think it's also important to note that this (function pointers not lambdas) is somewhat like how python holds methods in most classes, except instead of a list, it's a dictionary with function names pointing to the functions. In many but not all cases instance.func(args) is equivalent to instance.__dict__['func'](args) or type(class).__dict__['func'](args)
This question already has answers here:
What do lambda function closures capture?
(7 answers)
Closed 6 months ago.
While I was investigating a problem I had with lexical closures in Javascript code, I came along this problem in Python:
flist = []
for i in xrange(3):
def func(x): return x * i
flist.append(func)
for f in flist:
print f(2)
Note that this example mindfully avoids lambda. It prints "4 4 4", which is surprising. I'd expect "0 2 4".
This equivalent Perl code does it right:
my #flist = ();
foreach my $i (0 .. 2)
{
push(#flist, sub {$i * $_[0]});
}
foreach my $f (#flist)
{
print $f->(2), "\n";
}
"0 2 4" is printed.
Can you please explain the difference ?
Update:
The problem is not with i being global. This displays the same behavior:
flist = []
def outer():
for i in xrange(3):
def inner(x): return x * i
flist.append(inner)
outer()
#~ print i # commented because it causes an error
for f in flist:
print f(2)
As the commented line shows, i is unknown at that point. Still, it prints "4 4 4".
Python is actually behaving as defined. Three separate functions are created, but they each have the closure of the environment they're defined in - in this case, the global environment (or the outer function's environment if the loop is placed inside another function). This is exactly the problem, though - in this environment, i is modified, and the closures all refer to the same i.
Here is the best solution I can come up with - create a function creater and invoke that instead. This will force different environments for each of the functions created, with a different i in each one.
flist = []
for i in xrange(3):
def funcC(j):
def func(x): return x * j
return func
flist.append(funcC(i))
for f in flist:
print f(2)
This is what happens when you mix side effects and functional programming.
The functions defined in the loop keep accessing the same variable i while its value changes. At the end of the loop, all the functions point to the same variable, which is holding the last value in the loop: the effect is what reported in the example.
In order to evaluate i and use its value, a common pattern is to set it as a parameter default: parameter defaults are evaluated when the def statement is executed, and thus the value of the loop variable is frozen.
The following works as expected:
flist = []
for i in xrange(3):
def func(x, i=i): # the *value* of i is copied in func() environment
return x * i
flist.append(func)
for f in flist:
print f(2)
Here's how you do it using the functools library (which I'm not sure was available at the time the question was posed).
from functools import partial
flist = []
def func(i, x): return x * i
for i in range(3):
flist.append(partial(func, i))
for f in flist:
print(f(2))
Outputs 0 2 4, as expected.
look at this:
for f in flist:
print f.func_closure
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
It means they all point to the same i variable instance, which will have a value of 2 once the loop is over.
A readable solution:
for i in xrange(3):
def ffunc(i):
def func(x): return x * i
return func
flist.append(ffunc(i))
What is happening is that the variable i is captured, and the functions are returning the value it is bound to at the time it is called. In functional languages this kind of situation never arises, as i wouldn't be rebound. However with python, and also as you've seen with lisp, this is no longer true.
The difference with your scheme example is to do with the semantics of the do loop. Scheme is effectively creating a new i variable each time through the loop, rather than reusing an existing i binding as with the other languages. If you use a different variable created external to the loop and mutate it, you'll see the same behaviour in scheme. Try replacing your loop with:
(let ((ii 1)) (
(do ((i 1 (+ 1 i)))
((>= i 4))
(set! flist
(cons (lambda (x) (* ii x)) flist))
(set! ii i))
))
Take a look here for some further discussion of this.
[Edit] Possibly a better way to describe it is to think of the do loop as a macro which performs the following steps:
Define a lambda taking a single parameter (i), with a body defined by the body of the loop,
An immediate call of that lambda with appropriate values of i as its parameter.
ie. the equivalent to the below python:
flist = []
def loop_body(i): # extract body of the for loop to function
def func(x): return x*i
flist.append(func)
map(loop_body, xrange(3)) # for i in xrange(3): body
The i is no longer the one from the parent scope but a brand new variable in its own scope (ie. the parameter to the lambda) and so you get the behaviour you observe. Python doesn't have this implicit new scope, so the body of the for loop just shares the i variable.
The problem is that all of the local functions bind to the same environment and thus to the same i variable. The solution (workaround) is to create separate environments (stack frames) for each function (or lambda):
t = [ (lambda x: lambda y : x*y)(x) for x in range(5)]
>>> t[1](2)
2
>>> t[2](2)
4
I'm still not entirely convinced why in some languages this works one way, and in some another way. In Common Lisp it's like Python:
(defvar *flist* '())
(dotimes (i 3 t)
(setf *flist*
(cons (lambda (x) (* x i)) *flist*)))
(dolist (f *flist*)
(format t "~a~%" (funcall f 2)))
Prints "6 6 6" (note that here the list is from 1 to 3, and built in reverse").
While in Scheme it works like in Perl:
(define flist '())
(do ((i 1 (+ 1 i)))
((>= i 4))
(set! flist
(cons (lambda (x) (* i x)) flist)))
(map
(lambda (f)
(printf "~a~%" (f 2)))
flist)
Prints "6 4 2"
And as I've mentioned already, Javascript is in the Python/CL camp. It appears there is an implementation decision here, which different languages approach in distinct ways. I would love to understand what is the decision, exactly.
The variable i is a global, whose value is 2 at each time the function f is called.
I would be inclined to implement the behavior you're after as follows:
>>> class f:
... def __init__(self, multiplier): self.multiplier = multiplier
... def __call__(self, multiplicand): return self.multiplier*multiplicand
...
>>> flist = [f(i) for i in range(3)]
>>> [g(2) for g in flist]
[0, 2, 4]
Response to your update: It's not the globalness of i per se which is causing this behavior, it's the fact that it's a variable from an enclosing scope which has a fixed value over the times when f is called. In your second example, the value of i is taken from the scope of the kkk function, and nothing is changing that when you call the functions on flist.
The reasoning behind the behavior has already been explained, and multiple solutions have been posted, but I think this is the most pythonic (remember, everything in Python is an object!):
flist = []
for i in xrange(3):
def func(x): return x * func.i
func.i=i
flist.append(func)
for f in flist:
print f(2)
Claudiu's answer is pretty good, using a function generator, but piro's answer is a hack, to be honest, as it's making i into a "hidden" argument with a default value (it'll work fine, but it's not "pythonic").
I didn't like how solutions above created wrappers in the loop. Note: python 3.xx
flist = []
def func(i):
return lambda x: x * i
for i in range(3):
flist.append(func(i))
for f in flist:
print f(2)