This question already has answers here:
Creating lambda inside a loop [duplicate]
(3 answers)
Closed 6 years ago.
I'm encountering some strange behavior with lambda functions in a loop in python. When I try to assign lambda functions to dictionary entries in a list, and when other entries in the dictionary are used in the function, only the last time through the loop is the lambda operator evaluated. So all of the functions end up having the same value!
Below is stripped-down code that captures just the parts of what I'm trying that is behaving oddly. My actual code is more complex, not as trivial as this, so I'm looking for an explanation and, preferably, a workaround.
n=4
numbers=range(n)
entries = [dict() for x in numbers]
for number, entry in zip(numbers,entries):
n = number
entry["number"] = n
entry["number2"] = lambda x: n*1
for number in numbers:
print(entries[number]["number"], entries[number]["number2"](2))
The output is:
0 3
1 3
2 3
3 3
In other words, the dictionary entires that are just integers are fine, and were filled properly by the loop. But the lambda functions — which are trivial and should just return the same value as the "number" entries — are all set to the last pass through.
What's going on?
Try this
N=4
numbers=range(N)
entries = [dict() for x in numbers]
for number, entry in zip(numbers,entries):
entry["number"] = number
entry["number2"] = lambda x,n=number: n*1
for number in numbers:
print(entries[number]["number"], entries[number]["number2"](2))
It prints (python3)
0 0
1 1
2 2
3 3
To avoid confusion, n referred to different things in your code. I used it only at one place.
It is a closure problem.
By the end of your for loop, the n variable - which, unlike in static languages such as C#, is set to 3, which is then being accessed in the lambda expression. The variable value is not fixed; as another answer on the site points out, lambda expressions are fluid and will retain references to the variables involved instead of capturing the values at the time of creation. This question also discusses your issue.
To fix it, you need to give the lambdas new, local variable via default parameters:
entry["number2"] = lambda x, n=n: n*1
This creates a new variable in the lambda's scope, called n, which sets its default value to the "outside" value of n. Note that this is the solution endorsed by the official FAQ, as this answer by Adrien Plisson states.
Now, you can call your lambda like normal and ignore the optional parameter, with no ill effect.
EDIT: As originally stated by Sci Prog, this solution makes n = number redundant. Your final code will look similar to this:
lim = 4
numbers = range(lim)
entries = [dict() for x in numbers]
for number, entry in zip(numbers, entries):
entry["number"] = number
entry["number2"] = lambda x, n = number: n*1
for number in numbers:
print(entries[number]["number"], entries[number]["number2"](2))
You are probably reaching the problem that the method is created as referencing a variable n. The function is only evaluated after the loop so you are going to call the function which references n. If you're ok with having the function evaluated at the time of assignment you could put a function call around it:
(lambda x: n*1)(2)
or if you want to have the functions to use, have them reference the specific value you want. From your code you could use a default argument as a workaround:
entry["number"] = n
entry["number2"] = lambda x, n=n: n*1
The difference comes down to a question of memory addressing. I imagine it went something like this:
You: Python, please give me a variable called "n"
Python: Ok! Here it is, it is at memory slot 1
You: Cool! I will now create functions which say take that variable "n"
value (at memory slot 1) and multiply it by 1 and return that to me.
Python: Ok! Got it:
1. Take the value at memory slot 1.
2. Multiply by 1.
3. Return it to you.
You: Done with my looping, now evaluate those instructions!
Python: Ok! Now I will take the value of at memory slot 1 and multiply by 1
and give that to you.
You: Hey, I wanted each function to reference different values!
Python: I followed your instructions exactly!
Related
This question already has answers here:
Use value of variable in lambda expression [duplicate]
(3 answers)
Closed 7 years ago.
I've experienced some strange behaviour when storing lambda functions into a dictionary: If you try to pass some default value to a function in a loop, only the last default value is being used.
Here some minimal example:
#!/usr/bin/env python
# coding: utf-8
def myfct(one_value, another_value):
"do something with two int values"
return one_value + another_value
fct_dict = {'add_{}'.format(number): (lambda x: myfct(x, number))
for number in range(10)}
print('add_3(1): {}, id={}'.format(fct_dict['add_3'](1), id(fct_dict['add_3'])))
print('add_5(1): {}, id={}'.format(fct_dict['add_5'](1), id(fct_dict['add_5'])))
print('add_9(1): {}, id={}'.format(fct_dict['add_9'](1), id(fct_dict['add_9'])))
The output reads as follows
add_3(1): 10, id=140421083875280
add_5(1): 10, id=140421083875520
add_9(1): 10, id=140421083876000
You get dissimilar functions (id not identical) but every function uses the same second argument.
Can somebody explain what's going on?
The same holds with python2, python3, pypy...
The fix:
def make_closure(number):
return lambda x: myfct(x, number)
used as
{'add_{}'.format(number): make_closure(number) for number in range(10)}
The reason for this behaviour is, that the variable number (think: named memory location here) is the same during all iterations of the loop (though its actual value changes in each iteration). "Loop" here refers to the dictionary comprehension, which internally is based on a loop. All lambda instances created in the loop will close over the same "location", which retains the value last assigned to it (in the last iteration of the loop).
The following code is not what actually happens underneath. It is merely provided to shed light on the concepts:
# Think of a closure variable (like number) as being an instance
# of the following class
class Cell:
def __init__(self, init=None):
self.value = None
# Pretend, the compiler "desugars" the dictionary comprehension into
# something like this:
hidden_result_dict = {}
hidden_cell_number = Cell()
for number in range(10):
hidden_cell_number.value = number
hidden_result_dictionary['add_{}'.format(number)] = create_lambda_closure(hidden_cell_number)
All lambda closures created by the create_lambda_closure operation share the very same Cell instance and will grab the value attribute at run-time (i.e., when the closure is actually called). By that time, value will refer to last value ever assigned to it.
The value of hidden_result_dict is then answered as the result of the dict comprehension. (Again: this is only meant as be read on a "conceptual" level; it has no relation to the actual code executed by the Python VM).
number is a variable which has different value for each iteration of the dict comprehension. But when you do lambda x: myfct(x, number), it does not use value of number. It just creates a lambda method that will use value of number when it will be called/used. So when you use you add_{} methods, number has value 9 which is used in every call to myfct(x, number).
This question already has answers here:
Use value of variable in lambda expression [duplicate]
(3 answers)
Closed 7 years ago.
I've experienced some strange behaviour when storing lambda functions into a dictionary: If you try to pass some default value to a function in a loop, only the last default value is being used.
Here some minimal example:
#!/usr/bin/env python
# coding: utf-8
def myfct(one_value, another_value):
"do something with two int values"
return one_value + another_value
fct_dict = {'add_{}'.format(number): (lambda x: myfct(x, number))
for number in range(10)}
print('add_3(1): {}, id={}'.format(fct_dict['add_3'](1), id(fct_dict['add_3'])))
print('add_5(1): {}, id={}'.format(fct_dict['add_5'](1), id(fct_dict['add_5'])))
print('add_9(1): {}, id={}'.format(fct_dict['add_9'](1), id(fct_dict['add_9'])))
The output reads as follows
add_3(1): 10, id=140421083875280
add_5(1): 10, id=140421083875520
add_9(1): 10, id=140421083876000
You get dissimilar functions (id not identical) but every function uses the same second argument.
Can somebody explain what's going on?
The same holds with python2, python3, pypy...
The fix:
def make_closure(number):
return lambda x: myfct(x, number)
used as
{'add_{}'.format(number): make_closure(number) for number in range(10)}
The reason for this behaviour is, that the variable number (think: named memory location here) is the same during all iterations of the loop (though its actual value changes in each iteration). "Loop" here refers to the dictionary comprehension, which internally is based on a loop. All lambda instances created in the loop will close over the same "location", which retains the value last assigned to it (in the last iteration of the loop).
The following code is not what actually happens underneath. It is merely provided to shed light on the concepts:
# Think of a closure variable (like number) as being an instance
# of the following class
class Cell:
def __init__(self, init=None):
self.value = None
# Pretend, the compiler "desugars" the dictionary comprehension into
# something like this:
hidden_result_dict = {}
hidden_cell_number = Cell()
for number in range(10):
hidden_cell_number.value = number
hidden_result_dictionary['add_{}'.format(number)] = create_lambda_closure(hidden_cell_number)
All lambda closures created by the create_lambda_closure operation share the very same Cell instance and will grab the value attribute at run-time (i.e., when the closure is actually called). By that time, value will refer to last value ever assigned to it.
The value of hidden_result_dict is then answered as the result of the dict comprehension. (Again: this is only meant as be read on a "conceptual" level; it has no relation to the actual code executed by the Python VM).
number is a variable which has different value for each iteration of the dict comprehension. But when you do lambda x: myfct(x, number), it does not use value of number. It just creates a lambda method that will use value of number when it will be called/used. So when you use you add_{} methods, number has value 9 which is used in every call to myfct(x, number).
What do the following expression actually does?
list = [lambda n=n: lambda x: x+n for n in range(10)]
More specifically:
What does n=n mean?
What will be the content of 'list'?
What will be the output of
print(list[0](14)) and print(list[0]()(14))
and why?
What does n=n mean?
lambda lets you define functions that take parameters, just like def. And those parameters can have default argument values. So, lambda n=n: is the same as def foo(n=n):.
In fact, when faced with an expression that's too complicated for you to read, it's often worth unpacking into simple statements:
list = []
for n in range(10):
def spam(n=n):
def eggs(x):
return x+n
return eggs
list.append(spam)
Now, why would you want to create a parameter named n with default value n? Why not just lambda:? The official FAQ explains this, but let's try to summarize.
If you just write this:
funcs = [lambda: n for n in range(10)]
… what you get is 10 functions of no parameters, that are all closures over the same variable, n. Because n has the value 9 at the end of the loop, when called, they're all going to return 9.
But if you do this:
funcs = [lambda n=n: n for n in range(10)]
… what you get is 10 functions of one optional parameter n (which hides the closure n from view), whose default value is the value of n at the time each function was defined. So, when called with no arguments, the first one will return 0, the second 1, and so on.
In your case, of course, the functions aren't just returning n, they're returning a function that takes a parameter, adds n to it, and returns the result. But the idea is the same; you want them to return different functions, which add 0, 1, … 9 to their arguments, not all return equal functions that all add 9.
What will be the content of list?
list will be 10 functions of one optional parameter whose default values range from 0 to 9, each of which returns a function of one parameter. That returned function is a closure over the value of n from the outer function. So, when it's called, it returns its argument, x, plus the n variable that ranges from 0 through 9.
What will be the output of
print(list[0](14))
Here, you're calling the first outer function, list[0], with the argument 14. So, instead of its default value 0 for n, it's going to have 14. So, what you'll get is a function that takes one argument and adds 14 to it. But it will print out as something like:
<function <listcomp>.<lambda>.<locals>.<lambda> at 0x105f21f28>
That long mess is Python 3.4+ trying to be helpful by telling you where to find the function definition. Usually, when a function is nested this deeply, most of the steps along the way have names. In this case, you've got three layers of anonymous functions, so none of the names are very useful…
In order to see it do anything useful, you'll have to call it:
print(list[0](14)(20))
And this will give you 34.
You could also use the inspect module, or just dir, to poke around inside the function. For example, print(list[0](14).__code__.co_freevars[0], list[0](14).__closure__[0].cell_contents) will tell you that it's stashed the number 14 under the name n for use by its internal function.
…
print(list[0]()(14))
Here, you're again calling list[0], but this time with no argument, so its n gets the default value of 0. So, it returns a function that adds 0 to its argument. You then call that function with 14, so you get 14.
To answer the last part first:
In [1]: list = [lambda n=n: lambda x: x+n for n in range(10)]
In [2]: print(list[0](14))
<function <lambda> at 0x7f47b5ca7cf8>
In [3]: print(list[0]()(14))
14
Obtained by running the code. list bad name by the way as list is a python builtin gives you 10 lambda functions that don't do much - the first will return the original argument x, the second the argument + 1, ect. as n is stored as the index of the lambda by n=n local to that lambda.
I've read about python scopes and browsed questions here on stackoverflow but theres something i'd like to clarify.
I'm extracting a piece of code to a function, from what i used to it should take all the variables it's using as parameters. But, in python the variable address is determined at runtime so theres actually no need for the parameters. Since i'm new to python i wanted to know if there are other implications or conventions i should know about.
x = 5
x += 1
print x
is there any difference between the following refactoring of the above code :
def f(x):
x += 1
return x
x = 5
x = f(x)
print x
and:
def f():
x++
x = 5
f()
print x
If not then, is one of the ways more commonly used or preferred in python ?
It's preferred not to use global variables, if not absolutely necessary. That said, in the second example you'd need global x declared before you refer to x.
So, first way:
f takes an argument x
Increments x
returns x + 1
the global x is not affected
The second way:
def f():
global x
x += 1
x = 1
f()
f has no arguments
Increments the global x
P.S. Python has no ++ operator. x += 1 is used instead
I upvote the question for two reasons:
1)
importing the notation x++ from other languages (C++ and Java) is a venial sin; who has never been absent-minded ?
not testing codes deserves a downvote, that's right, because it denotes that no tests have been performed to try to obtain oneself a certain number of observations and that's baaad
however I find the desire to understand notions concerning scopes, namespaces, global/local more commendable than the faults to be reproached
2)
It is certainly an unsatisfactory situation to be downvoted because of an approximate question while receiving an upvoted answer that contains itself some inadequate terms according to me. Terminology is particularly important in subjects in which controversial debates and confusionning descriptions may happen.
For exemple, for
def f(x):
x++
return x
I wouldn't say that f receives an argument x, but that f receives x as an argument.
x is not incremented; rather: another object is created with a value resulting of the incrementation of the value of initial x
f doesn't returns x+1 , it returns the new object with the incremented value
EDIT 1
# f takes an argument x
In the call f(x) , I wouldn't say that f receives an argument x, because x isn't an argument in the absolute. x "becomes" an argument only relatively to a function, at the moment when it is passed to the function. So I rather say that in the call f(x) , the function f receives x AS an argument.
It may happen that one says "the argument Y" one time, as an understatement to express "Y , considered at a moment when it is passed as an argument to a function". This understatement is shorter; but what I think is that in a precise explanation such an easy way to express must be banned.
#Increments x
Refering to the code written in the question (case 1), this sentence is ambiguous because of the function written with a parameter x: does x refer to the x in the function or the x outside ? You'll say I nitpick, but for a newbie that doesn't know the data model and the working of functions well, the question is valid. What does f(x) mean in Python ? , is the x passed by value or passed by reference ? It could be that the operations inside f could change the x outside, why not, that's precisely what is in discussion, no ? That's why it is preferable to name the parameter of f with a different name than any of the names of the outside objects; I think the same as for the previous sentence: in a precise explanation, only cautious sentences and codes should be employed, to avoid confusionning ambiguities:
def f(N):
N += 1
return N
x = 5
x = f(x)
print x
And it should be explained that the instruction N += 1 triggers the creation of a new object with value incremented and that the local identifier N is rebound to this new object. And at this point, the global identifier x is still bound to the same object as before the call.
# returns x + 1
f doesn't returns x+1 , it returns the new object newly assigned to N with the incremented value
# the global x is not affected
What does it mean ? Before , global x has a value 5. After, global x has a value 6. During the execution of the operations inside f, the global x isn't affected, that's right: it is only when the new object (in fact its address..) is returned to the outside of f, and that the assignement x = f(x) is executed that global x is affected. But this sentence "the global x is not affected" is placed after "returns x + 1
". No really, I understand nothing in such explanations.
End of EDIT 1
.
.
Concerning the second case
def f():
global x
x += 1
I prefer to say that f has no argument than it has no parameter
It doesn't increments the global x, it provokes a new assignement of the global identifier x to a new object with incremented value.
EDIT 2
In this case, it isn't possible to use another name than x inside the function, otherwise the exemple would mean another thing.
def f():
global x
x += 1
x = 5
f()
print x
# f has no arguments
Why the hell did I write "I prefer to say that f has no argument than it has no parameter" to comment that ? I still wonder.
At first, I had written "I prefer to say that f has no parameter than it has no argument" , thinking to the definition of f . I don't know why, I "corrected" by reversing the sentence and the result expresses nothing of what I think. I am completely puzzled.
Precisely, I think that the correct manners to express are :
in the definition of the function, f -> HAS no PARAMETER,
in the call f() , f -> RECEIVES no ARGUMENTS.
# Increments the global x
It doesn't increments the global x, it provokes a new assignement of the global identifier x to a new object with incremented value.
End of EDIT 2
.
It may seem minor nuances , and I am sure that it is a way to express things in a condensed manner.
But for me it is very important because, when a newbie, I had a lot of difficulty because of this kind of imprecision to understand all these questions linked to the peculiar data model of Python that are badly explained in the official docs of Python and that are debated in awkard discussions, such as the one concerning "pass by value or pass by reference" for exemple (that has no sense in Python), in which the terms are floating from right to left and left to right like on a sea of vagueness.
I'm learning python using the tutorial on the official python website and came across this example:
>>> def make_incrementor(n):
... return lambda x: x + n
...
>>> f = make_incrementor(42)
>>> f(0)
42
>>> f(1)
43
Where does x get it's value from? I'm not familiar with how lambda works, I understand anonymous functions just fine from javascript but this has me stumped. Anyone care to shed some light? I'd be grateful.
Consider this. f is the object created by the make_incrementor function.
It is a lambda, an "anonymous function".
>>> f= lambda x: x+42
>>> f(10)
52
The value for x showed up when we applied f to a value.
First off, here's a translation into JavaScript:
function makeIncrementor(n) {
return function (x) {return x + n}
}
A lambda expression consists of three parts.
The word lambda (In JavaScript, the word function)
A list of parameters (In JavaScript, the part in (...))
Some expression to evaluate (In JavaScript, the part in {...}, automatically returned)
The list of parameters is the part before the colon. These parameters are visible within the scope of the expression. Whatever the expression evaluates to is returned as the result of the function.
Unlike in JavaScript, a lambda can only contain a single expression; you can't make an anonymous multi-line function with lambda.
Disclaimer: I have pretty much no Python background. This is going off my Scheme/lambda calculus knowledge.
make_incrementor defines a function to make functions. make_incrementor(42) returns a function with x bounded to the lambda, and n with a value of 42. When you call the function f, the argument of f replaces the bound variable.
lambda brings some lambda calculus to Python. In essence, this is what's happening: normally in lambda calculus a statement would look something like this:
sum = add 3 4
This would pass 3 and 4 to the add function and store the result in sum. However, we could also write something along the lines of add 3. Now, since add expects two arguments, we now get an object (or function, if you will) expecting one argument. That function will then call add with 3 as its first argument and whatever we pass it as the second argument. We can now do this:
func = add 3
sum = func 4
This will be equivalent to the previous example. However, you can now use func whenever you want to add 3 to something. (Granted, this doesn't seem useful in this example, but in more complex situations it is).
All this is closely related to currying, something very central in most functional languages. If you're interested in lambda calculus and its similarity to regular mathematics, I highly recommend that you take a look at Haskell.
When make_incrementor() is called it creates and returns a lambda function at that time. In the process the value of the argument n gets stored or remembered in the function object created. If you called it again with a different n a different function would be returned.
x gets its value from the parameter used to call f.
n gets its value from the parameter used to call make_incrementor.
make_incrementor is a function that returns a function:
def make_incrementor(n):
return lambda x: x + n
When we call make_incrementor, we bind the parameter 42 to n, and return the concrete function:
f = make_incrementor(42) # fill in n at this point and return a function
This function we will return will therefore look like:
lambda(x): x + 42
f is bound to the returned function, so f will conceptually look like:
def f(x):
return x + 42
When we call f, we provide the value for x
f(1) # x == 1
f(2) # x == 2
Here's how I understand lambda:
When we write x2, we often confuse two distinct ideas. Consider:
x2 has an odd number of factors when x is an integer.
x2 is larger than x when x>1.
x2 has derivative 2x.
x2 has an inverse, namely sqrt(x), for x>0.
The first two statements are about the square of a particular but unspecified number. In those statements "x" represents an arbitrary single thing, and x2 represents a single related thing.
The third and fourth statement are about x2, the function. But this is not clearly communicated in 3. and 4. except by context and shared understanding of Calculus. We need a notational device to distinguish between x2 the single (but arbitrary) value, and x2 the function. The device also needs to communicate that it is a function of x. Therefore lambda is invented:
"lambda x.x2" is written to make this distinction and communicate "the function of x whose value at x is x2". The lambda operator takes a name (x) and an expression (x2) and returns a function. This has the same consequences as normal function definition, except that the function doesn't automatically receive a name.