Dictionary comprehension with lambda functions gives wrong results - python

I tried the following code in Python 3.5.1:
>>> f = {x: (lambda y: x) for x in range(10)}
>>> f[5](3)
9
It's obvious that this should return 5. I don't understand where the other value comes from, and I wasn't able to find anything.
It seems like it's something related to reference - it always returns the answer of f[9], which is the last function assigned.
What's the error here, and how should this be done so that it works properly?

Python scoping is lexical. A closure will refer to the name and scope of the variable, not the actual object/value of the variable.
What happens is that each lambda is capturing the variable x not the value of x.
At the end of the loop the variable x is bound to 9, therefore every lambda will refer to this x whose value is 9.
Why #ChrisP's answer works:
make_func forces the value of x to be evaluated (as it is passed
into a function). Thus, the lambda is made with value of x currently
and we avoid the above scoping issue.
def make_func(x):
return lambda y: x
f = {x: make_func(x) for x in range(10)}

The following should work:
def make_func(x):
return lambda y: x
f = {x: make_func(x) for x in range(10)}
The x in your code ends up referring to the last x value, which is 9, but in mine it refers to the x in the function scope.

Related

Can a function using a global variable use the variable's value from when the function was declared rather than when it was called?

Here is a very simplified example of what I am trying to do:
x = 3
def f():
print(x)
x = 5
f() #f prints 5 but I want it to print 3.
Is there a way, when declaring the function, to turn x into a constant that points somewhere other than the global variable x? I can't provide arguments to the function.
This is a pretty common trick (you usually see it in lambda expressions that want to bind a particular value within a loop):
x = 3
def f(x=x):
print(x)
x = 5
f() # prints 3
The trick is that default parameter values are evaluated at the time of function definition, so in the expression x=x, the x on the right hand side is evaluated (producing the value 3) and then stored as the default value of the x parameter in the function (which shadows the x in the outer scope).
You could equivalently write:
x = 3
def f(n=x):
print(n)
x = 5
f() # prints 3
which has the same result, but doesn't shadow the x variable.
From what I understand, you seem to want x to hold two values simultaneously - which is what complex data structures are for. A list would work fine, or a dict:
>>> x = [3]
>>> def f():
... print(x[0]) # always refers to first element. Functionally constant.
...
>>> x.append(5)
>>> f()
3
>>>
However, it sounds like you really have an XY problem, where you're asking about your solution instead of your actual problem. Go back to your code and check if this seems to be the case. If so, we might be able to point you towards a better way of solving your real issue.

Why does the UnboundLocalError occur on the second variable of the flat comprehension?

I answered a question here: comprehension list in python2 works fine but i get an error in python3
OP's error was using the same variables for max range and indices:
x = 12
y = 10
z = 12
n = 100
ret_list = [ (x,y,z) for x in range(x+1) for y in range(y+1) for z in range(z+1) if x+y+z!=n ]
This is a Python-3 error only, and related to the scopes that were added to the comprehension to avoid the variables defined here "leaking". Changing the variable names fixes that.
The error is:
UnboundLocalError: local variable 'y' referenced before assignment
because outer, global y is shadowed by the local scope.
My question is: why do I get the error on y and not on z or x ?
EDIT: If I remove the loop on x, the error moves to z:
>> ret_list = [ (x,y,z) for y in range(y+1) for z in range(z+1) if x+y+z!=n ]
UnboundLocalError: local variable 'z' referenced before assignment
If I just do one loop:
ret_list = [ (x,y,z) for y in range(y+1) if x+y+z!=n ]
it works. So I'm suspecting that the first range function is evaluated before all the other expressions, which leaves the value of x intact. But the exact reason is still to be found. Using Python 3.4.3.
This behaviour is (implicitly) described in the reference documentation (emphasis mine).
However, aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don’t “leak” into the enclosing scope.
The iterable expression in the leftmost for clause is evaluated directly in the enclosing scope and then passed as an argument to the implictly [sic] nested scope. Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable. For example: [x*y for x in range(10) for y in range(x, x+10)].
This means that:
list_ = [(x, y) for x in range(x) for y in range(y)]
equivalent to:
def f(iter_):
for x in iter_:
for y in range(y):
yield x, y
list_ = list(f(iter(range(x))))
As the name x in for the leftmost iterable is read in the enclosing scope as opposed to the nested scope then there is no name conflict between these two uses of x. The same is not true for y, which is why it is where the UnboundLocalError occurs.
As to why this happens: a list comprehension is more-or-less syntactic sugar for list(<generator expression>), so it's going to be using the same code path as a generator expression (or at least behave in the same way). Generator expressions evaluate the iterable expression in the leftmost for clause to make error handling when the generator expression somewhat saner. Consider the following code:
y = None # line 1
gen = (x + 1 for x in range(y + 1)) # line 2
item = next(gen) # line 3
y is clearly the wrong type and so the addition will raise a TypeError. By evaluating range(y + 1) immediately that type error is raised on line 2 rather than line 3. Thus, it is easier to diagnose where and why the problem occurred. Had it occurred on line 3 then you might mistakenly assume that it was the x + 1 statement that caused the error.
There is a bug report here that mentions this behaviour. It was resolved as "not a bug" for reason that it is desirable that list comprehensions and generator expressions have the same behaviour.

list of lambdas in python [duplicate]

This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
I have seen this question, but i still cannot get why such simple example does not work:
mylist = ["alice", "bob", "greta"]
funcdict = dict(((y, lambda x: x==str(y)) for y in mylist))
funcdict['alice']("greta")
#True
funcdict['alice']("alice")
#False
funcdict['greta']("greta")
#True
How is it different from:
[(y, y) for y in mylist]
Why y is not evalueated within each step of iteration?
The y in the body of the lambda expression is just a name, unrelated to the y you use to iterate over mylist. As a free variable, a value of y is not found until you actually call the function, at which time it uses whatever value for y is in the calling scope.
To actually force y to have a value at definition time, you need to make it local to the body via an argument:
dict((y, lambda x, y=z: x == str(y)) for z in mylist)
((y, lambda x: x==str(y)) for y in mylist)
y inside the lambda is not bound at the time of the genration expression defined, but it's bound when it's called; When it's called, iteration is already done, so y references the last item greta.
One way to work around this is to use keyword argument, which is evaluated when the function/lambda is defined:
funcdict = dict((y, lambda x, y=y: x == y) for y in mylist)
funcdict = {y: lambda x, y=y: x == y for y in mylist} # dict-comprehension
or you can use partial:
funcdict = {y: partial(operator.eq, y) for y in mylist}
y is evaluated while the mylist is iterated.

if and else in python lambda expression

Please help me to understand how this works. Output is 4
a=4
b=7
x=lambda: a if 1 else b
lambda x: 'big' if x > 100 else 'small'
print(x())
First, let's remove this line as it doesn't do anything:
lambda x: 'big' if x > 100 else 'small'
This lambda expression is defined but never called. The fact that it's argument is also called x has nothing to do with the rest of the code.
Let's look at what remains:
a = 4
b = 7
x = lambda: a if 1 else b
print(x())
Here x becomes a function as it contains code. The lambda form can only contain expressions, not statements, so it has to use the expression form of if which is backward looking:
true-result if condition else false-result
In this case the condition is 1, which is always true, so the result of the function x() is always the value of a, assigned to 4 earlier in the code. Effectively, x() acts like:
def x():
return a
Understanding the differences between expressions and statements is key to understanding code like this.
Your x is always equals to 4, as it takes no arguments and if 1 is always True.
Then you have lambda expression that's not assigned to any variable, neither used elsewhere.
Eventualy, you print out x, which is always 4 as I said above.
P.S. I strongly suggest you to read Using lambda Functions from Dive into Python
Let me translate that for you.
You assign to x a lambda function with no arguments. Because 1 always evaluates as true, you always return the externally defined variable a, which evaluates as 4.
Then, you create a lambda function with one argument x, which you don't assign to a variable/access name, so it is lost forever.
Then, you call function x, which always returns a. Output is 4.

Python closures and cells (closed-over values)

What is the Python mechanism that makes it so that
[lambda: x for x in range(5)][2]()
is 4?
What is the usual trick for binding a copy of x to each lamba expression so that the above expression will equal 2?
My final solution:
for template, model in zip(model_templates, model_classes):
def create_known_parameters(known_parms):
return lambda self: [getattr(self, p.name)
for p in known_parms]
model.known_parameters = create_known_parameters(template.known_parms)
>>> [lambda x=x: x for x in range(5)][2]()
2
I usually use functools.partial:
[ partial(lambda x: x, x) for x in range(5) ]
Or, of course, you can do it yourself:
[ (lambda x: (lambda: x))(x) for x in range(5) ]
Since no one's answered the "what is the mechanism" part, and this surprised me when I first read it, here's a go:
This:
ls = [lambda: x for x in range(5)]
Is a bit like this:
ls = []
x = 0
ls.append(lambda: x)
x = 1
ls.append(lambda: x)
x = 2
ls.append(lambda: x)
x = 3
ls.append(lambda: x)
x = 4
ls.append(lambda: x)
Each of those lambdas has its own scope, but none of those scopes contain an x. So they're all going to be reading the value of x by looking in an outer scope, so of course they must all be referring to the same object. By the time any of them are called, the loop is done and that object is the integer 4.
So even though these lambdas look like functions involving only immutable values, they can still be affected by side effects, because they depend on the bindings in an outer scope, and that scope can change.
You could of course further change what these lambdas return by rebinding x, or make them throw an error by unbinding it. In the list comprehension version though, the x is only bound in a private scope inside the list comprehension, so you can't mess with it (as easily).
The solution is of course to arrange things so that each lambda has an x in its local scope (or at least some outer scope that is not shared between the lambdas), so that they can all refer to different objects. Ways to do that have been shown in the other answers.

Categories