We Keep Coding

Python lambda list: different behavior of list comprehension, for, and for in a function

Look at this code. I am creating 3 lists of lambda functions (stored in the variables plus_n, plus_n_, and plus_n__). They suppose to be exactly the same. However, only plus_n_ shows the expected behavior.
MAX=5
plus_n=[lambda x: x+i for i in range(MAX)]
plus_n_=[]
for i in range(MAX):
plus_n_.append(lambda x: x+i)
def all_plus_n():
plus_ns=[]
for i in range(MAX):
plus_ns.append(lambda x: x+i)
return plus_ns
plus_n__=all_plus_n()
for i in range(len(plus_n)):
print('plus_n[{}]({})={}'.format(i,3,plus_n[i](3)))
print('plus_n_[{}]({})={}'.format(i,3,plus_n_[i](3)))
print('plus_n__[{}]({})={}'.format(i,3,plus_n__[i](3)))
print()
The output:
plus_n[0](3)=7
plus_n_[0](3)=3
plus_n__[0](3)=7
plus_n[1](3)=7
plus_n_[1](3)=4
plus_n__[1](3)=7
plus_n[2](3)=7
plus_n_[2](3)=5
plus_n__[2](3)=7
plus_n[3](3)=7
plus_n_[3](3)=6
plus_n__[3](3)=7
plus_n[4](3)=7
plus_n_[4](3)=7
plus_n__[4](3)=7
See, the exact same code gives different results if it is on a function or in a comprehensive list...
So, what is the difference between the 3 approaches? What is happening?
If I want to use this variable in multiple functions, do I have to use it as a global variable? Because seems that I cant use a function to get the variable values...
Tks in advance.

This is a somewhat interesting variation on the usual question. Normally, the plus_n_ version wouldn't work either, but you happen to have reused i as the iteration variable for your testing at the end of the code. Since plus_n_ captures the global i, and the test loop also sets the global i, the lambda retrieved from plus_n_ uses the correct value each time through the loop - even though it's late binding on i. The list comprehension has its own scoped i which has a value of 4 after evaluation (and doesn't change after that); similarly for the loop in the function.
The clean, explicit, simple way to bind function parameters in Python is functools.partial from the standard library:
from functools import partial
MAX = 5
# Or we could use `operator.add`
def add(i, x):
return i + x
# Works as expected, whatever happens to `i` later in any scope
plus_n = [partial(add, i) for i in range(MAX)]
This way, each call to partial produces a callable object that binds its own value for i, rather than late-binding to the name i:
for j in plus_n:
i = {"this isn't even a valid operand": 'lol'} # utterly irrelevant
print(plus_n[j](3))
Do notice, however, that the parameters to be bound need to be at the beginning for this approach.
Another way to solve the specific example problem is to rely on bound method calls:
plus_n = [i.__add__ for i in range(MAX)]
You can also hand-roll your own currying, but why reinvent the wheel?
Finally, it is possible to use default parameters to lambdas to bind parameters - although I greatly dislike this method, since it is abusing the behaviour that causes another common problem and implying the existence of a parameter that could be overridden but isn't designed for it:
plus_n = [lambda x, i=i: x+i for i in range(MAX)]

How can I access lists that are generated within a function?

So I'm a bit stuck with some python homework... this is my first question btw sorry if it's formatted wrong.
There is a function "co-ordinates_generator()" that produces a list of co-ordinates, [[x, y], [x, y], [x, y]] etc.
I must create a function that draws upon these co-ordinates and prints pins onto a map accordingly, using those co-ordinates.
I thought I could achieve this by doing:
def place_coords():
for point in co-ordinates_generator():
if point[0] == '1':
place_pin() etc. etc....
This DOES work, however when I do this, the lists produced by "co_ordinates_generator()" are generated TWICE. So in other words, two lists are printed to my screen instead of just a single one that I am supposed to use.
My only assumption is that because in the part:
for point in co-ordiantes_generator():
I call upon co-ordinates_generator() and doing so triggers the function, causing it print and then also print again when I call place_coords(). Is this correct
If so, or otherwise, how would I go about fixing this? I've tried just deleting the "for _ in ____" part entirely but then that creates all sorts of troubles like "'pin' is not defined". And also the fact that I've written a whole heap of for-each loops based around using the "for _ in ___" part.
Sorry it's a long one! And thank you in advance.

In absence of your function I guess you are not returning any value from the function.
def co-ordinates_generator():
.... # your existing code
.... # your existing code
.... # your existing code
return co-ordiantes_variable # whatever name you have defined for the desired result.
result = co-ordinates_generator()
for point in result:
.... # your existing code
.... # your existing code
.... # your existing code

Retaining an objects attribute in "anonymous" function definition after object is potentially deleted [duplicate]

This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 8 years ago.
I'm writing some scientific code in Python which solves a complicated optimization problem and (ideally) returns a list of univariate functions which are a series of solution to the problem at particular points in time (a list of time-indexed, univariate policy functions, if you will).
The list of functions needs to be independent from the object which creates it. This is where I hit trouble -- inevitably, each function in the list needs to reference a value which was solved-for and stored in a list in the solver object.
Here's a quick little program that illustrates the problem I'm running into:
class test:
def __init__(self):
self.vals = np.arange(5)
def testfxns(self):
fxns = {}
for i in range(len(self.vals)):
def temp_func(x):
Z = self.vals[i]
return Z + x
fxns[i] = temp_func
#del temp_func # doesn't help
return fxns
t1 = test()
z = t1.testfxns()
print z[0](2)
print z[1](2)
print z[2](2)
print z[3](2)
print z[4](2)
The output of the print statements is all 6's -- I'd like it to be 2, 3, 4, 5, 6.
Is there any way to address this? I need those constants in the "Z" variable to be inserted into each iteration of the list of functions. The "poor man's" solution would be to print out each constant value and then just manually type out the list of functions, manually typing the constant values I need into different individual function definitions.
Is there any way to obtain the value of "self.vals[i]" in a "dereferenced" way, programatically? (Hopefully that makes sense.) Or alternatively, what's the terminology related to even solving this issue -- what terms should I be googling and/or researching in the documentation? (Some of the trouble is not even knowing where to be looking.)
Thanks so much. I know there may be multiple issues I'm running into here, so all help/guidance is appreciated.

This is a common problem with closures. The value of i is picked up when the function is called, not when it is created. The solution is to provide it as a keyword:
def temp_func(x, i=i):
Z = self.vals[i]
return Z + x
The keyword gets evaluated (and stored on the function object) when the function is created as opposed to when it is called.

Modifying variables in Python function is affecting variables with different names outside the function

I have a nested dictionary containing a bunch of data on a number of different objects (where I mean object in the non-programming sense of the word). The format of the dictionary is allData[i][someDataType], where i is a number designation of the object that I have data on, and someDataType is a specific data array associated with the object in question.
Now, I have a function that I have defined that requires a particular data array for a calculation to be performed for each object. The data array is called cleanFDF. So I feed this to my function, along with a bunch of other things it requires to work. I call it like this:
rm.analyze4complexity(allData[i]['cleanFDF'], other data, other data, other data)
Inside the function itself, I straight away re-assign the cleanFDF data to another variable name, namely clFDF. I.e. The end result is:
clFDF = allData[i]['cleanFDF']
I then have to zero out all of the data that lies below a certain threshold, as such:
clFDF[ clFDF < threshold ] = 0
OK - the function works as it is supposed to. But now when I try to plot the original cleanFDF data back in the main script, the entries that got zeroed out in clFDF are also zeroed out in allData[i]['cleanFDF']. WTF? Obviously something is happening here that I do not understand.
To make matters even weirder (from my point of view), I've tried to do a bodgy kludge to get around this by 'saving' the array to another variable before calling the function. I.e. I do
saveFDF = allData[i]['cleanFDF']
then run the function, then update the cleanFDF entry with the 'saved' data:
allData[i].update( {'cleanFDF':saveFDF} )
but somehow, simply by performing clFDF[ clFDF < threshold ] = 0 within the function modifies clFDF, saveFDF and allData[i]['cleanFDF'] in the main friggin' script, zeroing out all the entires at the same array indexes! It is like they are all associated global variables somehow, but I've made no such declarations anywhere...
I am a hopeless Python newbie, so no doubt I'm not understanding something about how it works. Any help would be greatly appreciated!

You are passing the value at allData[i]['cleanFDF'] by reference (decent explanation at https://stackoverflow.com/a/430958/337678). Any changes made to it will be made to the object it refers to, which is still the same object as the original, just assigned to a different variable.
Making a deep copy of the data will likely fix your issue (Python has a deepcopy library that should do the trick ;)).

Everything is a reference in Python.
def function(y):
y.append('yes')
return y
example = list()
function(example)
print(example)
it would return ['yes'] even though i am not directly changing the variable 'example'.
See Why does list.append evaluate to false?, Python append() vs. + operator on lists, why do these give different results?, Python lists append return value.

How are closures implemented?

"Learning Python, 4th Ed." mentions that:
the enclosing scope variable is looked up when the nested functions
are later called..
However, I thought that when a function exits, all of its local references disappear.
def makeActions():
acts = []
for i in range(5): # Tries to remember each i
acts.append(lambda x: i ** x) # All remember same last i!
return acts
makeActions()[n] is the same for every n because the variable i is somehow looked up at call time. How does Python look up this variable? Shouldn't it not exist at all because makeActions has already exited? Why doesn't Python do what the code intuitively suggests, and define each function by replacing i with its current value within the for loop as the loop is running?

I think it's pretty obvious what happens when you think of i as a name not some sort of value. Your lambda function does something like "take x: look up the value of i, calculate i**x" ... so when you actually run the function, it looks up i just then so i is 4.
You can also use the current number, but you have to make Python bind it to another name:
def makeActions():
def make_lambda( j ):
return lambda x: j * x # the j here is still a name, but now it wont change anymore
acts = []
for i in range(5):
# now you're pushing the current i as a value to another scope and
# bind it there, under a new name
acts.append(make_lambda(i))
return acts
It might seem confusing, because you often get taught that a variable and it's value are the same thing -- which is true, but only in languages that actually use variables. Python has no variables, but names instead.
About your comment, actually i can illustrate the point a bit better:
i = 5
myList = [i, i, i]
i = 6
print(myList) # myList is still [5, 5, 5].
You said you changed i to 6, that is not what actually happend: i=6 means "i have a value, 6 and i want to name it i". The fact that you already used i as a name matters nothing to Python, it will just reassign the name, not change it's value (that only works with variables).
You could say that in myList = [i, i, i], whatever value i currently points to (the number 5) gets three new names: mylist[0], mylist[1], mylist[2]. That's the same thing that happens when you call a function: The arguments are given new names. But that is probably going against any intuition about lists ...
This can explain the behavior in the example: You assign mylist[0]=5, mylist[1]=5, mylist[2]=5 - no wonder they don't change when you reassign the i. If i was something muteable, for example a list, then changing i would reflect on all entries in myList too, because you just have different names for the same value!
The simple fact that you can use mylist[0] on the left hand of a = proves that it is indeed a name. I like to call = the assign name operator: It takes a name on the left, and a expression on the right, then evaluates the expression (call function, look up the values behind names) until it has a value and finally gives the name to the value. It does not change anything.
For Marks comment about compiling functions:
Well, references (and pointers) only make sense when we have some sort of addressable memory. The values are stored somewhere in memory and references lead you that place. Using a reference means going to that place in memory and doing something with it. The problem is that none of these concepts are used by Python!
The Python VM has no concept of memory - values float somewhere in space and names are little tags connected to them (by a little red string). Names and values exist in separate worlds!
This makes a big difference when you compile a function. If you have references, you know the memory location of the object you refer to. Then you can simply replace then reference with this location.
Names on the other hand have no location, so what you have to do (during runtime) is follow that little red string and use whatever is on the other end. That is the way Python compiles functions: Where
ever there is a name in the code, it adds a instruction that will figure out what that name stands for.
So basically Python does fully compile functions, but names are compiled as lookups in the nesting namespaces, not as some sort of reference to memory.
When you use a name, the Python compiler will try to figure out where to which namespace it belongs to. This results in a instruction to load that name from the namespace it found.
Which brings you back to your original problem: In lambda x:x**i, the i is compiled as a lookup in the makeActions namespace (because i was used there). Python has no idea, nor does it care about the value behind it (it does not even have to be a valid name). One that code runs the i gets looked up in it's original namespace and gives the more or less expected value.

What happens when you create a closure:
The closure is constructed with a pointer to the frame (or roughly, block) that it was created in: in this case, the for block.
The closure actually assumes shared ownership of that frame, by incrementing the frame's ref count and stashing the pointer to that frame in the closure. That frame, in turn, keeps around references to the frames it was enclosed in, for variables that were captured further up the stack.
The value of i in that frame keeps changing as long as the for loop is running – each assignment to i updates the binding of i in that frame.
Once the for loop exits, the frame is popped off the stack, but it isn't thrown away as it might usually be! Instead, it's kept around because the closure's reference to the frame is still active. At this point, though, the value of i is no longer updated.
When the closure is invoked, it picks up whatever value of i is in the parent frame at the time of invocation. Since in the for loop you create closures, but don't actually invoke them, the value of i upon invocation will be the last value it had after all the looping was done.
Future calls to makeActions will create different frames. You won't reuse the for loop's previous frame, or update that previous frame's i value, in that case.
In short: frames are garbage-collected just like other Python objects, and in this case, an extra reference is kept around to the frame corresponding to the for block so it doesn't get destroyed when the for loop goes out of scope.
To get the effect you want, you need to have a new frame created for each value of i you want to capture, and each lambda needs to be created with a reference to that new frame. You won't get that from the for block itself, but you could get that from a call to a helper function which will establish the new frame. See THC4k's answer for one possible solution along these lines.

The local references persist because they're contained in the local scope, which the closure keeps a reference to.

I thought that when a function exits, all of its local references disappear.
Except for those locals which are closed over in a closure. Those do not disappear, even when the function to which they are local has returned.

Intuitively one might think i would be captured in its current state but that is not the case. Think of each layer as a dictionary of name value pairs.
Level 1:
acts
i
Level 2:
x
Every time you create a closure for the inner lambda you are capturing a reference to level one. I can only assume that the run-time will perform a look-up of the variable i, starting in level 2 and making its way to level 1. Since you are not executing these functions immediately they will all use the final value of i.
Experts?