This question already has answers here:
Python: Is the split function evaluated multiple times in a list comprehension?
(3 answers)
Closed 7 years ago.
Which one's a better way of doing list comprehension in python (in terms of computation time & cpu cycles).
In example (1) is the value f(r) evaluated in each iteration or is it evaluated once and cached ?
y = [x*f(r) for x in xlist]
c = f(r)
y = [x*c for x in xlist]
where
def f(r):
... some arbitrary function ...
It evaluates for every iteration. Look at this:
>>> def f():
... print("func")
...
>>> [f() for i in range(4)]
func
func
func
func
[None, None, None, None]
As you say, if f() has no side effects, storing the return value on a variable and using that variable instead is a lot faster solution.
I would probably choose the latter because the Python compiler doesn't know if the function has side-effects so it is called for each element.
Here is an easy way to find out:
>>> def f():
... print "called"
... return 1
...
>>> [1+f() for x in xrange(5)]
called
called
called
called
called
[2, 2, 2, 2, 2]
so yes, if the function is going to be the same each time then it is better to call it once outside the list comprehension.
The function f will be called for every element.
It is very complex for the compiler/interpreter to determine that the function need not to be called many times. It is then very probable that the function is called many times. Thus, using the second solution is always the best solution.
Functions have a non-trivial execution time compared to a name lookup, and caching the value is considered acceptable if the function is called many times and the same value is expected each time.
Python is probably free to do it once or many times, I'm not sure I would rely on any observed behavior. It might change in the next version.
If it's important to you to make sure the function is only called once, call it yourself and save the results.
Related
I am using python in operator with for loop and with if statement. My question is how is in implemented, that it behaves differently in this two cases: it iterates when using for loop and it checks if some element exists when using with if statement? Does this depend on implementation of for and if?
for i in x:
#iterates
if i in x:
#checks weather i in x or not
Membership testing with in is implemented via the method __contains__ (see Documentation). The different behaviour comes from the keyword before, for and if in your case.
Iteration with for is implemented such, that the method next is called and its return value is written to the iteration variable as long as the condition after the key word for is true. Membership testing in general is just a condition.
Code
A in B
Execution
B.__contains__(A) # returns boolean
Code
for A in B :
# Body
Execution
A = B.next()
if B.__contains__(A) :
# Body
A = B.next()
if B.__contains__(A) :
# Body
# ...
A = B.next()
if B.__contains__(A) :
# B.next() throws "StopIteration" if there is no more element
The keyword "in" in python solves different purposes based on "for" and "if". please look at this related link in stack overflow for more clarity
In many languages you'll find keywords that have multiple uses. This is simply an example of that. It's probably more helpful to think in terms of statements than thinking about the in keyword like an operator.
The statement x in y is a boolean-valued statement taking (assuming y is some appropriate collection) True if and only if the value of x is in the collection y. It is implemented with the __contains__ member function of y.
The statement for x in y: starts a loop, where each iteration x takes a different value from the collection y. This is implemented using the __iter__ member function of y and __next__ on the resulting iterator object.
There are other statements where the in keyword can appear, such as list comprehension or generator comprehension.
The reason is that for...in is something different from just in.
for x in y iterates over y.
if x in y calls y.__contains__(x).
The in keyword is an operator usually:
print(2 in [1, 2, 3]) # True
if 3 in range(7, 20):
print('3 in range!')
It corresponds to the object.__contains__ special method. The expression a in b corresponds to type(b).__contains__(a). Note that both a and b are names that are looked up.
In a for statement, in is not an operator. It is part of the for .. in .. syntax and separates the loop variable name from the iterable.
for thing in range(20):
print(thing) # thing is something else on each step
Note that for a in b only b is a name that is looked up. a is a name to bind to, similar to an assignment statement.
Python syntax has several constructs where the leading keyword defines the meaning of following keywords. For example, the as keyword has a different meaning in import and with:
# as aliases ExitStack
from contextlib import ExitStack as ContextStack
# as holds the result of ContextStack().__enter__()
with ContextStack() as stack:
...
It helps to think about such keywords not by implementation but by meaning. For example, a in b always means that "a is contained by b".
take the code:
SUM = sum(x() for x in xs)
I am writing some code that is in need of calling another function before each x() such that x() will compute the right value
is the only way to do this like so?
for x in xs: x.pre()
SUM = sum(x() for x in xs)
or
SUM = 0
for x in xs:
x.pre()
SUM += x()
incorporating x.pre into x is obviously possible but would make the real code exceptionally ugly and hard to read. is there some way of using generator expressions I am unaware of that would allow what I am trying to achieve?
eg:
SUM = sum(x(), x.pre() for x in xs)
thats obviously just an non-summable tuple generator
I would simply use the for-loop you already presented.
There are ways to do what you want in other ways. For example you could use functools.reduce with a customized function here instead of sum :
def pre_then_add(accumulated, new_one):
new_one.pre() # do some stuff so we get the right value
return accumulated + new_one() # add the value to the accumulated sum
and then it's just a matter of calling reduce:
import functools
functools.reduce(pre_then_add, xs, 0)
Note that this requires to give a "base value" otherwise the function is too simple but also that reduce with a customized function is not necessarily the most efficient or elegant way.
As pointed in the comments (thanks to #SteveJessop) another possibility using sum would be:
def pre_then_call(x):
x.pre();
return x()
sum(pre_then_call(x) for x in xs)
But just to repeat it: I would use the explicit loop. Those are far easier to understand in terms of what you are doing and why.
I'm trying to learn pure functional programming. But this code is puzzling me particularly the second line. I do not understand how the value 2 is passed to the variable x. Can somebody explain this nested lambda behaviour?
>>> square_func = lambda x: x**2
>>> function_product = lambda F, m: lambda x: F(x)*m
>>> square_func(2)
4
>>> function_product(square_func, 3)(2)
12
The inner lambda creates a function when the outer lambda is called. The outer lambda returns this function. This function is then called with the argument 2.
A good place to start would be to apply type to your definitions and see if it clarifies things. Also, I can't help but remark that something like Haskell would be a nicer place to start if you are interested in functional programming, even if you do not plan on using the language. That being said, here is what you get:
In [13]: type(square_func)
Out[13]: function
In [14]: type(function_product)
Out[14]: function
In [15]: type(square_func(2))
Out[15]: int
In [16]: type(function_product(square_func, 3))
Out[16]: function
In [17]: type(function_product(square_func, 3)(2))
Out[17]: int
So the puzzling part is the return type of function_product(square_func, 3), which is a function itself, one that is presumably intended to take a single number and return a single number. You could write it as:
f = function_product(square_func, 3)
f(2)
It's kind of a rule (or convention), if you follow the style guide, not to use lambda in the context you have used. The reason for this is exactly what made you turn to the internet in confusion. The flags are:
You are giving an anonymous function a name
There are multiple lambda. For a language that does not have nested anonymous functions, this is a code smell. There must be a better way
How about writing this way and reaping the benefits:
def function_product(F, m): # returns inner, a function
def inner(x): # takes x, and closes over F and m from
return F(x)*m # outer scope, hence a closure
return inner
See, everything is clear because it's more readable now. Avoid lambda, maybe except for callbacks,like in sorted(). Normal functions are objects, just like 10 and hello are.
Always do import this. :)
function_product asks for a function and a variable, you passed square_func and 3. Then, the inner function of function_product takes the result of the function you passed (square_func) and multiply it by the variable you passed, so:
square_func(2)*3 = 12
function_product(square_func, 3) returns other lamba function which can be defined implicitly like this:
lambda x: square_func(x)*3
next by calling the other lamba function you pass 2 to the variable x
This question already has answers here:
Creating lambda inside a loop [duplicate]
(3 answers)
Closed 6 years ago.
I'm encountering some strange behavior with lambda functions in a loop in python. When I try to assign lambda functions to dictionary entries in a list, and when other entries in the dictionary are used in the function, only the last time through the loop is the lambda operator evaluated. So all of the functions end up having the same value!
Below is stripped-down code that captures just the parts of what I'm trying that is behaving oddly. My actual code is more complex, not as trivial as this, so I'm looking for an explanation and, preferably, a workaround.
n=4
numbers=range(n)
entries = [dict() for x in numbers]
for number, entry in zip(numbers,entries):
n = number
entry["number"] = n
entry["number2"] = lambda x: n*1
for number in numbers:
print(entries[number]["number"], entries[number]["number2"](2))
The output is:
0 3
1 3
2 3
3 3
In other words, the dictionary entires that are just integers are fine, and were filled properly by the loop. But the lambda functions — which are trivial and should just return the same value as the "number" entries — are all set to the last pass through.
What's going on?
Try this
N=4
numbers=range(N)
entries = [dict() for x in numbers]
for number, entry in zip(numbers,entries):
entry["number"] = number
entry["number2"] = lambda x,n=number: n*1
for number in numbers:
print(entries[number]["number"], entries[number]["number2"](2))
It prints (python3)
0 0
1 1
2 2
3 3
To avoid confusion, n referred to different things in your code. I used it only at one place.
It is a closure problem.
By the end of your for loop, the n variable - which, unlike in static languages such as C#, is set to 3, which is then being accessed in the lambda expression. The variable value is not fixed; as another answer on the site points out, lambda expressions are fluid and will retain references to the variables involved instead of capturing the values at the time of creation. This question also discusses your issue.
To fix it, you need to give the lambdas new, local variable via default parameters:
entry["number2"] = lambda x, n=n: n*1
This creates a new variable in the lambda's scope, called n, which sets its default value to the "outside" value of n. Note that this is the solution endorsed by the official FAQ, as this answer by Adrien Plisson states.
Now, you can call your lambda like normal and ignore the optional parameter, with no ill effect.
EDIT: As originally stated by Sci Prog, this solution makes n = number redundant. Your final code will look similar to this:
lim = 4
numbers = range(lim)
entries = [dict() for x in numbers]
for number, entry in zip(numbers, entries):
entry["number"] = number
entry["number2"] = lambda x, n = number: n*1
for number in numbers:
print(entries[number]["number"], entries[number]["number2"](2))
You are probably reaching the problem that the method is created as referencing a variable n. The function is only evaluated after the loop so you are going to call the function which references n. If you're ok with having the function evaluated at the time of assignment you could put a function call around it:
(lambda x: n*1)(2)
or if you want to have the functions to use, have them reference the specific value you want. From your code you could use a default argument as a workaround:
entry["number"] = n
entry["number2"] = lambda x, n=n: n*1
The difference comes down to a question of memory addressing. I imagine it went something like this:
You: Python, please give me a variable called "n"
Python: Ok! Here it is, it is at memory slot 1
You: Cool! I will now create functions which say take that variable "n"
value (at memory slot 1) and multiply it by 1 and return that to me.
Python: Ok! Got it:
1. Take the value at memory slot 1.
2. Multiply by 1.
3. Return it to you.
You: Done with my looping, now evaluate those instructions!
Python: Ok! Now I will take the value of at memory slot 1 and multiply by 1
and give that to you.
You: Hey, I wanted each function to reference different values!
Python: I followed your instructions exactly!
This question already has answers here:
Creating functions (or lambdas) in a loop (or comprehension)
(6 answers)
Closed 6 months ago.
I am trying to use closures to eliminate a variable from a function signature (the application is to make writing all the functions needed for connecting Qt signals for an interface to control a largish number of parameters to the dictionary that stores the values ).
I do not understand why the case of using the lambda not wrapped in another function returns the last name for all cases.
names = ['a', 'b', 'c']
def test_fun(name, x):
print(name, x)
def gen_clousure(name):
return lambda x: test_fun(name, x)
funcs1 = [gen_clousure(n) for n in names]
funcs2 = [lambda x: test_fun(n, x) for n in names]
# this is what I want
In [88]: for f in funcs1:
....: f(1)
a 1
b 1
c 1
# I do not understand why I get this
In [89]: for f in funcs2:
....: f(1)
c 1
c 1
c 1
The reason is that closures (lambdas or otherwise) close over names, not values. When you define lambda x: test_fun(n, x), the n is not evaluated, because it is inside the function. It is evaluated when the function is called, at which time the value that is there is the last value from the loop.
You say at the beginning that you want to "use closures to eliminate a variable from a function signature", but it doesn't really work that way. (See below, though, for a way that may satisfy you, depending on what you mean by "eliminate".) Variables inside the function body will not be evaluated when the function is defined. In order to get the function to take a "snapshot" of the variable as it exists at function-definition time, you must pass the variable as an argument. The usual way to do this is to give the function an argument whose default value is the variable from the outer scope. Look at the difference between these two examples:
>>> stuff = [lambda x: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
4
4
4
>>> stuff = [lambda x, n=n: n+x for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
In the second example, passing n as an argument to the function "locks in" the current value of n to that function. You have to do something like this if you want to lock in the value in this way. (If it didn't work this way, things like global variables wouldn't work at all; it's essential that free variables be looked up at the time of use.)
Note that nothing about this behavior is specific to lambdas. The same scoping rules are in effect if you use def to define a function that references variables from the enclosing scope.
If you really want to, you can avoid adding the extra argument to your returned function, but to do so you must wrap that function in yet another function, like so:
>>> def makeFunc(n):
... return lambda x: x+n
>>> stuff = [makeFunc(n) for n in [1, 2, 3]]
>>> for f in stuff:
... print f(1)
2
3
4
Here, the inner lambda still looks up the value of n when it is called. But the n it refers to is no longer a global variable but a local variable inside the enclosing function makeFunc. A new value of this local variable is created every time makeFunc is called, and the returned lambda creates a closure that "saves" the local variable value that was in effect for that invocation of makeFunc. Thus each function created in the loop has its own "private" variable called x. (For this simple case, this can also be done using a lambda for the outer function --- stuff = [(lambda n: lambda x: x+n)(n) for n in [1, 2, 3]] --- but this is less readable.)
Notice that you still have to pass your n as an argument, it's just that, by doing it this way, you don't pass it as an argument to the same function that winds up going into the stuff list; instead you pass it as an argument to a helper function that creates the function you want to put into stuff. The advantage of using this two-function approach is that the returned function is "clean" and doesn't have the extra argument; this could be useful if you were wrapping functions that accepted a lot of arguments, in which case it could become confusing to remember where the n argument was in the list. The disadvantage is that, doing it this way, the process of making the functions is more complicated, since you need another enclosing function.
The upshot is that there is a tradeoff: you can make the function-creation process simpler (i.e., no need for two nested functions), but then you must make the resulting function a bit more complicated (i.e., it has this extra n=n argument). Or you can make the function simpler (i.e., it has no n=n argument), but then you must make the function-creation process more complicated (i.e., you need two nested functions to implement the mechanism).