Related
I'm a fairly experienced C/C++ (and to some degree, Java) programmer. I'm learning python, but I'm baffled at some strange (for my backgroung) behaviors of the language.
I'm learning about nested function and closures (reading "Learning Python", that seems a really good source for me).
I understand that if I nest a def inside a for loop when I call the created function, it looks up the last value of the captured loop variable (as it captures by reference, as a C++ programmer would put it)
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
and running the program the result is
>>> for f in funcs:
f()
3
3
3
3
Now, I was wrapping my head around this when I stumbled upon this (what to me seems) an inconsistency: if I do
for i in range(4):
funcs[i]()
0
1
2
3
more baffling, if I do
>>> i = 2
>>> funcs[i]()
2
and now, all functions in list returns 2:
for f in funcs:
f()
2
2
2
2
there must be some scope related question that I can't grasp
First, this creates a list of four functions.
funcs = []
for i in range(4):
def f():
print(i)
funcs.append(f)
Each of these functions looks up the value of i and then prints it.
This loops through the list of function and calls each one:
>>> for f in funcs:
f()
As stated above, these functions look up i, which is 3 right now due to the for i in range(4) loop that completed earlier, so you get four printouts of 3.
Now you loop again, using i as the loop variable:
for i in range(4):
funcs[i]()
0
1
2
3
The first time through the loop, i is 0, so when the function looks up i, it gets 0, and then prints that. Then it changes to 1, then 2, then 3.
The following code simply changes i in yet another way, and calls a function:
>>> i = 2
>>> funcs[i]()
2
You could've called any of those functions and they still would've printed 2, because that's the value of i now. You're just getting lost because you looped over range(4) to create these functions, then you looped over range(4) to index the list of functions, and you keep reusing i, and then you reassign i and also use it to index the list.
If you want each function's printed value of i to be fixed at what it was when you defined the function, the easiest way to do that is with a default argument, as those are evaluated when the function is defined rather than when it's called:
funcs = []
for i in range(4):
def f(num=i):
print(num)
funcs.append(f)
For the sake of completeness, this is an alternate implementation:
def getfunc(i):
return lambda: i
funcs = []
for i in range(5):
funcs.append(getfunc(i))
for item in funcs:
print(item())
Your functions
def f():
print(i)
print the current value of i.
If you write
for i in range(4):
funcs[i]()
then i is being set to 0,1,2,3 as you go through the loop. That's what for i in range(4) means.
If you write
for f in funcs:
f()
then i continues with whatever value it already had.
take some time for me to understand, actually in this example,
You will find f.__closure__ is None if you print it, ie. nothing related to closure, it's just about procedure the undefined local var i look for its value:
it can't find its value in local scope, finally find it in global scope (like python MRO)
There is no inconsistency here. The value of i in f() depends on the value of i from the parent scope. After you've run the first for i in range(4) i has the value of the last item in the range, which is 3, and thus all subsequent calls to f() will print 3
If you run
for i in range(4):
funcs[i]()
you redefine the value of i at each iteration step, and so you get 0,1,2,3 as the values printed by f. Doing
for x in range(4):
funcs[x]()
will not affect the value of i and so you'll get 3 as the value of i in all function calls
i'm trying to generate an endless stream of results given a function f and an initial value x
so first call should give the initial value, second call should give f(x), third call is f(x2) while x2 is the previous result of f(x) and so on..
what i have come up with:
def generate(f, x):
return itertools.repeat(lambda x: f(x))
which does not seem to work. any ideas? (i cant use yield in my code). also i cant use more than 1 line of code for this problem. any help would be appreciated.
also note that in a previous ex. i was asked to use the yield. with no problems:
while True:
yield x
x = f(x)
this works fine. but now.. no clue how to do it without
In Python 3.3, you can use itertools.accumulate:
import itertools
def generate(f, x):
return itertools.accumulate(itertools.repeat(x), lambda v,_:f(v))
for i, val in enumerate(generate(lambda x: 2*x, 3)):
print(val)
if i == 10:
break
I think this works:
import itertools as it
def g(f, x):
return it.chain([x],(setattr(g, 'x', f(getattr(g, 'x', x))) or getattr(g, 'x') for _ in it.count()))
def f(x):
return x + 1
gen = g(f, 1)
print next(gen)
print next(gen)
print next(gen)
print next(gen)
Of course, it relys on some sketchy behavior where I actually add an attribute to the function itself to keep the state. Basically, this function will only work the first time you call it. After that, all bets are off.
If we want to relax that restriction, we can use a temporary namespace. The problem is that to get a temporary namespace we need a unique class instance (or class, but an instance is cleaner and only requires 1 extra set of parenthesis). To make that happen in one line, we need to create a new function inline and use that as a default argument:
import itertools as it
def g(f, x):
return (lambda f, x, ns=type('foo', (object,), {})(): \
it.chain([x],
(setattr(ns, 'x', f(getattr(ns, 'x', x))) or getattr(ns, 'x')
for _ in it.count()))
)(f, x)
def f(x):
return x + 1
gen = g(f, 1)
print next(gen) == 1
print next(gen) == 2
print next(gen) == 3
print next(gen) == 4
print "first worked?"
gen2 = g(f, 2)
print next(gen2) == 2
print next(gen2) == 3
print next(gen2) == 4
I've broken it into a few lines, for readability, but it's a 1-liner at heart.
A version without any imports
(and the most robust one yet I believe).
def g(f, x):
return iter(lambda f=f, x=x, ns=type('foo', (object,), {'x':x}): ((getattr(ns, 'x'),setattr(ns, 'x', f(getattr(ns, 'x'))))[0]), object())
One trick here is the same as before. We create a lambda function with a mutable default argument to keep the state. Inside the function, we build a tuple. The first item is what we actually want, the second item is the return value of the setattr function which is used to update the state. In order to get rid of the itertools.chain, we set the initial value on the namespace to the value of x so the class is already initialzed to have the starting state. The second trick is that we use the two argument form of iter to get rid of it.count() which was only used to create an infinite iterable before. iter keeps calling the function you give it as the first argument until the return value is equal to the second argument. However, since my second argument is an instance of object, nothing returned from our function will ever be equal to it so we've effectively created an infinite iterable without itertools or yield! Come to think of it, I believe this last version is the most robust too. Previous versions had a bug where they relied on the truthfulness of the return value of f. I think they might have failed if f returned 0. This last version fixes that bug.
I'm guessing this is some sort of homework or assignment? As such, I'd say you should take a look at generator expressions. Though I agree with the other commenters that this seems an exercise of dubious value...
Python has an elegant way of automatically generating a counter variable in for loops: the enumerate function. This saves the need of initializing and incrementing a counter variable. Counter variables are also ugly because they are often useless once the loop is finished, yet their scope is not the scope of the loop, so they occupy the namespace without need (although I am not sure whether enumerate actually solves this).
My question is, whether there is a similar pythonic solution for while loops. enumerate won't work for while loops since enumerate returns an iterator. Ideally, the solution should be "pythonic" and not require function definitions.
For example:
x=0
c=0
while x<10:
x=int(raw_input())
print x,c
c+=1
In this case we would want to avoid initializing and incrementing c.
Clarification:
This can be done with an endless for loop with manual termination as some have suggested, but I am looking for a solution that makes the code clearer, and I don't think that solution makes the code clearer in this case.
Improvement (in readability, I'd say) to Ignacio's answer:
x = 0
for c in itertools.takewhile(lambda c: x < 10, itertools.count()):
x = int(raw_input())
print x, c
Advantages:
Only the while loop condition is in the loop header, not the side-effect raw_input.
The loop condition can depend on any condition that a normal while loop could. It's not necessary to "import" the variables referenced into the takewhile, as they are already visible in the lambda scope. Additionally it can depend on the count if you want, though not in this case.
Simplified: enumerate no longer appears at all.
Again with the itertools...
import itertools
for c, x in enumerate(
itertools.takewhile(lambda v: v < 10,
(int(raw_input()) for z in itertools.count())
)
):
print c, x
If you want zero initialization before the while loop, you can use a Singleton with a counter:
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
cls.count=0
else:
cls.count+=1
return cls._instance
Then there will only be one instance of Singleton and each additional instance just adds one:
>>> Singleton().count # initial instance
0
>>> Singleton().count
1
>>> Singleton().count
2
>>> Singleton().count
3
Then your while loop becomes:
while Singleton():
x=int(raw_input('x: '))
if x>10: break
print 'While loop executed',Singleton().count,'times'
Entering 1,2,3,11 it prints:
x: 1
x: 2
x: 3
x: 11
While loop executed 4 times
If you do not mind a single line initialization before the while loop, you can just subclass an interator:
import collections
class WhileEnum(collections.Iterator):
def __init__(self,stop=None):
self.stop=stop
self.count=0
def next(self): # '__next__' on Py 3, 'next' on Py 2
if self.stop is not None:
self.remaining=self.stop-self.count
if self.count>=self.stop: return False
self.count+=1
return True
def __call__(self):
return self.next()
Then your while loop becomes:
enu=WhileEnum()
while enu():
i=int(raw_input('x: '))
if i>10: break
print enu.count
I think the second is the far better approach. You can have multiple enumerators and you can also set a limit on how many loops to go:
limited_enum=WhileEnum(5)
I don't think it's possible to do what you want in the exact way you want it. If I understand right, you want a while loop that increments a counter each time through, without actually exposing a visible counter outside the scope of the loop. I think the way to do this would be to rewrite your while loop as a nonterminating for loop, and check the end condition manually. For your example code:
import itertools
x = 0
for c in itertools.count():
x = int(raw_input())
print x, c
if x >= 10:
break
The problem is that fundamentally you're doing iteration, with the counter. If you don't want to expose that counter, it needs to come from the loop construct. Without defining a new function, you're stuck with a standard loop and an explicit check.
On the other hand, you could probably also define a generator for this. You'd still be iterating, but you could at least wrap the check up in the loop construct.
I have a function
def f():
while True:
blah
I want to alter f in such a way that the caller could control the number of times the while loop in f runs, without altering much of the code in f (specially not adding a counter in f). Something like
def f(num_executions = True):
while num_executions:
blah()
f() will run an infinite loop
but f(an_expression_that_evaluates_to_true_n_times) will run the while loop n times.
What could such an expression be?
UPDATE:
I know, there are plenty of way to control how many times a loop will run, but the real question here is -
Can an expression in python evaluate to True for configurable number of times?
Some ideas I am toying with
-making an expression out of list = list[:-1]
-modifying default parameters of a function within a function
No need for a while-loop. Use a for-loop:
>>> def f(n):
... for _ in range(n):
... dostuff()
_ is used as a variable name in a for loop normally to be a placeholder. This loop loops through n amount of times. So f(5) would loop five times.
While I agree with the others that this is a bad idea, it is entirely (and easily) possible:
class BoolChange:
def __init__(self):
self.count = 0
def __bool__(self):
self.count += 1
return self.count <= 5
x = BoolChange()
while x:
print("Running")
This outputs Running five times, then exits.
The main reason this is a bad idea is that it means checking the state of the object modifies it, which is weird behaviour people won't expect. I can't imagine a good use case for this.
You can't do exactly what you describe. What is passed in python is not an expression, but a value. An object. An Immutable object in general evaluate to either True or to False. It will not change during the loop. Mutable object can change its truth value, but you can't make arbitrary object change during a general loop (which does not touch it in any way). In general, as have been said here, you really need to use for statement, or pass in a callable object (say, a function):
def f(is_true = lambda x : True):
while is_true():
blah()
Note that the reason that the callable solution is acceptable, while the "hiding in boolean" #Lattyware demonstrated is not, is that the additional coputation here is explicit - the () tells the reader that almost anything can happen here, depending on the object passed, and you don't need to know that the __bool__ method in this object is silently called and is expected to have side effect.
def f(c=-1):
while c:
print 'blah'
if c > 0: c -= 1
How about using a generator as a coroutine?
def blah():
print 'hi'
def f():
while True:
blah()
yield
x = f()
next(x) # "hi"
The generator here isn't be used for what it yields, but you get to control how many times it blahs externally, because it yields control ever time it blahs.
for i in range(3):
next(x) # blah blah blah
This will also work -
def foo(n=[1,2,3]):
foo.func_defaults = tuple([foo.func_defaults[0][:-1]],)
return n
while foo():
blah()
This question already has answers here:
What do lambda function closures capture?
(7 answers)
Closed 6 months ago.
While I was investigating a problem I had with lexical closures in Javascript code, I came along this problem in Python:
flist = []
for i in xrange(3):
def func(x): return x * i
flist.append(func)
for f in flist:
print f(2)
Note that this example mindfully avoids lambda. It prints "4 4 4", which is surprising. I'd expect "0 2 4".
This equivalent Perl code does it right:
my #flist = ();
foreach my $i (0 .. 2)
{
push(#flist, sub {$i * $_[0]});
}
foreach my $f (#flist)
{
print $f->(2), "\n";
}
"0 2 4" is printed.
Can you please explain the difference ?
Update:
The problem is not with i being global. This displays the same behavior:
flist = []
def outer():
for i in xrange(3):
def inner(x): return x * i
flist.append(inner)
outer()
#~ print i # commented because it causes an error
for f in flist:
print f(2)
As the commented line shows, i is unknown at that point. Still, it prints "4 4 4".
Python is actually behaving as defined. Three separate functions are created, but they each have the closure of the environment they're defined in - in this case, the global environment (or the outer function's environment if the loop is placed inside another function). This is exactly the problem, though - in this environment, i is modified, and the closures all refer to the same i.
Here is the best solution I can come up with - create a function creater and invoke that instead. This will force different environments for each of the functions created, with a different i in each one.
flist = []
for i in xrange(3):
def funcC(j):
def func(x): return x * j
return func
flist.append(funcC(i))
for f in flist:
print f(2)
This is what happens when you mix side effects and functional programming.
The functions defined in the loop keep accessing the same variable i while its value changes. At the end of the loop, all the functions point to the same variable, which is holding the last value in the loop: the effect is what reported in the example.
In order to evaluate i and use its value, a common pattern is to set it as a parameter default: parameter defaults are evaluated when the def statement is executed, and thus the value of the loop variable is frozen.
The following works as expected:
flist = []
for i in xrange(3):
def func(x, i=i): # the *value* of i is copied in func() environment
return x * i
flist.append(func)
for f in flist:
print f(2)
Here's how you do it using the functools library (which I'm not sure was available at the time the question was posed).
from functools import partial
flist = []
def func(i, x): return x * i
for i in range(3):
flist.append(partial(func, i))
for f in flist:
print(f(2))
Outputs 0 2 4, as expected.
look at this:
for f in flist:
print f.func_closure
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
(<cell at 0x00C980B0: int object at 0x009864B4>,)
It means they all point to the same i variable instance, which will have a value of 2 once the loop is over.
A readable solution:
for i in xrange(3):
def ffunc(i):
def func(x): return x * i
return func
flist.append(ffunc(i))
What is happening is that the variable i is captured, and the functions are returning the value it is bound to at the time it is called. In functional languages this kind of situation never arises, as i wouldn't be rebound. However with python, and also as you've seen with lisp, this is no longer true.
The difference with your scheme example is to do with the semantics of the do loop. Scheme is effectively creating a new i variable each time through the loop, rather than reusing an existing i binding as with the other languages. If you use a different variable created external to the loop and mutate it, you'll see the same behaviour in scheme. Try replacing your loop with:
(let ((ii 1)) (
(do ((i 1 (+ 1 i)))
((>= i 4))
(set! flist
(cons (lambda (x) (* ii x)) flist))
(set! ii i))
))
Take a look here for some further discussion of this.
[Edit] Possibly a better way to describe it is to think of the do loop as a macro which performs the following steps:
Define a lambda taking a single parameter (i), with a body defined by the body of the loop,
An immediate call of that lambda with appropriate values of i as its parameter.
ie. the equivalent to the below python:
flist = []
def loop_body(i): # extract body of the for loop to function
def func(x): return x*i
flist.append(func)
map(loop_body, xrange(3)) # for i in xrange(3): body
The i is no longer the one from the parent scope but a brand new variable in its own scope (ie. the parameter to the lambda) and so you get the behaviour you observe. Python doesn't have this implicit new scope, so the body of the for loop just shares the i variable.
The problem is that all of the local functions bind to the same environment and thus to the same i variable. The solution (workaround) is to create separate environments (stack frames) for each function (or lambda):
t = [ (lambda x: lambda y : x*y)(x) for x in range(5)]
>>> t[1](2)
2
>>> t[2](2)
4
I'm still not entirely convinced why in some languages this works one way, and in some another way. In Common Lisp it's like Python:
(defvar *flist* '())
(dotimes (i 3 t)
(setf *flist*
(cons (lambda (x) (* x i)) *flist*)))
(dolist (f *flist*)
(format t "~a~%" (funcall f 2)))
Prints "6 6 6" (note that here the list is from 1 to 3, and built in reverse").
While in Scheme it works like in Perl:
(define flist '())
(do ((i 1 (+ 1 i)))
((>= i 4))
(set! flist
(cons (lambda (x) (* i x)) flist)))
(map
(lambda (f)
(printf "~a~%" (f 2)))
flist)
Prints "6 4 2"
And as I've mentioned already, Javascript is in the Python/CL camp. It appears there is an implementation decision here, which different languages approach in distinct ways. I would love to understand what is the decision, exactly.
The variable i is a global, whose value is 2 at each time the function f is called.
I would be inclined to implement the behavior you're after as follows:
>>> class f:
... def __init__(self, multiplier): self.multiplier = multiplier
... def __call__(self, multiplicand): return self.multiplier*multiplicand
...
>>> flist = [f(i) for i in range(3)]
>>> [g(2) for g in flist]
[0, 2, 4]
Response to your update: It's not the globalness of i per se which is causing this behavior, it's the fact that it's a variable from an enclosing scope which has a fixed value over the times when f is called. In your second example, the value of i is taken from the scope of the kkk function, and nothing is changing that when you call the functions on flist.
The reasoning behind the behavior has already been explained, and multiple solutions have been posted, but I think this is the most pythonic (remember, everything in Python is an object!):
flist = []
for i in xrange(3):
def func(x): return x * func.i
func.i=i
flist.append(func)
for f in flist:
print f(2)
Claudiu's answer is pretty good, using a function generator, but piro's answer is a hack, to be honest, as it's making i into a "hidden" argument with a default value (it'll work fine, but it's not "pythonic").
I didn't like how solutions above created wrappers in the loop. Note: python 3.xx
flist = []
def func(i):
return lambda x: x * i
for i in range(3):
flist.append(func(i))
for f in flist:
print f(2)