Question
Please help pin-point the Python source code that implements the generator send part. I suppose somewhere in github This is Python version 3.8.7rc1 but not familiar with how the repository is organized.
Background
Having difficulty with what PEP 342 and documentation regarding the generator send(value). Hence trying to find out how it is implemented to understand.
there is no yield expression to receive a value when the generator has just been created
The value argument becomes the result of the **current yield expression**. The send() method returns the **next value yielded by the generator
Specification: Sending Values into Generators
Because generator-iterators begin execution at the top of the
generator's function body, there is no yield expression to receive a
value when the generator has just been created. Therefore, calling
send() with a non-None argument is prohibited when the generator
iterator has just started, and a TypeError is raised if this occurs
(presumably due to a logic error of some kind). Thus, before you can
communicate with a coroutine you must first call next() or send(None)
to advance its execution to the first yield expression.
generator.send(value)
Resumes the execution and “sends” a value into the generator function.
The value argument becomes the result of the current yield expression.
The send() method returns the next value yielded by the generator, or
raises StopIteration if the generator exits without yielding another
value. When send() is called to start the generator, it must be called
with None as the argument, because there is no yield expression that
could receive the value.
I suppose yield would be like a UNIX system call moving into a routine, inside which stack frame and execution pointer are saved and the generator co-routine is suspended. I think when save(value) is called, some tricks happen there and those are regarding the cryptic parts in the documents.
Although sent_value = (yield value) is one line statement, the blocking and resuming both happens in the same line, I think. The execution does not resume after the yield but within it, hence would like to know how block/resume are implemented. Also I believe next(generator) is the same with generator.send(None) and would like to verify.
Look here for class Generator, also look for this file, is a complete implementation of generators on C inside Python
Do I have to write
def count10():
for i in range(10):
yield i
gen = count10()
for j in gen:
print(j)
gen.close()
to save memory, or just
def count10():
for i in range(10):
yield i
for j in count10():
print(j)
In fact I would like to learn details of lifecycle of Python generator but failed to find relevant resources.
You don't need to close that generator.
close-ing a generator isn't about saving memory. (close-ing things is almost never about saving memory.) The idea behind the close method on a generator is that you might stop iterating over a generator while it's still in the middle of a try or with:
def gen():
with something_important():
yield from range(10)
for i in gen():
if i == 5:
break
close-ing a suspended generator throws a GeneratorExit exception into the generator, with the intent of triggering finally blocks and context manager __exit__ methods. Here, close would cause the generator to run the __exit__ method of something_important(). If you don't abandon a generator in the middle like this (or if your generator doesn't have any finally or with blocks, including in generators it delegates to with yield from), then close is unnecessary (and does nothing).
The memory management system usually runs close for you, but to really ensure prompt closure across Python implementations, you'd have to replace code like
for thing in gen():
...
with
with contextlib.closing(gen()) as generator:
for thing in generator:
...
I've never seen anyone do this.
The close method for generators came about in PEP 342:
Add a close() method for generator-iterators, which raises GeneratorExit at the point where the generator was paused. If the generator then raises StopIteration (by exiting normally, or due to already being closed) or GeneratorExit (by not catching the exception), close() returns to its caller. If the generator yields a value, a RuntimeError is raised. If the generator raises any other exception, it is propagated to the caller. close() does nothing if the generator has already exited due to an exception or normal exit.
Note the last sentence: close() does nothing if the generator has already exited due to an exception or normal exit.
I am trying to make a function that yields a value. Before anything could be done, I want to make sure function input is valid. Below code creates generator after execution. It raises exception only after next. Is there an elegant structure of the function which throws exception before next?
def foo(value):
if validate(value):
raise ValueError
yield 1
you can't check the value before using next, that is the whole point of using gnerators, from the docs:
Each yield temporarily suspends processing, remembering the location
execution state (including local variables and pending
try-statements). When the generator iterator resumes, it picks up
where it left off (in contrast to functions which start fresh on every
invocation).
what you can do it is to check the value before using the generator
Why in the example function terminates:
def func(iterable):
while True:
val = next(iterable)
yield val
but if I take off yield statement function will raise StopIteration exception?
EDIT: Sorry for misleading you guys. I know what generators are and how to use them. Of course when I said function terminates I didn't mean eager evaluation of function. I just implied that when I use function to produce generator:
gen = func(iterable)
in case of func it works and returns the same generator, but in case of func2:
def func2(iterable):
while True:
val = next(iterable)
it raises StopIteration instead of None return or infinite loop.
Let me be more specific. There is a function tee in itertools which is equivalent to:
def tee(iterable, n=2):
it = iter(iterable)
deques = [collections.deque() for i in range(n)]
def gen(mydeque):
while True:
if not mydeque: # when the local deque is empty
newval = next(it) # fetch a new value and
for d in deques: # load it to all the deques
d.append(newval)
yield mydeque.popleft()
return tuple(gen(d) for d in deques)
There is, in fact, some magic, because nested function gen has infinite loop without break statements. gen function terminates due to StopIteration exception when there is no items in it. But it terminates correctly (without raising exceptions), i.e. just stops loop. So the question is: where is StopIteration is handled?
Note: This question (and the original part of my answer to it) are only really meaningful for Python versions prior to 3.7. The behavior that was asked about no longer happens in 3.7 and later, thanks to changes described in PEP 479. So this question and the original answer are only really useful as historical artifacts. After the PEP was accepted, I added an additional section at the bottom of the answer which is more relevant to modern versions of Python.
To answer your question about where the StopIteration gets caught in the gen generator created inside of itertools.tee: it doesn't. It is up to the consumer of the tee results to catch the exception as they iterate.
First off, it's important to note that a generator function (which is any function with a yield statement in it, anywhere) is fundamentally different than a normal function. Instead of running the function's code when it is called, instead, you'll just get a generator object when you call the function. Only when you iterate over the generator will you run the code.
A generator function will never finish iterating without raising StopIteration (unless it raises some other exception instead). StopIteration is the signal from the generator that it is done, and it is not optional. If you reach a return statement or the end of the generator function's code without raising anything, Python will raise StopIteration for you!
This is different from regular functions, which return None if they reach the end without returning anything else. It ties in with the different ways that generators work, as I described above.
Here's an example generator function that will make it easy to see how StopIteration gets raised:
def simple_generator():
yield "foo"
yield "bar"
# StopIteration will be raised here automatically
Here's what happens when you consume it:
>>> g = simple_generator()
>>> next(g)
'foo'
>>> next(g)
'bar'
>>> next(g)
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
next(g)
StopIteration
Calling simple_generator always returns a generator object immediately (without running any of the code in the function). Each call of next on the generator object runs the code until the next yield statement, and returns the yielded value. If there is no more to get, StopIteration is raised.
Now, normally you don't see StopIteration exceptions. The reason for this is that you usually consume generators inside for loops. A for statement will automatically call next over and over until StopIteration gets raised. It will catch and suppress the StopIteration exception for you, so you don't need to mess around with try/except blocks to deal with it.
A for loop like for item in iterable: do_suff(item) is almost exactly equivalent to this while loop (the only difference being that a real for doesn't need a temporary variable to hold the iterator):
iterator = iter(iterable)
try:
while True:
item = next(iterator)
do_stuff(item)
except StopIteration:
pass
finally:
del iterator
The gen generator function you showed at the top is one exception. It uses the StopIteration exception produced by the iterator it is consuming as it's own signal that it is done being iterated on. That is, rather than catching the StopIteration and then breaking out of the loop, it simply lets the exception go uncaught (presumably to be caught by some higher level code).
Unrelated to the main question, there is one other thing I want to point out. In your code, you're calling next on an variable called iterable. If you take that name as documentation for what type of object you will get, this is not necessarily safe.
next is part of the iterator protocol, not the iterable (or container) protocol. It may work for some kinds of iterables (such as files and generators, as those types are their own iterators), but it will fail for others iterables, such as tuples and lists. The more correct approach is to call iter on your iterable value, then call next on the iterator you receive. (Or just use for loops, which call both iter and next for you at appropriate times!)
I just found my own answer in a Google search for a related question, and I feel I should update to point out that the answer above is not true in modern Python versions.
PEP 479 has made it an error to allow a StopIteration to bubble up uncaught from a generator function. If that happens, Python will turn it into a RuntimeError exception instead. This means that code like the examples in older versions of itertools that used a StopIteration to break out of a generator function needs to be modified. Usually you'll need to catch the exception with a try/except and then return.
Because this was a backwards incompatible change, it was phased in gradually. In Python 3.5, all code worked as before by default, but you could get the new behavior with from __future__ import generator_stop. In Python 3.6, unmodified code would still work, but it would give a warning. In Python 3.7 and later, the new behavior applies all the time.
When a function contains yield, calling it does not actually execute anything, it merely creates a generator object. Only iterating over this object will execute the code. So my guess is that you're merely calling the function, which means the function doesn't raise StopIteration because it is never being executed.
Given your function, and an iterable:
def func(iterable):
while True:
val = next(iterable)
yield val
iterable = iter([1, 2, 3])
This is the wrong way to call it:
func(iterable)
This is the right way:
for item in func(iterable):
# do something with item
You could also store the generator in a variable and call next() on it (or iterate over it in some other way):
gen = func(iterable)
print(next(gen)) # prints 1
print(next(gen)) # prints 2
print(next(gen)) # prints 3
print(next(gen)) # StopIteration
By the way, a better way to write your function is as follows:
def func(iterable):
for item in iterable:
yield item
Or in Python 3.3 and later:
def func(iterable):
yield from iter(iterable)
Of course, real generators are rarely so trivial. :-)
Without the yield, you iterate over the entire iterable without stopping to do anything with val. The while loop does not catch the StopIteration exception. An equivalent for loop would be:
def func(iterable):
for val in iterable:
pass
which does catch the StopIteration and simply exit the loop and thus return from the function.
You can explicitly catch the exception:
def func(iterable):
while True:
try:
val = next(iterable)
except StopIteration:
break
yield doesn't catch the StopIteration. What yield does for your function is it causes it to become a generator function rather than a regular function. Thus, the object returned from the function call is an iterable object (which calculates the next value when you ask it to with the next function (which gets called implicitly by a for loop)). If you leave the yield statement out of it, then python executes the entire while loop right away which ends up exhausting the iterable (if it is finite) and raising StopIteration right when you call it.
consider:
x = func(x for x in [])
next(x) #raises StopIteration
A for loop catches the exception -- That's how it knows when to stop calling next on the iterable you gave it.
Tested on Python 3.8, chunk as lazy generator
def split_to_chunk(size: int, iterable: Iterable) -> Iterable[Iterable]:
source_iter = iter(iterable)
while True:
batch_iter = itertools.islice(source_iter, size)
try:
yield itertools.chain([next(batch_iter)], batch_iter)
except StopIteration:
return
Why handling StopInteration error: https://www.python.org/dev/peps/pep-0479/
def sample_gen() -> Iterable[int]:
i = 0
while True:
yield i
i += 1
for chunk in split_to_chunk(7, sample_gen()):
pprint.pprint(list(chunk))
time.sleep(2)
Output:
[0, 1, 2, 3, 4, 5, 6]
[7, 8, 9, 10, 11, 12, 13]
[14, 15, 16, 17, 18, 19, 20]
[21, 22, 23, 24, 25, 26, 27]
............................
I'm curious about the difference between using raise StopIteration and a return statement in generators.
For example, is there any difference between these two functions?
def my_generator0(n):
for i in range(n):
yield i
if i >= 5:
return
def my_generator1(n):
for i in range(n):
yield i
if i >= 5:
raise StopIteration
I'm guessing the more "pythonic" way to do it is the second way (please correct me if I'm wrong), but as far as I can see both ways raise a StopIteration exception.
There's no need to explicitly raise StopIteration as that's what a bare return statement does for a generator function - so yes they're the same. But no, just using return is more Pythonic.
From: http://docs.python.org/2/reference/simple_stmts.html#the-return-statement (valid to Python 3.2)
In a generator function, the return statement is not allowed to include an expression_list. In that context, a bare return indicates that the generator is done and will cause StopIteration to be raised.
Or as #Bakuriu points out - the semantics of generators have changed slightly for Python 3.3, so the following is more appropriate:
In a generator function, the return statement indicates that the generator is done and will cause StopIteration to be raised. The returned value (if any) is used as an argument to construct StopIteration and becomes the StopIteration.value attribute.
As of late 2014 return is correct and raise StopIteration for ending a generator is on a depreciation schedule. See PEP 479 for full details.
Abstract
This PEP proposes a change to generators: when StopIteration is raised inside a generator, it is replaced with RuntimeError. (More precisely, this happens when the exception is about to bubble out of the generator's stack frame.) Because the change is backwards incompatible, the feature is initially introduced using a __future__ statement.
Acceptance
This PEP was accepted by the BDFL on November 22…
Rationale
The interaction of generators and StopIteration is currently somewhat surprising, and can conceal obscure bugs. An unexpected exception should not result in subtly altered behaviour, but should cause a noisy and easily-debugged traceback. Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.
…
That's true, they are equivalent except that one is readable whereas the other is obscure. This dates back to the very first version of generators (PEP 255, under "Specification: Return"), and the subsequent enhancements of (such as coroutines) do not change this. 3.3's yield from (PEP 380) extends that to return <expr> as syntactic sugar for raise StopIteration(<expr>), but that doesn't change the meaning of return;.