How can I check if the some blackbox generator is awaiting the value or it is returning the value now? I mean managing the following generator:
def gen():
a = yield
yield a
yield a+1
yield a+2
can be the following:
g = gen()
g.next()
print g.send(5)
print g.next()
print g.next()
and for the different generator, for example:
def gen():
a = yield
b = yield
yield a+b
it needs to be different also, for example:
g = gen()
g.next()
g.send(1)
print g.send(2)
So the question is how can I choose between sending value in generator and getting results from it only in case of blackbox (third-party) generator? I need to write the following code:
values = [1, 2, 3]
results = list()
g = gen()
g.next()
for v in values:
# needs this magic
if g.__awaits__: # in case of "x = yield" expression
results.append(g.send(v))
elif g.__yields__: # in case of "yield x" expression
results.append(g.next())
You can't, because there isn't any difference in terms of the allowable operations on those two "kinds" of generators. You can always send a value to the generator as soon as you have called next on it once, even if the generator doesn't do anything with the sent value:
>>> def gen():
... yield 1
... yield 2
... yield 3
>>> x = gen()
>>> next(x)
1
>>> x.send('hello')
2
>>> x.send('hello')
3
There's no way to tell whether the generator "needs" you to send a value except by reading the documentation for whatever generator you're using.
Related
[Python 3.8 | Spyder]
I was trying to introduce a simple definition that would either return [1,2,3] or yield the numbers sequentially.
The minimum working code is as follows:
def example(condition):
if condition:
yield 1
yield 2
yield 3
else:
return [1,2,3]
If one attempts to use the def, it will always return a generator. Even if the return appears first in the if/else pair:
def example(condition):
if not condition:
return [1,2,3]
else:
yield 1
yield 2
yield 3
It seems python is not ignoring the presence of the yields even if condition = False.
This is an unexpected behaviour.
You can return the generator in the form of a generator expression:
>>> def example(condition):
... if condition:
... return (i for i in (1, 2, 3))
... else:
... return [1, 2, 3]
>>> example(1)
<generator object example.<locals>.<genexpr> at 0x000000000257BBA0>
>>> example(0)
[1, 2, 3]
Or you can define the generator separately:
>>> def g():
... yield 1
... yield 2
... yield 3
...
>>> def example(condition):
... if condition:
... return g()
... else:
... return [1, 2, 3]
>>> example(1)
<generator object g at 0x000000000257BBA0>
>>> example(0)
[1, 2, 3]
Syntactically, a def statement with a yield statement in its suite defines a generator. It is inconsequential, and in fact undecidable, whether the yield is actually reached at runtime.
def another_example(): # this function has a well-defined type
if random.random() < 0.5:
yield 'What is the type of this function?'
return 'Of course it is a generator function!'
In order for a function to evaluate to either a generator or a non-generator value, use a separate definition for the generator or a generator-expression.
def example_generator():
yield 1
yield 2
yield 3
def example(condition):
if not condition:
return [1,2,3]
else:
return example_generator()
If both paths ultimately should provide the same values, it is semantically identically to make the choice outside of the function. This makes it sufficient to provide just a generator function; the result can be converted to another type externally.
def example():
yield 1
yield 2
yield 3
print("I prefer lists, such as:", list(example()))
I want to use next to skip one or more items returned from a generator. Here is a simplified example designed to skip one item per loop (in actual use, I'd test n and depending on the result, may repeat the next() and the generator is from a package I don't control):
def gen():
for i in range(10):
yield i
for g in gen():
n = next(gen())
print(g, n)
I expected the result to be
0 1
2 3
etc.
Instead I got
0 0
1 0
etc.
What am I doing wrong?
You're making a new generator each time you call gen(). Each new generator starts from 0.
Instead, you can call it once and capture the return value.
def gen():
for i in range(10):
yield i
x = gen()
for g in x:
n = next(x)
print(g, n)
I was messing around and noticed that the following code yields the value once, while I was expecting it to return a generator object.
def f():
yield (yield 1)
f().next() # returns 1
def g():
yield (yield (yield 1)
g().next() # returns 1
My question is what is the value of the yield expression and also why are we allowed to nest yield expression if the yield expression collapses?
The value of the yield expression after resuming depends on the method which resumed the execution. If __next__() is used (typically via either a for or the next() builtin) then the result is None. Otherwise, if send() is used, then the result will be the value passed in to that method.
So this:
def f():
yield (yield 1)
Is equivalent to this:
def f():
x = yield 1
yield x
Which in this case (since you're not using generator.send()) is equivalent to this:
def f():
yield 1
yield None
Your code is only looking at the first item yielded by the generator. If you instead call list() to consume the whole sequence, you'll see what I describe:
def f():
yield (yield 1)
def g():
yield (yield (yield 1))
print(list(f()))
print(list(g()))
Output:
$ python3 yield.py
[1, None]
[1, None, None]
If we iterate the generator manually (as you have), but .send() it values, then you can see that yield "returns" this value:
gen = f()
print(next(gen))
print(gen.send(42))
Output:
$ python3 yield_manual.py
1
42
This is my piece of code with two generators defined:
one_line_gen = (x for x in range(3))
def three_line_gen():
yield 0
yield 1
yield 2
When I execute:
for x in one_line_gen:
print x
for x in one_line_gen:
print x
The result is as expected:
0
1
2
However, if I execute:
for x in three_line_gen():
print x
for x in three_line_gen():
print x
The result is:
0
1
2
0
1
2
Why? I thought any generator can be used only once.
three_line_gen is not a generator, it's a function. What it returns when you call it is a generator, a brand new one each time you call it. Each time you put parenthesis like this:
three_line_gen()
It is a brand new generator to be iterated on. If however you were to first do
mygen = three_line_gen()
and iterate over mygen twice, the second time will fail as you expect.
no, you can not iterate over a generator twice. a generator is exhausted once you have iterated over it. you may make a copy of a generator with tee though:
from itertools import tee
one_line_gen = (x for x in range(3))
gen1, gen2 = tee(one_line_gen)
# or:
# gen1, gen2 = tee(x for x in range(3))
for item in gen1:
print(item)
for item in gen2:
print(item)
for the other issues see Ofer Sadan's answer.
Yes, generator can be used only once. but you have two generator object.
# Python 3
def three_line_gen():
yield 0
yield 1
yield 2
iterator = three_line_gen()
print(iterator)
for x in iterator:
print(id(iterator), x)
iterator2 = three_line_gen()
print(iterator2)
for x in iterator2:
print(id(iterator2), x)
And the result is:
<generator object three_line_gen at 0x1020401b0>
4328784304 0
4328784304 1
4328784304 2
<generator object three_line_gen at 0x1020401f8>
4328784376 0
4328784376 1
4328784376 2
Why? I thought any generator can be used only once.
Because every call to three_line_gen() creates a new generator.
Otherwise, you're correct that generators only run forward until exhausted.
Can generator be used more than once?
Yes, it is possible if the results are buffered outside the generator. One easy way is to use itertools.tee():
>>> from itertools import tee
>>> def three_line_gen():
yield 0
yield 1
yield 2
>>> t1, t2 = tee(three_line_gen())
>>> next(t1)
0
>>> next(t2)
0
>>> list(t1)
[1, 2]
>>> list(t2)
[1, 2]
Because in One liner is Generator Object while the three liner is a function.
They meant to be different.
These two are similar.
def three_line_gen_fun():
yield 0
yield 1
yield 2
three_line_gen = three_line_gen_fun()
one_line_gen = (x for x in range(3))
type(three_line_gen) == type(one_line_gen)
I'm trying to write a function that returns the next element of a generator and if it is at the end of the generator it resets it and returns the next result. The expected output of the code below would be:
1
2
3
1
2
However that is not what I get obviously. What am I doing that is incorrect?
a = '123'
def convert_to_generator(iterable):
return (x for x in iterable)
ag = convert_to_generator(a)
def get_next_item(gen, original):
try:
return next(gen)
except StopIteration:
gen = convert_to_generator(original)
get_next_item(gen, original)
for n in range(5):
print(get_next_item(ag,a))
1
2
3
None
None
Is itertools.cycle(iterable) a possible alternative?
You need to return the result of your recursive call:
return get_next_item(gen, original)
which still does not make this a working approach.
The generator ag used in your for-loop is not changed by the rebinding of the local variable gen in your function. It will stay exhausted...
As has been mentioned in the comments, check out itertools.cycle.
the easy way is just use itertools.cycle, otherwise you would need to remember the elements in the iterable if said iterable is an iterator (aka a generator) becase those can't be reset, if its not a iterator, you can reuse it many times.
the documentation include a example implementation
def cycle(iterable):
# cycle('ABCD') --> A B C D A B C D A B C D ...
saved = []
for element in iterable:
yield element
saved.append(element)
while saved:
for element in saved:
yield element
or for example, to do the reuse thing
def cycle(iterable):
# cycle('ABCD') --> A B C D A B C D A B C D ...
if iter(iterable) is iter(iterable): # is a iterator
saved = []
for element in iterable:
yield element
saved.append(element)
else:
saved = iterable
while saved:
for element in saved:
yield element
example use
test = cycle("123")
for i in range(5):
print(next(test))
now about your code, the problem is simple, it don't remember it state
def get_next_item(gen, original):
try:
return next(gen)
except StopIteration:
gen = convert_to_generator(original) # <-- the problem is here
get_next_item(gen, original) #and you should return something here
in the marked line a new generator is build, but you would need to update your ag variable outside this function to get the desire behavior, there are ways to do it, like changing your function to return the element and the generator, there are other ways, but they are not recommended or more complicated like building a class so it remember its state
get_next_item is a generator, that returns an iterator, that gives you the values it yields via the __next__ method. For that reason, your statement doesn't do anything.
What you want to do is this:
def get_next_item(gen, original):
try:
return next(gen)
except StopIteration:
gen = convert_to_generator(original)
for i in get_next_item(gen, original):
return i
or shorter, and completely equivalent (as long as gen has a __iter__ method, which it probably has):
def get_next_item(gen, original):
for i in gen:
yield i
for i in get_next_item(convert_to_generator(original)):
yield i
Or without recursion (which is a big problem in python, as it is 1. limited in depth and 2. slow):
def get_next_item(gen, original):
for i in gen:
yield i
while True:
for i in convert_to_generator(original):
yield i
If convert_to_generator is just a call to iter, it is even shorter:
def get_next_item(gen, original):
for i in gen:
yield i
while True:
for i in original:
yield i
or, with itertools:
import itertools
def get_next_item(gen, original):
return itertools.chain(gen, itertools.cycle(original))
and get_next_item is equivalent to itertools.cycle if gen is guaranteed to be an iterator for original.
Side note: You can exchange for i in x: yield i for yield from x (where x is some expression) with Python 3.3 or higher.