I have two functions func1 and func2 that are specific implementations of func0 that YIELDS its result:
def func0(parameter, **kwargs):
#do sth with kwargs and parameter
yield result # result is html
how should I refer to func0 inside the "specific" functions to make them yield their results? Is return ok?
def func1(**kwargs):
return func0(parameter=1, **kwargs)
def func2(**kwargs):
return func0(parameter=2, **kwargs)
In Python 3.3+, the normal way would be to use yield from. From the documentation:
PEP 380 adds the yield from expression, allowing a generator to delegate part of its operations to another generator. This allows a section of code containing yield to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator.
For Python 2.7 that's not possible, however. Here's an alternative that works instead:
def base_squared_generator(parameter):
yield parameter ** 2
def two_squared_generator():
yield next(base_squared_generator(parameter=2))
def three_squared_generator():
yield next(base_squared_generator(parameter=3))
print(next(two_squared_generator()))
print(next(three_squared_generator()))
Output
4
9
If you use return, then func1 will return the generator that is func0. Alternatively, if you use yield from, then the wrapping function becomes a generator itself, yielding the individual items from func0. The yielded elements are the same in both cases.
def func1(**kwargs):
return func0(parameter=1, **kwargs)
def func2(**kwargs):
yield from func0(parameter=1, **kwargs)
Note how func1 returns a func0-generator, while func2 returns a func2-generator.
>>> func1()
<generator object func0 at 0x7fe038147ea0>
>>> func2()
<generator object func2 at 0x7fe038147ee8>
>>> list(func1()) == list(func2())
True
Note that yield from was introduced in Python 3. In Python 2, you can achieve the samy by yielding from a loop.
def func2(**kwargs):
for x in func0(parameter=1, **kwargs):
yield x
You are returning generators from the functions.
You need to read about generator, it's not long, anyway is here a way to use it:
gen = func1(args...)
res = gen.next() # python 2
or
res = next(gen) # python 2 and 3
This is how i would do it:
def func0(a):
yield a**2
from functools import partial
func1 = partial(func0, a=1)
func2 = partial(func0, a=10)
print(next(func1())) # prints 1
print(next(func2())) # prints 100
You can take a look at partial there. As i said in the comments it essentially clones your function with some of its required parameters already set.
So if func0 yields so do its partials func1 and func2.
Related
Since Python 3.3, if a generator function returns a value, that becomes the value for the StopIteration exception that is raised. This can be collected a number of ways:
The value of a yield from expression, which implies the enclosing function is also a generator.
Wrapping a call to next() or .send() in a try/except block.
However, if I'm simply wanting to iterate over the generator in a for loop - the easiest way - there doesn't appear to be a way to collect the value of the StopIteration exception, and thus the return value. Im using a simple example where the generator yields values, and returns some kind of summary at the end (running totals, averages, timing statistics, etc).
for i in produce_values():
do_something(i)
values_summary = ....??
One way is to handle the loop myself:
values_iter = produce_values()
try:
while True:
i = next(values_iter)
do_something(i)
except StopIteration as e:
values_summary = e.value
But this throws away the simplicity of the for loop. I can't use yield from since that requires the calling code to be, itself, a generator. Is there a simpler way than the roll-ones-own for loop shown above?
You can think of the value attribute of StopIteration (and arguably StopIteration itself) as implementation details, not designed to be used in "normal" code.
Have a look at PEP 380 that specifies the yield from feature of Python 3.3: It discusses that some alternatives of using StopIteration to carry the return value where considered.
Since you are not supposed to get the return value in an ordinary for loop, there is no syntax for it. The same way as you are not supposed to catch the StopIteration explicitly.
A nice solution for your situation would be a small utility class (might be useful enough for the standard library):
class Generator:
def __init__(self, gen):
self.gen = gen
def __iter__(self):
self.value = yield from self.gen
This wraps any generator and catches its return value to be inspected later:
>>> def test():
... yield 1
... return 2
...
>>> gen = Generator(test())
>>> for i in gen:
... print(i)
...
1
>>> print(gen.value)
2
You could make a helper wrapper, that would catch the StopIteration and extract the value for you:
from functools import wraps
class ValueKeepingGenerator(object):
def __init__(self, g):
self.g = g
self.value = None
def __iter__(self):
self.value = yield from self.g
def keep_value(f):
#wraps(f)
def g(*args, **kwargs):
return ValueKeepingGenerator(f(*args, **kwargs))
return g
#keep_value
def f():
yield 1
yield 2
return "Hi"
v = f()
for x in v:
print(x)
print(v.value)
A light-weight way to handle the return value (one that doesn't involve instantiating an auxiliary class) is to use dependency injection.
Namely, one can pass in the function to handle / act on the return value using the following wrapper / helper generator function:
def handle_return(generator, func):
returned = yield from generator
func(returned)
For example, the following--
def generate():
yield 1
yield 2
return 3
def show_return(value):
print('returned: {}'.format(value))
for x in handle_return(generate(), show_return):
print(x)
results in--
1
2
returned: 3
The most obvious method I can think of for this would be a user defined type that would remember the summary for you..
>>> import random
>>> class ValueProducer:
... def produce_values(self, n):
... self._total = 0
... for i in range(n):
... r = random.randrange(n*100)
... self._total += r
... yield r
... self.value_summary = self._total/n
... return self.value_summary
...
>>> v = ValueProducer()
>>> for i in v.produce_values(3):
... print(i)
...
25
55
179
>>> print(v.value_summary)
86.33333333333333
>>>
Another light weight way sometimes appropriate is to yield the running summary in every generator step in addition to your primary value in a tuple. The loop stays simple with an extra binding which is still available afterwards:
for i, summary in produce_values():
do_something(i)
show_summary(summary)
This is especially useful if someone could use more than just the last summary value, e. g. updating a progress view.
Since Python 3.3, if a generator function returns a value, that becomes the value for the StopIteration exception that is raised. This can be collected a number of ways:
The value of a yield from expression, which implies the enclosing function is also a generator.
Wrapping a call to next() or .send() in a try/except block.
However, if I'm simply wanting to iterate over the generator in a for loop - the easiest way - there doesn't appear to be a way to collect the value of the StopIteration exception, and thus the return value. Im using a simple example where the generator yields values, and returns some kind of summary at the end (running totals, averages, timing statistics, etc).
for i in produce_values():
do_something(i)
values_summary = ....??
One way is to handle the loop myself:
values_iter = produce_values()
try:
while True:
i = next(values_iter)
do_something(i)
except StopIteration as e:
values_summary = e.value
But this throws away the simplicity of the for loop. I can't use yield from since that requires the calling code to be, itself, a generator. Is there a simpler way than the roll-ones-own for loop shown above?
You can think of the value attribute of StopIteration (and arguably StopIteration itself) as implementation details, not designed to be used in "normal" code.
Have a look at PEP 380 that specifies the yield from feature of Python 3.3: It discusses that some alternatives of using StopIteration to carry the return value where considered.
Since you are not supposed to get the return value in an ordinary for loop, there is no syntax for it. The same way as you are not supposed to catch the StopIteration explicitly.
A nice solution for your situation would be a small utility class (might be useful enough for the standard library):
class Generator:
def __init__(self, gen):
self.gen = gen
def __iter__(self):
self.value = yield from self.gen
This wraps any generator and catches its return value to be inspected later:
>>> def test():
... yield 1
... return 2
...
>>> gen = Generator(test())
>>> for i in gen:
... print(i)
...
1
>>> print(gen.value)
2
You could make a helper wrapper, that would catch the StopIteration and extract the value for you:
from functools import wraps
class ValueKeepingGenerator(object):
def __init__(self, g):
self.g = g
self.value = None
def __iter__(self):
self.value = yield from self.g
def keep_value(f):
#wraps(f)
def g(*args, **kwargs):
return ValueKeepingGenerator(f(*args, **kwargs))
return g
#keep_value
def f():
yield 1
yield 2
return "Hi"
v = f()
for x in v:
print(x)
print(v.value)
A light-weight way to handle the return value (one that doesn't involve instantiating an auxiliary class) is to use dependency injection.
Namely, one can pass in the function to handle / act on the return value using the following wrapper / helper generator function:
def handle_return(generator, func):
returned = yield from generator
func(returned)
For example, the following--
def generate():
yield 1
yield 2
return 3
def show_return(value):
print('returned: {}'.format(value))
for x in handle_return(generate(), show_return):
print(x)
results in--
1
2
returned: 3
The most obvious method I can think of for this would be a user defined type that would remember the summary for you..
>>> import random
>>> class ValueProducer:
... def produce_values(self, n):
... self._total = 0
... for i in range(n):
... r = random.randrange(n*100)
... self._total += r
... yield r
... self.value_summary = self._total/n
... return self.value_summary
...
>>> v = ValueProducer()
>>> for i in v.produce_values(3):
... print(i)
...
25
55
179
>>> print(v.value_summary)
86.33333333333333
>>>
Another light weight way sometimes appropriate is to yield the running summary in every generator step in addition to your primary value in a tuple. The loop stays simple with an extra binding which is still available afterwards:
for i, summary in produce_values():
do_something(i)
show_summary(summary)
This is especially useful if someone could use more than just the last summary value, e. g. updating a progress view.
Update: I've started a thread on python-ideas to propose additional syntax or a stdlib function for this purpose (i.e. specifying the first value sent by yield from). So far 0 replies... :/
How do I intercept the first yielded value of a subgenerator but delegate the rest of the iteration to the latter using yield from?
For example, suppose we have an arbitrary bidirectional generator subgen, and we want to wrap this in another generator gen. The purpose of gen is to intercept the first yielded value of subgen and delegate the rest of the generation—including sent values, thrown exceptions, .close(), etc.—to the sub-generator.
The first thing that might come to mind could be this:
def gen():
g = subgen()
first = next(g)
# do something with first...
yield "intercepted"
# delegate the rest
yield from g
But this is wrong, because when the caller .sends something back to the generator after getting the first value, it will end up as the value of the yield "intercepted" expression, which is ignored, and instead g will receive None as the first .send value, as part of the semantics of yield from.
So we might think to do this:
def gen():
g = subgen()
first = next(g)
# do something with first...
received = yield "intercepted"
g.send(received)
# delegate the rest
yield from g
But what we've done here is just moving the problem back by one step: as soon as we call g.send(received), the generator resumes its execution and doesn't stop until it reaches the next yield statement, whose value becomes the return value of the .send call. So we'd also have to intercept that and re-send it. And then send that, and that again, and so on... So this won't do.
Basically, what I'm asking for is a yield from with a way to customize what the first value sent to the generator is:
def gen():
g = subgen()
first = next(g)
# do something with first...
received = yield "intercepted"
# delegate the rest
yield from g start with received # pseudocode; not valid Python
...but without having to re-implement all of the semantics of yield from myself. That is, the laborious and poorly maintainable solution would be:
def adaptor(generator, init_send_value=None):
send = init_send_value
try:
while True:
send = yield generator.send(send)
except StopIteration as e:
return e.value
which is basically a bad re-implementation of yield from (it's missing handling of throw, close, etc.). Ideally I would like something more elegant and less redundant.
If you're trying to implement this generator wrapper as a generator function using yield from, then your question basically boils down to whether it is possible to specify the first value sent to the "yielded from" generator. Which it is not.
If you look at the formal specification of the yield from expression in PEP 380, you can see why. The specification contains a (surprisingly complex) piece of sample code that behaves the same as a yield from expression. The first few lines are:
_i = iter(EXPR)
try:
_y = next(_i)
except StopIteration as _e:
_r = _e.value
else:
...
You can see that the first thing that is done to the iterator is to call next() on it, which is basically equivalent to .send(None). There is no way to skip that step and your generator will always receive another None whenever yield from is used.
The solution I've come up with is to implement the generator protocol using a class instead of a generator function:
class Intercept:
def __init__(self, generator):
self._generator = generator
self._intercepted = False
def __next__(self):
return self.send(None)
def send(self, value):
yielded_value = self._generator.send(value)
# Intercept the first value yielded by the wrapped generator and
# replace it with a different value.
if not self._intercepted:
self._intercepted = True
print(f'Intercepted value: {yielded_value}')
yielded_value = 'intercepted'
return yielded_value
def throw(self, type, *args):
return self._generator.throw(type, *args)
def close(self):
self._generator.close()
__next__(), send(), throw(), close() are described in the Python Reference Manual.
The class wraps the generator passed to it when created will mimic its behavior. The only thing it changes is that the first value yielded by the generator is replaced by a different value before it is returned to the caller.
We can test the behavior with an example generator f() which yields two values and a function main() which sends values into the generator until the generator terminates:
def f():
y = yield 'first'
print(f'f(): {y}')
y = yield 'second'
print(f'f(): {y}')
def main():
value_to_send = 0
gen = f()
try:
x = gen.send(None)
while True:
print(f'main(): {x}')
# Send incrementing integers to the generator.
value_to_send += 1
x = gen.send(value_to_send)
except StopIteration:
print('main(): StopIteration')
main()
When ran, this example will produce the following output, showing which values arrive in the generator and which are returned by the generator:
main(): first
f(): 1
main(): second
f(): 2
main(): StopIteration
Wrapping the generator f() by changing the statement gen = f() to gen = Intercept(f()), produces the following output, showing that the first yielded value has been replaced:
Intercepted value: first
main(): intercepted
f(): 1
main(): second
f(): 2
As all other calls to any of the generator API are forwarded directly to the wrapped generator, it should behave equivalently to the wrapped generator itself.
If I understand the question, I think this works? Meaning, I ran this script and it did what I expected, which was to print all but the first line of the input file. But as long as the generator passed as the argument to the skip_first function can be iterator over, it should work.
def skip_first(thing):
_first = True
for _result in thing:
if _first:
_ first = False
continue
yield _result
inp = open("/var/tmp/test.txt")
for line in skip_first(inp):
print(line, end="")
I have a function which performs an expensive operation and is called often; but, the operation only needs to be performed once - its result could be cached.
I tried making an infinite generator but I didn't get the results I expected:
>>> def g():
... result = "foo"
... while True:
... yield result
...
>>> g()
<generator object g at 0x1093db230> # why didn't it give me "foo"?
Why isn't g a generator?
>>> g
<function g at 0x1093de488>
Edit: it's fine if this approach doesn't work, but I need something which performs exactly like a regular function, like so:
>>> [g() for x in range(3)]
["foo", "foo", "foo"]
g() is a generator function. Calling it returns the generator. You then need to use that generator to get your values. By looping, for example, or by calling next() on it:
gen = g()
value = next(gen)
Note that calling g() again will calculate the same value again and produce a new generator.
You may just want to use a global to cache the value. Storing it as an attribute on the function could work:
def g():
if not hasattr(g, '_cache'):
g._cache = 'foo'
return g._cache
A better way: #functools.lru_cache(maxsize=None). It's been backported to python 2.7, or you could just write your own.
I am occasionally guilty of doing:
def foo():
if hasattr(foo, 'cache'):
return foo.cache
# do work
foo.cache = result
return result
Here's a dead-simple caching decorator. It doesn't take into account any variations in parameters, it just returns the same result after the first call. There are fancier ones out there that cache the result for each combination of inputs ("memoization").
import functools
def callonce(func):
result = []
#functools.wraps(func)
def wrapper(*args, **kwargs):
if not result:
result.append(func(*args, **kwargs))
return result[0]
return wrapper
Usage:
#callonce
def long_running_function(x, y, z):
# do something expensive with x, y, and z, producing result
return result
If you would prefer to write your function as a generator for some reason (perhaps the result is slightly different on each call, but there's still a time-consuming initial setup, or else you just want C-style static variables that allow your function to remember some bit of state from one call to the next), you can use this decorator:
import functools
def gen2func(generator):
gen = []
#functools.wraps(generator)
def wrapper(*args, **kwargs):
if not gen:
gen.append(generator(*args, **kwargs))
return next(gen[0])
return wrapper
Usage:
#gen2func
def long_running_function_in_generator_form(x, y, z):
# do something expensive with x, y, and z, producing result
while True:
yield result
result += 1 # for example
A Python 2.5 or later version that uses .send() to allow parameters to be passed to each iteration of the generator is as follows (note that **kwargs are not supported):
import functools
def gen2func(generator):
gen = []
#functools.wraps(generator)
def wrapper(*args):
if not gen:
gen.append(generator(*args))
return next(gen[0])
return gen[0].send(args)
return wrapper
#gen2func
def function_with_static_vars(a, b, c):
# time-consuming initial setup goes here
# also initialize any "static" vars here
while True:
# do something with a, b, c
a, b, c = yield # get next a, b, c
A better option would be to use memoization. You can create a memoize decorator that you can use to wrap any function that you want to cache the results for. You can find some good implementations here.
You can also leverage Beaker and its cache.
Also it has a tons of extensions.
Suppose I have the following class:
class MyGen(object):
def next(self):
return X()
def send(self, x):
return f(x)
Is it possible to express it as a single function, using the yield keyword? Suppose I have g = MyGen(). Note that g.next() shouldn't call f(), and g.send(x) shouldn't call X(), but f() and X() could share some code.
This code will be almost equivalent:
def my_gen(x=None):
while True:
if x is None:
x = yield X()
else:
x = yield f(x)
One difference is that you can't send a value (other than None) to a generator before calling next() for the first time. Another difference is that sending None won't trigger calling f(), since the generator can't distinguish send(None) and next().
Sven's formulation is exactly the way to go, I just wanted to add that if you want to know more about generators, coroutines and such in Python, this site is the place to go.