I'm slowly getting to wrap my head around Python generators.
While it's not a real life problem for now, I'm still wondering why I can't return a generator from a function.
When I define a function with yield, it acts as a generator. But if I define it inside another function and try to return that instead, I get an ordinary function, i.e. not a generator with next method.
In other words, why the give_gen() approach in code below does not work?
#!/usr/bin/python
import time
def gen(d):
n = 0
while True:
n = n + d
time.sleep(0.5)
yield n
def give_gen(d):
def fn():
n = 0
while True:
n = n + d
time.sleep(0.5)
yield n
return fn
if __name__ == '__main__':
g = give_gen(3) # does not work
g = gen(3) # works well
while True:
print g.next()
# AttributeError: 'function' object has no attribute 'next'
# in case of give_gen
Why can't I return a generator from a function?
A generator function returns a generator only when called. Call fn to create the generator object:
return fn()
or call the returned object:
g = give_gen(3)()
You did call gen(); had you referred to just gen without calling it you'd have a reference to that function.
Related
Update: I've started a thread on python-ideas to propose additional syntax or a stdlib function for this purpose (i.e. specifying the first value sent by yield from). So far 0 replies... :/
How do I intercept the first yielded value of a subgenerator but delegate the rest of the iteration to the latter using yield from?
For example, suppose we have an arbitrary bidirectional generator subgen, and we want to wrap this in another generator gen. The purpose of gen is to intercept the first yielded value of subgen and delegate the rest of the generation—including sent values, thrown exceptions, .close(), etc.—to the sub-generator.
The first thing that might come to mind could be this:
def gen():
g = subgen()
first = next(g)
# do something with first...
yield "intercepted"
# delegate the rest
yield from g
But this is wrong, because when the caller .sends something back to the generator after getting the first value, it will end up as the value of the yield "intercepted" expression, which is ignored, and instead g will receive None as the first .send value, as part of the semantics of yield from.
So we might think to do this:
def gen():
g = subgen()
first = next(g)
# do something with first...
received = yield "intercepted"
g.send(received)
# delegate the rest
yield from g
But what we've done here is just moving the problem back by one step: as soon as we call g.send(received), the generator resumes its execution and doesn't stop until it reaches the next yield statement, whose value becomes the return value of the .send call. So we'd also have to intercept that and re-send it. And then send that, and that again, and so on... So this won't do.
Basically, what I'm asking for is a yield from with a way to customize what the first value sent to the generator is:
def gen():
g = subgen()
first = next(g)
# do something with first...
received = yield "intercepted"
# delegate the rest
yield from g start with received # pseudocode; not valid Python
...but without having to re-implement all of the semantics of yield from myself. That is, the laborious and poorly maintainable solution would be:
def adaptor(generator, init_send_value=None):
send = init_send_value
try:
while True:
send = yield generator.send(send)
except StopIteration as e:
return e.value
which is basically a bad re-implementation of yield from (it's missing handling of throw, close, etc.). Ideally I would like something more elegant and less redundant.
If you're trying to implement this generator wrapper as a generator function using yield from, then your question basically boils down to whether it is possible to specify the first value sent to the "yielded from" generator. Which it is not.
If you look at the formal specification of the yield from expression in PEP 380, you can see why. The specification contains a (surprisingly complex) piece of sample code that behaves the same as a yield from expression. The first few lines are:
_i = iter(EXPR)
try:
_y = next(_i)
except StopIteration as _e:
_r = _e.value
else:
...
You can see that the first thing that is done to the iterator is to call next() on it, which is basically equivalent to .send(None). There is no way to skip that step and your generator will always receive another None whenever yield from is used.
The solution I've come up with is to implement the generator protocol using a class instead of a generator function:
class Intercept:
def __init__(self, generator):
self._generator = generator
self._intercepted = False
def __next__(self):
return self.send(None)
def send(self, value):
yielded_value = self._generator.send(value)
# Intercept the first value yielded by the wrapped generator and
# replace it with a different value.
if not self._intercepted:
self._intercepted = True
print(f'Intercepted value: {yielded_value}')
yielded_value = 'intercepted'
return yielded_value
def throw(self, type, *args):
return self._generator.throw(type, *args)
def close(self):
self._generator.close()
__next__(), send(), throw(), close() are described in the Python Reference Manual.
The class wraps the generator passed to it when created will mimic its behavior. The only thing it changes is that the first value yielded by the generator is replaced by a different value before it is returned to the caller.
We can test the behavior with an example generator f() which yields two values and a function main() which sends values into the generator until the generator terminates:
def f():
y = yield 'first'
print(f'f(): {y}')
y = yield 'second'
print(f'f(): {y}')
def main():
value_to_send = 0
gen = f()
try:
x = gen.send(None)
while True:
print(f'main(): {x}')
# Send incrementing integers to the generator.
value_to_send += 1
x = gen.send(value_to_send)
except StopIteration:
print('main(): StopIteration')
main()
When ran, this example will produce the following output, showing which values arrive in the generator and which are returned by the generator:
main(): first
f(): 1
main(): second
f(): 2
main(): StopIteration
Wrapping the generator f() by changing the statement gen = f() to gen = Intercept(f()), produces the following output, showing that the first yielded value has been replaced:
Intercepted value: first
main(): intercepted
f(): 1
main(): second
f(): 2
As all other calls to any of the generator API are forwarded directly to the wrapped generator, it should behave equivalently to the wrapped generator itself.
If I understand the question, I think this works? Meaning, I ran this script and it did what I expected, which was to print all but the first line of the input file. But as long as the generator passed as the argument to the skip_first function can be iterator over, it should work.
def skip_first(thing):
_first = True
for _result in thing:
if _first:
_ first = False
continue
yield _result
inp = open("/var/tmp/test.txt")
for line in skip_first(inp):
print(line, end="")
I'm refreshing my memory about some python features that I didn't get yet, I'm learning from this python tutorial and there's an example that I don't fully understand. It's about a decorator counting calls to a function, here's the code:
def call_counter(func):
def helper(x):
helper.calls += 1
return func(x)
helper.calls = 0
return helper
#call_counter
def succ(x):
return x + 1
if __name__ == '__main__':
print(succ.calls)
for i in range(10):
print(succ(i))
print(succ.calls)
What I don't get here is why do we increment the calls of the function wrapper (helper.calls += 1) instead of the function calls itself, and why does it actually working?
The important thing to remember about decorators is that a decorator is a function that takes a function as an argument, and returns yet another function. The returned value - yet another function - is what will be called when the name of the original function is invoked.
This model can be very simple:
def my_decorator(fn):
print("Decorator was called")
return fn
In this case, the returned function is the same as the passed-in function. But that's usually not what you do. Usually, you return either a completely different function, or you return a function that somehow chains or wraps the original function.
In your example, which is a very common model, you have an inner function that is returned:
def helper(x):
helper.calls += 1
return func(x)
This inner function calls the original function (return func(x)) but it also increments the calls counter.
This inner function is being inserted as a "replacement" for whatever function is being decorated. So when your module foo.succ() function is looked up, the result is a reference to the inner helper function returned by the decorator. That function increments the call counter and then calls the originally-defined succ function.
When you decorate a function you "substitute" you're function with the wrapper.
In this example, after the decoration, when you call succ you are actually calling helper. So if you are counting calls you have to increase the helper calls.
You can check that once you decorate a function the name is binded tho the wrapper by checking the attribute _name_ of the decorated function:
def call_counter(func):
def helper(*args, **kwargs):
helper.calls += 1
print(helper.calls)
return func(*args, **kwargs)
helper.calls = 0
return helper
#call_counter
def succ(x):
return x + 1
succ(0)
>>> 1
succ(1)
>>> 2
print(succ.__name__)
>>> 'helper'
print(succ.calls)
>>> 2
Example with Class Decorator
When you decorate a function with the Class Decorator, every function has its own call_count. This is simplicity of OOP. Every time CallCountDecorator object is called, it will increase its own call_count attribute and print it.
class CallCountDecorator:
"""
A decorator that will count and print how many times the decorated function was called
"""
def __init__(self, inline_func):
self.call_count = 0
self.inline_func = inline_func
def __call__(self, *args, **kwargs):
self.call_count += 1
self._print_call_count()
return self.inline_func(*args, **kwargs)
def _print_call_count(self):
print(f"The {self.inline_func.__name__} called {self.call_count} times")
#CallCountDecorator
def function():
pass
#CallCountDecorator
def function2(a, b):
pass
if __name__ == "__main__":
function()
function2(1, b=2)
function()
function2(a=2, b=3)
function2(0, 1)
# OUTPUT
# --------------
# The function called 1 times
# The function2 called 1 times
# The function called 2 times
# The function2 called 2 times
# The function2 called 3 times
What I don't get here is why do we increment the calls of the function wrapper (helper.calls += 1) instead of the function calls itself, and why does it actually working?
I think to make it a generically useful decorator. You could do this
def succ(x):
succ.calls += 1
return x + 1
if __name__ == '__main__':
succ.calls = 0
print(succ.calls)
for i in range(10):
print(succ(i))
print(succ.calls)
which works just fine, but you would need to put the .calls +=1 in every function you wanted to apply this too, and initialise to 0 before you ran any of them. If you had a whole bunch of functions you wanted to count this is definitely nicer. Plus it initialises them to 0 at definition, which is nice.
As i understand it it works because it replaces the function succ with the helper function from within the decorator (which is redefined every time it decorates a function) so succ = helper and succ.calls = helper.calls. (although of course the name helper is only definied within the namespace of the decorator)
Does that make sense?
As I understand this (correct me if I'm wrong) the order you program executes is:
Register call_function.
Register succ.
While registering succ function interpreter finds a decorator so it executes call_function.
Your function returns an object which is a function (helper). And adds to this object field calls.
Now your function succ has been assigned to helper. So when you call your function, you're actually calling helper function, wrapped within a decorator. So every field you add to your helper function is accessible outside by addressing succ because those 2 variables refer to same thing.
So when you call succ() it's basically the same if you would do helper(*args, **argv)
Check this out:
def helper(x):
helper.calls += 1
return 2
helper.calls = 0
def call_counter(func):
return helper
#call_counter
def succ(x):
return x + 1
if __name__ == '__main__':
print(succ == helper) # prints true.
I'm kind of a new to python and i'm trying to run a code for a program, but i'm stuck in this part where i need a Function that always keep returning new data, here's a part of the code that always keep returning a zeros!
import time
def forloop():
for i in range(0,10000):
return i
while True:
time.sleep(0.25)
print(forloop())
When you call forloop() and do return i inside it, it returns from the function, and the next time you call forloop() it will start from the beginning. What you want to use are generator functions
You can use generator functions for this (using yield statement) to yield the values instead of return .
Example -
def forloop():
for i in range(0,10000):
yield i
x = forloop()
while True:
try :
time.sleep(0.25)
print(next(x))
except StopIteration:
x = forloop()
next(x) throws StopIteration exception if the generator has been exhausted, in which case we catch that exception and recreate the generator.
Your function returns immediately, and each time you call it you start at 0 again.
You can do what you want with a regular function; use a global and keep incrementing that:
_counter = -1
def increasing():
global _counter
_counter += 1
return counter
but a better idea would be to use a generator function:
def increasing():
counter = 0
while True:
yield counter
counter += 1
and you can use that in a loop:
for count in increasing():
print(count)
time.sleep(0.25)
The standard library already includes such a generator: the itertools.count() function does just that. Together with the next() function you can pretty much recreate your while loop:
from itertools import count
counter = count()
while True:
time.sleep(0.25)
print(next(counter))
If you wanted to continually loop over the the values 0 through to 9999, then you'd write a custom generator function that does just that:
def count_to_9999_and_restart():
while True:
for i in range(10000):
yield i
and you can use that generator:
counter = count_to_9999_and_restart()
while True:
time.sleep(0.25)
print(next(counter))
Try using a global variable to keep track of where the function reached inbetween calls:
counter = 0
def forloop():
global counter
counter += 1
return counter
I want to create functions and add them to a list, reusing the same name every time.
def fconstruct():
flist = []
for x in xrange(0,5):
def kol():
return x
flist.append(kol)
del kol #this doesn't fix the problem.
return flist
k = fconstruct()
However, this fails, even if i delete the function every loop, and no matter which of the functions in k i call, the result is the the same: 4, because the newest definition of kol has changed all the previous ones. For a simple function such as this,
kol = lambda: x
would work. However, i need to do this for a much more complex function
As solutions, i could store the function as a string in the list and use exec to call it.
I could generate disposable and random function names:
fname = '_'+str(random.random())
exec fname + ' = kol'
exec 'flist.append('+fname+')'
I could play around with this implementation of multiline lambdas: https://github.com/whaatt/Mu
None of these seem elegant, so what is the preferred way of doing this?
You have to use another function that generates the function you want with the x parameter set. Here I use the kol_factory (see also the answer to Closures in Python):
def fconstruct():
flist = []
# create a function g with the closure x
def kol_factory(y):
# y is local here
def kol():
# kol uses the y
return y
return kol
for x in xrange(0,5):
# we create a new g with x "burned-in"
flist.append(kol_factory(x))
return flist
for k in fconstruct():
print k()
You can define the factory function factory outside the fconstruct function:
def kol_factory(y):
# y is local here
def kol():
# kol uses the y
return y
return kol
def fconstruct():
flist = []
for x in xrange(0,5):
# we create a new kol_factory with x "burned-in"
flist.append(kol_factory(x))
return flist
for k in fconstruct():
print k()
When you're defining kol, you're establishing a closure around x. In fact, each time through the loop you're getting a closure around the same variable.
So while you have 5 different functions all named kol, they all return the value of the same variable. When the loop is finished, that variable's value is 4, so each function returns 4.
Consider this:
def fconstruct():
flist = []
for x in range(5):
def get_kol(y):
def kol():
return y
return kol
flist.append(get_kol(x))
return flist
In this case, the function get_kol() returns a function who's return value is get_kol()'s argument.
The closure in kol() is now around y, which is local to the get_kol() function, not the loop. Each time get_kol() is called, a new local y is created, so each kol() gets a different variable closure.
An alternative way is to create a partial function with functools.partial. This accomplishes the same thing (creates a function which, when called executes another function with arguments), and is a lot more powerful
def f(a): # whatever arguments you like
return a
# create a bunch of functions that return f(x)
flist = [functools.partial(f, x) for x in range(5)]
def g(a, b):
return a + b
# create a bunch of functions that take a single argument (b), but have a set to
# 0..4:
flist = [functools.partial(g, x) for x in range(5)]
some_g = flist[0]
some_g(1) # returns 0 + 1
I'm trying to implement a recursive Fibonacci series which returns the value at an index. It's a homework and needs to be done using multi-threading. This is what I've done so far. My question is how do I add the results from live_thread1 and live_thread2. The threads have to be created at every level in the recursion.
def Recursive(n):
if n< 2:
return n
else:
return Recursive(n- 1) + Recursive(n- 2)
def FibonacciThreads(n):
if n< 2:
return n
else:
thread1 = threading.Thread(target=FibonacciThreads,args=(n-1,))
thread2 = threading.Thread(target=FibonacciThreads,args=(n-2,))
thread1.start()
thread2.start()
thread1.join()
thread2.join()
return live_thread1+live_thread2
This is not possible, because you cannot retrieve the return-value of a function executed in another thread.
To implement the behavior you want, you have to make FibonacciThreads a callable object that stores the result as a member variable:
class FibonacciThreads(object):
def __init__(self):
self.result = None
def __call__(self, n):
# implement logic here as above
# instead of a return, store the result in self.result
You can use instances of this class like functions:
fib = FibonacciThreads() # create instance
fib(23) # calculate the number
print fib.result # retrieve the result
Note that, as I said in my comment, this is not a very smart use of threads. If this really is your assignment, it is a bad one.
You can pass a mutable object to the thread to use for storing the result. If you don't want to introduce a new data type, you can for example just use a single element list:
def fib(n, r):
if n < 2:
r[0] = n
else:
r1 = [None]
r2 = [None]
# Start fib() threads that use r1 and r2 for results.
...
# Sum the results of the threads.
r[0] = r1[0] + r2[0]
def FibonacciThreads(n):
r = [None]
fib(n, r)
return r[0]