I am reading the python asyncio library. The gather function confused me. Usually, we use it in the following way.
async def func(n):
await asyncio.sleep(n)
return n
result = await asyncio.gather(func(1), func(2)) # should be [1, 2]
It looks so naturally. There are two coroutines and result will be the list of their returns. However, I am confused by the implementation of asyncio.gather. See code. It returns a _GatheringFuture.
outer = _GatheringFuture(children, loop=loop)
return outer
The _GetheringFuture class is not like Task. It does not contain logic to call Future.set_result() or Future.set_execption(), so when we write await asyncio.gather(...), who sets the result/exception?
Ah. I see where set_result is called. It is inside def _done_callback(fut). So each coroutine in the input will execute this callback. The last finished one will set result of the outer future, i.e., _GatheringFuture.
Related
As mentioned in the title, what is the asyncio equivalent of Twisted's defer.succeed? Alternatively how do I create a coroutine object from a plain Python value?
Note that this is different from How do we call a normal function where a coroutine is expected?. I could always do something like the following but I was wondering if there was a better way.
value_to_convert_to_coroutine_object = 5
async def foo(bar):
return await asyncio.coroutine(lambda: value_to_convert_to_coroutine_object)()
defer.succeed() is nothing fancy and, as far as I know, there's not single asyncio function that replicates that.
def succeed(result):
f = Future()
f.set_result(result)
f.done()
return f
I recall on an asyncio IRC, that the authors didn't want to make functions that the user could simply write out on their own. Which might be why such a function doesn't exist.
As for the second part of your question, it seems like the solution you have is as simple as it gets. I've done something similar for unit tests.
The following asynchronous generator code is taked straight out of PEP525:
async def gen():
await asyncio.sleep(0.1)
v = yield 42
print(v)
await asyncio.sleep(0.2)
However when I call (with python3.6):
print(asyncio.iscoroutinefunction(gen), asyncio.iscoroutine(gen))
I get:
False, False
Why does asynchronous generator fails to identify as a coroutine function?
Is there another way to identify it as a coroutine function?
You want to use inspect.isasyncgenfunction() (and inspect.isasyncgen() for the result of calling gen()):
>>> import inspect
>>> print(inspect.isasyncgenfunction(gen), inspect.isasyncgen(gen()))
True True
There is no type hierarchy relationship between async functions and async generator functions.
Also, the only reason the asyncio.iscoroutine*() functions exist at all is to support the legacy generator-based #asyncio.coroutine decorator, which can't be used to create async generators. If you don't need to support older codebases that might still use those (so predating Python 3.5) I'd just stick with the inspect.is*() functions.
I have observed that the asyncio.run_coroutine_threadsafe function does not accept general awaitable objects, and I do not understand the reason for this restriction. Observe
import asyncio
async def native_coro():
return
#asyncio.coroutine
def generator_based_coro():
return
class Awaitable:
def __await__(self):
return asyncio.Future()
loop = asyncio.get_event_loop()
asyncio.run_coroutine_threadsafe(native_coro(), loop)
asyncio.run_coroutine_threadsafe(generator_based_coro(), loop)
asyncio.run_coroutine_threadsafe(Awaitable(), loop)
Running this with Python 3.6.6 yields
Traceback (most recent call last):
File "awaitable.py", line 24, in <module>
asyncio.run_coroutine_threadsafe(Awaitable(), loop)
File "~/.local/python3.6/lib/python3.6/asyncio/tasks.py", line 714, in run_coroutine_threadsafe
raise TypeError('A coroutine object is required')
TypeError: A coroutine object is required
where line 24 is asyncio.run_coroutine_threadsafe(Awaitable(), loop).
I know I can wrap my awaitable object in a coroutine defined like
awaitable = Awaitable()
async def wrapper():
return await awaitable
asyncio.run_coroutine_threadsafe(wrapper(), loop)
however my expectation was that the awaitable would be a valid argument directly to run_coroutine_threadsafe.
My questions are:
What is the reason for this restriction?
Is the wrapper function defined above the most conventional way to pass an awaitable to run_coroutine_threadsafe and other APIs that demand an async def or generator-defined coroutine?
What is the reason for this restriction?
Looking at the implementation, the reason is certainly not technical. Since the code already invokes ensure_future (rather than, say, create_task), it would automatically work, and work correctly, on any awaitable object.
The reason for the restriction can be found on the tracker. The function was added in 2015 as a result of a pull request. In the discussion on the related bpo issue the submitter explicitly requests the function be renamed to ensure_future_threadsafe (in parallel to ensure_future) and accept any kind of awaitable, a position seconded by Yury Selivanov. However, Guido was against the idea:
I'm against that idea. I don't really see a great important future for this method either way: It's just a little bit of glue between the threaded and asyncio worlds, and people will learn how to use it by finding an example.
[...]
But honestly I don't want to encourage flipping back and forth between threads and event loops; I see it as a necessary evil. The name we currently have is fine from the POV of someone coding in the threaded world who wants to hand off something to the asyncio world.
Why would someone in the threaded world have an asyncio.future that they need to wait for? That sounds like they're mixing up the two worlds -- or they should be writing asyncio code instead of threaded code.
There are other comments in a similar vein, but the above pretty much sums up the argument.
Is the wrapper function defined above the most conventional way to pass an awaitable to run_coroutine_threadsafe and other APIs that demand an async def or generator-defined coroutine?
If you actually need a coroutine object, something like wrapper is certainly a straightforward and correct way to get one.
If the only reason you're creating the wrapper is to call run_coroutine_threadsafe, but you're not actually interested in the result or the concurrent.futures.Future returned by run_coroutine_threadsafe, you can avoid the wrapping by calling call_soon_threadsafe directly:
loop.call_soon_threadsafe(asyncio.ensure_future, awaitable)
Environment: cooperative RTOS in C and micropython virtual machine is one of the tasks.
To make the VM not block the other RTOS tasks, I insert RTOS_sleep() in vm.c:DISPATCH() so that after every bytecode is executed, the VM relinquishes control to the next RTOS task.
I created a uPy interface to asynchronously obtain data from a physical data bus - could be CAN, SPI, ethernet - using producer-consumer design pattern.
Usage in uPy:
can_q = CANbus.queue()
message = can_q.get()
The implementation in C is such that can_q.get() does NOT block the RTOS: it polls a C-queue and if message is not received, it calls RTOS_sleep() to give another task the chance to fill the queue. Things are synchronized because the C-queue is only updated by another RTOS task and RTOS tasks only switch when RTOS_sleep() is called i.e. cooperative
The C-implementation is basically:
// gives chance for c-queue to be filled by other RTOS task
while(c_queue_empty() == true) RTOS_sleep();
return c_queue_get_message();
Although the Python statement can_q.get() does not block the RTOS, it does block the uPy script.
I'd like to rewrite it so I can use it with async def i.e. coroutine and have it not block the uPy script.
Not sure of the syntax but something like this:
can_q = CANbus.queue()
message = await can_q.get()
QUESTION
How do I write a C-function so I can await on it?
I would prefer a CPython and micropython answer but I would accept a CPython-only answer.
Note: this answer covers CPython and the asyncio framework. The concepts, however, should apply to other Python implementations as well as other async frameworks.
How do I write a C-function so I can await on it?
The simplest way to write a C function whose result can be awaited is by having it return an already made awaitable object, such as an asyncio.Future. Before returning the Future, the code must arrange for the future's result to be set by some asynchronous mechanism. All of these coroutine-based approaches assume that your program is running under some event loop that knows how to schedule the coroutines.
But returning a future isn't always enough - maybe we'd like to define an object with an arbitrary number of suspension points. Returning a future suspends only once (if the returned future is not complete), resumes once the future is completed, and that's it. An awaitable object equivalent to an async def that contains more than one await cannot be implemented by returning a future, it has to implement a protocol that coroutines normally implement. This is somewhat like an iterator implementing a custom __next__ and be used instead of a generator.
Defining a custom awaitable
To define our own awaitable type, we can turn to PEP 492, which specifies exactly which objects can be passed to await. Other than Python functions defined with async def, user-defined types can make objects awaitable by defining the __await__ special method, which Python/C maps to the tp_as_async.am_await part of the PyTypeObject struct.
What this means is that in Python/C, you must do the following:
specify a non-NULL value for the tp_as_async field of your extension type.
have its am_await member point to a C function that accepts an instance of your type and returns an instance of another extension type that implements the iterator protocol, i.e. defines tp_iter (trivially defined as PyIter_Self) and tp_iternext.
the iterator's tp_iternext must advance the coroutine's state machine. Each non-exceptional return from tp_iternext corresponds to a suspension, and the final StopIteration exception signifies the final return from the coroutine. The return value is stored in the value property of StopIteration.
For the coroutine to be useful, it must also be able to communicate with the event loop that drives it, so that it can specify when it is to be resumed after it has suspended. Most of coroutines defined by asyncio expect to be running under the asyncio event loop, and internally use asyncio.get_event_loop() (and/or accept an explicit loop argument) to obtain its services.
Example coroutine
To illustrate what the Python/C code needs to implement, let's consider simple coroutine expressed as a Python async def, such as this equivalent of asyncio.sleep():
async def my_sleep(n):
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.call_later(n, future.set_result, None)
await future
# we get back here after the timeout has elapsed, and
# immediately return
my_sleep creates a Future, arranges for it to complete (its result to become set) in n seconds, and suspends itself until the future completes. The last part uses await, where await x means "allow x to decide whether we will now suspend or keep executing". An incomplete future always decides to suspend, and the asyncio Task coroutine driver special-cases yielded futures to suspend them indefinitely and connects their completion to resuming the task. Suspension mechanisms of other event loops (curio etc) can differ in details, but the underlying idea is the same: await is an optional suspension of execution.
__await__() that returns a generator
To translate this to C, we have to get rid of the magic async def function definition, as well as of the await suspension point. Removing the async def is fairly simple: the equivalent ordinary function simply needs to return an object that implements __await__:
def my_sleep(n):
return _MySleep(n)
class _MySleep:
def __init__(self, n):
self.n = n
def __await__(self):
return _MySleepIter(self.n)
The __await__ method of the _MySleep object returned by my_sleep() will be automatically called by the await operator to convert an awaitable object (anything passed to await) to an iterator. This iterator will be used to ask the awaited object whether it chooses to suspend or to provide a value. This is much like how the for o in x statement calls x.__iter__() to convert the iterable x to a concrete iterator.
When the returned iterator chooses to suspend, it simply needs to produce a value. The meaning of the value, if any, will be interpreted by the coroutine driver, typically part of an event loop. When the iterator chooses to stop executing and return from await, it needs to stop iterating. Using a generator as a convenience iterator implementation, _MySleepIter would look like this:
def _MySleepIter(n):
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.call_later(n, future.set_result, None)
# yield from future.__await__()
for x in future.__await__():
yield x
As await x maps to yield from x.__await__(), our generator must exhaust the iterator returned by future.__await__(). The iterator returned by Future.__await__ will yield if the future is incomplete, and return the future's result (which we here ignore, but yield from actually provides) otherwise.
__await__() that returns a custom iterator
The final obstacle for a C implementation of my_sleep in C is the use of generator for _MySleepIter. Fortunately, any generator can be translated to a stateful iterator whose __next__ executes the piece of code up to the next await or return. __next__ implements a state machine version of the generator code, where yield is expressed by returning a value, and return by raising StopIteration. For example:
class _MySleepIter:
def __init__(self, n):
self.n = n
self.state = 0
def __iter__(self): # an iterator has to define __iter__
return self
def __next__(self):
if self.state == 0:
loop = asyncio.get_event_loop()
self.future = loop.create_future()
loop.call_later(self.n, self.future.set_result, None)
self.state = 1
if self.state == 1:
if not self.future.done():
return next(iter(self.future))
self.state = 2
if self.state == 2:
raise StopIteration
raise AssertionError("invalid state")
Translation to C
The above is quite some typing, but it works, and only uses constructs that can be defined with native Python/C functions.
Actually translating the two classes to C quite straightforward, but beyond the scope of this answer.
I come from the land of Twisted/Klein. I come in peace and to ask for Tornado help. I'm investigating Tornado and how its take on async differs from Twisted. Twisted has something similar to gen.coroutine which is defer.inlineCallbacks and I'm able to write async code like this:
kleinsample.py
#app.route('/endpoint/<int:n>')
#defer.inlineCallbacks
def myRoute(request, n):
jsonlist = []
for i in range(n):
yield jsonlist.append({'id': i})
return json.dumps(jsonlist)
curl cmd:
curl localhost:9000/json/2000
This endpoint will create a JSON string with n number of elements. n can be small or very big. I'm able to break it up in Twisted such that the event loop won't block using yield. Now here's how I tried to convert this into Tornado:
tornadosample.py
async def get(self, n):
jsonlist = []
for i in range(n):
await gen.Task(jsonlist.append, {'id': i}) # exception here
self.write(json.dumps(jsonlist))
The traceback:
TypeError: append() takes no keyword arguments
I'm confused about what I'm supposed to do to properly iterate each element in the loop so that the event loop doesn't get blocked. Does anyone know the "Tornado" way of doing this?
You cannot and must not await append, since it isn't a coroutine and doesn't return a Future. If you want to occasionally yield to allow other coroutines to proceed using Tornado's event loop, await gen.moment.
from tornado import gen
async def get(self, n):
jsonlist = []
for i in range(n):
jsonlist.append({'id': i})
if not i % 1000: # Yield control for a moment every 1k ops
await gen.moment
return json.dumps(jsonlist)
That said, unless this function is extremely CPU-intensive and requires hundreds of milliseconds or more to complete, you're probably better off just doing all your computation at once instead of taking multiple trips through the event loop before your function returns.
list.append() returns None, so it's a little misleading that your Klein sample looks like it's yielding some object. This is equivalent to jsonlist.append(...); yield as two separate statements. The tornado equivalent would be to do await gen.moment in place of the bare yield.
Also note that in Tornado, handlers produce their responses by calling self.write(), not by returning values, so the return statement should be self.write(json.dumps(jsonlist)).
Let's have a look at gen.Task docs:
Adapts a callback-based asynchronous function for use in coroutines.
Takes a function (and optional additional arguments) and runs it with those arguments plus a callback keyword argument. The argument passed to the callback is returned as the result of the yield expression.
Since append doesn't accept a keyword argument it doesn't know what to do with that callback kwarg and spits that exception.
What you could do is wrap append with your own function that does accept a callback kwarg or the approach showed in this answer.