I have observed that the asyncio.run_coroutine_threadsafe function does not accept general awaitable objects, and I do not understand the reason for this restriction. Observe
import asyncio
async def native_coro():
return
#asyncio.coroutine
def generator_based_coro():
return
class Awaitable:
def __await__(self):
return asyncio.Future()
loop = asyncio.get_event_loop()
asyncio.run_coroutine_threadsafe(native_coro(), loop)
asyncio.run_coroutine_threadsafe(generator_based_coro(), loop)
asyncio.run_coroutine_threadsafe(Awaitable(), loop)
Running this with Python 3.6.6 yields
Traceback (most recent call last):
File "awaitable.py", line 24, in <module>
asyncio.run_coroutine_threadsafe(Awaitable(), loop)
File "~/.local/python3.6/lib/python3.6/asyncio/tasks.py", line 714, in run_coroutine_threadsafe
raise TypeError('A coroutine object is required')
TypeError: A coroutine object is required
where line 24 is asyncio.run_coroutine_threadsafe(Awaitable(), loop).
I know I can wrap my awaitable object in a coroutine defined like
awaitable = Awaitable()
async def wrapper():
return await awaitable
asyncio.run_coroutine_threadsafe(wrapper(), loop)
however my expectation was that the awaitable would be a valid argument directly to run_coroutine_threadsafe.
My questions are:
What is the reason for this restriction?
Is the wrapper function defined above the most conventional way to pass an awaitable to run_coroutine_threadsafe and other APIs that demand an async def or generator-defined coroutine?
What is the reason for this restriction?
Looking at the implementation, the reason is certainly not technical. Since the code already invokes ensure_future (rather than, say, create_task), it would automatically work, and work correctly, on any awaitable object.
The reason for the restriction can be found on the tracker. The function was added in 2015 as a result of a pull request. In the discussion on the related bpo issue the submitter explicitly requests the function be renamed to ensure_future_threadsafe (in parallel to ensure_future) and accept any kind of awaitable, a position seconded by Yury Selivanov. However, Guido was against the idea:
I'm against that idea. I don't really see a great important future for this method either way: It's just a little bit of glue between the threaded and asyncio worlds, and people will learn how to use it by finding an example.
[...]
But honestly I don't want to encourage flipping back and forth between threads and event loops; I see it as a necessary evil. The name we currently have is fine from the POV of someone coding in the threaded world who wants to hand off something to the asyncio world.
Why would someone in the threaded world have an asyncio.future that they need to wait for? That sounds like they're mixing up the two worlds -- or they should be writing asyncio code instead of threaded code.
There are other comments in a similar vein, but the above pretty much sums up the argument.
Is the wrapper function defined above the most conventional way to pass an awaitable to run_coroutine_threadsafe and other APIs that demand an async def or generator-defined coroutine?
If you actually need a coroutine object, something like wrapper is certainly a straightforward and correct way to get one.
If the only reason you're creating the wrapper is to call run_coroutine_threadsafe, but you're not actually interested in the result or the concurrent.futures.Future returned by run_coroutine_threadsafe, you can avoid the wrapping by calling call_soon_threadsafe directly:
loop.call_soon_threadsafe(asyncio.ensure_future, awaitable)
Related
As mentioned in the title, what is the asyncio equivalent of Twisted's defer.succeed? Alternatively how do I create a coroutine object from a plain Python value?
Note that this is different from How do we call a normal function where a coroutine is expected?. I could always do something like the following but I was wondering if there was a better way.
value_to_convert_to_coroutine_object = 5
async def foo(bar):
return await asyncio.coroutine(lambda: value_to_convert_to_coroutine_object)()
defer.succeed() is nothing fancy and, as far as I know, there's not single asyncio function that replicates that.
def succeed(result):
f = Future()
f.set_result(result)
f.done()
return f
I recall on an asyncio IRC, that the authors didn't want to make functions that the user could simply write out on their own. Which might be why such a function doesn't exist.
As for the second part of your question, it seems like the solution you have is as simple as it gets. I've done something similar for unit tests.
Environment: cooperative RTOS in C and micropython virtual machine is one of the tasks.
To make the VM not block the other RTOS tasks, I insert RTOS_sleep() in vm.c:DISPATCH() so that after every bytecode is executed, the VM relinquishes control to the next RTOS task.
I created a uPy interface to asynchronously obtain data from a physical data bus - could be CAN, SPI, ethernet - using producer-consumer design pattern.
Usage in uPy:
can_q = CANbus.queue()
message = can_q.get()
The implementation in C is such that can_q.get() does NOT block the RTOS: it polls a C-queue and if message is not received, it calls RTOS_sleep() to give another task the chance to fill the queue. Things are synchronized because the C-queue is only updated by another RTOS task and RTOS tasks only switch when RTOS_sleep() is called i.e. cooperative
The C-implementation is basically:
// gives chance for c-queue to be filled by other RTOS task
while(c_queue_empty() == true) RTOS_sleep();
return c_queue_get_message();
Although the Python statement can_q.get() does not block the RTOS, it does block the uPy script.
I'd like to rewrite it so I can use it with async def i.e. coroutine and have it not block the uPy script.
Not sure of the syntax but something like this:
can_q = CANbus.queue()
message = await can_q.get()
QUESTION
How do I write a C-function so I can await on it?
I would prefer a CPython and micropython answer but I would accept a CPython-only answer.
Note: this answer covers CPython and the asyncio framework. The concepts, however, should apply to other Python implementations as well as other async frameworks.
How do I write a C-function so I can await on it?
The simplest way to write a C function whose result can be awaited is by having it return an already made awaitable object, such as an asyncio.Future. Before returning the Future, the code must arrange for the future's result to be set by some asynchronous mechanism. All of these coroutine-based approaches assume that your program is running under some event loop that knows how to schedule the coroutines.
But returning a future isn't always enough - maybe we'd like to define an object with an arbitrary number of suspension points. Returning a future suspends only once (if the returned future is not complete), resumes once the future is completed, and that's it. An awaitable object equivalent to an async def that contains more than one await cannot be implemented by returning a future, it has to implement a protocol that coroutines normally implement. This is somewhat like an iterator implementing a custom __next__ and be used instead of a generator.
Defining a custom awaitable
To define our own awaitable type, we can turn to PEP 492, which specifies exactly which objects can be passed to await. Other than Python functions defined with async def, user-defined types can make objects awaitable by defining the __await__ special method, which Python/C maps to the tp_as_async.am_await part of the PyTypeObject struct.
What this means is that in Python/C, you must do the following:
specify a non-NULL value for the tp_as_async field of your extension type.
have its am_await member point to a C function that accepts an instance of your type and returns an instance of another extension type that implements the iterator protocol, i.e. defines tp_iter (trivially defined as PyIter_Self) and tp_iternext.
the iterator's tp_iternext must advance the coroutine's state machine. Each non-exceptional return from tp_iternext corresponds to a suspension, and the final StopIteration exception signifies the final return from the coroutine. The return value is stored in the value property of StopIteration.
For the coroutine to be useful, it must also be able to communicate with the event loop that drives it, so that it can specify when it is to be resumed after it has suspended. Most of coroutines defined by asyncio expect to be running under the asyncio event loop, and internally use asyncio.get_event_loop() (and/or accept an explicit loop argument) to obtain its services.
Example coroutine
To illustrate what the Python/C code needs to implement, let's consider simple coroutine expressed as a Python async def, such as this equivalent of asyncio.sleep():
async def my_sleep(n):
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.call_later(n, future.set_result, None)
await future
# we get back here after the timeout has elapsed, and
# immediately return
my_sleep creates a Future, arranges for it to complete (its result to become set) in n seconds, and suspends itself until the future completes. The last part uses await, where await x means "allow x to decide whether we will now suspend or keep executing". An incomplete future always decides to suspend, and the asyncio Task coroutine driver special-cases yielded futures to suspend them indefinitely and connects their completion to resuming the task. Suspension mechanisms of other event loops (curio etc) can differ in details, but the underlying idea is the same: await is an optional suspension of execution.
__await__() that returns a generator
To translate this to C, we have to get rid of the magic async def function definition, as well as of the await suspension point. Removing the async def is fairly simple: the equivalent ordinary function simply needs to return an object that implements __await__:
def my_sleep(n):
return _MySleep(n)
class _MySleep:
def __init__(self, n):
self.n = n
def __await__(self):
return _MySleepIter(self.n)
The __await__ method of the _MySleep object returned by my_sleep() will be automatically called by the await operator to convert an awaitable object (anything passed to await) to an iterator. This iterator will be used to ask the awaited object whether it chooses to suspend or to provide a value. This is much like how the for o in x statement calls x.__iter__() to convert the iterable x to a concrete iterator.
When the returned iterator chooses to suspend, it simply needs to produce a value. The meaning of the value, if any, will be interpreted by the coroutine driver, typically part of an event loop. When the iterator chooses to stop executing and return from await, it needs to stop iterating. Using a generator as a convenience iterator implementation, _MySleepIter would look like this:
def _MySleepIter(n):
loop = asyncio.get_event_loop()
future = loop.create_future()
loop.call_later(n, future.set_result, None)
# yield from future.__await__()
for x in future.__await__():
yield x
As await x maps to yield from x.__await__(), our generator must exhaust the iterator returned by future.__await__(). The iterator returned by Future.__await__ will yield if the future is incomplete, and return the future's result (which we here ignore, but yield from actually provides) otherwise.
__await__() that returns a custom iterator
The final obstacle for a C implementation of my_sleep in C is the use of generator for _MySleepIter. Fortunately, any generator can be translated to a stateful iterator whose __next__ executes the piece of code up to the next await or return. __next__ implements a state machine version of the generator code, where yield is expressed by returning a value, and return by raising StopIteration. For example:
class _MySleepIter:
def __init__(self, n):
self.n = n
self.state = 0
def __iter__(self): # an iterator has to define __iter__
return self
def __next__(self):
if self.state == 0:
loop = asyncio.get_event_loop()
self.future = loop.create_future()
loop.call_later(self.n, self.future.set_result, None)
self.state = 1
if self.state == 1:
if not self.future.done():
return next(iter(self.future))
self.state = 2
if self.state == 2:
raise StopIteration
raise AssertionError("invalid state")
Translation to C
The above is quite some typing, but it works, and only uses constructs that can be defined with native Python/C functions.
Actually translating the two classes to C quite straightforward, but beyond the scope of this answer.
I'm a little bit confused by some asyncio functions. I see there is BaseEventLoop.create_task(coro) function to schedule a co-routine. The documentation for create_task says its a new function and for compatibility we should use asyncio.async(coro) which by referring to docs again I see is an alias for asyncio.ensure_future(coro) which again schedules the execution of a co-routine.
Meanwhile, I've been using Task(coro) for scheduling co-routine execution and that too seems to be working fine. so, what's the difference between all these?
As you've noticed, they all do the same thing.
asyncio.async had to be replaced with asyncio.ensure_future because in Python >= 3.5, async has been made a keyword[1].
create_task's raison d'etre[2]:
Third-party event loops can use their own subclass of Task for interoperability. In this case, the result type is a subclass of Task.
And this also means you should not create a Task directly, because different event loops might have different ways of creating a "Task".
Edit
Another important difference is that in addition to accepting coroutines, ensure_future also accepts any awaitable object; create_task on the other hand just accepts coroutines.
Future
A Future is an object that is supposed to have a result in the future.
Or more officially:
A Future is an awaitable object that represents an eventual result of an asynchronous operation.
Object is awaitable if it can be used in an await expression.
So, any Future can be await'ed.
Task
Docs:
Tasks are used to schedule coroutines concurrently.
When a coroutine is wrapped into a Task with functions like
asyncio.create_task() the coroutine is automatically scheduled to run
soon.
A Task actually is a Future:
A Future-like object that runs a Python coroutine.
Task is subclass of a Future.
asyncio.Taskinherits from Future all of its APIs except
Future.set_result() and Future.set_exception().
create_task(coro)
Docs:
Wraps the coro coroutine into a Task and schedules its execution.
Returns the Task object.
asyncio.ensure_future(obj, ...)
It actually ensures that obj is a Future. If it's not it creates a Task from obj. That's it.
Some conclusions
If obj already is a Future, ensure_future will return the same obj. In this case return value of a function may be a Future and not a Task - since Task is just a subclass of a Future.
With ensure_future you CAN do this:
# will create a Task
task = asyncio.ensure_future(coroutine())
# will ensure that it is a Task
task = asyncio.ensure_future(task)
or this:
# will create a Task
task = asyncio.create_task(coroutine())
# will ensure that it is a Task
task = asyncio.ensure_future(task)
So, you can call ensure_future on a Task.
But you CAN'T do the same with create_task:
# will create a Task
task = asyncio.create_task(coroutine())
# will raise an exception
task = asyncio.create_task(task)
TypeError: a coroutine was expected, got Task
Pay attention
According to docs:
create_task() is the preferred way for creating new Tasks.
Deprecated since version 3.10: Deprecation warning is emitted if obj
is not a Future-like object and loop is not specified and there is no
running event loop.
I'm just a newbie to Tornado but not Python.
I'm trying to write an async client for CouchDB(using couchdb package).
After 1 day research/Googling, I found every posts just simply using HttpAsyncClient for example and not telling why and how gen.coroutine works.
The source code just too complicated for me understand cause decorator after decorator HttpAsyncClient is a bad example to me...
Frankly, for ma it was only after reading source, that I (partially) understood the logic
It sort of works like this if you have decorated somefunction() with tornado.gen:
in statement yield somefunction() somefunction() is actually called
Since it's a wrapper, not your code is executed but tornado.coroutine.gen. It runs your code until the first yield in somefunction
If a Future(placehodler) is yielded(back to decorator code!), tornado says: "OK, when this future resolves, schedule a task(IOLoop callback) for IOLoop so that it calls some another_callback()".
To track how far the execution of your somefunction() is gone, Tornado maintains a special object called Runner. It remembers the last yield statement which blocked the execution of your somefunction() and is first run when decorator gen is executed.
After point 3., where Future was "registered", this Runner returns from its main run() method and decorator exits, returning Future of its own
When Future from point 3. is ready, it adds a task to IOLoop, which then calls another_callback. Latter is a special callback created by Tornado, shortly put, it run()s the same Runner as was running when in point 3. a newly resolved Future was yielded.
Runner uses .send() method to inject a value of newly resolved Future back to your somefuncion, which cause ait to be assigned to a variable (if any) in your function in statement like this:
a = yield some_other_coroutine_for_example_async_client_fetch()
OK, that's a gist, there are lots of deatails, some of which I canot wrap my head around, especially, exception handling, but HTH
The first answer covers a lot. I just want to let u know a simpler abstraction.
#Gen.coroutine
res = yield foo()
The coroutine idea is asyncly execute foo(), especially when foo takes a lot IO or network job.
The yield could fire up the foo() execution and transfer control out , say, to caller. And it leaves a Runner to register this foo() task as a future obj. Then when foo() successfully return a result, this is magic happen, the Runner will send the result back, in the form of yield statement execution result (tell the diff between value yield and yield statement execution result).
I'm trying to convert a synchronous library to use an internal asynchronous IO framework. I have several methods that look like this:
def foo:
....
sync_call_1() # synchronous blocking call
....
sync_call_2() # synchronous blocking call
....
return bar
For each of the synchronous functions (sync_call_*), I have written a corresponding async function that takes a a callback. E.g.
def async_call_1(callback=none):
# do the I/O
callback()
Now for the python newbie question -- whats the easiest way to translate the existing methods to use these new async methods instead? That is, the method foo() above needs to now be:
def async_foo(callback):
# Do the foo() stuff using async_call_*
callback()
One obvious choice is to pass a callback into each async method which effectively "resumes" the calling "foo" function, and then call the callback global at the very end of the method. However, that makes the code brittle, ugly and I would need to add a new callback for every call to an async_call_* method.
Is there an easy way to do that using a python idiom, such as a generator or coroutine?
UPDATE: take this with a grain of salt, as I'm out of touch with modern python async developments, including gevent and asyncio and don't actually have serious experience with async code.
There are 3 common approaches to thread-less async coding in Python:
Callbacks - ugly but workable, Twisted does this well.
Generators - nice but require all your code to follow the style.
Use Python implementation with real tasklets - Stackless (RIP) and greenlet.
Unfortunately, ideally the whole program should use one style, or things become complicated. If you are OK with your library exposing a fully synchronous interface, you are probably OK, but if you want several calls to your library to work in parallel, especially in parallel with other async code, then you need a common event "reactor" that can work with all the code.
So if you have (or expect the user to have) other async code in the application, adopting the same model is probably smart.
If you don't want to understand the whole mess, consider using bad old threads. They are also ugly, but work with everything else.
If you do want to understand how coroutines might help you - and how they might complicate you, David Beazley's "A Curious Course on Coroutines and Concurrency" is good stuff.
Greenlets might be actualy the cleanest way if you can use the extension. I don't have any experience with them, so can't say much.
There are several way for multiplexing tasks. We can't say what is the best for your case without deeper knowledge on what you are doing. Probably the most easiest/universal way is to use threads. Take a look at this question for some ideas.
You need to make function foo also async. How about this approach?
#make_async
def foo(somearg, callback):
# This function is now async. Expect a callback argument.
...
# change
# x = sync_call1(somearg, some_other_arg)
# to the following:
x = yield async_call1, somearg, some_other_arg
...
# same transformation again
y = yield async_call2, x
...
# change
# return bar
# to a callback call
callback(bar)
And make_async can be defined like this:
def make_async(f):
"""Decorator to convert sync function to async
using the above mentioned transformations"""
def g(*a, **kw):
async_call(f(*a, **kw))
return g
def async_call(it, value=None):
# This function is the core of async transformation.
try:
# send the current value to the iterator and
# expect function to call and args to pass to it
x = it.send(value)
except StopIteration:
return
func = x[0]
args = list(x[1:])
# define callback and append it to args
# (assuming that callback is always the last argument)
callback = lambda new_value: async_call(it, new_value)
args.append(callback)
func(*args)
CAUTION: I haven't tested this