We Keep Coding

Run event loop until all tasks are blocked in python

I am writing code that has some long-running coroutines that interact with each other. These coroutines can be blocked on await until something external happens. I want to be able to drive these coroutines in a unittest. The regular way of doing await on the coroutine doesn't work, because I want to be able to intercept something in the middle of their operation. I would also prefer not to mess with the coroutine internals either, unless there is something generic/reusable that can be done.
Ideally I would want to run an event loop until all tasks are currently blocked. This should be fairly easy to tell in an event loop implementation. Once everything is blocked, the event loop yields back control, where I can assert some state about the coroutines, and poke them externally. Then I can resume the loop until it gets blocked again. This would allow for deterministic simulation of tasks in an event loop.
Minimal example of the desired API:
import asyncio
from asyncio import Event
# Imagine this is a complicated "main" with many coroutines.
# But event is some external "mockable" event
# that can be used to drive in unit tests
async def wait_on_event(event: Event):
print("Waiting on event")
await event.wait()
print("Done waiting on event")
def test_deterministic():
loop = asyncio.get_event_loop()
event = Event()
task = loop.create_task(wait_on_event(event))
run_until_blocked_or_complete(loop) # define this magic function
# Should print "Waiting on event"
# can make some test assertions here
event.set()
run_until_blocked_or_complete(loop)
# Should print "Done waiting on event"
Anything like that possible? Or would this require writing a custom event loop just for tests?
Additionally, I am currently on Python 3.9 (AWS runtime limitation). If it's not possible to do this in 3.9, what version would support this?

This question has puzzled me since I first read it, because it's almost do-able with standard asyncio functions. The key is Alexander's "magic" is_not_blocked method, which I give verbatim below (except for moving it to the outer indentation level). I also use his wait_on_event method, and his test_deterministic_loop function. I added some extra tests to show how to start and stop other tasks, and how to drive the event loop step-by-step until all tasks are finished.
Instead of his DeterministicLoop class, I use a function run_until_blocked that makes only standard asyncio function calls. The two lines of code:
loop.call_soon(loop.stop)
loop.run_forever()
are a convenient means of advancing the loop by exactly one cycle. And asyncio already provides a method for obtaining all the tasks that run within a given event loop, so there is no need to store them independently.
A comment on the Alexander's "magic" method: if you look at the comments in the asyncio.Task code, the "private" variable _fut_waiter is described as an important invariant. That's very unlikely to change in future versions. So I think it's quite safe in practice.
import asyncio
from typing import Optional, cast
def _is_not_blocked(task: asyncio.Task):
# pylint: disable-next=protected-access
wait_for = cast(Optional[asyncio.Future], task._fut_waiter) # type: ignore
if wait_for is None:
return True
return wait_for.done()
def run_until_blocked():
"""Runs steps of the event loop until all tasks are blocked."""
loop = asyncio.get_event_loop()
# Always run one step.
loop.call_soon(loop.stop)
loop.run_forever()
# Continue running until all tasks are blocked
while any(_is_not_blocked(task) for task in asyncio.all_tasks(loop)):
loop.call_soon(loop.stop)
loop.run_forever()
# This coroutine could spawn many others. Keeping it simple here
async def wait_on_event(event: asyncio.Event) -> int:
print("Waiting")
await event.wait()
print("Done")
return 42
def test_deterministic_loop():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
event = asyncio.Event()
task = loop.create_task(wait_on_event(event))
assert not task.done()
run_until_blocked()
print("Task done", task.done())
assert not task.done()
print("Tasks running", asyncio.all_tasks(loop))
assert asyncio.all_tasks(loop)
event.set()
# You can start and stop tasks
loop.run_until_complete(asyncio.sleep(2.0))
run_until_blocked()
print("Task done", task.done())
assert task.done()
print("Tasks running", asyncio.all_tasks(loop))
assert task.result() == 42
assert not asyncio.all_tasks(loop)
# If you create a task you must loop run_until_blocked until
# the task is done.
task2 = loop.create_task(asyncio.sleep(2.0))
assert not task2.done()
while not task2.done():
assert asyncio.all_tasks(loop)
run_until_blocked()
assert task2.done()
assert not asyncio.all_tasks(loop)
test_deterministic_loop()

Yes, you can achieve this by creating a custom event loop policy and using a mock event loop in your test. The basic idea is to create a loop that only runs until all the coroutines are blocked, then yield control back to the test code to perform any necessary assertions or external pokes, and then continue running the loop until everything is blocked again, and so on.
import asyncio
class DeterministicEventLoopPolicy(asyncio.DefaultEventLoopPolicy):
def new_event_loop(self):
loop = super().new_event_loop()
loop._blocked = set()
return loop
def get_event_loop(self):
loop = super().get_event_loop()
if not hasattr(loop, "_blocked"):
loop._blocked = set()
return loop
def _enter_task(self, task):
super()._enter_task(task)
if not task._source_traceback:
task._source_traceback = asyncio.Task.current_task().get_stack()
task._loop._blocked.add(task)
def _leave_task(self, task):
super()._leave_task(task)
task._loop._blocked.discard(task)
def run_until_blocked(self, coro):
loop = self.new_event_loop()
asyncio.set_event_loop(loop)
try:
task = loop.create_task(coro)
while loop._blocked:
loop.run_until_complete(asyncio.sleep(0))
finally:
task.cancel()
loop.run_until_complete(task)
asyncio.set_event_loop(None)
This policy creates a new event loop with a _blocked set attribute that tracks the tasks that are currently blocked. When a new task is scheduled on the loop, the _enter_task method is called, and we add it to the _blocked set. When a task is completed or canceled, the _leave_task method is called, and we remove it from the _blocked set.
The run_until_blocked method takes a coroutine and runs the event loop until all the tasks are blocked. It creates a new event loop using the custom policy, schedules the coroutine on the loop, and then repeatedly runs the loop until the _blocked set is empty. This is the point where you can perform any necessary assertions or external pokes.
Here's an example usage of this policy:
async def wait_on_event(event: asyncio.Event):
print("Waiting on event")
await event.wait()
print("Done waiting on event")
def test_deterministic():
asyncio.set_event_loop_policy(DeterministicEventLoopPolicy())
event = asyncio.Event()
asyncio.get_event_loop().run_until_blocked(wait_on_event(event))
assert not event.is_set() # assert that the event has not been set yet
event.set() # set the event
asyncio.get_event_loop().run_until_blocked(wait_on_event(event))
assert event.is_set() # assert that the event has been set
asyncio.get_event_loop().close()
In this test, we create a new Event object and pass it to the wait_on_event coroutine. We use the run_until_blocked method to run the coroutine until it blocks on the event.wait() call. At this point, we can perform any necessary assertions, such as checking that the event has not been set yet. We then set the event, and call run_until_blocked again to resume the coroutine until it completes.
This pattern allows for deterministic simulation of tasks in an event loop and can be used to test coroutines that block on external events.
Hope this helps!

The default event loop simply runs everything that is scheduled in each "pass". If you simply schedule your pause with "loop.call_soon" after getting your tasks running, you should be called at the desired point:
import asyncio
async def worker(n=1):
await asyncio.sleep(n)
def event():
print("blah")
breakpoint()
print("bleh")
async def worker(id):
print(f"starting task {id}")
await asyncio.sleep(0.1)
print(f"ending task {id}")
async def main():
t = []
for id in (1,2,3):
t.append(asyncio.create_task(worker(id)))
loop = asyncio.get_running_loop()
loop.call_soon(event)
await asyncio.sleep(0.2)
And running this on the REPL:
In [8]: asyncio.run(main())
starting task 1
starting task 2
starting task 3
blah
> <ipython-input-3-450374919d79>(4)event()
-> print("bleh")
(Pdb)
Exception in callback event() at <ipython-input-3-450374919d79>:1
[...]
bdb.BdbQuit
ending task 1
ending task 2
ending task 3

After some experimenting I came up with something. Here is the usage first:
# This coroutine could spawn many others. Keeping it simple here
async def wait_on_event(event: asyncio.Event) -> int:
print("Waiting")
await event.wait()
print("Done")
return 42
def test_deterministic_loop():
loop = DeterministicLoop()
event = asyncio.Event()
task = loop.add_coro(wait_on_event(event))
assert not task.done()
loop.step()
# prints Waiting
assert not task.done()
assert not loop.done()
event.set()
loop.step()
# prints Done
assert task.done()
assert task.result() == 42
assert loop.done()
The implementation:
"""Module for testing facilities. Don't use these in production!"""
import asyncio
from enum import IntEnum
from typing import Any, Optional, TypeVar, cast
from collections.abc import Coroutine, Awaitable
def _get_other_tasks(loop: Optional[asyncio.AbstractEventLoop]) -> set[asyncio.Task]:
"""Get a set of currently scheduled tasks in an event loop that are not the current task"""
current = asyncio.current_task(loop)
tasks = asyncio.all_tasks(loop)
if current is not None:
tasks.discard(current)
return tasks
# Works on python 3.9, cannot guarantee on other versions
def _get_unblocked_tasks(tasks: set[asyncio.Task]) -> set[asyncio.Task]:
"""Get the subset of tasks that can make progress. This is the most magic
function, and is heavily dependent on eventloop implementation and python version"""
def is_not_blocked(task: asyncio.Task):
# pylint: disable-next=protected-access
wait_for = cast(Optional[asyncio.Future], task._fut_waiter) # type: ignore
if wait_for is None:
return True
return wait_for.done()
return set(filter(is_not_blocked, tasks))
class TasksState(IntEnum):
RUNNING = 0
BLOCKED = 1
DONE = 2
def _get_tasks_state(
prev_tasks: set[asyncio.Task], cur_tasks: set[asyncio.Task]
) -> TasksState:
"""Given set of tasks for previous and current pass of the event loop,
determine the overall state of the tasks. Are the tasks making progress,
blocked, or done?"""
if not cur_tasks:
return TasksState.DONE
unblocked: set[asyncio.Task] = _get_unblocked_tasks(cur_tasks)
# check if there are tasks that can make progress
if unblocked:
return TasksState.RUNNING
# if no tasks appear to make progress, check if this and last step the state
# has been constant
elif prev_tasks == cur_tasks:
return TasksState.BLOCKED
return TasksState.RUNNING
async def _stop_when_blocked():
"""Schedule this task to stop the event loop when all other tasks are
blocked, or they all complete"""
prev_tasks: set[asyncio.Task] = set()
loop = asyncio.get_running_loop()
while True:
tasks = _get_other_tasks(loop)
state = _get_tasks_state(prev_tasks, tasks)
prev_tasks = tasks
# stop the event loop if all other tasks cannot make progress
if state == TasksState.BLOCKED:
loop.stop()
# finish this task too, if no other tasks exist
if state == TasksState.DONE:
break
# yield back to the event loop
await asyncio.sleep(0.0)
loop.stop()
T = TypeVar("T")
class DeterministicLoop:
"""An event loop for writing deterministic tests."""
def __init__(self):
self.loop = asyncio.get_event_loop_policy().new_event_loop()
asyncio.set_event_loop(self.loop)
self.stepper_task = self.loop.create_task(_stop_when_blocked())
self.tasks: list[asyncio.Task] = []
def add_coro(self, coro: Coroutine[Any, Any, T]) -> asyncio.Task[T]:
"""Add a coroutine to the set of running coroutines, so they can be stepped through"""
if self.done():
raise RuntimeError("No point in adding more tasks. All tasks have finished")
task = self.loop.create_task(coro)
self.tasks.append(task)
return task
def step(self, awaitable: Optional[Awaitable[T]] = None) -> Optional[T]:
if self.done() or not self.tasks:
raise RuntimeError(
"No point in stepping. No tasks to step or all are finished"
)
step_future: Optional[asyncio.Future[T]] = None
if awaitable is not None:
step_future = asyncio.ensure_future(awaitable, loop=self.loop)
# stepper_task should halt us if we're blocked or all tasks are done
self.loop.run_forever()
if step_future is not None:
assert (
step_future.done()
), "Can't step the event loop, where the step function itself might get blocked"
return step_future.result()
return None
def done(self) -> bool:
return self.stepper_task.done()

Here is a simple generalized implementation.
If the loop finds that all tasks are stuck in some async instruction (Event, Semaphore, whatever) for some constant amount of iterations it will exit the loop context until run_until_blocked is called once again.
import asyncio
MAX_LOOP_ITER = 100
def run_until_blocked(loop):
global _tasks
_tasks = {}
while True:
loop.call_soon(loop.stop)
loop.run_forever()
for task in asyncio.all_tasks(loop=loop):
if task.done():
continue
lasti = task.get_stack()[-1].f_lasti
if task in _tasks and _tasks[task]["lasti"] == lasti:
_tasks[task]["iter"] += 1
else:
_tasks[task] = {"iter": 0, "lasti": lasti}
if all(val["iter"] < MAX_LOOP_ITER for val in _tasks.values()):
break
async def wait_on_event(event: asyncio.Event):
print("Waiting on event")
await event.wait()
print("Done waiting on event")
return 42
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
event = asyncio.Event()
coro = wait_on_event(event)
task = loop.create_task(coro)
run_until_blocked(loop)
event.set()
run_until_blocked(loop)
print(task.result())

How to do a "real concurrent" corroutines

I'm trying to do this:
"An event loop runs in a thread (typically the main thread) and executes all callbacks and Tasks in its thread. While a Task is running in the event loop, no other Tasks can run in the same thread. When a Task executes an await expression, the running Task gets suspended, and the event loop executes the next Task."
https://docs.python.org/3/library/asyncio-dev.html#concurrency-and-multithreading
And I did this ugly example:
import asyncio
async def print_message(message):
print(message)
async def int_sum(a, b):
await print_message('start_sum')
result = a + b
await print_message('end_sum')
return result
async def int_mul(a, b):
await print_message('start_mul')
result = a * b
await print_message('end_mul')
return result
async def main():
result = await asyncio.gather(int_sum(4, 3), int_mul(4, 3))
print(result)
asyncio.run(main())
With "secuential-like" results:
$ python async_test.py
start_sum
end_sum
start_mul
end_mul
[7, 12]
But I want a "corroutine-like" output:
$ python async_test.py
start_sum
start_mul
end_sum
end_mul
[7, 12]
How can I do that?
Note: I'm not looking for a asyncio.sleep(n) example, I'm looking for
"When a Task executes an await expression, the running Task gets suspended, and the event loop executes the next Task".

The point is, Tasks only give the control back to the event loop with yield statement. In your example you already have three active tasks(add asyncio.all_tasks() in the first line of int_sum coroutine to confirm1) but for example int_sum is not cooperating. it doesn't give the control back to the event loop. why ? Because you don't have any yield.
A simple fix to this is to change your print_message to:
async def print_message(message):
print(message)
await asyncio.sleep(0)
if you see the source code of asyncio.sleep:
async def sleep(delay, result=None):
"""Coroutine that completes after a given time (in seconds)."""
if delay <= 0:
await __sleep0()
return result
...
And this is the body of the __sleep0()(right above the sleep):
#types.coroutine
def __sleep0():
"""Skip one event loop run cycle.
This is a private helper for 'asyncio.sleep()', used
when the 'delay' is set to 0. It uses a bare 'yield'
expression (which Task.__step knows how to handle)
instead of creating a Future object.
"""
yield
Now your output should be:
start_sum
start_mul
end_sum
end_mul
[7, 12]
1 Note: you do have three tasks, asyncio.gather does that for you:
If any awaitable in aws is a coroutine, it is automatically scheduled
as a Task.

How to use asyncio.run_coroutine_threadsafe correctly?

In short, the problem is that the future returned by asyncio.run_coroutine_threadsafe is blocking when I call future.result()
The problem is also documented in the following question with (currently) no satisfactory answer: Future from asyncio.run_coroutine_threadsafe hangs forever?
What I'm trying to achieve is to call async code from sync code, where the sync code is actually itself wrapped in async code with an existing running event loop (to make things more concrete: it's a Jupyter notebook).
I would want to send async tasks from nested sync code to the existing 'outer' event loop and 'await' its results within the nested sync code. Implied constraint: I do not want to run those tasks on a new event loop (multiple reasons).
Since it's not possible to just 'await' an async result from sync code without blocking and without using asyncio.run which creates a new event loop, I thought using a separate thread would somehow help.
From the documentation description, asyncio.run_coroutine_threadsafe sounds like the perfect candidate.
But it's still blocking...
Bellow full snippet, with a timeout when calling the future's result.
How can I get this code to work correctly?
import asyncio
from concurrent.futures import ThreadPoolExecutor
async def gather_coroutines(*coroutines):
return await asyncio.gather(*coroutines)
def run_th_safe(loop, coroutines):
future = asyncio.run_coroutine_threadsafe(gather_coroutines(*coroutines), loop)
res = future.result(timeout=3) # **** BLOCKING *****
return res
def async2sync(*coroutines):
try:
loop = asyncio.get_running_loop()
except RuntimeError:
return asyncio.run(gather_coroutines(*coroutines))
# BLOW DOESN'T WORK BECAUSE run_th_safe IS BLOCKING
with ThreadPoolExecutor(max_workers=1) as ex:
thread_future = ex.submit(run_th_safe, loop, coroutines)
return thread_future.result()
# Testing
async def some_async_task(n):
"""Some async function to test"""
print('Task running with n =', n)
await asyncio.sleep(n/10)
print('Inside coro', n)
return list(range(n))
async def main_async():
coro3 = some_async_task(30)
coro1 = some_async_task(10)
coro2 = some_async_task(20)
results = async2sync(coro3, coro1, coro2)
return results
def main_sync():
coro3 = some_async_task(30)
coro1 = some_async_task(10)
coro2 = some_async_task(20)
results = async2sync(coro3, coro1, coro2)
return results
if __name__ == '__main__':
# Testing functionnality with asyncio.run()
# This works
print(main_sync())
# Testing functionnality with outer-loop (asyncio.run) and nested asyncio.run_coroutine_threadsafe
# **DOESN'T WORK**
print(asyncio.run(main_async()))

How to run a coroutine and wait it result from a sync func when the loop is running?

I have a code like the foolowing:
def render():
loop = asyncio.get_event_loop()
async def test():
await asyncio.sleep(2)
print("hi")
return 200
if loop.is_running():
result = asyncio.ensure_future(test())
else:
result = loop.run_until_complete(test())
When the loop is not running is quite easy, just use loop.run_until_complete and it return the coro result but if the loop is already running (my blocking code running in app which is already running the loop) I cannot use loop.run_until_complete since it will raise an exception; when I call asyncio.ensure_future the task gets scheduled and run, but I want to wait there for the result, does anybody knows how to do this? Docs are not very clear how to do this.
I tried passing a concurrent.futures.Future calling set_result inside the coro and then calling Future.result() on my blocking code, but it doesn't work, it blocks there and do not let anything else to run. ANy help would be appreciated.

To implement runner with the proposed design, you would need a way to single-step the event loop from a callback running inside it. Asyncio explicitly forbids recursive event loops, so this approach is a dead end.
Given that constraint, you have two options:
make render() itself a coroutine;
execute render() (and its callers) in a thread different than the thread that runs the asyncio event loop.
Assuming #1 is out of the question, you can implement the #2 variant of render() like this:
def render():
loop = _event_loop # can't call get_event_loop()
async def test():
await asyncio.sleep(2)
print("hi")
return 200
future = asyncio.run_coroutine_threadsafe(test(), loop)
result = future.result()
Note that you cannot use asyncio.get_event_loop() in render because the event loop is not (and should not be) set for that thread. Instead, the code that spawns the runner thread must call asyncio.get_event_loop() and send it to the thread, or just leave it in a global variable or a shared structure.

Waiting Synchronously for an Asynchronous Coroutine
If an asyncio event loop is already running by calling loop.run_forever, it will block the executing thread until loop.stop is called [see the docs]. Therefore, the only way for a synchronous wait is to run the event loop on a dedicated thread, schedule the asynchronous function on the loop and wait for it synchronously from another thread.
For this I have composed my own minimal solution following the answer by user4815162342. I have also added the parts for cleaning up the loop when all work is finished [see loop.close].
The main function in the code below runs the event loop on a dedicated thread, schedules several tasks on the event loop, plus the task the result of which is to be awaited synchronously. The synchronous wait will block until the desired result is ready. Finally, the loop is closed and cleaned up gracefully along with its thread.
The dedicated thread and the functions stop_loop, run_forever_safe, and await_sync can be encapsulated in a module or a class.
For thread-safery considerations, see section “Concurrency and Multithreading” in asyncio docs.
import asyncio
import threading
#----------------------------------------
def stop_loop(loop):
''' stops an event loop '''
loop.stop()
print (".: LOOP STOPPED:", loop.is_running())
def run_forever_safe(loop):
''' run a loop for ever and clean up after being stopped '''
loop.run_forever()
# NOTE: loop.run_forever returns after calling loop.stop
#-- cancell all tasks and close the loop gracefully
print(".: CLOSING LOOP...")
# source: <https://xinhuang.github.io/posts/2017-07-31-common-mistakes-using-python3-asyncio.html>
loop_tasks_all = asyncio.Task.all_tasks(loop=loop)
for task in loop_tasks_all: task.cancel()
# NOTE: `cancel` does not guarantee that the Task will be cancelled
for task in loop_tasks_all:
if not (task.done() or task.cancelled()):
try:
# wait for task cancellations
loop.run_until_complete(task)
except asyncio.CancelledError: pass
#END for
print(".: ALL TASKS CANCELLED.")
loop.close()
print(".: LOOP CLOSED:", loop.is_closed())
def await_sync(task):
''' synchronously waits for a task '''
while not task.done(): pass
print(".: AWAITED TASK DONE")
return task.result()
#----------------------------------------
async def asyncTask(loop, k):
''' asynchronous task '''
print("--start async task %s" % k)
await asyncio.sleep(3, loop=loop)
print("--end async task %s." % k)
key = "KEY#%s" % k
return key
def main():
loop = asyncio.new_event_loop() # construct a new event loop
#-- closures for running and stopping the event-loop
run_loop_forever = lambda: run_forever_safe(loop)
close_loop_safe = lambda: loop.call_soon_threadsafe(stop_loop, loop)
#-- make dedicated thread for running the event loop
thread = threading.Thread(target=run_loop_forever)
#-- add some tasks along with my particular task
myTask = asyncio.run_coroutine_threadsafe(asyncTask(loop, 100200300), loop=loop)
otherTasks = [asyncio.run_coroutine_threadsafe(asyncTask(loop, i), loop=loop)
for i in range(1, 10)]
#-- begin the thread to run the event-loop
print(".: EVENT-LOOP THREAD START")
thread.start()
#-- _synchronously_ wait for the result of my task
result = await_sync(myTask) # blocks until task is done
print("* final result of my task:", result)
#... do lots of work ...
print("*** ALL WORK DONE ***")
#========================================
# close the loop gracefully when everything is finished
close_loop_safe()
thread.join()
#----------------------------------------
main()

here is my case, my whole programe is async, but call some sync lib, then callback to my async func.
follow the answer by user4815162342.
import asyncio
async def asyncTask(k):
''' asynchronous task '''
print("--start async task %s" % k)
# await asyncio.sleep(3, loop=loop)
await asyncio.sleep(3)
print("--end async task %s." % k)
key = "KEY#%s" % k
return key
def my_callback():
print("here i want to call my async func!")
future = asyncio.run_coroutine_threadsafe(asyncTask(1), LOOP)
return future.result()
def sync_third_lib(cb):
print("here will call back to your code...")
cb()
async def main():
print("main start...")
print("call sync third lib ...")
await asyncio.to_thread(sync_third_lib, my_callback)
# await loop.run_in_executor(None, func=sync_third_lib)
print("another work...keep async...")
await asyncio.sleep(2)
print("done!")
LOOP = asyncio.get_event_loop()
LOOP.run_until_complete(main())

Conditional in a coroutine based on whether it was called again?

I am trying to translate this key "debouncing" logic from Javascript to Python.
function handle_key(key) {
if (this.state == null) {
this.state = ''
}
this.state += key
clearTimeout(this.timeout)
this.timeout = setTimeout(() => {
console.log(this.state)
}, 500)
}
handle_key('a')
handle_key('b')
The idea is that subsequent key presses extend the timeout. The Javascript version prints:
ab
I don't want to translate the JS timeout functions, I'd rather have idiomatic Python using asyncio. My attempt in Python (3.5) is below, but it doesn't work as global_state is not actually updated when I expect.
import asyncio
global_state = ''
#asyncio.coroutine
def handle_key(key):
global global_state
global_state += key
local_state = global_state
yield from asyncio.sleep(0.5)
#if another call hasn't modified global_state we print it
if local_state == global_state:
print(global_state)
#asyncio.coroutine
def main():
yield from handle_key('a')
yield from handle_key('b')
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(main())
It prints:
a
ab
I have looked into asyncio Event, Queue and Condition but it isn't clear to me how to use them for this. How would you implement the desired behavior using Python's asyncio?
EDIT
Some more details on how I'd like to use handle_keys. I have an async function that checks for key presses.
#asyncio.coroutine
def check_keys():
keys = driver.get_keys()
for key in keys:
yield from handle_key(key)
Which in turn is scheduled along with other program tasks
#asyncio.coroutine
def main():
while True:
yield from check_keys()
yield from do_other_stuff()
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(main())
Qeek's use of asyncio.create_task and asyncio.gather makes sense. But how would I use it within a loop like this? Or is there another way to schedule the async tasks that would allow handle_keys calls to "overlap"?
Actual code on GitHub if you are interested.

Why your code doesn't work now?
Both handle_key javascript functions don't block execution. Each just clear timeout callback and set new one. It happens immediately.
Coroutines work another way: using yield from or newer syntax await on coroutine means that we want to resume execution flow only after this coroutine if fully done:
async def a():
await asyncio.sleep(1)
async def main():
await a()
await b() # this line would be reached only after a() done - after 1 second delay
asyncio.sleep(0.5) in your code - is not setting callback by timeout, but code that should be done before handle_key finsihed.
Let's try to make code work
You can create task to start execution some coroutine "in background". You can also cancel task (just like you do with clearTimeout(this.timeout)) if you don't want it to be finished.
Python version that emulates your javascript snippet:
import asyncio
from contextlib import suppress
global_state = ''
timeout = None
async def handle_key(key):
global global_state, timeout
global_state += key
# cancel previous callback (clearTimeout(this.timeout))
if timeout:
timeout.cancel()
with suppress(asyncio.CancelledError):
await timeout
# set new callback (this.timeout = setTimeout ...)
async def callback():
await asyncio.sleep(0.5)
print(global_state)
timeout = asyncio.ensure_future(callback())
async def main():
await handle_key('a')
await handle_key('b')
# both handle_key functions done, but task isn't finished yet
# you need to await for task before exit main() coroutine and close loop
if timeout:
await timeout
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
finally:
loop.close()
Idiomatic?
While code above works, it is not how asyncio should be used. Your javascript code based on callbacks, while asyncio usually is about to avoid using of callbacks.
It's hard to demonstrate difference on your example since it's callback based by nature (key handling - is some sort of global callback) and doesn't have more async logic. But this understanding would be important later when you'll add more async operations.
Right now I advice you to read about async/await in modern javascript (it's similar to Python's async/await) and look at examples comparing it to callbacks/promises. This article looks good.
It'll help you understand how you can use coroutine-based approach in Python.
Upd:
Since buttons.check needs to periodically call driver.get_buttons() you'll have to use loop. But it can be done as task along with your event loop.
If you had some sort of button_handler(callback) (this is usually how different libs allow to handle user input) you could use it to set some asyncio.Future directly and avoid loop.
Consider possibility write some little gui app with asyncio from the beginning. I think it may help you to better understand how you can adapt your existing project.
Here's some pseudo-code that shows background task to handle
buttons and using asyncio to handle some simple UI events/states logic:
.
import asyncio
from contextlib import suppress
# GUI logic:
async def main():
while True:
print('We at main window, popup closed')
key = await key_pressed
if key == 'Enter':
print('Enter - open some popup')
await popup()
# this place wouldn't be reached until popup is not closed
print('Popup was closed')
elif key == 'Esc':
print('Esc - exit program')
return
async def popup():
while True:
key = await key_pressed
if key == 'Esc':
print('Esc inside popup, let us close it')
return
else:
print('Non escape key inside popup, play sound')
# Event loop logic:
async def button_check():
# Where 'key_pressed' is some global asyncio.Future
# that can be used by your coroutines to know some key is pressed
while True:
global key_pressed
for key in get_buttons():
key_pressed.set_result(key)
key_pressed = asyncio.Future()
await asyncio.sleep(0.01)
def run_my_loop(coro):
loop = asyncio.get_event_loop()
# Run background task to process input
buttons_task = asyncio.ensure_future(button_check())
try:
loop.run_until_complete(main())
finally:
# Shutdown task
buttons_task.cancel()
with suppress(asyncio.CancelledError):
loop.run_until_complete(buttons_task)
loop.close()
if __name__ == '__main__':
run_my_loop(main())

What's wrong
Basically the yield from xy() is very similar to the normal function call. The difference between function call and yield from is that the function call immediately start processing called function. The yield from statement insert called coroutine into queue inside event loop and give control to event loop and it decide which coroutine in it's queue will be processed.
Here is the explanation of what you code does:
It adds the main into event loop's queue.
The event loop start processing coroutine in the queue.
The queue contains only the main coroutine so it starts that.
The code hits the yield from handle_key('a').
It adds the handle_key('a') in the event loop's queue.
The event loop now contains the main and handle_key('a') but the main cannot be started because it is waiting for the result of the handle_key('a').
So the event loop starts the handle_key('a').
It will do some stuff until it hits the yield from asyncio.sleep(0.5).
Now the event loop contains main(), handle_key('a') and sleep(0.5).
The main() is waiting for result from handle_key('a').
The handle_key('a') is waiting for result from sleep(0.5).
The sleep has no dependency so it can be started.
The asyncio.sleep(0.5) returns None after 0.5 second.
The event loop takes the None and return it into the handle_key('a') coroutine.
The return value is ignored because it isn't assign into anything
The handle_key('a') prints the key (because nothing change the state)
The handle_key coroutine at the end return None (because there isn't return statement).
The None is returned to the main.
Again the return value is ignored.
The code hits the yield from handle_key('b') and start processing new key.
It run same steps from step 5 (but with the key b).
How to fix it
The main coroutinr replace with this:
#asyncio.coroutine
def main(loop=asyncio.get_event_loop()):
a_task = loop.create_task(handle_key('a'))
b_task = loop.create_task(handle_key('b'))
yield from asyncio.gather(a_task, b_task)
The loop.create_task adds into the event loop's queue the handle_key('a') and handle_key('b') and then the yield from asyncio.gather(a_task, b_task) give control to the event loop. The event loop from this point contains handle_key('a'), handle_key('b'), gather(...) and main().
The main() wiating for result from gather()
The gather() waiting until all tasks given as parameters are finished
The handle_key('a') and handle_key('b') has no dependencies so they can be started.
The event loop now contains 2 coroutine which can start but which one will it pick? Well... who knows it is implementation depended. So for better simulation of pressed keys this one replace should be a little better:
#asyncio.coroutine
def main(loop=asyncio.get_event_loop()):
a_task = loop.create_task(handle_key('a'))
yield from asyncio.sleep(0.1)
b_task = loop.create_task(handle_key('b'))
yield from asyncio.gather(a_task, b_task)
Python 3.5 bonus
From the documentation:
Coroutines used with asyncio may be implemented using the async def statement.
The async def type of coroutine was added in Python 3.5, and is recommended if there is no need to support older Python versions.
It means that you can replace:
#asyncio.coroutine
def main():
with newer statement
async def main():
If you start using the new syntax then you have to also replace yield from with await.