I have two tasks in a consumer/producer relationship, separated by a asyncio.Queue. If the producer task fails, I'd like the consumer task to also fail as soon as possible, and not wait indefinitely on the queue. The consumer task can be created(spawned) independently from the producer task.
In general terms, I'd like to implement a dependency between two tasks, such that the failure of one is also the failure of the other, while keeping those two tasks concurrent(i.e. one will not await the other directly).
What kind of solutions(e.g. patterns) could be used here?
Basically, I'm thinking of erlang's "links".
I think it may be possible to implement something similar using callbacks, i.e. asyncio.Task.add_done_callback
Thanks!
From the comment:
The behavior I'm trying to avoid is the consumer being oblivious to the producer's death and waiting indefinitely on the queue. I want the consumer to be notified of the producer's death, and have a chance to react. or just fail, and that even while it's also waiting on the queue.
Other than the answer presented by Yigal, another way is to set up a third task that monitors the two and cancels one when the other one finishes. This can be generalized to any two tasks:
async def cancel_when_done(source, target):
assert isinstance(source, asyncio.Task)
assert isinstance(target, asyncio.Task)
try:
await source
except:
# SOURCE is a task which we expect to be awaited by someone else
pass
target.cancel()
Now when setting up the producer and the consumer, you can link them with the above function. For example:
async def producer(q):
for i in itertools.count():
await q.put(i)
await asyncio.sleep(.2)
if i == 7:
1/0
async def consumer(q):
while True:
val = await q.get()
print('got', val)
async def main():
loop = asyncio.get_event_loop()
queue = asyncio.Queue()
p = loop.create_task(producer(queue))
c = loop.create_task(consumer(queue))
loop.create_task(cancel_when_done(p, c))
await asyncio.gather(p, c)
asyncio.get_event_loop().run_until_complete(main())
One way would be to propagate the exception through the queue, combined with delegation of the work handling:
class ValidWorkLoad:
async def do_work(self, handler):
await handler(self)
class HellBrokeLoose:
def __init__(self, exception):
self._exception = exception
async def do_work(self, handler):
raise self._exception
async def worker(name, queue):
async def handler(work_load):
print(f'{name} handled')
while True:
next_work = await queue.get()
try:
await next_work.do_work(handler)
except Exception as e:
print(f'{name} caught exception: {type(e)}: {e}')
break
finally:
queue.task_done()
async def producer(name, queue):
i = 0
while True:
try:
# Produce some work, or fail while trying
new_work = ValidWorkLoad()
i += 1
if i % 3 == 0:
raise ValueError(i)
await queue.put(new_work)
print(f'{name} produced')
await asyncio.sleep(0) # Preempt just for the sake of the example
except Exception as e:
print('Exception occurred')
await queue.put(HellBrokeLoose(e))
break
loop = asyncio.get_event_loop()
queue = asyncio.Queue(loop=loop)
producer_coro = producer('Producer', queue)
consumer_coro = worker('Consumer', queue)
loop.run_until_complete(asyncio.gather(producer_coro, consumer_coro))
loop.close()
Which outputs:
Producer produced
Consumer handled
Producer produced
Consumer handled
Exception occurred
Consumer caught exception: <class 'ValueError'>: 3
Alternatively you could skip the delegation, and designate an item that signals the worker to stop. When catching an exception in the producer you put that designated item in the queue.
Another possible solution:
import asyncio
def link_tasks(t1: Union[asyncio.Task, asyncio.Future], t2: Union[asyncio.Task, asyncio.Future]):
"""
Link the fate of two asyncio tasks,
such that the failure or cancellation of one
triggers the cancellation of the other
"""
def done_callback(other: asyncio.Task, t: asyncio.Task):
# TODO: log cancellation due to link propagation
if t.cancelled():
other.cancel()
elif t.exception():
other.cancel()
t1.add_done_callback(functools.partial(done_callback, t2))
t2.add_done_callback(functools.partial(done_callback, t1))
This uses asyncio.Task.add_done_callback to register callbacks that will cancel the other task if either one fails or is cancelled.
Related
If i run this code, it will hang without throwing ZeroDivisionError.
If i move await asyncio.gather(*tasks, return_exceptions=True)
above await queue.join(), it will finally throw ZeroDivisionError and stop.
If i then comment out 1 / 0 and run, it will execute everything, but will hang in the end.
Now the question is, how can i achive both:
Being able to see unexpected exceptions as in the case 2 above, and...
Actually stop when all task are done in the Queue
.
import asyncio
import random
import time
async def worker(name, queue):
while True:
print('Get a "work item" out of the queue.')
sleep_for = await queue.get()
print('Sleep for the "sleep_for" seconds.')
await asyncio.sleep(sleep_for)
# Error on purpose
1 / 0
print('Notify the queue that the "work item" has been processed.')
queue.task_done()
print(f'{name} has slept for {sleep_for:.2f} seconds')
async def main():
print('Create a queue that we will use to store our "workload".')
queue = asyncio.Queue()
print('Generate random timings and put them into the queue.')
total_sleep_time = 0
for _ in range(20):
sleep_for = random.uniform(0.05, 1.0)
total_sleep_time += sleep_for
queue.put_nowait(sleep_for)
print('Create three worker tasks to process the queue concurrently.')
tasks = []
for i in range(3):
task = asyncio.create_task(worker(f'worker-{i}', queue))
tasks.append(task)
print('Wait until the queue is fully processed.')
started_at = time.monotonic()
print('Joining queue')
await queue.join()
total_slept_for = time.monotonic() - started_at
print('Cancel our worker tasks.')
for task in tasks:
task.cancel()
print('Wait until all worker tasks are cancelled.')
await asyncio.gather(*tasks, return_exceptions=True)
print('====')
print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds')
print(f'total expected sleep time: {total_sleep_time:.2f} seconds')
asyncio.run(main())
There are several ways to approach this, but the central idea is that in asyncio, unlike in classic threading, it is straightforward to await multiple things at once.
For example, you can await queue.join() and the worker tasks, whichever completes first. Since workers don't complete normally (you cancel them later), a worker completing means that it has raised.
# convert queue.join() to a full-fledged task, so we can test
# whether it's done
queue_complete = asyncio.create_task(queue.join())
# wait for the queue to complete or one of the workers to exit
await asyncio.wait([queue_complete, *tasks], return_when=asyncio.FIRST_COMPLETED)
if not queue_complete.done():
# If the queue hasn't completed, it means one of the workers has
# raised - find it and propagate the exception. You can also
# use t.exception() to get the exception object. Canceling other
# tasks is another possibility.
for t in tasks:
if t.done():
t.result() # this will raise
A workaround (but ugly) solution: add try-catch block inside async def worker(...):, this will catch any exception in the code and prevent a no-ending loop.
Follow the same code as the question:
import asyncio
import random
import time
async def worker(name, queue):
while True:
try:
...
1 / 0 # Error code
...
except Exception as e:
print(e) # Show error
finanlly:
queue.task_done() # Make sure to clear the task
async def main():
...
asyncio.run(main())
While reading:
https://docs.python.org/3/library/asyncio-task.html#asyncio.Task.cancel
it seems that catching CancelledError is used for two purposes.
One is potentially preventing your task from getting cancelled.
The other is determining that something has cancelled the task you are awaiting.
How to tell the difference?
async def cancel_me():
try:
await asyncio.sleep(3600)
except asyncio.CancelledError:
raise
finally:
print('cancel_me(): after sleep')
async def main():
task = asyncio.create_task(cancel_me())
await asyncio.sleep(1)
task.cancel()
try:
await task
except asyncio.CancelledError:
# HERE: How do I know if `task` has been cancelled, or I AM being cancelled?
print("main(): cancel_me is cancelled now")
How to tell the difference [between ourselves getting canceled and the task we're awaiting getting canceled]?
Asyncio doesn't make it easy to tell the difference. When an outer task awaits an inner task, it is delegating control to inner one's coroutine. As a result, canceling either task injects a CancelledError into the exact same place: the inner-most await inside the inner task. This is why you cannot tell the which of the two tasks was canceled originally.
However, it is possible to circumvent the issue by breaking the chain of awaits and connecting the tasks using a completion callback instead. Cancellation of the inner task is then intercepted and detected in the callback:
class ChildCancelled(asyncio.CancelledError):
pass
async def detect_cancel(task):
cont = asyncio.get_event_loop().create_future()
def on_done(_):
if task.cancelled():
cont.set_exception(ChildCancelled())
elif task.exception() is not None:
cont.set_exception(task.exception())
else:
cont.set_result(task.result())
task.add_done_callback(on_done)
await cont
This is functionally equivalent to await task, except it doesn't await the inner task directly; it awaits a dummy future whose result is set after task completes. At this point we can replace the CancelledError (which we know must have come from cancellation of the inner task) with the more specific ChildCancelled. On the other hand, if the outer task is cancelled, that will show up as a regular CancelledError at await cont, and will be propagated as usual.
Here is some test code:
import asyncio, sys
# async def detect_cancel defined as above
async def cancel_me():
print('cancel_me')
try:
await asyncio.sleep(3600)
finally:
print('cancel_me(): after sleep')
async def parent(task):
await asyncio.sleep(.001)
try:
await detect_cancel(task)
except ChildCancelled:
print("parent(): child is cancelled now")
raise
except asyncio.CancelledError:
print("parent(): I am cancelled")
raise
async def main():
loop = asyncio.get_event_loop()
child_task = loop.create_task(cancel_me())
parent_task = loop.create_task(parent(child_task))
await asyncio.sleep(.1) # give a chance to child to start running
if sys.argv[1] == 'parent':
parent_task.cancel()
else:
child_task.cancel()
await asyncio.sleep(.5)
asyncio.get_event_loop().run_until_complete(main())
Note that with this implementation, cancelling the outer task will not automatically cancel the inner one, but that may be easily changed with an explicit call to child.cancel(), either in parent, or in detect_cancel itself.
Asyncio uses a similar approach to implement asyncio.shield().
Context
First, let's consider the wider context:
caller() --> your_coro() --> callee()
You are in control of your coroutine, but not in control of callers and only in partial control of callees.
By default, cancellation is effectively "propagated" both by up and down the stack:
(1)
caller1() ------------------+ (2)
+--> callee()
caller2() --> your_coro() --+
(4) (3)
In this diagram, semantically and very loosely, if caller1() is actively cancelled, then callee() gets cancelled, and then your coroutine gets cancelled and then caller2() gets cancelled. Roughly same happens if caller2() is actively cancelled.
(callee() is shared, and thus not a plain coroutine, rather a Task or a Future)
What alternate behaviour might you want?
Shield
If you want callee() to continue even if caller2() is cancelled, shield it:
callee_f = asyncio.ensure_future(callee())
async def your_coro():
# I might die, but I won't take callee down with me
await asyncio.shield(callee_f)
Reverse shield
If you allow callee() to die, but want your coroutine to continue, convert the exception:
async def reverse_shield(awaitable):
try:
return await awaitable
except asyncio.CancelledError:
raise Exception("custom")
async def your_coro():
await reverse_shield(callee_f)
# handle custom exception
Shield yourself
This one is questionable — normally you should allow your caller to cancel your coroutine.
A notable exception is when if your caller is a framework, and it's not configurable.
def your_coro():
async def inner():
...
return asyncio.shield(inner())
I am working on a project that uses the ccxt async library which requires all resources used by a certain class to be released with an explicit call to the class's .close() coroutine. I want to exit the program with ctrl+c and await the close coroutine in the exception. However, it is never awaited.
The application consists of the modules harvesters, strategies, traders, broker, and main (plus config and such). The broker initiates the strategies specified for an exchange and executes them. The strategy initiates the associated harvester which collects the necessary data. It also analyses the data and spawns a trader when there is a profitable opportunity. The main module creates a broker for each exchange and runs it. I have tried to catch the exception at each of these levels, but the close routine is never awaited. I'd prefer to catch it in the main module in order to close all exchange instances.
Harvester
async def harvest(self):
if not self.routes:
self.routes = await self.get_routes()
for route in self.routes:
self.logger.info("Harvesting route {}".format(route))
await asyncio.sleep(self.exchange.rateLimit / 1000)
yield await self.harvest_route(route)
Strategy
async def execute(self):
async for route_dct in self.harvester.harvest():
self.logger.debug("Route dictionary: {}".format(route_dct))
await self.try_route(route_dct)
Broker
async def run(self):
for strategy in self.strategies:
self.strategies[strategy] = getattr(
strategies, strategy)(self.share, self.exchange, self.currency)
while True:
try:
await self.execute_strategies()
except KeyboardInterrupt:
await safe_exit(self.exchange)
Main
async def main():
await load_exchanges()
await load_markets()
brokers = [Broker(
share,
exchanges[id]["api"],
currency,
exchanges[id]["strategies"]
) for id in exchanges]
futures = [broker.run() for broker in brokers]
for future in asyncio.as_completed(futures):
executed = await future
return executed
if __name__ == "__main__":
status = asyncio.run(main())
sys.exit(status)
I had expected the close() coroutine to be awaited, but I still get an error from the library that I must explicitly call it. Where do I catch the exception so that all exchange instances are closed properly?
Somewhere in your code should be entry point, where event loop is started.
Usually it is one of functions below:
loop.run_until_complete(main())
loop.run_forever()
asyncio.run(main())
When ctrl+C happens KeyboardInterrupt can be catched at this line. When it happened to execute some finalizing coroutine you can run event loop again.
This little example shows idea:
import asyncio
async def main():
print('Started, press ctrl+C')
await asyncio.sleep(10)
async def close():
print('Finalazing...')
await asyncio.sleep(1)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
except KeyboardInterrupt:
loop.run_until_complete(close())
finally:
print('Program finished')
I'm confused about how to use asyncio.Queue for a particular producer-consumer pattern in which both the producer and consumer operate concurrently and independently.
First, consider this example, which closely follows that from the docs for asyncio.Queue:
import asyncio
import random
import time
async def worker(name, queue):
while True:
sleep_for = await queue.get()
await asyncio.sleep(sleep_for)
queue.task_done()
print(f'{name} has slept for {sleep_for:0.2f} seconds')
async def main(n):
queue = asyncio.Queue()
total_sleep_time = 0
for _ in range(20):
sleep_for = random.uniform(0.05, 1.0)
total_sleep_time += sleep_for
queue.put_nowait(sleep_for)
tasks = []
for i in range(n):
task = asyncio.create_task(worker(f'worker-{i}', queue))
tasks.append(task)
started_at = time.monotonic()
await queue.join()
total_slept_for = time.monotonic() - started_at
for task in tasks:
task.cancel()
# Wait until all worker tasks are cancelled.
await asyncio.gather(*tasks, return_exceptions=True)
print('====')
print(f'3 workers slept in parallel for {total_slept_for:.2f} seconds')
print(f'total expected sleep time: {total_sleep_time:.2f} seconds')
if __name__ == '__main__':
import sys
n = 3 if len(sys.argv) == 1 else sys.argv[1]
asyncio.run(main())
There is one finer detail about this script: the items are put into the queue synchronously, with queue.put_nowait(sleep_for) over a conventional for-loop.
My goal is to create a script that uses async def worker() (or consumer()) and async def producer(). Both should be scheduled to run concurrently. No one consumer coroutine is explicitly tied to or chained from a producer.
How can I modify the program above so that the producer(s) is its own coroutine that can be scheduled concurrently with the consumers/workers?
There is a second example from PYMOTW. It requires the producer to know the number of consumers ahead of time, and uses None as a signal to the consumer that production is done.
How can I modify the program above so that the producer(s) is its own coroutine that can be scheduled concurrently with the consumers/workers?
The example can be generalized without changing its essential logic:
Move the insertion loop to a separate producer coroutine.
Start the consumers in the background, letting them process the items as they are produced.
With the consumers running, start the producers and wait for them to finish producing items, as with await producer() or await gather(*producers), etc.
Once all producers are done, wait for consumers to process the remaining items with await queue.join().
Cancel the consumers, all of which are now idly waiting for the queue to deliver the next item, which will never arrive as we know the producers are done.
Here is an example implementing the above:
import asyncio, random
async def rnd_sleep(t):
# sleep for T seconds on average
await asyncio.sleep(t * random.random() * 2)
async def producer(queue):
while True:
# produce a token and send it to a consumer
token = random.random()
print(f'produced {token}')
if token < .05:
break
await queue.put(token)
await rnd_sleep(.1)
async def consumer(queue):
while True:
token = await queue.get()
# process the token received from a producer
await rnd_sleep(.3)
queue.task_done()
print(f'consumed {token}')
async def main():
queue = asyncio.Queue()
# fire up the both producers and consumers
producers = [asyncio.create_task(producer(queue))
for _ in range(3)]
consumers = [asyncio.create_task(consumer(queue))
for _ in range(10)]
# with both producers and consumers running, wait for
# the producers to finish
await asyncio.gather(*producers)
print('---- done producing')
# wait for the remaining tasks to be processed
await queue.join()
# cancel the consumers, which are now idle
for c in consumers:
c.cancel()
asyncio.run(main())
Note that in real-life producers and consumers, especially those that involve network access, you probably want to catch IO-related exceptions that occur during processing. If the exception is recoverable, as most network-related exceptions are, you can simply catch the exception and log the error. You should still invoke task_done() because otherwise queue.join() will hang due to an unprocessed item. If it makes sense to re-try processing the item, you can return it into the queue prior to calling task_done(). For example:
# like the above, but handling exceptions during processing:
async def consumer(queue):
while True:
token = await queue.get()
try:
# this uses aiohttp or whatever
await process(token)
except aiohttp.ClientError as e:
print(f"Error processing token {token}: {e}")
# If it makes sense, return the token to the queue to be
# processed again. (You can use a counter to avoid
# processing a faulty token infinitely.)
#await queue.put(token)
queue.task_done()
print(f'consumed {token}')
where
This is on Linux, Python 3.5.1.
what
I'm developing a monitor process with asyncio, whose tasks at various places await on asyncio.sleep calls of various durations.
There are points in time when I would like to be able to interrupt all said asyncio.sleep calls and let all tasks proceed normally, but I can't find how to do that. An example is for graceful shutdown of the monitor process.
how (failed assumption)
I thought that I could send an ALRM signal to that effect, but the process dies. I tried catching the ALRM signal with:
def sigalrm_sent(signum, frame):
tse.logger.info("got SIGALRM")
signal.signal(signal.SIGALRM, sigalrm_sent)
Then I get the log line about catching SIGALRM, but the asyncio.sleep calls are not interrupted.
how (kludge)
At this point, I replaced all asyncio.sleep calls with calls to this coroutine:
async def interruptible_sleep(seconds):
while seconds > 0 and not tse.stop_requested:
duration = min(seconds, tse.TIME_QUANTUM)
await asyncio.sleep(duration)
seconds -= duration
So I only have to pick a TIME_QUANTUM that is not too small and not too large either.
but
Is there a way to interrupt all running asyncio.sleep calls and I am missing it?
Interrupting all running calls of asyncio.sleep seems a bit dangerous since it can be used in other parts of the code, for other purposes. Instead I would make a dedicated sleep coroutine that keeps track of its running calls. It is then possible to interrupt them all by canceling the corresponding tasks:
def make_sleep():
async def sleep(delay, result=None, *, loop=None):
coro = asyncio.sleep(delay, result=result, loop=loop)
task = asyncio.ensure_future(coro)
sleep.tasks.add(task)
try:
return await task
except asyncio.CancelledError:
return result
finally:
sleep.tasks.remove(task)
sleep.tasks = set()
sleep.cancel_all = lambda: sum(task.cancel() for task in sleep.tasks)
return sleep
Example:
async def main(sleep, loop):
for i in range(10):
loop.create_task(sleep(i))
await sleep(3)
nb_cancelled = sleep.cancel_all()
await asyncio.wait(sleep.tasks)
return nb_cancelled
sleep = make_sleep()
loop = asyncio.get_event_loop()
result = loop.run_until_complete(main(sleep, loop))
print(result) # Print '6'
For debugging purposes, loop.time = lambda: float('inf') also works.
Based on Vincent's answer, I used the following class (every instance of the class can cancel all its running .sleep tasks, allowing better compartmentalization):
class Sleeper:
"Group sleep calls allowing instant cancellation of all"
def __init__(self, loop):
self.loop = loop
self.tasks = set()
async def sleep(self, delay, result=None):
coro = aio.sleep(delay, result=result, loop=self.loop)
task = aio.ensure_future(coro)
self.tasks.add(task)
try:
return await task
except aio.CancelledError:
return result
finally:
self.tasks.remove(task)
def cancel_all_helper(self):
"Cancel all pending sleep tasks"
cancelled = set()
for task in self.tasks:
if task.cancel():
cancelled.add(task)
return cancelled
async def cancel_all(self):
"Coroutine cancelling tasks"
cancelled = self.cancel_all_helper()
await aio.wait(self.tasks)
self.tasks -= cancelled
return len(cancelled)