We Keep Coding

asyncio add_signal_handler not catching sigint or sigterm

I'm having some trouble debugging an issue, I have an asyncio project and I would like it to shutdown gracefully.
import asyncio
import signal
async def clean_loop(signal, loop):
print("something")
tasks = [t for t in asyncio.all_tasks() if t is not
asyncio.current_task()]
[task.cancel() for task in tasks]
await asyncio.gather(*tasks, return_exceptions=True)
loop.stop()
def main():
loop = asyncio.get_event_loop()
signals = (signal.SIGTERM, signal.SIGINT)
for s in signals:
loop.add_signal_handler(s, lambda s=s: asyncio.create_task(clean_loop(s, loop)))
task = loop.create_task(run(some_code))
loop.call_later(60, task.cancel)
try:
loop.run_until_complete(task)
except asyncio.CancelledError:
pass
finally:
loop.close()
if __name__ == "__main__":
main()
When I run my code and send a KeyboardInterrupt or TERM signal from a different screen, nothing seems to happen and it doesn't look like clean_loop is being called.
EDIT: I think I was able to isolate the problem a bit more, the code that I'm running within some_code which has an infinte loop and contains another asyncio.gather(*tasks) within it, when I comment it out I am able to catch the signal and clean_loop runs. Could anyone explain why this conflict is happening?

If you're on a Unix-like systems,
"""Add a handler for a signal. UNIX only.
it should work fine. The only thing is clean_loop itself is a task that getting destroyed with loop.stop() and because of that you see:
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<clean_loop() running at ...> wait_for=<_GatheringFuture finished result=[CancelledError('')]>>
It shouldn't be important though because it's just there to cancel tasks.
I've made some slight changes that have nothing to do with the actual question:
import asyncio
import signal
async def run():
for i in range(4):
print(i)
await asyncio.sleep(2)
async def clean_loop(signal, loop):
print("-----------------------handling signal")
tasks = asyncio.all_tasks() - {asyncio.current_task()}
for task in tasks:
task.cancel()
print("--------------------------reached here")
await asyncio.gather(*tasks, return_exceptions=True)
loop.stop()
def main():
loop = asyncio.new_event_loop()
signals = (signal.SIGTERM, signal.SIGINT)
for s in signals:
loop.add_signal_handler(s, lambda s=s: asyncio.create_task(clean_loop(s, loop)))
task = loop.create_task(run())
loop.call_later(60, task.cancel)
try:
loop.run_until_complete(task)
except asyncio.CancelledError:
pass
finally:
loop.close()
if __name__ == "__main__":
main()
output:
0
1
2
^C-----------------------handling signal
--------------------------reached here
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<clean_loop() running at ...> wait_for=<_GatheringFuture finished result=[CancelledError('')]>>
According to your edit:
Documentation of add_signal_handler says:
The callback will be invoked by loop, along with other queued
callbacks and runnable coroutines of that event loop.
Just like other coroutines it should be invoked by the loop in a cooperative way. In other words, a coroutine should give the control back to the event loop so that event loop can run another coroutine.
In your case, your coroutine which has infinite loop prevents event loop from running another coroutines especially the the callback which registered through add_signal_handler. It doesn't cooperate! That's why you though it didn't work. It sat idle in queue waiting for run.

python asyncio - cancelling a `to_thread` task won't stop the thread

With the following snippet, I can't figure why the infiniteTask is not cancelled (it keeps spamming "I'm still standing")
In debug mode, I can see that the Task stored in unfinished is indeed marked as Cancelled but obiously the thread is not cancelled / killed.
Why is the thread not killed when the wrapping task is cancelled ?
What should I do to stop the thread ?
import time
import asyncio
def quickTask():
time.sleep(1)
def infiniteTask():
while True:
time.sleep(1)
print("I'm still standing")
async def main():
finished, unfinished = await asyncio.wait({
asyncio.create_task(asyncio.to_thread(quickTask)),
asyncio.create_task(asyncio.to_thread(infiniteTask))
},
return_when = "FIRST_COMPLETED"
)
for task in unfinished:
task.cancel()
await asyncio.wait(unfinished)
print(" finished : " + str(len(finished))) # print '1'
print("unfinished : " + str(len(unfinished))) # print '1'
asyncio.run(main())

Cause
If we check the definition of asyncio.to_thread():
# python310/Lib/asyncio/threads.py
# ...
async def to_thread(func, /, *args, **kwargs):
"""Asynchronously run function *func* in a separate thread.
Any *args and **kwargs supplied for this function are directly passed
to *func*. Also, the current :class:`contextvars.Context` is propagated,
allowing context variables from the main thread to be accessed in the
separate thread.
Return a coroutine that can be awaited to get the eventual result of *func*.
"""
loop = events.get_running_loop()
ctx = contextvars.copy_context()
func_call = functools.partial(ctx.run, func, *args, **kwargs)
return await loop.run_in_executor(None, func_call)
It's actually a wrapper of loop.run_in_executor.
If we then go into how asyncio's test handle run_in_executor:
# python310/Lib/test/test_asyncio/threads.py
# ...
class EventLoopTestsMixin:
# ...
def test_run_in_executor_cancel(self):
called = False
def patched_call_soon(*args):
nonlocal called
called = True
def run():
time.sleep(0.05)
f2 = self.loop.run_in_executor(None, run)
f2.cancel()
self.loop.run_until_complete(
self.loop.shutdown_default_executor())
self.loop.close()
self.loop.call_soon = patched_call_soon
self.loop.call_soon_threadsafe = patched_call_soon
time.sleep(0.4)
self.assertFalse(called)
You can see it will wait for self.loop.shutdown_default_executor().
Now let's see how it looks like.
# event.pyi
# ...
class BaseEventLoop(events.AbstractEventLoop):
# ...
async def shutdown_default_executor(self):
"""Schedule the shutdown of the default executor."""
self._executor_shutdown_called = True
if self._default_executor is None:
return
future = self.create_future()
thread = threading.Thread(target=self._do_shutdown, args=(future,))
thread.start()
try:
await future
finally:
thread.join()
def _do_shutdown(self, future):
try:
self._default_executor.shutdown(wait=True)
self.call_soon_threadsafe(future.set_result, None)
except Exception as ex:
self.call_soon_threadsafe(future.set_exception, ex)
Here, we can see it creates another thread to wait for _do_shutdown, which then runs self._default_executor.shutdown with wait=True parameter.
Then where the shutdown is implemented:
# Python310/Lib/concurrent/futures/thread.py
# ...
class ThreadPoolExecutor(_base.Executor):
# ...
def shutdown(self, wait=True, *, cancel_futures=False):
with self._shutdown_lock:
self._shutdown = True
if cancel_futures:
# Drain all work items from the queue, and then cancel their
# associated futures.
while True:
try:
work_item = self._work_queue.get_nowait()
except queue.Empty:
break
if work_item is not None:
work_item.future.cancel()
# Send a wake-up to prevent threads calling
# _work_queue.get(block=True) from permanently blocking.
self._work_queue.put(None)
if wait:
for t in self._threads:
t.join()
When wait=True it decides to wait for all thread to be gracefully stops.
From all these we can't see any effort to actually cancel a thread.
To quote from Trio Documentation:
Cancellation is a tricky issue here, because neither Python nor the operating systems it runs on provide any general mechanism for cancelling an arbitrary synchronous function running in a thread. This function will always check for cancellation on entry, before starting the thread. But once the thread is running, there are two ways it can handle being cancelled:
If cancellable=False, the function ignores the cancellation and keeps going, just like if we had called sync_fn synchronously. This is the default behavior.
If cancellable=True, then this function immediately raises Cancelled. In this case the thread keeps running in background – we just abandon it to do whatever it’s going to do, and silently discard any return value or errors that it raises.
So, from these we can learn that there's no way to terminate infinite-loop running in thread.
Workaround
Since now we know we have to design what's going to run in thread with a bit more care, we need a way to signal the thread that we want to stop.
We can utilize Event for such cases.
import time
import asyncio
def blocking_func(event: asyncio.Event):
while not event.is_set():
time.sleep(1)
print("I'm still standing")
async def main():
event = asyncio.Event()
asyncio.create_task(asyncio.to_thread(blocking_func, event))
await asyncio.sleep(5)
# now lets stop
event.set()
asyncio.run(main())
By checking event on every loop we can see program terminating gracefully.
I'm still standing
I'm still standing
I'm still standing
I'm still standing
I'm still standing
Process finished with exit code 0

How to run a coroutine and wait it result from a sync func when the loop is running?

I have a code like the foolowing:
def render():
loop = asyncio.get_event_loop()
async def test():
await asyncio.sleep(2)
print("hi")
return 200
if loop.is_running():
result = asyncio.ensure_future(test())
else:
result = loop.run_until_complete(test())
When the loop is not running is quite easy, just use loop.run_until_complete and it return the coro result but if the loop is already running (my blocking code running in app which is already running the loop) I cannot use loop.run_until_complete since it will raise an exception; when I call asyncio.ensure_future the task gets scheduled and run, but I want to wait there for the result, does anybody knows how to do this? Docs are not very clear how to do this.
I tried passing a concurrent.futures.Future calling set_result inside the coro and then calling Future.result() on my blocking code, but it doesn't work, it blocks there and do not let anything else to run. ANy help would be appreciated.

To implement runner with the proposed design, you would need a way to single-step the event loop from a callback running inside it. Asyncio explicitly forbids recursive event loops, so this approach is a dead end.
Given that constraint, you have two options:
make render() itself a coroutine;
execute render() (and its callers) in a thread different than the thread that runs the asyncio event loop.
Assuming #1 is out of the question, you can implement the #2 variant of render() like this:
def render():
loop = _event_loop # can't call get_event_loop()
async def test():
await asyncio.sleep(2)
print("hi")
return 200
future = asyncio.run_coroutine_threadsafe(test(), loop)
result = future.result()
Note that you cannot use asyncio.get_event_loop() in render because the event loop is not (and should not be) set for that thread. Instead, the code that spawns the runner thread must call asyncio.get_event_loop() and send it to the thread, or just leave it in a global variable or a shared structure.

Waiting Synchronously for an Asynchronous Coroutine
If an asyncio event loop is already running by calling loop.run_forever, it will block the executing thread until loop.stop is called [see the docs]. Therefore, the only way for a synchronous wait is to run the event loop on a dedicated thread, schedule the asynchronous function on the loop and wait for it synchronously from another thread.
For this I have composed my own minimal solution following the answer by user4815162342. I have also added the parts for cleaning up the loop when all work is finished [see loop.close].
The main function in the code below runs the event loop on a dedicated thread, schedules several tasks on the event loop, plus the task the result of which is to be awaited synchronously. The synchronous wait will block until the desired result is ready. Finally, the loop is closed and cleaned up gracefully along with its thread.
The dedicated thread and the functions stop_loop, run_forever_safe, and await_sync can be encapsulated in a module or a class.
For thread-safery considerations, see section “Concurrency and Multithreading” in asyncio docs.
import asyncio
import threading
#----------------------------------------
def stop_loop(loop):
''' stops an event loop '''
loop.stop()
print (".: LOOP STOPPED:", loop.is_running())
def run_forever_safe(loop):
''' run a loop for ever and clean up after being stopped '''
loop.run_forever()
# NOTE: loop.run_forever returns after calling loop.stop
#-- cancell all tasks and close the loop gracefully
print(".: CLOSING LOOP...")
# source: <https://xinhuang.github.io/posts/2017-07-31-common-mistakes-using-python3-asyncio.html>
loop_tasks_all = asyncio.Task.all_tasks(loop=loop)
for task in loop_tasks_all: task.cancel()
# NOTE: `cancel` does not guarantee that the Task will be cancelled
for task in loop_tasks_all:
if not (task.done() or task.cancelled()):
try:
# wait for task cancellations
loop.run_until_complete(task)
except asyncio.CancelledError: pass
#END for
print(".: ALL TASKS CANCELLED.")
loop.close()
print(".: LOOP CLOSED:", loop.is_closed())
def await_sync(task):
''' synchronously waits for a task '''
while not task.done(): pass
print(".: AWAITED TASK DONE")
return task.result()
#----------------------------------------
async def asyncTask(loop, k):
''' asynchronous task '''
print("--start async task %s" % k)
await asyncio.sleep(3, loop=loop)
print("--end async task %s." % k)
key = "KEY#%s" % k
return key
def main():
loop = asyncio.new_event_loop() # construct a new event loop
#-- closures for running and stopping the event-loop
run_loop_forever = lambda: run_forever_safe(loop)
close_loop_safe = lambda: loop.call_soon_threadsafe(stop_loop, loop)
#-- make dedicated thread for running the event loop
thread = threading.Thread(target=run_loop_forever)
#-- add some tasks along with my particular task
myTask = asyncio.run_coroutine_threadsafe(asyncTask(loop, 100200300), loop=loop)
otherTasks = [asyncio.run_coroutine_threadsafe(asyncTask(loop, i), loop=loop)
for i in range(1, 10)]
#-- begin the thread to run the event-loop
print(".: EVENT-LOOP THREAD START")
thread.start()
#-- _synchronously_ wait for the result of my task
result = await_sync(myTask) # blocks until task is done
print("* final result of my task:", result)
#... do lots of work ...
print("*** ALL WORK DONE ***")
#========================================
# close the loop gracefully when everything is finished
close_loop_safe()
thread.join()
#----------------------------------------
main()

here is my case, my whole programe is async, but call some sync lib, then callback to my async func.
follow the answer by user4815162342.
import asyncio
async def asyncTask(k):
''' asynchronous task '''
print("--start async task %s" % k)
# await asyncio.sleep(3, loop=loop)
await asyncio.sleep(3)
print("--end async task %s." % k)
key = "KEY#%s" % k
return key
def my_callback():
print("here i want to call my async func!")
future = asyncio.run_coroutine_threadsafe(asyncTask(1), LOOP)
return future.result()
def sync_third_lib(cb):
print("here will call back to your code...")
cb()
async def main():
print("main start...")
print("call sync third lib ...")
await asyncio.to_thread(sync_third_lib, my_callback)
# await loop.run_in_executor(None, func=sync_third_lib)
print("another work...keep async...")
await asyncio.sleep(2)
print("done!")
LOOP = asyncio.get_event_loop()
LOOP.run_until_complete(main())

Exit program while tasks in default executor are still running

I have a program which executes some long-running, synchronous tasks and may need to exit even when some of those tasks haven't finished yet. The following is a simple program which start such a task in main(). main() immediately returns after that and thus the program should exit.
import asyncio, time
def synchronous_task():
for i in range(5):
print(i)
time.sleep(1)
async def main():
loop.run_in_executor(None, synchronous_task)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print('Main thread exiting.')
But instead I get this when running the script:
$ python3.5 test3.py
0
Main thread exiting.
1
2
3
4
What is the intended way to handle such a situation? The event loop's default executor already uses daemon threads and terminating the program without cleanup would be OK in my case, but I don't want to use os._exit().

Your question more about how to kill a thread. Look for this post for details.
Solution with event would be:
import asyncio, time
from functools import partial
import threading
import concurrent
def synchronous_task(ev):
for i in range(5):
if ev.isSet():
print('terminating thread')
return
print(i)
time.sleep(1)
async def main():
ev = threading.Event() # see: https://stackoverflow.com/a/325528/1113207
loop.run_in_executor(None, partial(synchronous_task, ev))
await asyncio.sleep(2) # give thread some time to execute, to see results
ev.set()
loop = asyncio.get_event_loop()
executor = concurrent.futures.ThreadPoolExecutor(5)
loop.set_default_executor(executor)
try:
loop.run_until_complete(main())
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
executor.shutdown(wait=True) # see: https://stackoverflow.com/a/32615276/1113207
loop.close()
print('Main thread exiting.')
Result:
0
1
terminating thread
Main thread exiting.
[Finished in 2.2s]

Please explain "Task was destroyed but it is pending!" after cancelling tasks

I am learning asyncio with Python 3.4.2 and I use it to continuously listen on an IPC bus, while gbulb listens on the DBus.
I created a function listen_to_ipc_channel_layer that continuously listens for incoming messages on the IPC channel and passes the message to message_handler.
I am also listening to SIGTERM and SIGINT. When I send a SIGTERM to the python process running the code you find at the bottom, the script should terminate gracefully.
The problem I am having is the following warning:
got signal 15: exit
Task was destroyed but it is pending!
task: <Task pending coro=<listen_to_ipc_channel_layer() running at /opt/mainloop-test.py:23> wait_for=<Future cancelled>>
Process finished with exit code 0
…with the following code:
import asyncio
import gbulb
import signal
import asgi_ipc as asgi
def main():
asyncio.async(listen_to_ipc_channel_layer())
loop = asyncio.get_event_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, ask_exit)
# Start listening on the Linux IPC bus for incoming messages
loop.run_forever()
loop.close()
#asyncio.coroutine
def listen_to_ipc_channel_layer():
"""Listens to the Linux IPC bus for messages"""
while True:
message_handler(message=channel_layer.receive(["my_channel"]))
try:
yield from asyncio.sleep(0.1)
except asyncio.CancelledError:
break
def ask_exit():
loop = asyncio.get_event_loop()
for task in asyncio.Task.all_tasks():
task.cancel()
loop.stop()
if __name__ == "__main__":
gbulb.install()
# Connect to the IPC bus
channel_layer = asgi.IPCChannelLayer(prefix="my_channel")
main()
I still only understand very little of asyncio, but I think I know what is going on. While waiting for yield from asyncio.sleep(0.1) the signal handler caught the SIGTERM and in that process it calls task.cancel().
Shouldn't this trigger the CancelledError within the while True: loop? (Because it is not, but that is how I understand "Calling cancel() will throw a CancelledError to the wrapped coroutine").
Eventually loop.stop() is called which stops the loop without waiting for either yield from asyncio.sleep(0.1) to return a result or even the whole coroutine listen_to_ipc_channel_layer.
Please correct me if I am wrong.
I think the only thing I need to do is to make my program wait for the yield from asyncio.sleep(0.1) to return a result and/or coroutine to break out the while loop and finish.
I believe I confuse a lot of things. Please help me get those things straight so that I can figure out how to gracefully close the event loop without warning.

The problem comes from closing the loop immediately after cancelling the tasks. As the cancel() docs state
"This arranges for a CancelledError to be thrown into the wrapped coroutine on the next cycle through the event loop."
Take this snippet of code:
import asyncio
import signal
async def pending_doom():
await asyncio.sleep(2)
print(">> Cancelling tasks now")
for task in asyncio.Task.all_tasks():
task.cancel()
print(">> Done cancelling tasks")
asyncio.get_event_loop().stop()
def ask_exit():
for task in asyncio.Task.all_tasks():
task.cancel()
async def looping_coro():
print("Executing coroutine")
while True:
try:
await asyncio.sleep(0.25)
except asyncio.CancelledError:
print("Got CancelledError")
break
print("Done waiting")
print("Done executing coroutine")
asyncio.get_event_loop().stop()
def main():
asyncio.async(pending_doom())
asyncio.async(looping_coro())
loop = asyncio.get_event_loop()
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, ask_exit)
loop.run_forever()
# I had to manually remove the handlers to
# avoid an exception on BaseEventLoop.__del__
for sig in (signal.SIGINT, signal.SIGTERM):
loop.remove_signal_handler(sig)
if __name__ == '__main__':
main()
Notice ask_exit cancels the tasks but does not stop the loop, on the next cycle looping_coro() stops it. The output if you cancel it is:
Executing coroutine
Done waiting
Done waiting
Done waiting
Done waiting
^CGot CancelledError
Done executing coroutine
Notice how pending_doom cancels and stops the loop immediately after. If you let it run until the pending_doom coroutines awakes from the sleep you can see the same warning you're getting:
Executing coroutine
Done waiting
Done waiting
Done waiting
Done waiting
Done waiting
Done waiting
Done waiting
>> Cancelling tasks now
>> Done cancelling tasks
Task was destroyed but it is pending!
task: <Task pending coro=<looping_coro() running at canceling_coroutines.py:24> wait_for=<Future cancelled>>

The meaning of the issue is that a loop doesn't have time to finish all the tasks.
This arranges for a CancelledError to be thrown into the wrapped coroutine on the next cycle through the event loop.
There is no chance to do a "next cycle" of the loop in your approach. To make it properly you should move a stop operation to a separate non-cyclic coroutine to give your loop a chance to finish.
Second significant thing is CancelledError raising.
Unlike Future.cancel(), this does not guarantee that the task will be cancelled: the exception might be caught and acted upon, delaying cancellation of the task or preventing cancellation completely. The task may also return a value or raise a different exception.
Immediately after this method is called, cancelled() will not return True (unless the task was already cancelled). A task will be marked as cancelled when the wrapped coroutine terminates with a CancelledError exception (even if cancel() was not called).
So after cleanup your coroutine must raise CancelledError to be marked as cancelled.
Using an extra coroutine to stop the loop is not an issue because it is not cyclic and be done immediately after execution.
def main():
loop = asyncio.get_event_loop()
asyncio.ensure_future(listen_to_ipc_channel_layer())
for sig in (signal.SIGINT, signal.SIGTERM):
loop.add_signal_handler(sig, ask_exit)
loop.run_forever()
print("Close")
loop.close()
#asyncio.coroutine
def listen_to_ipc_channel_layer():
while True:
try:
print("Running")
yield from asyncio.sleep(0.1)
except asyncio.CancelledError as e:
print("Break it out")
raise e # Raise a proper error
# Stop the loop concurrently
#asyncio.coroutine
def exit():
loop = asyncio.get_event_loop()
print("Stop")
loop.stop()
def ask_exit():
for task in asyncio.Task.all_tasks():
task.cancel()
asyncio.ensure_future(exit())
if __name__ == "__main__":
main()

I had this message and I believe it was caused by garbage collection of pending task. The Python developers were debating whether tasks created in asyncio should create strong references and decided they shouldn't (after 2 days of looking into this problem I strongly disagree! ... see the discussion here https://bugs.python.org/issue21163)
I created this utility for myself to make strong references to tasks and automatically clean it up (still need to test it thoroughly)...
import asyncio
#create a strong reference to tasks since asyncio doesn't do this for you
task_references = set()
def register_ensure_future(coro):
task = asyncio.ensure_future(coro)
task_references.add(task)
# Setup cleanup of strong reference on task completion...
def _on_completion(f):
task_references.remove(f)
task.add_done_callback(_on_completion)
return task
It seems to me that tasks should have a strong reference for as long as they are active! But asyncio doesn't do that for you so you can have some bad surprises once gc happens and long hours of debugging.

The reasons this happens is as explained by #Yeray Diaz Diaz
In my case, I wanted to cancel all the tasks that were not done after the first finished, so I ended up cancelling the extra jobs, then using loop._run_once() to run the loop a bit more and allow them to stop:
loop = asyncio.get_event_loop()
job = asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
tasks_finished,tasks_pending, = loop.run_until_complete(job)
tasks_done = [t for t in tasks_finished if t.exception() is None]
if tasks_done == 0:
raise Exception("Failed for all tasks.")
assert len(tasks_done) == 1
data = tasks_done[0].result()
for t in tasks_pending:
t.cancel()
t.cancel()
while not all([t.done() for t in tasks_pending]):
loop._run_once()