We Keep Coding

Django Channels. How to "try-except" the error when the "Event loop is closed"?

I have a trigger that sends a message to the layer group.
As it is a WebSocket, it works only on a webpage, but I have also an API point for that trigger because it also saves some object to DB.
So I need to handle(pass) an error when the trigger act when Websocket is closed (i.e. web page is closed).
So I think I need to "try except" that RuntimeError.
But where to make it? What code include in try-except?
I try to "try-except" the __init__ of Consumer Class, but to no avail.
Pls, advise.
Here is traceback:
Task exception was never retrieved
future: <Task finished name='Task-37' coro=<Connection.disconnect() done, defined at /Users/Ruslan/IT/CODE/my-blogpost-pet/venv/lib/python3.10/site-packages/redis/asyncio/connection.py:819> exception=RuntimeError('Event loop is closed')>
Traceback (most recent call last):
File "/Users/Ruslan/IT/CODE/my-blogpost-pet/venv/lib/python3.10/site-packages/redis/asyncio/connection.py", line 828, in disconnect
self._writer.close() # type: ignore[union-attr]
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/streams.py", line 337, in close
return self._transport.close()
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/selector_events.py", line 698, in close
self._loop.call_soon(self._call_connection_lost, None)
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 753, in call_soon
self._check_closed()
File "/usr/local/Cellar/python#3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 515, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

How to shutdown gracefully on keyboard interrupt when an asyncio task is performing _blocking_ work?

Update: asyncio simply does what it's told and you can handle these exceptions just fine - see my follow-up answer that I've marked as the solution to this question. Original question below, with slightly modified example to clarify the issue and its solution.
I've been trying to debug a library that I'm working on that relies heavily on asyncio. While working on some example code, I realised that performing a keyboard interrupt (CTRL-C) sometimes (rarely!) triggered the dreaded...
Task exception was never retrieved
I've tried hard to make sure that all tasks that I spin off handle asyncio.CancelledError gracefully, and after having spent way too many hours debugging this I realised that I only end up with this error message if one of the asyncio tasks is stuck on a blocking operation.
Blocking? You really shouldn't perform blocking work in tasks - that's why asyncio is kind enough to warn you about this. Run the below code...
import asyncio
from time import sleep
async def possibly_dangerous_sleep(i: int, use_blocking_sleep: bool = True):
try:
print(f"Sleep #{i}: Fine to cancel me within the next 2 seconds")
await asyncio.sleep(2)
if use_blocking_sleep:
print(
f"Sleep #{i}: Not fine to cancel me within the next 10 seconds UNLESS someone is"
" awaiting me, e.g. asyncio.gather()"
)
sleep(10)
else:
print(f"Sleep #{i}: Will sleep using asyncio.sleep(), nothing to see here")
await asyncio.sleep(10)
print(f"Sleep #{i}: Fine to cancel me now")
await asyncio.sleep(2)
except asyncio.CancelledError:
print(f"Sleep #{i}: So, I got cancelled...")
raise
def done_cb(task: asyncio.Task):
name = task.get_name()
try:
task.exception()
except asyncio.CancelledError:
print(f"Done: Task {name} was cancelled")
pass
except Exception as e:
print(f"Done: Task {name} didn't handle exception { e }")
else:
print(f"Done: Task {name} is simply done")
async def start_doing_stuff(collect_exceptions_when_gathering: bool = False):
tasks = []
for i in range(1, 7):
task = asyncio.create_task(
possibly_dangerous_sleep(i, use_blocking_sleep=True), name=str(i)
)
task.add_done_callback(done_cb)
tasks.append(task)
# await asyncio.sleep(3600)
results = await asyncio.gather(*tasks, return_exceptions=collect_exceptions_when_gathering)
if __name__ == "__main__":
try:
asyncio.run(start_doing_stuff(collect_exceptions_when_gathering=False), debug=True)
except KeyboardInterrupt:
print("User aborted through keyboard")
...and the debug console will tell you something along the lines of:
Executing <Task finished name='Task-2' coro=<possibly_dangerous_sleep() done, defined at ~/src/hej.py:5> result=None created at ~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/tasks.py:337> took 10.005 seconds
Rest assured that the above call to sleep(10) isn't the culprit in the library I'm working on, but it illustrates the issue I'm running into: if I try to interrupt the above test application within the first 2 to 12 seconds of it running, the debug console will end up with a hefty source traceback:
Fine to cancel me within the next 2 seconds
Not fine to cancel me within the next 10 seconds UNLESS someone is awaiting me, e.g. asyncio.gather()
^CDone with: <Task finished name='Task-2' coro=<possibly_dangerous_sleep() done, defined at ~/src/hej.py:5> exception=KeyboardInterrupt() created at ~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/tasks.py:337>
User aborted through keyboard
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<dangerous_sleep() done, defined at ~/src/hej.py:5> exception=KeyboardInterrupt() created at ~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/tasks.py:337>
source_traceback: Object created at (most recent call last):
File "~/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "~/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "~/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
cli.main()
File "~/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
run()
File "~/.vscode/extensions/ms-python.python-2021.12.1559732655/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
File "~/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 269, in run_path
return _run_module_code(code, init_globals, run_name,
File "~/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "~/.pyenv/versions/3.10.0/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "~/src/hej.py", line 37, in <module>
asyncio.run(start_doing_stuff(), debug=True)
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 628, in run_until_complete
self.run_forever()
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 595, in run_forever
self._run_once()
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 1873, in _run_once
handle._run()
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "~/src/hej.py", line 28, in start_doing_stuff
task = asyncio.create_task(dangerous_sleep())
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/tasks.py", line 337, in create_task
task = loop.create_task(coro)
Traceback (most recent call last):
File "~/src/hej.py", line 37, in <module>
asyncio.run(start_doing_stuff(), debug=True)
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 628, in run_until_complete
self.run_forever()
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 595, in run_forever
self._run_once()
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/base_events.py", line 1873, in _run_once
handle._run()
File "~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "~/src/hej.py", line 14, in dangerous_sleep
sleep(10)
KeyboardInterrupt
If I replace await asyncio.sleep(3600) with await asyncio.gather(task) (see the example code) and invoke CTRL-C, I instead get a very neat shutdown sequence in my debug console:
Fine to cancel me within the next 2 seconds
Not fine to cancel me within the next 10 seconds UNLESS someone is awaiting me, e.g. asyncio.gather()
^CDone with: <Task finished name='Task-2' coro=<possibly_dangerous_sleep() done, defined at ~/src/hej.py:5> exception=KeyboardInterrupt() created at ~/.pyenv/versions/3.10.0/lib/python3.10/asyncio/tasks.py:337>
User aborted through keyboard
Can someone explain to me if this is by design? I was expecting all asyncio tasks to be cancelled for me when asyncio.run() was interrupted (while cleaning up after itself).

Summary: You need to handle your exceptions, or asyncio will complain.
For background tasks (i.e. tasks that you don't explicitly wait for using gather())
You might think that trying to catch cancellation using except asyncio.CancelledError (and re-raising it) within your task would handle all types of cancellation. That's not the case. If your task is performing blocking work while being cancelled, you won't be able to catch the exception (e.g. KeyboardInterrupt) within the task itself. The safe bet here is to register a done callback using add_done_callback on your asyncio.Task. In this callback, check if there was an exception (see the updated example code in the question). If your task was stuck on blocking work while being cancelled, the done callback will tell you that the task was done (vs cancelled).
For a bunch of tasks that you await using gather()
If you use gather, you don't need to add done callbacks. Instead, ask it to return any exceptions and it will handle KeyboardInterrupt just fine. If you don't do this, the first exception being raised within any of its awaitables is immediately propagated to the task that awaits on gather(). In the case of a KeyboardInterrupt inside a task that's stuck doing blocking work, KeyboardInterrupt will be re-raised and you'll need to handle it. Alternatively, use try/except to handle any exceptions raised. Please try this yourself by setting the collect_exceptions_when_gathering variable in the example code.
Finally: the only thing I don't understand now is that I don't see any exception being raised if one calls gather() with a single task, not asking it to return exceptions. Try to modify the example code to have its range be range(1,2) and you won't get a messy stack trace on CTRL-C...?

Quart got an unexpected keyword argument 'loop' when run from bot.loop.create_task

I am trying to add a web ui to a discord bot using Quart. From what I've seen the appropriate way to do this is to from a instance of discord.Bot create a task and to run it. I'm currently doing it this way
def start():
bot.loop.create_task(app.run_task("0.0.0.0"))
load_commands()
print(f"loaded commands: {loaded_commands}")
bot.run(TOKEN)
however when this is run I get the following error
Task exception was never retrieved
future: <Task finished name='Task-1' coro=<serve() done, defined at /var/home/nullrequest/.local/share/virtualenvs/lunbot-ldM1Y48e/lib/python3.10/site-packages/hypercorn/asyncio/__init__.py:9> exception=TypeError("BaseEventLoop.create_server() got an unexpected keyword argument 'loop'")>
Traceback (most recent call last):
File "/var/home/nullrequest/.local/share/virtualenvs/lunbot-ldM1Y48e/lib/python3.10/site-packages/hypercorn/asyncio/__init__.py", line 39, in serve
await worker_serve(app, config, shutdown_trigger=shutdown_trigger)
File "/var/home/nullrequest/.local/share/virtualenvs/lunbot-ldM1Y48e/lib/python3.10/site-packages/hypercorn/asyncio/run.py", line 128, in worker_serve
await asyncio.start_server(
File "/usr/lib64/python3.10/asyncio/streams.py", line 84, in start_server
return await loop.create_server(factory, host, port, **kwds)
TypeError: BaseEventLoop.create_server() got an unexpected keyword argument 'loop'
Task was destroyed but it is pending!
task: <Task pending name='Task-3' coro=<Lifespan.handle_lifespan() running at /var/home/nullrequest/.local/share/virtualenvs/lunbot-ldM1Y48e/lib/python3.10/site-packages/hypercorn/asyncio/lifespan.py:30> wait_for=<Future pending cb=[Task.task_wakeup()]>>
I am using python 3.10rc1 if that makes any difference.

I have same error. Python 3.10 released on 04.10.2021. May be however it depends on the version of interpreter.
for message in client.iter_history(chat.id):
if message.from_user.id == user_id:
# My code...
My traceback:
Traceback (most recent call last):
File "D:\freelance\messagesParser\main.py", line 74, in <module>
main(sys.argv)
File "D:\freelance\messagesParser\main.py", line 64, in main
messages[chat] = get_messages(client, chat, **user_data)
File "D:\freelance\messagesParser\main.py", line 22, in get_messages
for message in client.iter_history(chat.id):
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\sync.py", line 59, in async_to_sync_wra
p
return loop.run_until_complete(consume_generator(coroutine))
File "C:\Program Files\Python310\lib\asyncio\base_events.py", line 641, in run_until_complete
return future.result()
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\sync.py", line 34, in consume_generator
return types.List([i async for i in coroutine])
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\sync.py", line 34, in <listcomp>
return types.List([i async for i in coroutine])
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\methods\messages\iter_history.py", line
79, in iter_history
messages = await self.get_history(
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\methods\messages\get_history.py", line
86, in get_history
messages = await utils.parse_messages(
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\utils.py", line 91, in parse_messages
parsed_messages.append(await types.Message._parse(client, message, users, chats, replies=0))
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\types\messages_and_media\message.py", l
ine 619, in _parse
sticker = await types.Sticker._parse(
File "D:\freelance\messagesParser\venv\lib\site-packages\pyrogram\types\messages_and_media\sticker.py", l
ine 135, in _parse
set_name = await Sticker._get_sticker_set_name(client.send, input_sticker_set_id)
File "D:\freelance\messagesParser\venv\lib\site-packages\async_lru.py", line 237, in wrapped
return (yield from asyncio.shield(fut, loop=_loop))
TypeError: shield() got an unexpected keyword argument 'loop'

Asyncio run_until_complete gives RunTimeError

I am trying to run a simple asyncio loop that I want to run continuously until Control + C is pressed, which should stop the main function and run a function to shutdown the loop. I am using the following code:
import asyncio
async def run():
while True:
print("Running")
async def shutdown():
print("Starting to shutdown")
await asyncio.sleep(1)
print("Finished sleeping")
try:
loop = asyncio.get_event_loop()
loop.create_task(run())
loop.run_forever()
except KeyboardInterrupt:
loop.stop()
loop.run_until_complete(asyncio.gather(*[shutdown()]))
When running this code and pressing Control + C the script doesn't shutdown gracefully as I expected (also based on this stackoverflow answer but I get the following error:
Traceback (most recent call last):
File "test.py", line 25, in <module>
loop.run_until_complete(asyncio.gather(*[shutdown()]))
File "/usr/lib/python3.7/asyncio/base_events.py", line 585, in run_until_complete
raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<run() done, defined at test.py:8> exception=KeyboardInterrupt()>
Traceback (most recent call last):
File "test.py", line 22, in <module>
loop.run_forever()
File "/usr/lib/python3.7/asyncio/base_events.py", line 541, in run_forever
self._run_once()
File "/usr/lib/python3.7/asyncio/base_events.py", line 1786, in _run_once
handle._run()
File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "test.py", line 10, in run
print("Running")
KeyboardInterrupt
Can anyone tell me where I am going wrong and how I should change my code to solve the issue?

How to debug a stuck asyncio coroutine in Python?

There are lots of coroutines in my production code, which are stuck at unknown position while processing request. I attached gdb with Python support extension to the process, but it doesn't show the exact line in the coroutine where the process is stuck, only primary stack trace. Here is a minimal example:
import asyncio
async def hello():
await asyncio.sleep(30)
print('hello world')
asyncio.run(hello())
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/lib/python3.8/asyncio/base_events.py", line 2335, in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
None, getaddr_func, host, port, family, type, proto, flags)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
File "main.py", line 7, in <module>
GDB shows a trace that ends on line 7, but the code is obviously stuck on line 4. How to make it show a more complete trace with nested coroutines?

You can use the aiodebug.log_slow_callbacks.enable(0.05)
Follow for more : https://pypi.org/project/aiodebug/