We Keep Coding

Problems using asyncio in Thread

There is no problem using asyncio as follows.
import asyncio
async def main():
await asyncio.sleep(1)
aaa = 1
print (aaa)
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
However, I have to use asyncio within Thread as shown in the simple code below, which results in an error as follows
import asyncio
import threading
async def main():
await asyncio.sleep(1)
aaa = 1
print (aaa)
def test():
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
t=threading.Thread(target=test)
t.start()
Error Message (RuntimeError: There is no current event loop in thread 'Thread-1'.)
Exception in thread Thread-1:
Traceback (most recent call last):
File "D:\Anaconda3\Lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "D:\Anaconda3\Lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "D:/Test/Test4/main.py", line 57, in test
loop = asyncio.get_event_loop()
File "D:\Anaconda3\Lib\asyncio\events.py", line 639, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'Thread-1'.
How can I use asyncio in Thread?
This is very important to me.
I've been struggling with this problem for two days.
I would appreciate your help.

Unless you need to work with a pre-existing event loop (not the case here) or support Python 3.6, use asyncio.run instead of manually handling the loop:
def test():
asyncio.run(main())
This always ensures that there is a valid event loop, and also handles proper cleanup.
This function always creates a new event loop and closes it at the end. It should be used as a main entry point for asyncio programs, and should ideally only be called once.
Note that when combining asyncio with threads, the event loop should generally be run in the main loop (see e.g. bpo#34697). Instead of spawning a thread for asyncio, prefer to spawn one for the synchronous parts of your program.
# launch synchronous work in child threads
t=threading.Thread(target=my_sync_tasks)
t.start()
# launch asynchronous work in main thread
asyncio.run(main())
t.join()

When using a thread other than the main thread you need to set the event loop
def test():
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(main())
loop.close()

Callback seems not to be executed when attached to another thread

What I want to do is, in the Python console:
Create an asyncio future
Attach a callback to this future
Set the future value (and so get at the same time the callback result)
To do this, I tried that, in the Python console:
from threading import Thread
import asyncio
def start_loop(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
new_loop = asyncio.new_event_loop()
t = Thread(target=start_loop, args=(new_loop,))
t.start()
fut = new_loop.create_future()
fut.add_done_callback(lambda _: print("Hey"))
fut.set_result(True)
I expect "Hey" to be printed, but actually I get nothing.
Additional information:
When I do:
import asyncio
loop = asyncio.get_event_loop()
fut = loop.create_future()
fut.add_done_callback(lambda x: print("Hey"))
fut.set_result(True)
loop.run_forever()
I get the expected result, but I had to call run_forever while the future is already set.
My questions are:
Why my first block of code does not do the job as expected ?
Is there a hypothetical syntax like the following existing ?
import asyncio
loop = asyncio.get_event_loop()
fut = loop.create_future()
# Hypothetical part
loop.run_forever_in_background()
# End of hypothetical part
fut.add_done_callback(lambda x: print("Hey"))
fut.set_result(True)
My high level purpose is to play with futures and asyncio directly in the console, to get a better understanding of it.

Pay attention when playing with asyncio loops, futures etc. in multithreading context.
Almost all asyncio objects are not thread safe, which is
typically not a problem unless there is code that works with them from
outside of a Task or a callback.
...
To schedule a coroutine object from a different OS thread, the
run_coroutine_threadsafe() function should be used. It returns a
concurrent.futures.Future to access the result:
from threading import Thread
import asyncio
def start_loop(loop):
asyncio.set_event_loop(loop)
loop.run_forever()
new_loop = asyncio.new_event_loop()
t = Thread(target=start_loop, args=(new_loop,))
t.start()
async def add_future():
fut = new_loop.create_future()
fut.add_done_callback(lambda _: print("Hey"))
fut.set_result(True)
asyncio.run_coroutine_threadsafe(add_future(), loop=new_loop)
Test:
$ python3 test.py
Hey
Another option is using loop.call_soon_threadsafe(callback, *args, context=None).
https://docs.python.org/3/library/asyncio-dev.html#asyncio-multithreading

Racing Two Tasks in Different Event Loops

I am using the Docker SDK, and I am trying to race a task that times out after some number of seconds against another task that waits on a Docker container to finish. In effect, I want to know if a given container finishes within the timeout I've set.
I have the following code to do it (adapted from this post):
container = # ... create container with Docker SDK
timeout = # ... some int
killed = None
# our tasks
async def __timeout():
await asyncio.sleep(timeout)
return True
async def __run():
container.wait()
return False
# loop and runner
wait_loop = asyncio.new_event_loop()
done, pending = wait_loop.run_until_complete(
asyncio.wait({__run(), __timeout()}, return_when=asyncio.FIRST_COMPLETED)
)
# result extraction
for task in done:
if killed is None:
killed = task.result()
# ... do something with result
# clean up
for task in pending:
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
wait_loop.run_until_complete(task)
wait_loop.close()
Unfortunately, I keep getting the following error:
File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
return future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
raise self._exception
File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
result = coro.throw(exc)
File "/usr/lib/python3.5/asyncio/tasks.py", line 347, in wait
return (yield from _wait(fs, timeout, return_when, loop))
File "/usr/lib/python3.5/asyncio/tasks.py", line 430, in _wait
yield from waiter
File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
yield self # This tells Task to wait for completion.
RuntimeError: Task <Task pending coro=<wait() running at /usr/lib/python3.5/asyncio/tasks.py:347> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> got Future <Future> pending> attached to a different loop
It seems I can't race with the wait task because it belongs to a different loop. Is there any way I can get around this error so that I can determine which task finishes first?

The problem is simple, there is one default loop in every thread. Which is set by asyncio.set_event_loop(loop). Then you can get this loop by loop = asyncio.get_event_loop().
So the problem is, mostly, some packages use asyncio.get_event_loop() by default to get current running loop. Take aiohttp as an example:
class aiohttp.ClientSession(*, connector=None, loop=None, cookies=None, headers=None, skip_auto_headers=None, auth=None, json_serialize=json.dumps, version=aiohttp.HttpVersion11, cookie_jar=None, read_timeout=None, conn_timeout=None, timeout=sentinel, raise_for_status=False, connector_owner=True, auto_decompress=True, requote_redirect_url=False, trust_env=False, trace_configs=None)
As you can see, it accepts loop parameter to specify running loop. But You can also just leave it blank to use asyncio.get_event_loop() by default.
Your problem is you are launching coroutines in a new created loop. But you cannot confirm that all your interal operations are also using this new created one. As they may use asyncio.get_event_loop(), they will be attached into another loop which is the default loop in current thread.
As far as I think, you don't really need to create a new one, but let users do that. Just like the example above, you accept an argument loop, and if it is None, use the default one.
Or you need to carefully inspect your code to ensure that every possible coroutine is using the loop you create.

"Asyncio Event Loop is Closed" when getting loop

When trying to run the asyncio hello world code example given in the docs:
import asyncio
async def hello_world():
print("Hello World!")
loop = asyncio.get_event_loop()
# Blocking call which returns when the hello_world() coroutine is done
loop.run_until_complete(hello_world())
loop.close()
I get the error:
RuntimeError: Event loop is closed
I am using python 3.5.3.

On Windows seems to be a problem with EventLoopPolicy, use this snippet to work around it:
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
asyncio.run(main())

You have already called loop.close() before you ran that sample piece of code, on the global event loop:
>>> import asyncio
>>> asyncio.get_event_loop().close()
>>> asyncio.get_event_loop().is_closed()
True
>>> asyncio.get_event_loop().run_until_complete(asyncio.sleep(1))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.../lib/python3.6/asyncio/base_events.py", line 443, in run_until_complete
self._check_closed()
File "/.../lib/python3.6/asyncio/base_events.py", line 357, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
You need to create a new loop:
loop = asyncio.new_event_loop()
You can set that as the new global loop with:
asyncio.set_event_loop(asyncio.new_event_loop())
and then just use asyncio.get_event_loop() again.
Alternatively, just restart your Python interpreter, the first time you try to get the global event loop you get a fresh new one, unclosed.
As of Python 3.7, the process of creating, managing, then closing the loop (as well as a few other resources) is handled for you when use asyncio.run(). It should be used instead of loop.run_until_complete(), and there is no need any more to first get or set the loop.

...and just in case:
import platform
if platform.system()=='Windows':
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
If you ever deploy your code in the cloud it might avoid painful debug.

asyncio event_loop declared outside of a class with asyncio.coroutine methods fails with "AttributeError: 'NoneType' object has no attribute 'select'"

Exploring Python 3.4.0's asyncio module, I am attempting to create a class with asyncio.coroutine methods that are called from an event_loop outside of the class.
My working code is below.
import asyncio
class Foo():
#asyncio.coroutine
def async_sleep(self):
print('about to async sleep')
yield from asyncio.sleep(1)
#asyncio.coroutine
def call_as(self):
print('about to call ass')
yield from self.async_sleep()
def run_loop(self):
loop = asyncio.get_event_loop()
loop.run_until_complete(self.call_as())
print('done with loop')
loop.close()
a = Foo()
a.run_loop()
loop = asyncio.get_event_loop()
loop.run_until_complete(a.call_as())
The call to a.run_loop() provides output as expected:
python3 async_class.py
about to call ass
about to async sleep
done with loop
However as soon as the event_loop attempts to process a.call_as() I get the following Traceback:
Traceback (most recent call last):
File "async_class.py", line 26, in <module>
doop.run_until_complete(asyncio.async(a.call_ass()))
File "/opt/boxen/homebrew/opt/pyenv/versions/3.4.0/lib/python3.4/asyncio/base_events.py", line 203, in run_until_complete
self.run_forever()
File "/opt/boxen/homebrew/opt/pyenv/versions/3.4.0/lib/python3.4/asyncio/base_events.py", line 184, in run_forever
self._run_once()
File "/opt/boxen/homebrew/opt/pyenv/versions/3.4.0/lib/python3.4/asyncio/base_events.py", line 778, in _run_once
event_list = self._selector.select(timeout)
AttributeError: 'NoneType' object has no attribute 'select'
I have attempted wrapping a.call_as() in an asyncio.Task(), asyncio.async() and the failure is the same.

As it turns out, the issue was with the context of the event loop.
asyncio magically creates an event loop for a thread at runtime. This event loop's context is set when .get_event_loop() is called.
In the above example, a.run_loop sets the event loop inside the context of Foo.run_loop.
One kicker of event loops is that there may only be one event loop per thread, and a given event loop can only process events in its context.
With that in mind, note that the loop = asyncio.get_event_loop() just after a.run_loop is asking to assign the thread's event loop to the __main__ context. Unfortunately, the event loop was already set to the context of Foo.run_loop, so a None type is set for the __main__ event loop.
Instead, it is necessary to create a new event loop and then set that event loop's context to __main__, i.e.
new_loop = asyncio.new_event_loop()
asyncio.set_event_loop(new_loop)
Only then will an event loop be properly set in the context of __main__, allowing for the proper execution of our now-modified new_loop.run_until_complete(a.call_as())

It is because you close the eventloop at the end of Foo.run_loop()
From BaseEventLoop.close
Close the event loop. The loop must not be running. Pending callbacks will be lost.
This clears the queues and shuts down the executor, but does not wait for the executor to finish.
This is idempotent and irreversible. No other methods should be called after this one.