I'm trying to run some IO blocking code, so I'm using to_thread to send the function to another thread. I tried several things, but in all cases, I seem to have to await the to_thread which just returns another coroutine (?!) that I then have to await again. For some reason this isn't quite clicking.
import asyncio
async def search(keyword_list):
coroutines = set()
for index, kw in enumerate(keyword_list):
coroutines.add(asyncio.to_thread(do_lookup, keyword, index))
for result in asyncio.as_completed(coroutines):
outcome, index = await result
# Do some magic with the outcome and index
# BUT, this doesn't work because `await result` apparently
# just returns ANOTHER coroutine!
async def do_lookup(keyword, index):
# Do long, blocking stuff here
print(f'running...{keyword} {index}')
return keyword, index
if __name__ == '__main__':
asyncio.run(search([1, 2, 3, 4]))
As I was copy/pasting and adapting my code to make a generic example, I discovered the problem here.
do_lookup is supposed to a synchronous function (because of the usage of to_thread), so by defining it async dev do_lookup I'm instead defining it as an asynchronous function, thereby causing the "double" await issue.
Simply redefining do_lookup without the async keyword did the trick!
Related
I'm trying to create a wrapper around an asyncio coroutine that allows the user to use it as a "normal" function.
To give a bit of context, this is a function inside a package that war originally not-async. For a series of reasons, I now need to have an async version of it. To avoid duplicating the whole code, I'm trying to create a wrapper to allow existing code (that doesn't use asyncio) to keep running without breaking back compatibility.
To make things more complicated, the majority of the users (it's a company code) use this code inside Spyder IDE.
To sort it, I did something like this
import asyncio
async def an_async_subfunction(t, tag):
print(f"I'm inside an_async_subfunction named {tag}")
await asyncio.sleep(t)
print(f"Leaving an_async_subfunction named {tag}")
async def an_async_function(n):
print(f"I'm inside an_async_function")
tasks = [an_async_subfunction(t, t) for t in range(n)]
await asyncio.gather(*tasks)
print(f"Leaving an_async_function")
async def main_async(n):
# the old main function, now become a corouting
await an_async_function(n)
return 'a result'
def main(*args):
# the wrapper exposed to the users
return asyncio.run(main_async(*args))
if __name__ == '__main__':
print('Normal version')
# The user can call the main function without bothering with asyncio
result = main(3)
print('Async version')
# ...or can use the async version of main if he wants
async def a_user_defined_async_function():
return await main_async(3)
result = asyncio.run(a_user_defined_async_function())
This works as expected, allowing the basic user to call main without bothering that it is a coroutine, while if a user wants to use main inside a custom-made async function, he can use main_async.
However, if you try to run this code in Spyder, you get the error:
RuntimeError: asyncio.run() cannot be called from a running event loop
This is caused by the fact that Spyder has its own event loop running as explained here.
I tried to fix it doing something like:
def main(*args):
if asyncio.get_event_loop().is_running():
return asyncio.create_task(main_async(*args)).result()
else:
return asyncio.run(main_async(*args))
This is now "Spyder-friendly" an it works inside Spyder without problems. The problem is that .result() is called before the Task created by asyncio.create_task is finished and an InvalidStateError exception is returned.
I can't put an await in front of create_task as main is not a coroutine, and I can't make main a coroutine, otherwise the whole thing would have been pointless.
Is there a solution to this mess?
I want to return the first element of async generator and handle the remainning values without return like fire and forget. How to make early return of coroutine in python?
After passing the iterator to asyncio.create_task, it doesn't print the remaining values.
import asyncio
import time
async def async_iter(num):
for i in range(num):
await asyncio.sleep(0.5)
yield i
async def handle_remains(it):
async for i in it:
print(i)
async def run() -> None:
it = async_iter(10)
async for i in it:
print(i)
break
# await handle_remains(it)
# want to make this `fire and forget`(no await), expecting just printing the remainning values.
asyncio.create_task(handle_remains(it))
return i
if __name__ == '__main__':
asyncio.run(run())
time.sleep(10)
You’re close with the code, but not quite there yet (see also my comments above). In short, creating the Task isn’t enough: the Task needs to run:
task = asyncio.create_task(handle_remains(it)) # Creates a Task in `pending` state.
await task # Run the task, i.e. execute the wrapped coroutine.
A Task, along with coroutines and Futures, is an “Awaitable”. In fact:
When a coroutine is wrapped into a Task with functions like asyncio.create_task() the coroutine is automatically scheduled to run soon.
Notice the “scheduled to run soon”, now you have to make sure to actually run the task by calling await, a keyword which…
is used to obtain a result of coroutine execution.
I am learning python asyncio and testing a lot of code using them.
Below is a code where I try to subscribe multiple Websocket streaming using asyncio and aiohttp.
I do not understand why when coro(item1, item2): is executed as a task, it does not go into the async with ... block. (i.e "A" is printed but not "B").
Could anyone help me understand the reason for this?
(I have already got a working code but , I simply want to understand what the mechanism behind this is.)
Code
import aiohttp
import asyncio
import json
async def coro(
item1,
item2):
print("A")
async with aiohttp.ClientSession() as session:
async with session.ws_connect(url='URL') as ws:
print("B")
await asyncio.gather(ws.send_json(item1),
ws.send_json(item2))
print("C")
async for msg in ws:
print(msg)
async def ws_connect(item1,
item2):
task = asyncio.create_task(coro(item1, item2))
return task
async def main():
item1 = {
"method": "subscribe",
"params": {'channel': "..."}
}
item2 = {
"method": "subscribe",
"params": {'channel': "..."}
}
ws_task = await ws_connect(item1, item2)
print("D")
asyncio.run(main())
Output
D
A
B is never printed because you never await the returned task, only the method which returned it.
The subtle mistake is in return task followed by await ws_connect(item1, item2).
TL;DR; return await task.
The key to understand the program's output is to know that the context switches in the asyncio event loop can only occur at few places, in particular at await expressions. At this point, the event loop might suspend the current coroutine and continue with another.
First, you create a ws_connect coroutine and immedietely await it, this forces the event loop to suspend main and actually run ws_connect because there is not anything else to run.
Since ws_connect contains none of those points which allow context switch, the coro() function never actually starts.
Only thing create_task does is binding the coroutine to the task object and adding it to the event loop's queue. But you never await it, you just return it as any ordinary return value. Okay, now the ws_connect() finishes and the event loop can choose to run any of the tasks, it chose to continue with main probably since it has been waiting on ws_connect().
Okay, main prints D and returns. Now what?
There is some extra await in asyncio.run which gives coro() a chance to start - hence the printed A (but only after D) yet nothing forces asyncio.run to wait on coro() so when the coro yields back to the context loop through async with, the run finishes and program exits which leaves coro() unfinished.
If you add an extra await asyncio.sleep(1) after print('D'), the loop will again suspend main for at least some time and continue with coro() and that would print B had the URL been correct.
Actually, the context switching is little bit more complicated because ordinary await on a coroutine usually does not switches unless the execution really needs to block on IO or something await asyncio.sleep(0) or yield* guarantees a true context switch without the extra blocking.
*yield from inside __await__ method.
The lesson here is simple - never return awaitables from async methods, it leads to exactly this kind of mistake. Always use return await by default, at worst you get runtime error in case the returned object is not actually awaitable(like return await some_string) and it can easily be spotted and fixed.
On the other hand, returning awaitables from ordinary functions is OK and makes it act like the function is asynchronous. Although one should be careful when mixing these two approaches. Personally, I prefer the first approach as it shifts the responsibility on the writer of the function, not the user which will be warned linters which usually do detect non-awaited corountine calls but not the returned awaitables. So another solution would to make ws_connect an ordinary function, then the await in await ws_connect would apply to the returned value(=the task), not the function itself.
I'd like to define what essentially is an asynchronous __del__ that closes a resource. Here's an example.
import asyncio
class Async:
async def close(self):
print('closing')
return self
def __del__(self):
print('destructing')
asyncio.ensure_future(self.close())
async def amain():
Async()
if __name__ == '__main__':
asyncio.run(amain())
This works, printing destructing and closing as expected. However, if the resource is defined outside an asynchronous function, __del__ is called, but closing is never performed.
def main():
Async()
No warning is raised here, but the prints reveal that closing was not done. The warning is issued if an asynchronous function has been run, but any instance is created outside of it.
def main2():
Async()
asyncio.run(amain())
RuntimeWarning: coroutine 'Async.close' was never awaited
This has been the subject in 1 and 2, but neither quite had what I was looking for, or maybe I didn't know how to look. Particularly the first question was about deleting a resource, and its answer suggested using asyncio.ensure_future, which was tested above. Python documentation suggests using the newer asyncio.create_task, but it straight up raises an error in the non-async case, there being no current loop. My final, desperate attempt was to use asyncio.run, which worked for the non-async case, but not for the asynchronous one, as calling run is prohibited in a thread that already has a running loop. Additionally, the documentation states that it should only be called once in a program.
I'm still new to async things. How could this be achieved?
A word on the use case, since asynchronous context managers were mentioned as the preferred alternative in comments. I agree, using them for short-term resource management would be ideal. However, my use case is different for two reasons.
Users of the class are not necessarily aware of the underlying resources. It is better user experience to hide closing the resource from a user who doesn't fiddle with the resource itself.
The class needs to be instantiated (or for it to be possible to instantiate it) in a synchronous context, and it is often created just once. For example, in a web server context the class would be instantiated in the global scope, after which its async functions would be used in the endpoint definitions.
For example:
asc = Async()
server.route('/', 'GET')
async def root():
return await asc.do_something(), 200
I'm open to other suggestions of implementing such a feature, but at this point even my curiosity for the possibility that this can be done is enough for me to want an answer to this specific question, not just the general problem.
Only thing that comes to mind is to run cleanup after the server shutdown. It'll look something like this:
asc = Async()
try:
asyncio.run(run_server()) # You already do it now somewhere
finally:
asyncio.run(asc.close())
Since asyncio.run creates new event loop each time, you may want to go even deeper and reuse the same event loop:
loop = asyncio.get_event_loop()
asc = Async()
try:
loop.run_until_complete(run_server())
finally:
loop.run_until_complete(asc.close())
It's absolutely ok to call run_until_complete multiple times as long as you know what you're doing.
Full example with your snippet:
import asyncio
class Async:
async def close(self):
print('closing')
return self
async def cleanup(self):
print('destructing')
await self.close()
loop = asyncio.get_event_loop()
asc = Async()
async def amain():
await asyncio.sleep(1) # Do something
if __name__ == '__main__':
try:
loop.run_until_complete(amain())
finally:
loop.run_until_complete(asc.cleanup())
loop.close()
Just like to understand async await syntax, so I am looking for some 'hello world' app without using asyncio at all.
So how to create simplest event loop using only Python syntax itself? The simplest code (from this Start async function without importing the asyncio package , further code is much more then hello world, that's why I am asking) looks like that:
async def cr():
while(True):
print(1)
cr().send(None)
It prints 1 infinitely, not so good.
So the 1st question is how to yield from coroutine back to the main flow? yield keyword makes coroutine async generator, not we expected.
I would also appreciate a simple application, like this
i.e. we have a coroutine which prints 1, then yields to event loop, then prints 2 then exits with return 3, and simple event loop, which push coroutine until return and consume result.
How about this?
import types
#types.coroutine
def simple_coroutine():
print(1)
yield
print(2)
return 3
future = simple_coroutine()
while True:
try: future.send(None)
except StopIteration as returned:
print('It has returned', returned.value)
break
I think your biggest problem is that you're mixing concepts. An async function is not the same as a coroutine. It is more appropriate to think of it as a way of combining coroutines. Same as ordinary def functions are a way of combining statements into functions. Yes, Python is highly reflective language, so def is also a statement, and what you get from your async function is also a coroutine---but you need to have something at the bottom, something to start with. (At the bottom, yielding is just yielding. At some intermediate level, it is awaiting---of something else, of course.) That's given to you through the types.coroutine decorator in the standard library.
If you have any more questions, feel free to ask.