Basically I have synchronous code something similar to this:
def f(.):
...
def call(.):
..Some sync code..
try:
resp = f(..)
...some more transformations of resp...
return ..
except:
..Some error handling..
async def af(.):
...
Basically I want to dynamically alter the code of call so that it can call and await af function instead of f. Is there a way around this? I have found syncit on github, but it doesn't seem to be a solution to me as you have to rewerite the code to async first and then downgrade it to sync. Any help will be appreciated.
In asyncio world every coroutine can be executied either inside other coroutine (using await) or by event loop blocking call (using run_until_complete()).
You can't await coroutine inside regular function since this function's call would be blocking, while coroutine to be awaited needs blocking event loop execution. This is how asyncio designed to work.
If you ok with idea of your call() function being blocking and have access to f() implementation, you can execute coroutine inside f() runnning event loop:
async def af(.):
...
def f(.):
loop = asyncio.get_event_loop()
return loop.run_until_complete(af())
def call(.):
..Some sync code..
try:
resp = f(..)
...some more transformations of resp...
return ..
except:
..Some error handling..
If you don't have access to f() implementation, I don't believe you would be able to alter call() to await coroutines (without some ugly monkey-patching).
I think rewriting call() to be async would be only good option then.
Here are two options mixing synchronous and asynchronous code.
Given
import asyncio
import trio
from unsync import unsync
Code
Option 1 - trio
Given sync code (call), call an async function via trio (async_div):
# Sync code
def call(x, y):
"""Return an awaited result."""
try:
resp = async_div(x, y)
except ZeroDivisionError:
return "Caught an exception."
return resp
# Wrapper
def async_div(*args):
"""Return results from an async function."""
return trio.run(_async_div, *args)
# Async code
async def _async_div(x, y):
"""Return an async result."""
await trio.sleep(3)
return x / y
Option 2 - unsync
Decorate the async code with #unsync and call the result():
# Sync code
def call(x, y):
"""Return an awaited result."""
try:
resp = async_div(x, y).result()
except ZeroDivisionError:
return "Caught an exception."
return resp
# Async code
#unsync
async def async_div(x, y):
"""Return an async result."""
await asyncio.sleep(3)
return x / y
Demo
After the prescribed delay (3 secs), the results are the same:
call(0, 1)
# 0.0
call(1, 0)
# 'Caught an exception.'
See Also
Some additional resources:
docs on trio
Talk Python interview with trio creator N. Smith for more details and his philosophy on asynchronous programming.
Python Bytes episode 73, item 6 on quick unsync details
M. Kennedy's webinar demo on using unsync to speed up requests
Related
In an API developed using FastAPI framework, I am using asyncio to make calls to solr collections, but when I am calling asyncio.run(man(a,b)) from a python file query.py then I am getting asyncio.run() cannot be called from a running event loop.
in controller.py
#router.api_route()
async def api_call(a,b):
#calling a function
resp = query(a,b)
return resp
in query.py
def query(a,b):
result = asyncio.run(man(a,b))
return result
in database_call.py
async def man(a,b)
async with aiohttp.ClientSession() as session:
url = ''
async with session.get(pokemon_url) as resp:
result = await resp.json()
return result
when I am calling asyncio.run(man(a,b)) from query then I am getting asyncio.run() cannot be called from a running event loop. Kindly help me resolve the issue.
I tried:
in query.py
def query(a,b):
loop = asyncio.get_event_loop
result = loop.create_task(man(a,b))
return result
then I am getting <coroutine object main at 0x0394999ejt>
The docs say that you should have only one call to asyncio.run in a program. "Should" doesn't mean the same thing as "must", so it's not a requirement. But it's a good guideline.
Solution 1: get rid of query entirely, and just await the coroutine man() directly.
#router.api_route()
async def api_call(a,b):
return await man(a, b)
Solution 2: declare query to be an async def function, and await it:
#router.api_route()
async def api_call(a,b):
#calling a function
return await query(a,b)
async def query(a,b):
return await man(a,b)
Solution 3: Do not declare query to be async def, but have it return an awaitable object. This is similar to what you tried in your last listing, but you need to await the result.
#router.api_route()
async def api_call(a,b):
#calling a function
return await query(a,b)
def query(a, b):
return asyncio.create_task(man(a, b))
Solution 4: run query in another thread using the asyncio.to_thread function.
Solution 5: run query in a ThreadPool or ProcessPool.
The whole idea of asyncio (or any other form of parallel processing) is to allow the use of a task or a coroutine that does not finish immediately. It runs at a later time, and its result is not available until enough time has passed. Whether you use threading, multiprocessing or asyncio the situation is the same. If you need the answer from the coroutine/task/function in order for your program to continue, you need to wait for it somehow.
I have a function in python that should kick off an asynchronous 'slow' function, and return before that 'slow' function completes.
async def doSomethingSlowButReturnQuickly():
asyncio.create_task(_mySlowFunction())
return "Returning early! Slow function is still running."
async def _mySlowFunction():
// Do some really slow stuff that isn't blocking for our other function.
Running this still seems to cause the 'return' to not happen until AFTER my async task has completed.
How do I correct this?
(And apologies, this is day #2 writing Python)
This isn't exactly what you asked, but you can run code in an async executor and continue while it is still running:
from concurrent.futures import ThreadPoolExecutor
from time import sleep
def blocking_code():
sleep(2)
print('inner')
return 'result'
def main():
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(blocking_code)
print('after')
print(future.result())
return
if __name__ == '__main__':
main()
Output:
after
inner
result
Maybe it's a dull answer, but if you want to stick with asyncio, I would advise to be sure using await in everyasync def functions.
async def doSomethingSlowButReturnQuickly():
task = asyncio.create_task(_mySlowFunction())
# must await something here!
return "Returning early! Slow function is still running."
async def _mySlowFunction():
# must await something here
If you don't need to get the slow function return value, why don't you directly call the two coroutines from a main function?
import asyncio
async def doSomethingSlowButReturnQuickly():
await asyncio.sleep(0.1) # must await something here!
return "Returning early! Slow function is still running."
async def _mySlowFunction():
await asyncio.sleep(10) # must await something here
async def main():
tasks = [_mySlowFunction(), doSomethingSlowButReturnQuickly()]
asyncio.gather(*tasks)
asyncio.run(main())
I am not much used to asyncio, so perhaps this question is trivial.
I have a code running asynchronously, which will run a callback when done (the callback can be callable or awaitable). I would like to wait for the callback to be called, with timeout. I sense that it is conceptually a task, but I am not sure how to create the task but wait for it somewhere else.
import asyncio, inspect
async def expensivefunction(callback):
# this is something which takes a lot of time
await asyncio.sleep(10)
# but eventually computes the result
result=10
# and calls the callback
callback(result)
if inspect.isawaitable(callback): await callback
# just print the result, for example
async def callback(result): print(result)
# main code async
async def myfunc():
await expensivefunction(callback=callback)
# this will wait for callback to be called within 5 seconds
# if not, exception is thrown
await asyncio.wait_for(...??,timeout=5)
asyncio.run(myfunc())
What would be the right approach to this?
Please find working example:
import asyncio
AWAIT_TIME = 5.0
async def expensive_function():
"""this is something which takes a lot of time"""
await asyncio.sleep(10)
result = 10
return result
def callback(fut: asyncio.Future):
"""just prints result. Callback should be sync function"""
if not fut.cancelled() and fut.done():
print(fut.result())
else:
print("No results")
async def amain():
"""Main async func in the app"""
# create task
task = asyncio.create_task(expensive_function())
task.add_done_callback(callback)
# try to await the task
try:
r = await asyncio.wait_for(task, timeout=AWAIT_TIME)
except asyncio.TimeoutError as ex:
print(ex)
else:
print(f"All work done fine: {r}")
finally:
print("App finished!")
if __name__ == '__main__':
asyncio.run(amain())
If any questions, please let me know.
I am using FastApi and have one endpoint.
I have two long running functions which I want to run concurrently using asyncio
Therefore, I have created two functions:
async def get_data_one():
return 'done_one'
async def get_data_two():
return 'done_two'
These functions get data from external webservices.
I want to execute them concurrently so I have created a another function that does it:
async def get_data():
loop = asyncio.get_event_loop()
asyncio.set_event_loop(loop)
task_1 = loop.create_task(get_data_one)
task_2 = loop.create_task(get_data_two)
tasks = (task_1, task_2)
first, second = loop.run_until_complete(asyncio.gather(*tasks))
loop.close()
# I will then perform cpu intensive computation on the results
# for now - assume i am just concatenating the results
return first + second
Finally, I have my endpoint:
#app.post("/result")
async def get_result_async():
r = await get_data()
return r
Even this simple example breaks and I get the following exception when I hit the endpoint:
RuntimeError: This event loop is already running
ERROR: _GatheringFuture exception was never retrieved
future: <_GatheringFuture finished exception=AttributeError("'function' object has no attribute 'send'",)>
AttributeError: 'function' object has no attribute 'send'
This is a simplified code but I would really appreciate how to do it the right way.
When in FastAPI context, you never need to run an asyncio loop; it's always running for as long as your server process lives.
Therefore, all you need is
import asyncio
async def get_data_one():
return "done_one"
async def get_data_two():
return "done_two"
async def get_data():
a, b = await asyncio.gather(get_data_one(), get_data_two())
return a + b
##route decorator here...
async def get_result_async():
r = await get_data()
print("Get_data said:", r)
return r
# (this is approximately what is done under the hood,
# presented here to make this a self-contained example)
asyncio.run(get_result_async())
It's as simple as:
async def get_data():
first, second = await asyncio.gather(
get_data_one(),
get_data_two(),
)
return first + second
I am trying to create a periodic task for an asyncio event loop as shown below, however I am getting a "RuntimeError: cannot reuse already awaited coroutine" exception. Apparently, asyncio does not allow for the same awaitable function to be awaited as discussed in this bug thread. This is how I tried to implement it:
import asyncio
class AsyncEventLoop:
def __init__(self):
self._loop = asyncio.get_event_loop()
def add_periodic_task(self, async_func, interval):
async def wrapper(_async_func, _interval):
while True:
await _async_func # This is where it goes wrong
await asyncio.sleep(_interval)
self._loop.create_task(wrapper(async_func, interval))
return
def start(self):
self._loop.run_forever()
return
Because of my while loop, the same awaitable function (_async_func) would be executed with a sleep interval in between. I got my inspiration for the implementation of periodic tasks from How can I periodically execute a function with asyncio?
.
From the bug thread mentioned above, I infer that the idea behind the RuntimeError was so that developers wouldn't accidentally await the same coroutine twice or more, as the coroutine would be marked as done and yield None instead of the result. Is there a way I can await the same function more than once?
It seems you are confusing async functions (coroutine functions) with coroutines - values that these async functions produce.
Consider this async function:
async def sample():
await asyncio.sleep(3.14)
You are passing result of its call: add_periodic_task(sample(), 5).
Instead, you should pass async function object itself: add_periodic_task(sample, 5), and call it within your wrapper:
while True:
await _async_func()
await asyncio.sleep(_interval)