We Keep Coding

Python asyncio wait and notify

I am trying to do something similar like C# ManualResetEvent but in Python.
I have attempted to do it in python but doesn't seem to work.
import asyncio
cond = asyncio.Condition()
async def main():
some_method()
cond.notify()
async def some_method():
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
main()
I want the some_method to start then wait until signaled to start again.

This code is not complete, first of all you need to use asyncio.run() to bootstrap the event loop - this is why your code is not running at all.
Secondly, some_method() never actually starts. You need to asynchronously start some_method() using asyncio.create_task(). When you call an "async def function" (the more correct term is coroutinefunction) it returns a coroutine object, this object needs to be driven by the event loop either by you awaiting it or using the before-mentioned function.
Your code should look more like this:
import asyncio
async def main():
cond = asyncio.Condition()
t = asyncio.create_task(some_method(cond))
# The event loop hasn't had any time to start the task
# until you await again. Sleeping for 0 seconds will let
# the event loop start the task before continuing.
await asyncio.sleep(0)
cond.notify()
# You should never really "fire and forget" tasks,
# the same way you never do with threading. Wait for
# it to complete before returning:
await t
async def some_method(cond):
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
asyncio.run(main())

asyncio: wait for async callback with timeout

I am not much used to asyncio, so perhaps this question is trivial.
I have a code running asynchronously, which will run a callback when done (the callback can be callable or awaitable). I would like to wait for the callback to be called, with timeout. I sense that it is conceptually a task, but I am not sure how to create the task but wait for it somewhere else.
import asyncio, inspect
async def expensivefunction(callback):
# this is something which takes a lot of time
await asyncio.sleep(10)
# but eventually computes the result
result=10
# and calls the callback
callback(result)
if inspect.isawaitable(callback): await callback
# just print the result, for example
async def callback(result): print(result)
# main code async
async def myfunc():
await expensivefunction(callback=callback)
# this will wait for callback to be called within 5 seconds
# if not, exception is thrown
await asyncio.wait_for(...??,timeout=5)
asyncio.run(myfunc())
What would be the right approach to this?

Please find working example:
import asyncio
AWAIT_TIME = 5.0
async def expensive_function():
"""this is something which takes a lot of time"""
await asyncio.sleep(10)
result = 10
return result
def callback(fut: asyncio.Future):
"""just prints result. Callback should be sync function"""
if not fut.cancelled() and fut.done():
print(fut.result())
else:
print("No results")
async def amain():
"""Main async func in the app"""
# create task
task = asyncio.create_task(expensive_function())
task.add_done_callback(callback)
# try to await the task
try:
r = await asyncio.wait_for(task, timeout=AWAIT_TIME)
except asyncio.TimeoutError as ex:
print(ex)
else:
print(f"All work done fine: {r}")
finally:
print("App finished!")
if __name__ == '__main__':
asyncio.run(amain())
If any questions, please let me know.

Run tasks immediately and then wait

I'm new to Python and have code similar to the following:
import time
import asyncio
async def my_async_function(i):
print("My function {}".format(i))
async def start():
requests = []
# Create multiple requests
for i in range(5):
print("Creating request #{}".format(i))
requests.append(my_async_function(i))
# Do some additional work here
print("Begin sleep")
time.sleep(10)
print("End sleep")
# Wait for all requests to finish
return await asyncio.gather(*requests)
asyncio.run(start())
No matter how long the "additional work" takes, the requests seem to only run after "End sleep". I'm guessing asyncio.gather is what actually begins to execute them. How can I have the requests (aka my_async_function()) start immediately, do additional work, and then wait for all to complete at the end?
Edit:
Per Krumelur's comments and my own findings, the following results in what I'm looking for:
import time
import asyncio
import random
async def my_async_function(i):
print("Begin function {}".format(i))
await asyncio.sleep(int(random.random() * 10))
print("End function {}".format(i))
async def start():
requests = []
# Create multiple requests
for i in range(10):
print("Creating request #{}".format(i))
requests.append(asyncio.create_task(my_async_function(i)))
# Do some additional work here
print("Begin sleep")
await asyncio.sleep(5)
print("End sleep")
# Wait for all requests to finish
return await asyncio.gather(*requests)
asyncio.run(start())
This only works if my_async_function and the "additional work" both are awaitable so that the event loop can give each of them execution time. You need create_task (if you know it's a coroutine) or ensure_future (if it could be a coroutine or future) to allow the requests to run immediately, otherwise they still end up running only when you gather.

time.sleep() is a synchronous operation
You’ll want to use the asynchronous sleep and await it,
E.g.
await asyncio.sleep(10)
Other async code will only run when the current task yields (I.e. typically when “await”ing something).
Using async code means you have to keep using async everywhere. Async operations are meant for I/O-bound applications. If “additional work” is mainly CPU-bound, you are better off using threads (but beware the global interpreter lock!)

Where do I catch the KeyboardInterrupt exception in this async setup

I am working on a project that uses the ccxt async library which requires all resources used by a certain class to be released with an explicit call to the class's .close() coroutine. I want to exit the program with ctrl+c and await the close coroutine in the exception. However, it is never awaited.
The application consists of the modules harvesters, strategies, traders, broker, and main (plus config and such). The broker initiates the strategies specified for an exchange and executes them. The strategy initiates the associated harvester which collects the necessary data. It also analyses the data and spawns a trader when there is a profitable opportunity. The main module creates a broker for each exchange and runs it. I have tried to catch the exception at each of these levels, but the close routine is never awaited. I'd prefer to catch it in the main module in order to close all exchange instances.
Harvester
async def harvest(self):
if not self.routes:
self.routes = await self.get_routes()
for route in self.routes:
self.logger.info("Harvesting route {}".format(route))
await asyncio.sleep(self.exchange.rateLimit / 1000)
yield await self.harvest_route(route)
Strategy
async def execute(self):
async for route_dct in self.harvester.harvest():
self.logger.debug("Route dictionary: {}".format(route_dct))
await self.try_route(route_dct)
Broker
async def run(self):
for strategy in self.strategies:
self.strategies[strategy] = getattr(
strategies, strategy)(self.share, self.exchange, self.currency)
while True:
try:
await self.execute_strategies()
except KeyboardInterrupt:
await safe_exit(self.exchange)
Main
async def main():
await load_exchanges()
await load_markets()
brokers = [Broker(
share,
exchanges[id]["api"],
currency,
exchanges[id]["strategies"]
) for id in exchanges]
futures = [broker.run() for broker in brokers]
for future in asyncio.as_completed(futures):
executed = await future
return executed
if __name__ == "__main__":
status = asyncio.run(main())
sys.exit(status)
I had expected the close() coroutine to be awaited, but I still get an error from the library that I must explicitly call it. Where do I catch the exception so that all exchange instances are closed properly?

Somewhere in your code should be entry point, where event loop is started.
Usually it is one of functions below:
loop.run_until_complete(main())
loop.run_forever()
asyncio.run(main())
When ctrl+C happens KeyboardInterrupt can be catched at this line. When it happened to execute some finalizing coroutine you can run event loop again.
This little example shows idea:
import asyncio
async def main():
print('Started, press ctrl+C')
await asyncio.sleep(10)
async def close():
print('Finalazing...')
await asyncio.sleep(1)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
except KeyboardInterrupt:
loop.run_until_complete(close())
finally:
print('Program finished')

Python await a function with loop inside

I am using the Sanic as the server and try to handle multiple request concurrently.
I have used the await for the encode function(I use for loop to simulate do something) but when I try the time curl http://0.0.0.0:8000/ in two separate consoles, it doesn't run concurrently.
I have searched google but only find event_loop but it is to schedule registered conroutines.
How do I await the for loop so the requests won't be blocked?
Thank you.
from sanic import Sanic
from sanic import response
from signal import signal, SIGINT
import asyncio
import uvloop
app = Sanic(__name__)
#app.route("/")
async def test(request):
# await asyncio.sleep(5)
await encode()
return response.json({"answer": "42"})
async def encode():
print('encode')
for i in range(0, 300000000):
pass
asyncio.set_event_loop(uvloop.new_event_loop())
server = app.create_server(host="0.0.0.0", port=8000)
loop = asyncio.get_event_loop()
task = asyncio.ensure_future(server)
signal(SIGINT, lambda s, f: loop.stop())
try:
loop.run_forever()
except:
loop.stop()

Running for i in range() is blocking. If you change that to put your await asyncio.sleep(5) into the encode method, you will see that it operates as expected.
#app.route("/")
async def test(request):
await encode()
return response.json({"answer": "42"})
async def encode():
print('encode')
await asyncio.sleep(5)
When you call await encode() and encode is a blocking method, then it still is going to block because you are not "awaiting" anything else. Your thread is still locked up.
You could also add another worker:
app.create_server(worker=2)
Try looking through this answer

Since the async handler is actually running in an eventloop, it is running asynchronously as callback rather than concurrently.
loop.run_forever() would call loop._run_once over and over again to run all the registered event, each await would stop the coroutine and yield control back to the eventloop and eventloop arrange to run the next event.
So basically if you don't want blocking in a long running for-loop, you need to manually hand over the control back to the eventloop inside the for-loop, see the issue about relinquishing control:
async def encode():
print('encode')
for i in range(0, 300000000):
await asyncio.sleep(0)
Here is a quote from Guido:
asyncio.sleep(0) means just that -- let any other tasks run and then
come back here.