As I read more, I feel more stupid about aysnc in python. So I decided to ask for a direct answer. How can I change the following code (using async or similar approaches) to achieve the desired result? Additionally, how can I do it in flask or sanic?
import time
def long_job():
print('long job started')
time.sleep(5)
print('long job ended')
def main_job():
long_job()
time.sleep(1)
print('main job returned')
main_job()
# expected result:
# 'long job started'
# 'main job returned'
# 'long job ended'
Basically, I do NOT want to await for long_job to end before returning my main_job. Thank you in advance. :)
Await asyncio's sleep() to yield time to other jobs (if you don't need to await something else).
Use create_task() instead of await to start a job without blocking.
Finally, you have to start the main job using the event loop.
# Written in Python 3.7
import asyncio
async def long_job():
print('long job started')
await asyncio.sleep(5)
print('long job ended')
async def main_job():
asyncio.create_task(long_job())
await asyncio.sleep(1)
print('main job returned')
Your framework should start the event loop, you don't have to start it yourself. You can await or call create_task on main_job() from an async def function called by your framework, depending on if you want to block or not.
If you want to test this without a framework, you'll have to start the loop yourself using asyncio.run(). This will stop immediately after its task completes, even if other tasks haven't finished yet. But this is easy enough to work around:
async def loop_job():
asyncio.create_task(main_job())
while len(asyncio.Task.all_tasks()) > 1: # Any task besides loop_job()?
await asyncio.sleep(0.2)
asyncio.run(loop_job())
If you're implementing a framework yourself, you can use the more primitive loop.run_forever(), but you'd have to stop() it yourself.
Related
I'm trying to run some code asynchronously. My expectation is that the test coroutine should not block the print(running first) statement. This is because I've dispatched it to the event loop, and should be seeing the output of this command logged first.
import asyncio
async def test():
await asyncio.sleep(5)
print("I should run second")
asyncio.run(test())
print('running first')
Does anyone have any tips on how to how this code run so that print('running first') is ran before print("I should run second")? I believe this code should be non-blocking, so I'm confused as to why the order of print messages isn't matching my expectation.
I believe this is what you want:
import asyncio
async def test():
await asyncio.sleep(5)
print("I should run second")
async def main():
task1 = asyncio.create_task(test())
print('running first')
await task1
asyncio.run(main())
A more detail explaination:
asyncio.run() will try to wait all of the task inside it to finish before it continues.
In your code, you are running asyncio.run(test()) first and it will continue ONLY IF test() IS ENDED and you awaited the sleep. so test() will end after the sleep and run the print then the main print.
This is why your code delay so long before running. The solution to it is simple. Fire the task without waiting, which is what asyncio.create_task() is doing, I created a task, fire it but wait it at the end.
btw normally when you are using async, you will have a ton of task in a list. If you want to wait it as a list you should use gather():
import asyncio
async def test():
await asyncio.sleep(5)
print("I should run second")
async def main():
task_list = []
for _ in range(100):
task_list.append(asyncio.create_task(test()))
print('running first')
await asyncio.gather(*task_list)
asyncio.run(main())
I am trying to do something similar like C# ManualResetEvent but in Python.
I have attempted to do it in python but doesn't seem to work.
import asyncio
cond = asyncio.Condition()
async def main():
some_method()
cond.notify()
async def some_method():
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
main()
I want the some_method to start then wait until signaled to start again.
This code is not complete, first of all you need to use asyncio.run() to bootstrap the event loop - this is why your code is not running at all.
Secondly, some_method() never actually starts. You need to asynchronously start some_method() using asyncio.create_task(). When you call an "async def function" (the more correct term is coroutinefunction) it returns a coroutine object, this object needs to be driven by the event loop either by you awaiting it or using the before-mentioned function.
Your code should look more like this:
import asyncio
async def main():
cond = asyncio.Condition()
t = asyncio.create_task(some_method(cond))
# The event loop hasn't had any time to start the task
# until you await again. Sleeping for 0 seconds will let
# the event loop start the task before continuing.
await asyncio.sleep(0)
cond.notify()
# You should never really "fire and forget" tasks,
# the same way you never do with threading. Wait for
# it to complete before returning:
await t
async def some_method(cond):
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
asyncio.run(main())
I'm new to Python and have code similar to the following:
import time
import asyncio
async def my_async_function(i):
print("My function {}".format(i))
async def start():
requests = []
# Create multiple requests
for i in range(5):
print("Creating request #{}".format(i))
requests.append(my_async_function(i))
# Do some additional work here
print("Begin sleep")
time.sleep(10)
print("End sleep")
# Wait for all requests to finish
return await asyncio.gather(*requests)
asyncio.run(start())
No matter how long the "additional work" takes, the requests seem to only run after "End sleep". I'm guessing asyncio.gather is what actually begins to execute them. How can I have the requests (aka my_async_function()) start immediately, do additional work, and then wait for all to complete at the end?
Edit:
Per Krumelur's comments and my own findings, the following results in what I'm looking for:
import time
import asyncio
import random
async def my_async_function(i):
print("Begin function {}".format(i))
await asyncio.sleep(int(random.random() * 10))
print("End function {}".format(i))
async def start():
requests = []
# Create multiple requests
for i in range(10):
print("Creating request #{}".format(i))
requests.append(asyncio.create_task(my_async_function(i)))
# Do some additional work here
print("Begin sleep")
await asyncio.sleep(5)
print("End sleep")
# Wait for all requests to finish
return await asyncio.gather(*requests)
asyncio.run(start())
This only works if my_async_function and the "additional work" both are awaitable so that the event loop can give each of them execution time. You need create_task (if you know it's a coroutine) or ensure_future (if it could be a coroutine or future) to allow the requests to run immediately, otherwise they still end up running only when you gather.
time.sleep() is a synchronous operation
You’ll want to use the asynchronous sleep and await it,
E.g.
await asyncio.sleep(10)
Other async code will only run when the current task yields (I.e. typically when “await”ing something).
Using async code means you have to keep using async everywhere. Async operations are meant for I/O-bound applications. If “additional work” is mainly CPU-bound, you are better off using threads (but beware the global interpreter lock!)
I have a queue which stored on Redis lists. I'm trying to create async consumer for this queue. But couldn't call async function inside loop. Its working like sync function when I call.
import asyncio
async def worker():
print("starting sleep")
await asyncio.sleep(2)
print("slept")
async def main():
while True:
await worker()
asyncio.run(main())
Here is a short and simple example of mine implemantation. I'm expecting to see 'starting sleep' messages until first 'slept' message, it means for 2 seconds.
main is literally awaiting the completion of worker. Until worker is done, main won't progress. async tasks don't run in the background like in multithreading.
What you want is to keep launching new workers without awaiting each one of them. However, if you just keep doing this in a loop like this:
while True:
worker()
then you will never see any output of those workers, since this is an endless loop which never gives anything else the chance to run. You'd need to "break" this loop in some way to allow workers to progress. Here's an example of that:
import asyncio
async def worker():
print("starting sleep")
await asyncio.sleep(2)
print("slept")
async def main():
while True:
asyncio.ensure_future(worker())
await asyncio.sleep(0.5)
asyncio.run(main())
This will produce the expected outcome:
starting sleep
starting sleep
starting sleep
starting sleep
slept
starting sleep
slept
...
The await inside main transfers control back to the event loop, which now has the chance to run the piled up worker tasks, When those worker tasks await, they in turn transfer control back to the event loop, which will transfer it back to either main or a worker as their awaited sleep completes.
Note that this is only for illustration purposes; if and when you interrupt this program, you'll see notices about unawaited tasks which haven't completed. You should keep track of your tasks and await them all to completion at the end somewhere.
Here is an example using asyncio.wait:
import asyncio
async def worker():
print("starting sleep")
await asyncio.sleep(2)
print("slept")
async def main():
tasks = [worker() for each in range(10)]
await asyncio.wait(tasks)
asyncio.run(main())
It spawns all the workers together.
I try to cancel process when timeout but asyncio.wait_for not working. How do i cancel process when reached time out. My code below:
import asyncio
async def process():
# do something take a long time like this
for i in range(0,10000000000,1):
for j in range(0,10000000000,1):
continue
print('done!')
async def main():
# I want to cancel process when reached timeout
try:
await asyncio.wait_for(process(), timeout=1.0)
except asyncio.TimeoutError:
print('timeout!')
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
This doesn't work because your process function is async in name only - it doesn't await anything. That means that it finishes in its entirety without giving the event loop a chance to interrupt it. Since asyncio is cooperative (as are other async/await based systems), such a function is not a correctly written async function and cannot be interrupted.
If you add an await asyncio.sleep(0.001) into the inner loop (or anything else that awaits something that actually suspends), your code will work fine.