This question already has answers here:
How to run functions in parallel?
(8 answers)
Closed 2 years ago.
I need to run two python functions simultaneously. How can I achieve that? Example:
import time
def short_task():
time.sleep(2)
def long_task():
time.sleep(4)
short_task()
long_task()
I expect the whole code to finish in 4 seconds (instead of 6).
It depends on what the functions are doing.
If it is something that involves waiting (such as time.sleep() or waiting for a response) use a Thread.
import time
from threading import Thread
def short_task():
time.sleep(2)
def long_task():
time.sleep(4)
Thread(target=short_task).start()
Thread(target=long_task).start()
If you have a long task that requires a lot of computing, use a Process:
import time
from multiprocessing import Process
def short_task():
time.sleep(2)
def long_task():
time.sleep(4)
if __name__ == '__main__':
Process(target=short_task).start()
Process(target=long_task).start()
One of your options is to use the asyncio module.
First of all, import the asyncio module with import asyncio. Replace your def functions to async def functions.
Second of all, replace time.sleep() with await asyncio.sleep(). You no longer need the time module because of this step.
Thirdly, create a new function, normally called main(). You can take the following code snippet for reference:
async def main():
task1 = asyncio.create_task(
short_task())
task2 = asyncio.create_task(
long_task())
await task1
await task2
Finally, run the whole main() code with asyncio.run(main()). Your final code should look like this:
import asyncio
async def short_task():
await asyncio.sleep(2)
async def long_task():
await asyncio.sleep(4)
async def main():
task1 = asyncio.create_task(
short_task())
task2 = asyncio.create_task(
long_task())
await task1
await task2
You may use a simple code snippet to proof that the whole process took 4 seconds.
Related
Is there a way to create a secondary asyncio loop(or prioritize an await) that when awaited does not pass control back to the main event loop buts awaits for those 'sub' functions? IE
import asyncio
async def priority1():
print("p1 before sleep")
await asyncio.sleep(11)
print("p1 after sleep")
async def priority2():
print("p2 before sleep")
await asyncio.sleep(11)
print("p2 after sleep")
async def foo():
while True:
print("foo before sleep")
#do not pass control back to main event loop here but run these 2 async
await asyncio.gather(priority1(),priority2())
print("foo after sleep")
async def bar():
while True:
print("bar before sleep")
await asyncio.sleep(5)
print("bar after sleep")
async def main():
await asyncio.gather(foo(),bar())
asyncio.run(main())
I would like foo to wait for priority1/2 to finish before passing control back to the main event loop.
Right now it would go:
foo before sleep
bar before sleep
p1 before sleep
p2 before sleep
bar after sleep
I would like to see:
foo before sleep
bar before sleep
p1 before sleep
p2 before sleep
p1 after sleep
p2 after sleep
bar after sleep
Is this possible?
thanks
It is not possible to run two event loops on the same thread. The code in asyncio and specs even look like they were thought in a way to permit that - but afterwards the API bent in a way it is no longer possible (for example, the explicit loop parameter for several of the relevant calls has been deprecated and removed)
In the same line, there is no way to prioritize a subset of tasks in the running loop. I answered a question on this line a couple weeks ago, and managed to get to a synchronization primitive to be used in place of asyncio.sleep which could take priority into account - but it requires all participating tasks to call it, so it would not be much different of a lock or something (I will link to it bellow - the idea is: your code call await custom.sleep() at certain points: it will only return when there are no other higher-prioritized tasks also calling that custom.sleep() ) -check it here: Execute asyncio task as soon as possible
When wrtting that code, I realized that it is possible to write an event loop which could take into account a "priority" attribute on tasks. But having a production-grade for this requires some non-hobby work: by using that loop, you could get what you are asking for, with no changes needed in the tasks code.
However, I think running a secondary loop in another thread, and then waiting synchronously on that thread to complete is a way to get your things accomplished.
import asyncio
import threading
def priority(func):
async def wrapper(*args, **kwargs):
result = None
def runner(*args, **kw):
nonlocal result
result = asyncio.run(func(*args, **kw))
t = threading.Thread(target=runner, args=args, kwargs=kwargs)
await asyncio.sleep(0)
t.start()
# if one wants to perform limited yields to the main loop, it should be done here
t.join()
return result
return wrapper
async def priority1():
print("p1 before sleep")
await asyncio.sleep(.11)
print("p1 after sleep")
async def priority2():
print("p2 before sleep")
await asyncio.sleep(.11)
print("p2 after sleep")
#priority
async def foo():
print("foo before sleep")
#do not pass control back to main event loop here but run these 2 async
await asyncio.gather(priority1(),priority2())
print("foo after sleep")
async def bar():
print("bar before sleep")
await asyncio.sleep(.05)
print("bar after sleep")
async def main():
await asyncio.gather(foo(),bar())
asyncio.run(main())
I am trying to do something similar like C# ManualResetEvent but in Python.
I have attempted to do it in python but doesn't seem to work.
import asyncio
cond = asyncio.Condition()
async def main():
some_method()
cond.notify()
async def some_method():
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
main()
I want the some_method to start then wait until signaled to start again.
This code is not complete, first of all you need to use asyncio.run() to bootstrap the event loop - this is why your code is not running at all.
Secondly, some_method() never actually starts. You need to asynchronously start some_method() using asyncio.create_task(). When you call an "async def function" (the more correct term is coroutinefunction) it returns a coroutine object, this object needs to be driven by the event loop either by you awaiting it or using the before-mentioned function.
Your code should look more like this:
import asyncio
async def main():
cond = asyncio.Condition()
t = asyncio.create_task(some_method(cond))
# The event loop hasn't had any time to start the task
# until you await again. Sleeping for 0 seconds will let
# the event loop start the task before continuing.
await asyncio.sleep(0)
cond.notify()
# You should never really "fire and forget" tasks,
# the same way you never do with threading. Wait for
# it to complete before returning:
await t
async def some_method(cond):
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
asyncio.run(main())
I'm a python beginner and taking from https://www.youtube.com/watch?v=iG6fr81xHKA&t=269s about the power of asyncio, I tried to use this example shown and repurpose it to execute 10 times. Here's a code snippet
def main(x):
print("Hello")
time.sleep(3)
print("World!")
And so I tried to do it in a asyncio fashion however it doesn't execute asynchronously.
Here's so far what I've tried. What am I doing wrong?
import time
import asyncio
async def main(x):
print(f"Starting Task {x}")
await asyncio.sleep(3)
print(f"Finished Task {x}")
async def async_io():
for i in range(10):
await main(i)
if __name__ == "__main__":
start_time = time.perf_counter()
asyncio.run(async_io())
print(f"Took {time.perf_counter() - start_time} secs")
I've also tried to use queue_task in asyncio.
Using await, by definition, waits for the task main to finish. So your code as-is is no different from the synchronous code you posted above. If you want to run them at the same time (asynchronously), while waiting for the results, you should use asyncio.gather or asyncio.wait instead.
async def async_io():
tasks = []
for i in range(10):
tasks += [main(i)]
await asyncio.gather(*tasks)
If you don't care to wait for all of the main() calls to finish, you can also just use asyncio.create_task(main(i)), which creates a Task object and schedule its execution in the background. In this case, def async_io() doesn't need to be async anymore.
I'm new to Python and have code similar to the following:
import time
import asyncio
async def my_async_function(i):
print("My function {}".format(i))
async def start():
requests = []
# Create multiple requests
for i in range(5):
print("Creating request #{}".format(i))
requests.append(my_async_function(i))
# Do some additional work here
print("Begin sleep")
time.sleep(10)
print("End sleep")
# Wait for all requests to finish
return await asyncio.gather(*requests)
asyncio.run(start())
No matter how long the "additional work" takes, the requests seem to only run after "End sleep". I'm guessing asyncio.gather is what actually begins to execute them. How can I have the requests (aka my_async_function()) start immediately, do additional work, and then wait for all to complete at the end?
Edit:
Per Krumelur's comments and my own findings, the following results in what I'm looking for:
import time
import asyncio
import random
async def my_async_function(i):
print("Begin function {}".format(i))
await asyncio.sleep(int(random.random() * 10))
print("End function {}".format(i))
async def start():
requests = []
# Create multiple requests
for i in range(10):
print("Creating request #{}".format(i))
requests.append(asyncio.create_task(my_async_function(i)))
# Do some additional work here
print("Begin sleep")
await asyncio.sleep(5)
print("End sleep")
# Wait for all requests to finish
return await asyncio.gather(*requests)
asyncio.run(start())
This only works if my_async_function and the "additional work" both are awaitable so that the event loop can give each of them execution time. You need create_task (if you know it's a coroutine) or ensure_future (if it could be a coroutine or future) to allow the requests to run immediately, otherwise they still end up running only when you gather.
time.sleep() is a synchronous operation
You’ll want to use the asynchronous sleep and await it,
E.g.
await asyncio.sleep(10)
Other async code will only run when the current task yields (I.e. typically when “await”ing something).
Using async code means you have to keep using async everywhere. Async operations are meant for I/O-bound applications. If “additional work” is mainly CPU-bound, you are better off using threads (but beware the global interpreter lock!)
I want to use async in python like c# or javascript.For example I want my program not to block next codes or maybe buttons in app while sending a request.I wrote some code about that.However I dont know whether this usage is true or false.I can't understand asyncio
import asyncio
import threading
import time
async def waitForMe(name):
for i in range(5):
await asyncio.sleep(1)
print(name)
async def main():
task1 = asyncio.create_task(waitForMe("task1"))
task2 = asyncio.create_task(waitForMe("task2"))
task3 = asyncio.create_task(waitForMe("task3"))
await task1
await task2
await task3
def mfunction():
asyncio.run(main())
t1=threading.Thread(target=mfunction)
t1.start()
for i in range(3):
time.sleep(1)
print("main")
I'd really recommend this excellent asyncio walkthrough, which should answer most of your questions.
A quote from the mentioned article in the light of your code:
[...] async IO is a single-threaded, single-process design: it uses cooperative multitasking, a term that [...] gives a feeling of concurrency despite using a single thread in a single process.
If you don't want your program to block while processing (IO) requests (as stated in your question), concurrency is sufficient (and you don't need (multi)threading)!
Concurrency [...] suggests that multiple tasks have the ability to run in an overlapping manner.
I'll repeat the exact example from the mentioned article, which has a similar structure as your example:
#!/usr/bin/env python3
# countasync.py
import asyncio
async def count():
print("One")
await asyncio.sleep(1)
print("Two")
async def main():
await asyncio.gather(count(), count(), count())
if __name__ == "__main__":
import time
s = time.perf_counter()
asyncio.run(main())
elapsed = time.perf_counter() - s
print(f"{__file__} executed in {elapsed:0.2f} seconds.")
This runs as follows:
$ python3 countasync.py
One
One
One
Two
Two
Two
countasync.py executed in 1.01 seconds.
Note that this examples uses asyncio.gather to kick-off the three count() processes in a non-blocking manner. Putting three await count() statements after one another won't work.
As far as I can see, this is exactly what you are looking for. As demonstrated, you don't need threading to achieve this.