I want to call two function concurrently in asyncio, I do it with loop.create_task, but I find that it is not concurrently actually.
Here is my code:
import asyncio
async def foo(v):
print(f"start {v}")
for i in range(5):
print(f"work in {v}")
print(f"done {v}")
def schedule_foo():
print("out start 1")
loop.create_task(foo(1))
print("out start 2")
loop.create_task(foo(2))
loop = asyncio.get_event_loop()
schedule_foo()
loop.run_forever()
this will output:
out start 1
out start 2
start 1
work in 1
work in 1
work in 1
work in 1
work in 1
done 1
start 2
work in 2
work in 2
work in 2
work in 2
work in 2
done 2
As you can see the main loop is async, but sub task is actually sync.
My question is how can I make the function run in concurrently actually?
You forgot to use await:
import asyncio
async def foo(v):
print(f"start {v}")
for i in range(5):
print(f"work in {v}")
await asyncio.sleep(0.1)
print(f"done {v}")
def schedule_foo():
print("out start 1")
loop.create_task(foo(1))
print("out start 2")
loop.create_task(foo(2))
loop = asyncio.get_event_loop()
schedule_foo()
loop.run_forever()
Related
I have the following program here
`
import datetime
import asyncio
import time
import math
async def count1():
s = 0
print('start time count 1: ' +str(datetime.datetime.now()))
for i in range(100000000):
s += math.cos(i)
print('end time count 1: ' +str(datetime.datetime.now()))
return s
async def count2():
s = 0
print('start time count 2: ' +str(datetime.datetime.now()))
for i in range(1000000):
s += math.cos(i)
print('end time count 2: ' +str(datetime.datetime.now()))
return s
async def main():
start_time = time.time()
task = asyncio.gather(count1(), count2())
results = await task
end_time = time.time()
print(f"Result 1: {results[0]}")
print(f"Result 2: {results[1]}")
print(f"Total time taken: {end_time - start_time:.2f} seconds")
asyncio.run(main())
The output is
start time count 1: 2023-02-16 12:26:19.322523
end time count 1: 2023-02-16 12:26:40.866866
start time count 2: 2023-02-16 12:26:40.868166
end time count 2: 2023-02-16 12:26:41.055005
Result 1: 1.534369444774577
Result 2: -0.28870546796843
Total time taken: 21.73 seconds
I am trying to get count1() and count2() to start working at the same time, as it seen it the output it's not happening. count2() only starts after count1() has ended, and I am not sure why.
I also tried replacing the lines in main() with:
task1 = asyncio.create_task(count1())
task2 = asyncio.create_task(count2())
result1 = await task1
result2 = await task2
not also does not result in count1() and count2() starting at the same time.
async is essentially cooperative multitasking. Neither function awaits, so they hog the entire process for as long as they run and don't yield to other functions.
You could add a strategic
if i % 10000 == 0:
await asyncio.sleep(0)
in the loop so they "give way" to other async coroutines.
Async doesn’t mean “running at the same time”. Async means that when one of the functions waits (typically for some I/O), the control can be passed to another function.
In practice async can be used e.g. to have one process wait for several responses from web at the same time. It won’t allow you to e.g. run the same computation on several CPU cores (this stuff is generally hard to achieve in Python due to the infamous GIL problem).
I have a function which constantly yields some objects, say 1 per second and a handler which works 2 seconds and handles this objects. For example:
from time import sleep
import asyncio
from datetime import datetime
def generator():
i = 0
while True:
yield i
i += 1
sleep(1)
def handler(number):
sleep(2)
if number % 2 == 0:
print(str(number) + ' is even')
else:
print(str(number) + ' is odd')
for number in generator():
handler(number)
So, for example '2 is even' is printed 6 seconds after the program starts. How do I reduce this time to 4 seconds ( 2 seconds for generator + 2 seconds for handler) using asyncio? I need to set up asynchronous handling of the numbers.
You need couple of changes here:
your generator currently is a "generator", change it to "asynchronous generator" so that you can use async for. This way it can give the control back to eventloop.
Use async version of sleep in asyncio library: asyncio.sleep. time.sleep doesn't cooperate with other tasks.
Change your handler sync function to a "coroutine".
import asyncio
async def generator():
i = 0
while True:
yield i
i += 1
await asyncio.sleep(1)
async def handler(number):
await asyncio.sleep(2)
if number % 2 == 0:
print(str(number) + " is even")
else:
print(str(number) + " is odd")
async def main():
async for number in generator():
asyncio.create_task(handler(number))
asyncio.run(main())
Now, your first task is main, asyncio.run() automatically creates it as Task. Then when this task is running, it iterates asynchronously through the generator(). The values are received then for each number, you create a new Task out of handler coroutine.
This way the sleep times are overlapped. When it waits for new number for 1 second, it also actually waits for handler() one second. Then when the number is received, one second of handler() task is already passed, it only needs 1 second.
You can see the number of tasks if you want:
async def main():
async for number in generator():
print(f"Number of all tasks: {len(asyncio.all_tasks())}")
asyncio.create_task(handler(number))
Because each handler sleeps 2 seconds, and your number generator sleeps 1 seconds, You see that in every iteration 2 Tasks are exist in event loop. Change await asyncio.sleep(1) to await asyncio.sleep(0.5) in generator coroutine, you will see that 4 tasks are in event loop in every iteration.
Answer to the comment:
Can I do the same thing if I use API which doesn't let me create
asynchronous generator, but just a normal one? Can I still
asynchronously handle yielded objects?
Yes you can. Just note that if you don't have asynchronous generator, you can't use async for, which means your iteration is synchronous. But, you have to do a little trick for it to work. When your main() task is being executed, it constantly get a value from generator generator and creates a Task for it, but it doesn't give a chance to other tasks to run. You need await asyncio.sleep(0):
import asyncio
import time
def generator():
i = 0
while True:
yield i
i += 1
time.sleep(1)
async def handler(number):
await asyncio.sleep(2)
if number % 2 == 0:
print(str(number) + " is even")
else:
print(str(number) + " is odd")
async def main():
for number in generator():
print(f"Number of all tasks: {len(asyncio.all_tasks())}")
asyncio.create_task(handler(number))
await asyncio.sleep(0)
asyncio.run(main())
I am creating a python code that does the following:
1 - creates an asynchronous function called "nums" than prints random numbers from 0 to 10 every 2 seconds.
2 - execute a function that prints "hello world" every 5 seconds.
3 - if the function "nums" prints 1 or 3, break the loop and end the program.
The thing is, it's not breaking out of the loop. What I am supposed to do? Thank you.
import asyncio
import random
async def nums():
while True:
x = print(random.randint(0, 10))
await asyncio.sleep(2)
if x == 1 or x == 3:
break
async def world():
while True:
print("hello world")
await asyncio.sleep(5)
async def main():
while True:
await asyncio.wait([nums(), world()])
if __name__ == '__main__':
asyncio.run(main())
You can use return_when=asyncio.FIRST_COMPLETED parameter in asyncio.wait:
import asyncio
import random
async def nums():
while True:
x = random.randint(0, 10)
print(x)
if x in {1, 3}
break
await asyncio.sleep(2)
async def world():
while True:
print("hello world")
await asyncio.sleep(5)
async def main():
task1 = asyncio.create_task(nums())
task2 = asyncio.create_task(world())
await asyncio.wait({task1, task2}, return_when=asyncio.FIRST_COMPLETED)
if __name__ == "__main__":
asyncio.run(main())
Prints (for example):
5
hello world
6
6
hello world
1
<program ends here>
I have an async method, as shown below.
I pass in lists of 1000 numbers, where the method will pass in each number to a helper function which will return something from a website.
I have a global variable called count, which i surround with locks to make sure it doesnt get changed by anything else
I use add_done_callback with the task to make this method async.
The goal is to keep sending a number in the list of 1000 numbers to the server, and only when the server returns data (can take anywhere from 0.1 to 2 seconds), to pause, write the data to a sql database, and then continue
The code works as expected without locks, or without making the callback function, (which is named 'function' below) asyncrounous. But adding locks gives me an error: RuntimeWarning: coroutine 'function' was never awaited self._context.run(self._callback, *self._args) RuntimeWarning: Enable tracemalloc to get the object allocation traceback
I am super new to async in python so any help/advice is greatly appriciated
My code is shown below. It is just a simple draft:
import time
import random
import asyncio
# from helper import get_message_from_server
async def get(number):
# get_message_from_server(number), which takes somewhere between 0.1 to 2 seconds
await asyncio.sleep(random.uniform(0.1, 2))
s = 'Done with number ' + number
return s
async def function(future, lock):
global count
print(future.result())
# write future.result() to db
acquired = await lock.acquire()
count -= 1 if (count > 1) else 0
lock.release()
async def main(numbers, lock):
global count
count = 0
for i, number in enumerate(numbers):
print('number:', number, 'count:', count)
acquired = await lock.acquire()
count += 1
lock.release()
task = asyncio.create_task(get(number))
task.add_done_callback(
lambda x: function(x, lock)
)
if (count == 50):
print('Reached 50')
await task
acquired = await lock.acquire()
count = 0
lock.release()
if (i == len(numbers) - 1):
await task
def make_numbers():
count = []
for i in range(1001):
count.append(str(i))
return count
if __name__ == '__main__':
numbers = make_numbers()
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
try:
loop.run_until_complete(main(numbers, lock))
except Exception as e:
pass
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.stop()
The above comment helped a lot
This is what the final working code looks like:
import time
import random
import asyncio
from functools import partial
# from helper import get_message_from_server
async def get(number):
# get_message_from_server(number), which takes somewhere between 0.1 to 2 seconds
await asyncio.sleep(random.uniform(0.1, 2))
s = 'Done with number ' + number
return s
def function(result, lock):
print(result.result())
async def count_decrement(lock):
global count
print('in count decrement')
acquired = await lock.acquire()
count -= 1 if (count > 1) else 0
lock.release()
asyncio.create_task(count_decrement(lock))
async def main(numbers, lock):
global count
count = 0
for i, number in enumerate(numbers):
print('number:', number, 'count:', count)
acquired = await lock.acquire()
count += 1
lock.release()
task = asyncio.create_task(get(number))
task.add_done_callback(partial(function, lock = lock))
if (count == 50):
print('Reached 50')
await task
acquired = await lock.acquire()
count = 0
lock.release()
if (i == len(numbers) - 1):
await task
def make_numbers():
count = []
for i in range(1001):
count.append(str(i))
return count
if __name__ == '__main__':
numbers = make_numbers()
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
try:
loop.run_until_complete(main(numbers, lock))
except Exception as e:
pass
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.stop()
I am learning asyncio library to do some tasks I want to achieve. I wrote the following code to teach myself on how to be able to switch to so another task while the original one is being executed. As you can see below, the summation() should be performed until a condition is met where it should jump to the secondaryTask(). After secondaryTask() is finished, it should return back to summation() where hopefully it gets finished. The potential results should be sum=1225 and mul=24.
import asyncio, time
async def summation():
print('Running summation from 0 to 50:')
sum = 0
for i in range(25):
sum = sum + i
if i != 25:
time.sleep(0.1)
else:
await asyncio.sleep(0) # pretend to be non-blocking work (Jump to the next task)
print('This message is shown because summation() is completed! sum= %d' % sum)
async def secondaryTask():
print('Do some secondaryTask here while summation() is on progress')
mul = 1
for i in range(1, 5):
mul = mul * i
time.sleep(0.1)
await asyncio.sleep(0)
print('This message is shown because secondaryTask() is completed! Mul= %d' % mul)
t0 = time.time()
ioloop = asyncio.get_event_loop()
tasks = [ioloop.create_task(summation()), ioloop.create_task(secondaryTask())]
wait_tasks = asyncio.wait(tasks)
ioloop.run_until_complete(wait_tasks)
ioloop.close()
t1 = time.time()
print('Total time= %.3f' % (t1-t0))
This code does not perform as expected because sum=300 as oppose to be sum=1225. Clearly that summation() does not continue while secondaryTask() is being processed. How can I modify summation() to be able to do the summation of the remaining 25 values on the background?
Thank you
It's just your careless. You want to run summation from 0 to 50, your summation function should be for i in range(50)