I am writing a minimalist code based on asyncio and composed of tasks, workers and a queue:
Workers look in the queue for a task and execute it
Tasks are represented as a random asyncio.sleep()
When a task is done it adds two more tasks to the queue
Two restrictions:
10 max worker at a time
100 tasks should be executed in total, the script should end then
To monitor the total number of tasks executed, I use a global variable task_queued updated each time a task is being added to the queue.
I'm sure there's a better, pythonic way, to do that and not using a global variable, but all the solution I came up with are much more complicated.
I'm missing something here, any clue ?
Here's my code:
import asyncio
from random import random
import sys
MAX_WORKERS = 10
MAX_TASKS = 100
task_queued = 0
async def task(queue, id="1"):
global task_queued
sleep_time = 0.5 + random()
print(' Begin task #{}'.format(id))
await asyncio.sleep(sleep_time)
if task_queued < MAX_TASKS:
await queue.put(id + ".1")
task_queued += 1
if task_queued < MAX_TASKS:
await queue.put(id + ".2")
task_queued += 1
print(' End task #{} ({} item(s) in the queue)'.format(id, queue.qsize()))
async def worker(worker_id, queue):
while True:
task_id = await queue.get()
print('Worker #{} takes charge of task {}'.format(worker_id, task_id))
await task(queue, task_id)
queue.task_done()
async def main():
global task_queued
print('Begin main \n')
queue = asyncio.Queue()
await queue.put("1") # We add one task to the queue
task_queued += 1
workers = [asyncio.create_task((worker(worker_id + 1, queue))) for worker_id in range(MAX_WORKERS)]
await queue.join()
print('Queue is empty, {} tasks completed'.format(task_queued))
for w in workers:
w.cancel()
print('\n End main')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
try:
loop.run_until_complete(main())
except KeyboardInterrupt:
print('\nBye bye')
sys.exit(0)
Thank to user4815162342 for the answer, here's the code if anyone is interested
import asyncio
from random import random
import sys
class Factory:
"""
Factory
"""
def __init__(self, max_workers, max_tasks):
self.task_queued = 0
self.max_workers = max_workers
self.max_tasks = max_tasks
self.queue = asyncio.Queue()
async def task(self, task_id):
sleep_time = 0.5 + random()
print(' Begin task #{}'.format(task_id))
await asyncio.sleep(sleep_time)
if self.task_queued < self.max_tasks:
await self.queue.put(task_id + ".1")
self.task_queued += 1
if self.task_queued < self.max_tasks:
await self.queue.put(task_id + ".2")
self.task_queued += 1
print(' End task #{} ({} item(s) in the queue)'.format(task_id, self.queue.qsize()))
async def worker(self, worker_id):
while True:
task_id = await self.queue.get()
print('Worker #{} takes charge of task {}'.format(worker_id, task_id))
await self.task(task_id)
self.queue.task_done()
async def organize_work(self):
print('Begin work \n')
await self.queue.put("1") # We add one task to the queue to start
self.task_queued += 1
workers = [asyncio.create_task((self.worker(worker_id + 1))) for worker_id in range(self.max_workers)]
await self.queue.join()
print('Queue is empty, {} tasks completed'.format(self.task_queued))
for w in workers:
w.cancel()
print('\nEnd work')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
factory = Factory(max_workers=3, max_tasks=50)
try:
loop.run_until_complete(factory.organize_work())
except KeyboardInterrupt:
print('\nBye bye')
sys.exit(0)
Related
I have a function which constantly yields some objects, say 1 per second and a handler which works 2 seconds and handles this objects. For example:
from time import sleep
import asyncio
from datetime import datetime
def generator():
i = 0
while True:
yield i
i += 1
sleep(1)
def handler(number):
sleep(2)
if number % 2 == 0:
print(str(number) + ' is even')
else:
print(str(number) + ' is odd')
for number in generator():
handler(number)
So, for example '2 is even' is printed 6 seconds after the program starts. How do I reduce this time to 4 seconds ( 2 seconds for generator + 2 seconds for handler) using asyncio? I need to set up asynchronous handling of the numbers.
You need couple of changes here:
your generator currently is a "generator", change it to "asynchronous generator" so that you can use async for. This way it can give the control back to eventloop.
Use async version of sleep in asyncio library: asyncio.sleep. time.sleep doesn't cooperate with other tasks.
Change your handler sync function to a "coroutine".
import asyncio
async def generator():
i = 0
while True:
yield i
i += 1
await asyncio.sleep(1)
async def handler(number):
await asyncio.sleep(2)
if number % 2 == 0:
print(str(number) + " is even")
else:
print(str(number) + " is odd")
async def main():
async for number in generator():
asyncio.create_task(handler(number))
asyncio.run(main())
Now, your first task is main, asyncio.run() automatically creates it as Task. Then when this task is running, it iterates asynchronously through the generator(). The values are received then for each number, you create a new Task out of handler coroutine.
This way the sleep times are overlapped. When it waits for new number for 1 second, it also actually waits for handler() one second. Then when the number is received, one second of handler() task is already passed, it only needs 1 second.
You can see the number of tasks if you want:
async def main():
async for number in generator():
print(f"Number of all tasks: {len(asyncio.all_tasks())}")
asyncio.create_task(handler(number))
Because each handler sleeps 2 seconds, and your number generator sleeps 1 seconds, You see that in every iteration 2 Tasks are exist in event loop. Change await asyncio.sleep(1) to await asyncio.sleep(0.5) in generator coroutine, you will see that 4 tasks are in event loop in every iteration.
Answer to the comment:
Can I do the same thing if I use API which doesn't let me create
asynchronous generator, but just a normal one? Can I still
asynchronously handle yielded objects?
Yes you can. Just note that if you don't have asynchronous generator, you can't use async for, which means your iteration is synchronous. But, you have to do a little trick for it to work. When your main() task is being executed, it constantly get a value from generator generator and creates a Task for it, but it doesn't give a chance to other tasks to run. You need await asyncio.sleep(0):
import asyncio
import time
def generator():
i = 0
while True:
yield i
i += 1
time.sleep(1)
async def handler(number):
await asyncio.sleep(2)
if number % 2 == 0:
print(str(number) + " is even")
else:
print(str(number) + " is odd")
async def main():
for number in generator():
print(f"Number of all tasks: {len(asyncio.all_tasks())}")
asyncio.create_task(handler(number))
await asyncio.sleep(0)
asyncio.run(main())
my code:
import asyncio
async def count(counter):
print(f"number of entries in the list {len(counter)}")
while True:
await asyncio.sleep(1 / 1000)
counter.append(1)
async def print_every_sec(counter):
while True:
await asyncio.sleep(1)
print(f"- 1 secund later. " f"number of entries in the list: {len(counter)}")
async def print_every_5_sec():
while True:
await asyncio.sleep(5)
print(f"---- 5 secund later")
async def print_every_10_sec():
while True:
await asyncio.sleep(10)
print(f"---------- 10 secund later")
async def main():
counter = list()
tasks = [
count(counter),
print_every_sec(counter),
print_every_5_sec(),
print_every_10_sec(),
]
await asyncio.gather(*tasks)
asyncio.run(main())
This is my conclusion but is not correct.
Correct conclusion around 1000 for each iteration.
I don't now what is it. This code works fine in online interpretations.
The assumption that asyncio.sleep(1 / 1000) (and to return control to other async routines) takes exactly one millisecond is not true.
Here's a more interesting example that records how long the sleep (and the time.perf_counter_ns() invocation) actually took:
import asyncio
import statistics
import time
max_count = 2500
async def count(counter):
while len(counter) < max_count:
t0 = time.perf_counter_ns()
await asyncio.sleep(1 / 1000)
t1 = time.perf_counter_ns()
counter.append((t1 - t0) / 1_000_000)
async def print_every_sec(counter):
while len(counter) < max_count:
await asyncio.sleep(1)
print(f'count: {len(counter)}; average: {statistics.mean(counter)} ms')
async def main():
counter = list()
tasks = [
count(counter),
print_every_sec(counter),
]
await asyncio.gather(*tasks)
asyncio.run(main())
On my Macbook, Python 3.9, etc, etc., the result is
count: 744; average: 1.341670
count: 1494; average: 1.33668
count: 2248; average: 1.33304
count: 2500; average: 1.325463428
so it takes 30% more than we expected to.
For sleeps of 10ms, the average is 11.84 ms. For sleeps of 100ms, the average is 102.9 ms.
I have an async method, as shown below.
I pass in lists of 1000 numbers, where the method will pass in each number to a helper function which will return something from a website.
I have a global variable called count, which i surround with locks to make sure it doesnt get changed by anything else
I use add_done_callback with the task to make this method async.
The goal is to keep sending a number in the list of 1000 numbers to the server, and only when the server returns data (can take anywhere from 0.1 to 2 seconds), to pause, write the data to a sql database, and then continue
The code works as expected without locks, or without making the callback function, (which is named 'function' below) asyncrounous. But adding locks gives me an error: RuntimeWarning: coroutine 'function' was never awaited self._context.run(self._callback, *self._args) RuntimeWarning: Enable tracemalloc to get the object allocation traceback
I am super new to async in python so any help/advice is greatly appriciated
My code is shown below. It is just a simple draft:
import time
import random
import asyncio
# from helper import get_message_from_server
async def get(number):
# get_message_from_server(number), which takes somewhere between 0.1 to 2 seconds
await asyncio.sleep(random.uniform(0.1, 2))
s = 'Done with number ' + number
return s
async def function(future, lock):
global count
print(future.result())
# write future.result() to db
acquired = await lock.acquire()
count -= 1 if (count > 1) else 0
lock.release()
async def main(numbers, lock):
global count
count = 0
for i, number in enumerate(numbers):
print('number:', number, 'count:', count)
acquired = await lock.acquire()
count += 1
lock.release()
task = asyncio.create_task(get(number))
task.add_done_callback(
lambda x: function(x, lock)
)
if (count == 50):
print('Reached 50')
await task
acquired = await lock.acquire()
count = 0
lock.release()
if (i == len(numbers) - 1):
await task
def make_numbers():
count = []
for i in range(1001):
count.append(str(i))
return count
if __name__ == '__main__':
numbers = make_numbers()
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
try:
loop.run_until_complete(main(numbers, lock))
except Exception as e:
pass
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.stop()
The above comment helped a lot
This is what the final working code looks like:
import time
import random
import asyncio
from functools import partial
# from helper import get_message_from_server
async def get(number):
# get_message_from_server(number), which takes somewhere between 0.1 to 2 seconds
await asyncio.sleep(random.uniform(0.1, 2))
s = 'Done with number ' + number
return s
def function(result, lock):
print(result.result())
async def count_decrement(lock):
global count
print('in count decrement')
acquired = await lock.acquire()
count -= 1 if (count > 1) else 0
lock.release()
asyncio.create_task(count_decrement(lock))
async def main(numbers, lock):
global count
count = 0
for i, number in enumerate(numbers):
print('number:', number, 'count:', count)
acquired = await lock.acquire()
count += 1
lock.release()
task = asyncio.create_task(get(number))
task.add_done_callback(partial(function, lock = lock))
if (count == 50):
print('Reached 50')
await task
acquired = await lock.acquire()
count = 0
lock.release()
if (i == len(numbers) - 1):
await task
def make_numbers():
count = []
for i in range(1001):
count.append(str(i))
return count
if __name__ == '__main__':
numbers = make_numbers()
loop = asyncio.get_event_loop()
lock = asyncio.Lock()
try:
loop.run_until_complete(main(numbers, lock))
except Exception as e:
pass
finally:
loop.run_until_complete(loop.shutdown_asyncgens())
loop.stop()
I am attempting to grasp Python's asyncio library, and I'm having a problem with it's timeout exception. I can't figure out why does the "asyncio.TimeoutError" exception in the "async def create" function always execute at the end of my program even though the timeout limit hasn't passed. I would appreciate Your expert advise and opinion :)
Thank you for your time.
import asyncio
import multiprocessing as mp
from enum import Enum
class Sensor(Enum):
GREEN = 0
RED = 1
class State(Enum):
NORMAL = 0
MEDIUM = 1
BURNED = 2
class Toaster:
def __init__(self, min = 20, max = 50, temp = 0, timer = 0, state = State.NORMAL, sensor = Sensor.GREEN):
self.min = min
self.max = max
self.temp = self.min
self.timer = timer
self.state = state
self.sensor = sensor
def display(self):
print("\nTimer state:", self.timer, "seconds")
print("Toast state:", self.state.name)
print("Sensor state:", self.sensor.name)
async def start(self):
while True:
if self.temp <= self.max:
await asyncio.sleep(0.1)
print("Temperature:", self.temp)
self.temp+=1
else:
print("\nMaximum temperature", self.max, "celsius reached")
await self.measure_state()
await self.restart()
break
async def restart(self):
while True:
if self.temp >= self.min:
await asyncio.sleep(0.1)
print("Temperature:", self.temp)
self.temp-=1
else:
self.sensor = Sensor.GREEN
print("\nMinimum temperature", self.min, "celsius reached")
break
async def validateInput(self, message):
valid = False
while not valid:
try:
userInput = int(input(message))
if userInput == 0 or userInput == 1:
valid = True
return userInput
else:
raise ValueError("\nInvalid value", userInput)
except ValueError as v:
print(v)
async def eject(self):
self.display()
message = "\nEject toast - 1(Yes), 0(No):"
try:
return await asyncio.wait_for(self.validateInput(message), timeout=1000)
except asyncio.TimeoutError:
print("Took too long - eject")
async def repeat(self):
message = "\nInject another toast - 1(Yes), 0(No):"
try:
return await asyncio.wait_for(self.validateInput(message), timeout=1000)
except asyncio.TimeoutError:
print("Took too long - repeat")
async def measure_state(self):
while True:
await asyncio.sleep(5)
self.timer+=50
if self.timer == 50:
print("\nToast is in it's", self.state.name, "state")
if await self.eject() == 1:
print("\nToast ejected")
if await self.repeat() == 1:
self.timer = 0
self.state = State.NORMAL
await self.measure_state()
break
elif self.timer == 100:
self.state = State.MEDIUM
self.sensor = Sensor.RED
print("\nToast is in it's", self.state.name, "state")
if await self.eject() == 1:
print("\nToast ejected")
if await self.repeat() == 1:
self.timer = 0
self.state = State.NORMAL
await self.measure_state()
break
elif self.timer >= 150:
self.state = State.BURNED
print("\nToast is in it's", self.state.name, "state, ejecting toast")
break
async def toaster(self):
message = "\nInsert a toast - 1(Yes), 0(No):"
while await self.validateInput(message) != 1:
print("\nPlease insert a toast")
print("\nToast inserted")
await self.start()
async def create(self):
x = loop.create_task(Toaster().toaster())
y = loop.create_task(Toaster().toaster())
z = loop.create_task(Toaster().toaster())
try:
await asyncio.wait([x, y, z], timeout=1000)
raise asyncio.TimeoutError("\nTook too long - create")
except asyncio.TimeoutError as t:
print(t)
x.cancel(), y.cancel(), z.cancel()
def get_process_count():
nproc = mp.cpu_count()
pool = mp.Pool(processes=nproc)
return pool
class Connector(Toaster):
pass
async def main():
connector = Connector()
result = get_process_count()
result.map(await connector.create())
await asyncio.gather(result)
if __name__ == "__main__":
loop = None
try:
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
except Exception as e:
pass
finally:
loop.close()
In create() you raise the exception immediately after waiting for tasks x,y,z to complete. Adding some prints to toaster and create shows that the three tasks finish so execution just resumes with the raise asyncio.TimeoutError... statement.
...
async def toaster(self):
message = "\nInsert a toast - 1(Yes), 0(No):"
while await self.validateInput(message) != 1:
print("\nPlease insert a toast")
print("\nToast inserted")
await self.start()
return 'FINISHED'
async def create(self):
x = loop.create_task(Toaster().toaster())
y = loop.create_task(Toaster().toaster())
z = loop.create_task(Toaster().toaster())
try:
await asyncio.wait([x, y, z], timeout=1000)
for thing in (x,y,z):
print(thing)
raise asyncio.TimeoutError("\nTook too long - create") # <-- you raise the exception Here!
except asyncio.TimeoutError as t:
print(t)
x.cancel(), y.cancel(), z.cancel()
Results in
>>>
...
...
Temperature: 20
Minimum temperature 20 celsius reached
Temperature: 20
Minimum temperature 20 celsius reached
Temperature: 20
Minimum temperature 20 celsius reached
<Task finished coro=<Toaster.toaster() done, defined at tmp.py:129> result='FINISHED'>
<Task finished coro=<Toaster.toaster() done, defined at tmp.py:129> result='FINISHED'>
<Task finished coro=<Toaster.toaster() done, defined at tmp.py:129> result='FINISHED'>
Took too long - create
I inserted 3 toasts and ejected them at first request.
I went through a dozen cycles of injecting and ejecting toast and it didn't
timeout but the exception was raised as soon as a declined to inject
more toast.
...why does the "asyncio.TimeoutError" exception in the "async def create" function always execute at the end of my program ...?
I would say you wrote it to do that - looks like that was your intention.
I am learning asyncio library to do some tasks I want to achieve. I wrote the following code to teach myself on how to be able to switch to so another task while the original one is being executed. As you can see below, the summation() should be performed until a condition is met where it should jump to the secondaryTask(). After secondaryTask() is finished, it should return back to summation() where hopefully it gets finished. The potential results should be sum=1225 and mul=24.
import asyncio, time
async def summation():
print('Running summation from 0 to 50:')
sum = 0
for i in range(25):
sum = sum + i
if i != 25:
time.sleep(0.1)
else:
await asyncio.sleep(0) # pretend to be non-blocking work (Jump to the next task)
print('This message is shown because summation() is completed! sum= %d' % sum)
async def secondaryTask():
print('Do some secondaryTask here while summation() is on progress')
mul = 1
for i in range(1, 5):
mul = mul * i
time.sleep(0.1)
await asyncio.sleep(0)
print('This message is shown because secondaryTask() is completed! Mul= %d' % mul)
t0 = time.time()
ioloop = asyncio.get_event_loop()
tasks = [ioloop.create_task(summation()), ioloop.create_task(secondaryTask())]
wait_tasks = asyncio.wait(tasks)
ioloop.run_until_complete(wait_tasks)
ioloop.close()
t1 = time.time()
print('Total time= %.3f' % (t1-t0))
This code does not perform as expected because sum=300 as oppose to be sum=1225. Clearly that summation() does not continue while secondaryTask() is being processed. How can I modify summation() to be able to do the summation of the remaining 25 values on the background?
Thank you
It's just your careless. You want to run summation from 0 to 50, your summation function should be for i in range(50)