In asyncio, it there a way to attach a callback function to a Task such that this callback function will run after the Task has been completed?
So far, the only way I can figure out is to use asyncio.completed in a loop, as shown below. But this requires 2 lists (tasks and cb_tasks) to hold all the tasks/futures.
Is there a better way to do this?
import asyncio
import random
class Foo:
async def start(self):
tasks = []
cb_tasks = []
# Start ten `do_work` tasks simultaneously
for i in range(10):
task = asyncio.create_task(self.do_work(i))
tasks.append(task)
# How to run `self.handle_work_done` as soon as this `task` is completed?
for f in asyncio.as_completed(tasks):
res = await f
t = asyncio.create_task(self.work_done_cb(res))
cb_tasks.append(t)
await asyncio.wait(tasks + cb_tasks)
async def do_work(self, i):
""" Simulate doing some work """
x = random.randint(1, 10)
await asyncio.sleep(x)
print(f"Finished work #{i}")
return x
async def work_done_cb(self, x):
""" Callback after `do_work` has been completed """
await asyncio.sleep(random.randint(1, 3))
print(f"Finished additional work {x}")
if __name__ == "__main__":
foo = Foo()
asyncio.run(foo.start())
I have a class Sensor which has one numeric value num. It has an async method update, which updates the num every second in an infinite loop.
I want to initiate the class and call the method update, keep it running and return the control to the main program to access the value of the num after some time. I am using asyncio but do not know how to invoke the method such that the concurrent running of the loop and variable access is feasible.
Code:
import asyncio
import time
class Sensor:
def __init__(self, start=0):
self.num = start
async def update(self):
print(f"Starting Updates{self.num}")
while True:
self.num += 1
print(self.num)
await asyncio.sleep(1)
if __name__ == "__main__":
print("Main")
sensor = Sensor()
# asyncio.run(sensor.update())
# asyncio.ensure_future(sensor.update())
future = asyncio.run_coroutine_threadsafe(sensor.update(), asyncio.new_event_loop())
print("We are back")
print(f"current value: {sensor.num}")
time.sleep(4)
print(f"current value: {sensor.num}")
This gives me output of 0 before and after waiting for 4 seconds, which means the update method is not running behind. run() does not return the control at all.
Which method should I call to invoke the infinite loop in the background?
To use asyncio.run_coroutine_threadsafe, you need to actually run the event loop, in a separate thread. For example:
sensor = Sensor()
loop = asyncio.new_event_loop()
threading.Thread(target=loop.run_forever).start()
future = asyncio.run_coroutine_threadsafe(sensor.update(), loop)
...
loop.call_soon_threadsafe(loop.stop)
According to documents for running tasks concurrently, you should use gather. For example, your code will be something like this:
import asyncio
import time
class Sensor:
def __init__(self, start=0):
self.num = start
async def update(self):
print(f"Starting Updates{self.num}")
while True:
self.num += 1
print(self.num)
await asyncio.sleep(1)
async def print_value(sensor):
print("We are back")
print(f"current value: {sensor.num}")
await asyncio.sleep(4)
print(f"current value: {sensor.num}")
async def main():
print("Main")
sensor = Sensor()
await asyncio.gather(sensor.update(), print_value(sensor))
if __name__ == "__main__":
asyncio.run(main())
I am connecting to aioredis from __init__ (I do not want to move it out since this means I have to some extra major changes). How can I wait for aioredis connection task in below __init__ example code and have it print self.sub and self.pub object? Currently it gives an error saying
abc.py:42> exception=AttributeError("'S' object has no attribute
'pub'")
I do see redis connections created and coro create_connetion done.
Is there a way to call blocking asyncio calls from __init__. If I replace asyncio.wait with asyncio.run_until_complete I get an error that roughly says
loop is already running.
asyncio.gather is
import sys, json
from addict import Dict
import asyncio
import aioredis
class S():
def __init__(self, opts):
print(asyncio.Task.all_tasks())
task = asyncio.wait(asyncio.create_task(self.create_connection()), return_when="ALL_COMPLETED")
print(asyncio.Task.all_tasks())
print(task)
print(self.pub, self.sub)
async def receive_message(self, channel):
while await channel.wait_message():
message = await channel.get_json()
await asyncio.create_task(self.callback_loop(Dict(json.loads(message))))
async def run_s(self):
asyncio.create_task(self.listen())
async def callback_loop(msg):
print(msg)
self.callback_loop = callback_loop
async def create_connection(self):
self.pub = await aioredis.create_redis("redis://c8:7070/0", password="abc")
self.sub = await aioredis.create_redis("redis://c8:7070/0", password="abc")
self.db = await aioredis.create_redis("redis://c8:7070/0", password="abc")
self.listener = await self.sub.subscribe(f"abc")
async def listen(self):
self.tsk = asyncio.ensure_future(self.receive_message(self.listener[0]))
await self.tsk
async def periodic(): #test function to show current tasks
number = 5
while True:
await asyncio.sleep(number)
print(asyncio.Task.all_tasks())
async def main(opts):
loop.create_task(periodic())
s = S(opts)
print(s.pub, s.sub)
loop.create_task(s.run_s())
if __name__ == "__main__":
loop = asyncio.get_event_loop()
main_task = loop.create_task(main(sys.argv[1:]))
loop.run_forever() #I DONT WANT TO MOVE THIS UNLESS IT IS NECESSARY
I think what you want to do is to make sure the function create_connections runs to completion BEFORE the S constructor. A way to do that is to rearrange your code a little bit. Move the create_connections function outside the class:
async def create_connection():
pub = await aioredis.create_redis("redis://c8:7070/0", password="abc")
sub = await aioredis.create_redis("redis://c8:7070/0", password="abc")
db = await aioredis.create_redis("redis://c8:7070/0", password="abc")
listener = await self.sub.subscribe(f"abc")
return pub, sub, db, listener
Now await that function before constructing S. So your main function becomes:
async def main(opts):
loop.create_task(periodic())
x = await create_connections()
s = S(opts, x) # pass the result of create_connections to S
print(s.pub, s.sub)
loop.create_task(s.run_s())
Now modify the S constructor to receive the objects created:
def __init__(self, opts, x):
self.pub, self.sub, self.db, self.listener = x
I'm not sure what you're trying to do with the return_when argument and the call to asyncio.wait. The create_connections function doesn't launch a set of parallel tasks, but rather awaits each of the calls before moving on to the next one. Perhaps you could improve performance by running the four calls in parallel but that's a different question.
I have the following code that executes a list of methods (_tick_watchers) every 10 seconds. While this is fine for most of the methods on the _tick_watchers list, there are some that I need to be executed only once every 5 minutes. Any ideas for a simple & neat way to do so?
async def on_tick(self):
while not self._exiting:
await asyncio.sleep(10)
now = time.time()
# call all tick watchers
for w in self._tick_watchers:
w(now)
Since asyncio doesn't come with a scheduler for repeating tasks, you can generalise your on_tick() into one:
import time
import asyncio
class App:
def my_func_1(self, now):
print('every second\t{}'.format(now))
def my_func_5(self, now):
print('every 5 seconds\t{}'.format(now))
def __init__(self):
self.exiting = False
async def scheduler(self, func, interval):
while not self.exiting:
now = time.time()
func(now)
await asyncio.sleep(interval)
# to combat drift, you could try something like this:
# await asyncio.sleep(interval + now - time.time())
async def run(self):
asyncio.ensure_future(self.scheduler(self.my_func_5, 5.0))
asyncio.ensure_future(self.scheduler(self.my_func_1, 1.0))
await asyncio.sleep(11)
self.exiting = True
asyncio.get_event_loop().run_until_complete(App().run()) # test run
Note that currently you do not start your task every 10 seconds: you wait for 10 seconds after the task exits. The small difference accumulates over time, which might be important.
You have some class for periodic execution? If so, you can add timeout parameter to it.
timeout in init:
def __init__(self, timeout=10):
self.timeout = timeout
and use it in tick handler:
async def on_tick(self):
while not self._exiting:
await asyncio.sleep(self.timeout)
# ...
Then create and run several instances of that class with different timeouts.
I'm migrating from tornado to asyncio, and I can't find the asyncio equivalent of tornado's PeriodicCallback. (A PeriodicCallback takes two arguments: the function to run and the number of milliseconds between calls.)
Is there such an equivalent in asyncio?
If not, what would be the cleanest way to implement this without running the risk of getting a RecursionError after a while?
For Python versions below 3.5:
import asyncio
#asyncio.coroutine
def periodic():
while True:
print('periodic')
yield from asyncio.sleep(1)
def stop():
task.cancel()
loop = asyncio.get_event_loop()
loop.call_later(5, stop)
task = loop.create_task(periodic())
try:
loop.run_until_complete(task)
except asyncio.CancelledError:
pass
For Python 3.5 and above:
import asyncio
async def periodic():
while True:
print('periodic')
await asyncio.sleep(1)
def stop():
task.cancel()
loop = asyncio.get_event_loop()
loop.call_later(5, stop)
task = loop.create_task(periodic())
try:
loop.run_until_complete(task)
except asyncio.CancelledError:
pass
When you feel that something should happen "in background" of your asyncio program, asyncio.Task might be good way to do it. You can read this post to see how to work with tasks.
Here's possible implementation of class that executes some function periodically:
import asyncio
from contextlib import suppress
class Periodic:
def __init__(self, func, time):
self.func = func
self.time = time
self.is_started = False
self._task = None
async def start(self):
if not self.is_started:
self.is_started = True
# Start task to call func periodically:
self._task = asyncio.ensure_future(self._run())
async def stop(self):
if self.is_started:
self.is_started = False
# Stop task and await it stopped:
self._task.cancel()
with suppress(asyncio.CancelledError):
await self._task
async def _run(self):
while True:
await asyncio.sleep(self.time)
self.func()
Let's test it:
async def main():
p = Periodic(lambda: print('test'), 1)
try:
print('Start')
await p.start()
await asyncio.sleep(3.1)
print('Stop')
await p.stop()
await asyncio.sleep(3.1)
print('Start')
await p.start()
await asyncio.sleep(3.1)
finally:
await p.stop() # we should stop task finally
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Output:
Start
test
test
test
Stop
Start
test
test
test
[Finished in 9.5s]
As you see on start we just start task that calls some functions and sleeps some time in endless loop. On stop we just cancel that task. Note, that task should be stopped at the moment program finished.
One more important thing that your callback shouldn't take much time to be executed (or it'll freeze your event loop). If you're planning to call some long-running func, you possibly would need to run it in executor.
A variant that may be helpful: if you want your recurring call to happen every n seconds instead of n seconds between the end of the last execution and the beginning of the next, and you don't want calls to overlap in time, the following is simpler:
async def repeat(interval, func, *args, **kwargs):
"""Run func every interval seconds.
If func has not finished before *interval*, will run again
immediately when the previous iteration finished.
*args and **kwargs are passed as the arguments to func.
"""
while True:
await asyncio.gather(
func(*args, **kwargs),
asyncio.sleep(interval),
)
And an example of using it to run a couple tasks in the background:
async def f():
await asyncio.sleep(1)
print('Hello')
async def g():
await asyncio.sleep(0.5)
print('Goodbye')
async def main():
t1 = asyncio.ensure_future(repeat(3, f))
t2 = asyncio.ensure_future(repeat(2, g))
await t1
await t2
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
There is no built-in support for periodic calls, no.
Just create your own scheduler loop that sleeps and executes any tasks scheduled:
import math, time
async def scheduler():
while True:
# sleep until the next whole second
now = time.time()
await asyncio.sleep(math.ceil(now) - now)
# execute any scheduled tasks
async for task in scheduled_tasks(time.time()):
await task()
The scheduled_tasks() iterator should produce tasks that are ready to be run at the given time. Note that producing the schedule and kicking off all the tasks could in theory take longer than 1 second; the idea here is that the scheduler yields all tasks that should have started since the last check.
Alternative version with decorator for python 3.7
import asyncio
import time
def periodic(period):
def scheduler(fcn):
async def wrapper(*args, **kwargs):
while True:
asyncio.create_task(fcn(*args, **kwargs))
await asyncio.sleep(period)
return wrapper
return scheduler
#periodic(2)
async def do_something(*args, **kwargs):
await asyncio.sleep(5) # Do some heavy calculation
print(time.time())
if __name__ == '__main__':
asyncio.run(do_something('Maluzinha do papai!', secret=42))
Based on #A. Jesse Jiryu Davis answer (with #Torkel Bjørnson-Langen and #ReWrite comments) this is an improvement which avoids drift.
import time
import asyncio
#asyncio.coroutine
def periodic(period):
def g_tick():
t = time.time()
count = 0
while True:
count += 1
yield max(t + count * period - time.time(), 0)
g = g_tick()
while True:
print('periodic', time.time())
yield from asyncio.sleep(next(g))
loop = asyncio.get_event_loop()
task = loop.create_task(periodic(1))
loop.call_later(5, task.cancel)
try:
loop.run_until_complete(task)
except asyncio.CancelledError:
pass
This solution uses the decoration concept from Fernando José Esteves de Souza, the drifting workaround from Wojciech Migda and a superclass in order to generate most elegant code as possible to deal with asynchronous periodic functions.
Without threading.Thread
The solution is comprised of the following files:
periodic_async_thread.py with the base class for you to subclass
a_periodic_thread.py with an example subclass
run_me.py with an example instantiation and run
The PeriodicAsyncThread class in the file periodic_async_thread.py:
import time
import asyncio
import abc
class PeriodicAsyncThread:
def __init__(self, period):
self.period = period
def periodic(self):
def scheduler(fcn):
async def wrapper(*args, **kwargs):
def g_tick():
t = time.time()
count = 0
while True:
count += 1
yield max(t + count * self.period - time.time(), 0)
g = g_tick()
while True:
# print('periodic', time.time())
asyncio.create_task(fcn(*args, **kwargs))
await asyncio.sleep(next(g))
return wrapper
return scheduler
#abc.abstractmethod
async def run(self, *args, **kwargs):
return
def start(self):
asyncio.run(self.run())
An example of a simple subclass APeriodicThread in the file a_periodic_thread.py:
from periodic_async_thread import PeriodicAsyncThread
import time
import asyncio
class APeriodicThread(PeriodicAsyncThread):
def __init__(self, period):
super().__init__(period)
self.run = self.periodic()(self.run)
async def run(self, *args, **kwargs):
await asyncio.sleep(2)
print(time.time())
Instantiating and running the example class in the file run_me.py:
from a_periodic_thread import APeriodicThread
apt = APeriodicThread(2)
apt.start()
This code represents an elegant solution that also mitigates the time drift problem of other solutions. The output is similar to:
1642711285.3898764
1642711287.390698
1642711289.3924973
1642711291.3920736
With threading.Thread
The solution is comprised of the following files:
async_thread.py with the canopy asynchronous thread class.
periodic_async_thread.py with the base class for you to subclass
a_periodic_thread.py with an example subclass
run_me.py with an example instantiation and run
The AsyncThread class in the file async_thread.py:
from threading import Thread
import asyncio
import abc
class AsyncThread(Thread):
def __init__(self, *args, **kwargs) -> None:
super().__init__(*args, **kwargs)
#abc.abstractmethod
async def async_run(self, *args, **kwargs):
pass
def run(self, *args, **kwargs):
# loop = asyncio.new_event_loop()
# asyncio.set_event_loop(loop)
# loop.run_until_complete(self.async_run(*args, **kwargs))
# loop.close()
asyncio.run(self.async_run(*args, **kwargs))
The PeriodicAsyncThread class in the file periodic_async_thread.py:
import time
import asyncio
from .async_thread import AsyncThread
class PeriodicAsyncThread(AsyncThread):
def __init__(self, period, *args, **kwargs):
self.period = period
super().__init__(*args, **kwargs)
self.async_run = self.periodic()(self.async_run)
def periodic(self):
def scheduler(fcn):
async def wrapper(*args, **kwargs):
def g_tick():
t = time.time()
count = 0
while True:
count += 1
yield max(t + count * self.period - time.time(), 0)
g = g_tick()
while True:
# print('periodic', time.time())
asyncio.create_task(fcn(*args, **kwargs))
await asyncio.sleep(next(g))
return wrapper
return scheduler
An example of a simple subclass APeriodicThread in the file a_periodic_thread.py:
import time
from threading import current_thread
from .periodic_async_thread import PeriodicAsyncThread
import asyncio
class APeriodicAsyncTHread(PeriodicAsyncThread):
async def async_run(self, *args, **kwargs):
print(f"{current_thread().name} {time.time()} Hi!")
await asyncio.sleep(1)
print(f"{current_thread().name} {time.time()} Bye!")
Instantiating and running the example class in the file run_me.py:
from .a_periodic_thread import APeriodicAsyncTHread
a = APeriodicAsyncTHread(2, name = "a periodic async thread")
a.start()
a.join()
This code represents an elegant solution that also mitigates the time drift problem of other solutions. The output is similar to:
a periodic async thread 1643726990.505269 Hi!
a periodic async thread 1643726991.5069854 Bye!
a periodic async thread 1643726992.506919 Hi!
a periodic async thread 1643726993.5089169 Bye!
a periodic async thread 1643726994.5076022 Hi!
a periodic async thread 1643726995.509422 Bye!
a periodic async thread 1643726996.5075526 Hi!
a periodic async thread 1643726997.5093904 Bye!
a periodic async thread 1643726998.5072556 Hi!
a periodic async thread 1643726999.5091035 Bye!
For multiple types of scheduling I'd recommend APSScheduler which has asyncio support.
I use it for a simple python process I can fire up using docker and just runs like a cron executing something weekly, until I kill the docker/process.
This is what I did to test my theory of periodic call backs using asyncio. I don't have experience using Tornado, so I'm not sure exactly how the periodic call backs work with it. I am used to using the after(ms, callback) method in Tkinter though, and this is what I came up with. While True: Just looks ugly to me even if it is asynchronous (more so than globals). The call_later(s, callback, *args) method uses seconds not milliseconds though.
import asyncio
my_var = 0
def update_forever(the_loop):
global my_var
print(my_var)
my_var += 1
# exit logic could be placed here
the_loop.call_later(3, update_forever, the_loop) # the method adds a delayed callback on completion
event_loop = asyncio.get_event_loop()
event_loop.call_soon(update_forever, event_loop)
event_loop.run_forever()