Sometimes there is some non-critical asynchronous operation that needs to happen but I don't want to wait for it to complete. In Tornado's coroutine implementation you can "fire & forget" an asynchronous function by simply ommitting the yield key-word.
I've been trying to figure out how to "fire & forget" with the new async/await syntax released in Python 3.5. E.g., a simplified code snippet:
async def async_foo():
print("Do some stuff asynchronously here...")
def bar():
async_foo() # fire and forget "async_foo()"
bar()
What happens though is that bar() never executes and instead we get a runtime warning:
RuntimeWarning: coroutine 'async_foo' was never awaited
async_foo() # fire and forget "async_foo()"
Upd:
Replace asyncio.ensure_future with asyncio.create_task everywhere if you're using Python >= 3.7 It's a newer, nicer way to spawn tasks.
asyncio.Task to "fire and forget"
According to python docs for asyncio.Task it is possible to start some coroutine to execute "in the background". The task created by asyncio.ensure_future won't block the execution (therefore the function will return immediately!). This looks like a way to "fire and forget" as you requested.
import asyncio
async def async_foo():
print("async_foo started")
await asyncio.sleep(1)
print("async_foo done")
async def main():
asyncio.ensure_future(async_foo()) # fire and forget async_foo()
# btw, you can also create tasks inside non-async funcs
print('Do some actions 1')
await asyncio.sleep(1)
print('Do some actions 2')
await asyncio.sleep(1)
print('Do some actions 3')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Output:
Do some actions 1
async_foo started
Do some actions 2
async_foo done
Do some actions 3
What if tasks are executing after the event loop has completed?
Note that asyncio expects tasks to be completed at the moment the event loop completes. So if you'll change main() to:
async def main():
asyncio.ensure_future(async_foo()) # fire and forget
print('Do some actions 1')
await asyncio.sleep(0.1)
print('Do some actions 2')
You'll get this warning after the program finished:
Task was destroyed but it is pending!
task: <Task pending coro=<async_foo() running at [...]
To prevent that you can just await all pending tasks after the event loop has completed:
async def main():
asyncio.ensure_future(async_foo()) # fire and forget
print('Do some actions 1')
await asyncio.sleep(0.1)
print('Do some actions 2')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
# Let's also finish all running tasks:
pending = asyncio.Task.all_tasks()
loop.run_until_complete(asyncio.gather(*pending))
Kill tasks instead of awaiting them
Sometimes you don't want to await tasks to be done (for example, some tasks may be created to run forever). In that case, you can just cancel() them instead of awaiting them:
import asyncio
from contextlib import suppress
async def echo_forever():
while True:
print("echo")
await asyncio.sleep(1)
async def main():
asyncio.ensure_future(echo_forever()) # fire and forget
print('Do some actions 1')
await asyncio.sleep(1)
print('Do some actions 2')
await asyncio.sleep(1)
print('Do some actions 3')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
# Let's also cancel all running tasks:
pending = asyncio.Task.all_tasks()
for task in pending:
task.cancel()
# Now we should await task to execute it's cancellation.
# Cancelled task raises asyncio.CancelledError that we can suppress:
with suppress(asyncio.CancelledError):
loop.run_until_complete(task)
Output:
Do some actions 1
echo
Do some actions 2
echo
Do some actions 3
echo
Output:
>>> Hello
>>> foo() started
>>> I didn't wait for foo()
>>> foo() completed
Here is the simple decorator function which pushes the execution to background and line of control moves to next line of the code.
The primary advantage is, you don't have to declare the function as await
import asyncio
import time
def fire_and_forget(f):
def wrapped(*args, **kwargs):
return asyncio.get_event_loop().run_in_executor(None, f, *args, *kwargs)
return wrapped
#fire_and_forget
def foo():
print("foo() started")
time.sleep(1)
print("foo() completed")
print("Hello")
foo()
print("I didn't wait for foo()")
Note: Check my other answer which does the same using plain thread without asyncio.
This is not entirely asynchronous execution, but maybe run_in_executor() is suitable for you.
def fire_and_forget(task, *args, **kwargs):
loop = asyncio.get_event_loop()
if callable(task):
return loop.run_in_executor(None, task, *args, **kwargs)
else:
raise TypeError('Task must be a callable')
def foo():
#asynchronous stuff here
fire_and_forget(foo)
For some reason if you are unable to use asyncio then here is the implementation using plain threads. Check my other answers and Sergey's answer too.
import threading, time
def fire_and_forget(f):
def wrapped():
threading.Thread(target=f).start()
return wrapped
#fire_and_forget
def foo():
print("foo() started")
time.sleep(1)
print("foo() completed")
print("Hello")
foo()
print("I didn't wait for foo()")
produces
>>> Hello
>>> foo() started
>>> I didn't wait for foo()
>>> foo() completed
def fire_and_forget(f):
def wrapped(*args, **kwargs):
threading.Thread(target=functools.partial(f, *args, **kwargs)).start()
return wrapped
is the better version of the above -- does not use asyncio
Related
Is there a way to create a secondary asyncio loop(or prioritize an await) that when awaited does not pass control back to the main event loop buts awaits for those 'sub' functions? IE
import asyncio
async def priority1():
print("p1 before sleep")
await asyncio.sleep(11)
print("p1 after sleep")
async def priority2():
print("p2 before sleep")
await asyncio.sleep(11)
print("p2 after sleep")
async def foo():
while True:
print("foo before sleep")
#do not pass control back to main event loop here but run these 2 async
await asyncio.gather(priority1(),priority2())
print("foo after sleep")
async def bar():
while True:
print("bar before sleep")
await asyncio.sleep(5)
print("bar after sleep")
async def main():
await asyncio.gather(foo(),bar())
asyncio.run(main())
I would like foo to wait for priority1/2 to finish before passing control back to the main event loop.
Right now it would go:
foo before sleep
bar before sleep
p1 before sleep
p2 before sleep
bar after sleep
I would like to see:
foo before sleep
bar before sleep
p1 before sleep
p2 before sleep
p1 after sleep
p2 after sleep
bar after sleep
Is this possible?
thanks
It is not possible to run two event loops on the same thread. The code in asyncio and specs even look like they were thought in a way to permit that - but afterwards the API bent in a way it is no longer possible (for example, the explicit loop parameter for several of the relevant calls has been deprecated and removed)
In the same line, there is no way to prioritize a subset of tasks in the running loop. I answered a question on this line a couple weeks ago, and managed to get to a synchronization primitive to be used in place of asyncio.sleep which could take priority into account - but it requires all participating tasks to call it, so it would not be much different of a lock or something (I will link to it bellow - the idea is: your code call await custom.sleep() at certain points: it will only return when there are no other higher-prioritized tasks also calling that custom.sleep() ) -check it here: Execute asyncio task as soon as possible
When wrtting that code, I realized that it is possible to write an event loop which could take into account a "priority" attribute on tasks. But having a production-grade for this requires some non-hobby work: by using that loop, you could get what you are asking for, with no changes needed in the tasks code.
However, I think running a secondary loop in another thread, and then waiting synchronously on that thread to complete is a way to get your things accomplished.
import asyncio
import threading
def priority(func):
async def wrapper(*args, **kwargs):
result = None
def runner(*args, **kw):
nonlocal result
result = asyncio.run(func(*args, **kw))
t = threading.Thread(target=runner, args=args, kwargs=kwargs)
await asyncio.sleep(0)
t.start()
# if one wants to perform limited yields to the main loop, it should be done here
t.join()
return result
return wrapper
async def priority1():
print("p1 before sleep")
await asyncio.sleep(.11)
print("p1 after sleep")
async def priority2():
print("p2 before sleep")
await asyncio.sleep(.11)
print("p2 after sleep")
#priority
async def foo():
print("foo before sleep")
#do not pass control back to main event loop here but run these 2 async
await asyncio.gather(priority1(),priority2())
print("foo after sleep")
async def bar():
print("bar before sleep")
await asyncio.sleep(.05)
print("bar after sleep")
async def main():
await asyncio.gather(foo(),bar())
asyncio.run(main())
I am trying to do something similar like C# ManualResetEvent but in Python.
I have attempted to do it in python but doesn't seem to work.
import asyncio
cond = asyncio.Condition()
async def main():
some_method()
cond.notify()
async def some_method():
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
main()
I want the some_method to start then wait until signaled to start again.
This code is not complete, first of all you need to use asyncio.run() to bootstrap the event loop - this is why your code is not running at all.
Secondly, some_method() never actually starts. You need to asynchronously start some_method() using asyncio.create_task(). When you call an "async def function" (the more correct term is coroutinefunction) it returns a coroutine object, this object needs to be driven by the event loop either by you awaiting it or using the before-mentioned function.
Your code should look more like this:
import asyncio
async def main():
cond = asyncio.Condition()
t = asyncio.create_task(some_method(cond))
# The event loop hasn't had any time to start the task
# until you await again. Sleeping for 0 seconds will let
# the event loop start the task before continuing.
await asyncio.sleep(0)
cond.notify()
# You should never really "fire and forget" tasks,
# the same way you never do with threading. Wait for
# it to complete before returning:
await t
async def some_method(cond):
print("Starting...")
await cond.acquire()
await cond.wait()
cond.release()
print("Finshed...")
asyncio.run(main())
I am not much used to asyncio, so perhaps this question is trivial.
I have a code running asynchronously, which will run a callback when done (the callback can be callable or awaitable). I would like to wait for the callback to be called, with timeout. I sense that it is conceptually a task, but I am not sure how to create the task but wait for it somewhere else.
import asyncio, inspect
async def expensivefunction(callback):
# this is something which takes a lot of time
await asyncio.sleep(10)
# but eventually computes the result
result=10
# and calls the callback
callback(result)
if inspect.isawaitable(callback): await callback
# just print the result, for example
async def callback(result): print(result)
# main code async
async def myfunc():
await expensivefunction(callback=callback)
# this will wait for callback to be called within 5 seconds
# if not, exception is thrown
await asyncio.wait_for(...??,timeout=5)
asyncio.run(myfunc())
What would be the right approach to this?
Please find working example:
import asyncio
AWAIT_TIME = 5.0
async def expensive_function():
"""this is something which takes a lot of time"""
await asyncio.sleep(10)
result = 10
return result
def callback(fut: asyncio.Future):
"""just prints result. Callback should be sync function"""
if not fut.cancelled() and fut.done():
print(fut.result())
else:
print("No results")
async def amain():
"""Main async func in the app"""
# create task
task = asyncio.create_task(expensive_function())
task.add_done_callback(callback)
# try to await the task
try:
r = await asyncio.wait_for(task, timeout=AWAIT_TIME)
except asyncio.TimeoutError as ex:
print(ex)
else:
print(f"All work done fine: {r}")
finally:
print("App finished!")
if __name__ == '__main__':
asyncio.run(amain())
If any questions, please let me know.
I want to make a timer which is started in a normal function, but in the timer function, it should be able to call an async function
I want to do something like this:
startTimer()
while True:
print("e")
def startTimer(waitForSeconds: int):
# Wait for `waitForSeconds`
await myAsyncFunc()
async def myAsyncFunc():
print("in my async func")
Where the while True loop should do its stuff and after waitForSeconds the timer the async function should execute an other async function, but waiting shouldn't block any other actions and doesn't need to be awaited
If something isn't understandable, I'm sorry, I'll try to explain it then
Thanks
If you want to run your synchronous and asynchronous code in parallel, you will need to run one of them in a separate thread. For example:
def sync_code():
while True:
print("e")
async def start_timer(secs):
await asyncio.sleep(secs)
await async_func()
async def main():
asyncio.create_task(start_timer(1))
loop = asyncio.get_event_loop()
# use run_in_executor to run sync code in a separate thread
# while this thread runs the event loop
await loop.run_in_executor(None, sync_code)
asyncio.run(main())
If the above is not acceptable for you (e.g. because it turns the whole program into an asyncio program), you can also run the event loop in a background thread, and submit tasks to it using asyncio.run_coroutine_threadsafe. That approach would allow startTimer to have the signature (and interface) like you wanted it:
def startTimer(waitForSeconds):
loop = asyncio.new_event_loop()
threading.Thread(daemon=True, target=loop.run_forever).start()
async def sleep_and_run():
await asyncio.sleep(waitForSeconds)
await myAsyncFunc()
asyncio.run_coroutine_threadsafe(sleep_and_run(), loop)
async def myAsyncFunc():
print("in my async func")
startTimer(1)
while True:
print("e")
I'm pretty sure that you are familiar with concurent processing, but you didn't show exactly what you want. So if I understand you correctly you want to have 2 processes. First is doing only while True, and the second process is the timer(waits e.g. 5s) and it will call async task. I assume that you are using asyncio according to tags:
import asyncio
async def myAsyncFunc():
print("in my async func")
async def call_after(delay):
await asyncio.sleep(delay)
await myAsyncFunc()
async def while_true():
while True:
await asyncio.sleep(1) # sleep here to avoid to large output
print("e")
async def main():
task1 = asyncio.create_task(
while_true())
task2 = asyncio.create_task(
call_after(5))
# Wait until both tasks are completed (should take
# around 2 seconds.)
await task1
await task2
asyncio.run(main())
where
This is on Linux, Python 3.5.1.
what
I'm developing a monitor process with asyncio, whose tasks at various places await on asyncio.sleep calls of various durations.
There are points in time when I would like to be able to interrupt all said asyncio.sleep calls and let all tasks proceed normally, but I can't find how to do that. An example is for graceful shutdown of the monitor process.
how (failed assumption)
I thought that I could send an ALRM signal to that effect, but the process dies. I tried catching the ALRM signal with:
def sigalrm_sent(signum, frame):
tse.logger.info("got SIGALRM")
signal.signal(signal.SIGALRM, sigalrm_sent)
Then I get the log line about catching SIGALRM, but the asyncio.sleep calls are not interrupted.
how (kludge)
At this point, I replaced all asyncio.sleep calls with calls to this coroutine:
async def interruptible_sleep(seconds):
while seconds > 0 and not tse.stop_requested:
duration = min(seconds, tse.TIME_QUANTUM)
await asyncio.sleep(duration)
seconds -= duration
So I only have to pick a TIME_QUANTUM that is not too small and not too large either.
but
Is there a way to interrupt all running asyncio.sleep calls and I am missing it?
Interrupting all running calls of asyncio.sleep seems a bit dangerous since it can be used in other parts of the code, for other purposes. Instead I would make a dedicated sleep coroutine that keeps track of its running calls. It is then possible to interrupt them all by canceling the corresponding tasks:
def make_sleep():
async def sleep(delay, result=None, *, loop=None):
coro = asyncio.sleep(delay, result=result, loop=loop)
task = asyncio.ensure_future(coro)
sleep.tasks.add(task)
try:
return await task
except asyncio.CancelledError:
return result
finally:
sleep.tasks.remove(task)
sleep.tasks = set()
sleep.cancel_all = lambda: sum(task.cancel() for task in sleep.tasks)
return sleep
Example:
async def main(sleep, loop):
for i in range(10):
loop.create_task(sleep(i))
await sleep(3)
nb_cancelled = sleep.cancel_all()
await asyncio.wait(sleep.tasks)
return nb_cancelled
sleep = make_sleep()
loop = asyncio.get_event_loop()
result = loop.run_until_complete(main(sleep, loop))
print(result) # Print '6'
For debugging purposes, loop.time = lambda: float('inf') also works.
Based on Vincent's answer, I used the following class (every instance of the class can cancel all its running .sleep tasks, allowing better compartmentalization):
class Sleeper:
"Group sleep calls allowing instant cancellation of all"
def __init__(self, loop):
self.loop = loop
self.tasks = set()
async def sleep(self, delay, result=None):
coro = aio.sleep(delay, result=result, loop=self.loop)
task = aio.ensure_future(coro)
self.tasks.add(task)
try:
return await task
except aio.CancelledError:
return result
finally:
self.tasks.remove(task)
def cancel_all_helper(self):
"Cancel all pending sleep tasks"
cancelled = set()
for task in self.tasks:
if task.cancel():
cancelled.add(task)
return cancelled
async def cancel_all(self):
"Coroutine cancelling tasks"
cancelled = self.cancel_all_helper()
await aio.wait(self.tasks)
self.tasks -= cancelled
return len(cancelled)