We Keep Coding

In Python, kick off an async function and return early

I have a function in python that should kick off an asynchronous 'slow' function, and return before that 'slow' function completes.
async def doSomethingSlowButReturnQuickly():
asyncio.create_task(_mySlowFunction())
return "Returning early! Slow function is still running."
async def _mySlowFunction():
// Do some really slow stuff that isn't blocking for our other function.
Running this still seems to cause the 'return' to not happen until AFTER my async task has completed.
How do I correct this?
(And apologies, this is day #2 writing Python)

This isn't exactly what you asked, but you can run code in an async executor and continue while it is still running:
from concurrent.futures import ThreadPoolExecutor
from time import sleep
def blocking_code():
sleep(2)
print('inner')
return 'result'
def main():
with ThreadPoolExecutor(max_workers=1) as executor:
future = executor.submit(blocking_code)
print('after')
print(future.result())
return
if __name__ == '__main__':
main()
Output:
after
inner
result

Maybe it's a dull answer, but if you want to stick with asyncio, I would advise to be sure using await in everyasync def functions.
async def doSomethingSlowButReturnQuickly():
task = asyncio.create_task(_mySlowFunction())
# must await something here!
return "Returning early! Slow function is still running."
async def _mySlowFunction():
# must await something here
If you don't need to get the slow function return value, why don't you directly call the two coroutines from a main function?
import asyncio
async def doSomethingSlowButReturnQuickly():
await asyncio.sleep(0.1) # must await something here!
return "Returning early! Slow function is still running."
async def _mySlowFunction():
await asyncio.sleep(10) # must await something here
async def main():
tasks = [_mySlowFunction(), doSomethingSlowButReturnQuickly()]
asyncio.gather(*tasks)
asyncio.run(main())

How to stop an async function

I want to find a way to stop the call of a function
Currently I found this method in function
from func_timeout import func_set_timeout
######## is ok #########
#func_set_timeout(timeout=2)
def is_ok_request():
import time
time.sleep(10)
is_ok_request()
But currently I can't stop the call in an async function
def down_file():
'''eg. this a third-party modules '''
time.sleep(100000)
async def timeout_func():
'''down a file times out 10s to exit'''
print("start connection mysql")
down_file()
print("end connection mysql")
async def main():
try:
await asyncio.wait_for(timeout_func(),timeout=1)
except asyncio.TimeoutError:
print('timeout')
asyncio.run(main())
help

There are several problems:
Don't call time.sleep() in async progams! Always await asyncio.sleep() instead.
The timeout of asyncio.wait (link) is the time when to stop waiting. It does not cancel anything. Use asyncio.wait_for (link) instead. It generates a TimeoutError that should be handled.
Not an error, but loop.run_until_complete() is not the recommended way to run an async program. Use asyncio.run() as the entry-point, it is like run_until_complete with a cleanup afterward.
Another issue: task = my_request(). It is not a task, it is a coroutine. In asyncio, the term task has a fixed meaning (link). The wait documentation warns, that it expects tasks and will not accept coroutines in future versions.
The code in its simplest form (actually, it is almost the same as an example in the linked docs):
import asyncio
import time
async def my_request():
await asyncio.sleep(10)
async def main():
try:
await asyncio.wait_for(my_request(), timeout=1)
except asyncio.TimeoutError:
print('timeout')
asyncio.run(main())

Alternative to asyncio.gather which I can keep adding coroutines to at runtime?

I need to be able to keep adding coroutines to the asyncio loop at runtime. I tried using create_task() thinking that this would do what I want, but it still needs to be awaited.
This is the code I had, not sure if there is a simple edit to make it work?
async def get_value_from_api():
global ASYNC_CLIENT
return ASYNC_CLIENT.get(api_address)
async def print_subs():
count = await get_value_from_api()
print(count)
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
time.sleep(0.1)
async def start():
global ASYNC_CLIENT
async with httpx.AsyncClient() as ASYNC_CLIENT:
await save_subs_loop()
asyncio.run(start())

I once created similar pattern when I was mixing trio and kivy, which was demonstration of running multiple coroutines asynchronously.
It use a trio.MemoryChannel which is roughly equivalent to asyncio.Queue, I'll just refer it as queue here.
Main idea is:
Wrap each task with class, which has run function.
Make class object's own async method to put object itself into queue when execution is done.
Create a global task-spawning loop to wait for the object in queue and schedule execution/create task for the object.
import asyncio
import traceback
import httpx
async def task_1(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/")
print(resp.read())
await asyncio.sleep(0.1) # without this would be IP ban
async def task_2(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/meow/")
print(resp.read())
await asyncio.sleep(0.5)
class CoroutineWrapper:
def __init__(self, queue: asyncio.Queue, coro_func, *param):
self.func = coro_func
self.param = param
self.queue = queue
async def run(self):
try:
await self.func(*self.param)
except Exception:
traceback.print_exc()
return
# put itself back into queue
await self.queue.put(self)
class KeepRunning:
def __init__(self):
# queue for gathering CoroutineWrapper
self.queue = asyncio.Queue()
def add_task(self, coro, *param):
wrapped = CoroutineWrapper(self.queue, coro, *param)
# add tasks to be executed in queue
self.queue.put_nowait(wrapped)
async def task_processor(self):
task: CoroutineWrapper
while task := await self.queue.get():
# wait for new CoroutineWrapper Object then schedule it's async method execution
asyncio.create_task(task.run())
async def main():
keep_running = KeepRunning()
async with httpx.AsyncClient() as client:
keep_running.add_task(task_1, client)
keep_running.add_task(task_2, client)
await keep_running.task_processor()
asyncio.run(main())
Server
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return str(time.time())
#app.route("/meow/")
def meow():
return "meow"
app.run()
Output:
b'meow'
b'1639920445.965701'
b'1639920446.0767004'
b'1639920446.1887035'
b'1639920446.2986999'
b'1639920446.4067013'
b'meow'
b'1639920446.516704'
b'1639920446.6267014'
...
You can see tasks running repeatedly on their own pace.
Old answer
Seems like you only want to cycle fixed amount of tasks.
In that case just iterate list of coroutine with itertools.cycle
But this is no different with synchronous, so lemme know if you need is asynchronous.
import asyncio
import itertools
import httpx
async def main_task(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/")
print(resp.read())
await asyncio.sleep(0.1) # without this would be IP ban
async def main():
async with httpx.AsyncClient() as client:
for coroutine in itertools.cycle([main_task]):
await coroutine(client)
asyncio.run(main())
Server:
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return str(time.time())
app.run()
Output:
b'1639918937.7694323'
b'1639918937.8804302'
b'1639918937.9914327'
b'1639918938.1014295'
b'1639918938.2124324'
b'1639918938.3204308'
...

asyncio.create_task() works as you describe it. The problem you are having here is that you create an infinite loop here:
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
time.sleep(0.1) # do not use time.sleep() in async code EVER
save_subs_loop() keeps creating tasks but control is never yielded back to the event loop, because there is no await in there. Try
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
await asyncio.sleep(0.1) # yield control back to loop to give tasks a chance to actually run
This problem is so common I'm thinking python should raise a RuntimeError if it detects time.sleep() within a coroutine :-)

You might want to try the TaskThread framework
It allows you to add tasks in runtime
Tasks are re-scheduled periodically (like in your while loop up there)
There is a consumer / producer framework built in (parent/child relationships) which you seem to need
disclaimer: I wrote TaskThread out of necessity & it's been a life saver.

python Make an async timer without waiting to finish

I want to make a timer which is started in a normal function, but in the timer function, it should be able to call an async function
I want to do something like this:
startTimer()
while True:
print("e")
def startTimer(waitForSeconds: int):
# Wait for `waitForSeconds`
await myAsyncFunc()
async def myAsyncFunc():
print("in my async func")
Where the while True loop should do its stuff and after waitForSeconds the timer the async function should execute an other async function, but waiting shouldn't block any other actions and doesn't need to be awaited
If something isn't understandable, I'm sorry, I'll try to explain it then
Thanks

If you want to run your synchronous and asynchronous code in parallel, you will need to run one of them in a separate thread. For example:
def sync_code():
while True:
print("e")
async def start_timer(secs):
await asyncio.sleep(secs)
await async_func()
async def main():
asyncio.create_task(start_timer(1))
loop = asyncio.get_event_loop()
# use run_in_executor to run sync code in a separate thread
# while this thread runs the event loop
await loop.run_in_executor(None, sync_code)
asyncio.run(main())
If the above is not acceptable for you (e.g. because it turns the whole program into an asyncio program), you can also run the event loop in a background thread, and submit tasks to it using asyncio.run_coroutine_threadsafe. That approach would allow startTimer to have the signature (and interface) like you wanted it:
def startTimer(waitForSeconds):
loop = asyncio.new_event_loop()
threading.Thread(daemon=True, target=loop.run_forever).start()
async def sleep_and_run():
await asyncio.sleep(waitForSeconds)
await myAsyncFunc()
asyncio.run_coroutine_threadsafe(sleep_and_run(), loop)
async def myAsyncFunc():
print("in my async func")
startTimer(1)
while True:
print("e")

I'm pretty sure that you are familiar with concurent processing, but you didn't show exactly what you want. So if I understand you correctly you want to have 2 processes. First is doing only while True, and the second process is the timer(waits e.g. 5s) and it will call async task. I assume that you are using asyncio according to tags:
import asyncio
async def myAsyncFunc():
print("in my async func")
async def call_after(delay):
await asyncio.sleep(delay)
await myAsyncFunc()
async def while_true():
while True:
await asyncio.sleep(1) # sleep here to avoid to large output
print("e")
async def main():
task1 = asyncio.create_task(
while_true())
task2 = asyncio.create_task(
call_after(5))
# Wait until both tasks are completed (should take
# around 2 seconds.)
await task1
await task2
asyncio.run(main())

"Fire and forget" python async/await

Sometimes there is some non-critical asynchronous operation that needs to happen but I don't want to wait for it to complete. In Tornado's coroutine implementation you can "fire & forget" an asynchronous function by simply ommitting the yield key-word.
I've been trying to figure out how to "fire & forget" with the new async/await syntax released in Python 3.5. E.g., a simplified code snippet:
async def async_foo():
print("Do some stuff asynchronously here...")
def bar():
async_foo() # fire and forget "async_foo()"
bar()
What happens though is that bar() never executes and instead we get a runtime warning:
RuntimeWarning: coroutine 'async_foo' was never awaited
async_foo() # fire and forget "async_foo()"

Upd:
Replace asyncio.ensure_future with asyncio.create_task everywhere if you're using Python >= 3.7 It's a newer, nicer way to spawn tasks.
asyncio.Task to "fire and forget"
According to python docs for asyncio.Task it is possible to start some coroutine to execute "in the background". The task created by asyncio.ensure_future won't block the execution (therefore the function will return immediately!). This looks like a way to "fire and forget" as you requested.
import asyncio
async def async_foo():
print("async_foo started")
await asyncio.sleep(1)
print("async_foo done")
async def main():
asyncio.ensure_future(async_foo()) # fire and forget async_foo()
# btw, you can also create tasks inside non-async funcs
print('Do some actions 1')
await asyncio.sleep(1)
print('Do some actions 2')
await asyncio.sleep(1)
print('Do some actions 3')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
Output:
Do some actions 1
async_foo started
Do some actions 2
async_foo done
Do some actions 3
What if tasks are executing after the event loop has completed?
Note that asyncio expects tasks to be completed at the moment the event loop completes. So if you'll change main() to:
async def main():
asyncio.ensure_future(async_foo()) # fire and forget
print('Do some actions 1')
await asyncio.sleep(0.1)
print('Do some actions 2')
You'll get this warning after the program finished:
Task was destroyed but it is pending!
task: <Task pending coro=<async_foo() running at [...]
To prevent that you can just await all pending tasks after the event loop has completed:
async def main():
asyncio.ensure_future(async_foo()) # fire and forget
print('Do some actions 1')
await asyncio.sleep(0.1)
print('Do some actions 2')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
# Let's also finish all running tasks:
pending = asyncio.Task.all_tasks()
loop.run_until_complete(asyncio.gather(*pending))
Kill tasks instead of awaiting them
Sometimes you don't want to await tasks to be done (for example, some tasks may be created to run forever). In that case, you can just cancel() them instead of awaiting them:
import asyncio
from contextlib import suppress
async def echo_forever():
while True:
print("echo")
await asyncio.sleep(1)
async def main():
asyncio.ensure_future(echo_forever()) # fire and forget
print('Do some actions 1')
await asyncio.sleep(1)
print('Do some actions 2')
await asyncio.sleep(1)
print('Do some actions 3')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
# Let's also cancel all running tasks:
pending = asyncio.Task.all_tasks()
for task in pending:
task.cancel()
# Now we should await task to execute it's cancellation.
# Cancelled task raises asyncio.CancelledError that we can suppress:
with suppress(asyncio.CancelledError):
loop.run_until_complete(task)
Output:
Do some actions 1
echo
Do some actions 2
echo
Do some actions 3
echo

Output:
>>> Hello
>>> foo() started
>>> I didn't wait for foo()
>>> foo() completed
Here is the simple decorator function which pushes the execution to background and line of control moves to next line of the code.
The primary advantage is, you don't have to declare the function as await
import asyncio
import time
def fire_and_forget(f):
def wrapped(*args, **kwargs):
return asyncio.get_event_loop().run_in_executor(None, f, *args, *kwargs)
return wrapped
#fire_and_forget
def foo():
print("foo() started")
time.sleep(1)
print("foo() completed")
print("Hello")
foo()
print("I didn't wait for foo()")
Note: Check my other answer which does the same using plain thread without asyncio.

This is not entirely asynchronous execution, but maybe run_in_executor() is suitable for you.
def fire_and_forget(task, *args, **kwargs):
loop = asyncio.get_event_loop()
if callable(task):
return loop.run_in_executor(None, task, *args, **kwargs)
else:
raise TypeError('Task must be a callable')
def foo():
#asynchronous stuff here
fire_and_forget(foo)

For some reason if you are unable to use asyncio then here is the implementation using plain threads. Check my other answers and Sergey's answer too.
import threading, time
def fire_and_forget(f):
def wrapped():
threading.Thread(target=f).start()
return wrapped
#fire_and_forget
def foo():
print("foo() started")
time.sleep(1)
print("foo() completed")
print("Hello")
foo()
print("I didn't wait for foo()")
produces
>>> Hello
>>> foo() started
>>> I didn't wait for foo()
>>> foo() completed

def fire_and_forget(f):
def wrapped(*args, **kwargs):
threading.Thread(target=functools.partial(f, *args, **kwargs)).start()
return wrapped
is the better version of the above -- does not use asyncio