We Keep Coding

Alternative to asyncio.gather which I can keep adding coroutines to at runtime?

I need to be able to keep adding coroutines to the asyncio loop at runtime. I tried using create_task() thinking that this would do what I want, but it still needs to be awaited.
This is the code I had, not sure if there is a simple edit to make it work?
async def get_value_from_api():
global ASYNC_CLIENT
return ASYNC_CLIENT.get(api_address)
async def print_subs():
count = await get_value_from_api()
print(count)
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
time.sleep(0.1)
async def start():
global ASYNC_CLIENT
async with httpx.AsyncClient() as ASYNC_CLIENT:
await save_subs_loop()
asyncio.run(start())

I once created similar pattern when I was mixing trio and kivy, which was demonstration of running multiple coroutines asynchronously.
It use a trio.MemoryChannel which is roughly equivalent to asyncio.Queue, I'll just refer it as queue here.
Main idea is:
Wrap each task with class, which has run function.
Make class object's own async method to put object itself into queue when execution is done.
Create a global task-spawning loop to wait for the object in queue and schedule execution/create task for the object.
import asyncio
import traceback
import httpx
async def task_1(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/")
print(resp.read())
await asyncio.sleep(0.1) # without this would be IP ban
async def task_2(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/meow/")
print(resp.read())
await asyncio.sleep(0.5)
class CoroutineWrapper:
def __init__(self, queue: asyncio.Queue, coro_func, *param):
self.func = coro_func
self.param = param
self.queue = queue
async def run(self):
try:
await self.func(*self.param)
except Exception:
traceback.print_exc()
return
# put itself back into queue
await self.queue.put(self)
class KeepRunning:
def __init__(self):
# queue for gathering CoroutineWrapper
self.queue = asyncio.Queue()
def add_task(self, coro, *param):
wrapped = CoroutineWrapper(self.queue, coro, *param)
# add tasks to be executed in queue
self.queue.put_nowait(wrapped)
async def task_processor(self):
task: CoroutineWrapper
while task := await self.queue.get():
# wait for new CoroutineWrapper Object then schedule it's async method execution
asyncio.create_task(task.run())
async def main():
keep_running = KeepRunning()
async with httpx.AsyncClient() as client:
keep_running.add_task(task_1, client)
keep_running.add_task(task_2, client)
await keep_running.task_processor()
asyncio.run(main())
Server
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return str(time.time())
#app.route("/meow/")
def meow():
return "meow"
app.run()
Output:
b'meow'
b'1639920445.965701'
b'1639920446.0767004'
b'1639920446.1887035'
b'1639920446.2986999'
b'1639920446.4067013'
b'meow'
b'1639920446.516704'
b'1639920446.6267014'
...
You can see tasks running repeatedly on their own pace.
Old answer
Seems like you only want to cycle fixed amount of tasks.
In that case just iterate list of coroutine with itertools.cycle
But this is no different with synchronous, so lemme know if you need is asynchronous.
import asyncio
import itertools
import httpx
async def main_task(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/")
print(resp.read())
await asyncio.sleep(0.1) # without this would be IP ban
async def main():
async with httpx.AsyncClient() as client:
for coroutine in itertools.cycle([main_task]):
await coroutine(client)
asyncio.run(main())
Server:
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return str(time.time())
app.run()
Output:
b'1639918937.7694323'
b'1639918937.8804302'
b'1639918937.9914327'
b'1639918938.1014295'
b'1639918938.2124324'
b'1639918938.3204308'
...

asyncio.create_task() works as you describe it. The problem you are having here is that you create an infinite loop here:
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
time.sleep(0.1) # do not use time.sleep() in async code EVER
save_subs_loop() keeps creating tasks but control is never yielded back to the event loop, because there is no await in there. Try
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
await asyncio.sleep(0.1) # yield control back to loop to give tasks a chance to actually run
This problem is so common I'm thinking python should raise a RuntimeError if it detects time.sleep() within a coroutine :-)

You might want to try the TaskThread framework
It allows you to add tasks in runtime
Tasks are re-scheduled periodically (like in your while loop up there)
There is a consumer / producer framework built in (parent/child relationships) which you seem to need
disclaimer: I wrote TaskThread out of necessity & it's been a life saver.

#tasks.loop() stopping commands from running until the loop is finished

I have a background loop involving selenium, so it takes a long time to finish executing. I noticed that the bot had a delay when responding to commands, and I found out that the processes inside #tasks.loop() needs to finish before the commands execute. For example:
from discord.ext import commands, tasks
import time
bot = commands.Bot(command_prefix='-')
#bot.command()
async def test(ctx):
await ctx.send('hi')
#tasks.loop(seconds=30)
async def loop():
print('h')
time.sleep(20)
print('i')
#bot.event
async def on_ready():
loop.start()
bot.run()
Here, if you do -test after it prints the letter h and before it prints the letter i, the bot will not respond until it prints the i and the loop finishes.
How would I make it so that the commands will be able to execute along with the loop? FYI my code doesn't have a time.sleep(), it was just an example.

If you have long running code then you should move it into separated function and run it with threading or `multiprocessing.
Here basic example with threading. It runs new thread in every loop. For something more complex it may need different menthod. It may need to run single thread before discord and use queue in loop to send information to thread.
from discord.ext import commands, tasks
import time
import threading
import os
bot = commands.Bot(command_prefix='-')
#bot.command()
async def test(ctx):
await ctx.send('hi')
def long_running_function():
print('long_running_function: start')
time.sleep(10)
print('long_running_function: end')
#tasks.loop(seconds=30)
async def loop():
print('h')
t = threading.Thread(target=long_running_function)
t.start()
print('i')
#bot.event
async def on_ready():
loop.start()
bot.run(os.getenv('DISCORD_TOKEN'))

python Make an async timer without waiting to finish

I want to make a timer which is started in a normal function, but in the timer function, it should be able to call an async function
I want to do something like this:
startTimer()
while True:
print("e")
def startTimer(waitForSeconds: int):
# Wait for `waitForSeconds`
await myAsyncFunc()
async def myAsyncFunc():
print("in my async func")
Where the while True loop should do its stuff and after waitForSeconds the timer the async function should execute an other async function, but waiting shouldn't block any other actions and doesn't need to be awaited
If something isn't understandable, I'm sorry, I'll try to explain it then
Thanks

If you want to run your synchronous and asynchronous code in parallel, you will need to run one of them in a separate thread. For example:
def sync_code():
while True:
print("e")
async def start_timer(secs):
await asyncio.sleep(secs)
await async_func()
async def main():
asyncio.create_task(start_timer(1))
loop = asyncio.get_event_loop()
# use run_in_executor to run sync code in a separate thread
# while this thread runs the event loop
await loop.run_in_executor(None, sync_code)
asyncio.run(main())
If the above is not acceptable for you (e.g. because it turns the whole program into an asyncio program), you can also run the event loop in a background thread, and submit tasks to it using asyncio.run_coroutine_threadsafe. That approach would allow startTimer to have the signature (and interface) like you wanted it:
def startTimer(waitForSeconds):
loop = asyncio.new_event_loop()
threading.Thread(daemon=True, target=loop.run_forever).start()
async def sleep_and_run():
await asyncio.sleep(waitForSeconds)
await myAsyncFunc()
asyncio.run_coroutine_threadsafe(sleep_and_run(), loop)
async def myAsyncFunc():
print("in my async func")
startTimer(1)
while True:
print("e")

I'm pretty sure that you are familiar with concurent processing, but you didn't show exactly what you want. So if I understand you correctly you want to have 2 processes. First is doing only while True, and the second process is the timer(waits e.g. 5s) and it will call async task. I assume that you are using asyncio according to tags:
import asyncio
async def myAsyncFunc():
print("in my async func")
async def call_after(delay):
await asyncio.sleep(delay)
await myAsyncFunc()
async def while_true():
while True:
await asyncio.sleep(1) # sleep here to avoid to large output
print("e")
async def main():
task1 = asyncio.create_task(
while_true())
task2 = asyncio.create_task(
call_after(5))
# Wait until both tasks are completed (should take
# around 2 seconds.)
await task1
await task2
asyncio.run(main())

Python multithreading with asyncio

I have a websocket subscribed to a client's news feed. And as soon as a receive a news, I want to take an action like the following:
from client import Client, SocketManager
import asyncio
from concurrent.futures import ThreadPoolExecutor
from time import time
class client(Client):
def __init__():
self.loop = asyncio.get_event_loop()
async def start_websockets(self):
self.sm = await SocketManager.create(self.loop, self, self.handle_evt)
await self.sm.subscribe()
while not self.received:
await asyncio.sleep(10, loop=self.loop)
async def handle_evt(self, msg):
self.msg = msg
subject1 = msg['data']['subjetct']
subject2 = msg['data']['subjetct2']
with ThreadPoolExecutor(max_workers=2) as executor:
future_buy = executor.submit(self.process, subject1)
future_sell = executor.submit(self.process, subject2)
future_buy.result()
future_sell.result()
def process(self, subject):
if 'type' in subject:
# do something
else:
# do something
async def main(self):
await self.start_websockets()
while True:
if time() > start + 60*5:
start = time()
print(self.msg)
def start(self):
self.loop.run_until_complete(self.main())
However, it looks like it get stuck in the ThreadPoolExecutor. I am probably missing a await somewhere, but i am not very familiar with asyncio.
Can someone help me out on this please?

Use asyncio's inbuilt executor and wrapper via loop.run_in_executor:
async def handle_evt(self, msg):
self.msg = msg
subject1 = msg['data']['subjetct']
subject2 = msg['data']['subjetct2']
loop = asyncio.get_running_loop()
future_buy = loop.run_in_executor(None, self.process, subject1)
future_sell = loop.run_in_executor(None, self.process, subject2)
# allow other coroutines to run while waiting
await future_buy
await future_sell
A concurrent.futures.Future is inherently synchronous – waiting for Future.result() blocks the current thread. Executing it inside an event loop thus blocks the event loop, including all coroutines.
The asyncio wrapper provides an asyncio.Future instead, which allows asynchronous waiting for results.

Python Asyncio schedule tasks weekly

I am writing a discord bot in Python, which already has the two async overwritten methods on_ready and on_message to react to messages in a certain way.
Now I wanted to add a function, which should be called once in a week. I tried something with asyncio.sleep() but I don't want to start the bot at that specific time and then sleep 604.800 sec (1 week in seconds) to repeat that function every week.
Here's what I got so far:
class MyClient(discord.Client):
async def on_ready(self):
#some initial stuff
self.loop.create_task(self.routine())
async def on_message(self, message):
#reply to chat messages
await message.channel.send("Reply")
async def routine(self):
while True:
print("I'm on that routine boi")
await asyncio.sleep(3)
It works so far, that I can use the bot in discord and get the outputs every 3 seconds.
But I'd prefer something like from the schedule module to use something like
scheduler.every().sunday.at("8:55").do(self.routine())
Any chance to do something similar in combination with asyncio?

The Event loop provides mechanisms to schedule callback functions to be called at some point in the future, as shown here in Python docs
class MyClient(discord.Client):
async def on_ready(self):
#some initial stuff
self.loop.create_task(self.routine())
async def on_message(self, message):
#reply to chat messages
await message.channel.send("Reply")
async def execute():
res = await loop.call_later(delay, callback, *args)
delay += delay + 604800
return res
async def routine(self):
while True:
print("I'm on that routine boi")
res = await loop.run_until_complete(execute)