We Keep Coding

Pass async function (changes channel name) into thread

I'm trying to pass an async function into a #tasks.loop thread since my commands don't work until the task is finished, if threading isn't used. However the task I'm trying to do every 5 minutes is to change the channel name, and it doesn't work. I get a Timeout context manager should be used error.
async def change_name()
channel = client.get_channel(chid)
await chid.edit(name='yes')
#tasks.loop(minutes=5)
async def loop():
_thread = threading.Thread(target=asyncio.run,args=())
_thread.start()
loop.start()

You can try python-worker for it (link)
from worker import async_worker
#async_worker
async def change_name()
channel = client.get_channel(chid)
await chid.edit(name='yes')
#tasks.loop(minutes=5)
async def loop():
await change_name()
loop.start()
your change_name will be run as thread automatically

Alternative to asyncio.gather which I can keep adding coroutines to at runtime?

I need to be able to keep adding coroutines to the asyncio loop at runtime. I tried using create_task() thinking that this would do what I want, but it still needs to be awaited.
This is the code I had, not sure if there is a simple edit to make it work?
async def get_value_from_api():
global ASYNC_CLIENT
return ASYNC_CLIENT.get(api_address)
async def print_subs():
count = await get_value_from_api()
print(count)
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
time.sleep(0.1)
async def start():
global ASYNC_CLIENT
async with httpx.AsyncClient() as ASYNC_CLIENT:
await save_subs_loop()
asyncio.run(start())

I once created similar pattern when I was mixing trio and kivy, which was demonstration of running multiple coroutines asynchronously.
It use a trio.MemoryChannel which is roughly equivalent to asyncio.Queue, I'll just refer it as queue here.
Main idea is:
Wrap each task with class, which has run function.
Make class object's own async method to put object itself into queue when execution is done.
Create a global task-spawning loop to wait for the object in queue and schedule execution/create task for the object.
import asyncio
import traceback
import httpx
async def task_1(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/")
print(resp.read())
await asyncio.sleep(0.1) # without this would be IP ban
async def task_2(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/meow/")
print(resp.read())
await asyncio.sleep(0.5)
class CoroutineWrapper:
def __init__(self, queue: asyncio.Queue, coro_func, *param):
self.func = coro_func
self.param = param
self.queue = queue
async def run(self):
try:
await self.func(*self.param)
except Exception:
traceback.print_exc()
return
# put itself back into queue
await self.queue.put(self)
class KeepRunning:
def __init__(self):
# queue for gathering CoroutineWrapper
self.queue = asyncio.Queue()
def add_task(self, coro, *param):
wrapped = CoroutineWrapper(self.queue, coro, *param)
# add tasks to be executed in queue
self.queue.put_nowait(wrapped)
async def task_processor(self):
task: CoroutineWrapper
while task := await self.queue.get():
# wait for new CoroutineWrapper Object then schedule it's async method execution
asyncio.create_task(task.run())
async def main():
keep_running = KeepRunning()
async with httpx.AsyncClient() as client:
keep_running.add_task(task_1, client)
keep_running.add_task(task_2, client)
await keep_running.task_processor()
asyncio.run(main())
Server
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return str(time.time())
#app.route("/meow/")
def meow():
return "meow"
app.run()
Output:
b'meow'
b'1639920445.965701'
b'1639920446.0767004'
b'1639920446.1887035'
b'1639920446.2986999'
b'1639920446.4067013'
b'meow'
b'1639920446.516704'
b'1639920446.6267014'
...
You can see tasks running repeatedly on their own pace.
Old answer
Seems like you only want to cycle fixed amount of tasks.
In that case just iterate list of coroutine with itertools.cycle
But this is no different with synchronous, so lemme know if you need is asynchronous.
import asyncio
import itertools
import httpx
async def main_task(client: httpx.AsyncClient):
resp = await client.get("http://127.0.0.1:5000/")
print(resp.read())
await asyncio.sleep(0.1) # without this would be IP ban
async def main():
async with httpx.AsyncClient() as client:
for coroutine in itertools.cycle([main_task]):
await coroutine(client)
asyncio.run(main())
Server:
import time
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return str(time.time())
app.run()
Output:
b'1639918937.7694323'
b'1639918937.8804302'
b'1639918937.9914327'
b'1639918938.1014295'
b'1639918938.2124324'
b'1639918938.3204308'
...

asyncio.create_task() works as you describe it. The problem you are having here is that you create an infinite loop here:
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
time.sleep(0.1) # do not use time.sleep() in async code EVER
save_subs_loop() keeps creating tasks but control is never yielded back to the event loop, because there is no await in there. Try
async def save_subs_loop():
while True:
asyncio.create_task(print_subs())
await asyncio.sleep(0.1) # yield control back to loop to give tasks a chance to actually run
This problem is so common I'm thinking python should raise a RuntimeError if it detects time.sleep() within a coroutine :-)

You might want to try the TaskThread framework
It allows you to add tasks in runtime
Tasks are re-scheduled periodically (like in your while loop up there)
There is a consumer / producer framework built in (parent/child relationships) which you seem to need
disclaimer: I wrote TaskThread out of necessity & it's been a life saver.

Trying to understand how to use multithreaded websockets in Python but seem to be stuck with one thread

I have this basic exchange monitor script. I'm trying to create one thread per symbol, apart from the main thread which is handling other work, and have them listen to public Gemini websocket endpoints. I'm getting the first thread running and printing exchange data to the console, but not the second one. I had expected to see data from both threads being printed at approximately the same time. I've tried using the threading library instead of asyncio and encountered the same situation.
I realize my two public API MarketWebsocket classes could be combined to be cleaner, I'm still trying to work out a way to easily add other symbols to the list. Thanks for any nudges in the right direction!
import asyncio
from websockets import connect
symbols_to_watch = [
"BTCUSD",
"ETHUSD"
]
class BTCMarketWebsocket:
disable = False
async def __aenter__(self):
symbol = symbols_to_watch[0]
self._conn = connect("wss://api.gemini.com/v1/marketdata/{}".format(symbol))
self.websocket = await self._conn.__aenter__()
return self
async def __aexit__(self, *args, **kwargs):
await self._conn.__aexit__(*args, **kwargs)
async def receive(self):
return await self.websocket.recv()
class ETHMarketWebsocket:
disable = False
async def __aenter__(self):
symbol = symbols_to_watch[1]
self._conn = connect("wss://api.gemini.com/v1/marketdata/{}".format(symbol))
self.websocket = await self._conn.__aenter__()
return self
async def __aexit__(self, *args, **kwargs):
await self._conn.__aexit__(*args, **kwargs)
async def receive(self):
return await self.websocket.recv()
async def btcMarketWebsocket():
async with BTCMarketWebsocket() as btcMarketWebsocket:
while not btcMarketWebsocket.disable:
print(await btcMarketWebsocket.receive())
async def ethMarketWebsocket():
async with ETHMarketWebsocket() as ethMarketWebsocket:
while not ethMarketWebsocket.disable:
print(await ethMarketWebsocket.receive())
if __name__ == '__main__':
asyncio.run(btcMarketWebsocket())
asyncio.run(ethMarketWebsocket())

You can do
async def multiple_tasks():
Tasks =[]
Tasks.append(btcMarketWebsocket())
Tasks.append(ethMarketWebsocket())
await asyncio.gather(*Tasks, return_exceptions=True)
if __name__ == '__main__':
asyncio.get_event_loop().run_until_complete(multiple_tasks())

Python multithreading with asyncio

I have a websocket subscribed to a client's news feed. And as soon as a receive a news, I want to take an action like the following:
from client import Client, SocketManager
import asyncio
from concurrent.futures import ThreadPoolExecutor
from time import time
class client(Client):
def __init__():
self.loop = asyncio.get_event_loop()
async def start_websockets(self):
self.sm = await SocketManager.create(self.loop, self, self.handle_evt)
await self.sm.subscribe()
while not self.received:
await asyncio.sleep(10, loop=self.loop)
async def handle_evt(self, msg):
self.msg = msg
subject1 = msg['data']['subjetct']
subject2 = msg['data']['subjetct2']
with ThreadPoolExecutor(max_workers=2) as executor:
future_buy = executor.submit(self.process, subject1)
future_sell = executor.submit(self.process, subject2)
future_buy.result()
future_sell.result()
def process(self, subject):
if 'type' in subject:
# do something
else:
# do something
async def main(self):
await self.start_websockets()
while True:
if time() > start + 60*5:
start = time()
print(self.msg)
def start(self):
self.loop.run_until_complete(self.main())
However, it looks like it get stuck in the ThreadPoolExecutor. I am probably missing a await somewhere, but i am not very familiar with asyncio.
Can someone help me out on this please?

Use asyncio's inbuilt executor and wrapper via loop.run_in_executor:
async def handle_evt(self, msg):
self.msg = msg
subject1 = msg['data']['subjetct']
subject2 = msg['data']['subjetct2']
loop = asyncio.get_running_loop()
future_buy = loop.run_in_executor(None, self.process, subject1)
future_sell = loop.run_in_executor(None, self.process, subject2)
# allow other coroutines to run while waiting
await future_buy
await future_sell
A concurrent.futures.Future is inherently synchronous – waiting for Future.result() blocks the current thread. Executing it inside an event loop thus blocks the event loop, including all coroutines.
The asyncio wrapper provides an asyncio.Future instead, which allows asynchronous waiting for results.

Python Asyncio schedule tasks weekly

I am writing a discord bot in Python, which already has the two async overwritten methods on_ready and on_message to react to messages in a certain way.
Now I wanted to add a function, which should be called once in a week. I tried something with asyncio.sleep() but I don't want to start the bot at that specific time and then sleep 604.800 sec (1 week in seconds) to repeat that function every week.
Here's what I got so far:
class MyClient(discord.Client):
async def on_ready(self):
#some initial stuff
self.loop.create_task(self.routine())
async def on_message(self, message):
#reply to chat messages
await message.channel.send("Reply")
async def routine(self):
while True:
print("I'm on that routine boi")
await asyncio.sleep(3)
It works so far, that I can use the bot in discord and get the outputs every 3 seconds.
But I'd prefer something like from the schedule module to use something like
scheduler.every().sunday.at("8:55").do(self.routine())
Any chance to do something similar in combination with asyncio?

The Event loop provides mechanisms to schedule callback functions to be called at some point in the future, as shown here in Python docs
class MyClient(discord.Client):
async def on_ready(self):
#some initial stuff
self.loop.create_task(self.routine())
async def on_message(self, message):
#reply to chat messages
await message.channel.send("Reply")
async def execute():
res = await loop.call_later(delay, callback, *args)
delay += delay + 604800
return res
async def routine(self):
while True:
print("I'm on that routine boi")
res = await loop.run_until_complete(execute)