We Keep Coding

How to intergrate telethon script into my event loop?

I have two bots, one is using pydle for IRC, like:
async def start_ircbot ():
try:
client = MyOwnBot(NICK,
realname=REALNAME,
sasl_username=SASL_USERNAME,
sasl_password=SASL_PASSWORD,
sasl_identity=SASL_IDENTITY,)
loop = asyncio.get_event_loop()
asyncio.ensure_future(client.connect(HOST, PORT, tls=True, tls_verify=False), loop=loop)
loop.run_forever()
loop.close()
except Exception as e:
print (e)
and another is using telethon for Telegram:
#client.on(events.NewMessage)
async def my_event_handler(event):
...
async def start_client ():
print ("Telegram monitor started...")
await client.start()
await client.run_until_disconnected()
Both of them work without problem separately.
Now, I want to integrate both of them, I tried to launch both of them in my main function like this,
import Notifier
...
async def main():
await asyncio.gather (Notifier.start_client (), start_ircbot ())
asyncio.run(main())
It starts without issue but my_event_handler seems never to get new messages. If I swap the order of functions:
await asyncio.gather (start_ircbot (), Notifier.start_client ())
The script will be stuck at launching, I suspect it has to be something within events loops and tried some different methods but without luck, could anyone shed light on this for me?

Newer Python versions are removing the loop parameter from most methods, so you should try to avoid using it. As long as you don't use asyncio.run (which creates a new loop) or you don't create a new loop yourself, both libraries should be using the default loop from the main thread (I can't speak for pydle, but Telethon does this).
As long as the asyncio event loop is running, Telethon should have no trouble receiving updates. You can use client.loop to make sure it's using the same loop:
tlclient = TelegramClient(...)
irclient = MyOwnBot(...)
#tlclient.on(events.NewMessage)
async def my_event_handler(event):
...
async def main():
await tlclient.start()
await irclient.connect(HOST, PORT, tls=True, tls_verify=False), loop=tlclient.tlclient)
await tlclient.run_until_disconnected() # similar to loop.run_forever() but stops when the client disconnects
client.loop.run_until_complete(main())

python3.6 async/await still works synchronously with fastAPI

I have a fastAPI app that posts two requests, one of them is longer (if it helps, they're Elasticsearch queries and I'm using the AsyncElasticsearch module which already returns coroutine). This is my attempt:
class my_module:
search_object = AsyncElasticsearch(url, port)
async def do_things(self):
resp1 = await search_object.search() #the longer one
print(check_resp1)
resp2 = await search_object.search() #the shorter one
print(check_resp2)
process(resp2)
process(resp1)
do_synchronous_things()
return thing
app = FastAPI()
#app.post("/")
async def service(user_input):
result = await my_module.do_things()
return results
What I observed is instead of awaiting resp1, by the time it got to check_resp1 it's already a full response, as if I didn't use async at all.
I'm new to python async, I knew my code wouldn't work, but I don't know how to fix it. As far as I understand, when interpreter sees await it starts the function then just moves on, which in this case should immediately post the next request. How do I make it do that?

Yes, that's correct the coroutine won't proceed until the results are ready. You can use asyncio.gather to run tasks concurrently:
import asyncio
async def task(msg):
print(f"START {msg}")
await asyncio.sleep(1)
print(f"END {msg}")
return msg
async def main():
await task("1")
await task("2")
results = await asyncio.gather(task("3"), task("4"))
print(results)
if __name__ == "__main__":
asyncio.run(main())
Test:
$ python test.py
START 1
END 1
START 2
END 2
START 3
START 4
END 3
END 4
['3', '4']
Alternatively you can use asyncio.as_completed to get the earliest next result:
for coro in asyncio.as_completed((task("5"), task("6"))):
earliest_result = await coro
print(earliest_result)
Update Fri 2 Apr 09:25:33 UTC 2021:
asyncio.run is available since Python 3.7+, in previous versions you will have to create and start the loop manually:
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

Explanation
The reason your code run synchronyously is that in do_things function, the code is executed as follow:
Schedule search_object.search() to execute
Wait till search_object.search() is finished and get the result
Schedule search_object.search() to execute
Wait till search_object.search() is finished and get the result
Execute (synchronyously) process(resp2)
Execute (synchronyously) process(resp1)
Execute (synchronyously) do_synchronous_things()
What you intended, is to make steps 1 and 3 executed before 2 and 4. You can make it easily with unsync library - here is the documentation.
How you can fix this
from unsync import unsync
class my_module:
search_object = AsyncElasticsearch(url, port)
#unsync
async def search1():
return await search_object.search()
#unsync
async def search2(): # not sure if this is any different to search1
return await search_object.search()
async def do_things(self):
task1, task2 = self.search1(), self.search2() # schedule tasks
resp1, resp2 = task1.result(), task2.result() # wait till tasks are executed
# you might also do similar trick with process function to run process(resp2) and process(resp1) concurrently
process(resp2)
process(resp1)
do_synchronous_things() # if this does not rely on resp1 and resp2 it might also be put into separate task to make the computation quicker. To do this use #unsync(cpu_bound=True) decorator
return thing
app = FastAPI()
#app.post("/")
async def service(user_input):
result = await my_module.do_things()
return results
More information
If you want to learn more about asyncio and asyncronyous programming, I recommend this tutorial. There is also similar case that you presented with a few possible solutions to make the coroutines run concurrently.
PS. Obviosuly I could not run this code, so you must debug it on your own.

Using OSC to update sliders in GANSpace

I'm trying to update the sliders in interactive.py from GANSpace with messages from python-osc.
Ideally the on_draw function should run after receiving a series of OSC messages.
But I'm having trouble with implementing it, because the serve.forever() function is blocking and i can't figure out how to implement the async version.
When i do like this:
def set_slider_osc(addr, args):
print("[{0}] ~ {1}".format(args, addr))
#print(re.split('/',addr)[2])
if "/0" in addr:
ui_state.sliders[0].set(args)
if "/1" in addr:
ui_state.sliders[1].set(args)
async def loop():
while not pending_close:
root.update()
app.update()
on_draw()
reposition_toolbar()
await asyncio.sleep(0)
async def init_main():
setup_model()
setup_ui()
resample_latent()
pending_close = False
dispatcher = dispatcher.Dispatcher()
dispatcher.map("/Chroma/*", set_slider_osc)
server = AsyncIOOSCUDPServer(("127.0.0.1", 8000), dispatcher, asyncio.get_event_loop())
transport, protocol = await server.create_serve_endpoint() # Create datagram endpoint and start serving
await loop() # Enter main loop of program
transport.close() # Clean up serve endpoint
root.destroy()
#
if __name__=='__main__':
await init_main()
I get the error that i try to run 'await' outside a function and another way i tried it seemed like there already was a main loop running and therefore couldn't run a new..
I know it's allot to aks people to install GANSpace, but hope someone may shine some light on how one could do this.

await cannot be used outside of a function defined with async def. You are trying to use it outside a function. The fix is to run init_main in an event loop.
import asyncio
asyncio.run(init_main())

Ways to optimize simple asyncio program where TCP clients are persistent

Using Python 3.7.4 and the asyncio package I'm trying to write an application that should spawn around 20000 (20k or more) TCP clients which then connect to a single server.
The clients then wait for a command from the server (received_data = await reader.read(4096)) and proceed to executing it (await loop.run_in_executor(...)) then send the response back to the server (writer.write(resp)).
After this cycle is completed, I sleep 100ms (await asyncio.sleep(100e-3)) in order to allow other coroutines to run.
The 20k clients should never disconnect and should process commands from the server indefinitely.
I'm interested in ways I can change the code to optimize it (barring the use of uvloop or directly implementing a Protocol since I saw in uvloop's docs this could improve the performance) beyond what it is capable now.
Let's assume that I cannot modify handle_request.
For example the await asyncio.sleep(100e-3) is especially bothering me, but I had to add it there, otherwise the impression was that no other coroutines ran other than the first one! Why could that be?
Say I remove the sleep (since in theory the other awaits should allow other coroutines to run), what else could I do?
Below is a minimal example of what my application looks like:
import asyncio
from collections import namedtuple
import logging
import os
import sys
logger = logging.getLogger(__name__)
should_exit = asyncio.Event()
def exit(signame, loop):
should_exit.set()
logger.warning('Exiting soon...')
def handle_request(received_data, entity):
logger.info('Backend logic here that consumes a bit of time depending on the entity and the received_data')
async def run_entity(entity, args):
logger.info(f'Running entity {entity}')
loop = asyncio.get_running_loop()
try:
reader, writer = await asyncio.open_connection(args.addr[0], int(args.addr[1]))
logger.debug(f'{entity} connected to {args.addr[0]}:{args.addr[1]}')
try:
while not should_exit.is_set():
received_data = await reader.read(4096)
if received_data:
logger.debug(f'{entity} received data {received_data}')
success, resp = await loop.run_in_executor(None, functools.partial(handle_request, received_data, entity))
if success:
logger.debug(f'{entity} sending response {resp}')
writer.write(resp)
await writer.drain()
await asyncio.sleep(100e-3)
except ConnectionResetError:
pass
except ConnectionRefusedError:
logger.warning(f'Connection refused by {args.addr[0]}:{args.addr[1]}.')
except Exception:
logger.exception('Details of unexpected error:')
logger.info(f'Stopped entity {entity}')
async def main(entities, args):
if os.name == 'posix':
loop = asyncio.get_running_loop()
loop.add_signal_handler(signal.SIGTERM, functools.partial(exit, signal.SIGTERM, loop))
loop.add_signal_handler(signal.SIGINT, functools.partial(exit, signal.SIGINT, loop))
tasks = (run_entity(entity, args) for entity in entities)
await asyncio.gather(*tasks)
if __name__ == '__main__':
ArgsReplacement = namedtuple('ArgsReplacement', ['addr'])
asyncio.run(main(range(20000), ArgsReplacement(addr=['127.0.0.1', '4242'])))

push with python + asyncio + websockets = missing messages

I'm trying to create a websocket server on which a single client will push its messages (I know this is not the usual way to use websocket but this part is not my choice). To this end, I'm using python 3.7 and websockets 7.0.
My issue is: Numerous messages pushed by the client are not received by the server.
Here is the simple code I'm using.
import asyncio
import websockets
async def get_tag_data(websocket, path):
# print('received')
data = await websocket.recv()
print(data)
loop = asyncio.get_event_loop()
anchors_server = websockets.serve(get_tag_data, 'localhost', 9001)
loop.run_until_complete(asyncio.gather(anchors_server))
loop.run_forever()
Conversely, when I try the python-websocket-server (which uses threads for reception), all my messages are correctly received.
As far as I understand asyncio & websockets, it is supposed to manage backpressure: messages sent while the server is busy (to process older messages) are "buffered" and will be treat shortly....
What am I missing? Do I need to thread the websocket reception with asyncio to avoid losing messages?
Thank you for your answers!

Ok, I get it.
My function was running only once, next messages was not buffered. The following code resolve the issue :
import asyncio
import websockets
import signal
import sys
async def get_tag_data(websocket, path):
while True:
async for data in websocket:
print(data)
loop = asyncio.get_event_loop()
anchors_server = websockets.serve(get_tag_data, '', 9001)
loop.run_until_complete(asyncio.gather(anchors_server))
loop.run_forever()
note the
while True:
async for data in websocket

Shoud be without while loop.
import asyncio
import websockets
import signal
import sys
async def get_tag_data(websocket, path):
async for data in websocket:
print(data)