I'm trying to open two web sockets - depth book and user socket.
Here's my code:
async def sockets(client):
bm = BinanceSocketManager(client)
ds = bm.depth_socket("BTCUSDT", depth=BinanceSocketManager.WEBSOCKET_DEPTH_5)
print("Started...")
async with ds as depth_socket:
while True:
res = await depth_socket.recv()
print(res)
await client.close_connection()
I need bm.user_socket() socket to be opened as well at the same time.
How can I have two of them opened at the same time with BinanceSocketManager?
I've amended your code because it wasn't working. My feeling is that you don't have a good understanding of how asyncio works. I would recommend you read this tutorial on asyncio event loops.
What you need is to run multiple coroutines using asyncio's ensure_future module which will enable you to schedule tasks and run them asynchronously.
What we are doing in the code below is we are creating separate tasks for depth_socket and user_socket which loop indefinately but also run asynchronously so they can return data whenever it is sent by the server without waiting for the other task to finish.
I think the problem you were having is that you were trying to put both depth_socket and user_socket in the same async coroutine so looping indefinately with depth_socket meant that you could never loop through the user_socket concurrently.
There's a section on running multiple coroutines in the tutorial link I have given you above which helped me a lot understand how this works.
Unfortunately I can't seem to connect to the binance testnet so I haven't been able to test if the user_socket task actually works when a user event occurs but it should as it's not throwing an error when connected to livenet. Let me know if you are getting trade events.
You will need to input your api key and secret of course.
import asyncio
from binance import AsyncClient, BinanceSocketManager
api_key = '...'
api_secret = '...'
async def task_depth_socket():
client = await AsyncClient.create(api_key, api_secret)
bm = BinanceSocketManager(client)
ds = bm.depth_socket("BTCUSDT", depth=BinanceSocketManager.WEBSOCKET_DEPTH_5)
async with ds as depth_socket:
while True:
res = await depth_socket.recv()
print(res)
async def task_user_socket():
client = await AsyncClient.create(api_key, api_secret)
bm = BinanceSocketManager(client)
us = bm.user_socket()
async with us as user_socket:
while True:
res = await user_socket.recv()
print(res)
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(task_depth_socket())
asyncio.ensure_future(task_user_socket())
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print("Closing Loop")
loop.close()
Related
SETUP: I have a large list (over 100+) of tasks (coroutines) that connect to a REST API database server. The coroutines use a client connection pool. I think that the client connection pool is cutting me off, because I am not able to get all my results. I also think that I could use a Semaphore to limit the concurrent connections to the API server, and get all my results before my script finishes. Here's a minimal example:
q = Queue(-1)
progress = tqdm(total=total_hits)
sem = asyncio.Semaphore(1)
for task in asyncio.as_completed(tasks):
async with sem:
res = await task
q.put_nowait(res["data"])
progress.update(len(res["data"]))
while res["links"].get("next", None) is not None:
res = await client.get_json_async(res["links"]["next"])
q.put_nowait(res["data"])
progress.update(len(res["data"]))
PROBLEM: I know that I have 10,000 data points to capture. However, I consistently only capture about half of those. I think it's because the client is limiting my TCP connections to the server.
Any ideas?
You can combine asyncio.Semaphore and asyncio.as_completed the following way:
import asyncio
import time
async def make_request(url: str, s: asyncio.Semaphore):
"""simulates request"""
async with s:
await asyncio.sleep(2)
return f"{url}: {time.monotonic()}"
async def amain():
"""main wrapper."""
s = asyncio.Semaphore(5)
tasks = [make_request(f"url-{i}" ,s) for i in range(26)]
for cor in asyncio.as_completed(tasks):
res = await cor
print(res)
if __name__ == '__main__':
asyncio.run(amain())
On the other hand if you need to limit number of requests due to API timelimit, you will probably need to use some TimeLimiter from any 3rd party library.
I am author one one of such libraries BucketRateLimiter and I will be grateful if you use it.
I have two bots, one is using pydle for IRC, like:
async def start_ircbot ():
try:
client = MyOwnBot(NICK,
realname=REALNAME,
sasl_username=SASL_USERNAME,
sasl_password=SASL_PASSWORD,
sasl_identity=SASL_IDENTITY,)
loop = asyncio.get_event_loop()
asyncio.ensure_future(client.connect(HOST, PORT, tls=True, tls_verify=False), loop=loop)
loop.run_forever()
loop.close()
except Exception as e:
print (e)
and another is using telethon for Telegram:
#client.on(events.NewMessage)
async def my_event_handler(event):
...
async def start_client ():
print ("Telegram monitor started...")
await client.start()
await client.run_until_disconnected()
Both of them work without problem separately.
Now, I want to integrate both of them, I tried to launch both of them in my main function like this,
import Notifier
...
async def main():
await asyncio.gather (Notifier.start_client (), start_ircbot ())
asyncio.run(main())
It starts without issue but my_event_handler seems never to get new messages. If I swap the order of functions:
await asyncio.gather (start_ircbot (), Notifier.start_client ())
The script will be stuck at launching, I suspect it has to be something within events loops and tried some different methods but without luck, could anyone shed light on this for me?
Newer Python versions are removing the loop parameter from most methods, so you should try to avoid using it. As long as you don't use asyncio.run (which creates a new loop) or you don't create a new loop yourself, both libraries should be using the default loop from the main thread (I can't speak for pydle, but Telethon does this).
As long as the asyncio event loop is running, Telethon should have no trouble receiving updates. You can use client.loop to make sure it's using the same loop:
tlclient = TelegramClient(...)
irclient = MyOwnBot(...)
#tlclient.on(events.NewMessage)
async def my_event_handler(event):
...
async def main():
await tlclient.start()
await irclient.connect(HOST, PORT, tls=True, tls_verify=False), loop=tlclient.tlclient)
await tlclient.run_until_disconnected() # similar to loop.run_forever() but stops when the client disconnects
client.loop.run_until_complete(main())
I'm trying to create a WebSocket command line client that waits for messages from a WebSocket server but waits for user input at the same time.
Regularly polling multiple online sources every second works fine on the server, (the one running at localhost:6789 in this example), but instead of using Python's normal sleep() method, it uses asyncio.sleep(), which makes sense because sleeping and asynchronously sleeping aren't the same thing, at least not under the hood.
Similarly, waiting for user input and asynchronously waiting for user input aren't the same thing, but I can't figure out how to asynchronously wait for user input in the same way that I can asynchronously wait for an arbitrary amount of seconds, so that the client can deal with incoming messages from the WebSocket server while simultaneously waiting for user input.
The comment below in the else-clause of monitor_cmd() hopefully explains what I'm getting at:
import asyncio
import json
import websockets
async def monitor_ws():
uri = 'ws://localhost:6789'
async with websockets.connect(uri) as websocket:
async for message in websocket:
print(json.dumps(json.loads(message), indent=2, sort_keys=True))
async def monitor_cmd():
while True:
sleep_instead = False
if sleep_instead:
await asyncio.sleep(1)
print('Sleeping works fine.')
else:
# Seems like I need the equivalent of:
# line = await asyncio.input('Is this your line? ')
line = input('Is this your line? ')
print(line)
try:
asyncio.get_event_loop().run_until_complete(asyncio.wait([
monitor_ws(),
monitor_cmd()
]))
except KeyboardInterrupt:
quit()
This code just waits for input indefinitely and does nothing else in the meantime, and I understand why. What I don't understand, is how to fix it. :)
Of course, if I'm thinking about this problem in the wrong way, I'd be very happy to learn how to remedy that as well.
You can use the aioconsole third-party package to interact with stdin in an asyncio-friendly manner:
line = await aioconsole.ainput('Is this your line? ')
Borrowing heavily from aioconsole, if you would rather avoid using an external library you could define your own async input function:
async def ainput(string: str) -> str:
await asyncio.get_event_loop().run_in_executor(
None, lambda s=string: sys.stdout.write(s+' '))
return await asyncio.get_event_loop().run_in_executor(
None, sys.stdin.readline)
Borrowing heavily from aioconsole, there are 2 ways to handle.
start a new daemon thread:
import sys
import asyncio
import threading
from concurrent.futures import Future
async def run_as_daemon(func, *args):
future = Future()
future.set_running_or_notify_cancel()
def daemon():
try:
result = func(*args)
except Exception as e:
future.set_exception(e)
else:
future.set_result(result)
threading.Thread(target=daemon, daemon=True).start()
return await asyncio.wrap_future(future)
async def main():
data = await run_as_daemon(sys.stdin.readline)
print(data)
if __name__ == "__main__":
asyncio.run(main())
use stream reader:
import sys
import asyncio
async def get_steam_reader(pipe) -> asyncio.StreamReader:
loop = asyncio.get_event_loop()
reader = asyncio.StreamReader(loop=loop)
protocol = asyncio.StreamReaderProtocol(reader)
await loop.connect_read_pipe(lambda: protocol, pipe)
return reader
async def main():
reader = await get_steam_reader(sys.stdin)
data = await reader.readline()
print(data)
if __name__ == "__main__":
asyncio.run(main())
Using Python 3.7.4 and the asyncio package I'm trying to write an application that should spawn around 20000 (20k or more) TCP clients which then connect to a single server.
The clients then wait for a command from the server (received_data = await reader.read(4096)) and proceed to executing it (await loop.run_in_executor(...)) then send the response back to the server (writer.write(resp)).
After this cycle is completed, I sleep 100ms (await asyncio.sleep(100e-3)) in order to allow other coroutines to run.
The 20k clients should never disconnect and should process commands from the server indefinitely.
I'm interested in ways I can change the code to optimize it (barring the use of uvloop or directly implementing a Protocol since I saw in uvloop's docs this could improve the performance) beyond what it is capable now.
Let's assume that I cannot modify handle_request.
For example the await asyncio.sleep(100e-3) is especially bothering me, but I had to add it there, otherwise the impression was that no other coroutines ran other than the first one! Why could that be?
Say I remove the sleep (since in theory the other awaits should allow other coroutines to run), what else could I do?
Below is a minimal example of what my application looks like:
import asyncio
from collections import namedtuple
import logging
import os
import sys
logger = logging.getLogger(__name__)
should_exit = asyncio.Event()
def exit(signame, loop):
should_exit.set()
logger.warning('Exiting soon...')
def handle_request(received_data, entity):
logger.info('Backend logic here that consumes a bit of time depending on the entity and the received_data')
async def run_entity(entity, args):
logger.info(f'Running entity {entity}')
loop = asyncio.get_running_loop()
try:
reader, writer = await asyncio.open_connection(args.addr[0], int(args.addr[1]))
logger.debug(f'{entity} connected to {args.addr[0]}:{args.addr[1]}')
try:
while not should_exit.is_set():
received_data = await reader.read(4096)
if received_data:
logger.debug(f'{entity} received data {received_data}')
success, resp = await loop.run_in_executor(None, functools.partial(handle_request, received_data, entity))
if success:
logger.debug(f'{entity} sending response {resp}')
writer.write(resp)
await writer.drain()
await asyncio.sleep(100e-3)
except ConnectionResetError:
pass
except ConnectionRefusedError:
logger.warning(f'Connection refused by {args.addr[0]}:{args.addr[1]}.')
except Exception:
logger.exception('Details of unexpected error:')
logger.info(f'Stopped entity {entity}')
async def main(entities, args):
if os.name == 'posix':
loop = asyncio.get_running_loop()
loop.add_signal_handler(signal.SIGTERM, functools.partial(exit, signal.SIGTERM, loop))
loop.add_signal_handler(signal.SIGINT, functools.partial(exit, signal.SIGINT, loop))
tasks = (run_entity(entity, args) for entity in entities)
await asyncio.gather(*tasks)
if __name__ == '__main__':
ArgsReplacement = namedtuple('ArgsReplacement', ['addr'])
asyncio.run(main(range(20000), ArgsReplacement(addr=['127.0.0.1', '4242'])))
I am trying to create a script in python that listens to multiple sockets using websockets and asyncio, the problem is that no matter what I do it only listen to the first socket I call.
I think its the infinite loop, what are my option to solve this? using threads for each sockets?
async def start_socket(self, event):
payload = json.dumps(event)
loop = asyncio.get_event_loop()
self.tasks.append(loop.create_task(
self.subscribe(event)))
# this should not block the rest of the code
await asyncio.gather(*tasks)
def test(self):
# I want to be able to add corotines at a different time
self.start_socket(event1)
# some code
self.start_socket(event2)
this is what I did eventually, that way its not blocking the main thread and all subscriptions are working in parallel.
def subscribe(self, payload):
ws = websocket.WebSocket(sslopt={"cert_reqs": ssl.CERT_NONE})
ws.connect(url)
ws.send(payload)
while True:
result = ws.recv()
print("Received '%s'" % result)
def start_thread(self, loop):
asyncio.set_event_loop(loop)
loop.run_forever()
def start_socket(self, **kwargs):
worker_loop = asyncio.new_event_loop()
worker = Thread(target=self.start_thread, args=(worker_loop,))
worker.start()
worker_loop.call_soon_threadsafe(self.subscribe, payload)
def listen(self):
self.start_socket(payload1)
# code
self.start_socket(payload2)
# code
self.start_socket(payload3)
Your code appears incomplete, but what you've shown has two issues. One is that run_until_complete accepts a coroutine object (or other kind of future), not a coroutine function. So it should be:
# note parentheses after your_async_function()
asyncio.get_event_loop().run_until_complete(your_async_function())
the problem is that no matter what I do it only listen to the first socket I call. I think its the infinite loop, what are my option to solve this? using threads for each sockets?
The infinite loop is not the problem, asyncio is designed to support such "infinite loops". The problem is that you are trying to do everything in one coroutine, whereas you should be creating one coroutine per websocket. This is not a problem, as coroutines are very lightweight.
For example (untested):
async def subscribe_all(self, payload):
loop = asyncio.get_event_loop()
# create a task for each URL
for url in url_list:
tasks.append(loop.create_task(self.subscribe_one(url, payload)))
# run all tasks in parallel
await asyncio.gather(*tasks)
async def subsribe_one(self, url, payload):
async with websockets.connect(url) as websocket:
await websocket.send(payload)
while True:
msg = await websocket.recv()
print(msg)
One way to efficiently listen to multiple websocket connections from a websocket server is to keep a list of connected clients and essentially juggle multiple conversations in parallel.
E.g. A simple server that sends random # to each connected client every few secs:
import os
import asyncio
import websockets
import random
websocket_clients = set()
async def handle_socket_connection(websocket, path):
"""Handles the whole lifecycle of each client's websocket connection."""
websocket_clients.add(websocket)
print(f'New connection from: {websocket.remote_address} ({len(websocket_clients)} total)')
try:
# This loop will keep listening on the socket until its closed.
async for raw_message in websocket:
print(f'Got: [{raw_message}] from socket [{id(websocket)}]')
except websockets.exceptions.ConnectionClosedError as cce:
pass
finally:
print(f'Disconnected from socket [{id(websocket)}]...')
websocket_clients.remove(websocket)
async def broadcast_random_number(loop):
"""Keeps sending a random # to each connected websocket client"""
while True:
for c in websocket_clients:
num = str(random.randint(10, 99))
print(f'Sending [{num}] to socket [{id(c)}]')
await c.send(num)
await asyncio.sleep(2)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
try:
socket_server = websockets.serve(handle_socket_connection, 'localhost', 6789)
print(f'Started socket server: {socket_server} ...')
loop.run_until_complete(socket_server)
loop.run_until_complete(broadcast_random_number(loop))
loop.run_forever()
finally:
loop.close()
print(f"Successfully shutdown [{loop}].")
A simple client that connects to the server and listens for the numbers:
import asyncio
import random
import websockets
async def handle_message():
uri = "ws://localhost:6789"
async with websockets.connect(uri) as websocket:
msg = 'Please send me a number...'
print(f'Sending [{msg}] to [{websocket}]')
await websocket.send(msg)
while True:
got_back = await websocket.recv()
print(f"Got: {got_back}")
asyncio.get_event_loop().run_until_complete(handle_message())
Mixing up threads and asyncio is more trouble than its worth and you still have code that will block on the most wasteful steps like network IO (which is the essential benefit of using asyncio).
You need to run each coroutine asynchronously in an event loop, call any blocking calls with await and define each method that interacts with any awaitable interactions with an async
See a working e.g.: https://github.com/adnantium/websocket_client_server