We Keep Coding

Streaming list of dictionaries in Python with asyncio

I am working on Python server/client application where the server receives some data from the client and based on this data it collects a list of dictionaries from an embedded k/v store and streams it back.
I put here a code that reproduces the error. There is a reason why I put everything into separate functions on server side (clients send different requests).
The problem is that the server sends faster than the client can consume and the client reads several responses at a time, sometimes it is just a part of the message which has been truncated. I thought writelines/readline pair will read from the socket appropriately, but I think I missed something. write/drain also overloads the socket and once multiple results are read the client failes because chunked serialized dictionary is read to orjson.loads.
What is the proper way to solve this problem? Thank you in advance!
Server:
import orjson
async def getResult(cnt : int):
await asyncio.sleep(0)
result = []
for i in range(cnt):
result.append({"key" : i})
return result
async def send(writer, list_of_dict):
for r in list_of_dict:
print(f"\nSending: {r}")
writer.writelines([orjson.dumps(r)])
await writer.drain()
# sending END signal
writer.writelines([orjson.dumps("END")])
await writer.drain()
async def handleClient(reader, writer):
addr = writer.get_extra_info('peername')
print(f"Connection from {addr}")
data = await reader.readline()
message = orjson.loads(data)
print(f"Received {message} from {addr}")
counter = message["send_me"]
responses = await getResult(counter)
await send(writer, responses)
print("Close the client socket")
writer.close()
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handleClient, '127.0.0.1', 4000, loop=loop)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
Client
import asyncio
import orjson
async def async_client(loop):
reader, writer = await asyncio.open_connection('127.0.0.1', 4000, loop=loop)
counter = 5
print(f"Request counter: {counter}")
# in real life the message is a complex dictionary
msg = {"send_me" : counter}
writer.writelines([orjson.dumps(msg)])
#without write_eof the server reader.readline() waits for data and blocks
if writer.can_write_eof():
writer.write_eof()
while True:
data = await reader.readline()
if data:
print(data)
r = orjson.loads(data)
print(f"Received: {r}")
if r == "END":
print("server completed")
break
else:
await asyncio.sleep(0.1)
print('Close the socket')
writer.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(async_client(loop))
loop.close()
Error:
>python echo_client.py
Request counter: 5
b'{"key":0}{"key":1}{"key":2}{"key":3}{"key":4}"END"'
Traceback (most recent call last):
File "echo_client.py", line 32, in <module>
loop.run_until_complete(async_client(loop))
File "C:\Program Files (x86)\Anaconda\lib\asyncio\base_events.py", line 587, in run_until_complete
return future.result()
File "echo_client.py", line 21, in async_client
r = orjson.loads(data)
orjson.JSONDecodeError: trailing characters at line 1 column 10: line 1 column 1 (char 0)

I think the problem is much simpler: writelines doesn't do what you think it does. It doesn't insert newline characters, it just writes any data you give it. This is why the readline() by your client picks up the payload and "END" concatenated together. This is also why you need write_eof in the other direction.
If you want to write a line, then just write a newline character (byte) after your payload. You can abstract that in a function that handles it for you:
async def write_msg(writer, msg):
writer.write(orjson.dumps(msg))
writer.write('\n')
await writer.drain()
async def read_msg(reader):
line = await reader.readline()
return orjson.loads(line)
You can use these on both the client and the server to communicate.
On an aside note, you should probably switch to the newer asyncio.run() API which creates and correctly tears down the event loop with a single async entry point. Your server setup would look like this:
async def main():
await asyncio.start_server(handleClient, '127.0.0.1', 4000)
await server.wait_closed()
asyncio.run(main())

Get async input from console in Python [duplicate]

I'm trying to create a WebSocket command line client that waits for messages from a WebSocket server but waits for user input at the same time.
Regularly polling multiple online sources every second works fine on the server, (the one running at localhost:6789 in this example), but instead of using Python's normal sleep() method, it uses asyncio.sleep(), which makes sense because sleeping and asynchronously sleeping aren't the same thing, at least not under the hood.
Similarly, waiting for user input and asynchronously waiting for user input aren't the same thing, but I can't figure out how to asynchronously wait for user input in the same way that I can asynchronously wait for an arbitrary amount of seconds, so that the client can deal with incoming messages from the WebSocket server while simultaneously waiting for user input.
The comment below in the else-clause of monitor_cmd() hopefully explains what I'm getting at:
import asyncio
import json
import websockets
async def monitor_ws():
uri = 'ws://localhost:6789'
async with websockets.connect(uri) as websocket:
async for message in websocket:
print(json.dumps(json.loads(message), indent=2, sort_keys=True))
async def monitor_cmd():
while True:
sleep_instead = False
if sleep_instead:
await asyncio.sleep(1)
print('Sleeping works fine.')
else:
# Seems like I need the equivalent of:
# line = await asyncio.input('Is this your line? ')
line = input('Is this your line? ')
print(line)
try:
asyncio.get_event_loop().run_until_complete(asyncio.wait([
monitor_ws(),
monitor_cmd()
]))
except KeyboardInterrupt:
quit()
This code just waits for input indefinitely and does nothing else in the meantime, and I understand why. What I don't understand, is how to fix it. :)
Of course, if I'm thinking about this problem in the wrong way, I'd be very happy to learn how to remedy that as well.

You can use the aioconsole third-party package to interact with stdin in an asyncio-friendly manner:
line = await aioconsole.ainput('Is this your line? ')

Borrowing heavily from aioconsole, if you would rather avoid using an external library you could define your own async input function:
async def ainput(string: str) -> str:
await asyncio.get_event_loop().run_in_executor(
None, lambda s=string: sys.stdout.write(s+' '))
return await asyncio.get_event_loop().run_in_executor(
None, sys.stdin.readline)

Borrowing heavily from aioconsole, there are 2 ways to handle.
start a new daemon thread:
import sys
import asyncio
import threading
from concurrent.futures import Future
async def run_as_daemon(func, *args):
future = Future()
future.set_running_or_notify_cancel()
def daemon():
try:
result = func(*args)
except Exception as e:
future.set_exception(e)
else:
future.set_result(result)
threading.Thread(target=daemon, daemon=True).start()
return await asyncio.wrap_future(future)
async def main():
data = await run_as_daemon(sys.stdin.readline)
print(data)
if __name__ == "__main__":
asyncio.run(main())
use stream reader:
import sys
import asyncio
async def get_steam_reader(pipe) -> asyncio.StreamReader:
loop = asyncio.get_event_loop()
reader = asyncio.StreamReader(loop=loop)
protocol = asyncio.StreamReaderProtocol(reader)
await loop.connect_read_pipe(lambda: protocol, pipe)
return reader
async def main():
reader = await get_steam_reader(sys.stdin)
data = await reader.readline()
print(data)
if __name__ == "__main__":
asyncio.run(main())

listen to multiple socket with websockets and asyncio

I am trying to create a script in python that listens to multiple sockets using websockets and asyncio, the problem is that no matter what I do it only listen to the first socket I call.
I think its the infinite loop, what are my option to solve this? using threads for each sockets?
async def start_socket(self, event):
payload = json.dumps(event)
loop = asyncio.get_event_loop()
self.tasks.append(loop.create_task(
self.subscribe(event)))
# this should not block the rest of the code
await asyncio.gather(*tasks)
def test(self):
# I want to be able to add corotines at a different time
self.start_socket(event1)
# some code
self.start_socket(event2)

this is what I did eventually, that way its not blocking the main thread and all subscriptions are working in parallel.
def subscribe(self, payload):
ws = websocket.WebSocket(sslopt={"cert_reqs": ssl.CERT_NONE})
ws.connect(url)
ws.send(payload)
while True:
result = ws.recv()
print("Received '%s'" % result)
def start_thread(self, loop):
asyncio.set_event_loop(loop)
loop.run_forever()
def start_socket(self, **kwargs):
worker_loop = asyncio.new_event_loop()
worker = Thread(target=self.start_thread, args=(worker_loop,))
worker.start()
worker_loop.call_soon_threadsafe(self.subscribe, payload)
def listen(self):
self.start_socket(payload1)
# code
self.start_socket(payload2)
# code
self.start_socket(payload3)

Your code appears incomplete, but what you've shown has two issues. One is that run_until_complete accepts a coroutine object (or other kind of future), not a coroutine function. So it should be:
# note parentheses after your_async_function()
asyncio.get_event_loop().run_until_complete(your_async_function())
the problem is that no matter what I do it only listen to the first socket I call. I think its the infinite loop, what are my option to solve this? using threads for each sockets?
The infinite loop is not the problem, asyncio is designed to support such "infinite loops". The problem is that you are trying to do everything in one coroutine, whereas you should be creating one coroutine per websocket. This is not a problem, as coroutines are very lightweight.
For example (untested):
async def subscribe_all(self, payload):
loop = asyncio.get_event_loop()
# create a task for each URL
for url in url_list:
tasks.append(loop.create_task(self.subscribe_one(url, payload)))
# run all tasks in parallel
await asyncio.gather(*tasks)
async def subsribe_one(self, url, payload):
async with websockets.connect(url) as websocket:
await websocket.send(payload)
while True:
msg = await websocket.recv()
print(msg)

One way to efficiently listen to multiple websocket connections from a websocket server is to keep a list of connected clients and essentially juggle multiple conversations in parallel.
E.g. A simple server that sends random # to each connected client every few secs:
import os
import asyncio
import websockets
import random
websocket_clients = set()
async def handle_socket_connection(websocket, path):
"""Handles the whole lifecycle of each client's websocket connection."""
websocket_clients.add(websocket)
print(f'New connection from: {websocket.remote_address} ({len(websocket_clients)} total)')
try:
# This loop will keep listening on the socket until its closed.
async for raw_message in websocket:
print(f'Got: [{raw_message}] from socket [{id(websocket)}]')
except websockets.exceptions.ConnectionClosedError as cce:
pass
finally:
print(f'Disconnected from socket [{id(websocket)}]...')
websocket_clients.remove(websocket)
async def broadcast_random_number(loop):
"""Keeps sending a random # to each connected websocket client"""
while True:
for c in websocket_clients:
num = str(random.randint(10, 99))
print(f'Sending [{num}] to socket [{id(c)}]')
await c.send(num)
await asyncio.sleep(2)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
try:
socket_server = websockets.serve(handle_socket_connection, 'localhost', 6789)
print(f'Started socket server: {socket_server} ...')
loop.run_until_complete(socket_server)
loop.run_until_complete(broadcast_random_number(loop))
loop.run_forever()
finally:
loop.close()
print(f"Successfully shutdown [{loop}].")
A simple client that connects to the server and listens for the numbers:
import asyncio
import random
import websockets
async def handle_message():
uri = "ws://localhost:6789"
async with websockets.connect(uri) as websocket:
msg = 'Please send me a number...'
print(f'Sending [{msg}] to [{websocket}]')
await websocket.send(msg)
while True:
got_back = await websocket.recv()
print(f"Got: {got_back}")
asyncio.get_event_loop().run_until_complete(handle_message())
Mixing up threads and asyncio is more trouble than its worth and you still have code that will block on the most wasteful steps like network IO (which is the essential benefit of using asyncio).
You need to run each coroutine asynchronously in an event loop, call any blocking calls with await and define each method that interacts with any awaitable interactions with an async
See a working e.g.: https://github.com/adnantium/websocket_client_server

I always get errors in closing handshake of Websockets in Python

I've coded a little server and client with Python3, Asyncio and Websockets.
I generate just for testing random numbers from a array, parse them into json and send them as websocket to the server. But often I get many errors because of closing handshakes.
Here is the code:
Server:
import asyncio
import websockets
import json
async def receiver(websocket, path):
ws = await websocket.recv()
print("< {}".format(ws))
start_server = websockets.serve(receiver, 'localhost', 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Client:
import asyncio
import websockets
import json
from array import *
import random
async def randomNumbers():
while True:
numbers = array('i', [1, 2, 3, 4, 5, 6, 7, 8, 9])
random.shuffle(numbers)
await sendWebsocket(numbers[0])
await asyncio.sleep(1)
async def sendWebsocket(number):
async with websockets.connect('ws://localhost:8765') as websocket:
ws = json.dumps({"number": number})
await websocket.send(ws)
print("> {}".format(ws))
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(randomNumbers())
loop.run_forever()
finally:
print("Client1 closed")
loop.close()
Error:
Error in closing handshake
Traceback (most recent call last):
File "C:\Users\dgred\AppData\Local\Programs\Python\Python36-32\lib\site-packages\websockets\server.py", line 145, in handler
yield from self.close()
File "C:\Users\dgred\AppData\Local\Programs\Python\Python36-32\lib\site-packages\websockets\protocol.py", line 370, in close
self.timeout, loop=self.loop)
File "C:\Users\dgred\AppData\Local\Programs\Python\Python36-32\lib\asyncio\tasks.py", line 358, in wait_for
return fut.result()
File "C:\Users\dgred\AppData\Local\Programs\Python\Python36-32\lib\site-packages\websockets\protocol.py", line 642, in write_frame
"in the {} state".format(self.state.name))
websockets.exceptions.InvalidState: Cannot write to a WebSocket in the CLOSING state
Can anyone help me here?
Thanks!

Here is a slight modification of your server side:
import asyncio
import websockets
import json
import signal
async def receiver(websocket, path):
while True:
try:
ws = await websocket.recv()
print("< {}".format(ws))
except websockets.ConnectionClosed:
print("Connection closed")
break
async def simple_server(stop):
async with websockets.serve(receiver, 'localhost', 8765):
await stop
loop = asyncio.get_event_loop()
stop = asyncio.Future()
loop.add_signal_handler(signal.SIGTERM, stop.set_result, None)
loop.run_until_complete(simple_server(stop))
There are more official examples available on websocket's GitHub

Web Sockets client throwing an exception once it starts receiving data

I am trying to establish a Web-Socket connection to a server and enter into receive mode.Once the client starts receiving the data, it immediately closes the connection with below exception
webSoc_Received = await websocket.recv()
File "/root/envname/lib/python3.6/site-packages/websockets/protocol.py", line 319, in recv
raise ConnectionClosed(self.close_code, self.close_reason)
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1007, no reason.
Client-side Code Snippet :
import asyncio
import websockets
async def connect_ws():
print("websockets.client module defines a simple WebSocket client API::::::")
async with websockets.client.connect(full_url,extra_headers=headers_conn1) as websocket:
print ("starting")
webSoc_Received = await websocket.recv()
print ("Ending")
Decode_data = zlib.decompress(webSoc_Received)
print(Decode_data)
asyncio.get_event_loop().run_until_complete(connect_ws())
Any thoughts on this?

You use run_until_complete() which completes once you started process. Instead, you should use .run_forever(). It will keep your socket open, until you close it.
EDIT:
You can do something like this:
loop = asyncio.get_event_loop()
loop.call_soon(connect_ws) # Calls connect_ws once the event loop starts
loop.run_forever()
Or:
loop = asyncio.get_event_loop()
loop.run_until_complete(connect_ws())
loop.run_forever()
Or if previous examples didn't succeed, you can try with following code:
import asyncio
#asyncio.coroutine
def periodic():
while True:
print("websockets.client module defines a simple WebSocket client API::::::")
with websockets.client.connect(full_url,extra_headers=headers_conn1) as websocket:
print ("starting")
webSoc_Received = websocket.recv()
print ("Ending")
Decode_data = zlib.decompress(webSoc_Received)
print(Decode_data)
def stop():
task.cancel()
task = asyncio.Task(periodic())
loop = asyncio.get_event_loop()
loop.call_later(5, stop)
try:
loop.run_until_complete(task)
except asyncio.CancelledError:
pass

From what I can tell, your current code will exit after it receives its first message.
Try changing your code to a consumer pattern, as mentioned in the websocket docs here:
https://websockets.readthedocs.io/en/stable/intro.html#common-patterns
import asyncio
import websockets
async def connect_ws():
print("websockets.client module defines a simple WebSocket client API::::::")
async with websockets.client.connect(full_url,extra_headers=headers_conn1) as websocket:
while True:
print ("starting")
webSoc_Received = await websocket.recv()
print ("Ending")
Decode_data = zlib.decompress(webSoc_Received)
print(Decode_data)