We Keep Coding

Python-Websockets: sends messages but cannot receive messages asynchronously

I'm trying to create a full duplex client that sends and receives asynchronously at the same time, using python's websockets package.
The server simply receives a message and echoes it back.
when the client sends all the messages, but doesn't receive anything at all, as if either the send is blocking the receive handler, or the handler is stuck and never updates the data.
However, the server ensures that it both received and sent the data, so I doubt that it's the problem.
I'm genuinely new to async, multithreading, and network programming in general, but this code will be reflected on an applicated that buffers audios from an incoming systems, and sends it to another service, also it can receive any messages from that service at any time regarding this session.
python 3.9.15
websockets==10.4
I've followed the tutorial on the official websockets documentation:
https://websockets.readthedocs.io/en/stable/howto/patterns.html#consumer-and-producer
Client Code:
`
import asyncio
import websockets
sent = []
received = []
URL = "ws://localhost:8001"
async def update_sent(message):
with open("sent.txt", "a+") as f:
print(message, file=f)
sent.append(message)
return 0
async def update_received(message):
with open("recv.txt", "a+") as f:
print(message, file=f)
received.append(message)
return 0
async def sending_handler(websocket):
while True:
try:
message = input("send message:>")
await websocket.send(message)
await update_sent(message)
except Exception as e:
print("Sender: connection closed due to Exception", e)
break
async def receive_handler(websocket):
while True:
try:
message = await websocket.recv()
await update_received(message)
except Exception as e:
print("Receiver: connection closed due to Exception", e)
break
async def full_duplex_handler(websocket):
receiving_task = asyncio.create_task(receive_handler(websocket))
sending_task = asyncio.create_task(sending_handler(websocket))
done, pending = await asyncio.wait([receiving_task, sending_task],
return_when=asyncio.FIRST_COMPLETED)
# return_when=asyncio.FIRST_EXCEPTION)
for task in pending:
print(task)
task.cancel()
async def gather_handler(websocket):
await asyncio.gather(
sending_handler(websocket),
receive_handler(websocket),
)
# using asyncio.wait
async def main_1(url=URL):
async with websockets.connect(url) as websocket:
await full_duplex_handler(websocket)
# using asyncio.gather
# async def main_2(url=URL):
# async with websockets.connect(url) as websocket:
# await gather_handler(websocket)
if __name__ == "__main__":
asyncio.run(main_1())
# asyncio.run(main_2())
`
Server code:
`
import asyncio
import websockets
msgs = []
sent = []
async def handle_send(websocket, message):
await websocket.send(message)
msgs.append(message)
async def handle_recv(websocket):
message = await websocket.recv()
sent.append(message)
return f"echo {message}"
async def handler(websocket):
while True:
try:
message = await handle_recv(websocket)
await handle_send(websocket, message)
except Exception as e:
print(e)
print(msgs)
print(sent)
break
async def main():
async with websockets.serve(handler, "localhost", 8001):
await asyncio.Future()
if __name__ == "__main__":
print("starting the server now")
asyncio.run(main())
`
After sending some messages, all sent and received messages should be written to a file,
but only sent messages are received and processed.

TL;DR
I've put a sleep statement:
await asyncio.sleep(0.02)
in the sending_handler while loop, and it resolved the problem,
apparently the issue was that the sender is way faster than the receiver, that it keeps locking the resources for its use, while the receiver is being blocked.
Any shorter sleep durations can't solve this problem.
final while loop:
async def sending_handler(websocket):
while True:
await asyncio.sleep(0.02)
try:
message = input("send message:>")
await websocket.send(message)
await update_sent(message)
except Exception as e:
print("Sender: connection closed due to Exception", e)
break
Hope this answer helps anyone else who faces the same problem

Asyncio Server with Timer in python

I'm using asyncio to run a straightforward server and a client. The server is a simple echo server with two "special" commands sent by the client, "quit" and "timer". The quit command closes the connection, and the timer command starts a timer that will print a message in the console (server and client) every second. The timer will stop when the client sends the "quit" command.
Unfortunately, I'm having some problems with the timer. It blocks the server and the client.
How can I solve this problem?
Server
import asyncio
import time
HOST = '127.0.0.1'
PORT = 9999
async def timer():
while True:
print('tick')
await asyncio.sleep(1)
async def handle_echo(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
'''Handle the echo protocol.'''
data = None
while True:
if(data == b'quit'):
writer.close() # Close the connection
await writer.wait_closed() # Wait for the connection to close
if(data == b'timer'):
timertask = asyncio.create_task(timer())
await timertask #<-- This line freezes the server and the client
else:
data = await reader.read(1024) # Read 256 bytes from the reader. Size of the message
msg = data.decode() # Decode the message
addr, port = writer.get_extra_info('peername') # Get the address of the client
print(f"Received {msg!r} from {addr}:{port!r}")
send_message = 'Message received: ' + msg
writer.write(send_message.encode()) # Echo the data back to the client
await writer.drain() # This will wait until everything is clear to move to the next thing.
async def run_server() -> None:
# Our awaitable callable.
# This callable is ran when the server recieves some data
# Question: Does it run when a client connects?
server = await asyncio.start_server(handle_echo, HOST, PORT)
async with server:
await server.serve_forever()
if __name__ == '__main__':
loop = asyncio.new_event_loop() # new_event_loop() is for python 3.10. For older versions, use get_event_loop()
loop.run_until_complete(run_server())
Client
import asyncio
import time
HOST = '127.0.0.1'
PORT = 9999
async def run_client() -> None:
# It's a coroutine. It will wait until the connection is established
reader, writer = await asyncio.open_connection(HOST, PORT)
while True:
message = input('Enter a message: ')
writer.write(message.encode())
await writer.drain()
data = await reader.read(1024)
if not data:
raise Exception('Socket not communicating with the client')
print(f"Received {data.decode()!r}")
if(message == 'quit'):
writer.write(b"quit")
writer.close()
await writer.wait_closed()
exit(2)
#break # Don't know if this is necessary
if __name__ == '__main__':
loop = asyncio.new_event_loop()
loop.run_until_complete(run_client())

I slightly updated your server code: removed the await timertask line (this task never ends, can be only canceled):
Server.py
import asyncio
import time
HOST = "127.0.0.1"
PORT = 9999
async def timer():
while True:
print("tick")
await asyncio.sleep(1)
async def handle_echo(
reader: asyncio.StreamReader, writer: asyncio.StreamWriter
) -> None:
# timer task that the client can start:
timer_task = None
while True:
data = await reader.read(1024)
msg = data.decode()
addr, port = writer.get_extra_info("peername")
print(f"Received {msg!r} from {addr}:{port!r}")
send_message = "Message received: " + msg
writer.write(send_message.encode())
await writer.drain()
if data == b"quit":
# cancel the timer_task (if any)
if timer_task:
timer_task.cancel()
await timer_task
writer.close()
await writer.wait_closed()
elif data == b"timer" and timer_task is None:
timer_task = asyncio.create_task(timer())
async def run_server() -> None:
server = await asyncio.start_server(handle_echo, HOST, PORT)
async with server:
await server.serve_forever()
if __name__ == "__main__":
loop = asyncio.new_event_loop()
loop.run_until_complete(run_server())
Client.py
import asyncio
import time
HOST = "127.0.0.1"
PORT = 9999
async def run_client() -> None:
# It's a coroutine. It will wait until the connection is established
reader, writer = await asyncio.open_connection(HOST, PORT)
while True:
message = input("Enter a message: ")
writer.write(message.encode())
await writer.drain()
data = await reader.read(1024)
if not data:
raise Exception("Socket not communicating with the client")
print(f"Received {data.decode()!r}")
if message == "quit":
writer.write(b"quit")
await writer.drain()
writer.close()
await writer.wait_closed()
exit(2)
if __name__ == "__main__":
loop = asyncio.new_event_loop()
loop.run_until_complete(run_client())
The server waits for a client's messages. If message timer arrives, starts a new timer task and continues to read other messages.
When quit arrives, server cancels the timer_task (if exists) and exits.

Python AsyncIO stream reader not reading properly

I have created server which reads message and the client will write the message(which contain header
message concat with body) .However my server not reading properly, At first time reading entire message and the second time it just read the header not reading the Body.
In server.py
import asyncio
from _socket import AF_INET
async def handle_echo(reader, writer):
data = await reader.read(1024)
print(data)
async def main():
server = await asyncio.start_server(
handle_echo, '127.0.0.1', 449, family=AF_INET)
addr = server.sockets[0].getsockname()
print(f'Serving on {addr}')
async with server:
await server.serve_forever()
asyncio.run(main())
In client snippet:
message = loadImage()
writer.write(len(json.dumps(message)).to_bytes(4, 'little') + json.dumps(message).encode("utf-8"))
await writer.drain()
And also checked in Wire shark same message for both client request.
both header and body length are same.
Could please help to resolve?

Streaming list of dictionaries in Python with asyncio

I am working on Python server/client application where the server receives some data from the client and based on this data it collects a list of dictionaries from an embedded k/v store and streams it back.
I put here a code that reproduces the error. There is a reason why I put everything into separate functions on server side (clients send different requests).
The problem is that the server sends faster than the client can consume and the client reads several responses at a time, sometimes it is just a part of the message which has been truncated. I thought writelines/readline pair will read from the socket appropriately, but I think I missed something. write/drain also overloads the socket and once multiple results are read the client failes because chunked serialized dictionary is read to orjson.loads.
What is the proper way to solve this problem? Thank you in advance!
Server:
import orjson
async def getResult(cnt : int):
await asyncio.sleep(0)
result = []
for i in range(cnt):
result.append({"key" : i})
return result
async def send(writer, list_of_dict):
for r in list_of_dict:
print(f"\nSending: {r}")
writer.writelines([orjson.dumps(r)])
await writer.drain()
# sending END signal
writer.writelines([orjson.dumps("END")])
await writer.drain()
async def handleClient(reader, writer):
addr = writer.get_extra_info('peername')
print(f"Connection from {addr}")
data = await reader.readline()
message = orjson.loads(data)
print(f"Received {message} from {addr}")
counter = message["send_me"]
responses = await getResult(counter)
await send(writer, responses)
print("Close the client socket")
writer.close()
loop = asyncio.get_event_loop()
coro = asyncio.start_server(handleClient, '127.0.0.1', 4000, loop=loop)
server = loop.run_until_complete(coro)
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
loop.run_forever()
except KeyboardInterrupt:
pass
# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()
Client
import asyncio
import orjson
async def async_client(loop):
reader, writer = await asyncio.open_connection('127.0.0.1', 4000, loop=loop)
counter = 5
print(f"Request counter: {counter}")
# in real life the message is a complex dictionary
msg = {"send_me" : counter}
writer.writelines([orjson.dumps(msg)])
#without write_eof the server reader.readline() waits for data and blocks
if writer.can_write_eof():
writer.write_eof()
while True:
data = await reader.readline()
if data:
print(data)
r = orjson.loads(data)
print(f"Received: {r}")
if r == "END":
print("server completed")
break
else:
await asyncio.sleep(0.1)
print('Close the socket')
writer.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(async_client(loop))
loop.close()
Error:
>python echo_client.py
Request counter: 5
b'{"key":0}{"key":1}{"key":2}{"key":3}{"key":4}"END"'
Traceback (most recent call last):
File "echo_client.py", line 32, in <module>
loop.run_until_complete(async_client(loop))
File "C:\Program Files (x86)\Anaconda\lib\asyncio\base_events.py", line 587, in run_until_complete
return future.result()
File "echo_client.py", line 21, in async_client
r = orjson.loads(data)
orjson.JSONDecodeError: trailing characters at line 1 column 10: line 1 column 1 (char 0)

I think the problem is much simpler: writelines doesn't do what you think it does. It doesn't insert newline characters, it just writes any data you give it. This is why the readline() by your client picks up the payload and "END" concatenated together. This is also why you need write_eof in the other direction.
If you want to write a line, then just write a newline character (byte) after your payload. You can abstract that in a function that handles it for you:
async def write_msg(writer, msg):
writer.write(orjson.dumps(msg))
writer.write('\n')
await writer.drain()
async def read_msg(reader):
line = await reader.readline()
return orjson.loads(line)
You can use these on both the client and the server to communicate.
On an aside note, you should probably switch to the newer asyncio.run() API which creates and correctly tears down the event loop with a single async entry point. Your server setup would look like this:
async def main():
await asyncio.start_server(handleClient, '127.0.0.1', 4000)
await server.wait_closed()
asyncio.run(main())

how to handle tcp client socket auto reconnect in python asyncio?

I am using python asyncio streams to connect to several socket servers, but when the server is down, my code can't auto reconnect.
What I need is that, when the server is down, my script will try to reconnect every 5 seconds, until connected and start to parse the data again.
import asyncio
server1 = {'host': '192.168.1.51', 'port': 11110}
server2 = {'host': '192.168.1.52', 'port': 11110}
async def tcp_client(host, port, loop):
print('connect to server {} {}'.format(host, str(port)))
reader, writer = await asyncio.open_connection(host, port, loop=loop)
while True:
data = await reader.read(100)
print('raw data received: {}'.format(data))
await asyncio.sleep(0.1)
loop = asyncio.get_event_loop()
try:
for server in [server1, server2]:
loop.run_until_complete(tcp_client(server['host'], server['port'], loop))
print('task added: connect to server {} {}'.format(server['host'], server['port']))
finally:
loop.close()
print('loop closed')

You can handle reconnection by simply looping over a try/except statement.
Additionally, asyncio.wait_for can be used to set a timeout on the read operation.
Consider this working example:
import asyncio
async def tcp_client(host, port):
reader, writer = await asyncio.open_connection(host, port)
try:
while not reader.at_eof():
data = await asyncio.wait_for(reader.read(100), 3.0)
print('raw data received: {}'.format(data))
finally:
writer.close()
async def tcp_reconnect(host, port):
server = '{} {}'.format(host, port)
while True:
print('Connecting to server {} ...'.format(server))
try:
await tcp_client(host, port)
except ConnectionRefusedError:
print('Connection to server {} failed!'.format(server))
except asyncio.TimeoutError:
print('Connection to server {} timed out!'.format(server))
else:
print('Connection to server {} is closed.'.format(server))
await asyncio.sleep(2.0)
async def main():
servers = [('localhost', 8888), ('localhost', 9999)]
coros = [tcp_reconnect(host, port) for host, port in servers]
await asyncio.gather(*coros)
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()