We Keep Coding

Python-Websockets: sends messages but cannot receive messages asynchronously

I'm trying to create a full duplex client that sends and receives asynchronously at the same time, using python's websockets package.
The server simply receives a message and echoes it back.
when the client sends all the messages, but doesn't receive anything at all, as if either the send is blocking the receive handler, or the handler is stuck and never updates the data.
However, the server ensures that it both received and sent the data, so I doubt that it's the problem.
I'm genuinely new to async, multithreading, and network programming in general, but this code will be reflected on an applicated that buffers audios from an incoming systems, and sends it to another service, also it can receive any messages from that service at any time regarding this session.
python 3.9.15
websockets==10.4
I've followed the tutorial on the official websockets documentation:
https://websockets.readthedocs.io/en/stable/howto/patterns.html#consumer-and-producer
Client Code:
`
import asyncio
import websockets
sent = []
received = []
URL = "ws://localhost:8001"
async def update_sent(message):
with open("sent.txt", "a+") as f:
print(message, file=f)
sent.append(message)
return 0
async def update_received(message):
with open("recv.txt", "a+") as f:
print(message, file=f)
received.append(message)
return 0
async def sending_handler(websocket):
while True:
try:
message = input("send message:>")
await websocket.send(message)
await update_sent(message)
except Exception as e:
print("Sender: connection closed due to Exception", e)
break
async def receive_handler(websocket):
while True:
try:
message = await websocket.recv()
await update_received(message)
except Exception as e:
print("Receiver: connection closed due to Exception", e)
break
async def full_duplex_handler(websocket):
receiving_task = asyncio.create_task(receive_handler(websocket))
sending_task = asyncio.create_task(sending_handler(websocket))
done, pending = await asyncio.wait([receiving_task, sending_task],
return_when=asyncio.FIRST_COMPLETED)
# return_when=asyncio.FIRST_EXCEPTION)
for task in pending:
print(task)
task.cancel()
async def gather_handler(websocket):
await asyncio.gather(
sending_handler(websocket),
receive_handler(websocket),
)
# using asyncio.wait
async def main_1(url=URL):
async with websockets.connect(url) as websocket:
await full_duplex_handler(websocket)
# using asyncio.gather
# async def main_2(url=URL):
# async with websockets.connect(url) as websocket:
# await gather_handler(websocket)
if __name__ == "__main__":
asyncio.run(main_1())
# asyncio.run(main_2())
`
Server code:
`
import asyncio
import websockets
msgs = []
sent = []
async def handle_send(websocket, message):
await websocket.send(message)
msgs.append(message)
async def handle_recv(websocket):
message = await websocket.recv()
sent.append(message)
return f"echo {message}"
async def handler(websocket):
while True:
try:
message = await handle_recv(websocket)
await handle_send(websocket, message)
except Exception as e:
print(e)
print(msgs)
print(sent)
break
async def main():
async with websockets.serve(handler, "localhost", 8001):
await asyncio.Future()
if __name__ == "__main__":
print("starting the server now")
asyncio.run(main())
`
After sending some messages, all sent and received messages should be written to a file,
but only sent messages are received and processed.

TL;DR
I've put a sleep statement:
await asyncio.sleep(0.02)
in the sending_handler while loop, and it resolved the problem,
apparently the issue was that the sender is way faster than the receiver, that it keeps locking the resources for its use, while the receiver is being blocked.
Any shorter sleep durations can't solve this problem.
final while loop:
async def sending_handler(websocket):
while True:
await asyncio.sleep(0.02)
try:
message = input("send message:>")
await websocket.send(message)
await update_sent(message)
except Exception as e:
print("Sender: connection closed due to Exception", e)
break
Hope this answer helps anyone else who faces the same problem

Create another thread for an await function

I'm working with Webserver for the first time, I had worked with socket and parallelism before but it was very different and simple, it didn't use Async as parallelism.
My goal is simple, I have my server and my client. In my client I want to create a separate thread to receive the messages that the server will send and in the previous thread do some other things, as in the code example (client.py):
from typing import Dict
import websockets
import asyncio
import json
URL = "my localhost webserver"
connection = None
async def listen() -> None:
global connection
input("Press enter to connect.")
async with websockets.connect(URL) as ws:
connection = ws
msg_initial: Dict[str,str] = get_dict()
await ws.send(json.dumps(msg_initial))
## This i want to be in another thread
await receive_msg()
print("I`m at listener`s thread")
# do some stuffs
async def recieve_msg() -> None:
while True:
msg = await connection.recv()
print(f"Server: {msg}")
asyncio.get_event_loop().run_until_complete(listen())
For me to get a message I need to use await in recv() but I don't know how to create a separate thread for that. I've already tried using threading to create a separate thread but it didn't work.
Does anyone know how to do this and if it is possible to do this?

It's not clear what you want to do can be done in the exact way you propose. In the following example I am connecting to an echo server. The most straightforward way of implementing what you are suggesting directly is to create a new thread to which the connection is passed. But this does not quite work:
import websockets
import asyncio
from threading import Thread
URL = "ws://localhost:4000"
async def listen() -> None:
async with websockets.connect(URL) as ws:
# pass connection:
t = Thread(target=receiver_thread, args=(ws,))
t.start()
# Generate some messages to be echoed back:
await ws.send('msg1')
await ws.send('msg2')
await ws.send('msg3')
await ws.send('msg4')
await ws.send('msg5')
def receiver_thread(connection):
print("I`m at listener`s thread")
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(receive_msg(connection))
async def receive_msg(connection) -> None:
while True:
msg = await connection.recv()
print(f"Server: {msg}")
asyncio.get_event_loop().run_until_complete(listen())
Prints:
I`m at listener`s thread
Server: msg1
Server: msg2
Server: msg3
Server: msg4
Server: msg5
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Program Files\Python38\lib\threading.py", line 932, in _bootstrap_inner
self.run()
File "C:\Program Files\Python38\lib\threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "C:\Ron\test\test.py", line 22, in receiver_thread
loop.run_until_complete(receive_msg(connection))
File "C:\Program Files\Python38\lib\asyncio\base_events.py", line 616, in run_until_complete
return future.result()
File "C:\Ron\test\test.py", line 29, in receive_msg
msg = await connection.recv()
File "C:\Program Files\Python38\lib\site-packages\websockets\legacy\protocol.py", line 404, in recv
await asyncio.wait(
File "C:\Program Files\Python38\lib\asyncio\tasks.py", line 424, in wait
fs = {ensure_future(f, loop=loop) for f in set(fs)}
File "C:\Program Files\Python38\lib\asyncio\tasks.py", line 424, in <setcomp>
fs = {ensure_future(f, loop=loop) for f in set(fs)}
File "C:\Program Files\Python38\lib\asyncio\tasks.py", line 667, in ensure_future
raise ValueError('The future belongs to a different loop than '
ValueError: The future belongs to a different loop than the one specified as the loop argument
The messages are received okay but the problem occurs in function receiver_thread on the statement:
loop.run_until_complete(receive_msg(connection))
By necessity the started thread has no running event loop and cannot use the event loop being used by function listen and so must create a new event loop. That would be fine if this thread/event loop were not using any resources (i.e. the connection) from a difference event loop:
import websockets
import asyncio
from threading import Thread
URL = "ws://localhost:4000"
async def listen() -> None:
async with websockets.connect(URL) as ws:
t = Thread(target=receiver_thread)
t.start()
def receiver_thread():
print("I`m at listener`s thread")
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(receive_msg())
async def receive_msg() -> None:
await asyncio.sleep(2)
print('I just slept for 2 seconds')
asyncio.get_event_loop().run_until_complete(listen())
Prints:
I`m at listener`s thread
I just slept for 2 seconds
I can see no real need to be running anything in threads based on the minimal code you have showed but assuming you omitted showing some processing of the received message for which asyncio alone is not sufficient, then perhaps all you need to do is receive the messages in the current running loop (in function listen) and use threading just for the processing of the message:
from typing import Dict
import websockets
import asyncio
import json
from threading import Thread
URL = "my localhost webserver"
async def listen() -> None:
input("Press enter to connect.")
async with websockets.connect(URL) as ws:
msg_initial: Dict[str,str] = get_dict()
await ws.send(json.dumps(msg_initial))
while True:
msg = await ws.recv()
print(f"Server: {msg}")
# Non-daemon threads so program will not end until these threads terminate:
t = Thread(target=process_msg, args=(msg,))
t.start()
asyncio.get_event_loop().run_until_complete(listen())
Update
Based on your last comment to my answer concerning creating a chat program, you should either implement this using pure multithreading or pure asyncio. Here is a rough outline using asyncio:
import websockets
import asyncio
import aioconsole
URL = "my localhost webserver"
async def receiver(connection):
while True:
msg = await connection.recv()
print(f"\nServer: {msg}")
async def sender(connection):
while True:
msg = await aioconsole.ainput('\nEnter msg: ')
await connection.send(msg)
async def chat() -> None:
async with websockets.connect(URL) as ws:
await asyncio.gather(
receiver(ws),
sender(ws)
)
asyncio.get_event_loop().run_until_complete(chat())
However, you may be limited in the type of user input you can do with asyncio. I would think, therefore, that multithreading might be a better approach.

Python Websocket only receives once from client

I want to send values from a for loop from the client-server but the server only receives the first value and the connection is cut shortly
Client
import asyncio
import websockets
import time
async def message():
async with websockets.connect("ws://-------:5051") as socket:
for i in range(20):
await socket.send(f"{i}")
print(i)
time.sleep(4)
asyncio.get_event_loop().run_until_complete(message())
Server
import asyncio
import websockets
async def consumer_handler(websocket,path):
client_type = await websocket.recv()
print(client_type)
start_server = websockets.serve(consumer_handler,"ws://-------:5051", 5051)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

So, your consumer_handler receives message once and finishes.
You need to add loop.
Try something like this:
async def consumer_handler(websocket, path):
async for msg in websocket:
print(msg)

Python websockets client keep connection open

In Python, I'm using "websockets" library for websocket client.
import asyncio
import websockets
async def init_sma_ws():
uri = "wss://echo.websocket.org/"
async with websockets.connect(uri) as websocket:
name = input("What's your name? ")
await websocket.send('name')
greeting = await websocket.recv()
The problem is the client websocket connection is disconnected once a response is received. I want the connection to remain open so that I can send and receive messages later.
What changes do I need to keep the websocket open and be able to send and receive messages later?

I think your websocket is disconnected due to exit from context manager after recv().
Such code works perfectly:
import asyncio
import websockets
async def init_sma_ws():
uri = "wss://echo.websocket.org/"
async with websockets.connect(uri) as websocket:
while True:
name = input("What's your name? ")
if name == 'exit':
break
await websocket.send(name)
print('Response:', await websocket.recv())
asyncio.run(init_sma_ws())

In your approach you used a asynchronous context manager which closes a connection when code in the block is executed. In the example below an infinite asynchronous iterator is used which keeps the connection open.
import asyncio
import websockets
async def main():
async for websocket in websockets.connect(...):
try:
...
except websockets.ConnectionClosed:
continue
asyncio.run(main())
More info in library's docs.

listen to multiple socket with websockets and asyncio

I am trying to create a script in python that listens to multiple sockets using websockets and asyncio, the problem is that no matter what I do it only listen to the first socket I call.
I think its the infinite loop, what are my option to solve this? using threads for each sockets?
async def start_socket(self, event):
payload = json.dumps(event)
loop = asyncio.get_event_loop()
self.tasks.append(loop.create_task(
self.subscribe(event)))
# this should not block the rest of the code
await asyncio.gather(*tasks)
def test(self):
# I want to be able to add corotines at a different time
self.start_socket(event1)
# some code
self.start_socket(event2)

this is what I did eventually, that way its not blocking the main thread and all subscriptions are working in parallel.
def subscribe(self, payload):
ws = websocket.WebSocket(sslopt={"cert_reqs": ssl.CERT_NONE})
ws.connect(url)
ws.send(payload)
while True:
result = ws.recv()
print("Received '%s'" % result)
def start_thread(self, loop):
asyncio.set_event_loop(loop)
loop.run_forever()
def start_socket(self, **kwargs):
worker_loop = asyncio.new_event_loop()
worker = Thread(target=self.start_thread, args=(worker_loop,))
worker.start()
worker_loop.call_soon_threadsafe(self.subscribe, payload)
def listen(self):
self.start_socket(payload1)
# code
self.start_socket(payload2)
# code
self.start_socket(payload3)

Your code appears incomplete, but what you've shown has two issues. One is that run_until_complete accepts a coroutine object (or other kind of future), not a coroutine function. So it should be:
# note parentheses after your_async_function()
asyncio.get_event_loop().run_until_complete(your_async_function())
the problem is that no matter what I do it only listen to the first socket I call. I think its the infinite loop, what are my option to solve this? using threads for each sockets?
The infinite loop is not the problem, asyncio is designed to support such "infinite loops". The problem is that you are trying to do everything in one coroutine, whereas you should be creating one coroutine per websocket. This is not a problem, as coroutines are very lightweight.
For example (untested):
async def subscribe_all(self, payload):
loop = asyncio.get_event_loop()
# create a task for each URL
for url in url_list:
tasks.append(loop.create_task(self.subscribe_one(url, payload)))
# run all tasks in parallel
await asyncio.gather(*tasks)
async def subsribe_one(self, url, payload):
async with websockets.connect(url) as websocket:
await websocket.send(payload)
while True:
msg = await websocket.recv()
print(msg)

One way to efficiently listen to multiple websocket connections from a websocket server is to keep a list of connected clients and essentially juggle multiple conversations in parallel.
E.g. A simple server that sends random # to each connected client every few secs:
import os
import asyncio
import websockets
import random
websocket_clients = set()
async def handle_socket_connection(websocket, path):
"""Handles the whole lifecycle of each client's websocket connection."""
websocket_clients.add(websocket)
print(f'New connection from: {websocket.remote_address} ({len(websocket_clients)} total)')
try:
# This loop will keep listening on the socket until its closed.
async for raw_message in websocket:
print(f'Got: [{raw_message}] from socket [{id(websocket)}]')
except websockets.exceptions.ConnectionClosedError as cce:
pass
finally:
print(f'Disconnected from socket [{id(websocket)}]...')
websocket_clients.remove(websocket)
async def broadcast_random_number(loop):
"""Keeps sending a random # to each connected websocket client"""
while True:
for c in websocket_clients:
num = str(random.randint(10, 99))
print(f'Sending [{num}] to socket [{id(c)}]')
await c.send(num)
await asyncio.sleep(2)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
try:
socket_server = websockets.serve(handle_socket_connection, 'localhost', 6789)
print(f'Started socket server: {socket_server} ...')
loop.run_until_complete(socket_server)
loop.run_until_complete(broadcast_random_number(loop))
loop.run_forever()
finally:
loop.close()
print(f"Successfully shutdown [{loop}].")
A simple client that connects to the server and listens for the numbers:
import asyncio
import random
import websockets
async def handle_message():
uri = "ws://localhost:6789"
async with websockets.connect(uri) as websocket:
msg = 'Please send me a number...'
print(f'Sending [{msg}] to [{websocket}]')
await websocket.send(msg)
while True:
got_back = await websocket.recv()
print(f"Got: {got_back}")
asyncio.get_event_loop().run_until_complete(handle_message())
Mixing up threads and asyncio is more trouble than its worth and you still have code that will block on the most wasteful steps like network IO (which is the essential benefit of using asyncio).
You need to run each coroutine asynchronously in an event loop, call any blocking calls with await and define each method that interacts with any awaitable interactions with an async
See a working e.g.: https://github.com/adnantium/websocket_client_server