I'm trying to create a websocket server on which a single client will push its messages (I know this is not the usual way to use websocket but this part is not my choice). To this end, I'm using python 3.7 and websockets 7.0.
My issue is: Numerous messages pushed by the client are not received by the server.
Here is the simple code I'm using.
import asyncio
import websockets
async def get_tag_data(websocket, path):
# print('received')
data = await websocket.recv()
print(data)
loop = asyncio.get_event_loop()
anchors_server = websockets.serve(get_tag_data, 'localhost', 9001)
loop.run_until_complete(asyncio.gather(anchors_server))
loop.run_forever()
Conversely, when I try the python-websocket-server (which uses threads for reception), all my messages are correctly received.
As far as I understand asyncio & websockets, it is supposed to manage backpressure: messages sent while the server is busy (to process older messages) are "buffered" and will be treat shortly....
What am I missing? Do I need to thread the websocket reception with asyncio to avoid losing messages?
Thank you for your answers!
Ok, I get it.
My function was running only once, next messages was not buffered. The following code resolve the issue :
import asyncio
import websockets
import signal
import sys
async def get_tag_data(websocket, path):
while True:
async for data in websocket:
print(data)
loop = asyncio.get_event_loop()
anchors_server = websockets.serve(get_tag_data, '', 9001)
loop.run_until_complete(asyncio.gather(anchors_server))
loop.run_forever()
note the
while True:
async for data in websocket
Shoud be without while loop.
import asyncio
import websockets
import signal
import sys
async def get_tag_data(websocket, path):
async for data in websocket:
print(data)
Related
I’m trying to write a multiplayer card game for a friend.
I want to make it simple enough that they (who are new-ish to Python) can easily understand the code, while still being fully featured.
I would like to have multiple clients connecting to the server, and it was my idea to send “messages”, which are JSON. As the messages arrived, they were appended to an “ID”, and then put in a queue, which is a list. The game engine code could just pop off messages, process them, and then append messages to the outgoing queue.
I was wondering what the easiest way to implement this would be, or if there is a simpler way I should be considering.
I’ve seen some people using socketserver, others using asyncio, and some people using threading.
I figured out a possible solution, using asyncio:
import asyncio
import json
outgoing = asyncio.Queue()
incoming = asyncio.Queue()
async def handle_incoming(reader):
buffer = ""
while True:
buffer += (await reader.read(1)).decode('utf-8')
if '\r\n' in buffer:
packet, buffer = buffer.split('\r\n', 1)
message = json.loads(packet)
await incoming.put(message)
async def handle_outgoing(writer):
while True:
message = await outgoing.get()
packet = json.dumps(message) + '\r\n'
writer.write(packet.encode('utf-8'))
await writer.drain()
async def handle_client(reader, writer):
asyncio.create_task(handle_incoming(reader))
asyncio.create_task(handle_outgoing(writer))
async def main():
# This is where you would do your main program logic
while True:
print(await incoming.get())
async def run_server():
asyncio.create_task(main())
server = await asyncio.start_server(handle_client, 'localhost', 15555)
async with server:
await server.serve_forever()
asyncio.run(run_server())
I have the following Python websocket server. It gets some usage data from my server and then sends it through a websocket connection:
#!/usr/bin/env python3
import json
import asyncio
import websockets
import websockets.server
import data_parser as dp
async def handler(websocket: websockets.server.WebSocketServerProtocol):
"""
Handles data connection and dispatches necessary data
"""
# Receive and parse the amount of days wanted
message = await websocket.recv()
json_message = json.loads(message)
days = json_message["days"]
for i, data in enumerate(dp.get_upto_nth_usage_data(days)):
await websocket.send(json.dumps(data, separators=(",", ":")))
to_send = {"finished": 1}
await websocket.send(json.dumps(to_send, separators=(",", ":")))
async def main():
async with websockets.serve(handler, "", 8001):
await asyncio.Future() # run forever
if __name__ == "__main__":
asyncio.run(main())
The problem is that I have to send large chunks of data(inside the for loop) over each websocket connection and when a new user tries to make a connection, he has to wait for the last user to receive all his data to be able to begin receiving his own. Is there a way to open a new port to handle new websocket connections or should I use another kind of protocol to send data (and if so, which would be the best choice)?
I have a large python app (now converted to python3) where the main code ran in an infinite loop.
while (True):
#Do loads of stuff, connect to rs485 devices, obtain their status etc.
I'm wanting to use websockets with this app. I've now converted it so the structure is:
def mainPoll():
#Do loads of stuff (omitted for brevity)
while (True):
mainPoll()
The websockets sample code ( from HERE) I have as a starting point is:
import asyncio
import datetime
import random
import websockets
async def time(websocket, path):
while True:
now = datetime.datetime.utcnow().isoformat() + "Z"
await websocket.send(now)
await asyncio.sleep(random.random() * 3)
start_server = websockets.serve(time, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
EDIT I've been trying to combine the two. Mycode below will run mainPoll() asynchronously once using create_task. Is there a better way to keep running it that calling create_task again at the end of mainPoll?
async def mainPoll():
#do loads of stuff (omitted for brevity)
#then create another task to run myself again?!?!?!
#IS THIS NEXT LINE THE BEST WAY?
asyncio.get_event_loop().create_task(mainPoll())
#end of mainPoll
async def xtime(websocket, path):
while True:
now = str(time.time())
await websocket.send(now)
await asyncio.sleep(random.random() * 3)
start_server = websockets.serve(xtime, "0.0.0.0", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
poll=asyncio.get_event_loop().create_task(mainPoll())
asyncio.get_event_loop().run_forever()
I'd really appreciate some pointers here as the websockets code doesn't seem to match any of the asyncio examples I can find. Many thanks.
mainPoll needs to be an async function. There is no need to re-create the task as shown in the question - it is fine for it to contain a while True loop, as long as something is awaited inside the loop. If you need blocking APIs there, use run_in_executor to await them without disturbing websockets.
I am using the following code to test that a local websocket server is running:
import asyncio
import websockets
async def hello():
async with websockets.connect('ws://127.0.0.1:8000/ws/registration/123') as websocket:
await websocket.send(json_data)
asyncio.get_event_loop().run_until_complete(hello())
Is there a simpler way to do this without using asyncio? Something such as:
import asyncio
import websockets
conn = websockets.connect('ws://127.0.0.1:8000/ws/registration/123')
conn.send('hello')
Basically, I'm just trying to find the simplest way to test to see if my websocket server is listening and receiving messages at a particular url.
Doesn't async_to_sync make this more complex? Why not just create a normal test_url function:
def test_url(url, data=""):
async def inner():
async with websockets.connect(url) as websocket:
await websocket.send(data)
return asyncio.get_event_loop().run_until_complete(inner())
test_url("ws://127.0.0.1:8000/ws/registration/123")
You can do the above by using async_to_sync, for example:
from asgiref.sync import async_to_sync
import websockets
def test_url(url, data=""):
conn = async_to_sync(websockets.connect)(url)
async_to_sync(conn.send)(data)
test_url("ws://127.0.0.1:8000/ws/registration/123")
Note that the "handshake" will probably not complete here because it needs to be accepted both ways, but the above should enable you to test to make sure that the urls are being routed properly, etc.
I am trying to create a script in python that listens to multiple sockets using websockets and asyncio, the problem is that no matter what I do it only listen to the first socket I call.
I think its the infinite loop, what are my option to solve this? using threads for each sockets?
async def start_socket(self, event):
payload = json.dumps(event)
loop = asyncio.get_event_loop()
self.tasks.append(loop.create_task(
self.subscribe(event)))
# this should not block the rest of the code
await asyncio.gather(*tasks)
def test(self):
# I want to be able to add corotines at a different time
self.start_socket(event1)
# some code
self.start_socket(event2)
this is what I did eventually, that way its not blocking the main thread and all subscriptions are working in parallel.
def subscribe(self, payload):
ws = websocket.WebSocket(sslopt={"cert_reqs": ssl.CERT_NONE})
ws.connect(url)
ws.send(payload)
while True:
result = ws.recv()
print("Received '%s'" % result)
def start_thread(self, loop):
asyncio.set_event_loop(loop)
loop.run_forever()
def start_socket(self, **kwargs):
worker_loop = asyncio.new_event_loop()
worker = Thread(target=self.start_thread, args=(worker_loop,))
worker.start()
worker_loop.call_soon_threadsafe(self.subscribe, payload)
def listen(self):
self.start_socket(payload1)
# code
self.start_socket(payload2)
# code
self.start_socket(payload3)
Your code appears incomplete, but what you've shown has two issues. One is that run_until_complete accepts a coroutine object (or other kind of future), not a coroutine function. So it should be:
# note parentheses after your_async_function()
asyncio.get_event_loop().run_until_complete(your_async_function())
the problem is that no matter what I do it only listen to the first socket I call. I think its the infinite loop, what are my option to solve this? using threads for each sockets?
The infinite loop is not the problem, asyncio is designed to support such "infinite loops". The problem is that you are trying to do everything in one coroutine, whereas you should be creating one coroutine per websocket. This is not a problem, as coroutines are very lightweight.
For example (untested):
async def subscribe_all(self, payload):
loop = asyncio.get_event_loop()
# create a task for each URL
for url in url_list:
tasks.append(loop.create_task(self.subscribe_one(url, payload)))
# run all tasks in parallel
await asyncio.gather(*tasks)
async def subsribe_one(self, url, payload):
async with websockets.connect(url) as websocket:
await websocket.send(payload)
while True:
msg = await websocket.recv()
print(msg)
One way to efficiently listen to multiple websocket connections from a websocket server is to keep a list of connected clients and essentially juggle multiple conversations in parallel.
E.g. A simple server that sends random # to each connected client every few secs:
import os
import asyncio
import websockets
import random
websocket_clients = set()
async def handle_socket_connection(websocket, path):
"""Handles the whole lifecycle of each client's websocket connection."""
websocket_clients.add(websocket)
print(f'New connection from: {websocket.remote_address} ({len(websocket_clients)} total)')
try:
# This loop will keep listening on the socket until its closed.
async for raw_message in websocket:
print(f'Got: [{raw_message}] from socket [{id(websocket)}]')
except websockets.exceptions.ConnectionClosedError as cce:
pass
finally:
print(f'Disconnected from socket [{id(websocket)}]...')
websocket_clients.remove(websocket)
async def broadcast_random_number(loop):
"""Keeps sending a random # to each connected websocket client"""
while True:
for c in websocket_clients:
num = str(random.randint(10, 99))
print(f'Sending [{num}] to socket [{id(c)}]')
await c.send(num)
await asyncio.sleep(2)
if __name__ == "__main__":
loop = asyncio.get_event_loop()
try:
socket_server = websockets.serve(handle_socket_connection, 'localhost', 6789)
print(f'Started socket server: {socket_server} ...')
loop.run_until_complete(socket_server)
loop.run_until_complete(broadcast_random_number(loop))
loop.run_forever()
finally:
loop.close()
print(f"Successfully shutdown [{loop}].")
A simple client that connects to the server and listens for the numbers:
import asyncio
import random
import websockets
async def handle_message():
uri = "ws://localhost:6789"
async with websockets.connect(uri) as websocket:
msg = 'Please send me a number...'
print(f'Sending [{msg}] to [{websocket}]')
await websocket.send(msg)
while True:
got_back = await websocket.recv()
print(f"Got: {got_back}")
asyncio.get_event_loop().run_until_complete(handle_message())
Mixing up threads and asyncio is more trouble than its worth and you still have code that will block on the most wasteful steps like network IO (which is the essential benefit of using asyncio).
You need to run each coroutine asynchronously in an event loop, call any blocking calls with await and define each method that interacts with any awaitable interactions with an async
See a working e.g.: https://github.com/adnantium/websocket_client_server