We Keep Coding

When is async/await needed in writing to socket in python asnycio.Protocol?

I'm confused as to when and why the async/await syntax would be needed, or not, when using the low level asyncio.Protocol approach in python. Suppose I have a subclass of asyncio.Protocol, say EchoClientProtocol, and that has a method send_message that external code can use to send any message to an echo server. For example:
import asyncio
class EchoClientProtocol(asyncio.Protocol):
def __init__(self, message):
self.transport = None
self.message = message
def send_message(self,txt):
self.message = txt
self.transport.write(self.message.encode())
print('Data sent: {!r}'.format(self.message))
def connection_made(self, transport):
self.transport=transport
self.send_message(self.message)
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def connection_lost(self, exc):
print('The server closed the connection')
self.on_con_lost.set_result(True)
We can create and run the client with code that uses the familiar async/away syntax, with some auto-reconnect logic built in:
async def main():
while True:
try:
message="Initial message here"
transport, protocol = await loop.create_connection(
lambda: EchoClientProtocol(message),
'127.0.0.1',
8888
)
except OSError:
print("Server not up, retrying in 5 seconds...")
await asyncio.sleep(5)
else:
break
loop = asyncio.get_event_loop()
asyncio.run(main())
The above client class EchoClientProtocol does not declare send_message as async, and does not await the self.transport.write() method. I have seen code like this in many examples online. Why would this work if it uses asyncio? Isn't the async/await syntax necessary? Why or why not?
Suppose I have two clients in the same python script and both have to run simultaneously without blocking each other. For example, suppose that one echo client connects to a server at port 8888 and another connect to another client server at port 8899. I can use async.gather to make sure that both run simultaneously, but in that case do I have to declare send_message as async, and do I have to await self.transport.write()? In other words, what would I have to change in the EchoClientProtocol above so that I could run two or more such clients in the same script (connecting to different servers on different ports) to make them both run without blocking each other?

Sending data to multiple websocket connections in Python

I have a server that gathers data from a bunch of GPS trackers, and want to ship this data out in real time to X connected clients via WebSockets. The trackers connect over TCP (each in their own thread) and send data regularly to the server. The data is merged in a thread called data_merger and put in that threads queue(). This mechanic works nicely and as intended, however I'm running into issues when I want to send this data to websocket connections.
I tried basing my solution on the websocket synchronization example, as this seemed like it applied to my usecase. I have a thread called outbound_worker that handles the websocket code. From thread.run():
def run(self):
self.data_merger.name = 'data_merger'
self.data_merger.start()
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
start_server = websockets.serve(self.handle_clients, 'localhost', self.port)
print("WebSocker server started for port %s at %s" % (self.port, datetime.now()))
loop.run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Then the handler method for the websocket server:
async def handle_clients(self, websocket, path):
while True:
try:
# Register the websocket to connected set
await self.register(websocket)
data = self.data_merger.queue.get()
await send_to_clients(data)
await asyncio.sleep(0.1)
except websockets.ConnectionClosed:
print("Connection closed")
await self.unregister(websocket)
break
async def send_to_clients(self, data):
data = json.dumps(data)
if self.connected:
await asyncio.wait([ws.send(data) for ws in self.connected])
The register() and unregister() methods are identical to the example I linked above. The client I'm using is a basic loop that prints the data received:
async def hello():
uri = "ws://localhost:64000"
while True:
async with websockets.connect(uri) as websocket:
print("Awaiting data...")
data = await websocket.recv()
#print(f"{data}")
print(f"{json.loads(data)}")
asyncio.get_event_loop().run_until_complete(hello())
As I am new to asynchronous calls in Python and websockets in general, I'm not sure if my approach is correct here. Since I am trying to push data right after registering a new connection, the code seems to halt at the await send_to_clients(data) line. Should I rather handle this in the data_merger thread and pass the connected set?
Another issue is that if I simply use the client_handler to register() and unregister() the new connections, it seems to just loop over the register() part and I'm unable to connect a second client.
I guess my questions can be condensed into the following:
How do I accept and manage multiple open connections over websocket, similar to a multithreaded socket server?
Is there a way to trigger a function call (for instance register() only on new websocket connections, similar to socket.listen() and socket.accept()?

Unable to send data to a specific websocket client in python

I'm writing a program with websockets in python. I've got an example server and client code running and they work well if only one client is connected. If there are multiple clients, data from the server will go randomly to one of the clients.
I would like for:
Server to keep track of the various clients connected
Server to be able to direct messages to a specific client out of multiple(For eg. 5) clients
websockets is the library I'm using.
Python version 3.7.2
Server Code:
import asyncio
import websockets
uri='localhost'
async def response(websocket, path):
msg = input("What do you want to send : ")
print("message:",msg)
await websocket.send(msg)
start_server = websockets.serve(response, uri, 5000)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Client Code:
import asyncio
import websockets
uri="ws://localhost:5000"
async def message():
async with websockets.connect(uri) as socket:
print(await socket.recv())
while True:
asyncio.get_event_loop().run_until_complete(message())
If I create 2 files with the client code as client1.py and client2.py, and send message from the server side, I get the sent data going to either on of the clients.
I would like to:
Server keeps track of the various clients connected
Server is able to direct messages to a specific client out of multiple clients
As I am just starting out with websockets, all input is appreciated.
In this output given, I intended to send all my messages to client 1, yet they got split up between client 1 and 2

"websocket" targets the current connection and if you say "websocket.send(msg)" you're sending a message to the client that has just connected and websocket is an object that is reusable while the client is connected. You can assign the websocket as a variable then send a message some other time as long as the connection is still opened.
NOT RECOMMENED
Requiring user's input from the server is not a good idea because now you're awaiting the server until it receives user inputs. Nothing really happens to the server while it's waiting for user's input and this may crush your server.
RECOMMENED
A client has to tell the server which connection / client to send the message to. I would recommend using a JSON format when sending messages within client's and the server and then convert the String to a python-dict since websocket requires only strings.
Click here to check out my GitHub repository. A websockets server made only Python.
SERVER EXAMPLE
Example on how you can send a packet to a specific client

I'm not familiar with asyncio, so I will try to get to the point with functions/threads;
Usually, my server side listens to one connection and once it accepts it, I have a function 'handler' that is threaded to each connection that gets accepted.
part of my server and handler:
def handler(conn, addr):
global data1
while True:
data = conn.recv(2048)
data1 = json.loads(data.decode())
# treat it as you need
while True:
s.listen(1)
conn, addr = s.accept()
print('Conectado com', addr[0],':', str(addr[1]))
thr = threading.Thread(target = handler, args = (conn, addr)).start()
Now, for the control of the clients and such, I always use a dictionary. The key should be the username or any other particular info. The value of the key is the 'conn' from that user. This way you can get the connection of user 'x' by its specific key.
Something like:
import socket
import time
import datetime as dt
import base64
import os
import json
import threading
HOST = ''
PORT = 12999
global s
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST,PORT))
whoto = {}
def autenticacao():
global data1
global conn
global nomeuser
user1 = data1[1]
passwd = data1[2]
auth = c.execute('SELECT usuario FROM fullinfo WHERE usuario= ? AND password = ?', (user1,passwd)).fetchone()
if auth is not None:
word = 'autenticado'
conn.sendall(word.encode())
nomeuser = auth[0]
whoto[nomeuser] = conn
I'm sorry i'm leaving it unreproducible, but my point is to show the 'algorithm'. This dictionary is what I use to keep record of who is online, the 'adress' (conn) of each client to send messages to single clients and such.
On the example above, I add the user (key) and conn (value) once it's authenticated inside my server.
Hope this helps. Good luck!

Python asyncio - starting coroutines in an infinite loop

I am making a simple server/client chat program in Python. This program should allow for multiple users to connect at once, and then execute their requests concurrently. For this, I am using the asyncio module and sockets.
async def accept_new_connections(socket):
socket.listen(1)
while True:
connection, client_address = sock.accept()
print("accepted conn")
asyncio.create_task(accept_commands(socket, connection))
async def accept_commands(socket, connection):
print("accept cmd started")
while True:
# get and execute commands
def main():
asyncio.run(accept_new_connections(socket))
main()
What I would hope to do is running accept_commands for each of the connections, which would then execute commands concurrently. However, the current code only starts accept_commands for the first connection, and blocks the while loop (the one in accept_new_connections).
Any idea what I need to change to have accept_command started for each of the connections instead?

It is tough to tell because your example does have the implementation of accept_commands, but based on your issue it is likely you need to use the async socket methods on event loop itself so that your coroutine can yield execution and let something else happen.
The below example shows how to do this. This starts a socket on port 8080 and will send back any data it receives back to the client. You can see this work concurrently by connecting two clients with netcat or telnet and sending data.
import asyncio
import socket
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.bind(('localhost', 8080))
socket.listen(1)
socket.setblocking(False)
loop = asyncio.new_event_loop()
async def main():
while True:
connection, client_address = await loop.sock_accept(socket)
print('connected')
loop.create_task(accept_commands(connection))
async def accept_commands(connection):
while True:
request = await loop.sock_recv(connection, 16)
await loop.sock_sendall(connection, request)
loop.run_until_complete(main())

Python3 asyncio: using infinite loop for multiple connections and proper connection closing

I have server, where I need to keep connection with client as long as possible. I need to allow for multiple clients connect to this server. Code:
class LoginServer(BaseServer):
def __init__(self, host, port):
super().__init__(host, port)
async def handle_connection(self, reader: StreamReader, writer: StreamWriter):
peername = writer.get_extra_info('peername')
Logger.info('[Login Server]: Accepted connection from {}'.format(peername))
auth = AuthManager(reader, writer)
while True:
try:
await auth.process()
except TimeoutError:
continue
finally:
await asyncio.sleep(1)
Logger.warning('[Login Server]: closing...')
writer.close()
#staticmethod
def create():
Logger.info('[Login Server]: init')
return LoginServer(Connection.LOGIN_SERVER_HOST.value, Connection.LOGIN_SERVER_PORT.value)
The problem: currently only one client can connect to this server. It seems socket do not closing properly. And because of this even previous client cannot reconnect. I think this is because infinite loop exists. How to fix this problem?

The while loop is correct.
If you wanted a server that waits on data from a client you would have the following loop in your handle_connection.
while 1:
data = await reader.read(100)
# Do something with the data
See the example echo server here for more details on reading / writing.
https://asyncio.readthedocs.io/en/latest/tcp_echo.html
Your problem is likely that this function doesn't return and is looping itself without await'g anything. That would mean the asyncio loop would never regain control so new connections could not be made.
await auth.process()