While looking over the new features in Python 3.x, I was intrigued by the asyncio library being added. After looking at the reference documentation, I decided to play around with it a little bit.
This worked well until I tried to make it work for multiple clients, and keep a list of all active/connected clients. This introduced a cyclic dependency between the server class and the session class.
Now, I've tried a few different ways to resolve this; however, there doesn't appear to be any way for me to get at this data directly from the server class through any method/functional call.
While I have been able to workaround this by using a "lazy" import, it seems like this my be indicative of either a poor design, a lack of understanding of the library itself, or a combination of both.
Code wise, I have a small sample put together. Am I missing an obvious solution here, or does my organization need to change to better support the functionality provided by the asyncio library?
__main__.py:
from network.server import Server
def main(args=None):
s = Server()
try:
s.run()
except KeyboardInterrupt:
pass
s.close()
if __name__ == "__main__":
main()
server.py:
import asyncio
from network.session import Session
class Server:
sessionList = []
def __init__(self):
self.handler = None
self.loop = asyncio.get_event_loop()
self.coro = self.loop.create_server(Session, 'localhost', 1234)
def run(self):
self.handler = self.loop.run_until_complete(self.coro)
print('Server Running On: {}'.format(self.handler.sockets[0].getsockname()))
self.loop.run_forever()
def close(self):
self.handler.close()
self.loop.run_until_complete(self.handler.wait_closed())
self.loop.close()
#staticmethod
def add_session(session):
Server.sessionList.append(session)
#staticmethod
def del_session(session):
Server.sessionList.remove(session)
session.py:
import asyncio
class Session(asyncio.Protocol):
def __init__(self):
from network.server import Server
self._transport = None
Server.add_session(self)
def connection_made(self, transport):
self._transport = transport
self._transport.write('Echo Server Example\r\n\r\n'.encode())
def data_received(self, data):
self._transport.write(data)
def eof_received(self):
self._transport.close()
def connection_lost(self, exc):
from network.server import Server
Server.del_session(self)
if exc is not None:
self._transport.close()
You may pass server instance into Session constructor:
self.loop.create_server(lambda: Session(self), 'localhost', 1234)
Storing sessionList as global object is not the best practice.
I recommend saving it as self.sessionList = [] in Server.__init__ and converting both add_session and del_session from staticmethod into regular methods.
Related
I would like to use Twisted as a client/server manager that is part of regular Python objects.
The solution I am trying to implement is to isolate Twisted in its own process using multiprocessing.Process, and communicate with this process through multiprocessing.Pipe.
I have coded the client/server logic with Twisted already, but now I am stuck at interfacing the multiprocessing.Pipe communication with the reactor.
I am a beginner with Twisted so I may be missing something obvious, but from what I understand about how reactors work, I guess the reactor is somehow supposed to poll from my multiprocessing.Pipe along with the sockets that it already seems to handle nicely. So my question is, how can I make the reactor listen to my multiprocessing.Pipe on top of what it is already doing please?
Thus far my code looks something like this:
class ServerProtocol(Protocol):
def __init__(self, server):
self._server = server
def connectionMade(self):
pass
def connectionLost(self, reason):
pass
def dataReceived(self, data):
pass
class ServerProtocolFactory(Factory):
protocol = ServerProtocol
def __init__(self, server):
self.server = server
def buildProtocol(self, addr):
return ServerProtocol(self.server)
class Server:
def __init__(self):
pass
def run(self, pipe):
"""
This is called in its own process
"""
from twisted.internet import reactor
endpoint = TCP4ServerEndpoint(reactor, self._port)
endpoint.listen(ServerProtocolFactory(self))
reactor.run() # main Twisted reactor loop
class MyObject:
def __init__(self):
self._pipe = Pipe()
self._server = Server()
self._p = Process(target=self._server.run, args=(self._pipe, ))
self._p.start()
def stop(self):
# I want to send some stop command through the Pipe here
self._p.join()
if __name__ == "__main__":
obj = MyObject()
# do stuff here
obj.stop()
I don't know if Twisted will work as run this way (i.e., as the target of a multiprocessing.Process). Let's assume it will, though.
multiprocessing.Pipe is documented as returning a two-tuple of multiprocessing.Connection objects. multiprocessing.Connection is documented as having a fileno method returning a file descriptor (or handle) used by the Connection.
If it is a file descriptor then there is probably a very easy path to integrating it with a Twisted reactor. Most Twisted reactors implement IReactorFDSet which has an addReader method which accepts an IReadDescriptor value.
Connection is not quite an IReadDescriptor but it is easily adapted to be one:
from attrs import define
from multiprocessing import Connection
from twisted.python.failure import Failure
#define
class ConnectionToDescriptor:
_conn: Connection
def fileno(self) -> int:
return self._conn.fileno()
def doRead(self) -> None:
some_data = self._conn.recv()
# Process some_data how you like
def connectionLost(self, reason: Failure) -> None:
self._conn.close()
If you wrap this around your read Connection and then pass the result to reactor.addReader the reactor will use fileno to figure out what to monitor for readiness and call doRead when there is something to read.
You could apply similar treatment to the write end of the pipe if you also want reactor-friendly support for sending bytes back to the parent process.
There is plenty of information and examples when it comes to connecting to one server with multiple clients. But I was wondering is there a way for one client to connect to two servers at the same time? Here is my situation:
I have a python client that brings data from one server, analyzes it and sends an appropriate command to another server. There seems to be less information on this issue, If I may call it.
Here is how I tried approaching the issue. First, I made a socketio.Client class, which would enable me to create two client instances. It did not work. What am I missing here?:
import socketio
class SocketClient(socketio.Client):
def __init__(self, server_ip):
self.server_ip = server_ip # server's ip address
self.sio = socketio.Client(logger=True)
def connect(self):
self.sio.connect(self.server_ip, namespaces=['/my_namespace'])
#self.sio.event
def connect_error(self, error):
print('connection error=> ', error)
#self.sio.event
def my_event(self, server_response):
# Here I have to take the server_response
# and send it to another server.
# How do I do it?
# self.sio.emit('some_event', server_response)
# that does not work, as I do not have the second client instance
pass
#self.sio.event
def my_other_event(self, server_response):
# process the response
pass
# initiate the two client instances:
if __name__ == '__main__':
first_client = SocketClient('http://192.168.100.103')
second_client = SocketClient('http://192.168.100.104')
first_client.connect()
second_client.connect()
after my first try did not work, I ditched the class-instance approach and went for functional one:
import socketio
first_client = socketio.Client()
second_client = socketio.Client()
#second_client.event
#first_client.event
def connect():
print(f'connected with id {first_client.sid}')
#second_client.event
#first_client.event
def connect_error(e):
print('Error=> ', e)
#second_client.event
#first_client.event
def disconnect():
print('disconnected')
#first_client.event
def my_event(server_response):
# Here I have to take the server_response
# and send it to another server.
second_client.emit('some_event', server_response) # is it even possible?
#second_client.event
def my_other_event(server_response):
# handle the response
pass
if __name__ == '__main__':
first_client.connect('http://192.168.100.103')
second_client.connect('http://192.168.100.104')
In both cases, I am technically creating two clients. I might as well make them into separate files like first_client.py and second_client.py.
See where I am going with this? The goal is to get the data from server one, process it and send it to the other server with ideally one client. Please forgive me if I am missing something very obvious here. Any help is much appreciated.
P.S. both servers are up and running without any problem.
I am using NameSpace to solve this problem.
first make a Namespace class
class MyCustomNamespace(socketio.AsyncClientNamespace):
async def on_connect(self):
print("I'm connected!")
async def on_disconnect(self):
print("I'm disconnected!")
async def on_my_event(self, data):
await self.emit('my_response', data)
async def on_message(self, data):
print("[echo]:", data)
class mysio:
def __init__(self) -> None:
global sio
self.sio = socketio.AsyncClient(logger=False, engineio_logger=False)
self.sio.register_namespace(MyCustomNamespace('/')) # bind
then make 2 clients.
since wait() will block the process, I use create_task().
async def main():
async def fun1():
sio1 = mysio().sio
await sio1.connect('http://192.168.3.85:11451')
await sio1.emit('message', b'11111110001')
await sio1.wait()
async def fun2():
sio2 = mysio().sio
await sio2.connect('http://localhost:8080')
await sio2.emit('message', 'from sio2')
await sio2.wait()
tasks = [asyncio.create_task(fun1()),asyncio.create_task(fun2()) ]
await asyncio.wait(tasks)
asyncio.run(main())
I'm trying to simulate a situation where data is received by a server periodically. In my set up, I run one process that sets up the server and another process that sets up a bunch of clients (suffices to think of a single client). I have set up some of the code by putting together bits and pieces mostly from here. The server/clients communicate by sending messages using transport.write. First, the server tells the clients to start (this works fine AFAIK). The clients report back to the server as they make progress. What has me confused is that I only get these intermittent messages at the very end when the client is done. This could be a problem with buffer flush and I tried (unsuccessfully) things like This. Also, each message is pretty large and I tried sending the same message multiple times so the buffer would get cleared.
I suspect what I am seeing is a problem with returning the control to the transport but I can't figure out how to do away with it.
Any help with this or any other issues that jump up to you is much appreciated.
Server:
from twisted.internet import reactor, protocol
import time
import serverSideAnalysis
import pdb
#import bson, json, msgpack
import _pickle as pickle # I expect the users to authenticate and not
# do anything malicious.
PORT = 9000
NUM = 1
local_scratch="/local/scratch"
class Hub(protocol.Protocol):
def __init__(self,factory, clients, nclients):
self.clients = clients
self.nclients = nclients
self.factory = factory
self.dispatcher = serverSideAnalysis.ServerTalker(NUM, self,
local_scratch)
def connectionMade(self):
print("connected to user" , (self))
if len(self.clients) < self.nclients:
self.factory.clients.append(self)
else:
self.factory.clients[self.nclients] = self
if len(self.clients) == NUM:
val = input("Looks like everyone is here, shall we start? (Y/N)")
while (val.upper() != "Y"):
time.sleep(20)
val = input("Looks like everyone is here, shall we start??? (Y/N)")
message = pickle.dumps({"TASK": "INIT", "SUBTASK":"STORE"})
self.message(message) # This reaches the client as I had expected
def message(self, command):
for c in self.factory.clients:
c.transport.write(command)
def connectionLost(self, reason):
self.factory.clients.remove(self)
self.nclients -= 1
def dataReceived(self, data):
if len(self.clients) == NUM:
self.dispatcher.dispatch(data)
class PauseTransport(protocol.Protocol):
def makeConnection(self, transport):
transport.pauseProducing()
class HubFactory(protocol.Factory):
def __init__(self, num):
self.clients = set([])
self.nclients = 0
self.totConnections = num
def buildProtocol(self, addr):
print(self.nclients)
if self.nclients < self.totConnections:
self.nclients += 1
return Hub(self, self.clients, self.nclients)
protocol = PauseTransport()
protocol.factory = self
return protocol
factory = HubFactory(NUM)
reactor.listenTCP(PORT, factory)
factory.clients = []
reactor.run()
Client:
from twisted.internet import reactor, protocol
import time
import clientSideAnalysis
import sys
HOST = 'localhost'
PORT = 9000
local_scratch="/local/scratch"
class MyClient(protocol.Protocol):
def connectionMade(self):
print("connected!")
self.factory.clients.append(self)
print ("clients are ", self.factory.clients)
self.cdispatcher = clientSideAnalysis.ServerTalker(analysis_file_name, local_scratch, self)
def clientConnectionLost(self, reason):
#TODO send warning
self.factory.clients.remove(self)
def dataReceived(self, data): #This is the problematic part I think
self.cdispatcher.dispatch(data)
print("1 sent")
time.sleep(10)
self.cdispatcher.dispatch(data)
print("2 sent")
time.sleep(10)
self.cdispatcher.dispatch(data)
time.sleep(10)
def message(self, data):
self.transport.write(data)
class MyClientFactory(protocol.ClientFactory):
protocol = MyClient
if __name__=="__main__":
analysis_file_name = sys.argv[1]
factory = MyClientFactory()
reactor.connectTCP(HOST, PORT, factory)
factory.clients = []
reactor.run()
The last bit of relevant information about what the dispatchers do.
In both cases, they load the message that has arrived (a dictionary) and do a few computations based on the content. Every once in a while, they use the message method to communicate with thier current values.
Finally, I'm using python 3.6. and twisted 18.9.0
The way you return control to the reactor from a Protocol.dataReceived method is you return from that method. For example:
def dataReceived(self, data):
self.cdispatcher.dispatch(data)
print("1 sent")
If you want more work to happen after this, you have some options. If you want the work to happen after some amount of time has passed, use reactor.callLater. If you want the work to happen after it is dispatched to another thread, use twisted.internet.threads.deferToThread. If you want the work to happen in response to some other event (for example, data being received), put it in the callback that handles that event (for example, dataReceived).
I have a module which makes blocking network requests to some TCP server and receive responses. I must integrate it into asyncio application. My module looks like this:
import socket
# class providing transport facilities
# can be implemented in any manner
# for example it can manage asyncio connection
class CustomTransport:
def __init__(self, host, port):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.host = host
self.port = port
def write(self, data):
left = len(data)
while left:
written = self.sock.send(data.encode('utf-8'))
left = left - written
data = data[written:]
def read(self, sz):
return self.sock.recv(sz)
def open(self):
self.sock.connect((self.host, self.port))
def close(self):
self.sock.shutdown(2)
# generated. shouldn't be modified
# however any transport can be passed
class HelloNetClient:
def __init__(self, transport):
self.transport = transport
def say_hello_net(self):
self.transport.write('hello')
response = self.transport.read(5)
return response
# can be modified
class HelloService:
def __init__(self):
# create transport for connection to echo TCP server
self.transport = CustomTransport('127.0.0.1', 6789)
self.hello_client = HelloNetClient(self.transport)
def say_hello(self):
print('Saying hello...')
return self.hello_client.say_hello_net()
def __enter__(self):
self.transport.open()
return self
def __exit__(self,exc_type, exc_val, exc_tb):
self.transport.close()
Usage:
def start_conversation():
with HelloService() as hs:
answer = hs.say_hello()
print(answer.decode('utf-8'))
if __name__ == "__main__":
start_conversation()
Now I see that only way to turn my module to be compatible with asyncio is to convert everything to coroutines and replace regular socket with asyncio-provided transport. But I don't want to touch generated code (HelloNetClient). Is it possible?
P.S. I want it to be used like this:
async def start_conversation():
async with HelloService() as hs:
answer = await hs.say_hello()
print(answer.decode('utf-8'))
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(start_conversation())
HelloService will probably need to use run_in_executor (which manages a thread pool) to run HelloNetClient methods in the background. For example:
async def say_hello(self):
print('Saying hello...')
loop = asyncio.get_event_loop()
return await loop.run_in_executor(None, self.hello_client.say_hello_net)
This is not an idiomatic use of asyncio, and you're missing out on some of its features - for example, you won't be able to create thousands of clients that all work in parallel, and you won't get reliable cancellation (ability to cancel() any task you wish). Nonetheless, simple usage will work just fine.
Unfortunately the ability to provide custom transports is not of help here because the middle tier, HelloNetClient, expects synchronous behavior. Even if you were to write a custom transport that hooked into asyncio, methods like say_hello_net would still wait for as long as it takes for the response to arrive, so HelloService would have to schedule them in a separate thread. For this reason your best bet is to use the default transport and connect the code with asyncio with the code in the service as shown above.
I'm trying to write a Server Side Events server which I can connect to with telnet and have the telnet content be pushed to a browser. The idea behind using Python and asyncio is to use as little CPU as possible as this will be running on a Raspberry Pi.
So far I have the following which uses a library found here: https://pypi.python.org/pypi/asyncio-sse/0.1 which uses asyncio.
And I have also copied a telnet server which uses asyncio as well.
Both work separately, but I have no idea how to tie both together. As I understand it, I need to call send() in the SSEHandler class from inside Telnet.data_received, but I don't know how to access it. Both of these 'servers' need to be running in a loop to accept new connections, or push data.
Can anyone help, or point me in another direction?
import asyncio
import sse
# Get an instance of the asyncio event loop
loop = asyncio.get_event_loop()
# Setup SSE address and port
sse_host, sse_port = '192.168.2.25', 8888
class Telnet(asyncio.Protocol):
def connection_made(self, transport):
print("Connection received!");
self.transport = transport
def data_received(self, data):
print(data)
self.transport.write(b'echo:')
self.transport.write(data)
# This is where I want to send data via SSE
# SSEHandler.send(data)
# Things I've tried :(
#loop.call_soon_threadsafe(SSEHandler.handle_request());
#loop.call_soon_threadsafe(sse_server.send("PAH!"));
def connection_lost(self, esc):
print("Connection lost!")
telnet_server.close()
class SSEHandler(sse.Handler):
#asyncio.coroutine
def handle_request(self):
self.send('Working')
# SSE server
sse_server = sse.serve(SSEHandler, sse_host, sse_port)
# Telnet server
telnet_server = loop.run_until_complete(loop.create_server(Telnet, '192.168.2.25', 7777))
#telnet_server.something = sse_server;
loop.run_until_complete(sse_server)
loop.run_until_complete(telnet_server.wait_closed())
Server side events are a sort of http protocol; and you may have any number of concurrent http requests in flight at any given moment, you may have zero if nobody is connected, or dozens. This nuance is all wrapped up in the two sse.serve and sse.Handler constructs; the former represents a single listening port, which dispatches each separate client request to the latter.
Additionally, sse.Handler.handle_request() is called once for each client, and the client is disconnected once that co-routine terminates. In your code, that coroutine terminates immediately, and so the client sees a single "Working" event. So, we need to wait, more-or-less forever. We can do that by yield froming an asyncio.Future().
The second issue is that we'll somehow need to get a hold of all of the separate instances of a SSEHandler() and use the send() method on each of them, somehow. Well, we can have each one self-register in their handle_request() methods; by adding each one to a dict which maps the individual handler instances to the future they are waiting on.
class SSEHandler(sse.Handler):
_instances = {}
#asyncio.coroutine
def handle_request(self):
self.send('Working')
my_future = asyncio.Future()
SSEHandler._instances[self] = my_future
yield from my_future
Now, to send an event to every listening we just visit all of the SSEHandler instances registered in the dict we created and using send() on each one.
class SSEHandler(sse.Handler):
#...
#classmethod
def broadcast(cls, message):
for instance, future in cls._instances.items():
instance.send(message)
class Telnet(asyncio.Protocol):
#...
def data_received(self, data):
#...
SSEHandler.broadcast(data.decode('ascii'))
lastly, your code exits when the telnet connection closes. that's fine, but we should clean-up at that time, too. Fortunately, that's just a matter of setting a result on all of the futures for all of the handlers
class SSEHandler(sse.Handler):
#...
#classmethod
def abort(cls):
for instance, future in cls._instances.items():
future.set_result(None)
cls._instances = {}
class Telnet(asyncio.Protocol):
#...
def connection_lost(self, esc):
print("Connection lost!")
SSEHandler.abort()
telnet_server.close()
here's a full, working dump in case my illustration is not obvious.
import asyncio
import sse
loop = asyncio.get_event_loop()
sse_host, sse_port = '0.0.0.0', 8888
class Telnet(asyncio.Protocol):
def connection_made(self, transport):
print("Connection received!");
self.transport = transport
def data_received(self, data):
SSEHandler.broadcast(data.decode('ascii'))
def connection_lost(self, esc):
print("Connection lost!")
SSEHandler.abort()
telnet_server.close()
class SSEHandler(sse.Handler):
_instances = {}
#classmethod
def broadcast(cls, message):
for instance, future in cls._instances.items():
instance.send(message)
#classmethod
def abort(cls):
for instance, future in cls._instances.items():
future.set_result(None)
cls._instances = {}
#asyncio.coroutine
def handle_request(self):
self.send('Working')
my_future = asyncio.Future()
SSEHandler._instances[self] = my_future
yield from my_future
sse_server = sse.serve(SSEHandler, sse_host, sse_port)
telnet_server = loop.run_until_complete(loop.create_server(Telnet, '0.0.0.0', 7777))
loop.run_until_complete(sse_server)
loop.run_until_complete(telnet_server.wait_closed())