We Keep Coding

map a request response in python twisted

I have started to learn Python twisted. I have a basic requirement to map the request and response for a TCP client. The client can send any request anytime based on some condition and it expects a specific response for that request from a server. How I could map a request-response pair. A sample code of what I want. Also, the server could send an arbitrary message which does not map to any of client request.
class MSOProtocol(Protocol):
def __init__(self):
self.client_id = uuid4()
self.messagePair = collections.OrderedDict()
def connectionMade(self):
log.msg("Device connected to TSIF")
self.doOperation()
def connectionLost(self, reason):
log.msg('Device lost connection because {}'.format(reason))
def sendMessage(self, data):
log.msg('Device sending...str(msg)'.format(data))
self.transport.write(data)
def dataReceived(self, data):
log.msg('Device received {}'.format(data))
#how do I map the response with my request
def doOperation(self):
CONDITION = "rainy"
if condition == "sunny":
self.sendMessage("sunny weather")
class DeviceFactory(ClientFactory):
def buildProtocol(self, addr):
return MSOProtocol()
def main():
log.startLogging(sys.stdout)
reactor.connectTCP('127.0.0.1', 16000, DeviceFactory())
reactor.run()
if __name__ == '__main__':
main()

return control to the transport

I'm trying to simulate a situation where data is received by a server periodically. In my set up, I run one process that sets up the server and another process that sets up a bunch of clients (suffices to think of a single client). I have set up some of the code by putting together bits and pieces mostly from here. The server/clients communicate by sending messages using transport.write. First, the server tells the clients to start (this works fine AFAIK). The clients report back to the server as they make progress. What has me confused is that I only get these intermittent messages at the very end when the client is done. This could be a problem with buffer flush and I tried (unsuccessfully) things like This. Also, each message is pretty large and I tried sending the same message multiple times so the buffer would get cleared.
I suspect what I am seeing is a problem with returning the control to the transport but I can't figure out how to do away with it.
Any help with this or any other issues that jump up to you is much appreciated.
Server:
from twisted.internet import reactor, protocol
import time
import serverSideAnalysis
import pdb
#import bson, json, msgpack
import _pickle as pickle # I expect the users to authenticate and not
# do anything malicious.
PORT = 9000
NUM = 1
local_scratch="/local/scratch"
class Hub(protocol.Protocol):
def __init__(self,factory, clients, nclients):
self.clients = clients
self.nclients = nclients
self.factory = factory
self.dispatcher = serverSideAnalysis.ServerTalker(NUM, self,
local_scratch)
def connectionMade(self):
print("connected to user" , (self))
if len(self.clients) < self.nclients:
self.factory.clients.append(self)
else:
self.factory.clients[self.nclients] = self
if len(self.clients) == NUM:
val = input("Looks like everyone is here, shall we start? (Y/N)")
while (val.upper() != "Y"):
time.sleep(20)
val = input("Looks like everyone is here, shall we start??? (Y/N)")
message = pickle.dumps({"TASK": "INIT", "SUBTASK":"STORE"})
self.message(message) # This reaches the client as I had expected
def message(self, command):
for c in self.factory.clients:
c.transport.write(command)
def connectionLost(self, reason):
self.factory.clients.remove(self)
self.nclients -= 1
def dataReceived(self, data):
if len(self.clients) == NUM:
self.dispatcher.dispatch(data)
class PauseTransport(protocol.Protocol):
def makeConnection(self, transport):
transport.pauseProducing()
class HubFactory(protocol.Factory):
def __init__(self, num):
self.clients = set([])
self.nclients = 0
self.totConnections = num
def buildProtocol(self, addr):
print(self.nclients)
if self.nclients < self.totConnections:
self.nclients += 1
return Hub(self, self.clients, self.nclients)
protocol = PauseTransport()
protocol.factory = self
return protocol
factory = HubFactory(NUM)
reactor.listenTCP(PORT, factory)
factory.clients = []
reactor.run()
Client:
from twisted.internet import reactor, protocol
import time
import clientSideAnalysis
import sys
HOST = 'localhost'
PORT = 9000
local_scratch="/local/scratch"
class MyClient(protocol.Protocol):
def connectionMade(self):
print("connected!")
self.factory.clients.append(self)
print ("clients are ", self.factory.clients)
self.cdispatcher = clientSideAnalysis.ServerTalker(analysis_file_name, local_scratch, self)
def clientConnectionLost(self, reason):
#TODO send warning
self.factory.clients.remove(self)
def dataReceived(self, data): #This is the problematic part I think
self.cdispatcher.dispatch(data)
print("1 sent")
time.sleep(10)
self.cdispatcher.dispatch(data)
print("2 sent")
time.sleep(10)
self.cdispatcher.dispatch(data)
time.sleep(10)
def message(self, data):
self.transport.write(data)
class MyClientFactory(protocol.ClientFactory):
protocol = MyClient
if __name__=="__main__":
analysis_file_name = sys.argv[1]
factory = MyClientFactory()
reactor.connectTCP(HOST, PORT, factory)
factory.clients = []
reactor.run()
The last bit of relevant information about what the dispatchers do.
In both cases, they load the message that has arrived (a dictionary) and do a few computations based on the content. Every once in a while, they use the message method to communicate with thier current values.
Finally, I'm using python 3.6. and twisted 18.9.0

The way you return control to the reactor from a Protocol.dataReceived method is you return from that method. For example:
def dataReceived(self, data):
self.cdispatcher.dispatch(data)
print("1 sent")
If you want more work to happen after this, you have some options. If you want the work to happen after some amount of time has passed, use reactor.callLater. If you want the work to happen after it is dispatched to another thread, use twisted.internet.threads.deferToThread. If you want the work to happen in response to some other event (for example, data being received), put it in the callback that handles that event (for example, dataReceived).

Use twisted ,how can I keep the client object

Here is my simple twisted server code:
from twisted.internet.protocol import Protocol,Factory
from twisted.internet import reactor
class MultiEcho(Protocol):
def __init__(self, factory):
self.recvBuf = ""
self.factory = factory
def connectionMade(self):
self.factory.echoers.append(self)
print self.factory.echoers
def dataReceived(self, data):
self.recvBuf += data
self.transport.write(data)
def connectionLost(self, reason):
self.factory.echoers.remove(self)
class MultiEchoFactory(Factory):
def __init__(self):
self.echoers=[]
def buildProtocol(self, addr):
return MultiEcho(self)
if __name__ == '__main__':
reactor.listenTCP(4321, MultiEchoFactory())
reactor.run()
Use 'telnet 127.0.0.1 4321' ,will print a list:
[<__main__.MultiEcho instance at 0x0000000002EA6B08>]
add another 'telnet 127.0.0.1 4321',the list is:
[<__main__.MultiEcho instance at 0x0000000002EA6B08>, <__main__.MultiEcho instance at 0x0000000002EA6EC8>]
I know a client is a protocol object's instance ,and the list is MultiEchoFactory.echoers,so I want to keep the list,
import them in another .py file use list[n].transport.write(somedata).
I try from myserver import MultiEchoFactory,but the MultiEchoFactory.echoers is empty.
I also rewrite the code
class MultiEchoFactory(Factory):
def __init__(self):
self.echoers=[]
to
class MultiEchoFactory(Factory):
echoers=[]
But I still can't use the protocol list in another .py file.
So,what should I do?

You need to save it in buildProtocol(), i.e.:
class MultiEchoFactory(Factory):
echoers=[]
def buildProtocol(self, addr):
echoer = MultiEcho(self)
MultiEchoFactory.echoers.append(echoer)
return echoer

python twisted man-in-the-middle implementation

What I need is a sort of man-in-the-middle implementation: I need a server who receives connections from clients (binary data with different lengths) and forwards the stream to a server it connects to (acting as a client), and then sends the data back from the server it is connected to, to the clients.
It actually works standing between the clients and the servers, and passing the data they exchange (which is a stream, so it continuously get from one side and sends to the other one).
The server is static, so it is always the same, and its address can even be hardcoded; however when a client drops the connection, this server must also drop the connection to the "real" server.
I've been looking around, but couldn't find a solution or an example for such a simple problem.
The code I've made works actually, but I have not yet managed to find how to put a reference into the server part that says "this is your assigned client", or into the client that says "this is your server". Here's my code:
#!/usr/bin/env python
from twisted.internet import protocol, reactor
from twisted.protocols import basic
client = None
server = None
class ServerProtocol(protocol.Protocol):
def connectionMade(self):
global server
factory = protocol.ClientFactory()
factory.protocol = ClientProtocol
server = self
reactor.connectTCP('localhost', 1324, factory)
def dataReceived(self, data):
global client
client.transport.write(data)
class ClientProtocol(protocol.Protocol):
def connectionMade(self):
global client
# Here's the instance of the client
client = self
def dataReceived(self, data):
global server
server.transport.write(data)
def main():
import sys
from twisted.python import log
log.startLogging(sys.stdout)
factory = protocol.ServerFactory()
factory.protocol = ServerProtocol
# Here's the instance of the server
server = ServerProtocol
reactor.listenTCP(2593, factory)
reactor.run()
if __name__ == '__main__':
main()
Now, the point is that the instance can't be contained into the global objects, and should be put inside the two classes: how?

I've managed to solve the issue by myself and, for future references (or to help anybody else who had this problem), here's the code I used to solve it.
I think both my solution and the one kindly given by jedwards work; now I just have to study his own a little more to be sure that what I've done is correct: this is my first application using the Twisted framework and studying somebody else's solution is the way to learn something new! :)
#!/usr/bin/env python
from twisted.internet import protocol, reactor
from twisted.protocols import basic
class ServerProtocol(protocol.Protocol):
def __init__(self):
self.buffer = None
self.client = None
def connectionMade(self):
factory = protocol.ClientFactory()
factory.protocol = ClientProtocol
factory.server = self
reactor.connectTCP('gameserver16.gamesnet.it', 2593, factory)
def dataReceived(self, data):
if (self.client != None):
self.client.write(data)
else:
self.buffer = data
def write(self, data):
self.transport.write(data)
print 'Server: ' + data.encode('hex')
class ClientProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.server.client = self
self.write(self.factory.server.buffer)
self.factory.server.buffer = ''
def dataReceived(self, data):
self.factory.server.write(data)
def write(self, data):
self.transport.write(data)
print 'Client: ' + data.encode('hex')
def main():
import sys
from twisted.python import log
log.startLogging(sys.stdout)
factory = protocol.ServerFactory()
factory.protocol = ServerProtocol
reactor.listenTCP(2593, factory)
reactor.run()
if __name__ == '__main__':
main()

Consider this approach
#!/usr/bin/env python
import sys
from twisted.internet import reactor
from twisted.internet.protocol import ServerFactory, ClientFactory, Protocol
from twisted.protocols import basic
from twisted.python import log
LISTEN_PORT = 2593
SERVER_PORT = 1234
class ServerProtocol(Protocol):
def connectionMade(self):
reactor.connectTCP('localhost', SERVER_PORT, MyClientFactory(self))
def dataReceived(self, data):
self.clientProtocol.transport.write(data)
class ClientProtocol(Protocol):
def connectionMade(self):
# Pass ServerProtocol a ref. to ClientProtocol
self.serverProtocol.clientProtocol = self;
def dataReceived(self, data):
self.serverProtocol.transport.write(data)
class MyServerFactory(ServerFactory):
protocol = ServerProtocol
def buildProtocol(self, addr):
# Create ServerProtocol
p = ServerFactory.buildProtocol(self, addr)
return p
class MyClientFactory(ClientFactory):
protocol = ClientProtocol
def __init__(self, serverProtocol_):
self.serverProtocol = serverProtocol_
def buildProtocol(self, addr):
# Create ClientProtocol
p = ClientFactory.buildProtocol(self,addr)
# Pass ClientProtocol a ref. to ServerProtocol
p.serverProtocol = self.serverProtocol
return p
def main():
log.startLogging(sys.stdout)
reactor.listenTCP(LISTEN_PORT, MyServerFactory())
reactor.run()
if __name__ == '__main__':
main()
The ServerProtcol instance, passes a reference of itself to the MyClientFactory constructor, which then sets tells the ClientProtcol what ServerProtocol instance it's associated with.
Similarly, when the ClientProtocol connection is established, it uses it's reference to the ServerProtocol to tell the ServerProtocol what ClientProtocol to use.
Note: There's no error checking in this code, so you may encounter errors regarding NoneType if things go wrong (for example, if the real server isn't listening).
The important lines are:
reactor.connectTCP('localhost', SERVER_PORT, MyClientFactory(self))
#...
def __init__(self, serverProtocol_):
self.serverProtocol = serverProtocol_
Here, you pass a reference to the ServerProtocol to the MyClientFactory constructor. It stores this reference in a member variable. You do this so that that when the client factory creates a ClientProtocol, it can pass the reference on:
# Pass ClientProtocol a ref. to ServerProtocol
p.serverProtocol = self.serverProtocol
Then, once the connection is made from your script to the real server, the reverse happens. The ClientProtocol gives the ServerProtocol a reference to itself:
# Pass ServerProtocol a ref. to ClientProtocol
self.serverProtocol.clientProtocol = self;
Finally, both protocols use the stored references of each other to send data when it is received:
def dataReceived(self, data):
self.clientProtocol.transport.write(data)
#...
def dataReceived(self, data):
self.serverProtocol.transport.write(data)

Python Chat Client not Remaining Open

I have recently started developing a server-client chat protocol for learning purposes (later I would like to do more with this communication, but for now this suffices. Needless to say, I am still early into the learning phase of this portion of Python, but I have modified some examples to a server and a client that I found online. The communication works well from what I've seen so far, but I have to relaunch the client every time I want to send a message to the server.
Here is the code:
server:
from twisted.internet import reactor, protocol
from twisted.protocols import basic
class Echo(protocol.Protocol):
def dataReceived(self, data):
"As soon as any data is received, write it back."
self.transport.write(data)
class MyChat(basic.LineReceiver):
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def dataReceived(self, data):
print "received", repr(data)
for c in self.factory.clients:
c.message(data)
def message(self, message):
self.transport.write(message + '\n')
def main():
"""This runs the protocol on port 8000"""
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
reactor.listenTCP(8000,factory)
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
client:
from twisted.internet import reactor, protocol
# a client protocol
class EchoClient(protocol.Protocol):
"""Once connected, send a message, then print the result."""
def connectionMade(self):
self.transport.write("hello, world!")
def dataReceived(self, data):
"As soon as any data is received, write it back."
print "Server said:", data
self.transport.loseConnection()
def connectionLost(self, reason):
print "connection lost"
class EchoFactory(protocol.ClientFactory):
protocol = EchoClient
def clientConnectionFailed(self, connector, reason):
connector.connect()
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
connector.connect()
print "Connection lost - goodbye!"
reactor.stop()
# this connects the protocol to a server runing on port 8000
def main():
f = EchoFactory()
client = EchoClient()
reactor.connectTCP("localhost", 8000, f)
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
What am I forgetting to add so that I can have multiple clients connected to the server and stay connected?
I looked here and here (the first one seems to be the same type of question) but I'm still confused as to how to fix this issue. Any suggestions are appreciated. Thanks!