What I need is a sort of man-in-the-middle implementation: I need a server who receives connections from clients (binary data with different lengths) and forwards the stream to a server it connects to (acting as a client), and then sends the data back from the server it is connected to, to the clients.
It actually works standing between the clients and the servers, and passing the data they exchange (which is a stream, so it continuously get from one side and sends to the other one).
The server is static, so it is always the same, and its address can even be hardcoded; however when a client drops the connection, this server must also drop the connection to the "real" server.
I've been looking around, but couldn't find a solution or an example for such a simple problem.
The code I've made works actually, but I have not yet managed to find how to put a reference into the server part that says "this is your assigned client", or into the client that says "this is your server". Here's my code:
#!/usr/bin/env python
from twisted.internet import protocol, reactor
from twisted.protocols import basic
client = None
server = None
class ServerProtocol(protocol.Protocol):
def connectionMade(self):
global server
factory = protocol.ClientFactory()
factory.protocol = ClientProtocol
server = self
reactor.connectTCP('localhost', 1324, factory)
def dataReceived(self, data):
global client
client.transport.write(data)
class ClientProtocol(protocol.Protocol):
def connectionMade(self):
global client
# Here's the instance of the client
client = self
def dataReceived(self, data):
global server
server.transport.write(data)
def main():
import sys
from twisted.python import log
log.startLogging(sys.stdout)
factory = protocol.ServerFactory()
factory.protocol = ServerProtocol
# Here's the instance of the server
server = ServerProtocol
reactor.listenTCP(2593, factory)
reactor.run()
if __name__ == '__main__':
main()
Now, the point is that the instance can't be contained into the global objects, and should be put inside the two classes: how?
I've managed to solve the issue by myself and, for future references (or to help anybody else who had this problem), here's the code I used to solve it.
I think both my solution and the one kindly given by jedwards work; now I just have to study his own a little more to be sure that what I've done is correct: this is my first application using the Twisted framework and studying somebody else's solution is the way to learn something new! :)
#!/usr/bin/env python
from twisted.internet import protocol, reactor
from twisted.protocols import basic
class ServerProtocol(protocol.Protocol):
def __init__(self):
self.buffer = None
self.client = None
def connectionMade(self):
factory = protocol.ClientFactory()
factory.protocol = ClientProtocol
factory.server = self
reactor.connectTCP('gameserver16.gamesnet.it', 2593, factory)
def dataReceived(self, data):
if (self.client != None):
self.client.write(data)
else:
self.buffer = data
def write(self, data):
self.transport.write(data)
print 'Server: ' + data.encode('hex')
class ClientProtocol(protocol.Protocol):
def connectionMade(self):
self.factory.server.client = self
self.write(self.factory.server.buffer)
self.factory.server.buffer = ''
def dataReceived(self, data):
self.factory.server.write(data)
def write(self, data):
self.transport.write(data)
print 'Client: ' + data.encode('hex')
def main():
import sys
from twisted.python import log
log.startLogging(sys.stdout)
factory = protocol.ServerFactory()
factory.protocol = ServerProtocol
reactor.listenTCP(2593, factory)
reactor.run()
if __name__ == '__main__':
main()
Consider this approach
#!/usr/bin/env python
import sys
from twisted.internet import reactor
from twisted.internet.protocol import ServerFactory, ClientFactory, Protocol
from twisted.protocols import basic
from twisted.python import log
LISTEN_PORT = 2593
SERVER_PORT = 1234
class ServerProtocol(Protocol):
def connectionMade(self):
reactor.connectTCP('localhost', SERVER_PORT, MyClientFactory(self))
def dataReceived(self, data):
self.clientProtocol.transport.write(data)
class ClientProtocol(Protocol):
def connectionMade(self):
# Pass ServerProtocol a ref. to ClientProtocol
self.serverProtocol.clientProtocol = self;
def dataReceived(self, data):
self.serverProtocol.transport.write(data)
class MyServerFactory(ServerFactory):
protocol = ServerProtocol
def buildProtocol(self, addr):
# Create ServerProtocol
p = ServerFactory.buildProtocol(self, addr)
return p
class MyClientFactory(ClientFactory):
protocol = ClientProtocol
def __init__(self, serverProtocol_):
self.serverProtocol = serverProtocol_
def buildProtocol(self, addr):
# Create ClientProtocol
p = ClientFactory.buildProtocol(self,addr)
# Pass ClientProtocol a ref. to ServerProtocol
p.serverProtocol = self.serverProtocol
return p
def main():
log.startLogging(sys.stdout)
reactor.listenTCP(LISTEN_PORT, MyServerFactory())
reactor.run()
if __name__ == '__main__':
main()
The ServerProtcol instance, passes a reference of itself to the MyClientFactory constructor, which then sets tells the ClientProtcol what ServerProtocol instance it's associated with.
Similarly, when the ClientProtocol connection is established, it uses it's reference to the ServerProtocol to tell the ServerProtocol what ClientProtocol to use.
Note: There's no error checking in this code, so you may encounter errors regarding NoneType if things go wrong (for example, if the real server isn't listening).
The important lines are:
reactor.connectTCP('localhost', SERVER_PORT, MyClientFactory(self))
#...
def __init__(self, serverProtocol_):
self.serverProtocol = serverProtocol_
Here, you pass a reference to the ServerProtocol to the MyClientFactory constructor. It stores this reference in a member variable. You do this so that that when the client factory creates a ClientProtocol, it can pass the reference on:
# Pass ClientProtocol a ref. to ServerProtocol
p.serverProtocol = self.serverProtocol
Then, once the connection is made from your script to the real server, the reverse happens. The ClientProtocol gives the ServerProtocol a reference to itself:
# Pass ServerProtocol a ref. to ClientProtocol
self.serverProtocol.clientProtocol = self;
Finally, both protocols use the stored references of each other to send data when it is received:
def dataReceived(self, data):
self.clientProtocol.transport.write(data)
#...
def dataReceived(self, data):
self.serverProtocol.transport.write(data)
Related
I'm trying to simulate a situation where data is received by a server periodically. In my set up, I run one process that sets up the server and another process that sets up a bunch of clients (suffices to think of a single client). I have set up some of the code by putting together bits and pieces mostly from here. The server/clients communicate by sending messages using transport.write. First, the server tells the clients to start (this works fine AFAIK). The clients report back to the server as they make progress. What has me confused is that I only get these intermittent messages at the very end when the client is done. This could be a problem with buffer flush and I tried (unsuccessfully) things like This. Also, each message is pretty large and I tried sending the same message multiple times so the buffer would get cleared.
I suspect what I am seeing is a problem with returning the control to the transport but I can't figure out how to do away with it.
Any help with this or any other issues that jump up to you is much appreciated.
Server:
from twisted.internet import reactor, protocol
import time
import serverSideAnalysis
import pdb
#import bson, json, msgpack
import _pickle as pickle # I expect the users to authenticate and not
# do anything malicious.
PORT = 9000
NUM = 1
local_scratch="/local/scratch"
class Hub(protocol.Protocol):
def __init__(self,factory, clients, nclients):
self.clients = clients
self.nclients = nclients
self.factory = factory
self.dispatcher = serverSideAnalysis.ServerTalker(NUM, self,
local_scratch)
def connectionMade(self):
print("connected to user" , (self))
if len(self.clients) < self.nclients:
self.factory.clients.append(self)
else:
self.factory.clients[self.nclients] = self
if len(self.clients) == NUM:
val = input("Looks like everyone is here, shall we start? (Y/N)")
while (val.upper() != "Y"):
time.sleep(20)
val = input("Looks like everyone is here, shall we start??? (Y/N)")
message = pickle.dumps({"TASK": "INIT", "SUBTASK":"STORE"})
self.message(message) # This reaches the client as I had expected
def message(self, command):
for c in self.factory.clients:
c.transport.write(command)
def connectionLost(self, reason):
self.factory.clients.remove(self)
self.nclients -= 1
def dataReceived(self, data):
if len(self.clients) == NUM:
self.dispatcher.dispatch(data)
class PauseTransport(protocol.Protocol):
def makeConnection(self, transport):
transport.pauseProducing()
class HubFactory(protocol.Factory):
def __init__(self, num):
self.clients = set([])
self.nclients = 0
self.totConnections = num
def buildProtocol(self, addr):
print(self.nclients)
if self.nclients < self.totConnections:
self.nclients += 1
return Hub(self, self.clients, self.nclients)
protocol = PauseTransport()
protocol.factory = self
return protocol
factory = HubFactory(NUM)
reactor.listenTCP(PORT, factory)
factory.clients = []
reactor.run()
Client:
from twisted.internet import reactor, protocol
import time
import clientSideAnalysis
import sys
HOST = 'localhost'
PORT = 9000
local_scratch="/local/scratch"
class MyClient(protocol.Protocol):
def connectionMade(self):
print("connected!")
self.factory.clients.append(self)
print ("clients are ", self.factory.clients)
self.cdispatcher = clientSideAnalysis.ServerTalker(analysis_file_name, local_scratch, self)
def clientConnectionLost(self, reason):
#TODO send warning
self.factory.clients.remove(self)
def dataReceived(self, data): #This is the problematic part I think
self.cdispatcher.dispatch(data)
print("1 sent")
time.sleep(10)
self.cdispatcher.dispatch(data)
print("2 sent")
time.sleep(10)
self.cdispatcher.dispatch(data)
time.sleep(10)
def message(self, data):
self.transport.write(data)
class MyClientFactory(protocol.ClientFactory):
protocol = MyClient
if __name__=="__main__":
analysis_file_name = sys.argv[1]
factory = MyClientFactory()
reactor.connectTCP(HOST, PORT, factory)
factory.clients = []
reactor.run()
The last bit of relevant information about what the dispatchers do.
In both cases, they load the message that has arrived (a dictionary) and do a few computations based on the content. Every once in a while, they use the message method to communicate with thier current values.
Finally, I'm using python 3.6. and twisted 18.9.0
The way you return control to the reactor from a Protocol.dataReceived method is you return from that method. For example:
def dataReceived(self, data):
self.cdispatcher.dispatch(data)
print("1 sent")
If you want more work to happen after this, you have some options. If you want the work to happen after some amount of time has passed, use reactor.callLater. If you want the work to happen after it is dispatched to another thread, use twisted.internet.threads.deferToThread. If you want the work to happen in response to some other event (for example, data being received), put it in the callback that handles that event (for example, dataReceived).
Here is my simple twisted server code:
from twisted.internet.protocol import Protocol,Factory
from twisted.internet import reactor
class MultiEcho(Protocol):
def __init__(self, factory):
self.recvBuf = ""
self.factory = factory
def connectionMade(self):
self.factory.echoers.append(self)
print self.factory.echoers
def dataReceived(self, data):
self.recvBuf += data
self.transport.write(data)
def connectionLost(self, reason):
self.factory.echoers.remove(self)
class MultiEchoFactory(Factory):
def __init__(self):
self.echoers=[]
def buildProtocol(self, addr):
return MultiEcho(self)
if __name__ == '__main__':
reactor.listenTCP(4321, MultiEchoFactory())
reactor.run()
Use 'telnet 127.0.0.1 4321' ,will print a list:
[<__main__.MultiEcho instance at 0x0000000002EA6B08>]
add another 'telnet 127.0.0.1 4321',the list is:
[<__main__.MultiEcho instance at 0x0000000002EA6B08>, <__main__.MultiEcho instance at 0x0000000002EA6EC8>]
I know a client is a protocol object's instance ,and the list is MultiEchoFactory.echoers,so I want to keep the list,
import them in another .py file use list[n].transport.write(somedata).
I try from myserver import MultiEchoFactory,but the MultiEchoFactory.echoers is empty.
I also rewrite the code
class MultiEchoFactory(Factory):
def __init__(self):
self.echoers=[]
to
class MultiEchoFactory(Factory):
echoers=[]
But I still can't use the protocol list in another .py file.
So,what should I do?
You need to save it in buildProtocol(), i.e.:
class MultiEchoFactory(Factory):
echoers=[]
def buildProtocol(self, addr):
echoer = MultiEcho(self)
MultiEchoFactory.echoers.append(echoer)
return echoer
CLIENT:
#!/usr/bin/env python
from twisted.internet import reactor, protocol
class EchoClient(protocol.Protocol):
def __init__(self, arg):
self.arg = arg
def connectionMade(self):
self.transport.write("hello, world!")
def dataReceived(self, data):
print "Server said:", data
self.transport.loseConnection()
def connectionLost(self, reason):
print "connection lost"
class EchoFactory(protocol.ClientFactory):
protocol = EchoClient
def buildProtocol(self, address):
proto = protocol.ClientFactory.buildProtocol(self, address, 12)
self.connectedProtocol = proto
return proto
def clientConnectionFailed(self, connector, reason):
print "Connection failed - goodbye!"
reactor.stop()
def clientConnectionLost(self, connector, reason):
print "Connection lost - goodbye!"
reactor.stop()
def main():
f = EchoFactory()
reactor.connectTCP("localhost", 8000, f)
reactor.run()
if __name__ == '__main__':
main()
SERVER:
#!/usr/bin/env python
from twisted.internet import reactor, protocol
from twisted.application import service, internet
class Echo(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
def main():
factory = protocol.ServerFactory()
factory.protocol = Echo
reactor.listenTCP(8000,factory)
reactor.run()
if __name__ == '__main__':
main()
ERROR:
exceptions.TypeError: buildProtocol() takes exactly 2 arguments (3 given)
QUESTION:
How can I get the EchoClient class in the CLIENT to accept parameters and assign instance variables ( such as arg in the EchoClient constructor above)? As noted below, it was previously suggested that I override the buildProtocol function, but my attempt at doing so has lead me to the above error. I am not really sure where to go from here. I suppose my question can be generalize to: how can I add instance variables to a protocol?
you wrote:
def buildProtocol(self, address):
proto = protocol.ClientFactory.buildProtocol(self, address, 12)
that is, you are overriding ClientFactory.buildProtocol and calling the parent class with a different signature than it knows how to handle.
Passing data from the factory to the client is only a little tricky. You can provide any __init__ you want to the factory, but twisted creates instances of IProtocol itself. Fortunately, most factories assign themselves to the factory attribute of the protocol, once it's ready to go:
class MyClientProtocol(protocol.Protocol):
def connectionMade(self):
# use self.factory here:
self.transport.write(self.factory.arg)
class MyClientFactory(protocol.ClientFactory):
protocol = MyClientProtocol
def __init__(self, arg):
self.arg = arg
In fact, the whole ProtocolFactory business is to support this kind of use; but be mindful; many instances of Protocol will share a single instance of their factory; use the factory for configuration but manage state in the protocol.
It's certainly possible that the way the standard family of Protocol/Factory implementations don't suit your needs, and that's also reasonable, so long as you fully implement the IProtocol and IProtocolFactory interfaces. The base classes exist because they handle most of the cases for you, not because they are the only possible implementation.
It's not clear from your question what exactly your tryed and what exactly the error was, but anyway you have to do two steps:
Make EchoClient's constructor take whatever arguments you need it to take and initialise whatever field you need it to initialise.
Override buildProtocol method in your factory to supply those arguments to your protocol.
I'm fairly new with both Python and Twisted so I may just not be understanding things properly but I seem to be stuck at a point where I need help.
What I want to do is use ReconnectingClientFactory on an SSL connection. I have it all running, but if the connection is dropped all data sent to the transport's write() method is simply dropped without any error. The actual method called is twisted.protocols.tls.TLSMemoryBIOProtocol.write().
Here's what I think is happening (starting from a working connection):
connection is lost
call to write() method (source code here) with some data
self.disconnecting is False so data passes to _write() method
_write method gets to _lostTLSConnection which is True and then just runs return
connection is regained but no data sent because it's not buffered any where
Here's the a reduced version of the client:
from OpenSSL import SSL
from twisted.internet.protocol import (Protocol, ReconnectingClientFactory)
from twisted.internet import (reactor, ssl)
import struct
class MetricsServer(Protocol):
streambuffer = bytearray()
def connectionMade(self):
self.transport.setTcpKeepAlive(True) # maintain the TCP connection
self.transport.setTcpNoDelay(False) # allow Nagle algorithm
print("connected to server")
def dataReceived(self, data):
print("from server:", data)
def connectionLost(self, reason):
self.connected = 0
print("server connection lost:", reason)
class MetricsServerFactory(ReconnectingClientFactory):
protocol = MetricsServer
maxDelay = 300 # maximum seconds between retries
factor = 1.6180339887498948
packet_sequence_number = 0
active_connection = None
def buildProtocol(self, addr):
self.resetDelay()
if self.active_connection == None:
self.active_connection = self.protocol()
return self.active_connection
def get_packet_sequence_number(self):
self.packet_sequence_number += 1
return self.packet_sequence_number
def send_data(self):
print ("sending ssl packet")
packet = struct.pack("!I", self.get_packet_sequence_number())
self.active_connection.transport.write(packet)
reactor.callLater(1.0, metrics_server.send_data)
class CtxFactory(ssl.ClientContextFactory):
def getContext(self):
self.method = SSL.TLSv1_METHOD
ctx = ssl.ClientContextFactory.getContext(self)
ctx.use_certificate_file('keys/client.crt')
ctx.use_privatekey_file('keys/client.key')
def verifyCallback(connection, x509, errnum, errdepth, ok):
return bool(ok)
ctx.set_verify(SSL.VERIFY_PEER, verifyCallback)
ctx.load_verify_locations("keys/ca.pem")
return ctx
if __name__ == "__main__":
metrics_server = MetricsServerFactory()
reactor.connectSSL('localhost', 8000, metrics_server, CtxFactory())
reactor.callLater(3.0, metrics_server.send_data)
reactor.run()
And here's a simple server that outputs the data it receives:
from OpenSSL import SSL
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
sent_back_data = False
def dataReceived(self, data):
print(' '.join("{0:02x}".format(x) for x in data))
def verifyCallback(connection, x509, errnum, errdepth, ok):
return bool(ok)
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
myContextFactory = ssl.DefaultOpenSSLContextFactory(
'keys/server.key', 'keys/server.crt'
)
ctx = myContextFactory.getContext()
ctx.set_verify(
SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
verifyCallback
)
ctx.load_verify_locations("keys/ca.pem")
reactor.listenSSL(8000, factory, myContextFactory)
reactor.run()
Process to recreate issue:
first you need to generate your own certs and CA for this to work
run the server first
run the client code
wait for some output on the server end and then end the program
notice the client continues to try and send data
restart the server end
notice the server end will continue receiving packets, but packets sent when the connection was lost are simply dropped
As a work-around I tried implementing my own buffer to send the data on re-connection, but ran into another issue. I want it to send the data when the connection is re-established and the only hook I can see is Protocol.connectionMade(). However, that method is called before the TLS handshaking is actually done so it ends up being caught by an exception handler in _write() and placed into another buffer to be sent later. But, that buffer only seems to be sent if data is received from the other end (which doesn't occur very often in my case, and would also mean that the data could arrive at the other end in the wrong order because a write() may be called before data is received). I also think another disconnection before data is received will also cause that buffer of data to just be wiped.
EDIT: added sample code for the first issue. It's probably odd that I have that active_connection in the Factory, but I am trying to make it work as a singleton.
Okay, I figured out the problem with my workaround... I was passing a bytearray to be written to the transport and then immediately cleared it afterwards not realizing that the write was being deferred until after I cleared the buffer. So, I pass a copy of the bytearray and it seems to be working now.
It still doesn't seem quite right that every call to the write has to be proceeded by a check to see if it's connected as the whole idea of the ReconnectingClientFactory is that it handles maintaining the connection for you. Also, I think it's possible to lose a connection between that if statement and when the write() is actually run, so it's still possible to lose data.
from OpenSSL import SSL
from twisted.internet.protocol import (Protocol, ReconnectingClientFactory)
from twisted.internet import (reactor, ssl)
import struct
class MetricsServer(Protocol):
streambuffer = bytearray()
def connectionMade(self):
self.transport.setTcpKeepAlive(True) # maintain the TCP connection
self.transport.setTcpNoDelay(False) # allow Nagle algorithm
print("connected to server")
if len(self.transport.factory.wrappedFactory.send_buffer) > 0:
self.transport.write(bytes(self.transport.factory.wrappedFactory.send_buffer))
self.transport.factory.wrappedFactory.send_buffer.clear()
def dataReceived(self, data):
print("from server:", data)
def connectionLost(self, reason):
self.connected = 0
print("server connection lost:", reason)
class MetricsServerFactory(ReconnectingClientFactory):
protocol = MetricsServer
maxDelay = 300 # maximum seconds between retries
factor = 1.6180339887498948
packet_sequence_number = 0
active_connection = None
send_buffer = bytearray()
def buildProtocol(self, addr):
self.resetDelay()
if self.active_connection == None:
self.active_connection = self.protocol()
return self.active_connection
def get_packet_sequence_number(self):
self.packet_sequence_number += 1
return self.packet_sequence_number
def send_data(self):
print ("sending ssl packet")
packet = struct.pack("!I", self.get_packet_sequence_number())
if self.active_connection and self.active_connection.connected:
self.active_connection.transport.write(packet)
else:
self.send_buffer.extend(packet)
reactor.callLater(1.0, metrics_server.send_data)
class CtxFactory(ssl.ClientContextFactory):
def getContext(self):
self.method = SSL.TLSv1_METHOD
ctx = ssl.ClientContextFactory.getContext(self)
ctx.use_certificate_file('keys/client.crt')
ctx.use_privatekey_file('keys/client.key')
def verifyCallback(connection, x509, errnum, errdepth, ok):
return bool(ok)
ctx.set_verify(SSL.VERIFY_PEER, verifyCallback)
ctx.load_verify_locations("keys/ca.pem")
return ctx
if __name__ == "__main__":
metrics_server = MetricsServerFactory()
reactor.connectSSL('localhost', 8000, metrics_server, CtxFactory())
reactor.callLater(3.0, metrics_server.send_data)
reactor.run()
I have a server application written in python using twisted and I'd like to know how to kill instances of my protocol (bottalk). Everytime I get a new client connection, I see the instance in memory (print Factory.clients) .. but let's say I want to kill one of those instances from the server side (drop a specific client connection)? Is this possible? I've tried looking for a phrase using lineReceived, then if it matches, self.transport.loseConnection(). But that doesn't seem to reference the instance anymore or something..
class bottalk(LineReceiver):
from os import linesep as delimiter
def connectionMade(self):
Factory.clients.append(self)
print Factory.clients
def lineReceived(self, line):
for bots in Factory.clients[1:]:
bots.message(line)
if line == "killme":
self.transport.loseConnection()
def message(self, message):
self.transport.write(message + '\n')
class botfactory(Factory):
def buildProtocol(self, addr):
return bottalk()
Factory.clients = []
stdio.StandardIO(bottalk())
reactor.listenTCP(8123, botfactory())
reactor.run()
You closed the TCP connection by calling loseConnection. But there's no code anywhere in your application that removes items from the clients list on the factory.
Try adding this to your protocol:
def connectionLost(self, reason):
Factory.clients.remove(self)
This will remove the protocol instance from the clients list when the protocol's connection is lost.
Also, you should consider not using the global Factory.clients to implement this functionality. It's bad for all the usual reasons globals are bad. Instead, give each protocol instance a reference to its factory and use that:
class botfactory(Factory):
def buildProtocol(self, addr):
protocol = bottalk()
protocol.factory = self
return protocol
factory = botfactory()
factory.clients = []
StandardIO(factory.buildProtocol(None))
reactor.listenTCP(8123, factory)
Now each bottalk instance can use self.factory.clients instead of Factory.clients.