I have a simple twisted TCP server that I am using to send and receive data from another application. I am able to communicate and send messages to this application once a connection has been established.
What I want to be able to do have an external module command when and what to send to the 3rd party application (instead of when connected). For example, here is my server code:
from twisted.internet import reactor, protocol
# Twisted
class Tcp(protocol.Protocol):
def dataReceived(self, data):
self.transport.write(data)
print 'data received: ', data
# do something with data
def connectionMade(self, msg):
# Want to only transport message when I command not immediately when connected
self.transport.write(msg)
class TcpFactory(protocol.Factory):
def buildProtocol(self, addr):
return Tcp()
reactor.listenTCP(8001, TcpFactory())
reactor.run()
I have another module (operated by a GUI interface) which defines what message to send and this message needs to be passed as 'msg' into connectionMade
What is the best way to do this?
Your 3rd party call will happen from within connectionMade and should not call it directly. The connectionMade is a Twisted callback and it's there to signal that a client is on the other side. It's up to you to dictate what happens after that. So you can do something like make a series of Deferreds/callbacks which take self.transport (this holds relevant client data). In those series of callbacks, you'll have your logic for
communicating with the 3rd party
validating client
sending a message to the client
This is the gist of it:
def connectionMade(self):
d = Deferred()
d.addCallback(self.thirdPartyComm).addErrBack(self.handleError)
d.addCallback(sendMsg)
d.callback(self.transport)
def thirdPartyComm(self, transport):
# logic for talking to 3rd party goes here
# 3rd party should validate the client then provide a msg
# or raise exception if client is invalid
return msg
def handleError(self, error):
# something went wrong, kill client connection
self.transport.loseConnection()
def sendMsg(self, msg):
self.transport.write(msg)
Related
I've a server as follow:-
class MyServerProtocol(WebSocketServerProtocol):
def onConnect(self, request):
print("Client connecting: {0}".format(request.peer))
def onOpen(self):
print("WebSocket connection open.")
def onMessage(self, payload, isBinary):
self.sendMessage(payload, isBinary)
def onClose(self, wasClean, code, reason):
print("WebSocket connection closed: {0}".format(reason))
The above code receives the message and replies back with the same message. However, my use case is different.
I'm processing some data, storing it into the DB and then finally want to send the data to the connected clients, or one type of data to specific set of connected clients and the other type to another.
How can I do it? How can I call sendMessage function outside the class?
You need to subclass WebSocketServerFactory to keep track of your connections.
Take a look at this example, which stores all the open connections in a list and broadcasts to all of them. You could easily adapt it to send specific messages to certain clients.
Are you sure you don't want to just use WAMP instead of raw WebSockets?
This might be a lot easier for your use-case, as "broadcast information to interested clients" is exactly what the Publish/Subscribe pattern is for.
I'm attempting to do the following:
connect as client to an existing websocket
process the streaming data received from this socket, and publish it on another websocket
I'm using twisted and autobahn to do so. I have managed to have the two parts working separately, by deriving a WebSocketClientProtocol for the client, and deriving an ApplicationSession in the second. The two run with the same reactor.
I am not sure however as to how to make them communicate. I would like to send a message on my server when the client receives a message, but I don't know how to get the running instance of the WebSocketClientProtocol...
Perhaps this isn't the right approach to do this either. What's the right way to do this?
I've been trying to solve similiar problem recently, here's what worked:
f = XLeagueBotFactory()
app = Application(f)
reactor.connectTCP("irc.gamesurge.net", 6667, f)
reactor.listenTCP(port, app, interface=host)
^ This is in if __name__ == "__main__":
class Application(web.Application):
def __init__(self, botfactory):
self.botfactory = botfactory
Define the instance as self, then in my instance I was sending it to another handler for http post request (using cyclone)
class requestvouch(web.RequestHandler):
def __init__(self, application, request, **kwargs):
super(requestvouch, self).__init__(application, request, **kwargs)
self.botfactory = application.botfactory
def msg(self, channel, msg):
bot = self.botfactory.getProtocolByName("XLeagueBot")
sendmsg(bot, channel, msg) # function that processed the msg through stuff like encoding and logging and then sent it to bot.msg() function that posts it to IRC (endpoint in my case)
def post(self):
msg = "What I'm sending to the protocol of the other thing"
self.msg("#xleague", msg)
Now the important part comes in factory
class XLeagueBotFactory(protocol.ClientFactory):
protocol = XLeagueBot
def __init__(self):
self.protocols = {}
def getProtocolByName(self, name):
return self.protocols.get(name)
def registerProtocol(self, protocol):
self.protocols[protocol.nickname] = protocol
def unregisterProtocol(self, protocol):
del self.protocols[protocol.nickname]
Finally in my client class:
class XLeagueBot(irc.IRCClient):
nickname = "XLeagueBot"
def connectionMade(self):
irc.IRCClient.connectionMade(self)
self.factory.registerProtocol(self)
def connectionLost(self, reason):
self.factory.unregisterProtocol(self)
irc.IRCClient.connectionLost(self, reason)
I'm not entirely sure that this code is perfect, or bugfree, but it should +- tell you how to deal with calling instance of protocol class. The problem afaik comes from name of instance protocol being generated inside of it's factory and not being sent elsewhere.
I started working with Twisted Framework, I wrote a TCP server and I connect to it throw Telnet, it works fine. Now I want to manage connections and connected clients( sending data, cutting connections, etc etc) using an GUI like PyUI or GTK..
this is my code
import sys
import os
from twisted.internet import reactor, protocol
from twisted.python import log
class Server(protocol.Protocol):
def dataReceived(self, data):
log.msg ("data received: %s"%data)
self.transport.write("you sent: %s"%data)
def connectionMade(self):
self.client_host = self.transport.getPeer().host
self.client_port = self.transport.getPeer().port
if len(self.factory.clients) >= self.factory.clients_max:
log.msg("Too many connections !!")
self.transport.write("Too many connections, sorry\n")
self.transport.loseConnection()
else:
self.factory.clients.append((self.client_host,self.client_port))
log.msg("connection from %s:%s\n"%(self.client_host,str(self.client_port)))
self.transport.write(
"Welcome %s:%s\n" %(self.client_host,str(self.client_port)))
def connectionLost(self, reason):
log.msg('Connection lost from %s:%s. Reason: %s\n' % (self.client_host,str(self.client_port),reason.getErrorMessage()))
if (self.client_host,self.client_port) in self.factory.clients:
self.factory.clients.remove((self.client_host,self.client_port))
class MyFactory(protocol.ServerFactory):
protocol = Server
def __init__(self, clients_max=10):
self.clients_max = clients_max
self.clients = []
def main():
"""This runs the protocol on port 8000"""
log.startLogging(sys.stdout)
reactor.listenTCP(8000,MyFactory)
reactor.run()
if __name__ == '__main__':
main()
Thanks.
If you want to write a single Python program (process) that runs both your UI and your networking, you will first need to choose an appropriate Twisted reactor that integrates with the UI toolkit's event loop. See here.
Next, you might start with something simple, like have a button that when pressed will send a text message to all currently connected clients.
Another thing: what clients will connect? Browsers (also)? If so, you might contemplate about using WebSocket instead of raw TCP.
I have an application where every websocket connection (within tornado open callback) creates a zmq.SUB socket to an existing zmq.FORWARDER device. Idea is to receive data from zmq as callbacks, which can then be relayed to frontend clients over websocket connection.
https://gist.github.com/abhinavsingh/6378134
ws.py
import zmq
from zmq.eventloop import ioloop
from zmq.eventloop.zmqstream import ZMQStream
ioloop.install()
from tornado.websocket import WebSocketHandler
from tornado.web import Application
from tornado.ioloop import IOLoop
ioloop = IOLoop.instance()
class ZMQPubSub(object):
def __init__(self, callback):
self.callback = callback
def connect(self):
self.context = zmq.Context()
self.socket = self.context.socket(zmq.SUB)
self.socket.connect('tcp://127.0.0.1:5560')
self.stream = ZMQStream(self.socket)
self.stream.on_recv(self.callback)
def subscribe(self, channel_id):
self.socket.setsockopt(zmq.SUBSCRIBE, channel_id)
class MyWebSocket(WebSocketHandler):
def open(self):
self.pubsub = ZMQPubSub(self.on_data)
self.pubsub.connect()
self.pubsub.subscribe("session_id")
print 'ws opened'
def on_message(self, message):
print message
def on_close(self):
print 'ws closed'
def on_data(self, data):
print data
def main():
application = Application([(r'/channel', MyWebSocket)])
application.listen(10001)
print 'starting ws on port 10001'
ioloop.start()
if __name__ == '__main__':
main()
forwarder.py
import zmq
def main():
try:
context = zmq.Context(1)
frontend = context.socket(zmq.SUB)
frontend.bind('tcp://*:5559')
frontend.setsockopt(zmq.SUBSCRIBE, '')
backend = context.socket(zmq.PUB)
backend.bind('tcp://*:5560')
print 'starting zmq forwarder'
zmq.device(zmq.FORWARDER, frontend, backend)
except KeyboardInterrupt:
pass
except Exception as e:
logger.exception(e)
finally:
frontend.close()
backend.close()
context.term()
if __name__ == '__main__':
main()
publish.py
import zmq
if __name__ == '__main__':
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.connect('tcp://127.0.0.1:5559')
socket.send('session_id helloworld')
print 'sent data for channel session_id'
However, my ZMQPubSub class doesn't seem like is receiving any data at all.
I further experimented and realized that I need to call ioloop.IOLoop.instance().start() after registering on_recv callback within ZMQPubSub. But, that will just block the execution.
I also tried passing main.ioloop instance to ZMQStream constructor but doesn't help either.
Is there a way by which I can bind ZMQStream to existing main.ioloop instance without blocking flow within MyWebSocket.open?
In your now complete example, simply change frontend in your forwarder to a PULL socket and your publisher socket to PUSH, and it should behave as you expect.
The general principles of socket choice that are relevant here:
use PUB/SUB when you want to send a message to everyone who is ready to receive it (may be no one)
use PUSH/PULL when you want to send a message to exactly one peer, waiting for them to be ready
it may appear initially that you just want PUB-SUB, but once you start looking at each socket pair, you realize that they are very different. The frontend-websocket connection is definitely PUB-SUB - you may have zero-to-many receivers, and you just want to send messages to everyone who happens to be available when a message comes through. But the backend side is different - there is only one receiver, and it definitely wants every message from the publishers.
So there you have it - backend should be PULL and frontend PUB. All your sockets:
PUSH -> [PULL-PUB] -> SUB
publisher.py: socket is PUSH, connected to backend in device.py
forwarder.py: backend is PULL, frontend is PUB
ws.py: SUB connects and subscribes to forwarder.frontend.
The relevant behavior that makes PUB/SUB fail on the backend in your case is the slow joiner syndrome, which is described in The Guide. Essentially, subscribers take a finite time to tell publishers about there subscriptions, so if you send a message immediately after opening a PUB socket, the odds are it hasn't been told that it has any subscribers yet, so it's just discarding messages.
ZeroMq subscribers have to subscribe on what messages they wish to receive; I don't see that in your code. I believe the Python way is this:
self.socket.setsockopt(zmq.SUBSCRIBE, "")
I've decided to dip my toe into the world of asynchronous python with the help of twisted. I've implemented some of the examples from the documentation, but I'm having a difficult time finding an example of the, very simple, client I'm trying to write.
In short I'd like a client which establishes a tcp connection with a server and then sends simple "\n" terminated string messages off of a queue object to the server. The server doesn't ever respond with any messages so my client is fully unidirectional. I /think/ that what I want is some combination of this example and the twisted.internet.protocols.basic.LineReceiver convenience protocol. This feels like it should be just about the simplest thing one could do in twisted, but none of the documentation or examples I've seen online seem to fit quite right.
What I have done is not used a Queue but I am illustrating the code that sends a line, once a connection is made. There are bunch of print stuff that will help you understand on what is going on.
Usual import stuff:
from twisted.web import proxy
from twisted.internet import reactor
from twisted.internet import protocol
from twisted.internet.protocol import ReconnectingClientFactory
from twisted.protocols import basic
from twisted.python import log
import sys
log.startLogging(sys.stdout)
You create a protocol derived from line receiver, set the delimiter.
In this case, I simply write a string "www" once the connection is made.
The key thing is to look at protocol interface at twisted.internet.interface.py and understand the various methods of protocol and what they do and when they are called.
class MyProtocol(basic.LineReceiver):
#def makeConnection(self, transport):
# print transport
def connectionLost(self, reason):
print reason
self.sendData = False
def connectionMade(self):
print "connection made"
self.delimiter = "\n"
self.sendData = True
print self.transport
self.sendFromQueue()
def sendFromQueue(self):
while self.sendData:
msg = dataQueue.get()
self.sendLine(msg)
# you need to handle empty queue
# Have another function to resume
Finally, A protocol factory that will create a protocol instance for every connection.
Look at method : buildProtcol.
class myProtocolFactory():
protocol = MyProtocol
def doStart(self):
pass
def startedConnecting(self, connectorInstance):
print connectorInstance
def buildProtocol(self, address):
print address
return self.protocol()
def clientConnectionLost(self, connection, reason):
print reason
print connection
def clientConnectionFailed(self, connection, reason):
print connection
print reason
def doStop(self):
pass
Now you use a connector to make a connection:
reactor.connectTCP('localhost', 50000, myProtocolFactory())
reactor.run()
I ran this and connected it to an server that simply prints what it receives and hence send no ack back. Here is the output:
1286906080.08 82 INFO 140735087148064 __main__ conn_made: client_address=127.0.0.1:50277
1286906080.08 83 DEBUG 140735087148064 __main__ created handler; waiting for loop
1286906080.08 83 DEBUG 140735087148064 __main__ handle_read
1286906080.08 83 DEBUG 140735087148064 __main__ after recv
'www\n'
Recieved: 4
The above example if not fault tolerant. To reconnect , when a connection is lost, you can derive your protocol factory from an existing twisted class - ReconnectingClientFactory.
Twisted has almost all the tools that you would need :)
class myProtocolFactory(ReconnectingClientFactory):
protocol = MyProtocol
def buildProtocol(self, address):
print address
return self.protocol()
For further reference
I suggest that you read : http://krondo.com/?page_id=1327
[Edited: As per comment below]