We Keep Coding

What is the best way of handling received messages in Twisted for Python?

I am a noob at Python and seeking some assistance with architecture. Here is my setup: I have a legacy client application written in LiveCode that runs in multiple locations to display synchronized information based on what the server demands. Think of this as a kiosk. This client piece is not going anywhere.
The server application is what I’m rewriting in Python. My goal is to have the server application running constantly, listening for client socket connections, and sending/receiving data to/from these clients. I have successfully passed messages between this LiveCode client application and a python script that uses Twisted for the socket handling, so now I need to start processing those messages. The code looks something like this:
from twisted.internet.protocol import Factory
from twisted.protocols.basic import LineReceiver
from twisted.internet import reactor
class MessageListener(LineReceiver):
def __init__(self, users):
self.users = users
self.name = None
def connectionMade(self):
d = self.transport.getHost()
print("Connection established with {}:{}".format(d.host, d.port))
def connectionLost(self, reason):
print("Connection lost: {}".format(reason))
if self.name in self.users:
del self.users[self.name]
def dataReceived(self, line):
d = self.transport.getHost()
print("Received message from {}:{}...{}".format(d.host, d.port, line))
self.handle_GOTDATA(line)
def handle_GOTDATA(self, msg):
#convert "msg" to string and parse it into the necessary chunks
#*****Go do something based on the requestor and the command*****
#Use if-elif or dictionary to determine which function to run
#based on what the string tells us.
#Should these functions be defined here or in a separate class?
class MessageListenerFactory(Factory):
def __init__(self):
self.users = {} # maps user names to Chat instances
def buildProtocol(self, addr):
return MessageListener(self.users)
reactor.listenTCP(50010, MessageListenerFactory())
reactor.run()
A couple of questions:
The handle_GOTDATA() function is where I will take the received message, parse it out into the chunks that tell me what to do with the data, then call a different function depending on what needs to be done with that data.
Do I just define all 20 of these functions in this same “MessageListener” class, or do I write a separate class to keep all of these functions? I might get 10 messages at about the same time, and they may need to call the same function, so I wasn’t sure the best architecture approach here.
I also want to build a GUI to interact with the server for some troubleshooting and monitoring on occasion. I’m familiar with Tkinter and it would be fine for this, and I can write the GUI in a separate file and have it connect to the server over a socket as well. But would I use the same socket listener implemented above and just pass it similar messages? Or should I build a separate class and factory to listen for the GUI connections?

If you're going to use LineReceiver, you should not override dataReceived. Instead, override lineReceived. If your protocol isn't line-oriented, you probably don't want to use LineReceiver. Also you may want to consider using the Tubes interface in either case.
Do I just define all 20 of these functions in this same “MessageListener” class, or do I write a separate class to keep all of these functions?
You should probably put them on a different class. If you put them on MessageListener then you will have more difficulty testing, refactoring, and maintaining the code because your protocol logic is tightly coupled to your application logic.
Define an explicit interface that MessageListener uses to dispatch high-level events representing network actions. Then implement that interface appropriately for your particular application. Later, you can implement it differently for another application. Or you can write test doubles. Or you can change your protocol without changing your application logic. Decoupling the two pieces gives you lots of extra flexibility compared to smashing them both into one class.
I might get 10 messages at about the same time, and they may need to call the same function, so I wasn’t sure the best architecture approach here.
I think this is orthogonal but if it's an important enough part of your protocol or application logic, you might want to consider building some kind of vectorization into the explicit interface I mentioned above. Instead of appObj.doOneThing(oneThing) perhaps you have appObj.doSeveralThings([oneThing, anotherThing]).
I also want to build a GUI to interact with the server for some troubleshooting and monitoring on occasion. I’m familiar with Tkinter and it would be fine for this, and I can write the GUI in a separate file and have it connect to the server over a socket as well. But would I use the same socket listener implemented above and just pass it similar messages? Or should I build a separate class and factory to listen for the GUI connections?
That depends on the interactions you want to perform, I suppose. If this is a GUI with extra privileges compared to the normal client, you need a protocol and server with authentication and authorization functionality or you must not use the same server and protocol because you risk giving clients those extra privileges.
If you just want to simulate a real client with a kind of debug-friendly interface that lets you easily interact with the server in ways that a client would normally interact with it, but with an interface that makes this easier for you, you can (and presumably should) connect to the same server.

Is there a library to interact via sockets with Python objects which are "running" on another machine?

So suppose on some server inside of some Python environment I have an object A. From some client I now want to interact with this object A transparently as if it were in the client environment. Like when from the client I call A.some_func(B,C), whatever library I'm using to do this Pickles up (A,"some_func",B,C), sends it to the server via Socket, the server runs the code, and sends the client back the result. My question is, does some library to do this or something similar exist?
Clearly there are/would be many limitations, including needing everything Picklable, probably unable to do this with many of the special methods, etc..., but I'd be interested in any sort of functionality at all.

select() function for simultaneously I/O

I am trying to implement a multi-threading server which can handle with simultaneously read/write from client.
The server method:
The client connects to the server, when each message starts with the name of the user they want to send the message to, followed by '|'. It looks something like that: "USER_NAME|DATA".
After receiving the data, the server knows by a dictionary of {socket:username} where to send the data. Everything works great, except the fact that the client can't handle with simultaneously reading and writing. I searched for a method to handle that and i found the select() function, but with a lack of examples- i couldn't integrate that function in my code.
therefore I have 2 questions:
Is the select() function should be on the server side? will it be more efficient?
Is someone can demonstrate with a simple example how the select() method should look in the client side?
Thanks in advance!!!

Though select() will work, you have to use threads if you want to do other things while the system is blocked on the select.
Have a look at glib's GIO library. There you can connect callbacks to the actions you want to monitor or act on, for example the 'connect's from clients.
Just open a socket, and use its file descriptor to hang a gio.add_watch on. Here's a mini-tutorial on using giochannels.

Making moves w/ websockets and python / django ( / twisted? )

The fun part of websockets is sending essentially unsolicited content from the server to the browser right?
Well, I'm using django-websocket by Gregor Müllegger. It's a really wonderful early crack at making websockets work in Django.
I have accomplished "hello world." The way this works is: when a request is a websocket, an object, websocket, is appended to the request object. Thus, I can, in the view interpreting the websocket, do something like:
request.websocket.send('We are the knights who say ni!')
That works fine. I get the message back in the browser like a charm.
But what if I want to do that without issuing a request from the browser at all?
OK, so first I save the websocket in the session dictionary:
request.session['websocket'] = request.websocket
Then, in a shell, I go and grab the session by session key. Sure enough, there's a websocket object in the session dictionary. Happy!
However, when I try to do:
>>> session.get_decoded()['websocket'].send('With a herring!')
I get:
Traceback (most recent call last):
File "<console>", line 1, in <module>
error: [Errno 9] Bad file descriptor
Sad. :-(
OK, so I don't know much of anything about sockets, but I know enough to sniff around in a debugger, and lo and behold, I see that the socket in my debugger (which is tied to the genuine websocket from the request) has fd=6, while the one that I grabbed from the session-saved websocket has fd=-1.
Can a socket-oriented person help me sort this stuff out?

I'm the author of django-websocket. I'm not a real expert in the topic of websockets and networking, however I think I have a decent understanding of whats going on. Sorry for going into great detail. Even if most of the answer isn't specific to your question it might help you at some other point. :-)
How websockets work
Let me explain shortly what a websocket is. A websocket starts as something that really looks like a plain HTTP request, established from the browser. It indicates through a HTTP header that it wants to "upgrade" the protocol to be a websocket instead of a HTTP request. If the server supports websockets, it agrees on the handshake and both - server and client - now know that they will use the established tcp socket formerly used for the HTTP request as a connection to interchange websocket messages.
Beside sending and waiting for messages, they have also of course the ability to close the connection at any time.
How django-websocket abuses the python's wsgi request environment to hijack the socket
Now lets get into the details of how django-websocket implements the "upgrading" of the HTTP request in a django request-response cylce.
Django usually uses the WSGI specification to talk to the webserver like apache or gunicorn etc. This specification was designed just with the very limited communication model of HTTP in mind. It assumes that it gets a HTTP request (only incoming data) and returns the response (only outgoing data). This makes it tricky to force django into the concept of a websocket where bidirectional communication is allowed.
What I'm doing in django-websocket to achieve this is that I dig very deeply into the internals of WSGI and django's request object to retrieve the underlaying socket. This tcp socket is then used to handle the upgrade the HTTP request to a websocket instance directly.
Now to your original question ...
I hope the above makes it obvious that when a websocket is established, there is no point in returning a HttpResponse. This is why you usually don't return anything in a view that is handled by django-websocket.
However I wanted to stick close to the concept of a view that holds the logic and returns data based on the input. This is why you should only use the code in your view to handle the websocket.
After you return from the view, the websocket is automatically closed. This is done for a reason: We don't want to keep the socket open for an undefined amount of time and relying on the client (the browser) to close it.
This is why you cannot access a websocket with django-websocket outside of your view. The file descriptor is then of course set to -1 indicating that its already closed.
Disclaimer
I explained above that I'm digging in the surrounding environment of django to get somehow -- in a very hackish way -- access to the underlaying socket. This is very fragile and also not supposed to work since WSGI is not designed for this! I also explained above that the websocket is closed after the view ends - however after the websocket closed down (AND closed the tcp socket), django's WSGI implementation tries to send a HTTP response - it doesn't know about websockets and thinks it is in a normal HTTP request-response cycle. But the socket is already closed an the sending will fail. This usually causes an exception in django.
This didn't affected my testings with the development server. The browser will never notice (you know .. the socket is already closed ;-) - but raising an unhandled error in every request is not a very good concept and may leak memory, doesn't handle database connection shutdown correctly and many athor things that will break at some point if you use django-websocket for more than experimenting.
This is why I would really advise you not to use websockets with django yet. It doesn't work by design. Django and especially WSGI would need a total overhaul to solve these problems (see this discussion for websockets and WSGI). Since then I would suggest using something like eventlet. Eventlet has a working websocket implementation (I borrowed some code from eventlet for the initial version of django-websocket) and since its just plain python code you can import your models and everything else from django. The only drawback is that you need a second webserver running just to handle websockets.

As Gregor Müllegger pointed out, Websockets can't be properly handled by WSGI, because that protocol never was designed to handle such a feature.
uWSGI, since version 1.9.11, can handle Websockets out of the box. Here uWSGI communicates with the application server using raw HTTP rather than the WSGI protocol. A server written that way, can therefore handle the protocol internals and keep the connection open over a long period. Having long living connections handled by a Django view is not a good idea either, because they then would block a worker thread, which is a limited resource.
The main purpose of Websockets, is to have the server push messages to the client in an asynchronous way. This can be a Django view triggered by other browsers (ex.: chat clients, multiplayer games), or an event triggered by, say django-celery (ex.: sport results). It therefore is fundamental for these Django services, to use a message queue for pushing messages to the client.
To handle this in a scalable way, I wrote django-websocket-redis, a Django module which can keep open all those long living Websocket connections in one single thread/process using Redis as the backend message queue.

You could give stargate a bash: http://boothead.github.com/stargate/ and http://pypi.python.org/pypi/stargate/.
It's built on top of pyramid and eventlet (I also contributed a fair bit of the websocket support and tests to eventlet). The big advantage of pyramid for this sort of thing is that it's got the concept of a resource which the url maps to, rather than just the result of a callable. So you end up with a graph of persistent resources that maps to your url structure and websocket connections are simply routed and connected to those resources.
So you end up only needing to do two things:
class YourView(WebSocketView):
def handler(self, websocket):
self.request.context.add_listener(websocket)
while True:
msg = websocket.wait()
# Do something with message
To receive messages
and
resource.send(some_other_message)
Here resource is an instance of a stargate.resource.WebSocketAwareContext (as is self.request.context) above and the send method sends the message to all clients connected with the add_listener method.
To publish a message to all of the connected clients you just call node.send(message)
I'm hopefully going to write up a little example app in the next week or two to demonstrate this a little better.
Feel free to ping me on github if you want some help with it.

request.websocket is probably get closed when you return from the request handler (view). The simple solution is to keep the handler alive (by not returning from the view). If your server is not multi-threaded you won't be able to accept any other simultaneous requests.

Removing clients from the reactor in Twisted

I have a simple TCP client which is connected to twisted using:
reactor.connectTCP(host, port, SomeClientFactory())
The program is able to receive a HUP signal to trigger a reload. I'd like to basically:
Remove the old clients
Reload config
Create new clients based upon new config
However, I can't seem to find a way to acheive the first of these points. Any tips?
Thanks

IReactorTCP.connectTCP returns an IConnector provider. As you can see on the definition of the IConnector interface, the disconnect method will do something like what you want. You can also use the protocol instance's transport attribute's loseConnection method, of course. The latter would be more suitable if there's any kind of cleanup you want the protocol to do before actually disconnecting, since you could put that work and a call to loseConnection at the end of a method like shutdown or quit or cleanup on the protocol class and then just call that.