I'm trying to do something seemingly simple, but I'm not sure how to go about it...
I have an application that publishes messages over a ZMQ PUB socket. I'd like to write a small server that subscribes to these updates and broadcasts them to all connected websocket clients.
This seems simple on the surface, but I don't know how to trigger an event in the twisted reactor when data is received through the subscription. I know ZMQ has an ioloop as well, but it is not the same as twisted...
What's the best way to go about this? I don't have to use twisted, some other python framework will do if something can deal with this better (rpclib?)
If you want to use ØMQ with Twisted, you can use txZMQ. The documentation on the linked page has a couple of basic examples which ought to get you started.
Related
I am working on a Flask-SocketIO application that integrates with zmq. The basic premise of the app is that a zmq message is received by the Flask-SocketIO web server, then that zmq message is converted to a SocketIO message which is sent along to the client (browser). I have the app working but not exactly the way that I want it to be working.
There is a need for the zmq event listener to be on a separate process than the main server process. I was able to use both Redis and RabbitMQ as a message queue for facilitating SocketIO emits from non-server processes. OK, great. So what's the problem?
The problem is that I'd really like to use zmq as a message queue instead of Redis or RabbitMQ since I'm already integrating with zmq in my app. So I read in the Flask-SocketIO docs that Kombu is the mechanism for supporting other types of message queues. OK, cool. But then I notice that zmq as a transport has been removed from the latest release of Kombu. And as far as I can tell it was only experimental when it was a transport option, like in Kombu 3.0.37.
My first approach was to just try zmq as a message transport via Kombu 3.0.37, but that is not working. I'm still trying to determine exactly why that is. But my best guess right now after looking at the source code a bit is that multiple processes are trying to open a PULL socket on the same port, which just doesn't work, even in a simple independent example. And that makes sense. On that front, my next step is to manually create a PULL socket independently of the zmq transport code, and somehow pass that in as a type of singleton for the zmq transport code to use.
Another approach that I'm working is getting zmq to work as a transport in a Hello World Kombu example. I swapped out the connection string in the sample code with zmq+tcp://localhost. This is where it becomes clear that I'm not understanding how to use zmq as a message queue transport. If I run the publisher code, I can send a message. But when I run the client code a few seconds later, it says that the queue is empty. That makes me think that in order for zmq to work as a transport, I may need an external zmq message broker of some sort, which I'm assuming I would need to put together on my own. But I haven't yet got my head around how that would work.
Any suggestions? Is this a waste of time to try to pursue zmq as a transport, or should I continue down this path? Miguel Grinberg (author of Flask-SocketIO) has graciously provided some direction, but I wanted to branch out a little to see if anyone else had some thoughts on the matter.
Recently I've been doing a lot of testing around different ways of serving our Django application. I've settled on uwsgi as it seems to fit our needs pretty well.
I've recently discovered that uwsgi also supports WebSockets and started looking into it and found some examples: https://github.com/unbit/uwsgi/blob/master/tests/
After running the example (websockets_chat.py) and taking a look through uwsgi's documention for their websockets implementation it appears as though you can only send broadcast, or global messages.
Has anyone managed to find a way to transmit a message to a particular user or does uwsgi not support that level of communication yet?
Cheers
There is nothing like broadcast or global messages in websockets specs. They only "upgrades" an http connection to a lower-level one. What you do with that connection is up to you. The examples show integration with redis as message exchanger but you are free to make other uses.
For your specific case you will need to build a shared list of connected users and implements routing. Remember, you cannot rely on node.js way as it is based on a single threaded setup so everything is way simpler. In uWSGI a websocket connection can happens on a thread, a process or a coroutine, so exchanging data between them is the key.
I need to have a tcp socket client connected to a server to send data and receive.
But this socket must be always on and i cannot open another socket.
I have always some data to send over the time and then later process the answer to the data sent previously.
If i could open many sockets, i think it was more easy. But in my case i have to send everything on the same socket asynchronously.
So the question is, what do you recommend to use within the Python ecosystem? (twisted, tornado, etc)
Should i consider node.js or another option?
I highly recommend Twisted for this:
It comes with out-of-the-box support for many TCP protocols.
It is easy to maintain a single connection, there is a ReconnectingClientFactory that will deal with disconnections and use exponential backoff, and LoopingCall makes it easy to implement a heartbeat.
Stateful protocols are also easy to implement and intermingle with complex business logic.
It's fun.
I have a service that is exactly like the one you mention (single login, stays on all the time, processes data). It's been on for months working like a champ.
Twisted is possibly hard to get your head around, but the tutorials here are a great start. Knowing Twisted will get you far in the long run!
"i have to send everything on the same socket asynchronously"
Add your data to a queue, have a separate thread taking items out of the queue and sending via socket.send()
I am looking to have my client "react" to a received message through some xml based communications medium. I was looking into xmpp with google talk, but I just need something that can quickly relay messages on an event based basis (i.e. without having a thread over a "check messages" function.)
I am using twisted to do the rest of my project, so if at all possible, it would be very helpful to use twisted for the rest.
wokkel is an extension of twisted words that makes it super easy to develop clients and components.
Here is echobot as an example.
I'm looking for a way to write a XMPP bot that would listen to a RabbitMQ queue and send messages to the XMPP channel notifying users of any new issues ( already got Nagios sending notifications to RabbitMQ).
I've tried using xmppy and it stopped working and I stumbled across SleekXMPP which looks fairly better.
I'm just wondering if I define a AMQP listener to automatically call the XMPP "send" method in the bot. So it would be listening both on AMQP and XMPP at the same time.
Thank you for your help!
Edit: Would BOSH be much better solution here ?
The most interesting part of your solution will be that many libraries in this space assume that they are the only event loop. You'll either need to put each in its own thread (seemingly easier, but fraught with lurking locking issues), use a non-blocking I/O approach like Twisted (but you'll need an AMQP library), or extract the socket file descriptors out of each of the libraries you're using and run select() or poll() over them to tell when there is data to read. Of these three, the Twisted approach seems easiest to me.
BOSH will just make the problem more difficult. Don't go that way.
This is really quite simple. I suggest that you start by writing an AMQP listener that simply prints out received messages. Once you get that working it should be obvious how to integrate that into an XMPP bot.
You can use ejabberd and a xmpp plugin like this https://github.com/rabbitmq/rabbitmq-xmpp