I can best describe what I'm looking for with an example of a simplified version. One of the demos for Tornado is a simple chat server:
https://github.com/facebook/tornado/blob/master/demos/chat/chatdemo.py
I'm interested in the MessageMixin class here. It keeps a static-length backlog of messages, and when new messages are available, it returns the slice of the message list that's new. Or that's what it appears to do. I know that I've implemented something like that before when writing a simple comet app.
So has anyone generalized this and added fancy things to it? I'm particularly interested in a way to manage many channels of communication and delete ones that haven't been used in a while. Persistence might also be useful.
Is this something an MQ can do?
Redis has a publish/subscribe feature, along with additional data structure-oriented commands which you can use to persist and expire the message backlog, list users in a given room, or other attributes associated with them. The protocol is text-based and is a superset of the Memcached commands.
Here is a description which uses chat as an example of pub/sub along with a Ruby example using Websocket, and a snippet in Python which uses Websocket, Tornado and Redis pub/sub to implement a simple chat room.
Based on the information in your question, a dedicated message queue (like RabbitMQ) may also be useful to you. It is hard to say without knowing what you need in the areas of message volume, fault-tolerance, replication, etc. Redis may also be what you're looking for, but if nothing else it is pretty simple and could help you get a prototype running quickly to further nail down your app's requirements.
Related
Currently I am designing and programming a piece of software that controls a series of devices. The software is planned to have a REST interface through which you would be able to control the software (and the devices) remotely.
Now, a very basic abstraction of the architecture could look something like this:
As you can note, the system is composed of a master controller, which then handles and monitors different modules that are not dependent on each other. Front End module is an example in the diagram, while others are general abstractions of modules, but they could be anything (Database module, MessageBus module, etc).
For the actual REST interface there are both data retrieval, data storage, as well as control commands that are being implemented.
My "problem" is that I can't decide on how these "commands" should be propagated down the line.
Some cases of possible commands:
Command requesting to turn on/off, restart, control a device handled by another module
Command requesting to restart/reload the software
Command to retrieve data from another module
Now I see a few possible ways of the actual logic implementation:
All received REST commands are dispatched through a message bus. In this case each request should receive a unique identifier which could be then used to retrieve the status of the request
All received REST commands make direct calls to other modules
Both of these have pros and cons:
The second method of doing everything could very easily fall into spaghetti code and would be hard to debug and expand upon, since there's a lot of multithreading utilization through different modules. But it is possibly the fastest way of handling a command and retrieving data. Especially since the project requires speed and responsiveness.
The first method lacks the advantages of the second method, however it would help to keep the code and architecture clean and clear of dependencies from other modules. Furthermore, a Console channel is also planned which could in theory use the same methodology for implementation.
There's another method that I thought of while brainstorming about the problem:
To force the REST channel to forward the incoming requests to the actual FronEnd module and then "wait" until it receives a response. The FrontEnd module would then have to directly call other modules for any information or actions requested.
This method, however, is not that "different" from the method nr 2.
Could anyone offer any advice? Perhaps ideas on the implementation or design decisions?
In case you are wondering, the software is being written in Python, but I don't think this is relevant to the question.
So basically we have decided to ditch the RESTful way and simply went for an approach using sockets (or websockets in particular).
The commands sent through the websockets are formatted as JSON and resemble REST in a way (basically a request contains a "URI", a "Action" [get, put, post, etc.] and a "body").
A command comes to the front end control part of the system and then is pushed to a message bus where another part of the system has already subscribed for these commands. After it processes the data or executes a command, the data is returned through the message bus and dispatched to the client through the websocket.
I'm designin a pub/sub architecture using ZMQ. I need maximum reliability and scalability and am kind of lost in the hell of possibilities provided.
At the moment, I got a set a publishers and subscribers, linked by a broker. The broker is a simple forwarder device exposing a frontend for publishers, and a backend for subscribers.
I need to handle the case when the broker crashes or disconnects, and improve the overall scalability.
Okay, so i thought of adding multiple brokers, the publishers would round robin the broker to send messages to, and the subscribers would just subscribe to all these brokers.
Then i needed a way to retrieve the list of possible brokers, so i wrote a name service that provides a list of brokers on demand. Publishers and subscribers ask this service which brokers to connect to.
I also wrote a kind of "lazy pirate" (i.e. try/retry one after the other) reliable name service in case the main name service falls.
I'm starting to think that i'm designing it wrong since the codebase is non stop increasing in size and complexity. I'm lost in the jungle of possibilities provided by ZMQ.
Maybe something router/dealer based would be usable here ?
Any advice greatly appreciated !
It's not possible to answer your question directly because it's predicated on so many assumptions, many of which are probably wrong.
You're getting lost because you're using the wrong approach. Consider 0MQ as a language, one that you don't know very well yet. If you start by trying to write "maximum reliability and scalability", you're going to end up with Godzilla's vomit.
So: use the approach I use in the Guide. Start with a minimal solution to the core message flow and get that working properly. Think very carefully about the right kind of sockets to use. Then make incremental improvements, each time testing fully to make sure you understand what is actually going on. Refactor the code regularly, as you find it growing. Continue until you have a stable minimal version 1. Do not aim for "maximum" anything at the start.
Finally, when you've understood the problem better, start again from scratch and again, build up a working model in several steps.
Repeat until you have totally dominated the problem and learned the best ways to solve it.
It seems like most of the complexity stems from trying to make the broker service persist in the event of a failure. Solving this at the application level gives you the highest degree of flexibility, but requires the most effort if you're starting from scratch.
Instead of handling this at the application level, you could instead handle this at the network level. Treat your brokers as you would any other simple network service and use an IP failover mechanism (e.g., pacemaker/corosync, UCARP, etc) to fail a virtual ip address over to the secondary service if the primary becomes unavailable.
This greatly simplifies your publishers and subscribers, because you don't need a name service. They only need to know about the single virtual ip address. ZMQ will take care of reconnecting to the service as necessary (i.e., when a failover occurs).
Mornink!
I need to design, write and implement wide system consisting of multiple unix servers performing different roles and running different services. The system must be bullet proof, robust and fast. Yeah, I know. ;) Since I dont know how to approach this task, I've decided to ask you for your opinion before I leave design stage. Here is how the workflow is about to flow:
users are interacting with website, where they set up demands for service
this demand is being stored (database?) and some kind of message to central system (clustered) is being sent about new demand in database/queue
central system picks up the demand and sends signals to various other systems (clusters) to perform their duties (parts of the demanded service setup)
when they are done, they send up message to central system or the website that the service is now being served
Now, what is the modern, robust, clean and efficient way of storing these requests in some kind of queue, and executing them? Should I send some signals, or should I let all subsystems check the queue/db of any sort for new data? What could be that queue, should it be a database? How to deal with the messages? I thought about opening single tcp connection and sending data over that, along with comands triggering actions/functions on the other end, but at closer inspection, there has to be other, better way. So I found Spring Python, that has been criticized for being so 90's-ish.
I know its a very wide question, but I really hope you can help me wrap my head around that design and not make something stupid here :)
Thanks in advance!
Some general ideas for you:
You could have a master-client approach. Requests would be inserted in the master, stored in a database. Master knows the state of each client (same db). Whenever there is a request, the master redirects it to a free client. The client reports back when has finished the task (including answers if any), making it able to receive a new task from the master (this removes the need for pooling).
Communication could be done using web-services. An HTTP request/post should solve every cases. No need to actually go down to the TCP level.
Just general ideas, hope they're useful.
There are a number of message queue technologies out there which are Python friendly which could serve quite well. The top two that I know of are ActiveMQ and RabbitMQ, which both play well with Python, plus I found this comparison which states that ActiveMQ currently (as of 18 months ago!) outperforms RabbitMQ.
I'm building a website where I hook people up so that they can anonymously vent to strangers. You either choose to be a listener, or a talker, and then you get catapulted into a one-on-one chat room.
The reason for the app's construction is because you often can't vent to friends, because your deepest vulnerabilities can often be leveraged against you later on. (Like it or not, this is a part of human nature. Sad.)
I'm looking for some insight into how I should architect everything. I found this neat tutorial, http://giantflyingsaucer.com/blog/?p=875, which suggests using python & stackless + flash. Someone else suggested I should try using p2p sockets, but I don't even know where to begin to look for info on that.
Any other suggestions? I'd like to keep it simple. :^)
Unless you expect super high load, this is simple enough that it doesn't really matter what you use on the backend: just pick something you're comfortable with. PHP, Python, Ruby, Even a bash script using CGI - your skill level with the language is likely to make more difference that the language features themselves.
I would use an XMPP server like ejabberd or OpenFire to power the backend. XMPP contains everything you need for creating chat/real-time applications. You can use a Flex/Flash Actionscript library like Actionscript 3 XIFF to communicate with the XMPP server.
Flash is user-unfriendly for UI (forms, etc) and it is relatively easy to do what you want using HTML and Javascript on the front-end.
One possible approach for reading the messages would be to regularly do an Ajax request from the server for any new messages. Format the new message and insert it into the DOM.
You will probably need to answer at least these questions before you continue, though:
1) Are you recreating IRQ (everyone sees your posts), or is this a random one-to-one chat, like chatroulette?
1a) Is this a way for a specific person to talk to another specific person, or is this more like twitter?
2) What is your plan for scaling up if this idea takes off? Memcached should probably be a method of last-resort ("bandaid over a bullet-hole"). What's your roadmap for eventually handling a large volume of messages?
3) Is there any way to ignore users? Talk to certain users? Hide your rants from users?
Hey Zach I had to create a socket server for a flash game I made. I built my server in C#, but I would use whatever language your familiar with. If you let me know what your most comfortable with I could try to help find a good tutorial.
The one thing I spent many hours on was getting flash to work from a website with a socket server. With the newer versions of Flash you need to send back a policy file. In my case this needed to be the first chunk of data sent back to the client when they connected to the socket server.
Not sure what to tell you about structuring the back end. I need to know a little bit more about your programming experience. I had an array of all user connections, and was placing them in different "Rooms" so they could play each other. So just some simple arrays and understanding how to send messages to the clients would help you here.
If you have any familiarity with C# I would have no problem sending you the source code for my socket server.
I have an XMPP server (likely — python, twisted, wokkel), which I prefer not to restart even in the development version, and I have some python module “worker” (which is interface to particular django project), which gets jid and message text and returns some response (text or XML, either way).
The question is, what would be the best way to connect them, considering that I may prefer to update the module part too often?
Another consideration is that it might be required to run multiple instances of “worker” for it all to be high-load-capable.
One possible way I see is implementing a thread in the server which checks if the module was changed and reload()s it if necessary.
The other way would be making something similar to fastcgi through sockets, although not HTTP-based.
My suggestion is:
Use RabbitMQ with XMPP adaptor.
Use Python carrot for AMQP since it can be used directly under Django.
I can't say that I understand all of your question, but the bit where you're asking how to connect django and twisted and multiple workers: I'd suggest using AMPQ. This gets you reliable message delivery, multiple consumers, persistence.
There's the txAMQP library for twisted.
https://launchpad.net/txamqp
A good primer to AMQP here, it's a good place to start:
http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/