I have a service provided by a REST API, with a Python library wrapping it using python-requests.
I have a 'dumb' user interface designed by a third party (not Python) to connect to a local XML-RPC.
Now I have to connect both ends and forward the XML-RPC calls to the REST API and return the results. It's mostly asynchronous and doesn't depend on results returning to the user in real-time. Most of the XML-RPC calls are supposed to return immediately, queue a task, and some other call will query the results later. Data is stored in an sqlite database until needed.
So, I decided to use twisted.web.xmlrpc for this middle layer and use the requests based lib for the remote calls and it works fine. I guess I'm blocking twisted's mainloop for a few seconds once in a while, but that's not a big deal.
The problem is that I also have to make some big file uploads from this middle layer to the HTTP server providing the REST API. I can't make those uploads using the requests based lib because it will block the twisted loop until the upload is finished.
I'd rather not use multithreading, and I really don't want to rewrite the python-requests based lib I have as a twisted client. Is there any way I can integrate requests into twisted's mainloop, or any other reasonable solution?
If you like requests' style of API, but want something that would work with Twisted, consider using treq. There are support libraries for writing interfaces which can be either synchronous or asynchronous depending on their caller's needs.
If you really want to use requests, but you don't want to block the main loop, you can invoke it with twisted.internet.threads.deferToThread. This is mostly transparent, and if your requests don't share any state you can almost ignore the fact that you're using multithreading.
But, ultimately, Jean-Paul's comment is correct; you are going to need to make some changes to the way this code works, if you want to change the way it works.
Related
Currently I am designing and programming a piece of software that controls a series of devices. The software is planned to have a REST interface through which you would be able to control the software (and the devices) remotely.
Now, a very basic abstraction of the architecture could look something like this:
As you can note, the system is composed of a master controller, which then handles and monitors different modules that are not dependent on each other. Front End module is an example in the diagram, while others are general abstractions of modules, but they could be anything (Database module, MessageBus module, etc).
For the actual REST interface there are both data retrieval, data storage, as well as control commands that are being implemented.
My "problem" is that I can't decide on how these "commands" should be propagated down the line.
Some cases of possible commands:
Command requesting to turn on/off, restart, control a device handled by another module
Command requesting to restart/reload the software
Command to retrieve data from another module
Now I see a few possible ways of the actual logic implementation:
All received REST commands are dispatched through a message bus. In this case each request should receive a unique identifier which could be then used to retrieve the status of the request
All received REST commands make direct calls to other modules
Both of these have pros and cons:
The second method of doing everything could very easily fall into spaghetti code and would be hard to debug and expand upon, since there's a lot of multithreading utilization through different modules. But it is possibly the fastest way of handling a command and retrieving data. Especially since the project requires speed and responsiveness.
The first method lacks the advantages of the second method, however it would help to keep the code and architecture clean and clear of dependencies from other modules. Furthermore, a Console channel is also planned which could in theory use the same methodology for implementation.
There's another method that I thought of while brainstorming about the problem:
To force the REST channel to forward the incoming requests to the actual FronEnd module and then "wait" until it receives a response. The FrontEnd module would then have to directly call other modules for any information or actions requested.
This method, however, is not that "different" from the method nr 2.
Could anyone offer any advice? Perhaps ideas on the implementation or design decisions?
In case you are wondering, the software is being written in Python, but I don't think this is relevant to the question.
So basically we have decided to ditch the RESTful way and simply went for an approach using sockets (or websockets in particular).
The commands sent through the websockets are formatted as JSON and resemble REST in a way (basically a request contains a "URI", a "Action" [get, put, post, etc.] and a "body").
A command comes to the front end control part of the system and then is pushed to a message bus where another part of the system has already subscribed for these commands. After it processes the data or executes a command, the data is returned through the message bus and dispatched to the client through the websocket.
I am writing a web application that uses nginx to serve static content and tornado to serve dynamic content. I was thinking of utilizing CouchBase as my datastore, but am having trouble locating a suitable client for use with the Tornado framework (i.e. asynchronous). Anybody know of one?
I've seen trombi: https://github.com/inoi/trombi but couldn't find much information on it. If anyone has had any experience with it (good or bad), I'd love to hear about it.
I would really recommend sticking with the Couchbase released code for Python. While it isn't technically asynchronous, the queries are so fast that it really doesn't factor into things. Its not like building out a query for a Database which could easily lock up continued actions for a period of time. Not to mention the fact there is a lot of load balancing and bucket management code that you would lose in most situations by trying to find some third party module for it.
Also you can always build a multiprocessing package to create sub-processes to handle removing these calls from the primary process stream and reduce the impact to almost nothing.
UPDATE
Another option is to use Tornado's internal callback functionality to offset the blocking process so it doesn't impair browsing. A method for this is described here: http://tornadogists.org/2185380/
I'm interested in something based on Jabber but I didn't find a free/opensource one so I'm thinking of writing one.
I've installed a Jabber server and now thinking about the ways in which I can write the client. I'm thinking of one of either these two methods.
1) An ajax call made to a jabber script running on the webserver that takes care of connecting to the server. But then I thought because of the dependencies involved in the jabber client, it might end up consuming too much memory when a few clients connect.
2) The other method is to run a client running as a daemon that takes care of all the heavy lifting. This way I need to have only one instance of the client that sends a spoofed message (sender's name as that of whatever the user entered on the site). A simple script running on the webserver talks to this daemon over some sort of API (XMLRPC or Msgpack maybe?)
I think #2 is better but I'm not sure. Are there other ways I can implement this? I'm considering using Perl or Python for this.
Jabber is usually called XMPP nowadays, and there are dozens of clients and servers, something for every language. If you are using Javascript (you mention Ajax), you probably want Strophe. Most servers are modular, so you only load the features you need (consider Tigase, ejabberd, or xmpppy). Writing your own is even worse an idea than it sounds.
BOSH
Install prosody because it is really eaSily installed and has BOSH support built-in. You could skip this but then you need to find out how to use BOSH via ejabberd.
use strophe.js to implement this(using BOSH). New browsers support cross-domain request(CORS -> read Proxy-less BOSH part). The old browsers you could use proxy or use flash in the middle as proxy.
read Professional XMPP Programming with JavaScript and jQuery to learn strophe. It even has chapters explaining how to create chat.
Node.js
Or you could consider installing node.js to create your chat system using socket.io.
I am trying to move away from CherryPy for a web service that I am working on and one alternative that I am considering is Tornado. Now, most of my requests look on the backend something like:
get POST data
see if I have it in cache (database access)
if not make multiple HTTP requests to some other web service which can take even a good few seconds depending on the number of requests
I keep hearing that one should not block the tornado main loop; I am wondering if all of the above code is executed in the post() method of a RequestHandler, does this mean that I am blocking the code ? And if so, what's the appropriate approach to use tornado with the above requirements.
Tornado comes shipped with an asynchronous (actually two iirc) http client (AsyncHTTPClient). Use that one if you need to do additional http requests.
The database lookup should also be done using an asynchronous client in order to not block the tornado ioloop/mainloop. I know there are a couple of tornado tailor made database clients (e.g redis, mongodb) out there. The mysql lib is included in the tornado distribution.
I have a Python-driven web interface powered by Apache 2.2 with mod_python and Python 2.4. I need to make an asynchronous process appear synchronous to users of this web interface.
When users access one module on this website:
An external SOAP interface will be contacted with a unique identifier and will respond with a number N
The external interface will respond asynchronously by contacting a SOAP server on my machine between 1 and 10 times (the number N tells us how many responses we will receive)
I need to somehow aggregate these responses and pass them to the original module which will display the information back to the user. The goal is to make the process appear synchronous to the user.
What is the best way to handle this synchronization issue? Is this something Twisted would be well-suited for?
I am not restricting myself to Python for the solution, though it is preferred because everything else on the server is in Python. I prefer a solution that is both scalable and will take a minimal amount of programming time (though I understand that these attributes are somewhat at odds).
Maybe you can use Orbited to get ajax push with long-lived HTTP connections to your web clients. Orbited is based on Twisted, so I think it makes sense to look at if you already know Twisted. Have a look at this tutorial to get started.