I'm new to Tornado, and working on a project that involves some rather complex routing. In most of the other frameworks I've used I've been able to isolate routing for testing, without spinning up a server or doing anything terribly complex. I'd prefer to use pytest as my testing framework, but I'm not sure it matters.
Is there a way to, say, create my project's instance of tornado.web.Application, and pass it arbitrary paths and assert which RequestHandler will be invoked based on that path?
No, it is not currently possible to test this in Tornado via any public interface (as of Tornado version 4.3).
It's straightforward to avoid spinning up a server, although it requires a nontrivial amount of code: the interface between HTTPServer and Application is well-defined and documented. The trickier part is the other side: there is no supported way to determine which handler will be invoked before that handler is invoked.
I generally recommend testing routing via end-to-end tests for this reason. You could also store your URL route list before passing it into Tornado, and do your tests against that - the internal logic of "take the first regex match" is pretty easy to replicate.
Related
I'm struggling with some architectural choices for a scalable internet-of-things application.
I've chosen to base my project on Twisted augmented with the Cyclone framework to provide many Tornado convenances (websockets, auth-decorators, secure-cookies, etc)
Using a Twisted core has worked beautifully for me. I have numerous IP protocol and hardware interfaces all of which turned out to have great library support inside of twisted (and adding new protocols and interfaces to my application are the most-likely angles I'll have project scope creep), all with Twisted needing very low CPU and providing for very high connection counts.
My problems are with second-order webapp functionality.
I pulled in Cyclone thinking that with it's auth goodies (OpenID, oauth, user-auth decorators and secure-cookies) it wouldn't take much to implement user/session/admin functionality in my webapp. After the 500+ lines of abstracting my database (via txmongo) and just building user logins it became clear I both:
Didn't understand how little Cyclone/Tornado bring in the user/session/admin space, and
Didn't understand the amount of code it takes to fill in the gaps if your trying to build a multi-user auth webapp
A friend pointed me at Flask, which initially I thought was completely redundant, until I found flask plugins. The combination of Flask-Login and Flask-Admin would completely cover my user, session and user-admin needs, negating me writing what I would guess to be about 2k lines of code. Unfortunately, the flask plugins are all rife with blocking code and calls to blocking libraries. I don't see them as compatible with my project even if WSGI containers are used given that the user/session functionality happens with every page load (additionally I don't see any short cuts that would allow me to port them to async world without work roughly equal to that of rewriting them)
My question is:
In the python async space (... hopefully in the Twisted space, given my protocol needs), are there any plugins or alternate frameworks that provide ready-to-go user/login/admin functionality similar to what is in Flask-Login and Flask-Admin?
P.S. I've looked at Klein as the obvious Twisted version of Flask, but it doesn't seem to have a plugin ecosystem, and I'm not finding any strong user/session/admin there.
P.P.S. By the time I wrote this question I had already written my own (crappy) user-login-session system. So what I'm really after is the "Admin" capability (automated CRUD functions on user-style records, including web UI rendering, all designed in a Twisted/async way). I asked about user/login in the question in case it turn out there is an already-integraded solution (such as flask-login and flask-admin) in which case I would happily drop my code and switch to that.
Do you really need everything async? Consider async WebSockets but sync page renders. If you must, add an async downstream proxy or load balancer which will virtually eliminate app server's IO overhead.
Here the simple webserver means a server that deal with simple HTTP request, just like the following one:
import BaseHTTPServer
class WebRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
def do_GET(self):
if self.path == ‘/foo’:
self.send_response(200)
self.do_something()
else:
self.send_error(404)
def do_something(self):
print ‘hello world’
server = BaseHTTPServer.HTTPServer((’127.0.0.1′,8080), WebRequestHandler)
server.serve_forever()
Despite of dealing with request of POST,PUT,DELETE methods, what is the difference between this simple server with Apache Web Server?
Or in other words, if i want to use python to implement a server which can be put into use of business, what also should i do?
It'd be greatly appreciated if the big picture of Apache Server is shown.
Or in other words, if i want to use python to implement a server which can be put into use of business, what also should i do?
There are already python-based web servers, such as CherryPy (which I think is intended to be a web server solution on the same stack level as Apache; it is more python-based though, and Apache has been around a lot longer).
If you wish to write a lightweight extremely simple webserver from scratch, there is probably nothing wrong with using BaseHTTPServer, other than perhaps a few outstanding design issues (I hear race conditions might permanently clog a socket until a thread dies).
Though I would not recommend it (alone) for business, some of the big boys use BaseHTTPServer with a bit of extra machinery:
http://www.cherrypy.org/browser/trunk/cherrypy/_cphttpserver.py?rev=583
To elaborate, Apache is the industry standard. It has a plethora of configuration options, a security team, vulnerability mailing lists I think, etc. It supports modules (e.g. mod_python). Python-based web servers also support python-based modules (maybe they might allow you access to non-python things) via something called a WSGI stack; a WSGI application can run on any python-based web server (and Apache too, which also has a modwsgi); I think they are narrower in scope than Apache modules.
Apache module examples: http://httpd.apache.org/docs/2.0/mod/
WSGI examples (not a valid comparison): http://wsgi.org/wsgi/Middleware_and_Utilities
I might code my own webserver if I'm doing something extremely lightweight, or if I needed massive control over webserver internals that the module interfaces could not provide, or if I was doing a personal project. I would not code my own server for a business unless I had significant experience with how real-world web servers worked. This is especially important from a security vulnerability point-of-view.
For example, I once wrote a web-based music player. I used a BaseHTTPServer to serve music out of a sandbox I had written to ensure that people couldn't access arbitrary files. Threading was a nightmare. (I recall a bug where you needed to pass special arguments to Popen since threading caused an implicit fork which would cause hangs on dangling file descriptors.) There were other various issues. The code needed to be refactored a lot. It can be very worthwhile for a personal project, but is a significant undertaking and not worth it for a business that just needs a website.
I know two startups who have been content using Pylons (using Paste) or Turbogears (using CherryPy) in the past, if you're looking for a lightweight python web server stack. Their default template systems are lacking though. The choice between Apache and a leaner more python-based web server may also depend on the skillset of your co-developers.
Apache is written in C and designed to be scalable while BaseHTTPServer is meant for local/testing/debugging environments.
So you shouldn't use BaseHTTPServer for any production sites.
Apache web server knows and supports the entire HTTP protocol, so it can deal with all the complications having to do with headers, keeping connections open, caching content, all the different HTTP response codes and their proper treatment, etc.
You'd have to understand the entire HTTP protocol and express it in code to go beyond your simple HTTP server.
Disclaimer: I'm not very familiar with any of the things mentioned in the question title.
Would it be possible to use a browser control (like Webkit) as a frontend for a WSGI app (using a framework like Flask) without starting a local WSGI server?
Basically the requests and responses are managed by a middle layer between the HTML UI and the WSGI backend. A certain URI could mean "Local", for instance "local://" or something similar, and will be routed to the embedded WSGI app with all the original headers etc.
You will lose any features that a normal WSGI server provides unless you implement it yourself or somehow embed a server that is also usable via an API instead of real HTTP requests.
Now that I think of it, this is the only real requirement: A WSGI server that is callable via an API and not just real HTTP requests.
I know the usefulness of this is questionable (and maybe doesn't even make sense). My question is whether this is at all possible?
EDIT: Here's another way of putting it:
I want a single codebase to be both a web app and a desktop app, using an HTML frontend and a Python backend. I don't want to run a server on any port for the desktop app. What's the easiest way to achieve this?
It is in theory possible to write your own WSGI container that implements a full API and adapts that to WSGI. flup might bring some inspiration.
Earlier today I saw exactly what you're asking for -- a way to call WSGI through an API without actually connecting over the network. However, it shouldn't be that hard.
On a side note, you might want to look at PySide, of particular interest to you may be the ability to bind python elements to DOM events, so if you're just looking to trigger python code that's an even shorter route.
If you give some more detail on what you're hoping to achieve we might be able to dial it in for you.
Reviving this, since we're facing the same problem and are about to scale things up from a single view/widget to the whole app.
What I did was to simply set the base URL to something where I serve static content, and from a QRC file that's easy:
html = jinjatemplate.render(...)
self._mainFrame.setHtml(html.decode('utf-8'), Qt.QUrl('qrc:///Orsync/html/'))
For the communication, our HTML uses AJAX over jQuery for most things. You could wrap that in a layer that either does $.post(...) or api.post(...) like this:
self._mainFrame.addToJavaScriptWindowObject('api', self._webapi)
You'd need to decode the URL and create a request object yourself, but maybe that's not too hard to do? We use very few URLs currently (who are mapped directly to python objects/functions) so it's easy to do the mapping ourselves.
Data that goes back is just sent using QMainFrame.evaluateJavaScript(...), either as a direct Qt call or as a bunch of code lines fetched using $.getScript(...) (which just evaluates the code received).
I'm currently rebuilding things a bit using CherryPy, and it maps urls -> Python objects straight off, so I'm hoping there's something to be gained by that.
Otherwise, I would wish one could run QWebKit over named pipes or something similarly localized and not a tcp-socket. :)
I'm writing unit tests for a portion of an application that runs as an HTTP server. The approach I have been trying to take is to import the module that contains the HTTP server, start it. Then, the unit tests will use urllib2 to connect, send data, and check the response.
Our HTTP server is using Twisted. One problem here is that I'm just not that familiar with Twisted :)
Now, I instantiate our HTTP server and start it in the setUp() method and then I stop it in the tearDown() method.
Problem is, Twisted doesn't appear to like this, and it will only run one unit test. After the first one, the reactor won't start anymore.
I've searched and searched and searched, and I just can't seem to find an answer that makes sense.
Am I taking the wrong approach entirely, or just missing something obvious?
Here's some info: Writing tests for Twisted code using Trial
You should also look at the -help of the trial command. There'a lot of good stuff in trial! But it's not always easy to do testing in a async application. Good luck!
I believe that for unit testing within Twisted you're supposed to use TwistedTrial (it's a core component, i.e., comes with the Twisted tarball in the twisted/trial directory). However, as the URL I've pointed to says, the doc is mostly by having a look through the source (including sources of various Twisted projects, as they're tested with Trial too).
As others mentioned, you should be using Trial for unit tests in Twisted.
You also should be unit testing from the bottom up - that's what the "unit" in unit testing implies. Test your data and logic before you test your interface. For a HTTP interface, you should be calling processGET, processPOST, etc with a mock request, but you should only be doing this after you've tested what these methods are calling. Each test should assume that the units tested elsewhere are working as designed.
If you're speaking HTTP, or you need a running server or other state, you're probably making higher level tests such as functional or integration tests. This isn't a bad thing, but you might want to rephrase your question.
There is a known bug with Twisted (that probably won't get fixed) where re-starting the reactor causes a crash.
This is why your unit tests don't work.
As well as using Trial you might want to consider seperate testing systems that talk to your HTTP server like a client will.
Webdriver - an API to drive a browser session around your site.
TestGen4Web - Firefox plugin that records interactions with site and can replay.
I'm quite new in Python world. I come from java and ABAP world, where their application server are able to handle stateful request.
Is it also possible in python using WSGI?
Or stateful and stateless are handled in other layer?
Usually, you don't work with "bare" WSGI. You work with web-frameworks, such as Pylons or TurboGears2.
And these contain a session-middleware, based on WSGI - called "Beaker". But if you work with the framework, you don't have to worry about that - you just use it.
But if you insist, you can of course use Beaker standalone.
I prefer working directly on wsgi, along with mako and psycopg.
It's good to know about Beaker, though I usually don't hold state in the server because I believe it reduces scalability. I either put it in the user's cookie, in the database tied to a token in the user's cookie, or in a redirect url.
Your question is a little vague and open-ended. First of all, WSGI itself isn't a framework, it's just the glue to connect a framework to the web server. Secondly, I'm not clear on what you mean when you say "state" -- do you mean storing information about a client on the server? If so, web frameworks (Pylons, Django, etc) allow you to store that kind of information in web session variables.