We've got a couple of quite complex sites written in Python based on Django, hosted by uwsgi. We also use nginx for some out-of-application things (like SSL termination).
I need to write an incredibly lightweight API which does nothing except validate an authentication token (against a different API) and respond with some info from the local file system.
I'd prefer to avoid using a whole Django site for what will be ~50 lines of code, however, since I develop on a windows machine, I'm not sure how best to host this tiny API for dev/testing. (Django's runserver command usually handles this for us) uWSGI seems like a good choice for final usage as it's already in use elsewhere but doesn't run on Windows.
I could always just code on one box, deploy and then test but it seems a little long-winded. Ideally, whatever I can find to host on windows will call my code in the same way as uwsgi
What's the best way to do this?
The most lightweight would be wsgiref.simple_server from the standard library which Django's runserver command uses internally.
Alternatively you can use
WSGI server from the Paste
This is a minimalistic WSGI server using Python’s built-in
BaseHTTPServer; if pyOpenSSL is installed, it also provides SSL
capabilities.
Waitress from the Pylons project
Waitress is meant to be a production-quality pure-Python WSGI server
with very acceptable performance. It has no dependencies except ones
which live in the Python standard library. It runs on CPython on
Unix and Windows under Python 2.6+ and Python 3.2+. It is also
known to run on PyPy 1.6.0 on UNIX. It supports HTTP/1.0 and HTTP/1.1.
WSGI server from CherryPy
A high-speed, production ready, thread pooled, generic HTTP server.
Related
I run a WSGI application from within my Python file, using the make_server command.
(I don't know if this is a good practice or whether it is more common to setup Apache or nginx for this purpose.)
I want to make this little server secure by adding SSL support. Where do I start?
Is the built-in WSGI server from Python considered safe? I want the connection to be really secure. I only recently came across WSGI and I thought it was very easy to use, especially from within Python. The app is only used as a proxy so has little functionality, but I do want it to be as secure as possible.
Basically WSGI is just an interface for communication between a web-server software (e.i. nginx or httpd) and your python script/app which contains some code to process requests (usually either application callable or applications list of callables).
You need to enable SSL on the web-server layer. But if you don't want to involve external software you may follow #Demz's advice and try using eventlet.wrap_ssl. Please find more relevant information here.
I've already looked at other threads on this, but most don't go into enough setup detail which is where I need help.
I have an Ubuntu based VPS running with nginx, serving PHP sites through php-cgi on port 9000.
I'd like to start doing more with Python, so I've written a deployment script which I essentially want to use as a post-receive hook on my local GitLab server as my first python script. I can run this script successfully by running python script.py on the command line but in order to use this as a post-receive hook I need it be able to access it via http.
I looked at this guide on the nginx wiki but partway down is says to:
And start the django fastcgi process:
python ./manage.py runfcgi host=127.0.0.1 port=8080
Now, like I said I am pretty new to python, and I have never used the Django framework. Can anyone assit on how I am supposed to start the fastcgi server? Do I replace ./manage.py with the name of my script? Any help would be appriciated as everything I've found online refers to working with Django.
Do I replace ./manage.py with the name of my script?
No. It's highly unlikely your script is a FastCGI server, or that it can accept HTTP requests of any kind since you mention running it over the command line. (From what little I know of FastCGI, an app supporting it has to be able to handle a stream of requests coming in over stdin in a specific format, so there's definitely some plumbing involved.)
I'd say the easiest approach would be to use some web framework just to act as HTTP/FastCGI middleware. For your use a "microframework" like Flask (or even Paste but I found the documentation inscrutable) sounds like it'd work fine. The idea would be to have two interfaces to your main code, one that can handle command line arguments, and one that can handle a HTTP request, ultimately both would just call one function that actually does the work. (If you want to keep the command-line version of the app.)
The Flask documentation also mentions using uWSGI or standalone workers as deployment options. I'm not familiar with the former; the latter I wouldn't recommend for a simple, low-traffic app for the same reasons as the approach in the next paragraph.
Considering you use a VPS, you might even be able to just run the app as a standalone server process using the http.server module, but I'm not sure that's the better choice unless you absolutely want to avoid using any sort of framework. You'd have to make sure the app starts up if the server is rebooted or that it restarts when it crashes and it seems easier to just have nginx do the job of the supervisor.
UPDATE: Scratch that, it seems that nginx won't handle supervising a FastCGI worker process for you, which would've been the main advantage of the approach. In light of that it doesn't matter which of the three approaches you use since you'll have to set up a service supervisor one way or the other. I'd say go with uWSGI since flup (which is needed for Flask+FastCGI) seems abandoned since 2011, and the uWSGI protocol is apparently supported in nginx natively. Otherwise you'd need to use a different webserver than nginx, one that will manage a FastCGI worker for you. If this is an option, I'd consider Cherokee, which can be configured using a web GUI.
tl;dr: you need to write a (very simple) webapp. While it is feasible to do this without a web framework of any kind, in my opinion using one is easier, since you some (nontrivial) plumbing for free and there's a lot of guidance available on how to deploy them.
I am a PHP developer and am I want to use python for a project. What is the most common and simplest way to run python script on a server like apache or lighttpd? I am having trouble understanding how server languages that are not PHP or ASP run on servers.
In Python we have something called WSGI. But I believe this is too much for you right now. Grab a Python web framework, like Django. It comes with a web server and with good documentation on how to deploy it later. Play with it and things will start to clear up.
The longer version is that Python is a general purpose language - it's not designed for the web like PHP is. So you need a bit of work the get it to do web stuff and we already have some good frameworks for that (Django is the easiest to start with, so that is why I'm recommending it to you).
In general you should understand how the web works. It uses HTTP as a protocol for communication, which is build on top of the TCP stack, so a web application is just a server, that uses sockets (PHP has them as well as Python) and understands HTTP. Python comes with one build in - the SimpleHTTPServer, but it is not very good for production uses (it's great for development, though). This is why there are things like mod_wsgi (Python specific), FastCGI (general purpouse). Those things are basically ways for a real, production grade, web server (Apache, nginx) to talk to our python app and feed it the HTTP they get.
You can install gunicorn a Pure Python WSGI HTTP Server for UNIX. http://gunicorn.org/.
Typically, Apache is used with mod_python. See this mod_python tutorial
I am deploying a WSGI application. There are many ways to skin this cat. I am currently using apache2 with mod-wsgi, but I can see some potential problems with this.
So how can it be done?
Apache Mod-wsgi (the other mod-wsgi's seem to not be worth it)
Pure Python web server eg paste, cherrypy, Spawning, Twisted.web
as 2 but with reverse proxy from nginx, apache2 etc, with good static file handling
Conversion to other protocol such as FCGI with a bridge (eg Flup) and running in a conventional web server.
More?
I want to know how you do it, and why it is the best way to do it. I would absolutely love you to bore me with details about the whats and the whys, application specific stuff, etc.
As always: It depends ;-)
When I don't need any apache features I am going with a pure python webserver like paste etc. Which one exactly depends on your application I guess and can be decided by doing some benchmarks. I always wanted to do some but never came to it. I guess Spawning might have some advantages in using non blocking IO out of the box but I had sometimes problems with it because of the patching it's doing.
You are always free to put a varnish in front as well of course.
If an Apache is required I am usually going with solution 3 so that I can keep processes separate. You can also more easily move processes to other servers etc. I simply like to keep things separate.
For static files I am using right now a separate server for a project which just serves static images/css/js. I am using lighttpd as webserver which has great performance (in this case I don't have a varnish in front anymore).
Another useful tool is supervisord for controlling and monitoring these services.
I am additionally using buildout for managing my deployments and development sandboxes (together with virtualenv).
I would absolutely love you to bore me with details about the whats and the whys, application specific stuff, etc
Ho. Well you asked for it!
Like Daniel I personally use Apache with mod_wsgi. It is still new enough that deploying it in some environments can be a struggle, but if you're compiling everything yourself anyway it's pretty easy. I've found it very reliable, even the early versions. Props to Graham Dumpleton for keeping control of it pretty much by himself.
However for me it's essential that WSGI applications work across all possible servers. There is a bit of a hole at the moment in this area: you have the WSGI standard telling you what a WSGI callable (application) does, but there's no standardisation of deployment; no single way to tell the web server where to find the application. There's also no standardised way to make the server reload the application when you've updated it.
The approach I've adopted is to put:
all application logic in modules/packages, preferably in classes
all website-specific customisations to be done by subclassing the main Application and overriding members
all server-specific deployment settings (eg. database connection factory, mail relay settings) as class __init__() parameters
one top-level ‘application.py’ script that initialises the Application class with the correct deployment settings for the current server, then runs the application in such a way that it can work deployed as a CGI script, a mod_wsgi WSGIScriptAlias (or Passenger, which apparently works the same way), or can be interacted with from the command line
a helper module that takes care of above deployment issues and allows the application to be reloaded when the modules the application is relying on change
So what the application.py looks like in the end is something like:
#!/usr/bin/env python
import os.path
basedir= os.path.dirname(__file__)
import MySQLdb
def dbfactory():
return MySQLdb.connect(db= 'myappdb', unix_socket= '/var/mysql/socket', user= 'u', passwd= 'p')
def appfactory():
import myapplication
return myapplication.Application(basedir, dbfactory, debug= False)
import wsgiwrap
ismain= __name__=='__main__'
libdir= os.path.join(basedir, 'system', 'lib')
application= wsgiwrap.Wrapper(appfactory, libdir, 10, ismain)
The wsgiwrap.Wrapper checks every 10 seconds to see if any of the application modules in libdir have been updated, and if so does some nasty sys.modules magic to unload them all reliably. Then appfactory() will be called again to get a new instance of the updated application.
(You can also use command line tools such as
./application.py setup
./application.py daemon
to run any setup and background-tasks hooks provided by the application callable — a bit like how distutils works. It also responds to start/stop/restart like an init script.)
Another trick I use is to put the deployment settings for multiple servers (development/testing/production) in the same application.py script, and sniff ‘socket.gethostname()’ to decide which server-specific bunch of settings to use.
At some point I might package wsgiwrap up and release it properly (possibly under a different name). In the meantime if you're interested, you can see a dogfood-development version at http://www.doxdesk.com/file/software/py/v/wsgiwrap-0.5.py.
The absolute easiest thing to deploy is CherryPy. Your web application can also become a standalone webserver. CherryPy is also a fairly fast server considering that it's written in pure Python. With that said, it's not Apache. Thus, I find that CherryPy is a good choice for lower volume webapps.
Other than that, I don't think there's any right or wrong answer to this question. Lots of high-volume websites have been built on the technologies you talk about, and I don't think you can go too wrong any of those ways (although I will say that I agree with mod-wsgi not being up to snuff on every non-apache server).
Also, I've been using isapi_wsgi to deploy python apps under IIS. It's a less than ideal setup, but it works and you don't always get to choose otherwise when you live in a windows-centric world.
Nginx reverse proxy and static file sharing + XSendfile + uploadprogress_module. Nothing beats it for the purpose.
On the WSGI side either Apache + mod_wsgi or cherrypy server. I like to use cherrypy wsgi server for applications on servers with less memory and less requests.
Reasoning:
I've done benchmarks with different tools for different popular solutions.
I have more experience with lower level TCP/IP than web development, especially http implementations. I'm more confident that I can recognize a good http server than I can recognize a good web framework.
I know Twisted much more than Django or Pylons. The http stack in Twisted is still not up to this but it will be there.
I'm using Google App Engine for an application I'm developing. It runs WSGI applications.
Here's a couple bits of info on it.
This is the first web-app I've ever really worked on, so I don't have a basis for comparison, but if you're a Google fan, you might want to look into it. I've had a lot of fun using it as my framework for learning.
TurboGears (2.0)
TurboGears 2.0 is leaving Beta within the next month (has been in it for plenty of time). 2.0 improves upon 1.0 series and attempts to give you best-of-breed WSGI stack, so it makes some default choices for you, if you want the least fuss.
it has the tg* tools for testing and deployment in 1.x series, but now transformed to paster equivalents in 2.0 series, which shoud seem familiar if you've expermiented with pylons.
tg-admin quickstart —> paster quickstart
tg-admin info —> paster tginfo
tg-admin toolbox –> paster toolbox
tg-admin shell –> paster shell
tg-admin sql create –> paster setup-app development.ini
Pylons
It you'd like to be more flexible in your WSGI stack (choice of ORM, choice of templater, choice of form-ing), Pylons is becoming the consolidated choice. This would be my recommended choice, since it offers excellent documentation and allows you to experiment with different components.
It is a pleasure to work with as a result, and works on under Apache (production deployment) or stand-alone (helpful for testing and experimenting stage).
so it follows, you can do both with Pylons:
2 option for testing stage (python standalone)
4 for scalable production purposes (FastCGI, assuming the database you choose can keep up)
The Pylons admin interface is very similar to TurboGears. Here's a toy standalone example:
$ paster create -t pylons helloworld
$ cd helloworld
$ paster serve --reload development.ini
for production-class deployment, you could refer to the setup guide of Apache + FastCGI + mod_rewrite is available here. this would scale up to most needs.
Apache httpd + mod_fcgid using web.py (which is a wsgi application).
Works like a charm.
We are using pure Paste for some of our web services. It is easy to deploy (with our internal deployment mechanism; we're not using Paste Deploy or anything like that) and it is nice to minimize the difference between production systems and what's running on developers' workstations. Caveat: we don't expect low latency out of Paste itself because of the heavyweight nature of our requests. In some crude benchmarking we did we weren't getting fantastic results; it just ended up being moot due to the expense of our typical request handler. So far it has worked fine.
Static data has been handled by completely separate (and somewhat "organically" grown) stacks, including the use of S3, Akamai, Apache and IIS, in various ways.
Apache+mod_wsgi,
Simple, clean. (only four lines of webserver config), easy for other sysadimns to get their head around.
Can I write web application that I can host on Windows(IIS web server) and Linux (Apache or lighttpd) without any changes?
CGI? Maybe something new? WSGI | FastCGI ?
Yes you can. But you can also use apache on windows. If you go the IIS way there's only CGI and it's pretty hard to set up. You can also use python based server like CherryPy which is pretty good and will work on all platforms with python.
Some frameworks like django support both CGI and WSGI, so you don't have to worry about the details of WSGI or CGI much.
If you ask me, WSGI is the future for python web apps.
web.py includes a server... It will do the trick for small jobs.
By the way, Apache works on windows.
Yes, if you use CGI, FastCGI or depending on your framework, even a self-contained web server (so IIS and Apache would be a reverse-proxy) then that would all work.
The difference will be the configuration of the OS-specific servers, and also your Python environment on each OS. So you may find yourself doing a small bit of work at the beginning to make sure your paths are right, etc.
A rather big Python based web framework is ZOPE.
Zope is an open source application server for building content management systems, intranets, portals, and custom applications. The Zope community consists of hundreds of companies and thousands of developers all over the world, working on building the platform and Zope applications. Zope is written in Python, a highly-productive, object-oriented scripting language
ZOPE is available on Linux and Windows, and you can use Python to write your Zope Web Apps (it includes a simpler templating system, too).
consider also the possibility of using web2Py, or XML-RPC implementation, or Twisted...
Writing python web apps is a topic on itself, but I would say that by default, it will be portable on multiple servers / platforms.
When developping python web applications, you will often use frameworks that provide their own web server. For performance reasons, you might want to place it behind apache, but it is not even necessary, however, you might get a performance boost by placing it behind an apache server.
Some of the most popular frameworks for web python are : Plone, Zope, CherryPy and TurboGears, only to name a few.
Under apache, you could also use python server pages through mod_python, and since apache runs on windows too, this would aslo be portable.