We Keep Coding

WSGI process reload modules

I'm trying to trigger a reload of my WSGI process when any file changes in the folder where it and all it's dependent modules are located.
I've read http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode and I thought I understood it, but this intermittent staleness makes me doubt myself. I'm running in daemon mode like this:
DocumentRoot /usr/local/www/mysite.com/public_html
WSGIScriptAlias /api /usr/local/www/mysite.com/server/server.py
WSGIPassAuthorization On
WSGIDaemonProcess mysite.com threads=15 python-path=/usr/local/www/mysite.com/server
WSGIProcessGroup mysite.com
server.py is the main WSGI application file and all the modules it imports (which are likely to change) are in the same folder as it.
This is what I've come up with and it seems to work most of the time but occasionally I get stuck modules (where I make a change to a source file and the process restarts but it seems to load the old code). Some caching issue? If the process is restarted, I thought the import would get the fresh code? I really want to avoid using reload(). Restarting Apache always unsticks it and picks up the changes.
#!/bin/bash
while true; do
inotifywait . -e modify,create --exclude server.py -q
if (($? == 0)); then
touch server.py
else
exit 0
fi
done
Am I right in thinking that this (or something like it) should work or am I barking up the wrong tree?
The root WSGI file (server.py) is quite small:
print "Server restart"
import sys, types, os, web
import api
import user # /api/user
import list # /api/list
#api.path('/info')
class info(api.Handler):
#api.params({
'params': {'echo': unicode }
})
def Post(self, data):
return api.JSON({'info': 'foo', 'echo': data['echo']})
#api.path('/(.*)')
class notfound(api.Handler):
def Get(self):
api.error('404 page not found')
app = web.application(api.urls(), globals())
if __name__ == '__main__':
app.run()
else:
application = app.wsgifunc()

How can I make a Python package which serves a website with Flask to work with Apache?

I have written a Python package hwrt (see installation instructions if you want to try it) which serves a website when executed with
$ hwrt serve
2014-12-04 20:27:07,182 INFO * Running on http://127.0.0.1:5000/
2014-12-04 20:27:07,183 INFO * Restarting with reloader
I would like to let it run on http://www.pythonanywhere.com, but when I start it there I get
19:19 ~ $ hwrt serve
2014-12-04 19:19:59,282 INFO * Running on http://127.0.0.1:5000/
Traceback (most recent call last):
File "/home/MartinThoma/.local/bin/hwrt", line 108, in <module>
main(args)
File "/home/MartinThoma/.local/bin/hwrt", line 102, in main
serve.main()
File "/home/MartinThoma/.local/lib/python2.7/site-packages/hwrt/serve.py", line 95, in main
app.run()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 739, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 613, in run_simple
test_socket.bind((hostname, port))
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
I only found this in the documentation:
Flask
never use app.run(), it will break your webapp. Just import the
app into your wsgi file...
By searching for wsgi file, I found mod_wsgi (Apache). However, I don't understand how I can adjust my current minimalistic Flask application to work with that. Currently, the script behind hwrt serve is:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Start a webserver which can record the data and work as a classifier."""
import pkg_resources
from flask import Flask, request, render_template
from flask_bootstrap import Bootstrap
import os
import json
# hwrt modules
import hwrt
import hwrt.utils as utils
def show_results(results, n=10):
"""Show the TOP n results of a classification."""
import nntoolkit
classification = nntoolkit.evaluate.show_results(results, n)
return "<pre>" + classification.replace("\n", "<br/>") + "</pre>"
# configuration
DEBUG = True
template_path = pkg_resources.resource_filename('hwrt', 'templates/')
# create our little application :)
app = Flask(__name__, template_folder=template_path)
Bootstrap(app)
app.config.from_object(__name__)
#app.route('/', methods=['POST', 'GET'])
def show_entries():
heartbeat = request.args.get('heartbeat', '')
return heartbeat
#app.route('/interactive', methods=['POST', 'GET'])
def interactive():
if request.method == 'POST':
raw_data_json = request.form['drawnJSON']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
print(model)
results = utils.evaluate_model_single_recording(model, raw_data_json)
# Show classification page
page = show_results(results, n=10)
page += 'back'
return page
else:
# Page where the user can enter a recording
return render_template('canvas.html')
def get_json_result(results, n=10):
s = []
for res in results[:min(len(results), n)]:
s.append({res['semantics']: res['probability']})
return json.dumps(s)
#app.route('/worker', methods=['POST', 'GET'])
def worker():
# Test with
# wget --post-data 'classify=%5B%5B%7B%22x%22%3A334%2C%22y%22%3A407%2C%22time%22%3A1417704378719%7D%5D%5D' http://127.0.0.1:5000/worker
if request.method == 'POST':
raw_data_json = request.form['classify']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
results = utils.evaluate_model_single_recording(model, raw_data_json)
return get_json_result(results, n=10)
else:
# Page where the user can enter a recording
return "Classification Worker (Version %s)" % hwrt.__version__
def get_parser():
"""Return the parser object for this script."""
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
parser = ArgumentParser(description=__doc__,
formatter_class=ArgumentDefaultsHelpFormatter)
return parser
def main():
app.run()
if __name__ == '__main__':
main()

Ok, a not so non-sequitur answer to your question is around what mod_wsgi does to interface with your app. A typical flask app would look something like this:
from flask import Flask
app = Flask(__name__)
app.route("/")
def hello():
return "Holy moly that tunnel was bright.. said Bit to NIC"
if __name__ == "__main__":
app.run()
Unfortunately, Apache has no way to know what to do with this (though the app would run happily on its own). In order to get the app and Apache to play nice together we're going to use something called mod_wsgi. What Mod_WSGI does that's important to us, is that it provides a known interface (a file type called wsgi) that's going to wrap our application and initialize it so that we can serve it through Apache.
I'm going to assume you are using a python virtual environment, but if you aren't you can omit the step that deals with this in the instructions below. If you're curious why virtual environments are so great, feel free read about the python ecosystem.
Also - you can include an extra flag (assuming you are running wsgi as a daemon) to automatically reload the daemon whenever you touch or alter your wsgi file. This is quite useful during development and debugging so I'll include is below.
Anyway, let's get started. I'll break this down to steps below.
Configuring Apache for mod_wsgi
Enable mod_wsgi in Apache:
sudo apt-get install libapache2-mod-wsgi
Edit your /etc/apache2/sites-available/<yoursite>.conf.
<VirtualHost interface:port>
WSGIDaemonProcess yourapp user=someUser processes=2 threads=15
WSGIProcessGroup yourapp
# In this case / refers to whatever relative URL path hosts flask
WSGIScriptAlias / /absolute/path/to/yourapp.wsgi
<Directory /path/to/your/main/py/file/ >
# Use good judgement here when server hardening, this assumes dev env
Order allow,deny
Allow from all
Require all granted
#The below enables 'auto-reload' of WSGI
WSGIScriptReloading On
</Directory>
# If you want to serve static files as well and bypass flask in those cases
Alias /relative/url/to/static/content/
<Directory /absolute/path/to/static/root/directory/>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Create your yourapp.wsgi file and put it in the appropriate place: Be wary of file permissions!
#!/usr/bin/python
import sys
import logging
# Activate virtual environment.
# If you are not using venv, skip this.
# But you really should be using it!
activate_this = "/path/to/venv/bin/activate_this.py"
execfile(activate_this, dict(__file__=activate_this))
# Handle logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/path/to/your/main/py/file/")
from YourMainPyFileName import app as application
application.secret_key = "your_secret_key"
Reload Apache and troubleshoot problems. I set this up probably every few weeks for a different project or idea I have and... I usually have to fix one thing or another when doing it from scratch. Don't despair though! Flask has great documentation on this.
Once you've done all this you should be at a place where flask runs all on its own. The sample flask app above is the actual code I use to verify everything works whenever I set this up.
This was left here in case it's some use, but is not really directly related to the question...
The answer here is to use x-send-file. This takes advantage of letting Apache do what it's good at (serving static content), while at the same time first letting flask (or other python framework) do it's work first. I do this often to let flask handle my auth layers in single page web apps and have so far been happy with the results.
Doing so requires two things:
First - Enable xsendfile on Apache2 sudo apt-get install libapache2-mod-xsendfile.
Second - Alter your apache2 configuration so allow x-send-file headers:
Alter your conf file in /etc/apache2/sites-available/<yoursite>.conf and add...
XSendFile On
XSendFilePath /path/to/static/directory
This can be entered top level within the <Virtualhost></Virtualhost> tags.
Don't forget to restart Apache sudo service apache2 restart.
Finally - Configure your flask app to use x-send-file in your app.py file:
app.user_x_sendfile = True
Note: Must be done after app initialization. Consequently can also be passed as an initialization parameter.
Flask has documentation on this (excerpt below):
use_x_sendfile
Enable this if you want to use the X-Sendfile feature. Keep in mind that the server has to support this. This only affects files sent with the send_file() method.
New in version 0.2.
This attribute can also be configured from the config with the USE_X_SENDFILE configuration key. Defaults to False.

I ran into a similar issue #moose was having. Getting connection refused and couldnt even telnet localhost 5000.
Turns out theres a ports.conf file i had to add Listen 5000
Happy days.

How to serve static files from my CherryPy applications

I have written a simple web app and I got stuck trying to figure out how to server static files. My static files folder is in a different folder /usr/lib/python2.6/site-packages/web3/static Below is my code, how do I add this static files folder configuration to my app.
#!/usr/bin/env python
import sys
import my_web.settings
from django.core.management import execute_from_command_line
from django.core.management import call_command
import os
import tempfile
import cherrypy.wsgiserver
import django.core.handlers.wsgi
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "my_web.settings")
# Since we use a throwaway database, it needs to be initialized
# every time Django starts.
if sys.argv[1] in ('runserver', 'shell'):
with tempfile.NamedTemporaryFile() as dbFile:
my_web.settings.DATABASES['default']['NAME'] = dbFile.name
call_command('syncdb', interactive=False)
server = cherrypy.wsgiserver.CherryPyWSGIServer(
('0.0.0.0', 8080), django.core.handlers.wsgi.WSGIHandler(),
server_name='localhost', numthreads = 10
)
try:
server.start()
except KeyboardInterrupt:
server.stop()

try this... and be sure the user running the app has permission to access the static path.
cherrypy.config.update({'tools.staticdir.on': True,
'tools.staticdir.dir': '/usr/lib/python2.6/site-packages/web3/static'
})
Hope this helps!

Sylvain Hellegouarch has posted a Django CherryPy integration recipe that nicely solves the problem of serving static Django content. See https://bitbucket.org/Lawouach/cherrypy-recipes/src/c8290261eefb82cb5694930f7236606082a941ff/frameworks/django_/?at=default
Just drop the three python files (init.py, httplogger.py, and djangoplugin.py) into the directory above your app and change the name of the directory holding your settings.py file in init.py to match the directory in your own source code tree.
I also removed the two WebSocketPlugin lines from init.py since that was not relevant for my django app.
There is some discussion about the recipe on Sylvain's blog at http://www.defuze.org/archives/262-hosting-a-django-application-on-a-cherrypy-server.html

Is there an alternative of RewriteRule / .htaccess for a Python http.server.HTTPServer?

Can my server module (with http.server.HTTPServer) use something like the RewriteRule for redirect all traffic into a single cgi script? I'd like to be able to do what's shown here in this other question, but for my python server.
Can it be done using something like .htaccess, or is there another way?
Also, can this be done even for a simple localhost development server?
I am serving files for development via, for example, http://localhost:8000/html/index.html, and I would like to hide the /html subfolder from the URL even in development.
How can that be achieved?

You can use a custom script to initialize your server and define your Routes in it, such as suggested in this article:
Python 2:
server.py
import os
import posixpath
import urllib
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
# modify this to add additional routes
ROUTES = ( <- this is the "good stuff", make aliases for your paths
# [url_prefix , directory_path]
['/media', '/var/www/media'],
['', '/var/www/site'] # empty string for the 'default' match
)
class RequestHandler(SimpleHTTPRequestHandler):
def translate_path(self, path):
"""translate path given routes"""
# set default root to cwd
root = os.getcwd()
# look up routes and set root directory accordingly
for pattern, rootdir in ROUTES:
if path.startswith(pattern):
# found match!
path = path[len(pattern):] # consume path up to pattern len
root = rootdir
break
# normalize path and prepend root directory
path = path.split('?',1)[0]
path = path.split('#',1)[0]
path = posixpath.normpath(urllib.unquote(path))
words = path.split('/')
words = filter(None, words)
path = root
for word in words:
drive, word = os.path.splitdrive(word)
head, word = os.path.split(word)
if word in (os.curdir, os.pardir):
continue
path = os.path.join(path, word)
return path
if __name__ == '__main__':
BaseHTTPServer.test(RequestHandler, BaseHTTPServer.HTTPServer)
Then run your script:
python server.py
Python 3:
In Python 3 the BaseHTTPServer and the SimpleHTTPServer modules have been merged into the http.server module. So you will have to modify the above script as follows:
Change the
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
to
import http.server
(if you do this then the calls should be modified to http.server.SimpleHTTPRequest etc.)
or
from http.server import BaseHTTPServer, SimpleHTTPServer, SimpleHTTPRequestHandler
(with this option calls remain the same as the original script)
then run the server script: python server.py
IN YOUR CASE:
You should modify the ROUTES variable to suite your needs. For example, you want to hide the /html folder from your url:
ROUTES = (
['', '/exact/path/to/folder/html'],
['/another_url_path', '/exact/path/to/another/folder'],
...
)
Now if you hit: http://localhost:8000/index.html you will be in your home page.
Note:
This script by default will serve the contained files in the folder that is in when executed to the domain url (ex. I have server.py on the Documents folder, then when I run it, http://localhost:8000 url will server my Documents folder).
You can change this behavior by the Routes (see the 'default' match comment on the code) or by placing the script in your projects root folder and start it from there.

John Moutafis's answer was helpful to get me started, but needed some fine tuning to run on python3, beyond his comments about imports.
The imports should be
from http.server import HTTPServer, SimpleHTTPRequestHandler
and you'll also need to change the urllib import to:
from urllib.parse import unquote
Then the main should be something like:
if __name__ == '__main__':
myServer = HTTPServer(('0.0.0.0', 8000), RequestHandler)
print("Ready to begin serving files.")
try:
myServer.serve_forever()
except KeyboardInterrupt:
pass
myServer.server_close()
print("Exiting.")

Is it possible to have SimpleHTTPServer serve files from two different directories?

If I do python -m SimpleHTTPServer it serves the files in the current directory.
My directory structure looks like this:
/protected/public
/protected/private
/test
I want to start the server in my /test directory and I want it to serve files in the /test directory. But I want all requests to the server starting with '/public' to be pulled from the /protected/public directory.
e.g.a request to http://localhost:8000/public/index.html would serve the file at /protected/public/index.html
Is this possible with the built in server or will I have to write a custom one?

I think it is absolutely possible to do that. You can start the server inside /test directory and override translate_path method of SimpleHTTPRequestHandler as follows:
import BaseHTTPServer
import SimpleHTTPServer
server_address = ("", 8888)
PUBLIC_RESOURCE_PREFIX = '/public'
PUBLIC_DIRECTORY = '/path/to/protected/public'
class MyRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
def translate_path(self, path):
if self.path.startswith(PUBLIC_RESOURCE_PREFIX):
if self.path == PUBLIC_RESOURCE_PREFIX or self.path == PUBLIC_RESOURCE_PREFIX + '/':
return PUBLIC_DIRECTORY + '/index.html'
else:
return PUBLIC_DIRECTORY + path[len(PUBLIC_RESOURCE_PREFIX):]
else:
return SimpleHTTPServer.SimpleHTTPRequestHandler.translate_path(self, path)
httpd = BaseHTTPServer.HTTPServer(server_address, MyRequestHandler)
httpd.serve_forever()
Hope this helps.

I think I have found the answer to this, basically it involves changing the current working directory, starting the server and then returning back to your original working directory.
This is how I achieved it, I've commented out two sets of options for you, as the solution for me was just moving to a folder within my app directory and then back up one level to the original app directory. But, you might want to go to an entire other directory in your file system and then return someplace else or not at all.
#Setup file server
import SimpleHTTPServer
import SocketServer
import os
PORT = 5002
# -- OPTION 1 --
#os.chdir(os.path.join(os.path.abspath(os.curdir),'PATH_TO_FOLDER_IN_APP_DIR'))
# -- OPTION 2 --
#os.chdir('PATH_TO_ROOT_DIRECTORY')
Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
httpd = SocketServer.TCPServer(("", PORT), Handler)
print "serving at port", PORT
httpd.serve_forever()
# -- OPTION 1 --
#os.chdir(os.path.abspath('..'))
# -- OPTION 2 --
#os.chdir('PATH_TO_ORIGINAL_WORKING_DIR')
Let me know how it works out!

I do not believe SimpleHTTPServer has this feature, however if you use a symbolic link inside of /test that points to /protected/public, that should effectively do the same thing.