We Keep Coding

What gets executed when running a python app on OpenShift?

I'm confused as to what file gets initially run when running a Flask application on OpenShift. There is a wsgi.py file that creates a wsgiref.simple_server but the comment above it says it is only for testing. Can anyone explain how the application get executed? I'm having a hard time wrapping my head around the process and where exactly the web server comes in to the picture vs the application code.
wsgi.py:
import os
virtenv = os.path.join(os.environ.get('OPENSHIFT_PYTHON_DIR','.'), 'virtenv')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
pass
#
# IMPORTANT: Put any additional includes below this line. If placed above this
# line, it's possible required libraries won't be in your searchable path
#
from flaskapp import app
#
# Below for testing only
#
if __name__ == '__main__':
from wsgiref.simple_server import make_server
host = app.config['HOST_NAME']
ip = app.config['HOST_IP']
httpd = make_server(host, ip, app)
httpd.handle_request()
flaskapp.py
import os
virtenv = os.path.join(os.environ.get('OPENSHIFT_PYTHON_DIR','.'), 'virtenv')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
pass
#
# IMPORTANT: Put any additional includes below this line. If placed above this
# line, it's possible required libraries won't be in your searchable path
#
from flaskapp import app
#
# Below for testing only
#
if __name__ == '__main__':
from wsgiref.simple_server import make_server
host = app.config['HOST_NAME']
ip = app.config['HOST_IP']
httpd = make_server(host, ip, app)
httpd.handle_request()

See my answer here from another SO question.
How to create app using pyramid into openshift?
I think my last commit in my github example uses the "wsgi.py" entry point. I prefer using "app.py" as the entry point. I find it is less problematic and more reliable.
Go ahead and rename "app.py disabled" to "app.py" and delete the wsgi.py.
I'm using pyramid instead of flask in the example but the setup is similar.

How can I make a Python package which serves a website with Flask to work with Apache?

I have written a Python package hwrt (see installation instructions if you want to try it) which serves a website when executed with
$ hwrt serve
2014-12-04 20:27:07,182 INFO * Running on http://127.0.0.1:5000/
2014-12-04 20:27:07,183 INFO * Restarting with reloader
I would like to let it run on http://www.pythonanywhere.com, but when I start it there I get
19:19 ~ $ hwrt serve
2014-12-04 19:19:59,282 INFO * Running on http://127.0.0.1:5000/
Traceback (most recent call last):
File "/home/MartinThoma/.local/bin/hwrt", line 108, in <module>
main(args)
File "/home/MartinThoma/.local/bin/hwrt", line 102, in main
serve.main()
File "/home/MartinThoma/.local/lib/python2.7/site-packages/hwrt/serve.py", line 95, in main
app.run()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 739, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 613, in run_simple
test_socket.bind((hostname, port))
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
I only found this in the documentation:
Flask
never use app.run(), it will break your webapp. Just import the
app into your wsgi file...
By searching for wsgi file, I found mod_wsgi (Apache). However, I don't understand how I can adjust my current minimalistic Flask application to work with that. Currently, the script behind hwrt serve is:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Start a webserver which can record the data and work as a classifier."""
import pkg_resources
from flask import Flask, request, render_template
from flask_bootstrap import Bootstrap
import os
import json
# hwrt modules
import hwrt
import hwrt.utils as utils
def show_results(results, n=10):
"""Show the TOP n results of a classification."""
import nntoolkit
classification = nntoolkit.evaluate.show_results(results, n)
return "<pre>" + classification.replace("\n", "<br/>") + "</pre>"
# configuration
DEBUG = True
template_path = pkg_resources.resource_filename('hwrt', 'templates/')
# create our little application :)
app = Flask(__name__, template_folder=template_path)
Bootstrap(app)
app.config.from_object(__name__)
#app.route('/', methods=['POST', 'GET'])
def show_entries():
heartbeat = request.args.get('heartbeat', '')
return heartbeat
#app.route('/interactive', methods=['POST', 'GET'])
def interactive():
if request.method == 'POST':
raw_data_json = request.form['drawnJSON']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
print(model)
results = utils.evaluate_model_single_recording(model, raw_data_json)
# Show classification page
page = show_results(results, n=10)
page += 'back'
return page
else:
# Page where the user can enter a recording
return render_template('canvas.html')
def get_json_result(results, n=10):
s = []
for res in results[:min(len(results), n)]:
s.append({res['semantics']: res['probability']})
return json.dumps(s)
#app.route('/worker', methods=['POST', 'GET'])
def worker():
# Test with
# wget --post-data 'classify=%5B%5B%7B%22x%22%3A334%2C%22y%22%3A407%2C%22time%22%3A1417704378719%7D%5D%5D' http://127.0.0.1:5000/worker
if request.method == 'POST':
raw_data_json = request.form['classify']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
results = utils.evaluate_model_single_recording(model, raw_data_json)
return get_json_result(results, n=10)
else:
# Page where the user can enter a recording
return "Classification Worker (Version %s)" % hwrt.__version__
def get_parser():
"""Return the parser object for this script."""
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
parser = ArgumentParser(description=__doc__,
formatter_class=ArgumentDefaultsHelpFormatter)
return parser
def main():
app.run()
if __name__ == '__main__':
main()

Ok, a not so non-sequitur answer to your question is around what mod_wsgi does to interface with your app. A typical flask app would look something like this:
from flask import Flask
app = Flask(__name__)
app.route("/")
def hello():
return "Holy moly that tunnel was bright.. said Bit to NIC"
if __name__ == "__main__":
app.run()
Unfortunately, Apache has no way to know what to do with this (though the app would run happily on its own). In order to get the app and Apache to play nice together we're going to use something called mod_wsgi. What Mod_WSGI does that's important to us, is that it provides a known interface (a file type called wsgi) that's going to wrap our application and initialize it so that we can serve it through Apache.
I'm going to assume you are using a python virtual environment, but if you aren't you can omit the step that deals with this in the instructions below. If you're curious why virtual environments are so great, feel free read about the python ecosystem.
Also - you can include an extra flag (assuming you are running wsgi as a daemon) to automatically reload the daemon whenever you touch or alter your wsgi file. This is quite useful during development and debugging so I'll include is below.
Anyway, let's get started. I'll break this down to steps below.
Configuring Apache for mod_wsgi
Enable mod_wsgi in Apache:
sudo apt-get install libapache2-mod-wsgi
Edit your /etc/apache2/sites-available/<yoursite>.conf.
<VirtualHost interface:port>
WSGIDaemonProcess yourapp user=someUser processes=2 threads=15
WSGIProcessGroup yourapp
# In this case / refers to whatever relative URL path hosts flask
WSGIScriptAlias / /absolute/path/to/yourapp.wsgi
<Directory /path/to/your/main/py/file/ >
# Use good judgement here when server hardening, this assumes dev env
Order allow,deny
Allow from all
Require all granted
#The below enables 'auto-reload' of WSGI
WSGIScriptReloading On
</Directory>
# If you want to serve static files as well and bypass flask in those cases
Alias /relative/url/to/static/content/
<Directory /absolute/path/to/static/root/directory/>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Create your yourapp.wsgi file and put it in the appropriate place: Be wary of file permissions!
#!/usr/bin/python
import sys
import logging
# Activate virtual environment.
# If you are not using venv, skip this.
# But you really should be using it!
activate_this = "/path/to/venv/bin/activate_this.py"
execfile(activate_this, dict(__file__=activate_this))
# Handle logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/path/to/your/main/py/file/")
from YourMainPyFileName import app as application
application.secret_key = "your_secret_key"
Reload Apache and troubleshoot problems. I set this up probably every few weeks for a different project or idea I have and... I usually have to fix one thing or another when doing it from scratch. Don't despair though! Flask has great documentation on this.
Once you've done all this you should be at a place where flask runs all on its own. The sample flask app above is the actual code I use to verify everything works whenever I set this up.
This was left here in case it's some use, but is not really directly related to the question...
The answer here is to use x-send-file. This takes advantage of letting Apache do what it's good at (serving static content), while at the same time first letting flask (or other python framework) do it's work first. I do this often to let flask handle my auth layers in single page web apps and have so far been happy with the results.
Doing so requires two things:
First - Enable xsendfile on Apache2 sudo apt-get install libapache2-mod-xsendfile.
Second - Alter your apache2 configuration so allow x-send-file headers:
Alter your conf file in /etc/apache2/sites-available/<yoursite>.conf and add...
XSendFile On
XSendFilePath /path/to/static/directory
This can be entered top level within the <Virtualhost></Virtualhost> tags.
Don't forget to restart Apache sudo service apache2 restart.
Finally - Configure your flask app to use x-send-file in your app.py file:
app.user_x_sendfile = True
Note: Must be done after app initialization. Consequently can also be passed as an initialization parameter.
Flask has documentation on this (excerpt below):
use_x_sendfile
Enable this if you want to use the X-Sendfile feature. Keep in mind that the server has to support this. This only affects files sent with the send_file() method.
New in version 0.2.
This attribute can also be configured from the config with the USE_X_SENDFILE configuration key. Defaults to False.

I ran into a similar issue #moose was having. Getting connection refused and couldnt even telnet localhost 5000.
Turns out theres a ports.conf file i had to add Listen 5000
Happy days.

How do I deal with URL reroutes in Python Bottle and Apache .htaccess?

I am currently trying to create a simple standalone application using Python Bottle.
My entire project is under pytest/, where I have dispatch.fcgi and .htaccess.
dispatch.fcgi:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import bottle
import os
from bottle import route, run, view
#route('<foo:path>')
#view('index')
def pytest(foo = ''):
return dict(foo=foo)
APP_ROOT = os.path.abspath(os.path.dirname(__file__))
bottle.TEMPLATE_PATH.append(os.path.join(APP_ROOT, 'templates'))
app = bottle.default_app()
if __name__ == '__main__':
from flup.server.fcgi import WSGIServer
WSGIServer(app).run()
.htaccess:
DirectoryIndex dispatch.fcgi
The following URLs give me the corresponding values of foo:
url.com/pytest/
> /pytest/
url.com/pytest/dispatch.fcgi
> /pytest/dispatch.fcgi
url.com/pytest/dispatch.fcgi/
> /
url.com/pytest/dispatch.fcgi/foo/bar
> /foo/bar
url.com/pytest/dispatch.fcgi/pytest/
> /pytest/
How can I make the URLs uniform? Should I deal with the rerouting with the .htaccess file or within the Python code? What would be considered most pythonic, or best practices?
I am running Python 2.6.6, Bottle 0.11.6, Flup 1.0.2, and Apache 2.2.24. I would also like to point out that I'm using shared hosting, and mod_wsgi is out of the question (if that makes a difference).
EDIT
This is what I expect to see:
url.com/pytest/
> <redirect to url.com/pytest/dispatch.fcgi>
url.com/pytest/dispatch.fcgi
> <empty string>
url.com/pytest/dispatch.fcgi/
> /
url.com/pytest/dispatch.fcgi/foo/bar
> /foo/bar
url.com/pytest/dispatch.fcgi/pytest/
> /pytest/
If there is a more efficient way of tackling this problem, please let me know.

Couple of thoughts. Hopefully some or all of these will help.
1) You can do the redirect from '/' to '/pytest/dispatch.fcgi' like this:
#route('/')
def home():
bottle.redirect('/pytest/dispatch.fcgi')
2) Can you use ScriptAlias instead of DirectoryIndex? I see you're on a shared environment, so I'm not sure. My bottle/apache servers use ScriptAlias (or WSGIScriptAlias) and it works perfectly there; and it'd make more clear the way your code interacts with apache.
3) If worse came to worst, could you hackishly detect the case where foo == '/pytest/dispatch.fcgi' and act accordingly? (E.g., treat it as empty string.)
Hope this helps. Please keep us posted!

Bottle seems to be confused because it expects a trailing slash, followed by parameters. For that reason I changed my .htaccess file to read like this:
DirectoryIndex dispatch.fcgi/
Another option would be to have all errors fall back onto the dispatch script. That can be done with mod_rewrite:
<IfModule mod_rewrite.c>
Options -MultiViews
# rewrite for current folder
RewriteEngine On
RewriteBase /pytest
# redirect to front controller
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ dispatch.fcgi/ [R=301,QSA,L]
</IfModule>
or FallbackResource:
FallbackResource /pytest/dispatch.fcgi/

Easy application logging/debugging with nginx, uwsgi, flask?

I'm not looking to turn on the dangerous debugging console, but my application is getting a 500 error and doesn't seem to be writing any output for me to investigate more deeply.
I saw this exchange on the mailing list, which led me to this page on logging errors.
However, I still find this very confusing and have a couple of questions:
(1) In which file should the stuff below go?
ADMINS = ['yourname#example.com']
if not app.debug:
import logging
from logging.handlers import SMTPHandler
mail_handler = SMTPHandler('127.0.0.1',
'server-error#example.com',
ADMINS, 'YourApplication Failed')
mail_handler.setLevel(logging.ERROR)
app.logger.addHandler(mail_handler)
...assuming the "getting bigger" file pattern for larger applications? __init__.py? config.py? run.py?
(2) I am overwhelmed by options there, and can't tell which I should use. Which loggers should I turn on, with what settings, to replicate the local python server debug I get to stdout when I run run.py? I find that default, local output stream very useful, more so than the interactive debugger in the page. Does anyone have a pattern they could share on setting up something replicating this with an nginx deployment, outputting to a log?
(3) Is there anything I need to change, not at the flask level, but in nginx, say in my /etc/nginx/sites-available/appname file, to enable logging?
UPDATE
Specifically, I'm looking for information like I get when python runs locally as to why, say, a package isn't working, or where my syntax error might be, or what variable doesn't exist:
$ python run.py
Traceback (most recent call last):
File "run.py", line 1, in <module>
from myappname import app
File "/home/me/myappname/myappname/__init__.py", line 27, in <module>
file_handler.setLevel(logging.debug)
File "/usr/lib/python2.7/logging/__init__.py", line 710, in setLevel
self.level = _checkLevel(level)
File "/usr/lib/python2.7/logging/__init__.py", line 190, in _checkLevel
raise TypeError("Level not an integer or a valid string: %r" % level)
When I run flask on a server, I never see this. I just get a uWSGI error in the browser, and have no idea which code was problematic. I would just like something like the above to be written to a file.
I notice also that setting the following logging didn't really write much to file, even when I turn the log way up to the DEBUG level:
from logging import FileHandler
file_handler = FileHandler('mylog.log')
file_handler.setLevel(logging.DEBUG)
app.logger.addHandler(file_handler)
mylog.log is blank, even when my application errors out.
I'll also add that I've tried to set debug = True in the following ways, in __init__.py:
app = Flask(__name__)
app.debug = True
app.config['DEBUG'] = True
from werkzeug.debug import DebuggedApplication
app.wsgi_app = DebuggedApplication(app.wsgi_app, True)
app.config.from_object('config')
app.config.update(DEBUG=True)
app.config['DEBUG'] = True
if __name__ == '__main__':
app.run(debug=True)
While in my config.py file, I have...
debug = True
Debug = True
DEBUG = True
Yet, no debugging happens, and without logging or debugging, this is rather hard to track down. Errors simply terminate the application with the un-useful browser message:
uWSGI Error
Python application not found

Set config['PROPAGATE_EXCEPTIONS'] to True when running app in production and you want tracebacks to be logged into log files. (I haven't tried with SMTP handler, though..)

The part where you create handlers, add to loggers etc. should be in the if __name__ == '__main__' clause, i.e. your main entry point. I assume that would be run.py.
I'm not sure I can answer this - it depends on what you want. I'd advise looking at the logging tutorial to see the various options available.
I don't believe you need to change anything at the nginx level.
Update: You might want to have an exception clause that covers uncaught exceptions, e.g.
if __name__ == '__main__':
try:
app.run(debug=True)
except Exception:
app.logger.exception('Failed')
which should write the traceback of any exception which occurred in app.run() to the log.

I know that this is a VERY old post, but I ran into the issue now, and it took me a bit to find the solution. Flask sends errors to the server. I was running Gunicorn with an upstart script on Ubuntu 14.04 LTS, and the place where I found the error logs was as follows:
/var/log/upstart/myapp.log
http://docs.gunicorn.org/en/stable/deploy.html#upstart
Just in case some other poor soul ends up in this situation.

CherryPy3 and IIS 6.0

I have a small Python web application using the Cherrypy framework. I am by no means an expert in web servers.
I got Cherrypy working with Apache using mod_python on our Ubuntu server. This time, however, I have to use Windows 2003 and IIS 6.0 to host my site.
The site runs perfectly as a stand alone server - I am just so lost when it comes to getting IIS running. I have spent the past day Googling and blindly trying any and everything to get this running.
I have all the various tools installed that websites have told me to (Python 2.6, CherrpyPy 3, ISAPI-WSGI, PyWin32) and have read all the documentation I can. This blog was the most helpful:
http://whatschrisdoing.com/blog/2008/07/10/turbogears-isapi-wsgi-iis/
But I am still lost as to what I need to run my site. I can't find any thorough examples or how-to's to even start with. I hope someone here can help!
Cheers.

I run CherryPy behind my IIS sites. There are several tricks to get it to work.
When running as the IIS Worker Process identity, you won't have the same permissions as you do when you run the site from your user process. Things will break. In particular, anything that wants to write to the file system will probably not work without some tweaking.
If you're using setuptools, you probably want to install your components with the -Z option (unzips all eggs).
Use win32traceutil to track down problems. Be sure that in your hook script that you're importing win32traceutil. Then, when you're attempting to access the web site, if anything goes wrong, make sure it gets printed to standard out, it'll get logged to the trace utility. Use 'python -m win32traceutil' to see the output from the trace.
It's important to understand the basic process to get an ISAPI application running. I suggest first getting a hello-world WSGI application running under ISAPI_WSGI. Here's an early version of a hook script I used to validate that I was getting CherryPy to work with my web server.
#!python
"""
Things to remember:
easy_install munges permissions on zip eggs.
anything that's installed in a user folder (i.e. setup develop) will probably not work.
There may still exist an issue with static files.
"""
import sys
import os
import isapi_wsgi
# change this to '/myapp' to have the site installed to only a virtual
# directory of the site.
site_root = '/'
if hasattr(sys, "isapidllhandle"):
import win32traceutil
appdir = os.path.dirname(__file__)
egg_cache = os.path.join(appdir, 'egg-tmp')
if not os.path.exists(egg_cache):
os.makedirs(egg_cache)
os.environ['PYTHON_EGG_CACHE'] = egg_cache
os.chdir(appdir)
import cherrypy
import traceback
class Root(object):
#cherrypy.expose
def index(self):
return 'Hai Werld'
def setup_application():
print "starting cherrypy application server"
#app_root = os.path.dirname(__file__)
#sys.path.append(app_root)
app = cherrypy.tree.mount(Root(), site_root)
print "successfully set up the application"
return app
def __ExtensionFactory__():
"The entry point for when the ISAPIDLL is triggered"
try:
# import the wsgi app creator
app = setup_application()
return isapi_wsgi.ISAPISimpleHandler(app)
except:
import traceback
traceback.print_exc()
f = open(os.path.join(appdir, 'critical error.txt'), 'w')
traceback.print_exc(file=f)
f.close()
def install_virtual_dir():
import isapi.install
params = isapi.install.ISAPIParameters()
# Setup the virtual directories - this is a list of directories our
# extension uses - in this case only 1.
# Each extension has a "script map" - this is the mapping of ISAPI
# extensions.
sm = [
isapi.install.ScriptMapParams(Extension="*", Flags=0)
]
vd = isapi.install.VirtualDirParameters(
Server="CherryPy Web Server",
Name=site_root,
Description = "CherryPy Application",
ScriptMaps = sm,
ScriptMapUpdate = "end",
)
params.VirtualDirs = [vd]
isapi.install.HandleCommandLine(params)
if __name__=='__main__':
# If run from the command-line, install ourselves.
install_virtual_dir()
This script does several things. It (a) acts as the installer, installing itself into IIS [install_virtual_dir], (b) contains the entry point when IIS loads the DLL [__ExtensionFactory__], and (c) it creates the CherryPy WSGI instance consumed by the ISAPI handler [setup_application].
If you place this in your \inetpub\cherrypy directory and run it, it will attempt to install itself to the root of your IIS web site named "CherryPy Web Server".
You're also welcome to take a look at my production web site code, which has refactored all of this into different modules.

OK, I got it working. Thanks to Jason and all his help. I needed to call
cherrypy.config.update({
'tools.sessions.on': True
})
return cherrypy.tree.mount(Root(), '/', config=path_to_config)
I had this in the config file under [/] but for some reason it did not like that. Now I can get my web app up and running - then I think I will try and work out why it needs that config update and doesn't like the config file I have...