We Keep Coding

How to import Django Settings to python standalone script

I'm having difficulty calling a django script. Here is what I'm currently doing in my root directory:
>>> import os
>>> os.environ['DJANGO_SETTINGS_MODULE'] = 'settings.py'
>>> from django.conf import settings
>>> settings.configure()
>>> settings.DATABASES
{}
The settings.DATABASES should not be empty, so I know I haven't initialized the project correct. How would I do this in django2.1? I used to be able to do this easily using import settings; setup_environ(settings), but not anymore.
Note: I'm looking to be able to run the above from any directory. Here is an example from trying to import my project from tmp:
(V) david$ cd /tmp && python
>>> import django
>>> from django.conf import settings
>>> settings.configure()
>>> django.setup()
>>> from users.models import *
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'users'

The 2.1 docs state that you need to use setup() now. Relevant snippet:
import django
from django.conf import settings
settings.configure()
django.setup()
# Now this script or any imported module can use any part of Django it needs.
from myapp import models
Here is the full documentation.

You don't need to run settings.configure(). To properly initialize, you could do something like:
if __name__ == '__main__':
import sys, os
sys.path.append(django_root)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from django.conf import settings
print (settings.DATABASES)

I have had this problem and my solution (script) is the result of hundreds of Django sites over the course of a decade.
Some solutions can inhibit the use of settings and this is the best working version of a stand alone script for running Django that I have been able to compile. This is faster than the Django shell generally (no need to exit and re-enter).
Above suggestions will work 98% of the time. Still, consider reviewing (and commenting) on this for a more robust way to run Django scripts. I have hit the "Exceptions" about 1-2x every year for a while until I got this script fully debugged via many projects.
"""
scratch.py
----->note the no .py-----v
clear; echo 'import scratch' | python manage.py shell
"""
from django.conf import settings
from django.core.wsgi import get_wsgi_application # turns on app, access to DB
print(f'- - - - - - - - - - - - - - - - - - - - ')
print(f'settings: {settings}')
print('settings.DEBUG: {}'.format(settings.DEBUG))
# settings.DEBUG = True
# etc
if not settings.DEBUG:
# keeps this off prod, usually on git ignore as well
print('SETTINGS NOT IN DEBUG, set to:')
print(settings.DEBUG)
raise(RuntimeError('Can not be run on production or when debug is False'))
application = get_wsgi_application()
print('READY!')
print(f'- - - - - - - - - - - - - - - - - - - - ')
# App ready, code below. Add imports, calls, etc below here.
Additionally this script can be run in an infinite loop, such as for monitoring, debugging, and other day to day Django operations. At the ned of the file just add the following, possibly with a sleep() and logging.
while True:
pass
Call this script scratch.py from the same directory as manage.py. Call using the following syntax:
echo 'import scratch' | python manage.py shell
If you change the file name, then the import will need to be edited. This syntax loads the settings without needing to call settings.configure(). We leverage manage.py and that seems to solve the edge cases.
Note: This will NOT auto-reload on with changes.

malformed header from script index.py Bad header

I want to run python code in apache2(ubuntu 14.04) server. I have followed these steps and getting Error:
Step 1:
Configuration 1: I have created a directory and file under
/var/www/cgi-bin
Configuration 2 : I have edited /etc/apache2/sites-available/000-default.conf
Alias /cgi-bin /var/www/cgi-bin
<Directory "/var/www/cgi-bin">
Options Indexes FollowSymLinks ExecCGI
AddHandler cgi-script .cgi .py
Allow from all
</Directory>
<Directory /var/www/cgi-bin>
Options All
</Directory>
Step 2:
and my python script is: index.py
#!/usr/bin/env python
import cgi;
import cgitb;cgitb.enable()
print "Content-Type: text/plain\n"
print "<b>Hello python</b>"
step 3:
When i ran through chrome browser using:
URL : http://localhost/cgi-bin/index.py
step 4:
I am getting this Error in error-log
malformed header from script 'index.py': Bad header: Hello Python

You should end your header with \r\n, then you must print out yet another \r\n to signal that the body is coming.
(In other words, it's interpreting your body as a Header because the headers were never terminated)

Try this script
#!/usr/bin/env python
import cgi;
import cgitb;cgitb.enable()
print "Content-Type: text/html"
print "" #use this double quote print statement to add a blank line in the script
print "<b>Hello python</b>"
There should be one line space between header and main html content. That's why we have to use extra print statement before starting html tags in script.

I had this issue with the flush mechanism when you need to print a file.
This code responds to a http request if it is called via e.g. apache2.
import sys
print("Content-type: image/png", end="\r\n\r\n", flush=True)
sys.stdout.buffer.write(bytes(open("file.png","rb").read()))
end="\r\n\r\n" adds an empty line to begin the body
flush=True forces python to print the lines as intended. In my case, the header was printed wrong.

WSGI process reload modules

I'm trying to trigger a reload of my WSGI process when any file changes in the folder where it and all it's dependent modules are located.
I've read http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode and I thought I understood it, but this intermittent staleness makes me doubt myself. I'm running in daemon mode like this:
DocumentRoot /usr/local/www/mysite.com/public_html
WSGIScriptAlias /api /usr/local/www/mysite.com/server/server.py
WSGIPassAuthorization On
WSGIDaemonProcess mysite.com threads=15 python-path=/usr/local/www/mysite.com/server
WSGIProcessGroup mysite.com
server.py is the main WSGI application file and all the modules it imports (which are likely to change) are in the same folder as it.
This is what I've come up with and it seems to work most of the time but occasionally I get stuck modules (where I make a change to a source file and the process restarts but it seems to load the old code). Some caching issue? If the process is restarted, I thought the import would get the fresh code? I really want to avoid using reload(). Restarting Apache always unsticks it and picks up the changes.
#!/bin/bash
while true; do
inotifywait . -e modify,create --exclude server.py -q
if (($? == 0)); then
touch server.py
else
exit 0
fi
done
Am I right in thinking that this (or something like it) should work or am I barking up the wrong tree?
The root WSGI file (server.py) is quite small:
print "Server restart"
import sys, types, os, web
import api
import user # /api/user
import list # /api/list
#api.path('/info')
class info(api.Handler):
#api.params({
'params': {'echo': unicode }
})
def Post(self, data):
return api.JSON({'info': 'foo', 'echo': data['echo']})
#api.path('/(.*)')
class notfound(api.Handler):
def Get(self):
api.error('404 page not found')
app = web.application(api.urls(), globals())
if __name__ == '__main__':
app.run()
else:
application = app.wsgifunc()

How can I make a Python package which serves a website with Flask to work with Apache?

I have written a Python package hwrt (see installation instructions if you want to try it) which serves a website when executed with
$ hwrt serve
2014-12-04 20:27:07,182 INFO * Running on http://127.0.0.1:5000/
2014-12-04 20:27:07,183 INFO * Restarting with reloader
I would like to let it run on http://www.pythonanywhere.com, but when I start it there I get
19:19 ~ $ hwrt serve
2014-12-04 19:19:59,282 INFO * Running on http://127.0.0.1:5000/
Traceback (most recent call last):
File "/home/MartinThoma/.local/bin/hwrt", line 108, in <module>
main(args)
File "/home/MartinThoma/.local/bin/hwrt", line 102, in main
serve.main()
File "/home/MartinThoma/.local/lib/python2.7/site-packages/hwrt/serve.py", line 95, in main
app.run()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 739, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 613, in run_simple
test_socket.bind((hostname, port))
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
I only found this in the documentation:
Flask
never use app.run(), it will break your webapp. Just import the
app into your wsgi file...
By searching for wsgi file, I found mod_wsgi (Apache). However, I don't understand how I can adjust my current minimalistic Flask application to work with that. Currently, the script behind hwrt serve is:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Start a webserver which can record the data and work as a classifier."""
import pkg_resources
from flask import Flask, request, render_template
from flask_bootstrap import Bootstrap
import os
import json
# hwrt modules
import hwrt
import hwrt.utils as utils
def show_results(results, n=10):
"""Show the TOP n results of a classification."""
import nntoolkit
classification = nntoolkit.evaluate.show_results(results, n)
return "<pre>" + classification.replace("\n", "<br/>") + "</pre>"
# configuration
DEBUG = True
template_path = pkg_resources.resource_filename('hwrt', 'templates/')
# create our little application :)
app = Flask(__name__, template_folder=template_path)
Bootstrap(app)
app.config.from_object(__name__)
#app.route('/', methods=['POST', 'GET'])
def show_entries():
heartbeat = request.args.get('heartbeat', '')
return heartbeat
#app.route('/interactive', methods=['POST', 'GET'])
def interactive():
if request.method == 'POST':
raw_data_json = request.form['drawnJSON']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
print(model)
results = utils.evaluate_model_single_recording(model, raw_data_json)
# Show classification page
page = show_results(results, n=10)
page += 'back'
return page
else:
# Page where the user can enter a recording
return render_template('canvas.html')
def get_json_result(results, n=10):
s = []
for res in results[:min(len(results), n)]:
s.append({res['semantics']: res['probability']})
return json.dumps(s)
#app.route('/worker', methods=['POST', 'GET'])
def worker():
# Test with
# wget --post-data 'classify=%5B%5B%7B%22x%22%3A334%2C%22y%22%3A407%2C%22time%22%3A1417704378719%7D%5D%5D' http://127.0.0.1:5000/worker
if request.method == 'POST':
raw_data_json = request.form['classify']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
results = utils.evaluate_model_single_recording(model, raw_data_json)
return get_json_result(results, n=10)
else:
# Page where the user can enter a recording
return "Classification Worker (Version %s)" % hwrt.__version__
def get_parser():
"""Return the parser object for this script."""
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
parser = ArgumentParser(description=__doc__,
formatter_class=ArgumentDefaultsHelpFormatter)
return parser
def main():
app.run()
if __name__ == '__main__':
main()

Ok, a not so non-sequitur answer to your question is around what mod_wsgi does to interface with your app. A typical flask app would look something like this:
from flask import Flask
app = Flask(__name__)
app.route("/")
def hello():
return "Holy moly that tunnel was bright.. said Bit to NIC"
if __name__ == "__main__":
app.run()
Unfortunately, Apache has no way to know what to do with this (though the app would run happily on its own). In order to get the app and Apache to play nice together we're going to use something called mod_wsgi. What Mod_WSGI does that's important to us, is that it provides a known interface (a file type called wsgi) that's going to wrap our application and initialize it so that we can serve it through Apache.
I'm going to assume you are using a python virtual environment, but if you aren't you can omit the step that deals with this in the instructions below. If you're curious why virtual environments are so great, feel free read about the python ecosystem.
Also - you can include an extra flag (assuming you are running wsgi as a daemon) to automatically reload the daemon whenever you touch or alter your wsgi file. This is quite useful during development and debugging so I'll include is below.
Anyway, let's get started. I'll break this down to steps below.
Configuring Apache for mod_wsgi
Enable mod_wsgi in Apache:
sudo apt-get install libapache2-mod-wsgi
Edit your /etc/apache2/sites-available/<yoursite>.conf.
<VirtualHost interface:port>
WSGIDaemonProcess yourapp user=someUser processes=2 threads=15
WSGIProcessGroup yourapp
# In this case / refers to whatever relative URL path hosts flask
WSGIScriptAlias / /absolute/path/to/yourapp.wsgi
<Directory /path/to/your/main/py/file/ >
# Use good judgement here when server hardening, this assumes dev env
Order allow,deny
Allow from all
Require all granted
#The below enables 'auto-reload' of WSGI
WSGIScriptReloading On
</Directory>
# If you want to serve static files as well and bypass flask in those cases
Alias /relative/url/to/static/content/
<Directory /absolute/path/to/static/root/directory/>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Create your yourapp.wsgi file and put it in the appropriate place: Be wary of file permissions!
#!/usr/bin/python
import sys
import logging
# Activate virtual environment.
# If you are not using venv, skip this.
# But you really should be using it!
activate_this = "/path/to/venv/bin/activate_this.py"
execfile(activate_this, dict(__file__=activate_this))
# Handle logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/path/to/your/main/py/file/")
from YourMainPyFileName import app as application
application.secret_key = "your_secret_key"
Reload Apache and troubleshoot problems. I set this up probably every few weeks for a different project or idea I have and... I usually have to fix one thing or another when doing it from scratch. Don't despair though! Flask has great documentation on this.
Once you've done all this you should be at a place where flask runs all on its own. The sample flask app above is the actual code I use to verify everything works whenever I set this up.
This was left here in case it's some use, but is not really directly related to the question...
The answer here is to use x-send-file. This takes advantage of letting Apache do what it's good at (serving static content), while at the same time first letting flask (or other python framework) do it's work first. I do this often to let flask handle my auth layers in single page web apps and have so far been happy with the results.
Doing so requires two things:
First - Enable xsendfile on Apache2 sudo apt-get install libapache2-mod-xsendfile.
Second - Alter your apache2 configuration so allow x-send-file headers:
Alter your conf file in /etc/apache2/sites-available/<yoursite>.conf and add...
XSendFile On
XSendFilePath /path/to/static/directory
This can be entered top level within the <Virtualhost></Virtualhost> tags.
Don't forget to restart Apache sudo service apache2 restart.
Finally - Configure your flask app to use x-send-file in your app.py file:
app.user_x_sendfile = True
Note: Must be done after app initialization. Consequently can also be passed as an initialization parameter.
Flask has documentation on this (excerpt below):
use_x_sendfile
Enable this if you want to use the X-Sendfile feature. Keep in mind that the server has to support this. This only affects files sent with the send_file() method.
New in version 0.2.
This attribute can also be configured from the config with the USE_X_SENDFILE configuration key. Defaults to False.

I ran into a similar issue #moose was having. Getting connection refused and couldnt even telnet localhost 5000.
Turns out theres a ports.conf file i had to add Listen 5000
Happy days.

How do I deal with URL reroutes in Python Bottle and Apache .htaccess?

I am currently trying to create a simple standalone application using Python Bottle.
My entire project is under pytest/, where I have dispatch.fcgi and .htaccess.
dispatch.fcgi:
#!/usr/bin/python
# -*- coding: utf-8 -*-
import bottle
import os
from bottle import route, run, view
#route('<foo:path>')
#view('index')
def pytest(foo = ''):
return dict(foo=foo)
APP_ROOT = os.path.abspath(os.path.dirname(__file__))
bottle.TEMPLATE_PATH.append(os.path.join(APP_ROOT, 'templates'))
app = bottle.default_app()
if __name__ == '__main__':
from flup.server.fcgi import WSGIServer
WSGIServer(app).run()
.htaccess:
DirectoryIndex dispatch.fcgi
The following URLs give me the corresponding values of foo:
url.com/pytest/
> /pytest/
url.com/pytest/dispatch.fcgi
> /pytest/dispatch.fcgi
url.com/pytest/dispatch.fcgi/
> /
url.com/pytest/dispatch.fcgi/foo/bar
> /foo/bar
url.com/pytest/dispatch.fcgi/pytest/
> /pytest/
How can I make the URLs uniform? Should I deal with the rerouting with the .htaccess file or within the Python code? What would be considered most pythonic, or best practices?
I am running Python 2.6.6, Bottle 0.11.6, Flup 1.0.2, and Apache 2.2.24. I would also like to point out that I'm using shared hosting, and mod_wsgi is out of the question (if that makes a difference).
EDIT
This is what I expect to see:
url.com/pytest/
> <redirect to url.com/pytest/dispatch.fcgi>
url.com/pytest/dispatch.fcgi
> <empty string>
url.com/pytest/dispatch.fcgi/
> /
url.com/pytest/dispatch.fcgi/foo/bar
> /foo/bar
url.com/pytest/dispatch.fcgi/pytest/
> /pytest/
If there is a more efficient way of tackling this problem, please let me know.

Couple of thoughts. Hopefully some or all of these will help.
1) You can do the redirect from '/' to '/pytest/dispatch.fcgi' like this:
#route('/')
def home():
bottle.redirect('/pytest/dispatch.fcgi')
2) Can you use ScriptAlias instead of DirectoryIndex? I see you're on a shared environment, so I'm not sure. My bottle/apache servers use ScriptAlias (or WSGIScriptAlias) and it works perfectly there; and it'd make more clear the way your code interacts with apache.
3) If worse came to worst, could you hackishly detect the case where foo == '/pytest/dispatch.fcgi' and act accordingly? (E.g., treat it as empty string.)
Hope this helps. Please keep us posted!

Bottle seems to be confused because it expects a trailing slash, followed by parameters. For that reason I changed my .htaccess file to read like this:
DirectoryIndex dispatch.fcgi/
Another option would be to have all errors fall back onto the dispatch script. That can be done with mod_rewrite:
<IfModule mod_rewrite.c>
Options -MultiViews
# rewrite for current folder
RewriteEngine On
RewriteBase /pytest
# redirect to front controller
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^ dispatch.fcgi/ [R=301,QSA,L]
</IfModule>
or FallbackResource:
FallbackResource /pytest/dispatch.fcgi/