We Keep Coding

How to deploy a Flask application on pythonanywhere with a folder as a module

Currently I am trying to a deploy my first FLASK application on PythonAnywhere.
Im not sure if this is the correct terminology but I have a folder as a module and there for I can't seem to find the correct way to deploy my application. I am not even sure where to start in resolving this issue. Any advice?
File and Folder Layout Snipped
my init.py code is:
import os
from flask import Flask
def create_app(test_config=None):
# create and configure the app
app = Flask(__name__, instance_relative_config=True)
app.config.from_mapping(
SECRET_KEY='secret',
DATABASE=os.path.join(app.instance_path, 'LAMA.sqlite'),
)
if test_config is None:
# load the instance config, if it exists, when not testing
app.config.from_pyfile('config.py', silent=True)
else:
# load the test config if passed in
app.config.from_mapping(test_config)
# ensure the instance folder exists
try:
os.makedirs(app.instance_path)
except OSError:
pass
# database
from . import db
db.init_app(app)
# authentication blueprint
from . import auth
app.register_blueprint(auth.bp)
# blog blueprint - the main index
# from . import blog
# app.register_blueprint(blog.bp)
# app.add_url_rule('/', endpoint='index')
# book blueprint
from . import book
app.register_blueprint(book.bp)
app.add_url_rule('/', endpoint='index')
return app
I have also followed the python debugging page where I have done the following:
>>> import LAMA
>>> print(LAMA)
<module 'LAMA' from '/home/ivanv257/LAMA_MAIN/LAMA/__init__.py'>
So at this stage in my WSGI configuration file I have:
import sys
path = '/home/ivanv257/LAMA_MAIN/LAMA/__init__.py'
if path not in sys.path:
sys.path.append(path)
from LAMA import app as application
I have also tried many other combinations such as
path = '/home/ivanv257/LAMA_MAIN/LAMA/'
from init import app as application
path = '/home/ivanv257/LAMA_MAIN/'
from init import app as application
path = '/home/ivanv257/LAMA_MAIN/'
from LAMA import app as application
my source code path is : /home/ivanv257/LAMA_MAIN/LAMA , although I have also tried different combinations such as /home/ivanv257/LAMA_MAIN/
ERROR DETAIL:
2018-12-08 10:05:32,028: Error running WSGI application
2018-12-08 10:05:32,030: ModuleNotFoundError: No module named 'LAMA'
2018-12-08 10:05:32,030: File "/var/www/ivanv257_pythonanywhere_com_wsgi.py", line 83, in <module>
2018-12-08 10:05:32,030: from LAMA import app as application # noqa

To solve my problem I changed the following (with some assistance) from lama import create_app:
import sys
path = '/home/ivanv257/LAMA_MAIN/LAMA'
if path not in sys.path:
sys.path.append(path)
from lama import create_app
application = create_app()
I also had to remove the from . to just imports
import db
db.init_app(app)
# authentication blueprint
import auth
app.register_blueprint(auth.bp)

You are close. To deploy your app, navigate to the webapps page on your user dashboard. If you have not done so already, click the "Add new a webapp" button and enter the desired name of the app. Then, on the same webapp page on the dashboard, scroll down to the "Code" section of the page. Click on the "source code" a href and add the absolute path (full path) to the lama_main directory storing your init.py file.
Next, click on "WSGI configuration file" link. In the WSGI configuration file for your app, set the correct path to the parent directory and import app from the init.py file:
import sys
# add your project directory to the sys.path
project_home = u'/home/your_user_name/lama_main'
if project_home not in sys.path:
sys.path = [project_home] + sys.path
# import flask app but need to call it "application" for WSGI to work
from init import app as application #note that the module being imported from must be the file with "app" defined.
Then, save the WSGI file and return to the webapp panel on your dashboard. Click the "Reload {your site name}" button. Now, you should be able to visit the site by clicking on the main link at the top of the page.

What gets executed when running a python app on OpenShift?

I'm confused as to what file gets initially run when running a Flask application on OpenShift. There is a wsgi.py file that creates a wsgiref.simple_server but the comment above it says it is only for testing. Can anyone explain how the application get executed? I'm having a hard time wrapping my head around the process and where exactly the web server comes in to the picture vs the application code.
wsgi.py:
import os
virtenv = os.path.join(os.environ.get('OPENSHIFT_PYTHON_DIR','.'), 'virtenv')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
pass
#
# IMPORTANT: Put any additional includes below this line. If placed above this
# line, it's possible required libraries won't be in your searchable path
#
from flaskapp import app
#
# Below for testing only
#
if __name__ == '__main__':
from wsgiref.simple_server import make_server
host = app.config['HOST_NAME']
ip = app.config['HOST_IP']
httpd = make_server(host, ip, app)
httpd.handle_request()
flaskapp.py
import os
virtenv = os.path.join(os.environ.get('OPENSHIFT_PYTHON_DIR','.'), 'virtenv')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
pass
#
# IMPORTANT: Put any additional includes below this line. If placed above this
# line, it's possible required libraries won't be in your searchable path
#
from flaskapp import app
#
# Below for testing only
#
if __name__ == '__main__':
from wsgiref.simple_server import make_server
host = app.config['HOST_NAME']
ip = app.config['HOST_IP']
httpd = make_server(host, ip, app)
httpd.handle_request()

See my answer here from another SO question.
How to create app using pyramid into openshift?
I think my last commit in my github example uses the "wsgi.py" entry point. I prefer using "app.py" as the entry point. I find it is less problematic and more reliable.
Go ahead and rename "app.py disabled" to "app.py" and delete the wsgi.py.
I'm using pyramid instead of flask in the example but the setup is similar.

Is there an alternative of RewriteRule / .htaccess for a Python http.server.HTTPServer?

Can my server module (with http.server.HTTPServer) use something like the RewriteRule for redirect all traffic into a single cgi script? I'd like to be able to do what's shown here in this other question, but for my python server.
Can it be done using something like .htaccess, or is there another way?
Also, can this be done even for a simple localhost development server?
I am serving files for development via, for example, http://localhost:8000/html/index.html, and I would like to hide the /html subfolder from the URL even in development.
How can that be achieved?

You can use a custom script to initialize your server and define your Routes in it, such as suggested in this article:
Python 2:
server.py
import os
import posixpath
import urllib
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
# modify this to add additional routes
ROUTES = ( <- this is the "good stuff", make aliases for your paths
# [url_prefix , directory_path]
['/media', '/var/www/media'],
['', '/var/www/site'] # empty string for the 'default' match
)
class RequestHandler(SimpleHTTPRequestHandler):
def translate_path(self, path):
"""translate path given routes"""
# set default root to cwd
root = os.getcwd()
# look up routes and set root directory accordingly
for pattern, rootdir in ROUTES:
if path.startswith(pattern):
# found match!
path = path[len(pattern):] # consume path up to pattern len
root = rootdir
break
# normalize path and prepend root directory
path = path.split('?',1)[0]
path = path.split('#',1)[0]
path = posixpath.normpath(urllib.unquote(path))
words = path.split('/')
words = filter(None, words)
path = root
for word in words:
drive, word = os.path.splitdrive(word)
head, word = os.path.split(word)
if word in (os.curdir, os.pardir):
continue
path = os.path.join(path, word)
return path
if __name__ == '__main__':
BaseHTTPServer.test(RequestHandler, BaseHTTPServer.HTTPServer)
Then run your script:
python server.py
Python 3:
In Python 3 the BaseHTTPServer and the SimpleHTTPServer modules have been merged into the http.server module. So you will have to modify the above script as follows:
Change the
import BaseHTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
to
import http.server
(if you do this then the calls should be modified to http.server.SimpleHTTPRequest etc.)
or
from http.server import BaseHTTPServer, SimpleHTTPServer, SimpleHTTPRequestHandler
(with this option calls remain the same as the original script)
then run the server script: python server.py
IN YOUR CASE:
You should modify the ROUTES variable to suite your needs. For example, you want to hide the /html folder from your url:
ROUTES = (
['', '/exact/path/to/folder/html'],
['/another_url_path', '/exact/path/to/another/folder'],
...
)
Now if you hit: http://localhost:8000/index.html you will be in your home page.
Note:
This script by default will serve the contained files in the folder that is in when executed to the domain url (ex. I have server.py on the Documents folder, then when I run it, http://localhost:8000 url will server my Documents folder).
You can change this behavior by the Routes (see the 'default' match comment on the code) or by placing the script in your projects root folder and start it from there.

John Moutafis's answer was helpful to get me started, but needed some fine tuning to run on python3, beyond his comments about imports.
The imports should be
from http.server import HTTPServer, SimpleHTTPRequestHandler
and you'll also need to change the urllib import to:
from urllib.parse import unquote
Then the main should be something like:
if __name__ == '__main__':
myServer = HTTPServer(('0.0.0.0', 8000), RequestHandler)
print("Ready to begin serving files.")
try:
myServer.serve_forever()
except KeyboardInterrupt:
pass
myServer.server_close()
print("Exiting.")

Writing doctests for pyramid web app which depend on settings in ini file

I would like to write doctests for my pyramid web app, using the webtest module. I tried it like this:
from my_webapp import main
from webtest import TestApp
app = TestApp(main({}))
result = app.get('/')
This raises a KeyError (because some.url is not known) when my code reaches this line:
url = request.registry.settings['some.url']
The value of some.url is specified in the paster ini file of my application. Is there a simple way to use my development.ini when running my test code? I did not yet fully understand how/when the ini file is loaded during pyramid start up, so it's hard to figure out where to load it while testing.

main is invoked with the contents of your ini file. A simple way to load your app from an ini is:
from pyramid.paster import get_app
app = get_app('testing.ini#main')
test_app = TestApp(app)
This expects "testing.ini" to be in the current working directory, so you may need to tweak that. If you'd like it to be relative to a spot in your tree you can use:
import os.path
import some_module
here = os.path.dirname(some_module.__file__)
app = get_app(os.path.join(here, 'testing.ini'))

ImportError and Django driving me crazy

OK, I have the following directory structure (it's a django project):
-> project
--> app
and within the app folder, there is a scraper.py file which needs to reference a class defined within models.py
I'm trying to do the following:
import urllib2
import os
import sys
import time
import datetime
import re
import BeautifulSoup
sys.path.append('/home/userspace/Development/')
os.environ['DJANGO_SETTINGS_MODULE'] = 'project.settings'
from project.app.models import ClassName
and this code just isn't working. I get an error of:
Traceback (most recent call last):
File "scraper.py", line 14, in
from project.app.models import ClassName
ImportError: No module named project.app.models
This code above used to work, but broke somewhere along the line and I'm extremely confused as to why I'm having problems. On SnowLeopard using python2.5.

import sys
sys.path.append ('/path/to/the/project')
from django.core.management import setup_environ
import settings
setup_environ(settings)
from app.models import MyModel

Whoa whoa whoa. You should never ever have to put your project name in any of your app code. You should be able to reuse app code across multiple projects with no changes. Pinax does this really well and I highly recommend checking it out for a lot of django best practices.
The worst thing you could do here is to hard code your absolute path into your app or settings. You shouldn't do this because it will break during deployment unless you do some import local_settings hacking.
If you have to access the project root directory, try what pinax has in settings.py...
import os.path
PROJECT_ROOT = os.path.abspath(os.path.dirname(__file__))
The thing is that it looks like you are trying to access the models module within the same app and this is waaay easier.
To import models.py inside scraper.py in the same directory just use import models or import models as app_models if you already have something named models in scraper.py (django.db.models for instance). Are you familiar with Python module conventions?
However, the best way is probably to stick with the django idiom, from ... import ... statement:
from app import models
If this doesn't work automatically, then something is wrong in your settings.py.

You don't indicate if project is located in /home/userspace/Development/. I'll assume that it is.
Make sure there's an (empty by default) file named __init__.py in project and another one in app.
EDIT: Next thing to try: Fire up the Python command line in the script's directory and try the following:
import project
import project.app as app
import project.app.models as models
models.__dict__.keys()
Do they all work? If so, what is the last line's output? If not, which dies first?