We Keep Coding

How gunicorn and shared variables work

I have a class that I instantiate in a request (it's a ML model that loads and takes a bit of time to configure on startup). The idea is to only do that once and have each request use the model for predictions. Does gunicorn instantiate the app every time?
Aka, will the model retrain every time a new request comes in?

It sounds like you could benefit from the application preloading:
http://docs.gunicorn.org/en/stable/settings.html#preload-app
This will let you load app code before spinning off your workers.

For those who are looking for how to share a variable between gunicorn workers without using Redis or Session, here is a good alternative with the awesome python dotenv:
The principle is to read and write shared variables from a file that could be done with open() but dotenv is perfect in this situation.
pip install python-dotenv
In app directory, create .env file:
├── .env
└── app.py
.env:
var1="value1"
var2="value2"
app.py: # flask app
from flask import Flask
import os
from dotenv import load_dotenv
app = Flask( __name__ )
# define the path explicitly if not in same folder
#env_path = os.path.dirname(os.path.realpath(__file__)) +'/../.env'
#load_dotenv(dotenv_path=env_path) # you may need a first load
def getdotenv(env):
try:
#global env_path
#load_dotenv(dotenv_path=env_path,override=True)
load_dotenv(override=True)
val = os.getenv(env)
return val
except :
return None
def setdotenv(key, value): # string
global env_path
if key :
if not value:
value = '\'\''
cmd = 'dotenv -f '+env_path+' set -- '+key+' '+value # set env variable
os.system(cmd)
#app.route('/get')
def index():
var1 = getdotenv('var1') # retreive value of variable var1
return var1
#app.route('/set')
def update():
setdotenv('newValue2') # set variable var2='newValue2'

Multiple settings with Django and AppEngine

On my current setup I have multiple settings files for local, staging and production environments. I just have different DJANGO_SETTINGS_MODULE value for each server.
But now I'm trying to move to AppEngine and I'd like to know how I could deploy to different AppEngine instances with different DJANGO_SETTINGS_MODULE environment variable values.
I know there's env_variables option on app.yaml but I didn't find any way I could override it in appcfg.py as I can do for version and application options.
Is there any way I can do what I need?

We use something similar to this for our webapp2 project inside the appengine_config.py file in our application root. The idea is that we swap out config based on the app id of the project. I made up constants for the example but it should give you enough to get started.
import os
from google.appengine.api import app_identity
app_id = app_identity.get_application_id()
if os.environ.get('SERVER_SOFTWARE', 'dev').lower().startswith('dev'):
os.environ['DJANGO_SETTINGS_MODULE'] = 'local settings'
elif app_id == MY_DEV_ID:
os.environ['DJANGO_SETTINGS_MODULE'] = 'dev settings'
elif app_id == MY_STAGING_ID:
os.environ['DJANGO_SETTINGS_MODULE'] = 'staging settings'
elif app_id == MY_PROD_ID:
os.environ['DJANGO_SETTINGS_MODULE'] = 'prod settings'
else:
raise ValueError("Unknown app id %" % app_id)

Determine if Django is running under the development server

Is there a way to determine if Django is running on localhost and setting the DEBUG variable in settings.py accordingly.
So that if I run the server locally it will set DEBUG to True and otherwise set it to False.
Localhost: python manage.py runserver
Not localhost: python manage.py runserver 0.0.0.0:8000

As suggested by Bernhard Vallant, you can just check for runserver in sys.argv.
You can just replace your DEBUG assignment in settings.py with this:
DEBUG = (sys.argv[1] == 'runserver')
You should also import sys somewhere in settings.py.

This is not the best approach, but it works :)
For something better you can use django-configurations
import sys
# Determine if in Production or Development
if (len(sys.argv) >= 2 and sys.argv[1] == 'runserver'):
DEBUG = True
#...
else:
DEBUG = False
#...
Or you can use it as one-liner as mentioned by little_birdie in the comments:
DEBUG = (len(sys.argv) > 1 and sys.argv[1] == 'runserver')

Could not have a permalink to this accepted and related answer to your question. So, just pasting it:-
server = request.META.get('wsgi.file_wrapper', None)
if server is not None and server.__module__ == 'django.core.servers.basehttp':
print 'inside dev'
Of course, wsgi.file_wrapper might be set on META, and have a class from a module named django.core.servers.basehttp by extreme coincidence on another server environment, but I hope this will have you covered.
PS: Please refer to How can I tell whether my Django application is running on development server or not? for more details

Simpler code.
you really should log it so you can know for sure
this works even if it's run in an environment that starts it a completely different way other than calling python. There may not be argv at position 1.
import sys
DEBUG = 'runserver' in sys.argv
print(f'DEBUG = {DEBUG}')

__import__ or import_module vs import when chaining multiple django settings files

I'm trying to modularize my django settings file so that it's easier for to deploy and manage our multiple environments.
I've got it set up to load my files in this order...
settings.py - settings common to all
config/country_XX - settings specific to the install for that country (XX could be US, CA, AU, etc)
config/developer_XX - settings specific to a particular developer's local dev environment
Both the secondary files are allowed to change values set in previous files
I find that if 1 loads 2 and 3 using the normal
from config.country_XX import *
the changes I make in those files are preserved.
If, however, 1 loads 2 and 3 using
__import__()
or
importlib.import_module()
the changes I make are not preserved.
I'd prefer to use import_module because it allows me to write cleaner code
import_module('config.country_' + country)
instead of
if country = 'AA':
from config.country_AA import *
elif: country == 'BB'
from config.country_BB import *
...
Here's what I've got... let me know what you think.
settings.py
import os
import sys
from django.utils import importlib
DEVELOPMENT = True
DEBUG = False
USES_24_HOUR_TIME = True
country_config = 'config.country_us'
developer_config = 'config.developer_jack'
try:
#importlib.import_module(country_config)
from config.country_us import *
if DEVELOPMENT:
#importlib.import_module(developer_config)
from config.developer_jack import *
except ImportError:
pass
config/country_us.py
import sys
globals().update(vars(sys.modules['settings']))
USES_24_HOUR_TIME = False
config/developer_jack.py
import sys
globals().update(vars(sys.modules['settings']))
DEBUG = True

Flask: app.config settings from .env &. flaskenv in mod_wsgi

I have spent quite a while trying to figure out how to set .env and .flaskenv configuration values in my flask backend in Google Cloud Platform server. I am using apache2, mod_wsgi, Flask, Python 3.6 and SQLAlchemy. My backend works fine locally on my Mac using pure Flask.
Having python-dotenv installed, running the flask command will set environment variables defined in the files .env and .flaskenv. This, however, does not work with wsgi. The request from apache is redirected to execute my run.wsgi-file. There is no mechanism (that I have knowledge about) to set the environment variables defined in .env and .flaskenv.
The minimun requirement is to pass to the application information if test or development environment should be used. From there I could within init.py populate app.config values from an object. However, being somehow able to use config-values from .env and .flaskenv would be far better. I would really appreciate if somebody had any good ideas here - the best practice to set app.config values in wsgi environment.
There are two posts where this same problem has been presented - they really do not have a best practice how to tackle this challenge (and I am sure I am not the only one having a hard time with this):
Why can't Flask can't see my environment variables from Apache (mod_wsgi)?
Apache SetEnv not working as expected with mod_wsgi
My run.wsgi:
import sys
sys.path.append("/var/www/contacts-api/venv/lib/python3.6/site-packages")
sys.path.insert(0,"/var/www/contacts-api/")
from contacts import create_app
app = create_app('settings.py')
app.run()
[3]:Allows you to configure an application using pre-set methods.
from flask_appconfig import AppConfig
def create_app(configfile=None):
app = Flask('myapp')
AppConfig(app, configfile)
return app
The application returned by create_app will, in order:
Load default settings from a module called myapp.default_config, if it exists. (method described in http://flask.pocoo.org/docs/config/#configuring-from-files )
Load settings from a configuration file whose name is given in the environment variable MYAPP_CONFIG (see link from 1.).
Load json or string values directly from environment variables that start with a prefix of MYAPP_, i.e. setting MYAPP_SQLALCHEMY_ECHO=true will cause the setting of SQLALCHEMY_ECHO to be True.
Any of these behaviors can be altered or disabled by passing the appropriate options to the constructor or init_app().
[4]: Using “ENV-only”
If you only want to use the environment-parsing functions of Flask-AppConfig, the appropriate functions are exposed:
from flask_appconfig.heroku import from_heroku_envvars
from flask_appconfig.env import from_envvars
# from environment variables. note that you need to set the prefix, as
# no auto-detection can be done without an app object
from_envvars(app.config, prefix=app.name.upper() + '_')
# also possible: parse heroku configuration values
# any dict-like object will do as the first parameter
from_heroku_envvars(app.config)

After reading more about this and trying many different things. I reached to the conclusion that there is no reasonable way for configuring a Flask-application using .env- and .flaskenv -files. I ended up using a method presented in Configuration Handling which enables managing development/testing/production-environments in a reasonable manner:
app = Flask(__name__)
app.config.from_object('yourapplication.default_settings')
app.config.from_envvar('YOURAPPLICATION_SETTINGS')
My run.wsgi (being used in google cloud platform compute instance):
import sys
import os
from contacts import create_app
sys.path.append("/var/www/myapp/venv/lib/python3.6/site-packages")
sys.path.insert(0,"/var/www/myapp/")
os.environ['SETTINGS_PLATFORM_SPECIFIC'] = "/path/settings_platform_specific.py"
os.environ['CONFIG_ENVIRONMENT'] = 'DevelopmentConfig'
app = create_app(')
app.run()
Locally on my mac I use run.py (for flask run):
import os
from contacts import create_app
os.environ['SETTINGS_PLATFORM_SPECIFIC'] ="/path/settings_platform_specific.py"
os.environ['CONFIG_ENVIRONMENT'] = 'DevelopmentConfig'
if __name__ == '__main__':
app = create_app()
app.run()
For app creation init.py
def create_app():
app = Flask(__name__, template_folder='templates')
app.config.from_object(f'contacts.settings_common.{os.environ.get("CONFIG_ENVIRONMENT")}')
app.config.from_envvar('SETTINGS_PLATFORM_SPECIFIC')
db.init_app(app)
babel.init_app(app)
mail.init_app(app)
bcrypt.init_app(app)
app.register_blueprint(routes)
create_db(app)
return app
At this point it looks like this works out fine for my purposes. The most important thing is that I can easily manage different environments and deploy the backend service to google platform using git.

I was wrestling with the same conundrum, wanting to use the same .env file I was using with docker-compose while developing with flask run. I ended up using python-dotenv, like so:
In .env:
DEBUG=True
APPLICATION_ROOT=${PWD}
In config.py:
import os
from dotenv import load_dotenv
load_dotenv()
class Config(object):
SECRET_KEY = os.getenv('SECRET_KEY') or 'development-secret'
DEBUG = os.getenv("DEBUG") or False
APPLICATION_ROOT = os.getenv("APPLICATION_ROOT") or os.getcwd()
I haven't experimented with it yet, but I may also give flask-env a try in combination with this solution.

The easy to go will be using load_dotenv() and from_mapping
from flask import Flask
from dotenv import load_dotenv , dotenv_values
load_dotenv()
app = Flask(__name__)
config = dotenv_values()
app.config.from_mapping(config)