We Keep Coding

How to run one part of code on local and another in production (heroku) in python? [duplicate]

I have a Django webapp, and I'd like to check if it's running on the Heroku stack (for conditional enabling of debugging, etc.) Is there any simple way to do this? An environment variable, perhaps?
I know I can probably also do it the other way around - that is, have it detect if it's running on a developer machine, but that just doesn't "sound right".

An ENV var seems to the most obvious way of doing this. Either look for an ENV var that you know exists, or set your own:
on_heroku = False
if 'YOUR_ENV_VAR' in os.environ:
on_heroku = True
more at: http://devcenter.heroku.com/articles/config-vars

Similar to what Neil suggested, I would do the following:
debug = True
if 'SOME_ENV_VAR' in os.environ:
debug = False
I've seen some people use if 'PORT' in os.environ: But the unfortunate thing is that the PORT variable is present when you run foreman start locally, so there is no way to distinguish between local testing with foreman and deployment on Heroku.
I'd also recommend using one of the env vars that:
Heroku has out of the box (rather than setting and checking for your own)
is unlikely to be found in your local environment
At the date of posting, Heroku has the following environ variables:
['PATH', 'PS1', 'COLUMNS', 'TERM', 'PORT', 'LINES', 'LANG', 'SHLVL', 'LIBRARY_PATH', 'PWD', 'LD_LIBRARY_PATH', 'PYTHONPATH', 'DYNO', 'PYTHONHASHSEED', 'PYTHONUNBUFFERED', 'PYTHONHOME', 'HOME', '_']
I generally go with if 'DYNO' in os.environ:, because it seems to be the most Heroku specific (who else would use the term dyno, right?).
And I also prefer to format it like an if-else statement because it's more explicit:
if 'DYNO' in os.environ:
debug = False
else:
debug = True

First set the environment variable ON_HEROKU on heroku:
$ heroku config:set ON_HEROKU=1
Then in settings.py
import os
# define if on heroku environment
ON_HEROKU = 'ON_HEROKU' in os.environ

Read more about it here: https://devcenter.heroku.com/articles/config-vars
My solution:
$ heroku config:set HEROKU=1
These environment variables are persistent – they will remain in place across deploys and app restarts – so unless you need to change values, you only need to set them once.
Then you can test its presence in your app.:
>>> 'HEROKU' in os.environ
True

The most reliable way would be to set an environment variable as above.
If that's not possible, there are a few signs you can look for in the filesystem, but they may not be / are not foolproof
Heroku instances all have the path /app - the files and scripts that are running will be under this too, so you can check for the presence of the directory and/or that the scripts are being run from under it.
There is an empty directory /etc/heroku
/etc/hosts may have some heroku related domains added
~ $ cat /etc/hosts
<snip>.dyno.rt.heroku.com
Any of these can and may change at any moment.
Your milage may vary

DATABASE_URL environment variable
in_heroku = False
if 'DATABASE_URL' in os.environ:
in_heroku = True
I think you need to enable the database for your app with:
heroku addons:create heroku-postgresql:hobby-dev
but it is free and likely what you are going to do anyways.
Heroku makes this environment variable available when running its apps, in particular for usage as:
import dj_database_url
if in_heroku:
DATABASES = {'default': dj_database_url.config()}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
Not foolproof as that variable might be defined locally, but convenient for simple cases.
heroku run env
might also show other possible variables like:
DYNO_RAM
WEB_CONCURRENCY
but I'm not sure if those are documented like DATABASE_URL.

Short version: check that the time zone is UTC/GMT:
if not 'ORIGINAL_TIMEZONE' in os.environ:
f = os.popen('date +%Z')
tz = f.read().upper()
os.environ['ORIGINAL_TIMEZONE']=tz
tz = os.environ['ORIGINAL_TIMEZONE']
if tz != '' and (not 'utc' in tz.lower()) and (not 'gmt' in tz.lower()):
print 'Definitely not running on Heroku (or in production in general)'
else:
print 'Assume that we are running on Heroku (or in production in general)'
This is more conservative than if tz=='UTC\n': if in doubt, assume that we are in production. Note that we are saving the timezone to an environment variable because settings.py may be executed more than once. In fact, the development server executes it twice, and the second time the system timezone is already 'UTC' (or whatever is in settings.TIMEZONE).
Long version:
making absolutely sure that we never run on Heroku with DEBUG=True, and that we never run the development server on Heroku even with DEBUG=False. From settings.py:
RUNNING_DEV_SERVER = (len(sys.argv) > 1) and (sys.argv[1] == 'runserver')
DEBUG = RUNNING_DEV_SERVER
TEMPLATE_DEBUG = DEBUG
# Detect the timezone
if not 'ORIGINAL_TIMEZONE' in os.environ:
f = os.popen('date +%Z')
tz = f.read().upper()
os.environ['ORIGINAL_TIMEZONE']=tz
print ('DEBUG: %d, RUNNING_DEV_SERVER: %d, system timezone: %s ' % (DEBUG, RUNNING_DEV_SERVER, tz))
if not (DEBUG or RUNNING_DEV_SERVER):
SECRET_KEY = os.environ['SECRET_KEY']
else:
print 'Running in DEBUG MODE! Hope this is not in production!'
SECRET_KEY = 'DEBUG_INSECURE_SECRET_KEY_ae$kh(7b%$+a fcw_bdnzl#)$t88x7h2-p%eg_ei5m=w&2p-)1+'
# But what if we are idiots and are still somehow running with DEBUG=True in production?!
# 1. Make sure SECRET_KEY is not set
assert not SECRET_KEY in os.environ
# 2. Make sure the timezone is not UTC or GMT (indicating production)
tz = os.environ['ORIGINAL_TIMEZONE']
assert tz != '' and (not 'UTC' in tz) and (not 'GMT' in tz)
# 3. Look for environment variables suggesting we are in PROD
for key in os.environ:
for red_flag in ['heroku', 'amazon', 'aws', 'prod', 'gondor']:
assert not red_flag in key.lower()
assert not red_flag in os.environ[key].lower()
If you really want to run the development server on Heroku, I suggest you add an environment variable specifying the date when you can do that. Then only proceed if this date is today. This way you'll have to change this variable before you begin development work, but if you forget to unset it, next day you will still be protected against accidentally running it in production. Of course, if you want to be super-conservative, you can also specify, say, a 1-hour window when exceptions apply.
Lastly, if you decided to adopt the approach suggested above, while you are at it, also install django-security, add djangosecurity to INSTALLED_APPS, and add to the end of your settings.py:
if not (DEBUG or RUNNING_DEV_SERVER):
### Security
SECURE_SSL_REDIRECT = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_HSTS_SECONDS = 86400000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_BROWSER_XSS_FILTER = True
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
CSRF_COOKIE_HTTPONLY = True # May have problems with Ajax
CSRF_COOKIE_SECURE = True

Apache Airflow: Control over logging [Disable/Adjust logging level]

I am using Airflow 1.7.1.3 installed using pip
I would like to limit the logging to ERROR level for the workflow being executed by the scheduler. Could not find anything beyond setting log files location in the settings.py file.
Also the online resources led me to this google group discussion here but not much info here as well
Any idea how to control logging in Airflow?

I tried below work around and it seems to be working to set LOGGING_LEVEL outside of settings.py:
Update settings.py:
Remove or comment line:
LOGGING_LEVEL = logging.INFO
Add line:
LOGGING_LEVEL = os.path.expanduser(conf.get('core', 'LOGGING_LEVEL'))
Update airflow.cfg configuration file:
Add line under [core]:
logging_level = WARN
Restart webserver and scheduler services
Use environment vaiable AIRFLOW__CORE__LOGGING_LEVEL=WARN.
See the official docs for details.

The logging functionality and its configuration will be changed in version 1.9 with this commit

As #Dimo Boyadzhiev pointed the change, Adding the more info path for documentaion.
File - $AIRFLOW_HOME/airflow.cfg
# Logging level
logging_level = INFO
fab_logging_level = WARN

Only solution I am aware of is changing LOGGING_LEVEL in settings.py file. Default level is set to INFO.
AIRFLOW_HOME = os.path.expanduser(conf.get('core', 'AIRFLOW_HOME'))
SQL_ALCHEMY_CONN = conf.get('core', 'SQL_ALCHEMY_CONN')
LOGGING_LEVEL = logging.INFO
DAGS_FOLDER = os.path.expanduser(conf.get('core', 'DAGS_FOLDER'))

If you are using docker-compose.yaml
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2}
environment:
&airflow-common-env
#the other parameters
AIRFLOW__CORE__LOGGING_LEVEL: DEBUG #add

Django best way to support developer specific config/settings

Here is the method my company devised, just wondering if anyone has anything better:
In Settings.py (at the bottom)
#...
try:
if socket.gethostname() == 'testsite':
from myir.local.TEST_settings import *
elif socket.gethostname() == 'prod':
from myir.local.PROD_settings import *
else:
from myir.local.DEV_settings import *
try:
# dev settings - don't commit local_settings.py
from proj.local.local_settings import *
except:
print "no local dev settings found..."
pass # intentionally do nothing.
except ImportError:
pass
local_settings.py:
DEBUG = True
LOGGING = { .. } # i usually keep maximum aount of logging possible in my dev environment.
... other configs you might want to override.

Django provides the environment variable DJANGO_SETTINGS_MODULE to specify the settings module to use. You can specify settings.my_prod_module there which enables settings to be different on production. Locally you can specify a different value.
You can also specify the value in your WSGI file:
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysite.settings.my_prod_module'

Determine if Django is running under the development server

Is there a way to determine if Django is running on localhost and setting the DEBUG variable in settings.py accordingly.
So that if I run the server locally it will set DEBUG to True and otherwise set it to False.
Localhost: python manage.py runserver
Not localhost: python manage.py runserver 0.0.0.0:8000

As suggested by Bernhard Vallant, you can just check for runserver in sys.argv.
You can just replace your DEBUG assignment in settings.py with this:
DEBUG = (sys.argv[1] == 'runserver')
You should also import sys somewhere in settings.py.

This is not the best approach, but it works :)
For something better you can use django-configurations
import sys
# Determine if in Production or Development
if (len(sys.argv) >= 2 and sys.argv[1] == 'runserver'):
DEBUG = True
#...
else:
DEBUG = False
#...
Or you can use it as one-liner as mentioned by little_birdie in the comments:
DEBUG = (len(sys.argv) > 1 and sys.argv[1] == 'runserver')

Could not have a permalink to this accepted and related answer to your question. So, just pasting it:-
server = request.META.get('wsgi.file_wrapper', None)
if server is not None and server.__module__ == 'django.core.servers.basehttp':
print 'inside dev'
Of course, wsgi.file_wrapper might be set on META, and have a class from a module named django.core.servers.basehttp by extreme coincidence on another server environment, but I hope this will have you covered.
PS: Please refer to How can I tell whether my Django application is running on development server or not? for more details

Simpler code.
you really should log it so you can know for sure
this works even if it's run in an environment that starts it a completely different way other than calling python. There may not be argv at position 1.
import sys
DEBUG = 'runserver' in sys.argv
print(f'DEBUG = {DEBUG}')

Google app engine bulkloader environment variables

I rely on python's os.environ to work out what configs my application should use (such as different API keys for different hosts).
It seems that bulkloader doesn't have access to these variables, is there anyway I can tell what the current version of my application or the current host is when bulkloader is running?
Usually I do this in my config_helper:
env = os.environ[ 'CURRENT_VERSION_ID' ].split( '.' )[ 0 ]
And bulkloader has reported a KeyError regarding CURRENT_VERSION_ID, so I used this:
if os.environ.get('HTTP_HOST'):
host = os.environ['HTTP_HOST']
else:
host = os.environ['SERVER_NAME']
if host is not None:
if host.find( 'locahost' ):
env = 'local'
elif host.find( 'prod-server' ):
env = 'prod'
elif host.find( 'dev-server' ):
env = 'dev'
elif host.find( 'stage-server' ):
env = 'stage'
os.environ[ 'CURRENT_VERSION_ID' ] = env + '.1'
However bulkloader complains that SERVER_NAME is an invalid object meaning that it also can't find HTTP_HOST.
Any other ideas?

Environment variables like HTTP_HOST and CURRENT_VERSION_ID are only available when your app is running as a web application.
Probably you can just pass the variables with env command as follows:
$ env CURRENT_VERSION_ID=local.1 bulkloader ....