when deploying a Django solution in production, I ran into a problem with compressed and compiled static files. I have Django running under a Nginx reverse proxy that also takes care of serving static files.
Some of the files (notably js and css files) are not found, and I notice that this happens because of the compression enabled by the following settings in settings.py
COMPRESS_ENABLED = True
COMPRESS_PRECOMPILERS = (
('text/coffeescript', 'coffee --compile --stdio'),
('text/less', 'lessc {infile} {outfile}'),
)
COMPRESS_JS_FILTERS = [
'compressor.filters.jsmin.JSMinFilter'
]
INSTALLED_APPS = (
# [..]
'compressor',
# [..]
)
So, what is the most correct way to freeze the precompilations and compressions in order to deploy them in production? Thanks
additional to the above comment, maybe its worth to look at this
https://github.com/jezdez/django_compressor
Hope this helps.
I guess you are looking for a way to compress files on your dev machine, before deploying:
./manage.py compress --force
You should also set COMPRESS_OFFLINE in settings:
COMPRESS_OFFLINE = True
if 'compress' in sys.argv:
# compressor related settings when called as management command
TEMPLATE_DEBUG = False
COMPRESS_ENABLED = True
Related
I have a Django webapp, and I'd like to check if it's running on the Heroku stack (for conditional enabling of debugging, etc.) Is there any simple way to do this? An environment variable, perhaps?
I know I can probably also do it the other way around - that is, have it detect if it's running on a developer machine, but that just doesn't "sound right".
An ENV var seems to the most obvious way of doing this. Either look for an ENV var that you know exists, or set your own:
on_heroku = False
if 'YOUR_ENV_VAR' in os.environ:
on_heroku = True
more at: http://devcenter.heroku.com/articles/config-vars
Similar to what Neil suggested, I would do the following:
debug = True
if 'SOME_ENV_VAR' in os.environ:
debug = False
I've seen some people use if 'PORT' in os.environ: But the unfortunate thing is that the PORT variable is present when you run foreman start locally, so there is no way to distinguish between local testing with foreman and deployment on Heroku.
I'd also recommend using one of the env vars that:
Heroku has out of the box (rather than setting and checking for your own)
is unlikely to be found in your local environment
At the date of posting, Heroku has the following environ variables:
['PATH', 'PS1', 'COLUMNS', 'TERM', 'PORT', 'LINES', 'LANG', 'SHLVL', 'LIBRARY_PATH', 'PWD', 'LD_LIBRARY_PATH', 'PYTHONPATH', 'DYNO', 'PYTHONHASHSEED', 'PYTHONUNBUFFERED', 'PYTHONHOME', 'HOME', '_']
I generally go with if 'DYNO' in os.environ:, because it seems to be the most Heroku specific (who else would use the term dyno, right?).
And I also prefer to format it like an if-else statement because it's more explicit:
if 'DYNO' in os.environ:
debug = False
else:
debug = True
First set the environment variable ON_HEROKU on heroku:
$ heroku config:set ON_HEROKU=1
Then in settings.py
import os
# define if on heroku environment
ON_HEROKU = 'ON_HEROKU' in os.environ
Read more about it here: https://devcenter.heroku.com/articles/config-vars
My solution:
$ heroku config:set HEROKU=1
These environment variables are persistent – they will remain in place across deploys and app restarts – so unless you need to change values, you only need to set them once.
Then you can test its presence in your app.:
>>> 'HEROKU' in os.environ
True
The most reliable way would be to set an environment variable as above.
If that's not possible, there are a few signs you can look for in the filesystem, but they may not be / are not foolproof
Heroku instances all have the path /app - the files and scripts that are running will be under this too, so you can check for the presence of the directory and/or that the scripts are being run from under it.
There is an empty directory /etc/heroku
/etc/hosts may have some heroku related domains added
~ $ cat /etc/hosts
<snip>.dyno.rt.heroku.com
Any of these can and may change at any moment.
Your milage may vary
DATABASE_URL environment variable
in_heroku = False
if 'DATABASE_URL' in os.environ:
in_heroku = True
I think you need to enable the database for your app with:
heroku addons:create heroku-postgresql:hobby-dev
but it is free and likely what you are going to do anyways.
Heroku makes this environment variable available when running its apps, in particular for usage as:
import dj_database_url
if in_heroku:
DATABASES = {'default': dj_database_url.config()}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
Not foolproof as that variable might be defined locally, but convenient for simple cases.
heroku run env
might also show other possible variables like:
DYNO_RAM
WEB_CONCURRENCY
but I'm not sure if those are documented like DATABASE_URL.
Short version: check that the time zone is UTC/GMT:
if not 'ORIGINAL_TIMEZONE' in os.environ:
f = os.popen('date +%Z')
tz = f.read().upper()
os.environ['ORIGINAL_TIMEZONE']=tz
tz = os.environ['ORIGINAL_TIMEZONE']
if tz != '' and (not 'utc' in tz.lower()) and (not 'gmt' in tz.lower()):
print 'Definitely not running on Heroku (or in production in general)'
else:
print 'Assume that we are running on Heroku (or in production in general)'
This is more conservative than if tz=='UTC\n': if in doubt, assume that we are in production. Note that we are saving the timezone to an environment variable because settings.py may be executed more than once. In fact, the development server executes it twice, and the second time the system timezone is already 'UTC' (or whatever is in settings.TIMEZONE).
Long version:
making absolutely sure that we never run on Heroku with DEBUG=True, and that we never run the development server on Heroku even with DEBUG=False. From settings.py:
RUNNING_DEV_SERVER = (len(sys.argv) > 1) and (sys.argv[1] == 'runserver')
DEBUG = RUNNING_DEV_SERVER
TEMPLATE_DEBUG = DEBUG
# Detect the timezone
if not 'ORIGINAL_TIMEZONE' in os.environ:
f = os.popen('date +%Z')
tz = f.read().upper()
os.environ['ORIGINAL_TIMEZONE']=tz
print ('DEBUG: %d, RUNNING_DEV_SERVER: %d, system timezone: %s ' % (DEBUG, RUNNING_DEV_SERVER, tz))
if not (DEBUG or RUNNING_DEV_SERVER):
SECRET_KEY = os.environ['SECRET_KEY']
else:
print 'Running in DEBUG MODE! Hope this is not in production!'
SECRET_KEY = 'DEBUG_INSECURE_SECRET_KEY_ae$kh(7b%$+a fcw_bdnzl#)$t88x7h2-p%eg_ei5m=w&2p-)1+'
# But what if we are idiots and are still somehow running with DEBUG=True in production?!
# 1. Make sure SECRET_KEY is not set
assert not SECRET_KEY in os.environ
# 2. Make sure the timezone is not UTC or GMT (indicating production)
tz = os.environ['ORIGINAL_TIMEZONE']
assert tz != '' and (not 'UTC' in tz) and (not 'GMT' in tz)
# 3. Look for environment variables suggesting we are in PROD
for key in os.environ:
for red_flag in ['heroku', 'amazon', 'aws', 'prod', 'gondor']:
assert not red_flag in key.lower()
assert not red_flag in os.environ[key].lower()
If you really want to run the development server on Heroku, I suggest you add an environment variable specifying the date when you can do that. Then only proceed if this date is today. This way you'll have to change this variable before you begin development work, but if you forget to unset it, next day you will still be protected against accidentally running it in production. Of course, if you want to be super-conservative, you can also specify, say, a 1-hour window when exceptions apply.
Lastly, if you decided to adopt the approach suggested above, while you are at it, also install django-security, add djangosecurity to INSTALLED_APPS, and add to the end of your settings.py:
if not (DEBUG or RUNNING_DEV_SERVER):
### Security
SECURE_SSL_REDIRECT = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_HSTS_SECONDS = 86400000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_BROWSER_XSS_FILTER = True
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
CSRF_COOKIE_HTTPONLY = True # May have problems with Ajax
CSRF_COOKIE_SECURE = True
I am using Airflow 1.7.1.3 installed using pip
I would like to limit the logging to ERROR level for the workflow being executed by the scheduler. Could not find anything beyond setting log files location in the settings.py file.
Also the online resources led me to this google group discussion here but not much info here as well
Any idea how to control logging in Airflow?
I tried below work around and it seems to be working to set LOGGING_LEVEL outside of settings.py:
Update settings.py:
Remove or comment line:
LOGGING_LEVEL = logging.INFO
Add line:
LOGGING_LEVEL = os.path.expanduser(conf.get('core', 'LOGGING_LEVEL'))
Update airflow.cfg configuration file:
Add line under [core]:
logging_level = WARN
Restart webserver and scheduler services
Use environment vaiable AIRFLOW__CORE__LOGGING_LEVEL=WARN.
See the official docs for details.
The logging functionality and its configuration will be changed in version 1.9 with this commit
As #Dimo Boyadzhiev pointed the change, Adding the more info path for documentaion.
File - $AIRFLOW_HOME/airflow.cfg
# Logging level
logging_level = INFO
fab_logging_level = WARN
Only solution I am aware of is changing LOGGING_LEVEL in settings.py file. Default level is set to INFO.
AIRFLOW_HOME = os.path.expanduser(conf.get('core', 'AIRFLOW_HOME'))
SQL_ALCHEMY_CONN = conf.get('core', 'SQL_ALCHEMY_CONN')
LOGGING_LEVEL = logging.INFO
DAGS_FOLDER = os.path.expanduser(conf.get('core', 'DAGS_FOLDER'))
If you are using docker-compose.yaml
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.1.2}
environment:
&airflow-common-env
#the other parameters
AIRFLOW__CORE__LOGGING_LEVEL: DEBUG #add
Following the instructions on https://github.com/linuxlewis/djorm-ext-pgfulltext I added a search_index to one of my existing PostgreSQL models, as follows:
search_index = VectorField()
objects = SearchManager(
fields = ('materials','origin','style'),
config = 'pg_catalog.english',
search_field = 'search_index',
auto_update_search_field = True
)
Using my virtual-env and having pgfulltext installed, with this version djorm-ext-pgfulltext==0.9.3 according to pip freeze, I cd to the directory where manage.pyis located. And tried to run:
./manage.py update_search_field app_name
Wich resulted in the following error:
Unknown command: 'update_search_field' Type 'manage.py help' for
usage.
Am I missing something on how this command is used? Should I add the index directly to my db instance with psql? And if so, how could I achieve this?
Any help will be much appreciated.
"Unknown command"
...this usually means you forgot to add the app to your INSTALLED_APPS in settings.py
That's not surprising since it seems they forgot to mention that on the readme for the app, but you need something like:
INSTALLED_APPS = (
# ... (your existing apps)
'djorm_pgfulltext',
)
Django needs to be told about the app in this way so that it knows to look for the management/commands directory in the app.
Is there a way to determine if Django is running on localhost and setting the DEBUG variable in settings.py accordingly.
So that if I run the server locally it will set DEBUG to True and otherwise set it to False.
Localhost: python manage.py runserver
Not localhost: python manage.py runserver 0.0.0.0:8000
As suggested by Bernhard Vallant, you can just check for runserver in sys.argv.
You can just replace your DEBUG assignment in settings.py with this:
DEBUG = (sys.argv[1] == 'runserver')
You should also import sys somewhere in settings.py.
This is not the best approach, but it works :)
For something better you can use django-configurations
import sys
# Determine if in Production or Development
if (len(sys.argv) >= 2 and sys.argv[1] == 'runserver'):
DEBUG = True
#...
else:
DEBUG = False
#...
Or you can use it as one-liner as mentioned by little_birdie in the comments:
DEBUG = (len(sys.argv) > 1 and sys.argv[1] == 'runserver')
Could not have a permalink to this accepted and related answer to your question. So, just pasting it:-
server = request.META.get('wsgi.file_wrapper', None)
if server is not None and server.__module__ == 'django.core.servers.basehttp':
print 'inside dev'
Of course, wsgi.file_wrapper might be set on META, and have a class from a module named django.core.servers.basehttp by extreme coincidence on another server environment, but I hope this will have you covered.
PS: Please refer to How can I tell whether my Django application is running on development server or not? for more details
Simpler code.
you really should log it so you can know for sure
this works even if it's run in an environment that starts it a completely different way other than calling python. There may not be argv at position 1.
import sys
DEBUG = 'runserver' in sys.argv
print(f'DEBUG = {DEBUG}')
I have a Django webapp, and I'd like to check if it's running on the Heroku stack (for conditional enabling of debugging, etc.) Is there any simple way to do this? An environment variable, perhaps?
I know I can probably also do it the other way around - that is, have it detect if it's running on a developer machine, but that just doesn't "sound right".
An ENV var seems to the most obvious way of doing this. Either look for an ENV var that you know exists, or set your own:
on_heroku = False
if 'YOUR_ENV_VAR' in os.environ:
on_heroku = True
more at: http://devcenter.heroku.com/articles/config-vars
Similar to what Neil suggested, I would do the following:
debug = True
if 'SOME_ENV_VAR' in os.environ:
debug = False
I've seen some people use if 'PORT' in os.environ: But the unfortunate thing is that the PORT variable is present when you run foreman start locally, so there is no way to distinguish between local testing with foreman and deployment on Heroku.
I'd also recommend using one of the env vars that:
Heroku has out of the box (rather than setting and checking for your own)
is unlikely to be found in your local environment
At the date of posting, Heroku has the following environ variables:
['PATH', 'PS1', 'COLUMNS', 'TERM', 'PORT', 'LINES', 'LANG', 'SHLVL', 'LIBRARY_PATH', 'PWD', 'LD_LIBRARY_PATH', 'PYTHONPATH', 'DYNO', 'PYTHONHASHSEED', 'PYTHONUNBUFFERED', 'PYTHONHOME', 'HOME', '_']
I generally go with if 'DYNO' in os.environ:, because it seems to be the most Heroku specific (who else would use the term dyno, right?).
And I also prefer to format it like an if-else statement because it's more explicit:
if 'DYNO' in os.environ:
debug = False
else:
debug = True
First set the environment variable ON_HEROKU on heroku:
$ heroku config:set ON_HEROKU=1
Then in settings.py
import os
# define if on heroku environment
ON_HEROKU = 'ON_HEROKU' in os.environ
Read more about it here: https://devcenter.heroku.com/articles/config-vars
My solution:
$ heroku config:set HEROKU=1
These environment variables are persistent – they will remain in place across deploys and app restarts – so unless you need to change values, you only need to set them once.
Then you can test its presence in your app.:
>>> 'HEROKU' in os.environ
True
The most reliable way would be to set an environment variable as above.
If that's not possible, there are a few signs you can look for in the filesystem, but they may not be / are not foolproof
Heroku instances all have the path /app - the files and scripts that are running will be under this too, so you can check for the presence of the directory and/or that the scripts are being run from under it.
There is an empty directory /etc/heroku
/etc/hosts may have some heroku related domains added
~ $ cat /etc/hosts
<snip>.dyno.rt.heroku.com
Any of these can and may change at any moment.
Your milage may vary
DATABASE_URL environment variable
in_heroku = False
if 'DATABASE_URL' in os.environ:
in_heroku = True
I think you need to enable the database for your app with:
heroku addons:create heroku-postgresql:hobby-dev
but it is free and likely what you are going to do anyways.
Heroku makes this environment variable available when running its apps, in particular for usage as:
import dj_database_url
if in_heroku:
DATABASES = {'default': dj_database_url.config()}
else:
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
}
}
Not foolproof as that variable might be defined locally, but convenient for simple cases.
heroku run env
might also show other possible variables like:
DYNO_RAM
WEB_CONCURRENCY
but I'm not sure if those are documented like DATABASE_URL.
Short version: check that the time zone is UTC/GMT:
if not 'ORIGINAL_TIMEZONE' in os.environ:
f = os.popen('date +%Z')
tz = f.read().upper()
os.environ['ORIGINAL_TIMEZONE']=tz
tz = os.environ['ORIGINAL_TIMEZONE']
if tz != '' and (not 'utc' in tz.lower()) and (not 'gmt' in tz.lower()):
print 'Definitely not running on Heroku (or in production in general)'
else:
print 'Assume that we are running on Heroku (or in production in general)'
This is more conservative than if tz=='UTC\n': if in doubt, assume that we are in production. Note that we are saving the timezone to an environment variable because settings.py may be executed more than once. In fact, the development server executes it twice, and the second time the system timezone is already 'UTC' (or whatever is in settings.TIMEZONE).
Long version:
making absolutely sure that we never run on Heroku with DEBUG=True, and that we never run the development server on Heroku even with DEBUG=False. From settings.py:
RUNNING_DEV_SERVER = (len(sys.argv) > 1) and (sys.argv[1] == 'runserver')
DEBUG = RUNNING_DEV_SERVER
TEMPLATE_DEBUG = DEBUG
# Detect the timezone
if not 'ORIGINAL_TIMEZONE' in os.environ:
f = os.popen('date +%Z')
tz = f.read().upper()
os.environ['ORIGINAL_TIMEZONE']=tz
print ('DEBUG: %d, RUNNING_DEV_SERVER: %d, system timezone: %s ' % (DEBUG, RUNNING_DEV_SERVER, tz))
if not (DEBUG or RUNNING_DEV_SERVER):
SECRET_KEY = os.environ['SECRET_KEY']
else:
print 'Running in DEBUG MODE! Hope this is not in production!'
SECRET_KEY = 'DEBUG_INSECURE_SECRET_KEY_ae$kh(7b%$+a fcw_bdnzl#)$t88x7h2-p%eg_ei5m=w&2p-)1+'
# But what if we are idiots and are still somehow running with DEBUG=True in production?!
# 1. Make sure SECRET_KEY is not set
assert not SECRET_KEY in os.environ
# 2. Make sure the timezone is not UTC or GMT (indicating production)
tz = os.environ['ORIGINAL_TIMEZONE']
assert tz != '' and (not 'UTC' in tz) and (not 'GMT' in tz)
# 3. Look for environment variables suggesting we are in PROD
for key in os.environ:
for red_flag in ['heroku', 'amazon', 'aws', 'prod', 'gondor']:
assert not red_flag in key.lower()
assert not red_flag in os.environ[key].lower()
If you really want to run the development server on Heroku, I suggest you add an environment variable specifying the date when you can do that. Then only proceed if this date is today. This way you'll have to change this variable before you begin development work, but if you forget to unset it, next day you will still be protected against accidentally running it in production. Of course, if you want to be super-conservative, you can also specify, say, a 1-hour window when exceptions apply.
Lastly, if you decided to adopt the approach suggested above, while you are at it, also install django-security, add djangosecurity to INSTALLED_APPS, and add to the end of your settings.py:
if not (DEBUG or RUNNING_DEV_SERVER):
### Security
SECURE_SSL_REDIRECT = True
SECURE_CONTENT_TYPE_NOSNIFF = True
SECURE_HSTS_SECONDS = 86400000
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_BROWSER_XSS_FILTER = True
SESSION_COOKIE_SECURE = True
SESSION_COOKIE_HTTPONLY = True
CSRF_COOKIE_HTTPONLY = True # May have problems with Ajax
CSRF_COOKIE_SECURE = True