I'm pretty new to doing sysadmin stuff for my development and to the django framework. I want to have a different username/password for my local dev station and to my production environment.
I'm using dotcloud as the server. I can write a post install script (in python, bash, whatever) and it will execute it on every new push.
However I don't know how to go about this. Do I need to write this myself? is there a python/django build automation tools that will help me with that?
Clarification: how can I change debug=false in settings.py to true just on the server?
The django standard way is to use an environmanet variable DJANGO_SETTINGS_MODULE. Point it to different settings and let both import a common settings module for common things:
# settings_production.py
from settings_common import *
DEBUG = False
DATABASES = {...}
# settings_development.py
from settings_common import *
DEBUG = True
DATABASES = {...}
# settings_common.py
INSTALLED_APPS = (...) # etc
You can also use an alternative strategy of using one main settings and import names from another one depending on some system condition, like getting os.platform.node() or socket.gethostname() and switch over that value (or part of it).
reversed_hostname_parts = socket.gethostname().split('.').reverse()
host_specific = {
('com', 'dotcloud'): 'production',
('local'): 'dev',
}
for index in range(len(reversed_hostname_parts)):
identifier = tuple(reversed_hostname_parts[:index+1])
if identifier in host_specific:
extra_settings = host_specific[identifier]
break
else: # executed when the loop has not been `break`ed
extra_settings = 'dev' # any default value
if extra_settings == 'dev':
from development_settings import *
elif extra_settings == 'production':
from production_settings import *
EDIT: added link
See https://code.djangoproject.com/wiki/SplitSettings for other strategies.
I usually import my development settings at the end of production settings.py, if my project is residing on a local directory structure.
You can also store your DB settings and other settings that are different in production and development in a separate file and remove them from your SVN, Git of whatever you use.
Just add this at the end of your settings.py:
try:
from myapp.specific_settings import *
except ImportError:
pass
In this case, specific_settings will be different in production and development environment.
If you want to dynamically choose between development and production servers use this at the end of settings:
import os
directory = os.path.dirname(__file__)
if directory == '/home/yourname/development/':
from myapp.development_settings import *
else:
from myapp.production_settings import *
Note that I wrote this on top of my head and there might be some bugs in it. I will verify that when I go home.
Related
Based on the Configuration Handling Documents for Flask the section of Configuring from Files mentions a possibility to configure the App using files however it provides no example or mention of files that are not Python Files.
Is it possible to configure apps via files like config.yml or config.toml?
My Current flask app has configurations for two distinct databases and since I am using flask-restplus there are additional configurations for Swagger documentations.
Snippet:
from flask import Flask
app = Flask(__name__)
def configure_app(flask_app):
# MongoDB Setting
flask_app.config['MONGO_URI'] = 'mongodb://user:password#mongo_db_endpoint:37018/myDB?authSource=admin'
flask_app.config['MONGO_DBNAME'] = 'myDB'
# InfluxDB Setting
flask_app.config['INFLUXDB_HOST'] = 'my_influxdb_endpoint'
flask_app.config['INFLUXDB_PORT'] = 8086
flask_app.config['INFLUXDB_USER'] = 'influx_user'
flask_app.config['INFLUXDB_PASSWORD'] = 'influx_password'
flask_app.config['INFLUXDB_SSL'] = True
flask_app.config['INFLUXDB_VERIFY_SSL'] = False
flask_app.config['INFLUXDB_DATABASE'] = 'IoTData'
# Flask-Restplus Swagger Configuration
flask_app.config['RESTPLUS_SWAGGER_UI_DOC_EXPANSION'] = 'list'
flask_app.config['RESTPLUS_VALIDATE'] = True
flask_app.config['RESTPLUS_MASK_SWAGGER'] = False
flask_app.config['ERROR_404_HELP'] = False
def main():
configure_app(app)
if __name__ == "__main__":
main()
I would like to avoid setting large number of Environment Variables and wish to configure them using a config.toml file?
How is this achieved in flask?
You can use the .cfg files and from_envvar to achieve this. Create config file with all your environment variables.
my_config.cfg
MONGO_URI=mongodb://user:password#mongo_db_endpoint:37018
..
..
ERROR_404_HELP=False
Then set the env var APP_ENVS=my_config.cfg. Now all you need to do is use from_envvars given by Flask.
def configure_app(flask_app):
flask_app.config.from_envvar('APP_ENVS')
# configure any other things
# register blue prints if you have any
Quoting from documentation:
Configuring from Data Files
It is also possible to load configuration from a file in a format of
your choice using from_file(). For example to load from a TOML file:
import toml
app.config.from_file("config.toml", load=toml.load)
Or from a JSON file:
import json
app.config.from_file("config.json", load=json.load)
EDIT: The above feature is new for v2.0.
Link to the documentation reference:
Class Flask.config, method from_file(filename, load, silent=False)
I would like certain parts of my code to not run while it is being run locally.
This is because, I am having trouble installing certain dependencies locally, for the code to run.
Specifically, memcache doesn't work for me locally.
#app.route('/some_url_route/')
#cache.cached(timeout=2000) #ignore this locally
def show_a_page():
How would the app somehow ignore the cache section of the code above, when running locally?
In my code I follow a Django-esq model and have a main settings.py file I keep all my settings in.
In that file putting DEBUG = True for your local environment (and False for production) I then use:
from settings import DEBUG
if DEBUG:
# Do this as it's development
else:
# Do this as it's production
So in your cache decorator include a similar line that only checks memcached if DEBUG=False
You can then load all these settings into your Flask setup as detailed in the configuration documentation.
If you're using Flask-Cache, then just edit the settings:
if app.debug:
app.settings.CACHE_TYPE = 'null' # the cache that doesn't cache
cache = Cache(app)
...
A better approach would be to have separate settings for production and development. I use a class-based approach:
class BaseSettings(object):
...
class DevelopmentSettings(BaseSettings):
DEBUG = True
CACHE_TYPE = 'null'
...
class ProductionSettings(BaseSettings):
CACHE_TYPE = 'memcached'
...
And then import the appropriate object when you setup your app (config.py is the name of the file which contains the settings):
app.config.from_object('config.DevelopmentConfig')
I have following code written in python in order to communicate with ExistDB using eulexistdb module.
from eulexistdb import db
class TryExist:
def __init__(self):
self.db = db.ExistDB(server_url="http://localhost:8899/exist")
def get_data(self, query):
result = list()
qresult = self.db.executeQuery(query)
hits = self.db.getHits(qresult)
for i in range(hits):
result.append(str(self.db.retrieve(qresult, i)))
return result
query = '''
let $x:= doc("/db/sample/books.xml")
return $x/bookstore/book/author/text()
'''
a = TryExist()
response = a.get_data(query)
print response
I am amazed that this code runs fine in Aptana Studio 3 giving me the output I want, but when running from other IDE or using command "python.exe myfile.py" brings following error:
django.core.exceptions.ImproperlyConfigured: Requested setting EXISTDB_TIMEOUT, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
I used my own localsetting.py to solve the problem using following code:
import os
# must be set before importing anything from django
os.environ['DJANGO_SETTINGS_MODULE'] = 'localsettings'
... writing link for existdb here...
Then I get error as:
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
How do I configure the setting in Django to suit for ExistDB? Help me here please..
Never Mind. I found the answer with little research from this site. What I did was created a localsetting.py file with following configurations.
EXISTDB_SERVER_USER = 'user'
EXISTDB_SERVER_PASSWORD = 'admin'
EXISTDB_SERVER_URL = "http://localhost:8899/exist"
EXISTDB_ROOT_COLLECTION = "/db"
and in my main file myfile.py I used :
from localsettings import EXISTDB_SERVER_URL
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'localsettings.py'
and In the class TryExist I changed in __ init __() as:
def __init__(self):
self.db = db.ExistDB(server_url=EXISTDB_SERVER_URL)
PS: Using only os.environ['DJANGO_SETTINGS_MODULE'] = 'localsettings' brings the django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty..
The reason your code works in an IDE but not at the command line is probably that you have a difference in what Python environments are used to run your code.
I've done a couple of tests:
Virtualenv with eulexistdb installed but not Django. eulexistdb tries to load django.conf but fails and so does not try to get its configuration from a Django configuration. Ultimately, your code runs without error.
Virtualenv with 'eulexistdb*and* Django:eulexistdbtries to loaddjango.conf` and succeed. I then tries to get is configuration from the Django configuration but fails. I get the same error you describe in your question.
To prevent the error in the presence of a Django installation, the problem can be fixed by adding a Django configuration like you did in your accepted self-answer. But if the code you are writing does not otherwise use Django, that's a bit of a roundabout way to get your code to run. The most direct way to fix the problem is to simply add a timeout parameter to the code that creates the ExistDB instance:
self.db = db.ExistDB(
server_url="http://localhost:8080/exist", timeout=None)
If you do this, then there won't be any error. Setting the timeout to None leaves the default behavior in place but prevents eulexistdb from looking for a Django configuration.
In trying to find a place to store and save settings beyond settings.py and the database, I used an environment.json for environment variables. I import these in settings.py.
My problem is that when I try to change or store new values in my environment, env, settings.py does not notice the change - perhaps because the time and number of times settings.py is read by Django.
Is there a way I would be able to use my environment variables the way I want like attempted below?
# settings.py
import json
with open('/home/dotcloud/environment.json') as f:
env = json.load(f)
EMAIL_HOST = env.get('EMAIL_PORT', '500')
# views.py
import json
def site_configuration(request):
with open('/home/dotcloud/environment.json') as f:
env = json.load(f)
if request.method == 'POST':
os.environ['EMAIL_PORT'] = request.POST['email_port']
return render(request, ...)
# python manage.py shell demo
>>> import json
>>> with open('/home/dotcloud/environment.json') as f:
... env = json.load(f)
...
>>> project_settings.EMAIL_PORT
'500'
>>> env['EMAIL_PORT']
Traceback (most recent call last):
File "<console>", line 1, in <module>
KeyError: 'EMAIL_PORT'
>>> env['EMAIL_PORT'] = "123"
>>> env['EMAIL_PORT']
'123'
>>> project_settings.EMAIL_PORT
'500'
>>> project_settings.EMAIL_PORT == env['EMAIL_PORT']
False'
And if not, how else could I store changeable settings that are retrieved by settings.py somewhere in my Django project?
You might want to look into foreman (GitHub) or honcho (GitHub). Both of these look for a .env file in your current directory from which to load local environment variables.
My .env looks like this for most projects (I use dj-database-url for database configuration):
DATABASE_URL=sqlite://localhost/local.db
SECRET_KEY=<a secret key>
DEBUG=True
In your settings.py file, you can load these settings from os.environ like this:
import os
DEBUG = os.environ.get('DEBUG', False)
If there are required settings, you can assert their presence before trying to set them:
assert 'SECRET_KEY' in os.environ, 'Set SECRET_KEY in your .env file!'
SECRET_KEY = os.environ['SECRET_KEY']
I've been using this method of handling local settings for the last few projects I've started and I think it works really well. One caveat is to never commit your .env to source control. These are local settings that exist only for the current configuration and should be recreated for a different environment.
I see the question changed slightly, the original answers are still below but this one has a slightly different answer:
First, make sure you are using the right settings.py (print 'This file is being loaded' should do the trick).
Second, personally I would advise against using json files for config since it is less dynamic than Python files, but it should work regardless.
My recommended way of doing something like this:
create a base_settings.py file with your standard settings
create a settings.py which will be your default settings import. This file should have a from base_settings import * at the top to inherit the base settings.
If you want to have a custom settings file, dotcloud_settings.py for example, simply add the from dotcloud_settings import settings (or base_settings) and set the environment variable DJANGO_SETTINGS_MODULE to dotcloud_settings or your_project.dotcloud_settings depending on your setup.
Do note that you should be very careful with importing Django modules from these settings files. If any module does a from django.conf import settings it will stop parsing your settings after that point.
As for using json files, roughly the same principle of course:
Once again, make sure you don't have anything that imports django.conf.settings here
Make all of the variables within your json file global to your settings file:
import json
with open('/home/dotcloud/environment.json') as f:
env = json.load(f)
# A little hack to make all variables within our env global
globals().update(env)
Regardless though, I'd recommend turning this around and letting the settings file import this module instead.
Also, Django doesn't listen to environment variables by default (besides the DJANGO_SETTINGS_MODULE so that might be the problem too.
Hey, I'm trying to work out with /remote_api with a django-patch app engine app i got running.
i want to select a few rows from my online production app locally.
i cant seem to manage to do so, everything authenticates fine, it doesnt breaks on imports, but when i try to fetch something it just doesnt print anything.
Placed the test python inside my local app dir.
#!/usr/bin/env python
#
import os
import sys
# Hardwire in appengine modules to PYTHONPATH
# or use wrapper to do it more elegantly
appengine_dirs = ['myworkingpath']
sys.path.extend(appengine_dirs)
# Add your models to path
my_root_dir = os.path.abspath(os.path.dirname(__file__))
sys.path.insert(0, my_root_dir)
from google.appengine.ext import db
from google.appengine.ext.remote_api import remote_api_stub
import getpass
APP_NAME = 'Myappname'
os.environ['AUTH_DOMAIN'] = 'gmail.com'
os.environ['USER_EMAIL'] = 'myuser#gmail.com'
def auth_func():
return (raw_input('Username:'), getpass.getpass('Password:'))
# Use local dev server by passing in as parameter:
# servername='localhost:8080'
# Otherwise, remote_api assumes you are targeting APP_NAME.appspot.com
remote_api_stub.ConfigureRemoteDatastore(APP_NAME,
'/remote_api', auth_func)
# Do stuff like your code was running on App Engine
from channel.models import Channel, Channel2Operator
myresults = mymodel.all().fetch(10)
for result in myresults:
print result.key()
it doesn't give any error or print anything. so does the remote_api console example google got. when i print the myresults i get [].
App Engine patch monkeypatches the ext.db module, mutilating the kind names. You need to make sure you import App Engine patch from your script, to give it the opportunity to mangle things as per usual, or you won't see any data returned.