We Keep Coding

Configure Python Flask RESTplus app via TOML file

Based on the Configuration Handling Documents for Flask the section of Configuring from Files mentions a possibility to configure the App using files however it provides no example or mention of files that are not Python Files.
Is it possible to configure apps via files like config.yml or config.toml?
My Current flask app has configurations for two distinct databases and since I am using flask-restplus there are additional configurations for Swagger documentations.
Snippet:
from flask import Flask
app = Flask(__name__)
def configure_app(flask_app):
# MongoDB Setting
flask_app.config['MONGO_URI'] = 'mongodb://user:password#mongo_db_endpoint:37018/myDB?authSource=admin'
flask_app.config['MONGO_DBNAME'] = 'myDB'
# InfluxDB Setting
flask_app.config['INFLUXDB_HOST'] = 'my_influxdb_endpoint'
flask_app.config['INFLUXDB_PORT'] = 8086
flask_app.config['INFLUXDB_USER'] = 'influx_user'
flask_app.config['INFLUXDB_PASSWORD'] = 'influx_password'
flask_app.config['INFLUXDB_SSL'] = True
flask_app.config['INFLUXDB_VERIFY_SSL'] = False
flask_app.config['INFLUXDB_DATABASE'] = 'IoTData'
# Flask-Restplus Swagger Configuration
flask_app.config['RESTPLUS_SWAGGER_UI_DOC_EXPANSION'] = 'list'
flask_app.config['RESTPLUS_VALIDATE'] = True
flask_app.config['RESTPLUS_MASK_SWAGGER'] = False
flask_app.config['ERROR_404_HELP'] = False
def main():
configure_app(app)
if __name__ == "__main__":
main()
I would like to avoid setting large number of Environment Variables and wish to configure them using a config.toml file?
How is this achieved in flask?

You can use the .cfg files and from_envvar to achieve this. Create config file with all your environment variables.
my_config.cfg
MONGO_URI=mongodb://user:password#mongo_db_endpoint:37018
..
..
ERROR_404_HELP=False
Then set the env var APP_ENVS=my_config.cfg. Now all you need to do is use from_envvars given by Flask.
def configure_app(flask_app):
flask_app.config.from_envvar('APP_ENVS')
# configure any other things
# register blue prints if you have any

Quoting from documentation:
Configuring from Data Files
It is also possible to load configuration from a file in a format of
your choice using from_file(). For example to load from a TOML file:
import toml
app.config.from_file("config.toml", load=toml.load)
Or from a JSON file:
import json
app.config.from_file("config.json", load=json.load)
EDIT: The above feature is new for v2.0.
Link to the documentation reference:
Class Flask.config, method from_file(filename, load, silent=False)

Change static folder from config in Flask

Has anyone tried this snippet flask config based static folder code cnippet?
The code:
import flask
class MyFlask(flask.Flask):
#property
def static_folder(self):
if self.config.get('STATIC_FOLDER') is not None:
return os.path.join(self.root_path,
self.config.get('STATIC_FOLDER'))
#static_folder.setter
def static_folder(self, value):
self.config.get('STATIC_FOLDER') = value
# Now these are equivalent:
app = Flask(__name__, static_folder='foo')
app = MyFlask(__name__)
app.config['STATIC_FOLDER'] = 'foo'
In my case in complains about this line:
self.config.get('STATIC_FOLDER') = value
The error message: Can't assign to function call
Does anyone how to set the static_folder from the config.py file in Flask?

Okay, I assume you want to use a custom path to the static folder for whatever reason. I wanted to do the same for the sake of better app modularity.
Here's my app folder structure:
instance/
core/
|_templates/
|_static/
|_views.py
run.py
config.py
As you can see, my static folder is inside the core folder.
In run.py, you can do the following:
app = Flask(__name__, static_url_path=None)
if __name__ == '__main__':
app.config.from_object('config')
# config file has STATIC_FOLDER='/core/static'
app.static_url_path=app.config.get('STATIC_FOLDER')
# set the absolute path to the static folder
app.static_folder=app.root_path + app.static_url_path
print(app.static_url_path)
print(app.static_folder)
app.run(
host=app.config.get('HOST'),
port=app.config.get('PORT'),
threaded=True
)
This is what I did, and it works perfectly fine. I'm using flask 0.12.

I don't know anything about that snippet, but
some_function(...) = some_value
is never valid Python (Python doesn't have l-values). It looks like config has a dict-like interface, so the offending line should probably just be
self.config['STATIC_FOLDER'] = value
Probably a copy-and-paste error from the getter definition above the setter.

app = Flask(__name__, static_url_path="/STATIC_FOLDER", static_folder='STATIC_FOLDER')

Yes, In one of my projects I am using/setting a custom path for STATIC_FOLDER. You can set the path to STATIC_FOLDER in config.py like below:
STATIC_PATH = '<project-name>/<path-to-static-folder>/'
ex:
STATIC_PATH = 'myApp/static/'
If you can write your project structure then I can answer it as per your requirements.

FYI if you want your directory to be outside of the server directory the best solutions I found so far are either to make a copy of your directory into the server directory before startup in your main(), or to create a symlink.

Fixtures for Mongoengine with Django not working

I am using Mongoengine(version: 0.9.0 ) with Django(version: 1.8).
This is my settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.dummy'
}
}
MONGO_DBNAME = "mydatabasename"
MONGO_HOSTNAME = "localhost"
connect(MONGO_DBNAME, host=MONGO_HOSTNAME)
I want to have fixtures for the application. I have created initial_data.json in myapp/fixtures/ location.
When I run the command python manage.py dumpdata , I get the following error :
CommandError: Unable to serialize database: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
Questions:
1) Any workaround for this problem ?
2) Is there any other way to load the initial data ?
References at this link
Thank you

Mongoengine itsn a backend(in django terminology). Its has own models (schemas) and DOM (like ORM in docuemnt db's) but it dont have a Django backend adapters.
You can use it. But there is issue while workind with out-of-box Django solution like Tests, Fixtures, etc.
You need to write your own loader, sadenly but true.
I see 2 options here:
You can try to use Django MongoDB Engine
You can write your own loader for mongodb
Ill write my own fixture loader for tests.
I have a json file where mapped all fixture file ill need to load to db.
So a fast example here:
import bson
import os
from django.conf import settings
from mongoengine.connection import get_db
def _get_db(self):
self.db = get_db()
def _load_fixtures(self, clear_before_load=True):
"""
Load to db a fixtures from folder fixtures/{{DB_NAME}}/{{COLLECTION_NAME}} before each test.
In file fixtures.json mapped collection name and file name for it.
"""
fixture_path = lambda file_name: os.path.join(settings.FIXTURES_DIR, self.db.name, file_name)
with open(settings.COLLECTION_FIXTURES_PATH) as file_object:
db_collections = loads(file_object.read())
for collection_name, filename in db_collections.items():
collection = self.db[collection_name]
if clear_before_load:
collection.remove()
path = fixture_path(filename)
if os.path.exists(path) and os.path.isfile(path):
with open(path, 'r') as raw_data:
collection_data = bson.decode_all(raw_data.read())
for document in collection_data:
collection.save(document)

There is no support for fixtures on mongoengine, and I don't think the mongoengine team is continuing the plugin as of version 0.9.0.
What I ended up doing to load initial data for mongoDB is to create a script called startup.py in my project folder.
startup.py:
from {{app}}.models import Sample
def init():
if Sample.objects(name="test").count() == 0: # a flag to prevent initial data repetition
Sample(name="test").save()
Next is to run this script on Django's startup. The entry point of Django project is when DJANGO_SETTINGS_MODULE is first loaded at wsgi.py:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "{{project_name}}.settings")
import {{project_name}}.startup as startup
startup.init()
application = get_wsgi_application()
With this setup, when you run python manage.py runserver, the init() on startup.py will run and the data you set will be inserted to the DB.
Hope this helps.

How to add your app specific settings in pyramid rest framework?

I am new to pyramid.
The Issue is I am not able to figure out how app specific settings (key value pairs) work in pyramid.
This is what I have done after various google searches and other stackoverflow answers:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
if '__file__' in global_config:
settings.update(
load_sensitive_settings(global_config['__file__'], global_config))
config = Configurator(settings=settings)
config.include('pyramid_chameleon')
# config.add_static_view('static', 'static', cache_max_age=3600)
# config.add_route('home', '/')
config.add_route(
'tags',
'/tags', request_method='POST', accept='application/json', )
config.scan()
return config.make_wsgi_app()
def load_sensitive_settings(configurationPath, defaultByKey):
'Load sensitive settings from hidden configuration file'
# configFolder, configName = os.path.split(configurationPath)
# sensitivePath = os.path.join(configFolder, '.' + configName)
sensitivePath = configurationPath
settings = {}
configParser = ConfigParser.ConfigParser(defaultByKey)
if not configParser.read(sensitivePath):
log.warn('Could not open %s' % sensitivePath)
return settings
settings.update(configParser.items('custom'))
return settings
I have a file where I try to fetch settings like this:
from pyramid.threadlocal import get_current_registry
settings = get_current_registry().settings
value = settings['my_key']
But I always get settings object as None.
This is how I am defining my custom settings in development.ini
[custom]
my_key = ''
This is how I start my server in develpoment
pserve development.ini
I have read that request.settings can give me settings, But that approach is not feasible for me as my key contains the name of a file which is 1.5GBs and it has to be present in memory all the time. It takes around 5 minutes to load that file in server, hence cannot load the file on demand.
Please advice.
Thanks a lot for all the help in advance.
Update:
Thanks to all the answers provided, I finally made it work.
This is how my main function looks like:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
config = Configurator(settings=settings)
config.include('pyramid_chameleon')
if '__file__' in global_config:
init_config(global_config['__file__'])
And I made a config file, this is how my config file looks like:
import ConfigParser
settings = dict()
def init_config(filename):
config = ConfigParser.ConfigParser()
config.read(filename)
settings_dict = config.items('custom')
settings.update(settings_dict)
Now wherever I want settings, I just do:
from projectname.config import settings
settings.get('my_key')
And I put my app specific settings (development/production.py) like this
[custom]
my_key = value
Regards
HM

Easiest way is putting your settings to the app main section with dot separated names. Example:
[app:main]
websauna.site_name = Trees
websauna.site_tag_line = Enjoy
websauna.site_url = http://localhost:6543
websauna.site_author = Trees team
Then you can do:
my_settings_value = request.registry.settings.get("websauna.site_name", "Default value)
WSGI pipeline does not bring you settings from other sections and you need to reparse the INI file with ConfigParser if you want to access the other sections (as far as I know).
If you need to load a lot of data during development time just store a filename in settings and load the file when you need to access the data, so that you don't slow the web server startup.

Here is my working solution:
config.ini
[APP.CONFIG]
url = http://....
[SMTP.CONFIG]
smtp.server = ...
smtp.port = 25
smtp.login = ...
smtp.password = ...
smtp.from = ...
[DB.CONFIG]
db.database=...
db.host=...
db.port=..
db.user=...
db.password=...
config.py
import configparser
config = configparser.ConfigParser()
config._interpolation = configparser.ExtendedInterpolation()
config.read(encoding='utf-8', filenames=['path to file/config.ini'])
smtp = config['SMTP.CONFIG']
db = config['DB.CONFIG']
mail = config['APP.CONFIG']
And how i use it in APP
from config import db
host = db['db.host']

If, like me, you are using PasteDeploy with Pyramid, the Pyramid docs here explain how you can use a [DEFAULT] section in your .ini configuration file to hold your custom parameters.
You might also benefit from reading the documentation on .ini files, since it gives some snippets which make it all much clearer.

Cannot use environment variables for settings in Django

In trying to find a place to store and save settings beyond settings.py and the database, I used an environment.json for environment variables. I import these in settings.py.
My problem is that when I try to change or store new values in my environment, env, settings.py does not notice the change - perhaps because the time and number of times settings.py is read by Django.
Is there a way I would be able to use my environment variables the way I want like attempted below?
# settings.py
import json
with open('/home/dotcloud/environment.json') as f:
env = json.load(f)
EMAIL_HOST = env.get('EMAIL_PORT', '500')
# views.py
import json
def site_configuration(request):
with open('/home/dotcloud/environment.json') as f:
env = json.load(f)
if request.method == 'POST':
os.environ['EMAIL_PORT'] = request.POST['email_port']
return render(request, ...)
# python manage.py shell demo
>>> import json
>>> with open('/home/dotcloud/environment.json') as f:
... env = json.load(f)
...
>>> project_settings.EMAIL_PORT
'500'
>>> env['EMAIL_PORT']
Traceback (most recent call last):
File "<console>", line 1, in <module>
KeyError: 'EMAIL_PORT'
>>> env['EMAIL_PORT'] = "123"
>>> env['EMAIL_PORT']
'123'
>>> project_settings.EMAIL_PORT
'500'
>>> project_settings.EMAIL_PORT == env['EMAIL_PORT']
False'
And if not, how else could I store changeable settings that are retrieved by settings.py somewhere in my Django project?

You might want to look into foreman (GitHub) or honcho (GitHub). Both of these look for a .env file in your current directory from which to load local environment variables.
My .env looks like this for most projects (I use dj-database-url for database configuration):
DATABASE_URL=sqlite://localhost/local.db
SECRET_KEY=<a secret key>
DEBUG=True
In your settings.py file, you can load these settings from os.environ like this:
import os
DEBUG = os.environ.get('DEBUG', False)
If there are required settings, you can assert their presence before trying to set them:
assert 'SECRET_KEY' in os.environ, 'Set SECRET_KEY in your .env file!'
SECRET_KEY = os.environ['SECRET_KEY']
I've been using this method of handling local settings for the last few projects I've started and I think it works really well. One caveat is to never commit your .env to source control. These are local settings that exist only for the current configuration and should be recreated for a different environment.

I see the question changed slightly, the original answers are still below but this one has a slightly different answer:
First, make sure you are using the right settings.py (print 'This file is being loaded' should do the trick).
Second, personally I would advise against using json files for config since it is less dynamic than Python files, but it should work regardless.
My recommended way of doing something like this:
create a base_settings.py file with your standard settings
create a settings.py which will be your default settings import. This file should have a from base_settings import * at the top to inherit the base settings.
If you want to have a custom settings file, dotcloud_settings.py for example, simply add the from dotcloud_settings import settings (or base_settings) and set the environment variable DJANGO_SETTINGS_MODULE to dotcloud_settings or your_project.dotcloud_settings depending on your setup.
Do note that you should be very careful with importing Django modules from these settings files. If any module does a from django.conf import settings it will stop parsing your settings after that point.
As for using json files, roughly the same principle of course:
Once again, make sure you don't have anything that imports django.conf.settings here
Make all of the variables within your json file global to your settings file:
import json
with open('/home/dotcloud/environment.json') as f:
env = json.load(f)
# A little hack to make all variables within our env global
globals().update(env)
Regardless though, I'd recommend turning this around and letting the settings file import this module instead.
Also, Django doesn't listen to environment variables by default (besides the DJANGO_SETTINGS_MODULE so that might be the problem too.