We Keep Coding

Configure Python Flask RESTplus app via TOML file

Based on the Configuration Handling Documents for Flask the section of Configuring from Files mentions a possibility to configure the App using files however it provides no example or mention of files that are not Python Files.
Is it possible to configure apps via files like config.yml or config.toml?
My Current flask app has configurations for two distinct databases and since I am using flask-restplus there are additional configurations for Swagger documentations.
Snippet:
from flask import Flask
app = Flask(__name__)
def configure_app(flask_app):
# MongoDB Setting
flask_app.config['MONGO_URI'] = 'mongodb://user:password#mongo_db_endpoint:37018/myDB?authSource=admin'
flask_app.config['MONGO_DBNAME'] = 'myDB'
# InfluxDB Setting
flask_app.config['INFLUXDB_HOST'] = 'my_influxdb_endpoint'
flask_app.config['INFLUXDB_PORT'] = 8086
flask_app.config['INFLUXDB_USER'] = 'influx_user'
flask_app.config['INFLUXDB_PASSWORD'] = 'influx_password'
flask_app.config['INFLUXDB_SSL'] = True
flask_app.config['INFLUXDB_VERIFY_SSL'] = False
flask_app.config['INFLUXDB_DATABASE'] = 'IoTData'
# Flask-Restplus Swagger Configuration
flask_app.config['RESTPLUS_SWAGGER_UI_DOC_EXPANSION'] = 'list'
flask_app.config['RESTPLUS_VALIDATE'] = True
flask_app.config['RESTPLUS_MASK_SWAGGER'] = False
flask_app.config['ERROR_404_HELP'] = False
def main():
configure_app(app)
if __name__ == "__main__":
main()
I would like to avoid setting large number of Environment Variables and wish to configure them using a config.toml file?
How is this achieved in flask?

You can use the .cfg files and from_envvar to achieve this. Create config file with all your environment variables.
my_config.cfg
MONGO_URI=mongodb://user:password#mongo_db_endpoint:37018
..
..
ERROR_404_HELP=False
Then set the env var APP_ENVS=my_config.cfg. Now all you need to do is use from_envvars given by Flask.
def configure_app(flask_app):
flask_app.config.from_envvar('APP_ENVS')
# configure any other things
# register blue prints if you have any

Quoting from documentation:
Configuring from Data Files
It is also possible to load configuration from a file in a format of
your choice using from_file(). For example to load from a TOML file:
import toml
app.config.from_file("config.toml", load=toml.load)
Or from a JSON file:
import json
app.config.from_file("config.json", load=json.load)
EDIT: The above feature is new for v2.0.
Link to the documentation reference:
Class Flask.config, method from_file(filename, load, silent=False)

Retrieving config from a blueprint in Sanic app

I have a Sanic application, and want to retrieve app.config from a blueprint as it holds MONGO_URL, and I will pass it to a repository class from the blueprint.
However, I could not find how to get app.config in a blueprint. I have also checked Flask solutions, but they are not applicable to Sanic.
My app.py:
from sanic import Sanic
from routes.authentication import auth_route
from routes.user import user_route
app = Sanic(__name__)
app.blueprint(auth_route, url_prefix="/auth")
app.blueprint(user_route, url_prefix="/user")
app.config.from_envvar('TWEETBOX_CONFIG')
app.run(host='127.0.0.1', port=8000, debug=True)
My auth blueprint:
import jwt
from sanic import Blueprint
from sanic.response import json, redirect
from domain.user import User
from repository.user_repository import UserRepository
...
auth_route = Blueprint('authentication')
mongo_url = ?????
user_repository = UserRepository(mongo_url)
...
#auth_route.route('/signin')
async def redirect_user(request):
...

The Sanic way...
Inside a view method, you can access the app instance from the request object. And, therefore access your configuration.
#auth_route.route('/signin')
async def redirect_user(request):
configuration = request.app.config
2021-10-10 Update
There are two newer ways to get to the configuration values (or, perhaps more accuratlely getting the application instance from which you can get the configuration). The first version might be more on point to answering the question of how to get to the config from the blueprint. However, the second option is probably the preferred method since it is precisely intended for this kind of use.
Alternative #1
Blueprints have access to the Sanic applications they are attached to beginning with v21.3.
Therefore, if you have a blueprint object, you can trace that back to the application instance, and therefore also the config value.
app = Sanic("MyApp")
bp = Blueprint("MyBlueprint")
app.blueprint(bp)
assert bp.apps[0] is app
The Blueprint.apps property is a set because it is possible to attach a single blueprint to multiple applications.
Alternative #2
Sanic has a built-in method for retrieving an application instance from the global scope beginning in v20.12. This means that once an application has been instantiated, you can retrieve it using: Sanic.get_app().
app = Sanic("MyApp")
assert Sanic.get_app() is app
This method will only work if there is a single Sanic instance available. If you have multiple application instances, you will need to use the optional name argument:
app1 = Sanic("MyApp")
app2 = Sanic("MyOtherApp")
assert Sanic.get_app("MyApp") is app1

I would suggest a slightly different approach, based on the 12 Factor App (very interesting read which, among others, provides a nice guideline on how to protect and isolate your sensitive info).
The general idea is to place your sensitive and configuration variables in a file that is going to be gitignored and therefore will only be available locally.
I will try to present the method I tend to use in order to be as close as possible to the 12 Factor guidelines:
Create a .env file with your project variables in it:
MONGO_URL=http://no_peeking_this_is_secret:port/
SENSITIVE_PASSWORD=for_your_eyes_only
CONFIG_OPTION_1=config_this
DEBUG=True
...
(Important) Add .env and .env.* on your .gitignore file, thus protecting your sensitive info from been uploaded to GitHub.
Create an env.example (be careful not to name it with a . in the beginning, because it will get ignored).
In that file, you can put an example of the expected configuration in order to be reproducible by simply copy, paste, rename to .env.
In a file named settings.py, use decouple.config to read your config file into variables:
from decouple import config
MONGO_URL = config('MONGO_URL')
CONFIG_OPTION_1 = config('CONFIG_OPTION_1', default='')
DEBUG = config('DEBUG', cast=bool, default=True)
...
Now you can use these variables wherever is necessary for your implementation:
myblueprint.py:
import settings
...
auth_route = Blueprint('authentication')
mongo_url = settings.MONGO_URL
user_repository = UserRepository(mongo_url)
...
As a finisher, I would like to point out that this method is framework (and even language) agnostic so you can use it on Sanic as well as Flask and everywhere you need it!

I think you can create a config.py to save your configuration, just like
config.py
config = {
'MONGO_URL'：'127.0.0.1:27017'
}
and use it in app.py
from config import config
mongo_url = config['MONGO_URL']

There is a variable named current_app in Flask. You can use current_app.config["MONGO_URL"].
But I am not familiar with Sanic.

Setting Heroku DATABASE_URL to Pyramid Config

Goal: Set the Heroku DATABASE_URL variable to sqlalchemy.url=postgres://... settings in __init__.py and development.ini file. Currently, I am connecting directly to the database address (which can change).
Issue as reported by Heroku support:
If you have hard coded the database connection string into your ini
file that is probably not the best idea. While it will work for now,
if sometime in the future we need to change where you database is
running (which does happen for various reasons) then your application
will no longer connect to your database. If your database does move,
we do keep the DATABASE_URL updated so your application should use
this. Maybe change sqlalchemy.url to sqlalchemy.url = os.environ.get('DATABASE_URL') if this is not what it is already set to.
However, the address sqlalchemy.url = os.environ.get('DATABASE_URL') does NOT work. It crashes my app. I have even attempted: sqlalchemy.url = postgresql://os.environ.get('DATABASE_URL'), sqlalchemy.url = postgres://os.environ.get('DATABASE_URL'), and finally sqlalchemy.url = postgres://'DATABASE_URL'. All of which do NOT work.
SQLALCEHMY engine_config setup: docs
sqlalchemy.engine_from_config(configuration, prefix='sqlalchemy.',
**kwargs)
Create a new Engine instance using a configuration dictionary.
The dictionary is typically produced from a config file.
The keys of interest to engine_from_config() should be prefixed, e.g.
sqlalchemy.url, sqlalchemy.echo, etc. The ‘prefix’ argument indicates
the prefix to be searched for. Each matching key (after the prefix is
stripped) is treated as though it were the corresponding keyword
argument to a create_engine() call.
The only required key is (assuming the default prefix) sqlalchemy.url,
which provides the database URL.
A select set of keyword arguments will be “coerced” to their expected
type based on string values. The set of arguments is extensible
per-dialect using the engine_config_types accessor.
Parameters: configuration – A dictionary (typically produced from a
config file, but this is not a requirement). Items whose keys start
with the value of ‘prefix’ will have that prefix stripped, and will
then be passed to create_engine. prefix – Prefix to match and then
strip from keys in ‘configuration’. kwargs – Each keyword argument to
engine_from_config() itself overrides the corresponding item taken
from the ‘configuration’ dictionary. Keyword arguments should not be
prefixed.
Outside Example (doesn't work for me):
I believe the issue is with the way settings and engine are setup in my app. I found this tutorial helpful, but my code is different: Environment Variables in Pyramid
What we ultimately want to do is dynamically set the sqlalchemy.url to
the value of our DATABASE_URL environment variable.
learning_journal/init.py is where our .ini file’s configuration
gets bound to our Pyramid app. Before the current settings get added
to the Configurator, we can use os.environ to bring in our
environment’s DATABASE_URL.
# __init__.py
import os
from pyramid.config import Configurator
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
settings["sqlalchemy.url"] = os.environ["DATABASE_URL"]
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
Because we should always try to keep code DRY (and prevent future confusion), remove the sqlalchemy.url keyword from development.ini.
MY CODE:
init.py
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application"""
#global_config argument is a dictionary of key/value pairs mentioned in the [DEFAULT] section of an development.ini file
# **settings argument collects another set of arbitrary key/value pairs
#The main function first creates a SQLAlchemy database engine using sqlalchemy.engine_from_config() from the sqlalchemy. prefixed settings in the development.ini file's [app:main] section. This will be a URI (something like sqlite://):
engine = engine_from_config(settings, 'sqlalchemy.')
Session.configure(bind=engine)
Base.metadata.bind = engine
...
config.include('pyramid_jinja2')
config.include('pyramid_mailer')
config.add_static_view('static', 'static', cache_max_age=3600)
development.ini
#former db:
#sqlalchemy.url = postgres://localhost/NOTSSdb
#works, but unstable should db move:
sqlalchemy.url = postgres://ijfbcvuyifb.....
initialize_db script:
def main(argv=sys.argv):
if len(argv) < 2:
usage(argv)
config_uri = argv[1]
options = parse_vars(argv[2:])
setup_logging(config_uri)
settings = get_appsettings(config_uri, options=options)
engine = engine_from_config(settings)

How can I access a custom section in a Pyramid .ini file?

I'm currently writing a data collection service for multiple services. There are probably 5 different API Endpoints with differing hosts & port numbers. I wanted to create a settings file for this but thought that the .ini should be a better place, or so I thought...
My development.ini looks something like this:
[app:main]
use = egg:awesomeproject
auth.tkt = 'abc'
auth.secret = 'I love python'
mongodb.host = 'somehost'
mongodb.port= 6379
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543
[user:sg:qa]
host = 127.0.0.1
port = 1234
[customer:sg:qa]
host = 127.0.0.2
port = 4567
I tried to access the custom sections within a pyramid event like such:
def add_api_path(event):
request = event.request
settings = request.registry.settings
_type = 'customer:sg:qa'
base_config = settings[_type]
But that didn't work, because settings is actually a dict of the [app:main] attributes. Can someone teach me the way to access the sections the Pyramid way? I read about another way, using ConfigParser, but I wanted to ask if there's any other easier way in Pyramid first.

If you want to do that you'll have to parse the config file yourself. The section-isolation behavior you're seeing is intentional.
def main(global_conf, **settings):
parser = ConfigParser({'here': global_conf['__here__']})
parser.read(global_conf['__file__'])
for k, v in parser.items('user:sg:qa'):
settings['user:sg:qa:' + k] = v
config = Configurator(settings=settings)
Then later you can grab the settings:
request.registry.settings['user:sg:qa:host']
update
In Pyramid 1.9 the ini parsing was made pluggable and a new library was created to assist in loading arbitrary sections of the file in a standard way. Below is the updated example:
import plaster
def main(global_conf, **settings):
user_settings = plaster.get_settings(global_conf['__file__'], 'user:sg:qa')
for k, v in user_settings.items():
settings['user:sg:qa:' + k] = v
config = Configurator(settings=settings)

Implementing Sqlalchemy beaker caching in pyramid framework

As per the example provided by sqlalchemy documentation to cache a sqlalchemy query we are suppose to do this
from caching_query import FromCache
# load Person objects. cache the result under the namespace "all_people".
print "loading people...."
people = Session.query(Person).options(FromCache("default", "all_people")).all()
I have the following configuration for beaker in development.ini
cache.regions = day, hour, minute, second
cache.type = file
cache.data_dir = %(here)s/cache/sess/data
cache.lock_dir = %(here)s/cache/sess/lock
cache.second.expire = 1
cache.minute.expire = 60
cache.hour.expire = 3600
cache.day.expire = 86400
When i use the above example code in my application data is not cached in the cache folder, so i am assuming memory based caching is the default, Is it possible to switch sqlalchemy cache type to file based? or am i getting it wrong?

Your question is missing some details, but let me try:
the first parameter passed to FromCache() is a name of a Beaker cache region, it should match one of the configured regions, which is not the case here. Or perhaps you configure default region in the code (I'd expect BeakerException being thrown if region is unknown)?
you need pyramid_beaker module installed and included in Pyramid's project configuration. I suggest you follow pyramid_beaker manual's Setup section.
you need some extra code in __init__.py of your application in order to propagate .ini file settings to Beaker. This is described in Beaker cache region support section of the manual.
And here's a working sample from my current project, configuring both Beaker-based sessions and caching (all irrelevant parts removed):
from pyramid.config import Configurator
from pyramid_beaker import set_cache_regions_from_settings
from pyramid_beaker import session_factory_from_settings
def main(global_config, **settings):
# Configure Beaker caching/sessions
set_cache_regions_from_settings(settings)
session_factory = session_factory_from_settings(settings)
config = Configurator(settings=settings)
config.set_session_factory(session_factory)
config.include('pyramid_beaker')
config.add_static_view('static', 'static', cache_max_age=3600)
config.add_route('home', '/')
config.scan()
return config.make_wsgi_app()