We Keep Coding

What is the best method for setting up a config file in Python

I realise this question as been asked before (What's the best practice using a settings file in Python?) but seeing as this was asked 7 years ago, I feel it is valid to discuss again seeing as how technologies have evolved.
I have a python project that requires different configurations to be used based on the value of an environment variable. Since making use of the environment variable to choose a config file is simple enough, my question is as follows:
What format is seen as the best practice in the software industry for setting up a configuration file in python, when multiple configurations are needed based on the environment?
I realise that python comes with a ConfigParser module but I was wondering if it might be better to use a format such as YAML or JSON because of there raise in popularity due to their ease of use across languages. Which format is seen as easier to maintain when you have multiple configurations?

If you really want to use an environment-based YAML configuration, you could do so like this:
config.py
import yaml
import os
config = None
filename = getenv('env', 'default').lower()
script_dir = os.path.dirname(__file__)
abs_file_path = os.path.join(script_dir, filename)
with open(abs_file_path, 'r') as stream:
try:
config = yaml.load(stream)
except yaml.YAMLError as exc:
print(exc)

I think looking at the standard configuration for a Python Django settings module is a good example of this, since the Python Django web framework is extremely popular for commercial projects and therefore is representative of the software industry.
It doesn't get too fancy with JSON or YAML config files - It simply uses a python module called settings.py that can be imported into any other module that needs to access the settings. Environment variable based settings are also defined there. Here is a link to an example settings.py file for Django on Github:
https://github.com/deis/example-python-django/blob/master/helloworld/settings.py

This is really late to the party, but this is what I use and I'm pleased with it (if you're open to a pure Python solution). I like it because my configurations can be set automatically based on where this is deployed using environment variables. I haven't been using this that long so if someone sees an issue, I'm all ears.
Structure:
|--settings
|--__init__.py
|--config.py
config.py
class Common(object):
XYZ_API_KEY = 'AJSKDF234328942FJKDJ32'
XYZ_API_SECRET = 'KDJFKJ234df234fFW3424##ewrFEWF'
class Local(Common):
DB_URI = 'local/db/uri'
DEBUG = True
class Production(Common):
DB_URI = 'remote/db/uri'
DEBUG = False
class Staging(Production):
DEBUG = True
__init__.py
from settings.config import Local, Production, Staging
import os
config_space = os.getenv('CONFIG_SPACE', None)
if config_space:
if config_space == 'LOCAL':
auto_config = Local
elif config_space == 'STAGING':
auto_config = Staging
elif config_space == 'PRODUCTION':
auto_config = Production
else:
auto_config = None
raise EnvironmentError(f'CONFIG_SPACE is unexpected value: {config_space}')
else:
raise EnvironmentError('CONFIG_SPACE environment variable is not set!')
If my environment variable is set in each place where my app exists, I can bring this into my modules as needed:
from settings import auto_config as cfg

That really depends on your requirements, rather than the format's popularity. For instance, if you just need simple key-value pairs, an INI file would be more than enough. As soon as you need complex structures (e.g., arrays or dictionaries), I'd go for JSON or YAML. JSON simply stores data (it's more intended for automated data flow between systems), while YAML is better for human-generated (or maintained, or read) files, as it has comments, you can reference values elsewhere in the file... And on top of that, if you want robustness, flexibility, and means to check the correct structure of the file (but don't care much about the manual edition of the data), I'd go for XML.

I recommend giving trapdoor a try for turn-key configuration (disclaimer: I'm the author of trapdoor).

I also like to take advantage of the fact that you do not have to compile Python source and use plain Python files for configuration. But in the real world you may have multiple environments, each requires a different configuration, and you may also want to read some (mostly sensitive) information from env vars or files that are not in source control (to prevent committing those by mistake).
That's why I wrote this library: https://github.com/davidohana/kofiko,
which let you use plain Python files for configuration, but is also able to override those config settings from .ini or env-vars, and also support customization for different environments.
Blog post about it: https://medium.com/swlh/code-first-configuration-approach-for-python-f975469433b9

In the Pyramid web framework, how do I source sensitive settings into development.ini / production.ini from an external file?

I'd like to keep development.ini and production.ini under version control, but for security reason would not want the sqlalchemy.url connection string to be stored, as this would contain the username and password used for the database connection.
What's the canonical way, in Pyramid, of sourcing this setting from an additional external file?
Edit
In addition to solution using the environment variable, I came up with this solution after asking around on #pyramid:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
# Read db password from config file outside of version control
secret_cfg = ConfigParser()
secret_cfg.read(settings['secrets'])
dbpass = secret_cfg.get("secrets", "dbpass")
settings['sqlalchemy.url'] = settings['connstr'] % (dbpass,)

I looked into this a lot and played with a lot of different approaches. However, Pyramid is so flexible, and the .ini config parser is so minimal in what it does for you, that there doesn't seem to be a de facto answer.
In my scenario, I tried having a production.example.ini in version control at first that got copied on the production server with the details filled in, but this got hairy, as updates to the example didn't get translated to the copy, and so the copy had to be re-created any time a change was made. Also, I started using Heroku, so files not in version control never made it into the deployment.
Then, there's the encrypted config approach. Which, I don't like the paradigm. Imagine a sysadmin being responsible for maintaining the production environment, but he or she is unable to change the location of a database or environment-specific setting without running it back through version control. It's really nice to have the separation between environment and code as much as possible so those changes can be made on the fly without version control revisions.
My ultimate solution was to have some values that looked like this:
[app:main]
sqlalchemy.url = ${SQLALCHEMY_URL}
Then, on the production server, I would set the environment variable SQLALCHEMY_URL to point to the database. This even allowed me to use the same configuration file for staging and production, which is nice.
In my Pyramid init, I just expanded the environment variable value using os.path.expandvars:
sqlalchemy_url = os.path.expandvars(settings.get('sqlalchemy.url'))
engine = create_engine(sqlalchemy_url)
And, if you want to get fancy with it and automatically replace all the environment variables in your settings dictionary, I made this little helper method for my projects:
def expandvars_dict(settings):
"""Expands all environment variables in a settings dictionary."""
return dict((key, os.path.expandvars(value)) for
key, value in settings.iteritems())
Use it like this in your main app entry point:
settings = expandvars_dict(settings)

The whole point of the separate ini files in Pyramid is that you do not have to version control all of them and that they can contain different settings for different scenarios (development/production/testing). Your production.ini almost always should not be in the same VCS as your source code.

I found this way for loading secrets from a extra configuration and from the env.
from pyramid.config import Configurator
from paste.deploy import appconfig
from os import path
__all__ = [ "main" ]
def _load_secrets(global_config, settings):
""" Helper to load secrets from a secrets config and
from env (in that order).
"""
if "drawstack.secrets" in settings:
secrets_config = appconfig('config:' + settings["drawstack.secrets"],
relative_to=path.dirname(global_config['__file__']))
for k, v in secrets_config.items():
if k == "here" or k == "__file__":
continue
settings[k] = v
if "ENV_DB_URL" in global_config:
settings["sqlalchemy.url"] = global_config["ENV_DB_URL"]
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
_load_secrets(global_config, settings)
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
The code above, will load any variables from the value of the config key drawstack.secrets and after that it tries to load DB_URL from the enviornment.
drawstack.secrets can be relative to the original config file OR absolute.

passing **settings info to unittest from nose

I'm running my unit tests using nose.
I have .ini files such as production.ini, development.ini, local.ini. Finally, I have a test.ini file which looks like:
[app:main]
use = config:local.ini
# Add additional test specific configuration options as necessary.
sqlalchemy.url = sqlite:///%(here)s/tests.db
In my test class I want to setup the database as I would in my app server code. Something like:
engine = engine_from_config(settings)
initialize_sql(engine)
dbfixture = SQLAlchemyFixture(
env=model,
engine=engine,
style=NamedDataStyle()
)
How does nose pass 'settings' to my test code?
I've been reading the following link for some guidance, but I haven't been able to connect all the dots. http://farmdev.com/projects/fixture/using-fixture-with-pylons.html
Thanks much!

You will need to parse the settings from the INI file yourself. Pylons used to do this automatically for you by just hard-coding a load for "test.ini". The two options you have are 1) just load the INI settings via settings = paste.deploy.appconfig('test.ini') or 2) loading the actual WSGI app yourself, like if you wanted to use it via WebTest app = pyramid.paster.get_app('test.ini') which would parse the INI file and return an actual WSGI app. Unfortunately that route doesn't give you access to the INI file directly, it automatically just passes the settings to your app's startup function main(global_conf, **settings).
You may also find the Pyramid docs on functional tests useful.

Running scripts within Pyramid framework (ie without a server)

I have a fair bit of experience with PHP frameworks and Python for scripting so am now taking the step to Pyramid.
I'd like to know what is the 'correct' way to run a script in Pyramid. That is, how should I set it up so that it is part of the application and has access to config and thus database but does not run through paster (or whatever WSGI).
As an example, say I have a web application which while a user is offline grabs Facebook updates through a web service. I want to write a script to poll that service and store in the database ready for next login.
How should I do this in terms of:
Adding variables in the ini file
Starting the script correctly
I understand the basics of Python modules and packages; however I don't fully understand Configurator/Paster/package setup, wherein I suspect the answer lies.
Thanks
Update:
Thanks, this seems along the lines of what I am looking for. I note that you have to follow a certain structure (eg have summary and parser attributes set) and that the function called command() will always be run. My test code now looks something like this:
class AwesomeCommand(Command):
max_args = 2
min_args = 2
usage = "NAME"
# These are required
summary = "Say hello!"
group_name = "My Package Name"
# Required:
parser = Command.standard_parser(verbose=True)
def command(self):
# Load the config file/section
config_file, section_name = self.args
# What next?
I'm now stuck as to how to get the settings themselves. For example, in init.py you can do this:
engine = engine_from_config(settings, 'sqlalchemy.')
What do I need to do to transform the config file into the settings?
EDIT: The (simpler) way to do this in Pylons is here:
Run Pylons controller as separate app?

As of Pyramid 1.1, this is handled by the framework:
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/commandline.html#writing-a-script

paster starts an application given an ini file that describes that application. the "serve" command is a built in command for starting a wsgi application and serving it. BUT, you can write other commands.
from paste.script.command import Command
class AwesomeCommand(Command):
def command(self):
print "the awesome thing it does"
and then register them as entry points in your setup.py.
setup(...
entry_points="""
[paste.app_factory]
.....
[paste.global_paster_command]
myawesome-command = mypackage.path.to.command:AwesomeCommand """)
pyramid adds it's own commands this way like the pshell command.

After going to the pylons discuss list, I came up with an answer. Hope this helps somebody:
#Bring in pyramid application--------------------------------------
import pyramid
from paste.deploy import appconfig
config_file = '/path_to_config_file/configname.ini'
name = 'app_name'
config_name = 'config:%s' % config_file
here_dir = os.getcwd()
conf = appconfig(config_name, name, relative_to=here_dir)
from main_package import main
app = main(conf.global_conf, **conf.local_conf)
#--------------------------------------------------------------------------

you need to make view for that action and then run it using:
paster request development.ini /url_to_your_view

CherryPy combine file and dictionary based configuration

I'm setting up a CherryPy application and would like to have the majority of my configuration settings in a .conf file like this:
[global]
server.socketPort = 8080
server.threadPool = 10
server.environment = "production"
However I would also like to setup a few with a dictionary in code like this:
conf = {'/': {'tools.staticdir.on': True,
'tools.staticdir.dir': os.path.join(current_dir, 'templates')}}
cherrypy.quickstart(HelloWorld(), config=conf)
Is it possible to combine both configs into one and then pass it into the config quickstart option?

quickstart is for quick sites. If you're doing anything as complex as having multiple configs, it's time to graduate. Look at the source code for the quickstart function (it's not scary!): you're going to unpack that into your startup script. So instead of quickstart, write this:
cherrypy.config.update(conffile)
cherrypy.config.update(confdict)
app = cherrypy.tree.mount(HelloWorld(), '/', conffile)
app.merge(confdict)
if hasattr(cherrypy.engine, "signal_handler"):
cherrypy.engine.signal_handler.subscribe()
if hasattr(cherrypy.engine, "console_control_handler"):
cherrypy.engine.console_control_handler.subscribe()
cherrypy.engine.start()
cherrypy.engine.block()
We've essentially added two lines to the quickstart code. First, we have an extra call to config.update; that merges the config dict into the global config. Second, app.merge(confdict); that's for merging multiple configs into each app.
It's perfectly OK to do these in the opposite order if you want the file config to override the dict. It's also OK to stick the dict-based config in HelloWorld._cp_config as described in the docs.

Those are two different configurations. Cherrypy has two configurations: One is the global config and the other is application config. You can use both normally:
cherrypy.config.update('my_file.ini')
cherrypy.quickstart(HelloWorld(), config=conf)
Please note that your example config file is wrong -- instead of server.socketPort it should be server.socket_port and instead of server.threadPool it should be server.threadpool. Check config docs for more information.