I am working on a django project that I need to run it with Docker. In this project I have multiples .env files: .env.dev, .env.prod, .env.staging. Is there a right way to manage all this file with the package python-decouple? I've search for a workaround to deal with this challenge and do not find any kind of answer, not even on the official documentation.
Can I use something like:
# dont works that way, it's just a dummie example
python manage.py runserver --env-file=.env.prod
or maybe any way to setting or override the file I need to use?
Instead of importing decouple.config and doing the usual config('SOME_ENV_VAR'), create a new decouple.Config object using RepositoryEnv('/path/to/.env.prod').
from decouple import Config, RepositoryEnv
DOTENV_FILE = '/home/user/my-project/.env.prod'
env_config = Config(RepositoryEnv(DOTENV_FILE))
# use the Config().get() method as you normally would since
# decouple.config uses that internally.
# i.e. config('SECRET_KEY') = env_config.get('SECRET_KEY')
SECRET_KEY = env_config.get('SECRET_KEY')
Related
Is there a way to ignore all of the errors in certain packages within my project?
Some of the code in my project is compiled Protocol Buffers code which doesn't pass a MyPy check. It all lives within a directory /myproj/generated/proto.
Here's what I have in my mypy config file:
[mypy-myproject.generated]
ignore_missing_imports = True
ignore_errors = True
What can I add to this to make it ignore all error messages generated from an analysis of anything that's inside of myproject.generated?
This is a duplicate of a question on GitHub.
You can use a glob.
[mypy-myproject.generated.*]
ignore_errors = True
But you have to assure that you have __init__.py in /generated
You can also use ignore an entire folder or file via the exclude option.
Here is an example of an ini file:
[mypy]
exclude = generated
Well, of course it's a crutch. I do not pretend that this is the most correct way.
I realise this question as been asked before (What's the best practice using a settings file in Python?) but seeing as this was asked 7 years ago, I feel it is valid to discuss again seeing as how technologies have evolved.
I have a python project that requires different configurations to be used based on the value of an environment variable. Since making use of the environment variable to choose a config file is simple enough, my question is as follows:
What format is seen as the best practice in the software industry for setting up a configuration file in python, when multiple configurations are needed based on the environment?
I realise that python comes with a ConfigParser module but I was wondering if it might be better to use a format such as YAML or JSON because of there raise in popularity due to their ease of use across languages. Which format is seen as easier to maintain when you have multiple configurations?
If you really want to use an environment-based YAML configuration, you could do so like this:
config.py
import yaml
import os
config = None
filename = getenv('env', 'default').lower()
script_dir = os.path.dirname(__file__)
abs_file_path = os.path.join(script_dir, filename)
with open(abs_file_path, 'r') as stream:
try:
config = yaml.load(stream)
except yaml.YAMLError as exc:
print(exc)
I think looking at the standard configuration for a Python Django settings module is a good example of this, since the Python Django web framework is extremely popular for commercial projects and therefore is representative of the software industry.
It doesn't get too fancy with JSON or YAML config files - It simply uses a python module called settings.py that can be imported into any other module that needs to access the settings. Environment variable based settings are also defined there. Here is a link to an example settings.py file for Django on Github:
https://github.com/deis/example-python-django/blob/master/helloworld/settings.py
This is really late to the party, but this is what I use and I'm pleased with it (if you're open to a pure Python solution). I like it because my configurations can be set automatically based on where this is deployed using environment variables. I haven't been using this that long so if someone sees an issue, I'm all ears.
Structure:
|--settings
|--__init__.py
|--config.py
config.py
class Common(object):
XYZ_API_KEY = 'AJSKDF234328942FJKDJ32'
XYZ_API_SECRET = 'KDJFKJ234df234fFW3424##ewrFEWF'
class Local(Common):
DB_URI = 'local/db/uri'
DEBUG = True
class Production(Common):
DB_URI = 'remote/db/uri'
DEBUG = False
class Staging(Production):
DEBUG = True
__init__.py
from settings.config import Local, Production, Staging
import os
config_space = os.getenv('CONFIG_SPACE', None)
if config_space:
if config_space == 'LOCAL':
auto_config = Local
elif config_space == 'STAGING':
auto_config = Staging
elif config_space == 'PRODUCTION':
auto_config = Production
else:
auto_config = None
raise EnvironmentError(f'CONFIG_SPACE is unexpected value: {config_space}')
else:
raise EnvironmentError('CONFIG_SPACE environment variable is not set!')
If my environment variable is set in each place where my app exists, I can bring this into my modules as needed:
from settings import auto_config as cfg
That really depends on your requirements, rather than the format's popularity. For instance, if you just need simple key-value pairs, an INI file would be more than enough. As soon as you need complex structures (e.g., arrays or dictionaries), I'd go for JSON or YAML. JSON simply stores data (it's more intended for automated data flow between systems), while YAML is better for human-generated (or maintained, or read) files, as it has comments, you can reference values elsewhere in the file... And on top of that, if you want robustness, flexibility, and means to check the correct structure of the file (but don't care much about the manual edition of the data), I'd go for XML.
I recommend giving trapdoor a try for turn-key configuration (disclaimer: I'm the author of trapdoor).
I also like to take advantage of the fact that you do not have to compile Python source and use plain Python files for configuration. But in the real world you may have multiple environments, each requires a different configuration, and you may also want to read some (mostly sensitive) information from env vars or files that are not in source control (to prevent committing those by mistake).
That's why I wrote this library: https://github.com/davidohana/kofiko,
which let you use plain Python files for configuration, but is also able to override those config settings from .ini or env-vars, and also support customization for different environments.
Blog post about it: https://medium.com/swlh/code-first-configuration-approach-for-python-f975469433b9
I have a scrapy project that writes the data it scrapes to a database. It was based on this great tutorial: http://newcoder.io/scrape/part-3/
I have hit an issue now that I am trying to write some integration tests for the project. I am following the guidelines here: Scrapy Unit Testing
It's not clear to me how best to pass in the appropriate database settings. I'd like the tests to use their own database that I can ensure is in a known state before the tests start running.
So just import settings won't do the trick as, if the project is being run in test mode then it needs to use a different settings file.
I am familiar with Ruby on Rails projects where you specify a RAILS_ENV environment variable, and based on this environment variable, the framework will use settings from different files. Is there a similar concept that can apply when testing scrapy projects? Or is there a more pythonic alternative approach?
In the end I edited the settings.py file to support using an environment variable to determine which additional files to get the settings from, like this:
from importlib import import_module
import logging
import os
SCRAPY_ENV=os.environ.get('SCRAPY_ENV',None)
if SCRAPY_ENV == None:
raise ValueError("Must set SCRAPY_ENV environment var")
# Load if file exists; incorporate any names started with an
# uppercase letter into globals()
def load_extra_settings(fname):
if not os.path.isfile("config/%s.py" % fname):
logger = logging.getLogger(__name__)
logger.warning("Couldn't find %s, skipping" % fname)
return
mdl=import_module("config.%s" % fname)
names = [x for x in mdl.__dict__ if x[0].isupper()]
globals().update({k: getattr(mdl,k) for k in names})
load_extra_settings("secrets")
load_extra_settings("secrets_%s" % SCRAPY_ENV)
load_extra_settings("settings_%s" % SCRAPY_ENV)
I made an example github repo showing how this worked: https://github.com/alanbuxton/scrapy_local_settings
Keen to find out if there is a neater way of doing it.
I'd like to keep development.ini and production.ini under version control, but for security reason would not want the sqlalchemy.url connection string to be stored, as this would contain the username and password used for the database connection.
What's the canonical way, in Pyramid, of sourcing this setting from an additional external file?
Edit
In addition to solution using the environment variable, I came up with this solution after asking around on #pyramid:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
# Read db password from config file outside of version control
secret_cfg = ConfigParser()
secret_cfg.read(settings['secrets'])
dbpass = secret_cfg.get("secrets", "dbpass")
settings['sqlalchemy.url'] = settings['connstr'] % (dbpass,)
I looked into this a lot and played with a lot of different approaches. However, Pyramid is so flexible, and the .ini config parser is so minimal in what it does for you, that there doesn't seem to be a de facto answer.
In my scenario, I tried having a production.example.ini in version control at first that got copied on the production server with the details filled in, but this got hairy, as updates to the example didn't get translated to the copy, and so the copy had to be re-created any time a change was made. Also, I started using Heroku, so files not in version control never made it into the deployment.
Then, there's the encrypted config approach. Which, I don't like the paradigm. Imagine a sysadmin being responsible for maintaining the production environment, but he or she is unable to change the location of a database or environment-specific setting without running it back through version control. It's really nice to have the separation between environment and code as much as possible so those changes can be made on the fly without version control revisions.
My ultimate solution was to have some values that looked like this:
[app:main]
sqlalchemy.url = ${SQLALCHEMY_URL}
Then, on the production server, I would set the environment variable SQLALCHEMY_URL to point to the database. This even allowed me to use the same configuration file for staging and production, which is nice.
In my Pyramid init, I just expanded the environment variable value using os.path.expandvars:
sqlalchemy_url = os.path.expandvars(settings.get('sqlalchemy.url'))
engine = create_engine(sqlalchemy_url)
And, if you want to get fancy with it and automatically replace all the environment variables in your settings dictionary, I made this little helper method for my projects:
def expandvars_dict(settings):
"""Expands all environment variables in a settings dictionary."""
return dict((key, os.path.expandvars(value)) for
key, value in settings.iteritems())
Use it like this in your main app entry point:
settings = expandvars_dict(settings)
The whole point of the separate ini files in Pyramid is that you do not have to version control all of them and that they can contain different settings for different scenarios (development/production/testing). Your production.ini almost always should not be in the same VCS as your source code.
I found this way for loading secrets from a extra configuration and from the env.
from pyramid.config import Configurator
from paste.deploy import appconfig
from os import path
__all__ = [ "main" ]
def _load_secrets(global_config, settings):
""" Helper to load secrets from a secrets config and
from env (in that order).
"""
if "drawstack.secrets" in settings:
secrets_config = appconfig('config:' + settings["drawstack.secrets"],
relative_to=path.dirname(global_config['__file__']))
for k, v in secrets_config.items():
if k == "here" or k == "__file__":
continue
settings[k] = v
if "ENV_DB_URL" in global_config:
settings["sqlalchemy.url"] = global_config["ENV_DB_URL"]
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
_load_secrets(global_config, settings)
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
The code above, will load any variables from the value of the config key drawstack.secrets and after that it tries to load DB_URL from the enviornment.
drawstack.secrets can be relative to the original config file OR absolute.
I'm running my unit tests using nose.
I have .ini files such as production.ini, development.ini, local.ini. Finally, I have a test.ini file which looks like:
[app:main]
use = config:local.ini
# Add additional test specific configuration options as necessary.
sqlalchemy.url = sqlite:///%(here)s/tests.db
In my test class I want to setup the database as I would in my app server code. Something like:
engine = engine_from_config(settings)
initialize_sql(engine)
dbfixture = SQLAlchemyFixture(
env=model,
engine=engine,
style=NamedDataStyle()
)
How does nose pass 'settings' to my test code?
I've been reading the following link for some guidance, but I haven't been able to connect all the dots. http://farmdev.com/projects/fixture/using-fixture-with-pylons.html
Thanks much!
You will need to parse the settings from the INI file yourself. Pylons used to do this automatically for you by just hard-coding a load for "test.ini". The two options you have are 1) just load the INI settings via settings = paste.deploy.appconfig('test.ini') or 2) loading the actual WSGI app yourself, like if you wanted to use it via WebTest app = pyramid.paster.get_app('test.ini') which would parse the INI file and return an actual WSGI app. Unfortunately that route doesn't give you access to the INI file directly, it automatically just passes the settings to your app's startup function main(global_conf, **settings).
You may also find the Pyramid docs on functional tests useful.