Hi I am relatively new to programming and building my first flask project and I haven't been able to figure out if I should prefer accessing environment variables by using dotenv / load_dotenv or using them from a config.py file.
I understand the config route is more flexible but my question is specifically to do with environment variables.
Is there a best practice here? [I am building a simple app that will be hosted externally]
Best practices dictate that any value which is secret should not be hard-coded into any files which persist with the project or are checked into source control. Your config file is very likely to be saved in source control, so it should not store secrets, but instead load them from the environment variables set at execution time of the app. For example, let's say you are configuring an SMTP relay:
MAIL_PORT is a value that is not secret and not likely to change so it is a good candidate to be set in your config file.
MAIL_PASSWORD is a secret value that you do not want saved in your project's repository, so it should be loaded from the host's environment variables.
In this example, your config file might contain entries that look something like this:
MAIL_PORT = 465
MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')
Beyond evaluating whether or not a config value is a secret, also consider how often the value will change and how hard it would be to make that change. Anything hard-coded into your config file will require changing the file and adding a new commit to your source control, possibly even triggering a full CI/CD pipeline process. If the value were instead loaded from environment variables then this value could be changed by simply stopping the application, exporting the new value as an environment variable, and restarting the application.
Dotenv files are simply a convenience for grouping a number of variables together and auto-loading them to be read by your configuration. A .env file is not always used as these values can be manually exported when the app is invoked or handled by another system responsible for starting or scaling your application. Do not check .env or .flaskenv files into your source control.
Related
Say I have some API-key in my project that I don't wanna share into git repository, then I have to use environment variables. Now, why shouldn't I blatantly set the environmental variable on my local machine (like PATH) instead of making .env file and downloading python-dotenv library to my project for the sake of doing actually the same thing?
.env file is just manner to store this variables in file that can use packages for example dotenv to read as an os.environ varaible. so in short it is manner of storage of configuration.
many times your gitignore will have .env thus users can store the API key with .env file on local machine to help ease of use and ensuring dont accidently leave api keys within committed git files
if you just do a bear os.environ['API-KEY'] = 'stuff' within code it will store in commited git file (and env variables are there within a run of python process, not their permanently between sessions, thus why storing in file is preferred)
with this there are types of way to store configuration dynaconf is a great package and package shows many of the other types of configuration files
I'm using GAE to deploy my app and I have some environment variables that I would like to pass to my GAE instance. For example, every time I use the DB, the unix socket assignment is currently like this:
unix_socket='<my_path_to_my_sockets>/{instance}'
.format(instance=properties.get_property('data.instance'))
That is great but the problem is that it's a shared code and everytime someone makes a local test it changes the path and push the changes to the repository. When someone pull the new changes then it needs to change the in order to make the db requests because everyone have a different path to the sockets changes too. So I've created the following statement:
unix_socket= (os.environ['CLOUDSQL_INSTANCES_DIR']
if 'CLOUDSQL_INSTANCES_DIR' in os.environ
else '/home/cpinam/cloudsql/')
+ properties.get_property('cloud.instance')
So, if someone has an environment variable in its system, then it takes the variable and avoid the absolute path. The problem is that this environment variable is not referring to my computer but GAE instance.
The question is, how can I take my environment variable instead of any environment variable of server instance? Is it possible?
PS: I've know that I can pass environment variables through the app.yaml file but that implies to modify the file.
Thank you
App Engine does not support what you want in the way that you want it.
There are a few alternative approaches that you might want to consider. It sounds like your primary constraint is wanting to enable individual developers to store alternative configuration to what they might find in production. You might want to consider allowing developers to store said configuration in their local Datastore instances.
Your code would look something like:
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
unix_socket = os.environ['CLOUDSQL_INSTANCES_DIR']
...
else:
unix_socket = your_configuration_type.get("CLOUDSQL_INSTANCES_DIR")
...
Another alternative would be the approach outlined here where you'd store the relevant env vars in your own version of client_secrets.json and made sure that file was listed in .gitignore.
This can be currently done using the original App Engine SDK's appcfg.py command for deployment. This is not possible using the gcloud App Engine deployment as far as I know.
You can define a default environment variable in your app.yaml file:
env_variables:
HOST_NAME: ''
Pass your environment variable using -E option of the appcfg.py command -
-E NAME:VALUE. Ex : -E HOST_NAME:WOAH
-E description : Set an environment variable, potentially overriding an env_variable value from app.yaml file (flag may be repeated to set multiple variables).
I have a Flask application that I want to deploy to Amazon Elastic Beanstalk, but have the following problem:
Our configuration contains a lot of secret API keys, so we don't have it checked into Git.
Elastic Beanstalk only deploys the files that are committed to Git, so our config doesn't get uploaded as part of the deployment process.
Elastic Beanstalk offers a way to set and read environment variables - but Flask typically expects environment variables to be set in Python files that are imported at runtime.
Is there a way we can get our configuration file automatically uploaded to AWS EB alongside the source code when we deploy it? I considered having it in a file that gets created via configuration in .ebextensions/... but that would just need to be in Git anyway, defeating the object.
Or do we have to (a) convert all our code to read configuration from environment variables, and (b) create some sort of pre-startup script that injects the values in the current config scripts into the environment? It wouldn't be ideal to have to maintain 2 totally different ways of reading configuration data into our app depending on whether it's running on AWS or not.
I have seen this question/answer - How to specify sensitive environment variables at deploy time with Elastic Beanstalk - and I don't think that adequately addresses the issue because it is not very scalable to large numbers of config options nor deals with the fact that Flask typically expects its config in a different format.
If you are doing a 12 factor application, the simplest way to handle this kind of configuration is to lookup environment variables by default, and fall back on your hard-coded configurations:
from os import environ
from flask.config import Config
class EnvConfiguration(Config):
"""Configuration sub-class that checks its environment
If no environment variable is available,
it falls back to the hard-coded value provided"""
def __getitem__(self, key):
environ.get(key, super(EnvConfig, self).__getitem__(key))
You can then override either Flask.config_class (v1+) or Flask.make_config to ensure that the configuration used is the one you want:
class CustomApp(Flask):
# 1.0+ way
config_class = EnvConfiguration
# 0.8+ way (you only need one of these)
def make_config(self, instance_relative=False):
config = super(CustomApp, self).make_config(instance_relative=instance_relative)
config = EnvConfig(**config)
return config
I have sensitive data (database passwords) in settings.py and I was advised to upload my Django project to a github repository before pushing it to Heroku on their "Getting Started with Django on Heroku". If I put settings.py in .gitignore, then presumably it won't get deployed with my project. How can I prevent settings.py from being exposed but still get it deployed with my project ?
You can use environment variables (with heroku config:add SECRET=my-secret) to setup sensitive data and retrieve them in your settings with:
SECRET = os.environ.get('SECRET', 'my-default-secret-key')
If you don't want to be able to start your app without having set up some data, use this syntax instead:
SECRET = os.environ['SECRET']
It will raise an exception if you didn't set the SECRET environment variable.
You should use a tool designed for factoring out sensitive data. I use YamJam https://pypi.python.org/pypi/yamjam/ . It allows all the advantages of the os.environ method but is simpler -- you still have to set those environ variables, you'll need to put them in a script/ rc file somewhere. YamJam eliminates these questions and stores these config settings in a config store outside of the project. This allows you to have different settings for dev, staging and production.
from YamJam import yamjam
secret = yamjam()['myproject']['secret']
Is the basic usage. And like the os.environ method, it is not framework specific, you can use it with Django or any other app/framework. I've tried them all, multiple settings.py files, brittle logic of if/then and environment wrangling. In the end, I switched to yamjam and haven't regretted it.
My app uses Python with cherrypy to provide a web service. I want to set up a different config file, depending on if my application is started on my local machine or on my remote server.
Is there any easy way to check if my application is started from the server or on my local machine? If not, maybe I could pass some parameters, when running python myApp.py, which myApp.py will then be passed to myApp.py? Of course an automatic solution would be nicer.
Create a local configuration file and put a variable named environment inside it. Assign it dev for local env and production for production, and anything else you'd like. Just set it once, then reuse it everywhere -
from local_settings.py import environment
if environment == 'dev':
debug = True
# anything you'd like
If you're using any VCS like git, and using that to deploy, ignore the local_settings.py file. Local settings files are also handy for saving sensitive data that should not be public in any VCS repo, like API keys and all.