I'm using GAE to deploy my app and I have some environment variables that I would like to pass to my GAE instance. For example, every time I use the DB, the unix socket assignment is currently like this:
unix_socket='<my_path_to_my_sockets>/{instance}'
.format(instance=properties.get_property('data.instance'))
That is great but the problem is that it's a shared code and everytime someone makes a local test it changes the path and push the changes to the repository. When someone pull the new changes then it needs to change the in order to make the db requests because everyone have a different path to the sockets changes too. So I've created the following statement:
unix_socket= (os.environ['CLOUDSQL_INSTANCES_DIR']
if 'CLOUDSQL_INSTANCES_DIR' in os.environ
else '/home/cpinam/cloudsql/')
+ properties.get_property('cloud.instance')
So, if someone has an environment variable in its system, then it takes the variable and avoid the absolute path. The problem is that this environment variable is not referring to my computer but GAE instance.
The question is, how can I take my environment variable instead of any environment variable of server instance? Is it possible?
PS: I've know that I can pass environment variables through the app.yaml file but that implies to modify the file.
Thank you
App Engine does not support what you want in the way that you want it.
There are a few alternative approaches that you might want to consider. It sounds like your primary constraint is wanting to enable individual developers to store alternative configuration to what they might find in production. You might want to consider allowing developers to store said configuration in their local Datastore instances.
Your code would look something like:
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine/'):
unix_socket = os.environ['CLOUDSQL_INSTANCES_DIR']
...
else:
unix_socket = your_configuration_type.get("CLOUDSQL_INSTANCES_DIR")
...
Another alternative would be the approach outlined here where you'd store the relevant env vars in your own version of client_secrets.json and made sure that file was listed in .gitignore.
This can be currently done using the original App Engine SDK's appcfg.py command for deployment. This is not possible using the gcloud App Engine deployment as far as I know.
You can define a default environment variable in your app.yaml file:
env_variables:
HOST_NAME: ''
Pass your environment variable using -E option of the appcfg.py command -
-E NAME:VALUE. Ex : -E HOST_NAME:WOAH
-E description : Set an environment variable, potentially overriding an env_variable value from app.yaml file (flag may be repeated to set multiple variables).
Related
I'm trying to upload files to an S3 bucket and I'm using this tutorial. I'm setting config variables in the terminal using heroku config:set and when I enter heroku config in the terminal, my variables appear.
However, if I access S3_BUCKET in code when running locally, the value returned is None:
S3_BUCKET = os.environ.get('S3_BUCKET')
print("value of S3_BUCKET: " + S3_BUCKET)
This prints None.
Does anyone know why this doesn't work? The frontend of my application is in React, if that matters, but all of the bucket upload code is done in python.
heroku config:set sets variables on Heroku. Like their name suggests, environment variables are specific to a particular environment. Settings set this way have no impact on your local machine.
There are several ways to set environment variables locally. One common method involves putting them into an untracked .env file. Many tools, including heroku local, read that file and add variables found there to the environment when running locally.
Heroku recommends this, though you will have to make sure whatever tooling you use picks up those variables. Aside from heroku local, Visual Studio Code understands .env files for Python projects. So does Pipenv. There are also standalone tools for this like direnv.
Hi I am relatively new to programming and building my first flask project and I haven't been able to figure out if I should prefer accessing environment variables by using dotenv / load_dotenv or using them from a config.py file.
I understand the config route is more flexible but my question is specifically to do with environment variables.
Is there a best practice here? [I am building a simple app that will be hosted externally]
Best practices dictate that any value which is secret should not be hard-coded into any files which persist with the project or are checked into source control. Your config file is very likely to be saved in source control, so it should not store secrets, but instead load them from the environment variables set at execution time of the app. For example, let's say you are configuring an SMTP relay:
MAIL_PORT is a value that is not secret and not likely to change so it is a good candidate to be set in your config file.
MAIL_PASSWORD is a secret value that you do not want saved in your project's repository, so it should be loaded from the host's environment variables.
In this example, your config file might contain entries that look something like this:
MAIL_PORT = 465
MAIL_PASSWORD = os.environ.get('MAIL_PASSWORD')
Beyond evaluating whether or not a config value is a secret, also consider how often the value will change and how hard it would be to make that change. Anything hard-coded into your config file will require changing the file and adding a new commit to your source control, possibly even triggering a full CI/CD pipeline process. If the value were instead loaded from environment variables then this value could be changed by simply stopping the application, exporting the new value as an environment variable, and restarting the application.
Dotenv files are simply a convenience for grouping a number of variables together and auto-loading them to be read by your configuration. A .env file is not always used as these values can be manually exported when the app is invoked or handled by another system responsible for starting or scaling your application. Do not check .env or .flaskenv files into your source control.
Is DEBUG == False supposed to mean that the app is running in production environment?
At least, that's what I see occasionally on the internet. But what do I put in settings.py then? Okay, I can put local settings to, say, settings_local.py and import it from settings.py. But if some settings depend on environment, than I've got to put them after import statement. There more I think about it, the more I don't like it. And you?
As an answer to the question:
Is DEBUG == False supposed to mean that the app is running in production environment?
DEBUG is a configuration that you define in your setting.py file.
If set to True, in case of un-handled exception it displays the complete stack-trace along with the values of all the declared variables.
If set to False, your server just returns the 500 status code without any stack-trace.
In production, you must have DEBUG set to False in order to prevent potential risk of security breach, and other information which you wouldn't want your user to know.
In order to use different settings configuration on different environment, create different settings file. And in your deployment script, start the server using --settings=<my-settings.py> parameter, via which you can use different settings on different environment.
Benefits of using this approach:
Your settings will be modular based on each environment
You may import the master_settings.py containing the base configuration in the environmnet_configuration.py and override the values that you want to change in that environment.
If you have huge team, each developer may have their own local_settings.py which they can add to the code repository without any risk of modifying the server configuration. You can add these local settings to .gitnore if you use git or .hginore if you Mercurial for Code Version Control. That way local settings won't even be the part of actual code base keeping it clean.
Use django-configurations to define a some alternative configurations and set the environment variable DJANGO_CONFIGURATION on each machine running the code to choose one.
You're free to define the classes however you'd like, but I'd recommend defining a Common class, which everything inherits from, and then Dev, Test, and Prod classes.
Anything involving system configuration should be pulled from environment variables (eg database connection, cache connection etc).
Have fun!
To add to #anonymous answer, here's the script (manage.sh) I came up with:
#!/usr/bin/env bash
DIR=$(dirname -- "$(readlink -f -- "$0")")
export PYTHONPATH="$DIR${PYTHONPATH:+:$PYTHONPATH}"
if [ -e <app>/local_settings.py ]; then
export DJANGO_SETTINGS_MODULE=<app>.local_settings
else
export DJANGO_SETTINGS_MODULE=<app>.settings
fi
django-admin "$#"
It uses <app>.local_settings if exists. Otherwise it falls back to <app>.settings.
Alternatively, you can just edit your manage.py.
I deploy my project to GAE over Github. There is some foreign API-key which I don't want to save in repository and make them public. Is it possible to set an environment variable for a project in GAE control panel so I can catch it in my application?
You can store your keys in datastore. Later when you need them in the code, you can fetch them from datastore and cache them by memcache.
I prefer using the Datastore for keys like this. See the code in my answer at Securely storing environment variables in GAE with app.yaml
That code auto-generates placeholder values that you can then update from the developer console. Also, it uses the ndb library, so reading the keys is fast.
You can define environment variables in configuration file for App Engine application. In case of Python, it is app.yaml
env_variables:
MY_ENV_VAR: 'some_value'
You can find more details here.
There is no such a thing like project parameters that can be defined in Developers Console at the moment.
AWS does not properly explain how to manage different deployment environments on the beanstalk with relation to different environments and how to save those settings in your source control repo.
They clearly explain how to setup your python.config in .ebextensions like so:
"aws:elasticbeanstalk:container:python:environment":
DJANGO_SETTINGS_MODULE: "settings"
SERVER_ROOT: "/opt/python/current/app/"
However, if you want to have multiple environments like staging and prod you currently have to swap out your configuration files. Whats worse, how do you plan to retain this in your source control tree for shared environments like staging? It appears that every time you push you will need these configuration environment settings.
I've also found that AWS doesnt let me deploy unstaged changes which means writing a script to handle my deployments isnt an option either. What am I missing here?
Haven't tried it, but it appears that you can pass DJANGO_SETTINGS_MODULE not through configuration file, but through container's own parameters. You can update it through Environment Details -> Edit Configuration -> Container section of Beanstalk console.
Just as idea:
Create multiple environments "production", "staging", etc
Configure each with relevant DJANGO_SETTINGS_MODULE value
Remove DJANGO_SETTINGS_MODULE value from .ebextensions
Deploy application to pre-created environment
I did some digging on this in the past and it seems that they like you to use eb branch to configure the different environments and then configure the option differently within the optionsettings locally on the eb client level (when you init the branch and stuff).
When you think about it, the environment configuration (i.e. DJANGO_SETTINGS_MODULE) should be managed separately from the application code, so I just assume keep it out of the ebextensions and set it up when I navigate to a new environment. If I switch to an existing one need to make sure that value is set properly for the env I want to play in.