I'm developing a simple FastAPI app and I'm using Pydantic for storing app settings.
Some settings are populated from the environment variables set by Ansible deployment tools but some other settings are needed to be set explicitly from a separate env file.
So I have this in config.py
class Settings(BaseSettings):
# Project wide settings
PROJECT_MODE: str = getenv("PROJECT_MODE", "sandbox")
VERSION: str
class Config:
env_file = "config.txt"
And I have this config.txt
VERSION="0.0.1"
So project_mode env var is being set by deployment script and version is being set from the env file. The reason for that is that we'd like to keep deployment script similar across all projects, so any custom vars are populated from the project specific env files.
But the problem is that when I run the app, it fails with:
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
VERSION
field required (type=value_error.missing)
So how can I populate Pydantic settings model from the local ENV file?
If the environment file isn't being picked up, most of the time it's because it isn't placed in the current working directory. In your application in needs to be in the directory where the application is run from (or if the application manages the CWD itself, to where it expects to find it).
In particular when running tests this can be a bit confusing, and you might have to configure your IDE to run tests with the CWD set to the project root if you run the tests from your IDE.
The path of env_file is relative to the current working directory, which confused me as well. In order to always use a path relative to the config module I set it up like this:
env_file: f"{pathlib.Path(__file__).resolve().parent}/config.txt"
Related
I have a Django app running on Google AppEngine Standard environment. I've set up a cloud build trigger from my master branch in Github to run the following steps:
steps:
- name: 'python:3.7'
entrypoint: python3
args: ['-m', 'pip', 'install', '--target', '.', '--requirement', 'requirements.txt']
- name: 'python:3.7'
entrypoint: python3
args: ['./manage.py', 'collectstatic', '--noinput']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', 'app.yaml']
env:
- 'SHORT_SHA=$SHORT_SHA'
- 'TAG_NAME=$TAG_NAME'
I can see under the Execution Details tab on Cloud Build that the variables were actually set.
The problem is, SHORT_SHA and TAG_NAME aren't accessible from my Django app (followed instructions at https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values#using_user-defined_substitutions)! But if I set them in my app.yaml file with hardcoded values under env_variables, then my Django app can access those hardcoded values (and the values set in my build don't overwrite those hardcoded in app.yaml).
Why is this? Am I accessing them/setting them incorrectly? Should I be setting them in app.yaml somehow?
I even printed the whole os.environ dictionary in one of my views to see if they were just there with different names or something, but they're not present in there.
Not the cleanest solution, but I used this medium post as a guidance to my solution. I hypothesize that runserver command isn't being passed those env variables, and that those variables are only used for the app deploy command.
Write a Python script to dump the current environment variables in a .env file in project dir
In your settings file, read env variables from the .env file (I used django-environ library for this)
Add a step to cloud build file that runs your new Python script and pass env variables in that step (you're essentially dumping these variables into a .env file in this step)
- name: 'python:3.7'
entrypoint: python3
args: ['./create_env_file.py']
env:
- 'SHORT_SHA=$SHORT_SHA'
- 'TAG_NAME=$TAG_NAME'
Set the variables through Substitution Variables section in Edit Trigger page in Cloud Build
Now your application should have these env variables when app deploy happens
Suppose i my web application running with following settings
LOG_DIR = "/var/log/main"
This variable defines where the log should go.
Now then i run my tests and i have test_settings like
from settings import *
LOG_DIR = "/var/log/test"
Now i want to know that does that mean while my test script is running then my main application logs will also go to test folder because i have chnaged the global variable.
For my integration testiing i need to change variables if am afraid that if that will affect the main application or not .Like my application depends upon
Shell ENV variables . I wanted to chnage that for my tests. but i am afraid that if that will chnage the main running application.
This is not for PROD but for other testing applications environment
Running a django process will not affect the settings of another process. Just make sure to explicitly pass the --settings flag to your manage.py script when you run your tests/dev server/ etc..
e.g.
python manage.py test --settings project/settings/test.py
python manage.py runserver --settings project/settings
I prefer having a set of shell scripts that get sourced when I run my test environment (you can configure that with a test runner, see the docs).
I have an app where I want to run it from aws_wsgi.py instead of application.py, as there are several different entry points depending on where we are hosting it. For this reason, I would like to be able to change the WSGIPath variable to point to the correct location.
This, in an .ebextensions .config file, does not work:
option_settings:
- namespace: "aws:elasticbeanstalk:container:python"
option_name: WSGIPath
value: "/opt/python/current/app/aws_wsgi.py"
The environment attempts to use 'application.py' despite the above lines. No error appears to be emitted. Other parts of that same config file work perfectly, such as packages commands to get the system to install some yum packages. I can confirm the config files are getting uploaded in the logs:
INFO: Creating new application version using project code
WARNING: You have uncommitted changes.
INFO: Getting version label from git with git-describe
Creating application version archive "0_3_0-507-ga36f".
INFO: creating zip using git archive HEAD
INFO: git archive output: .ebextensions/
.ebextensions/01-weave_server_eb.config
.ebextensions/02-weave_server_eb_lxml_dependencies.config
.ebextensions/03-weave_server_eb_nltk_data.config
.ebextensions/04-weave_server_eb_entity_data.config
.ebextensions/05-weave_server_eb_geography_data.config
.gitattributes
.gitignore
...etc.
We run with a saved configuration, i.e. via eb create --cfg Live, and in the dashboard, that configuration shows that WSGIPath is "application.py". But there is nowhere to change that value in the dashboard. It seems like it is a built-in value that overrides the data we send with the above config file.
I tried adding it as an environment variable via the dashboard, but that goes in the aws:elasticbeanstalk:application:environment namespace and does not affect how the application is started up in the first place. (I checked this by using eb config to download the configuration.)
Maybe I could add a section in the file retrieved by eb config, but I heard that doing that will start to override .ebextensions files, and I have several important commands in my .ebextensions files that I need to continue using. (But see the comment below.) It's not clear from the documentation how .ebextensions data translates to and compares with config files used by eb config but the .ebextensions files are well-documented and reasonably convenient so I'd rather not break those if possible!
If I retrieve the configuration on the server via eb config get Live, it contains the following (numerous API keys removed):
EnvironmentConfigurationMetadata:
Description: Includes API keys for live operation
DateModified: '1437734273000'
DateCreated: '1437734273000'
AWSConfigurationTemplateVersion: 1.1.0.0
EnvironmentTier:
Name: WebServer
Type: Standard
SolutionStack: 64bit Amazon Linux 2015.03 v1.4.3 running Python 2.7
OptionSettings:
aws:elb:loadbalancer:
CrossZone: true
aws:elasticbeanstalk:command:
BatchSize: '30'
BatchSizeType: Percentage
aws:autoscaling:launchconfiguration:
IamInstanceProfile: aws-elasticbeanstalk-ec2-role
EC2KeyName: aws-eb
InstanceType: t2.micro
aws:elb:policies:
ConnectionDrainingEnabled: true
aws:autoscaling:updatepolicy:rollingupdate:
RollingUpdateType: Health
RollingUpdateEnabled: true
aws:elasticbeanstalk:application:environment:
DATA_DIR: /opt/python/current/app/data
WSGIPath: /opt/python/current/aws_wsgi.py
aws:elb:healthcheck:
Interval: '30'
(NB. The WSGIPath environment variable there is invalid - but I am unable to remove it from the configuration due to bugs in the AWS Dashboard. It appears to have no effect anyway.)
How do I get AWS to respect my chosen WSGIPath?
I'm following the approach in Two Scoops of Django: Best Practices for Django 1.6 regarding multiple settings files. I'm using Django 1.7 and virtualenvwrapper.
My setup is as follows:
project/
app1/
app2/
project/
__init__.py
settings/
__init__.py
base.py
local.py
production.py
manage.py
I'm a bit confused as to how Django knows which settings file to use. I do not want to specify the settings file every time I run manage.py. I would rather like to set the DJANG_SETTINGS_MODULE environmental variable as explained in omouse anser here:
What confuses me is in the wsgi.py file there is a line:
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "{{ project_name }}.settings.production")
Is this file only used in the production server? What happens if I already have a DJANGO_SETTINGS_MODULE environmental variable defined on the server?
When running it locally, I understand I need to set the DJANGO_SETTINGS_MODULE env variable every time I open the console. I've read here that I can define a postactivate hook in virtualenvwrapper. This hook will then create the environmental variables that I require everytime I activate the environment.
Is this the recommended way of ensuring the correct DJANGO_SETTINGS_MODULE env variable is loaded on my local machine? Would I also need to setup a similar file on my hosting server? I'm planning on using PythonAnywhere for hosting.
Lastly, if I run a staging server, how would I tell Django to load the staging settings file? The staging server is the practically the same as the production server, so I guess need a different wsgi.py file for the staging server, but that seems like a anti-pattern.
os.environ.setdefault only sets the value if it is not set. When you run in production, export the environment variable DJANGO_SETTINGS_MODULE and set it to your production/staging settings file, and you don't have to set anything when running in development (if you set it by default to your development settings). This is the DRY-est method.
The method with a local_settings.py (which is most of the times kept out of the repo!) is not best practice and should be avoided.
I have a Python application that is set up using the new New Relic configuration variables in the dotcloud.yml file, which works fine.
However I want to run a sandbox instance as a test/staging environment, so I want to be able to set the environment of the newrelic agent so that it uses the different configuration sections of the ini configuration. My dotcloud.yml is set up as follows:
www:
type: python
config:
python_version: 'v2.7'
enable_newrelic: True
environment:
NEW_RELIC_LICENSE_KEY: *****************************************
NEW_RELIC_APP_NAME: Application Name
NEW_RELIC_LOG: /var/log/supervisor/newrelic.log
NEW_RELIC_LOG_LEVEL: info
NEW_RELIC_CONFIG_FILE: /home/dotcloud/current/newrelic.ini
I have custom environment variables so that the sanbox is set as "test" and the live application is set to "production"
I am then calling the following in my uswsgi.py
NEWRELIC_CONFIG = os.environ.get('NEW_RELIC_CONFIG_FILE')
ENVIRONMENT = os.environ.get('MY_ENVIRONMENT', 'test')
newrelic.agent.initialize(NEWRELIC_CONFIG, ENVIRONMENT)
However the dotcloud instance is already enabling newrelic because I get this in the uwsgi.log file:
Sun Nov 18 18:50:12 2012 - unable to load app 0 (mountpoint='') (callable not found or import error)
Traceback (most recent call last):
File "/home/dotcloud/current/wsgi.py", line 15, in <module>
newrelic.agent.initialize(NEWRELIC_CONFIG, ENVIRONMENT)
File "/opt/ve/2.7/local/lib/python2.7/site-packages/newrelic-1.8.0.13/newrelic/config.py", line 1414, in initialize
log_file, log_level)
File "/opt/ve/2.7/local/lib/python2.7/site-packages/newrelic-1.8.0.13/newrelic/config.py", line 340, in _load_configuration
'environment "%s".' % (_config_file, _environment))
newrelic.api.exceptions.ConfigurationError: Configuration has already been done against differing configuration file or environment. Prior configuration file used was "/home/dotcloud/current/newrelic.ini" and environment "None".
So it would seem that the newrelic agent is being initialised before uwsgi.py is called.
So my question is:
Is there a way to initialise the newrelic environment?
The easiest way to do this, without changing any code would be to do the following.
Create a new sandbox app on dotCloud (see http://docs.dotcloud.com/0.9/guides/flavors/ for more information about creating apps in sandbox mode)
$ dotcloud create -f sandbox <app_name>
Deploy your code to the new sandbox app.
$ dotcloud push
Now you should have the same code running in both your live and sandbox apps. But because you want to change some of the ENV variables for the sandbox app, you need to do one more step.
According to this page http://docs.dotcloud.com/0.9/guides/environment/#adding-environment-variables there are 2 different ways of adding ENV variables.
Using the dotcloud.yml's environment section.
Using the dotcloud env cli command
Whereas dotcloud.yml allows you to define different environment variables for each service, dotcloud env set environment variables for the whole application. Moreover, environment variables set with dotcloud env supersede environment variables defined in dotcloud.yml.
That means that if we want to have different values for our sandbox app, we just need to run a dotcloud env command to set those variables on the sandbox app, which will override the ones in your dotcloud.yml
If we just want to change on variable we would run this command.
$ dotcloud env set NEW_RELIC_APP_NAME='Test Application Name'
If we want to update more then one at a time we would do the following.
$ dotcloud env set \
'NEW_RELIC_APP_NAME="Test Application Name"' \
'NEW_RELIC_LOG_LEVEL=debug'
To make sure that you have your env varibles set correctly you can run the following command.
$ dotcloud env list
Notes
The commands above, are using the new dotCloud 0.9.x CLI, if you are using the older one, you will need to either upgrade to the new one, or refer to the documentation for the old CLI http://docs.dotcloud.com/0.4/guides/environment/
When you set your environment variables it will restart your application so that it can install the variables, so to limit your downtime, set all of them in one command.
Unless they are doing something odd, you should be able to override the app_name supplied by the agent configuration file by doing:
import newrelic.agent
newrelic.agent.global_settings().app_name = 'Test Application Name'
Don't call newrelic.agent.initialize() a second time.
This will only work if app_name is listing a single application to report data to.