We Keep Coding

Pydantic validation error for BaseSettings model with local ENV file

I'm developing a simple FastAPI app and I'm using Pydantic for storing app settings.
Some settings are populated from the environment variables set by Ansible deployment tools but some other settings are needed to be set explicitly from a separate env file.
So I have this in config.py
class Settings(BaseSettings):
# Project wide settings
PROJECT_MODE: str = getenv("PROJECT_MODE", "sandbox")
VERSION: str
class Config:
env_file = "config.txt"
And I have this config.txt
VERSION="0.0.1"
So project_mode env var is being set by deployment script and version is being set from the env file. The reason for that is that we'd like to keep deployment script similar across all projects, so any custom vars are populated from the project specific env files.
But the problem is that when I run the app, it fails with:
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
VERSION
field required (type=value_error.missing)
So how can I populate Pydantic settings model from the local ENV file?

If the environment file isn't being picked up, most of the time it's because it isn't placed in the current working directory. In your application in needs to be in the directory where the application is run from (or if the application manages the CWD itself, to where it expects to find it).
In particular when running tests this can be a bit confusing, and you might have to configure your IDE to run tests with the CWD set to the project root if you run the tests from your IDE.

The path of env_file is relative to the current working directory, which confused me as well. In order to always use a path relative to the config module I set it up like this:
env_file: f"{pathlib.Path(__file__).resolve().parent}/config.txt"

Heroku Python Local Environment Variables

Working on a heroku django project.
Goal: I want to run the server locally with same code on cloud too (makes sense).
Problem:
Environments differ from a linux server (heroku) to a local PC windows system.
Local environment variables differ from cloud Heroku config vars.
Heroku config vars can be setup easily using the CLI heroku config:set TIMES=2.
While setting up local env vars is a total mess.
I tried the following in cmd:
py -c "import os;os.environ['Times']=2" # To set an env var
Then ran py -c "import os;os.environ.get('Times','Not Found')" stdout: "Not Found".
After a bit of research it appeared to be that such env vars are stored temporarily per process/session usage.
Solution theory: Redirect os.environ to .env file of the root heroku project instead of the PC env vars. So I found this tool direnv perfect for Unix-like OSs but not available for Windows.
views.py code (runs perfect on cloud, sick on the local machine):
import os
import requests
from django.shortcuts import render
from django.http import HttpResponse
from .models import Greeting
def index(request):
# get method takes 2 parameters (env_var_string,return value if var is not found)
times = int(os.environ.get('TIMES',3))
return HttpResponse('<p>'+ 'Hello! ' * times+ '</p>')
def db(request):
greeting = Greeting()
greeting.save()
greetings = Greeting.objects.all()
return render(request, "db.html", {"greetings": greetings})
Main Question: Is there a proper way to hide secrets locally in windows and access them by os.environ['KEY']?
Another solution theory: I was wondering if a python virtual environment has it's own environment variables. If yes i activate a venv locally without affecting the cloud. Therefore os.environ['KEY'] is redirected to the venv variables. Again it's just a theory.

You can use environment variables which you can get via os.environ['KEY'].
The same code will work on both local development and on Heroku.
On Heroku define these variables using ConfigVars heroku config:set KEY=val while locally (on Windows for example) define the same variables in an .env file (use dotenv to load them). The .env file is never committed with the source code.

Cloud Build env variables not passed to Django app on GAE

I have a Django app running on Google AppEngine Standard environment. I've set up a cloud build trigger from my master branch in Github to run the following steps:
steps:
- name: 'python:3.7'
entrypoint: python3
args: ['-m', 'pip', 'install', '--target', '.', '--requirement', 'requirements.txt']
- name: 'python:3.7'
entrypoint: python3
args: ['./manage.py', 'collectstatic', '--noinput']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy', 'app.yaml']
env:
- 'SHORT_SHA=$SHORT_SHA'
- 'TAG_NAME=$TAG_NAME'
I can see under the Execution Details tab on Cloud Build that the variables were actually set.
The problem is, SHORT_SHA and TAG_NAME aren't accessible from my Django app (followed instructions at https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values#using_user-defined_substitutions)! But if I set them in my app.yaml file with hardcoded values under env_variables, then my Django app can access those hardcoded values (and the values set in my build don't overwrite those hardcoded in app.yaml).
Why is this? Am I accessing them/setting them incorrectly? Should I be setting them in app.yaml somehow?
I even printed the whole os.environ dictionary in one of my views to see if they were just there with different names or something, but they're not present in there.

Not the cleanest solution, but I used this medium post as a guidance to my solution. I hypothesize that runserver command isn't being passed those env variables, and that those variables are only used for the app deploy command.
Write a Python script to dump the current environment variables in a .env file in project dir
In your settings file, read env variables from the .env file (I used django-environ library for this)
Add a step to cloud build file that runs your new Python script and pass env variables in that step (you're essentially dumping these variables into a .env file in this step)
- name: 'python:3.7'
entrypoint: python3
args: ['./create_env_file.py']
env:
- 'SHORT_SHA=$SHORT_SHA'
- 'TAG_NAME=$TAG_NAME'
Set the variables through Substitution Variables section in Edit Trigger page in Cloud Build
Now your application should have these env variables when app deploy happens

Where do I set environment variables on my Django Digital Ocean server?

I'm running my Django project on my Ubuntu 16.04 Digital Ocean server running Gunicorn/Nginx. I have my whole project except my settings.py file so am looking do add that in now - however don't want to hardcode the SECRET_KEY - so am looking to define an environment variable like it says in the Django docs: SECRET_KEY = os.environ['SECRET_KEY'].
Where do I define this variable? Is it in my gunicorn config file (/etc/systemd/system/gunicorn.service)

You can create environmental variables inside your .bashrc file in your home folder.
Just open the .bashrc file from home folder
sudo vi ~/.bashrc
And then at the end of the file, add your variable
export SECRET_KEY='your secret key'
then save it, and try running source command on the file so as to enable the variable(So that it gets applied without restarting the system)
source ~/.bashrc

How does django settings tied to web application

Suppose i my web application running with following settings
LOG_DIR = "/var/log/main"
This variable defines where the log should go.
Now then i run my tests and i have test_settings like
from settings import *
LOG_DIR = "/var/log/test"
Now i want to know that does that mean while my test script is running then my main application logs will also go to test folder because i have chnaged the global variable.
For my integration testiing i need to change variables if am afraid that if that will affect the main application or not .Like my application depends upon
Shell ENV variables . I wanted to chnage that for my tests. but i am afraid that if that will chnage the main running application.
This is not for PROD but for other testing applications environment

Running a django process will not affect the settings of another process. Just make sure to explicitly pass the --settings flag to your manage.py script when you run your tests/dev server/ etc..
e.g.
python manage.py test --settings project/settings/test.py
python manage.py runserver --settings project/settings
I prefer having a set of shell scripts that get sourced when I run my test environment (you can configure that with a test runner, see the docs).