Working on a heroku django project.
Goal: I want to run the server locally with same code on cloud too (makes sense).
Problem:
Environments differ from a linux server (heroku) to a local PC windows system.
Local environment variables differ from cloud Heroku config vars.
Heroku config vars can be setup easily using the CLI heroku config:set TIMES=2.
While setting up local env vars is a total mess.
I tried the following in cmd:
py -c "import os;os.environ['Times']=2" # To set an env var
Then ran py -c "import os;os.environ.get('Times','Not Found')" stdout: "Not Found".
After a bit of research it appeared to be that such env vars are stored temporarily per process/session usage.
Solution theory: Redirect os.environ to .env file of the root heroku project instead of the PC env vars. So I found this tool direnv perfect for Unix-like OSs but not available for Windows.
views.py code (runs perfect on cloud, sick on the local machine):
import os
import requests
from django.shortcuts import render
from django.http import HttpResponse
from .models import Greeting
def index(request):
# get method takes 2 parameters (env_var_string,return value if var is not found)
times = int(os.environ.get('TIMES',3))
return HttpResponse('<p>'+ 'Hello! ' * times+ '</p>')
def db(request):
greeting = Greeting()
greeting.save()
greetings = Greeting.objects.all()
return render(request, "db.html", {"greetings": greetings})
Main Question: Is there a proper way to hide secrets locally in windows and access them by os.environ['KEY']?
Another solution theory: I was wondering if a python virtual environment has it's own environment variables. If yes i activate a venv locally without affecting the cloud. Therefore os.environ['KEY'] is redirected to the venv variables. Again it's just a theory.
You can use environment variables which you can get via os.environ['KEY'].
The same code will work on both local development and on Heroku.
On Heroku define these variables using ConfigVars heroku config:set KEY=val while locally (on Windows for example) define the same variables in an .env file (use dotenv to load them). The .env file is never committed with the source code.
Related
I'm developing a simple FastAPI app and I'm using Pydantic for storing app settings.
Some settings are populated from the environment variables set by Ansible deployment tools but some other settings are needed to be set explicitly from a separate env file.
So I have this in config.py
class Settings(BaseSettings):
# Project wide settings
PROJECT_MODE: str = getenv("PROJECT_MODE", "sandbox")
VERSION: str
class Config:
env_file = "config.txt"
And I have this config.txt
VERSION="0.0.1"
So project_mode env var is being set by deployment script and version is being set from the env file. The reason for that is that we'd like to keep deployment script similar across all projects, so any custom vars are populated from the project specific env files.
But the problem is that when I run the app, it fails with:
pydantic.error_wrappers.ValidationError: 1 validation error for Settings
VERSION
field required (type=value_error.missing)
So how can I populate Pydantic settings model from the local ENV file?
If the environment file isn't being picked up, most of the time it's because it isn't placed in the current working directory. In your application in needs to be in the directory where the application is run from (or if the application manages the CWD itself, to where it expects to find it).
In particular when running tests this can be a bit confusing, and you might have to configure your IDE to run tests with the CWD set to the project root if you run the tests from your IDE.
The path of env_file is relative to the current working directory, which confused me as well. In order to always use a path relative to the config module I set it up like this:
env_file: f"{pathlib.Path(__file__).resolve().parent}/config.txt"
I have a weird problem with some code I want to run. The code itself should not be the problem since it is downloaded from a Udemy class and not modified:
# coding=utf-8
from flask import Flask, render_template, request
app = Flask(__name__)
#app.route("/")
def hello():
items = ["Apfel", "Birne", "Banane"]
return render_template("start.html", name="Max Mustermann", items=items)
#app.route("/test")
def test():
name = request.args.get("name")
return render_template("test.html", name=name)
I found online that, to start the emulated webserver(?) I have to rund the following temrinal commands before I can see the output:
(base) Christophs-MBP:13-23 chris$ export FLASK_APP=run.py run flask
(base) Christophs-MBP:13-23 chris$ export FLASK_APP=run.py run flask
(base) Christophs-MBP:13-23 chris$ export FLASK_APP=run.py
(base) Christophs-MBP:13-23 chris$ run flask
bash: run: command not found
No reaction to my terminal commands
Basically there is no reaction to the command to start the server(?).It should reply with "Running on 127.0.0.1:5000" as soon as I've run the command once.
If I go to my browser, there is no page when I address http://127.0.0.1:5000. What am I doing wrong? I am pretty new to Python and an absolute rookie regarding the terminal. Not sure if I broke something there, since trying to install pyenv to manage my Python installs better as recommended by a friend does not work either (I cannot update the SDK headers as described on RealPython
What are the export statements?
On Mac, export key=value creates a new (or updates an existing one) environment variable - the tutorial most likely simply asked you to provide one where key is FLASK_APP and value is a path to your app.
To verify it's been saved correctly, you can list the variables by just typing export in the terminal and finding out what's inside each of the environment variables on your system (if you want to only view FLASK_APP you can type export | grep FLASK_APP).
Why do you need FLASK_APP?
When you call flask run in your terminal, you will see the following message:
Error: Could not locate a Flask application. You did not provide the "FLASK_APP" environment variable, and a "wsgi.py" or "app.py"
module was not found in the current directory.
I presume your file is called run.py, therefore none of the conditions are met. You could rename run.py to app.py and simply type flask run in the terminal, but you can also type export FLASK_APP=<path-to-run.py>. It seems the tutorial author decided to do the latter. Keep in mind that if you rename your file to app.py you will need to run flask run within the directory that file lives in. You can change directory in the terminal using cd command.
Why do you get bash: run: command not found?
bash is a language running inside your terminal, and it only knows of a few commands - it is not aware of any run commands. It does however know about flask command once you have installed it on your machine. Within the command's output there is a part which includes a run command:
Commands:
routes Show the routes for the app.
run Run a development server.
shell Run a shell in the app context.
Therefore, what you want to do is type flask run instead of just run in your terminal.
On my production server, I've set environment variables both inside and outside my virtualenv (only because I don't understand this issue going on) including a variable HELLO_WORLD_PROD which I've set to '1'. in the python interpreter, both inside and out my venv, os.environ.get('HELLO_WORLD_PROD') == '1' returns True. In my settings folder, I have:
import os
if os.environ.get('HELLO_WORLD_PROD') == '1':
from hello_world.settings.prod import * # noqa
else:
from hello_world.settings.dev import * # noqa
Both prod.py and dev.py inherit from base.py, and in base DEBUG = False, and only in dev.py does DEBUG = True.
However, when I trigger an error through the browser, I'm seeing the debug page.
I'm using nginx and gunicorn. Why is my application importing the wrong settings file?
You can see my gunicorn conf here
Thanks in advance for your patience!
I was using sudo service gunicorn start to run gunicorn. The problem is service strips all environment variables but TERM, PATH and LANG. To fix it, in my exec line in my gunicorn.conf I added the environment variables there using the --env flag, like exec env/bin/gunicorn --env HELLO_WORLD_PROD=1 --env DB_PASSWORD=secret etc.
I'm starting a docker container the following way:
docker run -e IP_AD=192.168.99.100 -p 80:80 flask_app
I'm simply trying to pass an IP Address to the flask application so that something can be loaded from my application. This resource will change from environment to environment, so this is the reason I would like to pass it as an environment variable.
Later, I would like to use this variable but from the context of the running flask application. How can I load IP_AD from my flask application and use it as a python variable?
I've tried doing this:
import os
os.environ.get('IP_AD')
But it does not seem to be loading anything. What is the correct way to load IP_AD passed from docker run -e
create file .env like this
model_path=./model/data_fs.csv
then install library .env by doing
pip3 install python-dotenv==0.17.1
add this to your code to load all your .env variable
from dotenv import load_dotenv
load_dotenv()
so you can access it
if os.getenv("model_path", None) is not None:
return os.getenv("model_path", None)
# otherwise raise an exception
raise Exception("no .env file found")
you can try like this
import os
os.environ["IP_AD"]
I have a Python application that is set up using the new New Relic configuration variables in the dotcloud.yml file, which works fine.
However I want to run a sandbox instance as a test/staging environment, so I want to be able to set the environment of the newrelic agent so that it uses the different configuration sections of the ini configuration. My dotcloud.yml is set up as follows:
www:
type: python
config:
python_version: 'v2.7'
enable_newrelic: True
environment:
NEW_RELIC_LICENSE_KEY: *****************************************
NEW_RELIC_APP_NAME: Application Name
NEW_RELIC_LOG: /var/log/supervisor/newrelic.log
NEW_RELIC_LOG_LEVEL: info
NEW_RELIC_CONFIG_FILE: /home/dotcloud/current/newrelic.ini
I have custom environment variables so that the sanbox is set as "test" and the live application is set to "production"
I am then calling the following in my uswsgi.py
NEWRELIC_CONFIG = os.environ.get('NEW_RELIC_CONFIG_FILE')
ENVIRONMENT = os.environ.get('MY_ENVIRONMENT', 'test')
newrelic.agent.initialize(NEWRELIC_CONFIG, ENVIRONMENT)
However the dotcloud instance is already enabling newrelic because I get this in the uwsgi.log file:
Sun Nov 18 18:50:12 2012 - unable to load app 0 (mountpoint='') (callable not found or import error)
Traceback (most recent call last):
File "/home/dotcloud/current/wsgi.py", line 15, in <module>
newrelic.agent.initialize(NEWRELIC_CONFIG, ENVIRONMENT)
File "/opt/ve/2.7/local/lib/python2.7/site-packages/newrelic-1.8.0.13/newrelic/config.py", line 1414, in initialize
log_file, log_level)
File "/opt/ve/2.7/local/lib/python2.7/site-packages/newrelic-1.8.0.13/newrelic/config.py", line 340, in _load_configuration
'environment "%s".' % (_config_file, _environment))
newrelic.api.exceptions.ConfigurationError: Configuration has already been done against differing configuration file or environment. Prior configuration file used was "/home/dotcloud/current/newrelic.ini" and environment "None".
So it would seem that the newrelic agent is being initialised before uwsgi.py is called.
So my question is:
Is there a way to initialise the newrelic environment?
The easiest way to do this, without changing any code would be to do the following.
Create a new sandbox app on dotCloud (see http://docs.dotcloud.com/0.9/guides/flavors/ for more information about creating apps in sandbox mode)
$ dotcloud create -f sandbox <app_name>
Deploy your code to the new sandbox app.
$ dotcloud push
Now you should have the same code running in both your live and sandbox apps. But because you want to change some of the ENV variables for the sandbox app, you need to do one more step.
According to this page http://docs.dotcloud.com/0.9/guides/environment/#adding-environment-variables there are 2 different ways of adding ENV variables.
Using the dotcloud.yml's environment section.
Using the dotcloud env cli command
Whereas dotcloud.yml allows you to define different environment variables for each service, dotcloud env set environment variables for the whole application. Moreover, environment variables set with dotcloud env supersede environment variables defined in dotcloud.yml.
That means that if we want to have different values for our sandbox app, we just need to run a dotcloud env command to set those variables on the sandbox app, which will override the ones in your dotcloud.yml
If we just want to change on variable we would run this command.
$ dotcloud env set NEW_RELIC_APP_NAME='Test Application Name'
If we want to update more then one at a time we would do the following.
$ dotcloud env set \
'NEW_RELIC_APP_NAME="Test Application Name"' \
'NEW_RELIC_LOG_LEVEL=debug'
To make sure that you have your env varibles set correctly you can run the following command.
$ dotcloud env list
Notes
The commands above, are using the new dotCloud 0.9.x CLI, if you are using the older one, you will need to either upgrade to the new one, or refer to the documentation for the old CLI http://docs.dotcloud.com/0.4/guides/environment/
When you set your environment variables it will restart your application so that it can install the variables, so to limit your downtime, set all of them in one command.
Unless they are doing something odd, you should be able to override the app_name supplied by the agent configuration file by doing:
import newrelic.agent
newrelic.agent.global_settings().app_name = 'Test Application Name'
Don't call newrelic.agent.initialize() a second time.
This will only work if app_name is listing a single application to report data to.