I'm using fastapi-azure-auth to make call to my API impossible, if the user is not logged in (doesn't pass a valid token in the API call from the UI to be precise).
My question doesn't have anything to do with this particular library, it's about FastAPI in general.
I use a class (SingleTenantAzureAuthorizationCodeBearer) which is callable. It is used in two cases:
api.onevent("startup") - to connect to Azure
as a dependency in routes that user wants to have authentication in
To initialize it, it requires some things like Azure IDs etc. I provide those via a config file.
The problem is, this class is created when the modules get evaluated, so the values from the config file would have to be already present.
So, I have this:
dependecies.py
azure_scheme = SingleTenantAzureAuthorizationCodeBearer(
app_client_id=settings.APP_CLIENT_ID,
tenant_id=settings.TENANT_ID,
scopes={
f'api://{settings.APP_CLIENT_ID}/user_impersonation': 'user_impersonation',
}
)
api.py
from .dependencies import azure_scheme
api = FastAPI(
title="foo"
)
def init_api() -> FastAPI:
# I want to read configuration here
api.swagger_ui.init_oauth = {"clientID": config.CLIENT_ID}
return api
#api.on_event('startup')
async def load_config() -> None:
"""
Load OpenID config on startup.
"""
await azure_scheme.openid_config.load_config()
#api.get("/", dependencies=[Depends(azure_scheme)])
def test():
return {"hello": "world"}
Then I'd run the app with gunicorn -k uvicorn.workers.UvicornWorker foo:init_api().
So, for example, the Depends part will get evaluated before init_api, before reading the config. I would have to read the config file before that happens. And I don't want to do that, I'd like to control when the config reading happens (that's why I have init_api function where I initialize the logging and other stuff).
My question would be: is there a way to first read the config then initialize a dependency like SingleTenantAzureAuthorizationCodeBearer so I can use the values from config for this initialization?
#Edit
api.py:
from fastapi import Depends, FastAPI, Response
from fastapi.middleware.cors import CORSMiddleware
from .config import get_config
from .dependencies import get_azure_scheme
api = FastAPI(
title="Foo",
swagger_ui_oauth2_redirect_url="/oauth2-redirect",
)
api.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
def init_api() -> FastAPI:
api.swagger_ui_init_oauth = {
"usePkceWithAuthorizationCodeGrant": True,
"clientId": get_config().client_id,
}
return api
#api.get("/test", dependencies=[Depends(get_azure_scheme)])
def test():
return Response(status_code=200)
config.py:
import os
from functools import lru_cache
import toml
from pydantic import BaseSettings
class Settings(BaseSettings):
client_id: str
tenant_id: str
#lru_cache
def get_config():
with open(os.getenv("CONFIG_PATH", ""), mode="r") as config_file:
config_data = toml.load(config_file)
return Settings(
client_id=config_data["azure"]["CLIENT_ID"], tenant_id=config_data["azure"]["TENANT_ID"]
)
dependencies.py:
from fastapi import Depends
from fastapi_azure_auth import SingleTenantAzureAuthorizationCodeBearer
from .config import Settings, get_config
def get_azure_scheme(config: Settings = Depends(get_config)):
return SingleTenantAzureAuthorizationCodeBearer(
app_client_id=config.client_id,
tenant_id=config.tenant_id,
scopes={
f"api://{config.client_id}/user": "user",
},
)
I have different set of config files initialized under a Config class and would need them updated dynamically in the run time.
config.py
# Package import
import reusable.common as common
import reusable.JSON_utils as JSON
class Config:
def __init__(self):
# run_config
run_file = common.file_path('run_config.JSON')
self.run_dict = JSON.read(run_file)
self.env_list = list(self.run_dict.keys())
# connection_config
self.conn_file = common.file_path('connection_config.JSON')
self.conn_dict = JSON.read(self.conn_file)
# snapshot_config
self.snap_file = common.file_path('snapshot_config.JSON')
self.snap_dict = JSON.read(self.snap_file)
For example I have to iterate through different environments like (DEV, STAGE, QA, PROD) and want to update conn_dict('env') to 'QA' from 'DEV' after DEV tasks are completed. Currently I have the update dict/JSON code in main() but I want to have this as a method inside the Config class
# Package import
import reusable.config as config
import reusable.JSON_utils as JSON
config_obj = config.Config()
for env in config_obj.env_list:
config_obj.conn_dict['env'] = env
JSON.write(config_obj.conn_dict, config_obj.conn_file)
src_id_list = config_obj.run_dict.get(env).get('snapshot')
# do stuff in the current env
for src_id in src_id_list:
config_obj.snap_dict['source_id'] = str(src_id)
JSON.write(config_obj.snap_dict, config_obj.snap_file)
# do stuff for the current data source
Q1 Which method is the optimal and conventional way for this. Class or instant or static method? kindly explain as I'm not clear with those completely
Q2 Can we have a unified method inside the class that takes the dict variable and function as parameters to update the dict and JSON file. If yes how it can be achieved?
If I have a folder structure like the attached picture. The test.config.py is an empty file.
The default.py contains name variables indicate default folder root and addin information
local_root = r'c:\temp\project\cache'
local_input = local_root + r'inputs'
local_output = local_root + r'outputs'
addin_location = r'c:\user\...
addin_name = r'project_addin'
addin_version = r'1.1'
The setting.py contains name variables to overwrite addin information for testing.
addin_location = r'd:\user\...
addin_name = r'project_addin'
addin_version = r'2.1'
I want to import all variables from default.py and all variables from setting.py to init.py. Then overwrite variables with the same names imported from default.py use setattr(). i.e the addin_location, addin_name, and addin_version in default.py share the same name as variables in setting.py, thus overwrite those with setting.py.
Lastly, for any test.py files in test folder, it cannot refers to any of the variables using default.names or setting.names, but instead use config.names (basically the config.py should contain all variable names from default.py with overwritten information from setting.py, so that the codes in test.py only refer to the variable names in config.py). I have manually updated all reference to config.py but don't know how to put all variable names to config.py as it is an empty file. I think need to write some functions in init.py to dump those variable names to config.py
Thanks for the help.
Do you need to keep a track of those config in a file or do you just want to access all information from the combine default.py + any setting.py overwrite ?
For the latter, I think you can use dataclasses:
# default.py
from dataclasses import dataclass
# Here I am defining the default parameters for the class instance
#dataclass
class Config:
local_root: str = r'c:\temp\project\cache'
local_input: str = local_root + r'inputs'
local_output: str = local_root + r'outputs'
addin_location: str = r'c:\user\...'
addin_name: str = r'project_addin'
addin_version: str = r'1.1'
Then in setting you can instantiate the class by replacing by placing all overwriting values in a dict.
# setting.py
from .default import Config
overwrite_settings = {
"addin_location": r'd:\user\...',
"addin_name": r'project_addin',
"addin_version": r'2.1'
}
config = Config(**overwrite_settings)
Then in your test file you can import the config object and access the variable as follow:
# test.py
from .setting import config
my_path_root = config.local_root
And if you want to use init just instantiate the class in the init.py file and import the config with it:
#init.py
from .setting import overwrite_settings
from .default import Config
config = Config(**overwrite_settings)
For the former I am not sure how to save directly into a ready to use python file. Especially at run time, it might be tricky.
But if you want to keep a track of the config that you run you can add a
__post_init__ method to your class in order to save it to a json for example:
# default.py
from dataclasses import dataclass
#dataclass
class Config:
local_root: str = r'c:\temp\project\cache'
local_input: str = local_root + r'inputs'
local_output: str = local_root + r'outputs'
addin_location: str = r'c:\user\...'
addin_name: str = r'project_addin'
addin_version: str = r'1.1'
def __post_init__(self):
with open('run_config.json', 'r') as file:
json.dump(self.__dict__, file)
Hope this helps.
I need to parse the environment value from a config file or from os environments in a class.
I am looking for a way to have a default for the env variable in case the environment is not found in neither the os.environ nor in the parsed config file.
I have done this: but I am not sure it is the right place? is the __init__ the right place to define those variables? that are to be re-used later on to establish db connections?
import yaml
import os
from socket import gethostname
class wrapper(object):
with open('config') as fd:
config = yaml.safe_load(fd)
hostname = gethostname()
def __init__(self, env='prod'):
self.db_server = None
self.db_default_user = None
self.db_connection = None
for envmt,data in self.config.items():
if self.hostname in data.get('host'):
env = envmt
#override by environment variable
if 'CMS_ENV' in os.environ:
env = os.environ['CMS_ENV']
# Might be overwritten by ENV variables
db_default_user = self.config[env]['db_default_user']
db_server = self.config[env]['db_server']
def db_conn(self):
user = self.db_default_user
the question is how to define a default value to env to fallback to 'prod'? should this defined at class level or while initializing the instance.
the variable dev is used to get the right db_server and correct user_name to connect and fetch data from a mssql db I must make sure it is defined.
this is the content of the config file:
test:
hosts: [vmtest,vmtest2]
db_server: cmreplsta01.netdev.deutsche-boerse.de
db_default_user: example\DB-user
prod:
hosts: [vmprod,vmprod2]
db_server: cmsdb.io.deutsche-boerse.de
db_default_user: example\DB-userprod
I detect the hostname where the script runs the load the db_server and db_default_user accordingly. if the script is running from a host not in the config then i default the env to 'prod' an use the prod values.
i have not been able to find any similar question that would fit my use case.
Why not just use a dictionary to store the defaults, then update the values from the environment?
Something like:
import os
defaults = {'EDITOR': 'nano'}
defaults.update(...) # e.g. your config file
defaults.update(os.environ)
Edit
If you don't want to clutter up your defaults variable with all the system environment variables, you can filter it to only contain the variables which are common between the original defaults and the environment:
import os
defaults = {'EDITOR': 'nano'}
common_variables = os.environ & env.keys() # set intersection
filtered_env = {k:os.environ[k] for k in common_variables}
defaults.update(filtered_env)
I'm trying to use Python's PyYAML to create a custom tag that will allow me to retrieve environment variables with my YAML.
import os
import yaml
class EnvTag(yaml.YAMLObject):
yaml_tag = u'!Env'
def __init__(self, env_var):
self.env_var = env_var
def __repr__(self):
return os.environ.get(self.env_var)
settings_file = open('conf/defaults.yaml', 'r')
settings = yaml.load(settings_file)
And inside of defaults.yaml is simply:
example: !ENV foo
The error I keep getting:
yaml.constructor.ConstructorError:
could not determine a constructor for the tag '!ENV' in
"defaults.yaml", line 1, column 10
I plan to have more than one custom tag as well (assuming I can get this one working)
Your PyYAML class had a few problems:
yaml_tag is case sensitive, so !Env and !ENV are different tags.
So, as per the documentation, yaml.YAMLObject uses meta-classes to define itself, and has default to_yaml and from_yaml functions for those cases. By default, however, those functions require that your argument to your custom tag (in this case !ENV) be a mapping. So, to work with the default functions, your defaults.yaml file must look like this (just for example) instead:
example: !ENV {env_var: "PWD", test: "test"}
Your code will then work unchanged, in my case print(settings) now results in {'example': /home/Fred} But you're using load instead of safe_load -- in their answer below, Anthon pointed out that this is dangerous because the parsed YAML can overwrite/read data anywhere on the disk.
You can still easily use your YAML file format, example: !ENV foo—you just have to define an appropriate to_yaml and from_yaml in class EnvTag, ones that can parse and emit scalar variables like the string "foo".
So:
import os
import yaml
class EnvTag(yaml.YAMLObject):
yaml_tag = u'!ENV'
def __init__(self, env_var):
self.env_var = env_var
def __repr__(self):
v = os.environ.get(self.env_var) or ''
return 'EnvTag({}, contains={})'.format(self.env_var, v)
#classmethod
def from_yaml(cls, loader, node):
return EnvTag(node.value)
#classmethod
def to_yaml(cls, dumper, data):
return dumper.represent_scalar(cls.yaml_tag, data.env_var)
# Required for safe_load
yaml.SafeLoader.add_constructor('!ENV', EnvTag.from_yaml)
# Required for safe_dump
yaml.SafeDumper.add_multi_representer(EnvTag, EnvTag.to_yaml)
settings_file = open('defaults.yaml', 'r')
settings = yaml.safe_load(settings_file)
print(settings)
s = yaml.safe_dump(settings)
print(s)
When this program is run, it outputs:
{'example': EnvTag(foo, contains=)}
{example: !ENV 'foo'}
This code has the benefit of (1) using the original pyyaml, so nothing extra to install and (2) adding a representer. :)
I'd like to share how I resolved this as an addendum to the great answers above provided by Anthon and Fredrick Brennan. Thank you for your help.
In my opinion, the PyYAML document isn't real clear as to when you might want to add a constructor via a class (or "metaclass magic" as described in the doc), which may involve re-defining from_yaml and to_yaml, or simply adding a constructor using yaml.add_constructor.
In fact, the doc states:
You may define your own application-specific tags. The easiest way to do it is to define a subclass of yaml.YAMLObject
I would argue the opposite is true for simpler use-cases. Here's how I managed to implement my custom tag.
config/__init__.py
import yaml
import os
environment = os.environ.get('PYTHON_ENV', 'development')
def __env_constructor(loader, node):
value = loader.construct_scalar(node)
return os.environ.get(value)
yaml.add_constructor(u'!ENV', __env_constructor)
# Load and Parse Config
__defaults = open('config/defaults.yaml', 'r').read()
__env_config = open('config/%s.yaml' % environment, 'r').read()
__yaml_contents = ''.join([__defaults, __env_config])
__parsed_yaml = yaml.safe_load(__yaml_contents)
settings = __parsed_yaml[environment]
With this, I can now have a seperate yaml for each environment using an env PTYHON_ENV (default.yaml, development.yaml, test.yaml, production.yaml). And each can now reference ENV variables.
Example default.yaml:
defaults: &default
app:
host: '0.0.0.0'
port: 500
Example production.yaml:
production:
<<: *defaults
app:
host: !ENV APP_HOST
port: !ENV APP_PORT
To use:
from config import settings
"""
If PYTHON_ENV == 'production', prints value of APP_PORT
If PYTHON_ENV != 'production', prints default 5000
"""
print(settings['app']['port'])
If your goal is to find and replace environment variables (as strings) defined in your yaml file, you can use the following approach:
example.yaml:
foo: !ENV "Some string with ${VAR1} and ${VAR2}"
example.py:
import yaml
# Define the function that replaces your env vars
def env_var_replacement(loader, node):
replacements = {
'${VAR1}': 'foo',
'${VAR2}': 'bar',
}
s = node.value
for k, v in replacements.items():
s = s.replace(k, v)
return s
# Define a loader class that will contain your custom logic
class EnvLoader(yaml.SafeLoader):
pass
# Add the tag to your loader
EnvLoader.add_constructor('!ENV', env_var_replacement)
# Now, use your custom loader to load the file:
with open('example.yaml') as yaml_file:
loaded_dict = yaml.load(yaml_file, Loader=EnvLoader)
# Prints: "Some string with foo and bar"
print(loaded_dict['foo'])
It's worth noting, you don't necessarily need to create a custom EnvLoader class. You can call add_constructor directly on the SafeLoader class or the yaml module itself. However, this can have an unintended side-effect of adding your loader globally to all other modules that rely on those loaders, which could potentially cuase problems if those other modules have their own custom logic for loading that !ENV tag.
There are several problems with your code:
!Env in your YAML file is not the same as !ENV in your code.
You are missing the classmethod from_yaml that has to be provided for EnvTag.
Your YAML document specifies a scalar for !Env, but the subclassing mechanism for yaml.YAMLObject calls construct_yaml_object which in turn calls construct_mapping so a scalar is not allowed.
You are using .load(). This is unsafe, unless you have complete control over the YAML input, now and in the future. Unsafe in the sense that uncontrolled YAML can e.g. wipe or upload any information from your disc. PyYAML doesn't warn you for that possible loss.
PyYAML only supports most of YAML 1.1, the latest YAML specification is 1.2 (from 2009).
You should consistently indent your code at 4 spaces at every level (or 3 spaces, but not 4 at the first and 3 a the next level).
your __repr__ doesn't return a string if the environment variable is not set, which will throw an error.
So change your code to:
import sys
import os
from ruamel import yaml
yaml_str = """\
example: !Env foo
"""
class EnvTag:
yaml_tag = u'!Env'
def __init__(self, env_var):
self.env_var = env_var
def __repr__(self):
return os.environ.get(self.env_var, '')
#staticmethod
def yaml_constructor(loader, node):
return EnvTag(loader.construct_scalar(node))
yaml.add_constructor(EnvTag.yaml_tag, EnvTag.yaml_constructor,
constructor=yaml.SafeConstructor)
data = yaml.safe_load(yaml_str)
print(data)
os.environ['foo'] = 'Hello world!'
print(data)
which gives:
{'example': }
{'example': Hello world!}
Please note that I am using ruamel.yaml (disclaimer: I am the author of that package), so you can use YAML 1.2 (or 1.1) in your YAML file. With minor changes you can do the above with the old PyYAML as well.
You can do this by subclassing of YAMLObject as well, and in a safe way:
import sys
import os
from ruamel import yaml
yaml_str = """\
example: !Env foo
"""
yaml.YAMLObject.yaml_constructor = yaml.SafeConstructor
class EnvTag(yaml.YAMLObject):
yaml_tag = u'!Env'
def __init__(self, env_var):
self.env_var = env_var
def __repr__(self):
return os.environ.get(self.env_var, '')
#classmethod
def from_yaml(cls, loader, node):
return EnvTag(loader.construct_scalar(node))
data = yaml.safe_load(yaml_str)
print(data)
os.environ['foo'] = 'Hello world!'
print(data)
This will give you the same results as above.