I need to parse the environment value from a config file or from os environments in a class.
I am looking for a way to have a default for the env variable in case the environment is not found in neither the os.environ nor in the parsed config file.
I have done this: but I am not sure it is the right place? is the __init__ the right place to define those variables? that are to be re-used later on to establish db connections?
import yaml
import os
from socket import gethostname
class wrapper(object):
with open('config') as fd:
config = yaml.safe_load(fd)
hostname = gethostname()
def __init__(self, env='prod'):
self.db_server = None
self.db_default_user = None
self.db_connection = None
for envmt,data in self.config.items():
if self.hostname in data.get('host'):
env = envmt
#override by environment variable
if 'CMS_ENV' in os.environ:
env = os.environ['CMS_ENV']
# Might be overwritten by ENV variables
db_default_user = self.config[env]['db_default_user']
db_server = self.config[env]['db_server']
def db_conn(self):
user = self.db_default_user
the question is how to define a default value to env to fallback to 'prod'? should this defined at class level or while initializing the instance.
the variable dev is used to get the right db_server and correct user_name to connect and fetch data from a mssql db I must make sure it is defined.
this is the content of the config file:
test:
hosts: [vmtest,vmtest2]
db_server: cmreplsta01.netdev.deutsche-boerse.de
db_default_user: example\DB-user
prod:
hosts: [vmprod,vmprod2]
db_server: cmsdb.io.deutsche-boerse.de
db_default_user: example\DB-userprod
I detect the hostname where the script runs the load the db_server and db_default_user accordingly. if the script is running from a host not in the config then i default the env to 'prod' an use the prod values.
i have not been able to find any similar question that would fit my use case.
Why not just use a dictionary to store the defaults, then update the values from the environment?
Something like:
import os
defaults = {'EDITOR': 'nano'}
defaults.update(...) # e.g. your config file
defaults.update(os.environ)
Edit
If you don't want to clutter up your defaults variable with all the system environment variables, you can filter it to only contain the variables which are common between the original defaults and the environment:
import os
defaults = {'EDITOR': 'nano'}
common_variables = os.environ & env.keys() # set intersection
filtered_env = {k:os.environ[k] for k in common_variables}
defaults.update(filtered_env)
Related
I have different set of config files initialized under a Config class and would need them updated dynamically in the run time.
config.py
# Package import
import reusable.common as common
import reusable.JSON_utils as JSON
class Config:
def __init__(self):
# run_config
run_file = common.file_path('run_config.JSON')
self.run_dict = JSON.read(run_file)
self.env_list = list(self.run_dict.keys())
# connection_config
self.conn_file = common.file_path('connection_config.JSON')
self.conn_dict = JSON.read(self.conn_file)
# snapshot_config
self.snap_file = common.file_path('snapshot_config.JSON')
self.snap_dict = JSON.read(self.snap_file)
For example I have to iterate through different environments like (DEV, STAGE, QA, PROD) and want to update conn_dict('env') to 'QA' from 'DEV' after DEV tasks are completed. Currently I have the update dict/JSON code in main() but I want to have this as a method inside the Config class
# Package import
import reusable.config as config
import reusable.JSON_utils as JSON
config_obj = config.Config()
for env in config_obj.env_list:
config_obj.conn_dict['env'] = env
JSON.write(config_obj.conn_dict, config_obj.conn_file)
src_id_list = config_obj.run_dict.get(env).get('snapshot')
# do stuff in the current env
for src_id in src_id_list:
config_obj.snap_dict['source_id'] = str(src_id)
JSON.write(config_obj.snap_dict, config_obj.snap_file)
# do stuff for the current data source
Q1 Which method is the optimal and conventional way for this. Class or instant or static method? kindly explain as I'm not clear with those completely
Q2 Can we have a unified method inside the class that takes the dict variable and function as parameters to update the dict and JSON file. If yes how it can be achieved?
I've been struggling to achieve this for a while now and it seems that I can't find my way around this. I have the following main entry point for my FastAPI project:
from fastapi import FastAPI
from starlette.middleware.cors import CORSMiddleware
from starlette.responses import RedirectResponse
from app.core.config import get_api_settings
from app.api.api import api_router
def get_app() -> FastAPI:
api_settings = get_api_settings()
server = FastAPI(**api_settings.fastapi_kwargs)
server.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
server.include_router(api_router, prefix="/api")
#server.get("/", include_in_schema=False)
def redirect_to_docs() -> RedirectResponse:
return RedirectResponse(api_settings.docs_url)
return server
app = get_app()
Nothing too fancy so far. As you can see, I'm importing get_api_settings which holds my entire service config and it looks like this:
from functools import lru_cache
from typing import Any, Dict
from pydantic import BaseSettings
class APISettings(BaseSettings):
"""This class enables the configuration of your FastAPI instance
through the use of environment variables.
Any of the instance attributes can be overridden upon instantiation by
either passing the desired value to the initializer, or by setting the
corresponding environment variable.
Attribute `xxx_yyy` corresponds to environment variable `API_XXX_YYY`.
So, for example, to override `api_prefix`, you would set the environment
variable `API_PREFIX`.
Note that assignments to variables are also validated, ensuring that
even if you make runtime-modifications to the config, they should have
the correct types.
"""
# fastapi.applications.FastAPI initializer kwargs
debug: bool = False
docs_url: str = "/docs"
openapi_prefix: str = ""
openapi_url: str = "/openapi.json"
redoc_url: str = "/redoc"
title: str = "Api Backend"
version: str = "0.1.0"
# Custom settings
disable_docs: bool = False
environment: str
#property
def fastapi_kwargs(self) -> Dict[str, Any]:
"""This returns a dictionary of the most commonly used keyword
arguments when initializing a FastAPI instance.
If `self.disable_docs` is True, the various docs-related arguments
are disabled, preventing spec from being published.
"""
fastapi_kwargs: Dict[str, Any] = {
"debug": self.debug,
"docs_url": self.docs_url,
"openapi_prefix": self.openapi_prefix,
"openapi_url": self.openapi_url,
"redoc_url": self.redoc_url,
"title": self.title,
"version": self.version
}
if self.disable_docs:
fastapi_kwargs.update({
"docs_url": None,
"openapi_url": None,
"redoc_url": None
})
return fastapi_kwargs
class Config:
case_sensitive = True
# env_file should be dynamic depending on the
# `environment` env variable
env_file = ""
env_prefix = ""
validate_assignment = True
#lru_cache()
def get_api_settings() -> APISettings:
"""This function returns a cached instance of the APISettings object.
Caching is used to prevent re-reading the environment every time the API
settings are used in an endpoint.
If you want to change an environment variable and reset the cache
(e.g., during testing), this can be done using the `lru_cache` instance
method `get_api_settings.cache_clear()`.
"""
return APISettings()
I'm trying to prepare this service for multiple environments:
dev
stage
prod
For each of the above, I have three different .env files as follow:
core/configs/dev.env
core/configs/stage.env
core/configs/prod.env
As an example, here is how a .env file looks like:
environment=dev
frontend_service_url=http://localhost:3000
What I can't get my head around is how to dynamically set the env_file = "" in my Config class based on the environment attribute in my APISettings BaseSettings class.
Reading through Pydantic's docs I thought I can use the customise_sources classmethod to do something like this:
def load_envpath_settings(settings: BaseSettings):
environment = # not sure how to access it here
for env in ("dev", "stage", "prod"):
if environment == env:
return f"app/configs/{environment}.env"
class APISettings(BaseSettings):
# ...
class Config:
case_sensitive = True
# env_file = "app/configs/dev.env"
env_prefix = ""
validate_assignment = True
#classmethod
def customise_sources(cls, init_settings, env_settings, file_secret_settings):
return (
init_settings,
load_envpath_settings,
env_settings,
file_secret_settings,
)
but I couldn't find a way to access the environment in my load_envpath_settings. Any idea how to solve this? Or if there's another way to do it? I've also tried creating another #property in my APISettings class which which would basically be the same as the load_envpath_settings but I couldn't refer it back in the Config class.
First; usually you'd copy the file you want to have active into the .env file, and then just load that. If you however want to let that .env file control which of the configurations that are active:
You can have two sets of configuration - one that loads the initial configuration (i.e. which environment is the active one) from .env, and one that loads the actual application settings from the core/configs/<environment>.env file.
class AppSettings(BaseSettings):
environment:str = 'development'
This would be affected by the configuration given in .env (which is the default file name). You'd then use this value to load the API configuration by using the _env_file parameter, which is support on all BaseSettings instances.
def get_app_settings() -> AppSettings:
return AppSettings()
def get_api_settings() -> APISettings:
app_settings = get_app_settings()
return APISettings(_env_file=f'core/configs/{app_settings.environment}.env') # or os.path.join() and friends
Goal: Set the Heroku DATABASE_URL variable to sqlalchemy.url=postgres://... settings in __init__.py and development.ini file. Currently, I am connecting directly to the database address (which can change).
Issue as reported by Heroku support:
If you have hard coded the database connection string into your ini
file that is probably not the best idea. While it will work for now,
if sometime in the future we need to change where you database is
running (which does happen for various reasons) then your application
will no longer connect to your database. If your database does move,
we do keep the DATABASE_URL updated so your application should use
this. Maybe change sqlalchemy.url to sqlalchemy.url = os.environ.get('DATABASE_URL') if this is not what it is already set to.
However, the address sqlalchemy.url = os.environ.get('DATABASE_URL') does NOT work. It crashes my app. I have even attempted: sqlalchemy.url = postgresql://os.environ.get('DATABASE_URL'), sqlalchemy.url = postgres://os.environ.get('DATABASE_URL'), and finally sqlalchemy.url = postgres://'DATABASE_URL'. All of which do NOT work.
SQLALCEHMY engine_config setup: docs
sqlalchemy.engine_from_config(configuration, prefix='sqlalchemy.',
**kwargs)
Create a new Engine instance using a configuration dictionary.
The dictionary is typically produced from a config file.
The keys of interest to engine_from_config() should be prefixed, e.g.
sqlalchemy.url, sqlalchemy.echo, etc. The ‘prefix’ argument indicates
the prefix to be searched for. Each matching key (after the prefix is
stripped) is treated as though it were the corresponding keyword
argument to a create_engine() call.
The only required key is (assuming the default prefix) sqlalchemy.url,
which provides the database URL.
A select set of keyword arguments will be “coerced” to their expected
type based on string values. The set of arguments is extensible
per-dialect using the engine_config_types accessor.
Parameters: configuration – A dictionary (typically produced from a
config file, but this is not a requirement). Items whose keys start
with the value of ‘prefix’ will have that prefix stripped, and will
then be passed to create_engine. prefix – Prefix to match and then
strip from keys in ‘configuration’. kwargs – Each keyword argument to
engine_from_config() itself overrides the corresponding item taken
from the ‘configuration’ dictionary. Keyword arguments should not be
prefixed.
Outside Example (doesn't work for me):
I believe the issue is with the way settings and engine are setup in my app. I found this tutorial helpful, but my code is different: Environment Variables in Pyramid
What we ultimately want to do is dynamically set the sqlalchemy.url to
the value of our DATABASE_URL environment variable.
learning_journal/init.py is where our .ini file’s configuration
gets bound to our Pyramid app. Before the current settings get added
to the Configurator, we can use os.environ to bring in our
environment’s DATABASE_URL.
# __init__.py
import os
from pyramid.config import Configurator
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
settings["sqlalchemy.url"] = os.environ["DATABASE_URL"]
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
Because we should always try to keep code DRY (and prevent future confusion), remove the sqlalchemy.url keyword from development.ini.
MY CODE:
init.py
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application"""
#global_config argument is a dictionary of key/value pairs mentioned in the [DEFAULT] section of an development.ini file
# **settings argument collects another set of arbitrary key/value pairs
#The main function first creates a SQLAlchemy database engine using sqlalchemy.engine_from_config() from the sqlalchemy. prefixed settings in the development.ini file's [app:main] section. This will be a URI (something like sqlite://):
engine = engine_from_config(settings, 'sqlalchemy.')
Session.configure(bind=engine)
Base.metadata.bind = engine
...
config.include('pyramid_jinja2')
config.include('pyramid_mailer')
config.add_static_view('static', 'static', cache_max_age=3600)
development.ini
#former db:
#sqlalchemy.url = postgres://localhost/NOTSSdb
#works, but unstable should db move:
sqlalchemy.url = postgres://ijfbcvuyifb.....
initialize_db script:
def main(argv=sys.argv):
if len(argv) < 2:
usage(argv)
config_uri = argv[1]
options = parse_vars(argv[2:])
setup_logging(config_uri)
settings = get_appsettings(config_uri, options=options)
engine = engine_from_config(settings)
I have the following Pyramid .ini file:
[DEFAULT]
redis.host = localhost
redis.port = 6379
redis.db = 0
[app:main]
...
# beaker session
session.type = redis
session.url = localhost:6379
In the app:main section's session.url I want to use what's defined under DEFAULT section's redis.host and redis.port.
In my understanding everything under DEFAULT section is global and is passed to other sections. But if I want to reuse a settings from DEFAULT and assign it a different name under other sections how do I do that?
I'm looking at the same way I can reference section entry in buildout .cfg files using ${<section name>:<entry>}.
session.url = %(redis.host)s:%(redis.port)s
Should do the trick.
I want to change the env.hosts dynamically because sometimes I want to deploy to one machine first, check if ok then deploy to many machines.
Currently I need to set env.hosts first, how could I set the env.hosts in a method and not in global at script start?
Yes you can set env.hosts dynamically. One common pattern we use is:
from fabric.api import env
def staging():
env.hosts = ['XXX.XXX.XXX.XXX', ]
def production():
env.hosts = ['YYY.YYY.YYY.YYY', 'ZZZ.ZZZ.ZZZ.ZZZ', ]
def deploy():
# Do something...
You would use this to chain the tasks such as fab staging deploy or fab production deploy.
Kind of late to the party, but I achieved this with ec2 like so (note in EC2 you do not know what the ip/hostname may be, generally speaking - so you almost have to go dynamic to really account for how the environment/systems could come up - another option would be to use dyndns, but then this would still be useful):
from fabric.api import *
import datetime
import time
import urllib2
import ConfigParser
from platform_util import *
config = ConfigParser.RawConfigParser()
#task
def load_config(configfile=None):
'''
***REQUIRED*** Pass in the configuration to use - usage load_config:</path/to/config.cfg>
'''
if configfile != None:
# Load up our config file
config.read(configfile)
# Key/secret needed for aws interaction with boto
# (anyone help figure out a better way to do this with sub modules, please don't say classes :-) )
global aws_key
global aws_sec
aws_key = config.get("main","aws_key")
aws_sec = config.get("main","aws_sec")
# Stuff for fabric
env.user = config.get("main","fabric_ssh_user")
env.key_filename = config.get("main","fabric_ssh_key_filename")
env.parallel = config.get("main","fabric_default_parallel")
# Load our role definitions for fabric
for i in config.sections():
if i != "main":
hostlist = []
if config.get(i,"use-regex") == 'yes':
for x in get_running_instances_by_regex(aws_key,aws_sec,config.get(i,"security-group"),config.get(i,"pattern")):
hostlist.append(x.private_ip_address)
env.roledefs[i] = hostlist
else:
for x in get_running_instances(aws_key,aws_sec,config.get(i,"security-group")):
hostlist.append(x.private_ip_address)
env.roledefs[i] = hostlist
if config.has_option(i,"base-group"):
if config.get(i,"base-group") == 'yes':
print "%s is a base group" % i
print env.roledefs[i]
# env["basegroups"][i] = True
where get_running_instances and get_running_instances_by_regex are utility functions that make use of boto (http://code.google.com/p/boto/)
ex:
import logging
import re
from boto.ec2.connection import EC2Connection
from boto.ec2.securitygroup import SecurityGroup
from boto.ec2.instance import Instance
from boto.s3.key import Key
########################################
# B-O-F get_instances
########################################
def get_instances(access_key=None, secret_key=None, security_group=None):
'''
Get all instances. Only within a security group if specified., doesnt' matter their state (running/stopped/etc)
'''
logging.debug('get_instances()')
conn = EC2Connection(aws_access_key_id=access_key, aws_secret_access_key=secret_key)
if security_group:
sg = SecurityGroup(connection=conn, name=security_group)
instances = sg.instances()
return instances
else:
instances = conn.get_all_instances()
return instances
Here is a sample of what my config looked like:
# Config file for fabric toolset
#
# This specific configuration is for <whatever> related hosts
#
#
[main]
aws_key = <key>
aws_sec = <secret>
fabric_ssh_user = <your_user>
fabric_ssh_key_filename = /path/to/your/.ssh/<whatever>.pem
fabric_default_parallel = 1
#
# Groupings - Fabric knows them as roledefs (check env dict)
#
# Production groupings
[app-prod]
security-group = app-prod
use-regex = no
pattern =
[db-prod]
security-group = db-prod
use-regex = no
pattern =
[db-prod-masters]
security-group = db-prod
use-regex = yes
pattern = mysql-[d-s]01
Yet another new answer to an old question. :) But I just recently found myself attempting to dynamically set hosts, and really have to disagree with the main answer. My idea of dynamic, or at least what I was attempting to do, was take an instance DNS-name that was just created by boto, and access that instance with a fab command. I couldn't do fab staging deploy, because the instance doesn't exist at fabfile-editing time.
Fortunately, fabric does support a truly dynamic host-assignment with execute. (It's possible this didn't exist when the question was first asked, of course, but now it does). Execute allows you to define both a function to be called, and the env.hosts it should use for that command. For example:
def create_EC2_box(data=fab_base_data):
conn = boto.ec2.connect_to_region(region)
reservations = conn.run_instances(image_id=image_id, ...)
...
return instance.public_dns_name
def _ping_box():
run('uname -a')
run('tail /var/log/cloud-init-output.log')
def build_box():
box_name = create_EC2_box(fab_base_data)
new_hosts = [box_name]
# new_hosts = ['ec2-54-152-152-123.compute-1.amazonaws.com'] # testing
execute(_ping_box, hosts=new_hosts)
Now I can do fab build_box, and it will fire one boto call that creates an instance, and another fabric call that runs on the new instance - without having to define the instance-name at edit-time.