I have different set of config files initialized under a Config class and would need them updated dynamically in the run time.
config.py
# Package import
import reusable.common as common
import reusable.JSON_utils as JSON
class Config:
def __init__(self):
# run_config
run_file = common.file_path('run_config.JSON')
self.run_dict = JSON.read(run_file)
self.env_list = list(self.run_dict.keys())
# connection_config
self.conn_file = common.file_path('connection_config.JSON')
self.conn_dict = JSON.read(self.conn_file)
# snapshot_config
self.snap_file = common.file_path('snapshot_config.JSON')
self.snap_dict = JSON.read(self.snap_file)
For example I have to iterate through different environments like (DEV, STAGE, QA, PROD) and want to update conn_dict('env') to 'QA' from 'DEV' after DEV tasks are completed. Currently I have the update dict/JSON code in main() but I want to have this as a method inside the Config class
# Package import
import reusable.config as config
import reusable.JSON_utils as JSON
config_obj = config.Config()
for env in config_obj.env_list:
config_obj.conn_dict['env'] = env
JSON.write(config_obj.conn_dict, config_obj.conn_file)
src_id_list = config_obj.run_dict.get(env).get('snapshot')
# do stuff in the current env
for src_id in src_id_list:
config_obj.snap_dict['source_id'] = str(src_id)
JSON.write(config_obj.snap_dict, config_obj.snap_file)
# do stuff for the current data source
Q1 Which method is the optimal and conventional way for this. Class or instant or static method? kindly explain as I'm not clear with those completely
Q2 Can we have a unified method inside the class that takes the dict variable and function as parameters to update the dict and JSON file. If yes how it can be achieved?
Related
If I have a folder structure like the attached picture. The test.config.py is an empty file.
The default.py contains name variables indicate default folder root and addin information
local_root = r'c:\temp\project\cache'
local_input = local_root + r'inputs'
local_output = local_root + r'outputs'
addin_location = r'c:\user\...
addin_name = r'project_addin'
addin_version = r'1.1'
The setting.py contains name variables to overwrite addin information for testing.
addin_location = r'd:\user\...
addin_name = r'project_addin'
addin_version = r'2.1'
I want to import all variables from default.py and all variables from setting.py to init.py. Then overwrite variables with the same names imported from default.py use setattr(). i.e the addin_location, addin_name, and addin_version in default.py share the same name as variables in setting.py, thus overwrite those with setting.py.
Lastly, for any test.py files in test folder, it cannot refers to any of the variables using default.names or setting.names, but instead use config.names (basically the config.py should contain all variable names from default.py with overwritten information from setting.py, so that the codes in test.py only refer to the variable names in config.py). I have manually updated all reference to config.py but don't know how to put all variable names to config.py as it is an empty file. I think need to write some functions in init.py to dump those variable names to config.py
Thanks for the help.
Do you need to keep a track of those config in a file or do you just want to access all information from the combine default.py + any setting.py overwrite ?
For the latter, I think you can use dataclasses:
# default.py
from dataclasses import dataclass
# Here I am defining the default parameters for the class instance
#dataclass
class Config:
local_root: str = r'c:\temp\project\cache'
local_input: str = local_root + r'inputs'
local_output: str = local_root + r'outputs'
addin_location: str = r'c:\user\...'
addin_name: str = r'project_addin'
addin_version: str = r'1.1'
Then in setting you can instantiate the class by replacing by placing all overwriting values in a dict.
# setting.py
from .default import Config
overwrite_settings = {
"addin_location": r'd:\user\...',
"addin_name": r'project_addin',
"addin_version": r'2.1'
}
config = Config(**overwrite_settings)
Then in your test file you can import the config object and access the variable as follow:
# test.py
from .setting import config
my_path_root = config.local_root
And if you want to use init just instantiate the class in the init.py file and import the config with it:
#init.py
from .setting import overwrite_settings
from .default import Config
config = Config(**overwrite_settings)
For the former I am not sure how to save directly into a ready to use python file. Especially at run time, it might be tricky.
But if you want to keep a track of the config that you run you can add a
__post_init__ method to your class in order to save it to a json for example:
# default.py
from dataclasses import dataclass
#dataclass
class Config:
local_root: str = r'c:\temp\project\cache'
local_input: str = local_root + r'inputs'
local_output: str = local_root + r'outputs'
addin_location: str = r'c:\user\...'
addin_name: str = r'project_addin'
addin_version: str = r'1.1'
def __post_init__(self):
with open('run_config.json', 'r') as file:
json.dump(self.__dict__, file)
Hope this helps.
I need to parse the environment value from a config file or from os environments in a class.
I am looking for a way to have a default for the env variable in case the environment is not found in neither the os.environ nor in the parsed config file.
I have done this: but I am not sure it is the right place? is the __init__ the right place to define those variables? that are to be re-used later on to establish db connections?
import yaml
import os
from socket import gethostname
class wrapper(object):
with open('config') as fd:
config = yaml.safe_load(fd)
hostname = gethostname()
def __init__(self, env='prod'):
self.db_server = None
self.db_default_user = None
self.db_connection = None
for envmt,data in self.config.items():
if self.hostname in data.get('host'):
env = envmt
#override by environment variable
if 'CMS_ENV' in os.environ:
env = os.environ['CMS_ENV']
# Might be overwritten by ENV variables
db_default_user = self.config[env]['db_default_user']
db_server = self.config[env]['db_server']
def db_conn(self):
user = self.db_default_user
the question is how to define a default value to env to fallback to 'prod'? should this defined at class level or while initializing the instance.
the variable dev is used to get the right db_server and correct user_name to connect and fetch data from a mssql db I must make sure it is defined.
this is the content of the config file:
test:
hosts: [vmtest,vmtest2]
db_server: cmreplsta01.netdev.deutsche-boerse.de
db_default_user: example\DB-user
prod:
hosts: [vmprod,vmprod2]
db_server: cmsdb.io.deutsche-boerse.de
db_default_user: example\DB-userprod
I detect the hostname where the script runs the load the db_server and db_default_user accordingly. if the script is running from a host not in the config then i default the env to 'prod' an use the prod values.
i have not been able to find any similar question that would fit my use case.
Why not just use a dictionary to store the defaults, then update the values from the environment?
Something like:
import os
defaults = {'EDITOR': 'nano'}
defaults.update(...) # e.g. your config file
defaults.update(os.environ)
Edit
If you don't want to clutter up your defaults variable with all the system environment variables, you can filter it to only contain the variables which are common between the original defaults and the environment:
import os
defaults = {'EDITOR': 'nano'}
common_variables = os.environ & env.keys() # set intersection
filtered_env = {k:os.environ[k] for k in common_variables}
defaults.update(filtered_env)
I've been struggling to achieve this for a while now and it seems that I can't find my way around this. I have the following main entry point for my FastAPI project:
from fastapi import FastAPI
from starlette.middleware.cors import CORSMiddleware
from starlette.responses import RedirectResponse
from app.core.config import get_api_settings
from app.api.api import api_router
def get_app() -> FastAPI:
api_settings = get_api_settings()
server = FastAPI(**api_settings.fastapi_kwargs)
server.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
server.include_router(api_router, prefix="/api")
#server.get("/", include_in_schema=False)
def redirect_to_docs() -> RedirectResponse:
return RedirectResponse(api_settings.docs_url)
return server
app = get_app()
Nothing too fancy so far. As you can see, I'm importing get_api_settings which holds my entire service config and it looks like this:
from functools import lru_cache
from typing import Any, Dict
from pydantic import BaseSettings
class APISettings(BaseSettings):
"""This class enables the configuration of your FastAPI instance
through the use of environment variables.
Any of the instance attributes can be overridden upon instantiation by
either passing the desired value to the initializer, or by setting the
corresponding environment variable.
Attribute `xxx_yyy` corresponds to environment variable `API_XXX_YYY`.
So, for example, to override `api_prefix`, you would set the environment
variable `API_PREFIX`.
Note that assignments to variables are also validated, ensuring that
even if you make runtime-modifications to the config, they should have
the correct types.
"""
# fastapi.applications.FastAPI initializer kwargs
debug: bool = False
docs_url: str = "/docs"
openapi_prefix: str = ""
openapi_url: str = "/openapi.json"
redoc_url: str = "/redoc"
title: str = "Api Backend"
version: str = "0.1.0"
# Custom settings
disable_docs: bool = False
environment: str
#property
def fastapi_kwargs(self) -> Dict[str, Any]:
"""This returns a dictionary of the most commonly used keyword
arguments when initializing a FastAPI instance.
If `self.disable_docs` is True, the various docs-related arguments
are disabled, preventing spec from being published.
"""
fastapi_kwargs: Dict[str, Any] = {
"debug": self.debug,
"docs_url": self.docs_url,
"openapi_prefix": self.openapi_prefix,
"openapi_url": self.openapi_url,
"redoc_url": self.redoc_url,
"title": self.title,
"version": self.version
}
if self.disable_docs:
fastapi_kwargs.update({
"docs_url": None,
"openapi_url": None,
"redoc_url": None
})
return fastapi_kwargs
class Config:
case_sensitive = True
# env_file should be dynamic depending on the
# `environment` env variable
env_file = ""
env_prefix = ""
validate_assignment = True
#lru_cache()
def get_api_settings() -> APISettings:
"""This function returns a cached instance of the APISettings object.
Caching is used to prevent re-reading the environment every time the API
settings are used in an endpoint.
If you want to change an environment variable and reset the cache
(e.g., during testing), this can be done using the `lru_cache` instance
method `get_api_settings.cache_clear()`.
"""
return APISettings()
I'm trying to prepare this service for multiple environments:
dev
stage
prod
For each of the above, I have three different .env files as follow:
core/configs/dev.env
core/configs/stage.env
core/configs/prod.env
As an example, here is how a .env file looks like:
environment=dev
frontend_service_url=http://localhost:3000
What I can't get my head around is how to dynamically set the env_file = "" in my Config class based on the environment attribute in my APISettings BaseSettings class.
Reading through Pydantic's docs I thought I can use the customise_sources classmethod to do something like this:
def load_envpath_settings(settings: BaseSettings):
environment = # not sure how to access it here
for env in ("dev", "stage", "prod"):
if environment == env:
return f"app/configs/{environment}.env"
class APISettings(BaseSettings):
# ...
class Config:
case_sensitive = True
# env_file = "app/configs/dev.env"
env_prefix = ""
validate_assignment = True
#classmethod
def customise_sources(cls, init_settings, env_settings, file_secret_settings):
return (
init_settings,
load_envpath_settings,
env_settings,
file_secret_settings,
)
but I couldn't find a way to access the environment in my load_envpath_settings. Any idea how to solve this? Or if there's another way to do it? I've also tried creating another #property in my APISettings class which which would basically be the same as the load_envpath_settings but I couldn't refer it back in the Config class.
First; usually you'd copy the file you want to have active into the .env file, and then just load that. If you however want to let that .env file control which of the configurations that are active:
You can have two sets of configuration - one that loads the initial configuration (i.e. which environment is the active one) from .env, and one that loads the actual application settings from the core/configs/<environment>.env file.
class AppSettings(BaseSettings):
environment:str = 'development'
This would be affected by the configuration given in .env (which is the default file name). You'd then use this value to load the API configuration by using the _env_file parameter, which is support on all BaseSettings instances.
def get_app_settings() -> AppSettings:
return AppSettings()
def get_api_settings() -> APISettings:
app_settings = get_app_settings()
return APISettings(_env_file=f'core/configs/{app_settings.environment}.env') # or os.path.join() and friends
I want to move from Ansible to Nornir. In Ansbile I use dynamic inventory, where I use this python script to reference the host_var folder:
import json
import yaml
import glob
groups = {}
for hostfilename in glob.glob('./host_vars/*.yml'):
with open(hostfilename, 'r') as hostfile:
host = yaml.load(hostfile, Loader=yaml.FullLoader)
for hostgroup in host['host_groups']:
if hostgroup not in groups.keys():
groups[ hostgroup ] = { 'hosts': [] }
groups[ hostgroup ]['hosts'].append( host['hostname'] )
print(json.dumps(groups))
Question:
How can I use my existing Ansible Inventory in Nornir.
nornir.plugins.inventory.ansible.AnsibleInventory can only be used with 1x host.yaml file not with many, at least this is my understanding
Edit: Goal is to create always new Inventory files on every run. The workflow would be to generate the inventory yaml files in host_vars and then use it during the play.
Can somebody please help me?
Thanks
F.
If I understood you correctly, you want each yaml file in the host_vars folder to be interpreted as one host and its data. This feature is not part of base Nornir, but can be implemented via a custom inventory plugin.
The custom inventory plugin should implement a load() method that returns an Inventory-type object that Nornir can then use normally (see here for an example of the SimpleInventory implementation). I came up with this snippet adapted from the code that was given:
import os
import yaml
import glob
import pathlib
from nornir.core.inventory import (
Inventory,
Hosts,
Host,
Groups,
Group)
def map_host_data(host_dict):
return({
'hostname' : host_dict['hostname'],
'port': host_dict.get('port',22),
'username' : host_dict['username'],
'password' : host_dict['password'],
'platform' : host_dict['platform'],
'data' : host_dict.get('data', None)
})
class DynamicInventory:
def __init__(self, inventory_dir: str = "host_vars/") -> None:
self.inventory_dir = pathlib.Path(inventory_dir).expanduser()
def load(self):
hosts = Hosts()
groups = Groups()
for hostfilename in glob.glob(f"{self.inventory_dir}/*.yaml"):
with open(hostfilename,'r') as hostfile:
host_name = os.path.basename(hostfilename).replace('.yaml','')
host = yaml.load(hostfile, Loader=yaml.FullLoader)
for hostgroup in host['host_groups']:
if hostgroup not in groups.keys():
group = Group(name=hostgroup)
groups[hostgroup] = group
hosts[host_name] = Host(name=host_name, **map_host_data(host))
return Inventory(hosts=hosts,groups=groups,defaults={})
I'm assuming you're using Nornir >= 3 (which you really should), so don't forget to register your plugin if using it on your configuration. Assuming you put the above code under plugins/inventory.py:
from nornir import InitNornir
from plugins.inventory import DynamicInventory
from nornir.core.plugins.inventory import InventoryPluginRegister
InventoryPluginRegister.register("DynamicInventoryPlugin",DynamicInventory)
nr = InitNornir(inventory={'plugin': 'DynamicInventoryPlugin'},
runner={'plugin': 'threaded','options': {'num_workers': 20}})
This of course ignores some features (such as setting defaults), but can be modified to add more features that better match your current setup.
I want to change the env.hosts dynamically because sometimes I want to deploy to one machine first, check if ok then deploy to many machines.
Currently I need to set env.hosts first, how could I set the env.hosts in a method and not in global at script start?
Yes you can set env.hosts dynamically. One common pattern we use is:
from fabric.api import env
def staging():
env.hosts = ['XXX.XXX.XXX.XXX', ]
def production():
env.hosts = ['YYY.YYY.YYY.YYY', 'ZZZ.ZZZ.ZZZ.ZZZ', ]
def deploy():
# Do something...
You would use this to chain the tasks such as fab staging deploy or fab production deploy.
Kind of late to the party, but I achieved this with ec2 like so (note in EC2 you do not know what the ip/hostname may be, generally speaking - so you almost have to go dynamic to really account for how the environment/systems could come up - another option would be to use dyndns, but then this would still be useful):
from fabric.api import *
import datetime
import time
import urllib2
import ConfigParser
from platform_util import *
config = ConfigParser.RawConfigParser()
#task
def load_config(configfile=None):
'''
***REQUIRED*** Pass in the configuration to use - usage load_config:</path/to/config.cfg>
'''
if configfile != None:
# Load up our config file
config.read(configfile)
# Key/secret needed for aws interaction with boto
# (anyone help figure out a better way to do this with sub modules, please don't say classes :-) )
global aws_key
global aws_sec
aws_key = config.get("main","aws_key")
aws_sec = config.get("main","aws_sec")
# Stuff for fabric
env.user = config.get("main","fabric_ssh_user")
env.key_filename = config.get("main","fabric_ssh_key_filename")
env.parallel = config.get("main","fabric_default_parallel")
# Load our role definitions for fabric
for i in config.sections():
if i != "main":
hostlist = []
if config.get(i,"use-regex") == 'yes':
for x in get_running_instances_by_regex(aws_key,aws_sec,config.get(i,"security-group"),config.get(i,"pattern")):
hostlist.append(x.private_ip_address)
env.roledefs[i] = hostlist
else:
for x in get_running_instances(aws_key,aws_sec,config.get(i,"security-group")):
hostlist.append(x.private_ip_address)
env.roledefs[i] = hostlist
if config.has_option(i,"base-group"):
if config.get(i,"base-group") == 'yes':
print "%s is a base group" % i
print env.roledefs[i]
# env["basegroups"][i] = True
where get_running_instances and get_running_instances_by_regex are utility functions that make use of boto (http://code.google.com/p/boto/)
ex:
import logging
import re
from boto.ec2.connection import EC2Connection
from boto.ec2.securitygroup import SecurityGroup
from boto.ec2.instance import Instance
from boto.s3.key import Key
########################################
# B-O-F get_instances
########################################
def get_instances(access_key=None, secret_key=None, security_group=None):
'''
Get all instances. Only within a security group if specified., doesnt' matter their state (running/stopped/etc)
'''
logging.debug('get_instances()')
conn = EC2Connection(aws_access_key_id=access_key, aws_secret_access_key=secret_key)
if security_group:
sg = SecurityGroup(connection=conn, name=security_group)
instances = sg.instances()
return instances
else:
instances = conn.get_all_instances()
return instances
Here is a sample of what my config looked like:
# Config file for fabric toolset
#
# This specific configuration is for <whatever> related hosts
#
#
[main]
aws_key = <key>
aws_sec = <secret>
fabric_ssh_user = <your_user>
fabric_ssh_key_filename = /path/to/your/.ssh/<whatever>.pem
fabric_default_parallel = 1
#
# Groupings - Fabric knows them as roledefs (check env dict)
#
# Production groupings
[app-prod]
security-group = app-prod
use-regex = no
pattern =
[db-prod]
security-group = db-prod
use-regex = no
pattern =
[db-prod-masters]
security-group = db-prod
use-regex = yes
pattern = mysql-[d-s]01
Yet another new answer to an old question. :) But I just recently found myself attempting to dynamically set hosts, and really have to disagree with the main answer. My idea of dynamic, or at least what I was attempting to do, was take an instance DNS-name that was just created by boto, and access that instance with a fab command. I couldn't do fab staging deploy, because the instance doesn't exist at fabfile-editing time.
Fortunately, fabric does support a truly dynamic host-assignment with execute. (It's possible this didn't exist when the question was first asked, of course, but now it does). Execute allows you to define both a function to be called, and the env.hosts it should use for that command. For example:
def create_EC2_box(data=fab_base_data):
conn = boto.ec2.connect_to_region(region)
reservations = conn.run_instances(image_id=image_id, ...)
...
return instance.public_dns_name
def _ping_box():
run('uname -a')
run('tail /var/log/cloud-init-output.log')
def build_box():
box_name = create_EC2_box(fab_base_data)
new_hosts = [box_name]
# new_hosts = ['ec2-54-152-152-123.compute-1.amazonaws.com'] # testing
execute(_ping_box, hosts=new_hosts)
Now I can do fab build_box, and it will fire one boto call that creates an instance, and another fabric call that runs on the new instance - without having to define the instance-name at edit-time.