We Keep Coding

Default value for a variable in a class

I need to parse the environment value from a config file or from os environments in a class.
I am looking for a way to have a default for the env variable in case the environment is not found in neither the os.environ nor in the parsed config file.
I have done this: but I am not sure it is the right place? is the __init__ the right place to define those variables? that are to be re-used later on to establish db connections?
import yaml
import os
from socket import gethostname
class wrapper(object):
with open('config') as fd:
config = yaml.safe_load(fd)
hostname = gethostname()
def __init__(self, env='prod'):
self.db_server = None
self.db_default_user = None
self.db_connection = None
for envmt,data in self.config.items():
if self.hostname in data.get('host'):
env = envmt
#override by environment variable
if 'CMS_ENV' in os.environ:
env = os.environ['CMS_ENV']
# Might be overwritten by ENV variables
db_default_user = self.config[env]['db_default_user']
db_server = self.config[env]['db_server']
def db_conn(self):
user = self.db_default_user
the question is how to define a default value to env to fallback to 'prod'? should this defined at class level or while initializing the instance.
the variable dev is used to get the right db_server and correct user_name to connect and fetch data from a mssql db I must make sure it is defined.
this is the content of the config file:
test:
hosts: [vmtest,vmtest2]
db_server: cmreplsta01.netdev.deutsche-boerse.de
db_default_user: example\DB-user
prod:
hosts: [vmprod,vmprod2]
db_server: cmsdb.io.deutsche-boerse.de
db_default_user: example\DB-userprod
I detect the hostname where the script runs the load the db_server and db_default_user accordingly. if the script is running from a host not in the config then i default the env to 'prod' an use the prod values.
i have not been able to find any similar question that would fit my use case.

Why not just use a dictionary to store the defaults, then update the values from the environment?
Something like:
import os
defaults = {'EDITOR': 'nano'}
defaults.update(...) # e.g. your config file
defaults.update(os.environ)
Edit
If you don't want to clutter up your defaults variable with all the system environment variables, you can filter it to only contain the variables which are common between the original defaults and the environment:
import os
defaults = {'EDITOR': 'nano'}
common_variables = os.environ & env.keys() # set intersection
filtered_env = {k:os.environ[k] for k in common_variables}
defaults.update(filtered_env)

ImproperlyConfigured Set the xxxx environment variable - django-environ

In my settings.py i'm using django-environ like this:
import os
import environ
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
env = environ.Env(
SECRET_KEY=str,
)
env_path = os.path.join(BASE_DIR, '.env')
environ.Env.read_env('.env')
SECRET_KEY = env('SECRET_KEY')
My .env file looks like this
SECRET_KEY = ******************
However, when I run the app i get
django.core.exceptions.ImproperlyConfigured: Set the SECRET_KEY
environment variable
The .env file is found and lines are being read from it, so there's no problem with finding it, but somehow it doesn't work.
While following the execution thread i discovered that the regex for interpreting the .env lines is returning nothing
environ/environ.py
for line in content.splitlines():
m1 = re.match(r'\A(?:export )?([A-Za-z_0-9]+)=(.*)\Z', line)
if m1:
key, val = m1.group(1), m1.group(2)
m2 = re.match(r"\A'(.*)'\Z", val)
if m2:
val = m2.group(1)
m3 = re.match(r'\A"(.*)"\Z', val)
if m3:
val = re.sub(r'\\(.)', r'\1', m3.group(1))
cls.ENVIRON.setdefault(key, str(val))enter code here
re.match(r'\A(?:export )?([A-Za-z_0-9]+)=(.*)\Z', line) is returning none
Am i missing something here?

I found the answer by checking the regex responsible for interpreting the .env lines - \A(?:export )?([A-Za-z_0-9]+)=(.*)\Z
There should be no spaces between the key and the value
Problem:
KEY = VALUE
Good:
KEY=VALUE

How to hide SQL Plus password in Python script

I created Python script which truncate Oracle table. I use SQL Plus, but the problem is that I have to hide password which is plain text now. I have arguments like these:
db_name = "DB_NAME"
db_user = "DB_USER"
db_password = "DB_PASS"
Then I run command like:
sqlplus_delete_table = 'echo "TRUNCATE TABLE ' + db_user + '.' + table + ' DROP STORAGE;"'
sqlplus_connection = db_user + '/' + db_password + '#' + db_name
os.system(sqlplus_delete_table + ' | sqlplus -s ' + sqlplus_connection)
Everything works fine, but the problem is password. As I know, SQL Plus does not use jceks files. So what are other solutions to hide password?

You can use a solution like Django's SECRET_KEY, which I store in a file that is not in the project repository. From this file I load the keys like this in settings.py:
with open(os.path.join(ROOT_DIR, 'etc/secret_key.txt')) as f:
SECRET_KEY = f.read().strip()
In the above example the contents of the text file is just the key, but you can use structured formats such as JSON, YAML, or even a Python file and import it.
Example of Python secret file:
# secret.py
DB_PSSWD='pswd'
DB_USER='user'
In your source code simply:
import secret
print(DB_USER)
Example of YAML secret file:
# secret.yaml
db_psswd: pswd
db_user: user
In your source code simply:
import yaml
with open('secret.yaml') as yaml_secret:
rules = yaml.load(cfg)
print(rules['db_user'])

On Linux it's possible to create bash-script like:
# sql.env
export db_PSSWD='pswd'
export db_USER='user'
Before running python, run bash-script to initialize environment variables:
source sql.env
Then, in python:
db_psswd = os.environ.get("db_PSSWD")
db_user = os.environ.get("db_USER")

How to add your app specific settings in pyramid rest framework?

I am new to pyramid.
The Issue is I am not able to figure out how app specific settings (key value pairs) work in pyramid.
This is what I have done after various google searches and other stackoverflow answers:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
if '__file__' in global_config:
settings.update(
load_sensitive_settings(global_config['__file__'], global_config))
config = Configurator(settings=settings)
config.include('pyramid_chameleon')
# config.add_static_view('static', 'static', cache_max_age=3600)
# config.add_route('home', '/')
config.add_route(
'tags',
'/tags', request_method='POST', accept='application/json', )
config.scan()
return config.make_wsgi_app()
def load_sensitive_settings(configurationPath, defaultByKey):
'Load sensitive settings from hidden configuration file'
# configFolder, configName = os.path.split(configurationPath)
# sensitivePath = os.path.join(configFolder, '.' + configName)
sensitivePath = configurationPath
settings = {}
configParser = ConfigParser.ConfigParser(defaultByKey)
if not configParser.read(sensitivePath):
log.warn('Could not open %s' % sensitivePath)
return settings
settings.update(configParser.items('custom'))
return settings
I have a file where I try to fetch settings like this:
from pyramid.threadlocal import get_current_registry
settings = get_current_registry().settings
value = settings['my_key']
But I always get settings object as None.
This is how I am defining my custom settings in development.ini
[custom]
my_key = ''
This is how I start my server in develpoment
pserve development.ini
I have read that request.settings can give me settings, But that approach is not feasible for me as my key contains the name of a file which is 1.5GBs and it has to be present in memory all the time. It takes around 5 minutes to load that file in server, hence cannot load the file on demand.
Please advice.
Thanks a lot for all the help in advance.
Update:
Thanks to all the answers provided, I finally made it work.
This is how my main function looks like:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
config = Configurator(settings=settings)
config.include('pyramid_chameleon')
if '__file__' in global_config:
init_config(global_config['__file__'])
And I made a config file, this is how my config file looks like:
import ConfigParser
settings = dict()
def init_config(filename):
config = ConfigParser.ConfigParser()
config.read(filename)
settings_dict = config.items('custom')
settings.update(settings_dict)
Now wherever I want settings, I just do:
from projectname.config import settings
settings.get('my_key')
And I put my app specific settings (development/production.py) like this
[custom]
my_key = value
Regards
HM

Easiest way is putting your settings to the app main section with dot separated names. Example:
[app:main]
websauna.site_name = Trees
websauna.site_tag_line = Enjoy
websauna.site_url = http://localhost:6543
websauna.site_author = Trees team
Then you can do:
my_settings_value = request.registry.settings.get("websauna.site_name", "Default value)
WSGI pipeline does not bring you settings from other sections and you need to reparse the INI file with ConfigParser if you want to access the other sections (as far as I know).
If you need to load a lot of data during development time just store a filename in settings and load the file when you need to access the data, so that you don't slow the web server startup.

Here is my working solution:
config.ini
[APP.CONFIG]
url = http://....
[SMTP.CONFIG]
smtp.server = ...
smtp.port = 25
smtp.login = ...
smtp.password = ...
smtp.from = ...
[DB.CONFIG]
db.database=...
db.host=...
db.port=..
db.user=...
db.password=...
config.py
import configparser
config = configparser.ConfigParser()
config._interpolation = configparser.ExtendedInterpolation()
config.read(encoding='utf-8', filenames=['path to file/config.ini'])
smtp = config['SMTP.CONFIG']
db = config['DB.CONFIG']
mail = config['APP.CONFIG']
And how i use it in APP
from config import db
host = db['db.host']

If, like me, you are using PasteDeploy with Pyramid, the Pyramid docs here explain how you can use a [DEFAULT] section in your .ini configuration file to hold your custom parameters.
You might also benefit from reading the documentation on .ini files, since it gives some snippets which make it all much clearer.

Pass variables to development.ini on Pyramid

Using Pyramid + MySQL (sqlalchemy), I have the development.ini file and since I'm working with a team, everyone needs different database url on development.ini. Is there anyway to pass variables to development.ini files:
Something like this:
myconfig.py
DB_USER = 'user'
DB_PASSWORD = 'pass'
DB_DATABASE = 'db_name'
DB_HOSTNAME = 'localhost'
development.ini
sqlalchemy.url = mysql://DB_USER:DB_PASSWORD#DB_HOSTNAME/DB_DATABASE
That way, every developer can have a non-versioned myconfig.py.

Sounds kinda complicated - import variables from Python into an .ini file only to load that file back into Python :)
How about that:
try:
from .myconfig import DB_STRING
engine = sa.create_engine(DB_STRING, echo=False)
except ImportError:
# ... no myconfig.py found - proceed to configuring the engine from .ini file