I have the following Pyramid .ini file:
[DEFAULT]
redis.host = localhost
redis.port = 6379
redis.db = 0
[app:main]
...
# beaker session
session.type = redis
session.url = localhost:6379
In the app:main section's session.url I want to use what's defined under DEFAULT section's redis.host and redis.port.
In my understanding everything under DEFAULT section is global and is passed to other sections. But if I want to reuse a settings from DEFAULT and assign it a different name under other sections how do I do that?
I'm looking at the same way I can reference section entry in buildout .cfg files using ${<section name>:<entry>}.
session.url = %(redis.host)s:%(redis.port)s
Should do the trick.
Related
I have a django web app, with a middleware that intercepts incoming requests, extracts user details (added to header by upstream middleware) in the request header, and checks if user is authorized to access the page if user is member of a distribution group.
I'm using django-environ to manage my environment variables so i can modify the list of DL Groups which can access my page without changing the code.
# in Middleware, only included important part of code
from django.conf import settings
MINIMAL_MEMBERSHIP = settings.AUTHORIZED_MEMBERSHIP_REQUIREMENT # This should pass in as a list
server_object = Server(LDAP_SERVER)
conn = Connection(server_object, LDAP_USER, LDAP_PASS, client_strategy=SAFE_SYNC, auto_bind=True)
status, result, response, _ = conn.search(
search_base=requester_dn,
search_filter = '(objectClass=User)',
attributes = ['memberOf']
)
authorized = False
requester_membership_list = response[0]['raw_attributes']['memberOf']
for membership in requester_membership_list:
ad_group_name = membership.decode('utf-8')
if ad_group_name in MINIMAL_MEMBERSHIP:
authorized = True
break
# In settings.py
AUTHORIZED_MEMBERSHIP_REQUIREMENT = env.list('AUTHORIZED_MEMBERSHIP_REQUIREMENT')
# In .env
AUTHORIZED_MEMBERSHIP_REQUIREMENT="CN=Virtualisation team,OU=Distribution Group,OU=Exchange,OU=APPS,DC=xxx,DC=xxx,DC=xxx,DC=com",
According to django-environ, you can read .env as a list like
# .env
LIST_ENV=one,two,three,four
# settings.py
LIST_ENV=env.list(LIST_ENV)
print(LIST_ENV) # outputs ['one', 'two', 'three', 'four']
But understandably ldap DN format will break this as a full DN is already delimited by commas, so:
# .env
DN_LIST="CN=1,OU=1,OU=1,OU=1,DC=xxx,DC=xxx,DC=xxx,DC=com","CN=2,OU=2,OU=2,OU=2,DC=xxx,DC=xxx,DC=xxx,DC=com"
# settings.py
DN_LIST=env.list(DN_LIST)
# Actual Behavior
print(DN_LIST)
# DN_LIST = ['CN=1', 'OU=1', 'OU=1', ...]
# len(DN_LIST) will output 16
I would like achieve this:
# Behavior i want
print(DN_LIST)
# DN_LIST = ["CN=1,OU=1,OU=1,OU=1,DC=xxx,DC=xxx,DC=xxx,DC=com","CN=2,OU=2,OU=2,OU=2,DC=xxx,DC=xxx,DC=xxx,DC=com"]
# len(DN_LIST) will output 2
Is there anyway to do this, or maybe any alternatives to manage the list from .env without modifying the code if i need to add/remove new groups?
Thank you
Sometimes taking a night off provides a simple solution to an otherwise simple problem.. heh.
Decided to take the DN List in .env as a string and have the DNs delimited by something (e.g. ;) and manually split/process it into a list.
# .env
AUTHORIZED_MEMBERSHIP_REQUIREMENT=CN=1,OU=1,OU=1,OU=1,DC=xxx,DC=xxx,DC=xxx,DC=com;CN=2,OU=2,OU=2,OU=2,DC=xxx,DC=xxx,DC=xxx,DC=com
# .settings.py
AUTHORIZED_MEMBERSHIP_REQUIREMENT = env.str('AUTHORIZED_MEMBERSHIP_REQUIREMENT').split(';')
This is about how you read the shell environment variable (.env) into python
DN_LIST="CN=1,OU=1,OU=1,OU=1,DC=xxx" ## instead this
DN_LIST_NEW='["CN=1,OU=1,OU=1,OU=1,DC=xxx","CN=2,OU=2,OU=2,OU=2,DC=xxx"]' ## please you this
Off-topic: define these DN in .env means if we want to change the DN values, we have to change in .env and **restart/deploy** the application , which is not good. You should think about put these DN values into a table in database
Goal: Set the Heroku DATABASE_URL variable to sqlalchemy.url=postgres://... settings in __init__.py and development.ini file. Currently, I am connecting directly to the database address (which can change).
Issue as reported by Heroku support:
If you have hard coded the database connection string into your ini
file that is probably not the best idea. While it will work for now,
if sometime in the future we need to change where you database is
running (which does happen for various reasons) then your application
will no longer connect to your database. If your database does move,
we do keep the DATABASE_URL updated so your application should use
this. Maybe change sqlalchemy.url to sqlalchemy.url = os.environ.get('DATABASE_URL') if this is not what it is already set to.
However, the address sqlalchemy.url = os.environ.get('DATABASE_URL') does NOT work. It crashes my app. I have even attempted: sqlalchemy.url = postgresql://os.environ.get('DATABASE_URL'), sqlalchemy.url = postgres://os.environ.get('DATABASE_URL'), and finally sqlalchemy.url = postgres://'DATABASE_URL'. All of which do NOT work.
SQLALCEHMY engine_config setup: docs
sqlalchemy.engine_from_config(configuration, prefix='sqlalchemy.',
**kwargs)
Create a new Engine instance using a configuration dictionary.
The dictionary is typically produced from a config file.
The keys of interest to engine_from_config() should be prefixed, e.g.
sqlalchemy.url, sqlalchemy.echo, etc. The ‘prefix’ argument indicates
the prefix to be searched for. Each matching key (after the prefix is
stripped) is treated as though it were the corresponding keyword
argument to a create_engine() call.
The only required key is (assuming the default prefix) sqlalchemy.url,
which provides the database URL.
A select set of keyword arguments will be “coerced” to their expected
type based on string values. The set of arguments is extensible
per-dialect using the engine_config_types accessor.
Parameters: configuration – A dictionary (typically produced from a
config file, but this is not a requirement). Items whose keys start
with the value of ‘prefix’ will have that prefix stripped, and will
then be passed to create_engine. prefix – Prefix to match and then
strip from keys in ‘configuration’. kwargs – Each keyword argument to
engine_from_config() itself overrides the corresponding item taken
from the ‘configuration’ dictionary. Keyword arguments should not be
prefixed.
Outside Example (doesn't work for me):
I believe the issue is with the way settings and engine are setup in my app. I found this tutorial helpful, but my code is different: Environment Variables in Pyramid
What we ultimately want to do is dynamically set the sqlalchemy.url to
the value of our DATABASE_URL environment variable.
learning_journal/init.py is where our .ini file’s configuration
gets bound to our Pyramid app. Before the current settings get added
to the Configurator, we can use os.environ to bring in our
environment’s DATABASE_URL.
# __init__.py
import os
from pyramid.config import Configurator
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
settings["sqlalchemy.url"] = os.environ["DATABASE_URL"]
config = Configurator(settings=settings)
config.include('pyramid_jinja2')
config.include('.models')
config.include('.routes')
config.scan()
return config.make_wsgi_app()
Because we should always try to keep code DRY (and prevent future confusion), remove the sqlalchemy.url keyword from development.ini.
MY CODE:
init.py
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application"""
#global_config argument is a dictionary of key/value pairs mentioned in the [DEFAULT] section of an development.ini file
# **settings argument collects another set of arbitrary key/value pairs
#The main function first creates a SQLAlchemy database engine using sqlalchemy.engine_from_config() from the sqlalchemy. prefixed settings in the development.ini file's [app:main] section. This will be a URI (something like sqlite://):
engine = engine_from_config(settings, 'sqlalchemy.')
Session.configure(bind=engine)
Base.metadata.bind = engine
...
config.include('pyramid_jinja2')
config.include('pyramid_mailer')
config.add_static_view('static', 'static', cache_max_age=3600)
development.ini
#former db:
#sqlalchemy.url = postgres://localhost/NOTSSdb
#works, but unstable should db move:
sqlalchemy.url = postgres://ijfbcvuyifb.....
initialize_db script:
def main(argv=sys.argv):
if len(argv) < 2:
usage(argv)
config_uri = argv[1]
options = parse_vars(argv[2:])
setup_logging(config_uri)
settings = get_appsettings(config_uri, options=options)
engine = engine_from_config(settings)
I am new to pyramid.
The Issue is I am not able to figure out how app specific settings (key value pairs) work in pyramid.
This is what I have done after various google searches and other stackoverflow answers:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
if '__file__' in global_config:
settings.update(
load_sensitive_settings(global_config['__file__'], global_config))
config = Configurator(settings=settings)
config.include('pyramid_chameleon')
# config.add_static_view('static', 'static', cache_max_age=3600)
# config.add_route('home', '/')
config.add_route(
'tags',
'/tags', request_method='POST', accept='application/json', )
config.scan()
return config.make_wsgi_app()
def load_sensitive_settings(configurationPath, defaultByKey):
'Load sensitive settings from hidden configuration file'
# configFolder, configName = os.path.split(configurationPath)
# sensitivePath = os.path.join(configFolder, '.' + configName)
sensitivePath = configurationPath
settings = {}
configParser = ConfigParser.ConfigParser(defaultByKey)
if not configParser.read(sensitivePath):
log.warn('Could not open %s' % sensitivePath)
return settings
settings.update(configParser.items('custom'))
return settings
I have a file where I try to fetch settings like this:
from pyramid.threadlocal import get_current_registry
settings = get_current_registry().settings
value = settings['my_key']
But I always get settings object as None.
This is how I am defining my custom settings in development.ini
[custom]
my_key = ''
This is how I start my server in develpoment
pserve development.ini
I have read that request.settings can give me settings, But that approach is not feasible for me as my key contains the name of a file which is 1.5GBs and it has to be present in memory all the time. It takes around 5 minutes to load that file in server, hence cannot load the file on demand.
Please advice.
Thanks a lot for all the help in advance.
Update:
Thanks to all the answers provided, I finally made it work.
This is how my main function looks like:
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
config = Configurator(settings=settings)
config.include('pyramid_chameleon')
if '__file__' in global_config:
init_config(global_config['__file__'])
And I made a config file, this is how my config file looks like:
import ConfigParser
settings = dict()
def init_config(filename):
config = ConfigParser.ConfigParser()
config.read(filename)
settings_dict = config.items('custom')
settings.update(settings_dict)
Now wherever I want settings, I just do:
from projectname.config import settings
settings.get('my_key')
And I put my app specific settings (development/production.py) like this
[custom]
my_key = value
Regards
HM
Easiest way is putting your settings to the app main section with dot separated names. Example:
[app:main]
websauna.site_name = Trees
websauna.site_tag_line = Enjoy
websauna.site_url = http://localhost:6543
websauna.site_author = Trees team
Then you can do:
my_settings_value = request.registry.settings.get("websauna.site_name", "Default value)
WSGI pipeline does not bring you settings from other sections and you need to reparse the INI file with ConfigParser if you want to access the other sections (as far as I know).
If you need to load a lot of data during development time just store a filename in settings and load the file when you need to access the data, so that you don't slow the web server startup.
Here is my working solution:
config.ini
[APP.CONFIG]
url = http://....
[SMTP.CONFIG]
smtp.server = ...
smtp.port = 25
smtp.login = ...
smtp.password = ...
smtp.from = ...
[DB.CONFIG]
db.database=...
db.host=...
db.port=..
db.user=...
db.password=...
config.py
import configparser
config = configparser.ConfigParser()
config._interpolation = configparser.ExtendedInterpolation()
config.read(encoding='utf-8', filenames=['path to file/config.ini'])
smtp = config['SMTP.CONFIG']
db = config['DB.CONFIG']
mail = config['APP.CONFIG']
And how i use it in APP
from config import db
host = db['db.host']
If, like me, you are using PasteDeploy with Pyramid, the Pyramid docs here explain how you can use a [DEFAULT] section in your .ini configuration file to hold your custom parameters.
You might also benefit from reading the documentation on .ini files, since it gives some snippets which make it all much clearer.
I have a Python project where I am using a Twilio number (201-282-1111), but I do not want this private number to be in my public Git repository. I could make a new repo and just not include the values, but I would like it be easy enough for non techie people to use my script.
My question is, what can I do to mask the fake number provided below? Also, I have Twilio account_sid and auth_token that I would like to mask as well.
client.sms.messages.create(
body = "Your song was not found",
to = phoneNum,
from_ = "+2012821111")
account_sid = "XX1a116061fe67df9ab9eb0bc3c7ed0111"
auth_token = "1x11bc1b6555c52a93762f46d45861xx"
client = TwilioRestClient(account_sid,auth_token)
Use a config file. On one of my projects, this is what I did:
Install ConfigParser.
Create a file named config (or another name, your choice) that looks like this:
[Config]
account_sid=<your sid>
auth_token=<token>
phone_number=<your number>
Edit your .gitignore file and add config to it, this will keep Git from committing your config file to the repository
[optional] Add a config.sample file, which is a copy of the config file but with your values set to defaults or dummy values. Commit this one to Git. This helps other people set up your app later - all they have to do is copy the file to config and put in their credentials.
To access the config values, use this code:
config = ConfigParser.RawConfigParser()
config.read('config')
try:
your_value = config.get("Config", "your_value")
print your_value
except ConfigParser.NoOptionError:
print "'your_value' isn't defined in the config file!"
Another option would be to use an environment variable or to simply ask the user for their credentials when your app starts.
I'm currently writing a data collection service for multiple services. There are probably 5 different API Endpoints with differing hosts & port numbers. I wanted to create a settings file for this but thought that the .ini should be a better place, or so I thought...
My development.ini looks something like this:
[app:main]
use = egg:awesomeproject
auth.tkt = 'abc'
auth.secret = 'I love python'
mongodb.host = 'somehost'
mongodb.port= 6379
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543
[user:sg:qa]
host = 127.0.0.1
port = 1234
[customer:sg:qa]
host = 127.0.0.2
port = 4567
I tried to access the custom sections within a pyramid event like such:
def add_api_path(event):
request = event.request
settings = request.registry.settings
_type = 'customer:sg:qa'
base_config = settings[_type]
But that didn't work, because settings is actually a dict of the [app:main] attributes. Can someone teach me the way to access the sections the Pyramid way? I read about another way, using ConfigParser, but I wanted to ask if there's any other easier way in Pyramid first.
If you want to do that you'll have to parse the config file yourself. The section-isolation behavior you're seeing is intentional.
def main(global_conf, **settings):
parser = ConfigParser({'here': global_conf['__here__']})
parser.read(global_conf['__file__'])
for k, v in parser.items('user:sg:qa'):
settings['user:sg:qa:' + k] = v
config = Configurator(settings=settings)
Then later you can grab the settings:
request.registry.settings['user:sg:qa:host']
update
In Pyramid 1.9 the ini parsing was made pluggable and a new library was created to assist in loading arbitrary sections of the file in a standard way. Below is the updated example:
import plaster
def main(global_conf, **settings):
user_settings = plaster.get_settings(global_conf['__file__'], 'user:sg:qa')
for k, v in user_settings.items():
settings['user:sg:qa:' + k] = v
config = Configurator(settings=settings)