I am creating a piece of code that will have multiple files that need to reference a single config. The reason is that our IT department will use puppet to manage this config file in case any changes are required in the future. We don't want to do a release to change the configuration. I've seen a few projects that have multiple configs in different places but I really do not like this idea and I'd prefer to have a single source. My thought was to create a specific config.py file that can be called anywhere in my code that will ask for the user to input the location of the file.
Is this a good way or is there a better way to do this?
import configparser
class Config(object):
def __init__(self,conf):
self._cfg = configparser.ConfigParser()
self._cfg.read(conf)
def get_conf_value(self,property):
if property not in self._cfg.sections:
return None
return self._cfg.sections[property]
If so, if I have a Main.py while, what's the best way to have the scheduler pass the config location and then reference it across all of my files in my Python Package?
You can create a config.json file in json format. You can read the contents at startup using the json.load function from the json library.
Related
I am writing a Python program that polls various sources and tracks the metrics they output. This clearly has lots of personal credentials, including usernames, passwords and API keys. I want to be able to open source the program, but keep the credentials secret.
At the moment, I have simply got a file called config.py which contains all the sensitive credentials. I have copied this to a file called EXAMPLE_config.py and removed all the sensitive information. The first line is
# Add your information to the below then rename this file to config.py
I was planning to put a gitignore on config.py and to git the EXAMPLE_config.py file. While this works, it does seem a bit inefficient; every time I add a new credential to config.py, I would also need to add the variable name to EXAMPLE_config.py.
What is the best way to share this program on via git, on GitHub for example, without sharing the sensitive information? I have seen Configuration files in Python, however this suggests many options. I have also seen Remove sensitive files and their commits from Git history, however I want to prevent the data from ever being shared in the first place. Is there an accepted pythonic or general standard?
one of the most common way to deal with secrets in open source applications are environment-variables and .env files.
Example: https://medium.com/codait/environment-variables-or-keeping-your-secrets-secret-in-a-node-js-app-99019dfff716
Something I like to do is having a secret.py, which just reads all secrets and makes them available to the rest of the program in form of a python module, like this:
import os
reddit_client_id = os.environ['reddit_client_id']
reddit_client_secret = os.environ['reddit_client_secret']
reddit_user = os.environ['reddit_user']
reddit_password = os.environ['reddit_password']
streamable_pass = os.environ['streamable_pass']
streamable_user = os.environ['streamable_user']
source
The actual secrets are then stored in an .env file, which is ignored via .gitignore and read with docker-compose (in this particular case).
To expand on my comment: I typically have a config_local.py with all information, included in .gitignore. Then all relevant information is read by an interface module config.py, which is commited to github and throws a detailed error message including a config_local template for copy and paste if any of the relevant information is missing. The advantage is obviously that this is self-explanatory if I pass the repository to someone else. Works for me so far.
Example:
repo_name/config.py (commited):
try:
import config_local
PASSWORD = config_local.PASSWORD
MYPATH = config_local.MYPATH
# could also make this more fine-grained or define default values
except Exception as e:
print(e) # to get info on whether variables are missing or the file was not found
logger.error('''
Please set configuration parameters in
repo_name/config_local.py, e.g.
PASSWORD = 'password1234'
MYPATH = '/path'
'''
)
# could also raise something here, if variables are strictly required
repo_name/conf_local.py (in .gitignore):
PASSWORD = 'password1234'
MYPATH = '/path'
My Revit Add-in reads at some point a text file, that could be located anywhere. In my current implementation, the path to the text file is hardcoded. I'd like to avoid that, so that when I distribute the Add-in to other people, it doesn't simply crash.
Ideally I'd like to give them the ability of specifying their own location for that file on their computer, and that they don't need to re-specify it every time they re-launch the Add-in!
In other words, I'd like to store once and for all this information. And if you close and re-open Revit, the location is still stored somewhere when you re-use the Addin.
This question is actually similar to this one, except that I'd need a solution when developing in Python (pyRevit). Any help?
if you're developing you addon in pyRevit, then you can use the pyrevit.script module to get the configuration for that script.
Ask user for the file location (pyrevit.forms.save_file helps) and then save the file path in the script configuration. pyRevit handles this automatically and saves the information inside its master configuration file at %appdata%/pyRevit
from pyrevit import script
config = script.get_config()
config.filelocation = 'path/to/your/file'
script.save_config()
And then later, read the configuration like this:
from pyrevit import script
config = script.get_config()
print(config.filelocation)
# or to get the config safely
print(config.get_option('filelocation', None)
I implemented two other ways to store Revit add-in settings in the HoloLens Escape Path Waypoint JSON Exporter:
Store add-in option settings in XML using the .NET System.Configuration.ApplicationSettingsBase class
Store add-in option settings in JSON using custom solution and JavaScriptSerializer class
Both solutions are well suited for what you need.
Check them out in the ExportWaypointsJson GitHub repository.
Good day.
I set up a separate project from the main one called myproject-celery. It is a buildout based project which contains the async part of my project. For convenience I want to have a file, that will be containing this machine's configuration. I know that celery provides the python config file, but I do not like this configuration style.
Let's say I have a configuration in a Yaml config file named myproject.yaml
What I want to achieve:
./bin/celery worker --config /absolute/path/to/project/myproject.yaml --app myproject.celery
The problem really is that I want to specify the file's location, because it can change. I tried writing a custom loader class, but I failed, cause I do not even know why and when the many custom methods of this class are called (the only doc that I found is http://docs.celeryproject.org/en/latest/reference/celery.loaders.base.html?highlight=loader#id1 and It's no help for me). I tried to do something on import phase for the app module, but I can not pass the filepath to that module's code... The only solution that I came up with was using a custom ENV param that will contain the path, but I do not see why can't it be a launch param like in most apps, that I use(refering to pyramid with it's paster serve myproject.ini)
So the question:
What do I have to do to set up the config from a file that I could specify by an absolute path?
EDIT:
The question was not answered, sow I posted an issue on celery's github. Will wait for a response.
https://github.com/celery/celery/issues/1100
Looking at celery.loaders.base it looks like the method you want to override is read_configuration:
from celery.datastructures import DictAttr
from celery.loaders.base import BaseLoader
class YAMLLoader(BaseLoader):
def read_configuration():
# Load YAML file here and return a DictAttr instance
I have a project with 10 different python files. It has classes and functions - pretty much the lot.
I do want to share specific data that will represent the settings in the project between all the project files.
I came up with creating a settings.py file:
settings = {}
settings['max_bitrate'] = 160000
settings['dl_dir'] = r"C:\Downloads"
and then I import the class from every file.
Is there a more suitable way to do it?
I'm probably a little old-school in this regard, but in my latest project, I created a config file in /etc, then created a config module that uses ConfigParser to read it in and make it available, and import that config module wherever I need to read settings.
Your method sounds good to me, and has the advantage that you can easily change the implementation of the settings module, for example to use configuration files or the windows registry, or to provided a read only API.
I am trying to use python for translating a set of templates to a set of configuration files based on values taken from a main configuration file. However, I am having certain issues. Consider the following example of a template file.
file1.cfg.template
%(CLIENT1)s %(HOST1)s %(PORT1)d C %(COMPID1)s
%(CLIENT2)s %(HOST2)s %(PORT2)d C %(COMPID2)s
This file contains an entry for each client. There are hundreds of config files like this and I don't want to have logic for each type of config file. Python should do the replacements and generate config files automatically given a set of global values read from a main xml config file. However, in the above example, if CLIENT2 does not exist, how do I delete that line? I expect Python would generate the config file using something like this:
os.open("file1.cfg.template").read() % myhash
where myhash is hash of configuration parameters from the main config file which may not contain CLIENT2 at all. In the case it does not contain CLIENT2, I want that line to disappear from the file. Is it possible to insert some 'IF' block in the file and have python evaluate it?
Thanks for your help. Any suggestions most welcome.
Sounds like you may have outgrown your originally simple home-grown templating solution. Maybe you should move to something like Jinja? It might be less of a headache to simply implement a third-party solution than it would be to create/continue to maintain your own solution.
Other options:
cheetah
mako
Maybe you can use a standalone Django template.
How do I use Django templates without the rest of Django? - Stack Overflow
Given that the files already exist, I would set default values for things like CLIENT2 (assuming you know ahead of time all possible keys). You can probably set the default value to something unusual so you can do
config = os.open("file1.cfg.template").read() % myhash
config = [l for l in config.split('\n') if <l does not have unusual text>].join('\n')
I agree with others that in the long term a more robust template would be better.