I am trying to use python for translating a set of templates to a set of configuration files based on values taken from a main configuration file. However, I am having certain issues. Consider the following example of a template file.
file1.cfg.template
%(CLIENT1)s %(HOST1)s %(PORT1)d C %(COMPID1)s
%(CLIENT2)s %(HOST2)s %(PORT2)d C %(COMPID2)s
This file contains an entry for each client. There are hundreds of config files like this and I don't want to have logic for each type of config file. Python should do the replacements and generate config files automatically given a set of global values read from a main xml config file. However, in the above example, if CLIENT2 does not exist, how do I delete that line? I expect Python would generate the config file using something like this:
os.open("file1.cfg.template").read() % myhash
where myhash is hash of configuration parameters from the main config file which may not contain CLIENT2 at all. In the case it does not contain CLIENT2, I want that line to disappear from the file. Is it possible to insert some 'IF' block in the file and have python evaluate it?
Thanks for your help. Any suggestions most welcome.
Sounds like you may have outgrown your originally simple home-grown templating solution. Maybe you should move to something like Jinja? It might be less of a headache to simply implement a third-party solution than it would be to create/continue to maintain your own solution.
Other options:
cheetah
mako
Maybe you can use a standalone Django template.
How do I use Django templates without the rest of Django? - Stack Overflow
Given that the files already exist, I would set default values for things like CLIENT2 (assuming you know ahead of time all possible keys). You can probably set the default value to something unusual so you can do
config = os.open("file1.cfg.template").read() % myhash
config = [l for l in config.split('\n') if <l does not have unusual text>].join('\n')
I agree with others that in the long term a more robust template would be better.
Related
I am creating a piece of code that will have multiple files that need to reference a single config. The reason is that our IT department will use puppet to manage this config file in case any changes are required in the future. We don't want to do a release to change the configuration. I've seen a few projects that have multiple configs in different places but I really do not like this idea and I'd prefer to have a single source. My thought was to create a specific config.py file that can be called anywhere in my code that will ask for the user to input the location of the file.
Is this a good way or is there a better way to do this?
import configparser
class Config(object):
def __init__(self,conf):
self._cfg = configparser.ConfigParser()
self._cfg.read(conf)
def get_conf_value(self,property):
if property not in self._cfg.sections:
return None
return self._cfg.sections[property]
If so, if I have a Main.py while, what's the best way to have the scheduler pass the config location and then reference it across all of my files in my Python Package?
You can create a config.json file in json format. You can read the contents at startup using the json.load function from the json library.
I'm using absl-py's python gflags for my project.
They're a convenient way of specifying configurations, but I frequently like to run the same configuration, and saving command line invocations is not preferable because they are hard to read and maintain.
I'd like to instead be able to maintain YAML text files that specify flag values. For instance, a program that might be invoked as
python myapp.py --some_int_arg 3 --noboolean_value
could also be invoked as
python myapp.py --from_config config.yaml
where the contents of config.yaml are
some_int_arg: 3
boolean_value: false
and analogously it'd be nice to be able to construct the file above given the first invocation.
What is the most idiomatic way of
accepting a configuration file path on the command line and having that override all absl.flags.FLAGS values?
converting an absl.flags.FLAGS instance to a text representation (preferably YAML) that can then be reinvoked using (1)?
Edit. There seems to be an easy way to accept a configuration file, using --flagfile, which should just contain each flag's values on a new line. However, my question (2) still stands, how would I be able to recover such a file? Currently the only solution seems to be to crawl flags.FLAGS and stringify each flag value manually.
FLAGS.flags_into_string() should return a string that can be put in a flagfile. Or FLAGS.append_flags_into_file('path/to/flagfile.txt') will append the flags to that file.
Does this work for you?
I have some input data that is user configurable, so i do not want to hard code it. Like the data path, result path etc.
Can you please suggest the best way to handle this data? Should i keep them in an excel or notepad and then read at run time? Or is there a better way to handle it?
Thanks
There are a lot of ways to do it.
Configuration file
You can store configuration in separate file in YAML, JSON, INI or any other format. There are a lot of tools and libraries for parsing and loading such configurations. Take a look on this article. Such approach is good for rarely changed configuration like services credentials, but it's not really good for configuration that changes very often.
Environment variables
Also, you can store configuration inside environment variables. Take a look on py-env-config. You can hard-code default configuration values but allow user to override them using environment variables.
Script arguments
If you are writing a script, you can always pass all configuration as command-line arguments/options. Manuals. Such approach is good of configs that changes very often (almost every script execution).
EDIT
I'll suggest you to use configuration file for this constants.
I am looking for the method Django uses to generate unique filename when we upload a file.
For example, if I upload a file called test.csv twice in the same directory, the first one will be saved as test.csv and the second file will be saved as test_2.csv. I already tried to find how Django manages that, but I only found django.utils.text.get_valid_filename which could be useful, but that not what I am looking for...
I already saw other topics with random naming solution, that not what I am looking for here :) I really trying to understand how Django manage that problem.
I actually take a closer look with your help and a found something :)
So basically I have to do something like:
from django.core.files.storage import FileSystemStorage
fss = FileSystemStorage()
filepath = fss.get_available_name(filepath)
Thank you all :)
PS: If you are interesting, the comment from django.core.file.storage.FileSystemStorage._save says:
There's a potential race condition between get_available_name and
saving the file; it's possible that two threads might return the
same name, at which point all sorts of fun happens. So we need to
try to create the file, but if it already exists we have to go back
to get_available_name() and try again.
If you see the implementation of class django.core.files.storage.Storage you will know how Django 1.6 manages the file names.
Look into the save method of this class. In this the line
name = self.get_available_name(name)
is doing the trick.
This is the default implementation of getting the new file name before saving the file. If you want to write your own version (like the file should be overridden) then consider writing your own custom storage system
Actually, you were on the right track.
From the docs,
Internally, Django uses a django.core.files.File instance any time it
needs to represent a file.
And also,
Behind the scenes, Django delegates decisions about how and where to
store files to a file storage system
Which means that, when the file is uploaded, using the default storage (FileSystemStorage), Django delegates the naming (or the available name), behind the scene, for the file to the storage, which then uses: get_available_name(name).
So, If you want to change the way files are named when uploaded, you need to add a custom file storage, which would basically only override get_available_name. The documentation on the matter is here.
I want to use Python to make a configuration file generator. My roughly idea is feeding input with template files and some XML files with the real settings. Then use the program to generate the real configuration files.
Example:
[template file]
server_IP = %serverip%
server_name = %servername%
[XML file]
<serverinfo>
<server ip="x.x.x.x" name="host1" />
<server ip="x.x.x.x" name="host2" />
</serverinfo>
and then get output configuration file like this
[server.ini]
[server1]
server_IP = x.x.x.x
server_name = host1
[server2]
server_IP = x.x.x.x
server_name = host2
I got several questions:
Is there any open source configuration generator program? (what could be the keyword), I wonder if there's anything can be added/modified in the design.
Does Python have good XML parser module?
Is it good idea to use XML file to save the original settings? I've been thinking to use Excel since it's more intuitive to maintain, but harder for program to parse. Not sure how people deal with this.
Hope the community can give me some suggestions. Thanks a lot!
EDIT:
In scenario that there are dozens of these output ini files. I am concerning 2 things.
there are dozens of ip/hostname and related informations, that may requires to be managed by human, so XML format would be a bit inconvenient. What could be the most convenient structure to manage those information? (Excel would be a handy tool to batch modify and look up info)
In case of need to add some extra line into ini files, I need a efficient way by just modify the template file and add extra info into the source file (may be the Excel file or whatever), then whole bunches of ini files can be generated quickly.
I recommend using excellent ConfigObj library by Michael Foord. It can read/write configuration files, even with nested sections.
I don't know if there are any open source configuration generators.
Python has several xml parser modules, the newest (and perhaps most pythonic) of which is ElementTree. You can find additional documentation for it at the developer's site.
I recommend avoiding xml in your configuration files if possible. A simple flat file full of name/value pairs is much easier for humans to work with. If you need a level or two of structure, ini-style files ought to do nicely, and python comes with a built-in parser for them.
It's terrific to use xml for any configuration file. Best choice in any interpreted language is to use code file and simply exec or import it. Pickle is also good, but not human readable.
I've used the minidom module. That would probably work for you.
You need some template engine, look at string.Template