This question already has answers here:
How to save application settings in a config file?
(3 answers)
Closed 9 years ago.
I received a project from developer who left our company. Not too complex, but it doesn't look very nice.
So here is the question:
Application has some modules and one is "settings" which stores some app. options (not all possible options, lets say just two: foo and bar).
When application is started it reads options from command line (using argparse):
parser.add_argument('--foo', action='store_true')
parser.add_argument('--bar', action='store_true')
parser.add_argument('--baz', action='store_true')
And then it performs this nasty thing:
for name, val in parser.parse_args(sys.argv[1:])._get_kwargs():
setattr(sys.modules['settings'], name, val)
First: I think this is dirty, non-pythonic hack. And second, it is simply inconvenient to use such code, because when I need to use settings.baz, IDE complaints that it doesn't exist.
The intention of this hack is to make options parsed from command line available in all modules that are used in application further.
I'm thinking about something like singleton pattern, but I only used it once in PHP, and don't know if this correct solution in python. And even if it is, can someone show example?
I'm noob in python and on SO, please be kind to me :)
Thanks.
p.s. I'm sorry for possible mistakes in my English
Modules in Python are singleton objects, and using one to store the settings used by the other modules would be a very Pythonic
The second line of the "nasty thing" is just setting the attributes of a module named settings and so isn't that bad. What's worse is the _get_kwargs() part of the first line which is accessing a private attribute of the argparse.Namespace object returned by parser.parse_args() to get the names and values of the settings parsed from the command-line. A slightly better way to do it might be something like this:
import settings # possibly empty .py file
for name, val in vars(parser.parse_args(sys.argv[1:])).iteritems():
setattr(settings, name, val)
However this won't fix your IDE problems because the IDE doesn't know the name of settings added dynamically. A simple way to fix that would be to define all the possible attributes with some kind of default values in a settings.py module instead of having an empty one.
The first time a module is imported an entry for it is added to the sys.modules dictionary with its name as the key and an instance of types.ModuleType as a value. Subsequent imports will first check to see if an entry for it already exists and will skip reloading the file if it does -- which is why I claim they're basically singleton objects. Modifications made to its attributes will immediately be visible to other modules that have imported it or do so afterwards, so it's generally a good data sharing mechanism within an application.
Look this Config (A hierarchical, easy-to-use, powerful configuration module for Python )
Detailed doc & examples
Related
Say I create a simple web server using Flask, and allowing people to query certain things that I modulated in different python files, using the __import__ statement, would doing this with user supplied information be considered a security risk?
Example:
from flask import Flask
app = Flask(__name__)
#app.route("/<author>/<book>/<chapter>")
def index(author, book, chapter):
return getattr(__import__(author), book)(chapter)
# OR
return getattr(__import__("books." + author), book)(chapter)
I've seen a case like this recently when reviewing code, however it didn't feel right to me.
It is entirely insecure, and your system is wide open to attack. Your first return line doesn't limit what kind of names can be imported, which means the user can execute any arbitrary callable in any importable Python module.
That includes:
/pickle/loads/<url-encoded pickle data>
A pickle is a stack language that lets you execute arbitrary Python code, and the attacker can take full control of your server.
Even a prefixed __import__ would be insecure if an attacker can also place a file on your file system in the PYTHONPATH; all they need is a books directory earlier in the path. They can then use this route to have the file executed in your Flask process, again letting them take full control.
I would not use __import__ at all here. Just import those modules at the start and use a dictionary mapping author to the already imported module. You can use __import__ still to discover those modules on start-up, but you now remove the option to load arbitrary code from the filesystem.
Allowing untrusted data to direct calling arbitrary objects in modules should also be avoided (including getattr()). Again, an attacker that has limited access to the system could exploit this path to widen the crack considerably. Always limit the input to a whitelist of possible options (like the modules you loaded at the start, and per module, what objects can actually be called within).
More than being a security risk, it is a bad idea e.g. I could easily crash your web app by visiting the url:
/sys/exit/anything
translating to:
...
getattr(__import__('sys'), 'exit')('anything')
Don't give the possibility to import/execute just about anything to your users. Restrict the possibilities by using say a dictionary of permissible imports, as #MartijnPieters as clearly pointed out.
I am creating an application consisting of several modules. There is one main.py file which will be the file to run the application. The main.py file will load the configuration file(s) and put them in the 'config'-variable. It will also import the application-module-file (the file which holds the source-code of the application itself, a.k.a. application-class) and start the instance.
I am not very experienced in coding Python, and my biggest question is if I am doing it the right way, by using a main-file to handle all needed stuff (loading configuration-files for example). The problem I am having right now is that I cannot access the 'config'-variable that was defined in the main.py-file from any other module and/or Python-file.
Is it possible to make a global variable for configuration-values exc.? I know in PHP I used to create a singleton object which holds all the specific global arguments. I could also create a global 'ROOT'-variable to hold the full path to the root of the application, which is needed to load/import new files, this is also not possible in Python as far as I know.
I hope someone can help me out of this or send me in the right direction so I can continue working on this project.
The answer seems to be by Matthias:
Use from AppName.modules import settings and then access the data in the module with settings.value. According to PEP-8, the style guide for Python code, wildcard imports should be avoided and would in fact lead to undesirable behaviour in this case.
Thanks you all for the help!
I've been reading stackoverflow for years, and this is my first post. I've tried searching, but I can't really find anything that I both understand, and matches my scenario. Yes, I'm total OOP newbie, so please explain things as plainly as possible.
OK, I'm trying to write a Python script that calls Rsync to make a backup.
I'm essentially calling the script from root's crontab. Because this opens up security issues, I'm going to be reading in the directories that need to be backed up (and a few options for the rsync command for each directory) from a configuration file in a form that the ConfigParser module will understand.
So, I'm at a point where I want to create objects to represent backup directory. My question is thus:
Do I make a separate object factory class, and send it all of the relevant information that was gleaned while parsing the config file? Alternatively, do I put all the object creation stuff in a method in my existing configuration parsing class?
I hope at least some of that makes sense.
Please note this is both for production use and a learning project for me to learn Object Oriented programming and design, so yes, it is probably overkill to go OOP on this, but I want to learn this stuff!
Thanks for your advice and help!
Here's some of what I'm got so far (pseudo code):
import ConfigParser
import Logging
import os
class ParseConf(object):
[set up logging facilities]
def __init__(self,conffile = "/etc/pybackup/pyback.conf"):
self.conffile = conffile
self.confvals = {}
def getdirs():
create configparser instance
read config file
add values to dictionary self.confvals
def getdirobj():
either create a list of objects here or send self.confvals
to a factory object that returns a list of objects.
What is the most universal and best application configurations management method? I want to have these properties in order to have "good configuration management":
A list of all available properties and their default values in one
place.
A list of properties which can be changed by an app user, also in one
place.
When I retrieve a specific property, it's value is returned from the
2nd list (user changeable configs) or if it's not there, from the
first list.
So far, what I did was hard coding the 1st list as an object (more specific as a dict), wrote .conf file used by ConfigParser to make an app user to easily change some of the properties (2nd list), and wrote a public method on the config object to retrieve a property by it's name or if it's not there, raise an exception. In the end, one object was responsible for managing all the stuff (parsing file, raising exception, overriding properties etc.) But I was wondering, if there's a built-in library which does more or less the same thing, or even a better way to manage configuration, which takes into account all the KISS, DRY and other principles (I'm not always successful to do that with this method)?
Thanks in advance.
Create a default settings module which contains your desired default settings. Create a second module intended to be used by the the user with a from default_settings import * statement at the top, and instructing the user to write any replacements into this module instead.
Python is rather expressive, so in most cases, if you can expect the user to understand it on any level, you can use a Python module itself as the configuration file.
I am developing a project that requires a single configuration file whose data is used by multiple modules.
My question is: what is the common approach to that? should i read the configuration file from each
of my modules (files) or is there any other way to do it?
I was thinking to have a module named config.py that reads the configuration files and whenever I need a config I do import config and then do something like config.data['teamsdir'] get the 'teamsdir' property (for example).
response: opted for the conf.py approach then since it it is modular, flexible and simple
I can just put the configuration data directly in the file, latter if i want to read from a json file a xml file or multiple sources i just change the conf.py and make sure the data is accessed the same way.
accepted answer: chose "Alex Martelli" response because it was the most complete. voted up other answers because they where good and useful too.
I like the approach of a single config.py module whose body (when first imported) parses one or more configuration-data files and sets its own "global variables" appropriately -- though I'd favor config.teamdata over the round-about config.data['teamdata'] approach.
This assumes configuration settings are read-only once loaded (except maybe in unit-testing scenarios, where the test-code will be doing its own artificial setting of config variables to properly exercise the code-under-test) -- it basically exploits the nature of a module as the simplest Pythonic form of "singleton" (when you don't need subclassing or other features supported only by classes and not by modules, of course).
"One or more" configuration files (e.g. first one somewhere in /etc for general default settings, then one under /usr/local for site-specific overrides thereof, then again possibly one in the user's home directory for user specific settings) is a common and useful pattern.
The approach you describe is ok. If you want to add support for user config files, you can use execfile(os.path.expanduser("~/.yourprogram/config.py")).
One nice approach is to parse the config file(s) into a Python object when the application starts and pass this object around to all classes and modules requiring access to the configuration.
This may save a lot of time parsing the config.
If you want to share your config across different machines, you could perhaps put it on a web server and do import like this:
import urllib2
confstr = urllib2.urlopen("http://yourhost/config.py").read()
exec(confstr)
And if you want to share it across different languages, perhaps you can use JSON to encode and parse the configuration:
import urllib2
import simplejson
confstr = urllib2.urlopen("http://yourhost/config.py").read()
config = simplejson.loads(confstr)