I designed a configuration mechanism in Python, where certain objects can operate in special ways to define problems in our domain.
The user specifies the problem by using this objects in a "config-file" manner. For instance:
# run configuration
CASES = [
('Case 1', Item('item1') + Item('item2') + Item('item3')),
('Case 2', Item('item1') + Item('item4')),
]
DATA = {
'Case 1' = {'Piece 1': 'path 1'},
'Case 2' = {'Piece 1': 'path 2'},
}
The Item objects are, of course, defined in a specific module. In order to use them you have to issue an import statement: from models import Item (of course, my actual imports are more complex, not a single one).
I would like the user to simply write the configuration presented, without having to import anything (users very easily can forget this).
I thought of reading the file as text, and creating a secondary text file with all the appropriate imports at the top, write that to a file, and import that file, but this seems clumsy.
Any advice?
Edit:
The workflow of my system is somewhat similar to Django, in that the user defines the "Settings" in a python file, and runs a script which imports that Settings file and does things with it. That is where I would like this functionality, to tell Python "given this namespace (where Item means something in particular), the user will provide a script - execute it and hand me the result so that I can spawn the different runs".
From the eval help:
>>> help(eval)
Help on built-in function eval in module __builtin__:
eval(...)
eval(source[, globals[, locals]]) -> value
Evaluate the source in the context of globals and locals.
The source may be a string representing a Python expression
or a code object as returned by compile().
The globals must be a dictionary and locals can be any mapping,
defaulting to the current globals and locals.
If only globals is given, locals defaults to it.
That is, you can pass in an arbitrary dictionary to use as the namespace for an eval call.
with open(source) as f:
eval(f.read, globals(), {'Item': Item})
Why have you decided that the user needs to write their configuration file in pure Python? There are many simple human-writable languages you could use instead. Have a look at ConfigParser, for instance, which reads basic configuration files of the sort Windows uses.
[cases]
case 1: item1 + item2 + item3
case 2: item1 + item4
[data]
case 1: piece1 - path1
case 2: piece1 - path2
1) the first thing that i have in mind is to offer to the user the generation of your config file; how so ?
you can add an argument in the script that launch your application :
$ python application_run.py --generate_settings
this will generate a config file with a skeleton of different import that the user should not have to add every time, something like this:
import sys
from models import Item
# Complete the information here please !!!
CASES = []
DATA = {}
2) a second way is to use execfile() , you can for this create a script that will read the settings.py:
root_settings.py
# All import defined Here.
from Model import Item
...
execfile('settings.py')
And now to read the settings file info just import the root_settings, by the way all variable that have been defined in settings.py are now in root_settings.py namespace .
Related
I am in the habit of using raw_input(...) for certain debugging. However, in python3 this has changed to input(...). Is there a way to define an alias at the top of my project, such as:
# __init__.py
raw_input = input
I tried the above, but it only worked in the file I added it to, and not any other files in that directory. I'd like this to work basically in every file within my python repository.
You can define all aliases in a separate file (e.g. aliases.py) then import said file where needed (i.e. import aliases).
The con with this method that you'll be referencing the alias through aliases.alias unless you make the import stricter (i.e. from aliases import raw_input) or if you don't care about avoiding a wildcard import (i.e. from aliases import *).
Additionally, if you don't mind another import in the aliases file you can use the builtins namespace:
import builtins
builtins.raw_input = input
You still have to define all aliases separate file (e.g. aliases.py) then import said file where needed (i.e. import aliases) but the advantage of using the builtins namespace is that you can use that import exactly as given.
You can do it by creating a module for creating the renaming function and then importing it to every file you want to like this:
First the module function declaration in alias.py
def raw_input(a):
return input(a)
Secondly, import to another file:
from alias import raw_input
x = raw_input("hello world")
print(x)
Sadly, you will have to make the import of the module to every file you want to use the renamed function.
Hope it works for you!
Put this at the top, and you will get exactly what you want.
import builtins
builtins.raw_input = builtins.input
It is guaranteed to work, but generally considered a bad practice (everybody will be confused with where is that raw_input defined)
I have a small project where I need to initially introduce and use some quite large number of variables.
Obviously, I can make some configure file, where I set up all the values of the variables. I have made just some Python file, where I give values:
value_a = 'something'
value_b = 'something'
value_c = 5.0
and call the file conf.py. When I do from conf import *, I have all the variables with values initialized.
Nevertheless, I have different modules in project with different subroutines (methods in it) and I want to have all those values from conf.py known in every method and in every module.
Obviously, I can do from conf import * in every module and/or import conf in every subroutine, but is it the best way how to implement the initialization of variables?
Using a module as you describe is a viable way to setup configuration values for a script, but there are a few reasons you might be better off with something else.
A few others in the comments have pointed out that import * is frowned upon because it clutters up the root namespace with lots of variable names making it much easier to accidentally have name conflicts. Keeping them under the module name (ex: conf.varname) helps from an organizational standpoint in keeping track of names and preventing conflict.
If you plan to distribute code that requires configuration, using a .py module opens up your code to arbitrary code execution of anything that gets typed in that file. This is where things like ".ini .json .cfg etc" files are very useful. As an added bonus by using a common format (like json) it makes the configuration easy to port to other languages if a colleague is using a different language but needs to work on the same project. Off the top of my head, python includes libraries for .xml .json and .ini files.
In my opinion a solution that I would implement for that issue is to create a conf.py, as you said, and then define in it global variables as dictionary structures in order to be properly organized and easy-to-use on the modules that will be imported.
For example:
globals = {'value_a': 'something a',
'value_b': 'something b',
'value_c': '5.0',
'allowed_platforms': {
'windows': 'value 1',
'os x': 'value 2',
'linux': 'value 3'
},
'hosts': ['host a', 'host b', 'host c'],
...
}
You need to avoid the from some_module import * statement because you could put a lot of imports into the namespace and because it's not explicit about what is importing. So doing at the top of each module from your_package.conf import globals you could use it without the need of importing explicitly every single variable that you want to use or without importing the entire module. I prefer that solution, and also could be better if you use json files to store the info of that global variables and then read and serialize them in the conf.py module before being imported in your required modules.
I generally agree with ##Aaron. What he outlined is very general / portable and safe.
Since import * is an antipattern, you could easily do import config and then reference its values like config.varname.
I think it's fine to use .py files when needed. Aaron's point is good, but as long as the config is controlled by the person running the app, there's no security issue. The main reason to allow .py files is when some of the config items need to be derived from other config items, or looked up / loaded at run time. If there's no need for that (config is 100% flat and static) then .json or another flat file approach as Aaron mentioned would be best.
In my program I have a package filled with various .py files each containing a class definition. I want to make a list where each entry is an instance of one of those classes. In addition, my program doesn't know how many files are in the package or what the files or classes are called, so I can't just import each file. Ideally, I should be able to modify the contents of the package (take out files, put new ones in, etc.) without having to rewrite other parts of the program. Is there a way to do this?
Originally, I had a 'if __name__ == '__main__': return foo()' line in each file and tried to append to the list using execfile(), but obviously this doesn't work. Any ideas?
Sorry if this is kinda vague. I'll try to clarify if needed. I'm using Python 2.5.4.
EDIT:
My program is a random character generator for Dungeons and Dragons. I made a package for every major data type the program needs. I have a package for Classes, Races, Items, etc. and when making a character, my program makes a list of each data type that it can sort through when making a character. For example, when equipping a character, the program can look at the Weapon list and filter out all the weapons that are unsuitable for that character and then randomly choose from the ones that remain.
I don't want to specify file names because I would like the ability to easily add to this program later. If later on down the road I wanted to add more weapon types to the program, I could just write a few new class descriptions and drop them in the Weapons package, and the program could use them without me needing to edit any other code.
This sounds like a bit of a bad design. It would probably be better if you elaborate on the problem and we can help you to solve it some other way. However, what you want isn't hard:
import types
import my_package
my_package_members = [getattr(my_package, i) for i in dir(my_package)]
my_modules = [i for i in my_package_members if type(i) == types.ModuleType]
instances = []
for my_module in my_modules:
my_module_members = [getattr(my_module, i) for i in dir(my_module)]
my_classes = [i for i in my_module_members
if type(i) in (types.TypeType, types.ClassType)]
for my_class in my_classes:
instances.append(my_class())
EDIT: Simplified the code a bit.
To acheive this you are going to need to do the following things:
Have your code enumerate the source files containing your code.
For each source file, import the code specified in the file into a new module.
For each module, locate all the classes contained, instantiate each one and add it to your final list.
To take each part in turn:
To enumerate the source files, use os.walk and os.path to find the files and build full paths to the source.
To import code from a given source file dynamically, you can do execfile(my_file) in my_dict where my_file is the full path to your source file and my_dict is a dictionary to return the resulting code in (any classes declared in the source file would become members of this dict for example). Note you only need to use this method if the files you are importing are not part of a valid python module/package hierarchy (with an init.py file in the package) - if they are you can use import() instead.
To enumerate the classes declared in a given module you could use inspect.getmembers().
If you're willing to do a bit more work, you can use pkg_resource's entry points to advertise and discover the relevant classes. The Fedora Account System uses this to provide plugin functionality.
Assuming, first, that all your modules exist as .py files in the package's directory:
import inspect, glob, os, sys
def thelistyouwant(pathtothepackage):
sys.path.insert(0, pathtothepackage)
result = []
for fn in glob.glob(os.path.join(pathtothepackage, '*.py')):
if fn.startswith('_'): continue # no __init__ or other private modules
m = __import__(fn[:-3])
classes = inspect.getmembers(m, inspect.isclass)
if len(classes) != 1:
print>>sys.stderr, "Skipping %s (%d != 1 classes!)" % (fn, len(classes))
continue
n, c = classes[0]
try:
result.append(c())
except TypeError:
print>>sys.stderr, "Skipping %s, can't build a %s()" % (fn, n)
del sys.path[0]
return result
Further assumptions: each module should have exactly 1 class (otherwise it's skipped with a warning) instantiable without arguments (ditto ditto); you don't want to look at __init__.py (if any; actually this code does not require the path to be an actual package, any directory will do, so __init__.py may or may not be present) nor any module whose name starts with an underscore ("private" modules of the package).
I want to replace settings.py in my Django system with a module implemented as a directory settings containing __init__.py. This will try to import a module named after the server, thus allowing for per-server settings.
If I don't know the name of a module before I import it then I can't use the import keyword but must instead use the __import__ function. But this does not add the contents of the module to the settings module. I need the equivalent of from MACHINE_NAME import *. Or I need a way to iterate over vars(m) (where m is the loaded module) and add them to the current namespace. But I can't work out how to refer to the current namespace in order to make the assignment. In other words, I can't use setattr(x, ..) or modify x.__dict__, because I don't know what to use for x.
I can't think of much else to try now apart from using exec. This seems a little feeble to me. Am I missing some aspect of Pythonic introspection that would allow me to manipulate the current scope while still in it?
For similar situation where based on lang setting I import different messages in messages.py module it is something like
# set values in current namespace
for name in vars(messages):
v = getattr(messages, name)
globals()[name] = v
Btw why do you want to create a package for settings.py? whatever you want to do can be done in settings.py directly?
I would like to load a .py file at runtime. This .py file is basically a config file with the following format:
var1=value
var2=value
predicate_function=func line : <return true or false>
Once this file is loaded, I would like to be able to access var1, var2 and predicate_function. For each line, I'll pass it to the predicate function, and if it returns false, I'll ignore it.
In any case, I'm not sure how to load a python file at runtime and access its variables.
Clarification: there may be any number of these config files that I need to pass to the main program and I won't know their names until runtime. Google tells me I should use __import__. I'm not sure how to correctly use that method and then access the variables of the imported file.
As written in the python official documentation, if you just want to import a module by name, you can look it up in the sys.modules dictionary after using __import__.
Supposing your configuration is in myproject.mymodule, you would do like that :
module_name = 'myproject.mymodule'
import sys
__import__(module_name)
mymodule = sys.modules[module_name]
# Then you can just access your variables and functions
print mymodule.var1
print mymodule.var2
# etc...
You can also use the return value of __import__ statement but you will have to understand fully how python works with namespaces and scopes.
You just need to be able to dynamically specify the imports and then dynamically get at the variables.
Let's say your config file is bar.py and looks like this:
x = 3
y = 4
def f(x): return (x<4)
Then your code should look like this:
import sys
# somehow modnames should be a list of strings that are the names of config files
#
# you can do this more dynamically depending on what you're doing
modnames = ['bar']
for modname in modnames:
exec('import %s' % modname)
for modname in modnames:
mod = sys.modules[modname]
for k in mod.__dict__:
if k[:2] != '__':
print modname, k, mod.__dict__[k]
I get this output:
bar f <function f at 0x7f2354eb4cf8>
bar x 3
bar y 4
Then you at least have all the variables and functions. I didn't quite get what you wanted from the predicate functions, but maybe you can get that on your own now.
To access another Python module, you import it. execfile has been mentioned by a couple people, but it is messy and dangerous. execfile clutters your namespace, possibly even messing up the code you are running. When you want to access another Python source file, use the import statement.
Even better would be not to use a Python file for configuration at all, but rather to use the builtin module ConfigParser or a serialization format like JSON. This way your configuration files don't allow execution of arbitrary (possibly malicious) code, doesn't require people to know Python to configure your program, and can easily be altered programatically.
If the imported module is on the regular search path, you can use __import__.
If you need to load the module from an arbitrary path in the filesystem, use imp.load_module.
Be sure to consider the security implications of loading arbitrary user-specified code.
In Python 2.*, execfile works (I recommend passing a specific dictionary and accessing the variables from there -- as the note in the docs says, execfile can't affect the calling function's locals() dictionary).
In Python 3.*, execfile has been removed, so do, instead:
with open('thefile.py') as f:
exec(f.read(), somedict)
Since the Python version hasn't been clearly mentioned, it is worth pointing out that the imp module has been deprecated in newer Python versions in favor of the importlib module. Example here.
I'm kinda late to the party, but I want to present an alternative answer nonetheless.
If you want to import code without affecting the global module namespace, you can create an anonymous module (using types.ModuleType) and load arbitrary code in it (using compile and exec). For instance, like this:
import types
filename = "/path/to/your/file.py"
with open(filename) as fp:
code = compile(fp.read(), filename, "exec")
config_module = types.ModuleType("<config>")
exec code in config_module.__dict__
You can then access the variables as config_module.var1, &c.
If you want to have a configuration file that will only be edited by the user when the program isn't running, just import it as a normal python file
ie.
main.py:
import config
print config.var1
config.py:
var="var12"
var2 = 100.5
try the imp module : http://docs.python.org/library/imp.html