I have a system, where you can modify, which modules will be loaded (and run; "module" is not necessarily python module, it can combine several modules). The program can run module A and B. Now, I want to have an option that every module can define (add) its own parameters. Let's say A wants to have -n and B wants to have -s for something. But there is one common parameter -c, which the main system itself needs. What is the best way to achieve this?
So far I have been using single optparse.OptionParser instance and passed it to every module, when they are initialized. Then the module can modify (add a new parameter) if needed.
You should consider moving to a library that supports the concept of sub-parsers, such as argparse (which deprecates optparse anyway), so that each library can create its own parser rules, and the main program can just combine them.
When I had this problem I ended up using a class derived from ArgumentParser that added the ability to register callback functions that would be executed once the arguments were parsed:
import argparse
class ArgumentParser(argparse.ArgumentParser):
def __init__(self, *p, **kw):
super(ArgumentParser, self).__init__(*p, **kw)
self._reactions = []
def add_reaction(self, handler):
self._reactions.append(handler)
def parse_known_args(self, args=None, namespace=None):
(args, argv) = super(ArgumentParser, self).parse_known_args(args, namespace)
for reaction in self._reactions:
reaction(args)
return (args, argv)
This way the parser object still needs to be passed to all the modules to register their command line switches, but the modules can react to the switches "on their own":
def arguments_parsed(args):
if args.datafile:
load_stuff(args.datafile)
def add_arguments(ap):
ap.add_argument('--datafile',
help="Load additional input data")
ap.add_reaction(arguments_parsed)
This uses argparse, but the same could probably be done with optparse.
It is not tested with advanced features like subparsers and probably won't work there, but could easily be extended to do so.
Related
For a project involving multiple scripts (data processing, model tuning, model training, testing, etc.) I may keep all my <class 'argparse.ArgumentParser'> objects as the return value of functions for each script task in a module I name cli.py. However, in the calling script itself (__main__), after calling args = parser.parse_args(), if I type args. in VS Code I get no suggested attributes that are specific to that object. How can I get suggested attributes of argparse.NameSpace objects in VS Code?
E.g.,
"""Module for command line interface for different scripts."""
import argparse
def some_task_parser(description: str) -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=description)
parser.add_argument('some_arg')
return parser
then
"""Script for executing some task."""
from cli import some_task_parser
if __name__ == '__main__':
parser = some_task_parser('arbitrary task')
args = parser.parse_args()
print(args.) # I WANT SUGGESTED ATTRIBUTES (SUCH AS args.some_arg) TO POP UP!!
I don't know about VSCode, in an ipython session, the tab complete does show the available attributes of Namespace. argparse.Namespace is a relatively simple object class. What you want, I think, are just the attributes of this object.
In [238]: args = argparse.Namespace(test='one', foo='three', bar=3)
In [239]: args
Out[239]: Namespace(bar=3, foo='three', test='one')
In [240]: args.
bar test
foo args.txt
But those attributes will not be available during development. They are put there by the parsing action, at run time. They cannot be reliably deduced from the parser setup. Certainly argparse does not provide such a help.
I often recommend that developers include a
print(args)
during the debugging phase, so they get a clear idea of what the parser does.
Use args.__dict__ to get the names and values. If you need just the argument names, then [key for key in args.__dict__.keys()] will give the list of them.
(A simplified form of the problem.) I'm writing an API involving some Python components. These might be functions, but for concreteness let's say they're objects. I want to be able to parse options for the various components from the command line.
from argparse import ArgumentParser
class Foo(object):
def __init__(self, foo_options):
"""do stuff with options"""
"""..."""
class Bar(object):
def __init__(sef, bar_options):
"""..."""
def foo_parser():
"""(could also be a Foo method)"""
p = ArgumentParser()
p.add_argument('--option1')
#...
return p
def bar_parser(): "..."
But now I want to be able to build larger components:
def larger_component(options):
f1 = Foo(options.foo1)
f2 = Foo(options.foo2)
b = Bar(options.bar)
# ... do stuff with these pieces
Fine. But how to write the appropriate parser? We might wish for something like this:
def larger_parser(): # probably need to take some prefix/ns arguments
# general options to be overridden by p1, p2
# (this could be done automagically or by hand in `larger_component`):
p = foo_parser(prefix=None, namespace='foo')
p1 = foo_parser(prefix='first-foo-', namespace='foo1')
p2 = foo_parser(prefix='second-foo-', namespace='foo2')
b = bar_parser()
# (you wouldn't actually specify the prefix/namespace twice: )
return combine_parsers([(p1, namespace='foo1', prefix='first-foo-'),
(p2,...),p,b])
larger_component(larger_parser().parse_args())
# CLI should accept --foo1-option1, --foo2-option1, --option1 (*)
which looks a bit like argparse's parents feature if you forget that we want prefixing (so as to be able to add multiple parsers of the same type)
and probably namespacing (so that we can build tree-structured namespaces to reflect the structure of the components).
Of course, we want larger_component and larger_parser to be composable in the same way, and the namespace object passed to a certain component should always have the same internal shape/naming structure.
The trouble seems to be that the argparse API is basically about mutating your parsers, but querying them is more difficult - if you turned a
datatype into a parser directly, you could just walk these objects. I managed to hack something that somewhat works if the user writes a bunch of functions to add arguments to parsers by hand, but each add_argument call must then take a prefix, and the whole thing becomes quite inscrutable and probably non-composable. (You could abstract over this at the cost of duplicating some parts of the internal data structures ...). I also tried to subclass the parser and group objects ...
You could imagine this might be possible using a more algebraic CLI-parsing API, but I don't think rewriting argparse is a good solution here.
Is there a known/straightforward way to do this?
Some thoughts that may help you construct the larger parser:
parser = argparse.ArgumentParser(...)
arg1 = parser.add_argument('--foo',...)
Now arg1 is a reference to the Action object created by add_argument. I'd suggest doing this in an interactive shell and looking at its attributes. Or at least print its repr. You can also experiment with modifying attributes.
Most of what a parser 'knows' about the arguments is contained in these actions. In a sense a parser is an object that 'contains' a bunch of 'actions'.
Look also at:
parser._actions
This is the parser's master list of actions, which will include the default help as well as the ones you add.
The parents mechanism copies Action references from the parent to the child. Note, it does not make copies of the Action objects. It also recreates argument groups - but these groups only serve to group help lines. They have nothing to do with parsing.
args1, extras = parser.parse_known_args(argv, namespace)
is very useful when dealing with multiple parsers. With it, each parser can handle the arguments it knows about, and pass the rest on to others. Try to understand the inputs and outputs to that method.
We have talked about composite Namespace objects in earlier SO questions. The default argparse.Namespace class is a simple object class with a repr method. The parser just uses hasattr, getattr and setattr, trying to be as non-specific as it can. You could construct a more elaborate namespace class.
argparse subcommands with nested namespaces
You can also customize the Action classes. That's where most values are inserted into the Namespace (though defaults are set elsewhere).
IPython uses argparse, both for the main call, and internally for magic commands. It constructs many arguments from config files. Thus it is possible to set many values either with default configs, custom configs, or at the last moment via the commandline arguments.
You might be able to use the concept of composing actions to achieve the functionality that you need. You can build actions that modify the namespace, dest, etc as you need and then compose them with:
def compose_actions(*actions):
"""Compose many argparse actions into one callable action.
Args:
*actions: The actions to compose.
Returns:
argparse.Action: Composed action.
"""
class ComposableAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
for action in actions:
action(option_string, self.dest).__call__(parser,
namespace,
values,
option_string)
return ComposableAction
See example: https://gist.github.com/mnm364/edee068a5cebbfac43547b57b7c842f1
I am building a very basic platform in the form of a Python 2.7 module. This module has a read-eval-print loop where entered user commands are mapped to function calls. Since I am trying to make it easy to build plugin modules for my platform, the function calls will be from my Main module to an arbitrary plugin module. I'd like a plugin builder to be able to specify the command that he wants to trigger his function, so I've been looking for a Pythonic way to remotely enter a mapping in the command->function dict in the Main module from the plugin module.
I've looked at several things:
Method name parsing: the Main module would import the plugin module
and scan it for method names that match a certain format. For
example, it might add the download_file_command(file) method to its
dict as "download file" -> download_file_command. However, getting a
concise, easy-to-type command name (say, "dl") requires that the
function's name also be short, which isn't good for code
readability. It also requires the plugin developer to conform to a
precise naming format.
Cross-module decorators: decorators would let
the plugin developer name his function whatever he wants and simply
add something like #Main.register("dl"), but they would necessarily
require that I both modify another module's namespace and keep
global state in the Main module. I understand this is very bad.
Same-module decorators: using the same logic as above, I could add a
decorator that adds the function's name to some command name->function mapping local to the
plugin module and retrieve the mapping to the Main module with an
API call. This requires that certain methods always be present or
inherited though, and - if my understanding of decorators is correct - the function will only register itself the first time it is run and will unnecessarily re-register itself every subsequent time
thereafter.
Thus, what I really need is a Pythonic way to annotate a function with the command name that should trigger it, and that way can't be the function's name. I need to be able to extract the command name->function mapping when I import the module, and any less work on the plugin developer's side is a big plus.
Thanks for the help, and my apologies if there are any flaws in my Python understanding; I'm relatively new to the language.
Building or Standing on the first part of #ericstalbot's answer, you might find it convenient to use a decorator like the following.
################################################################################
import functools
def register(command_name):
def wrapped(fn):
#functools.wraps(fn)
def wrapped_f(*args, **kwargs):
return fn(*args, **kwargs)
wrapped_f.__doc__ += "(command=%s)" % command_name
wrapped_f.command_name = command_name
return wrapped_f
return wrapped
################################################################################
#register('cp')
def copy_all_the_files(*args, **kwargs):
"""Copy many files."""
print "copy_all_the_files:", args, kwargs
################################################################################
print "Command Name: ", copy_all_the_files.command_name
print "Docstring : ", copy_all_the_files.__doc__
copy_all_the_files("a", "b", keep=True)
Output when run:
Command Name: cp
Docstring : Copy many files.(command=cp)
copy_all_the_files: ('a', 'b') {'keep': True}
User-defined functions can have arbitrary attributes. So you could specify that plug-in functions have an attribute with a certain name. For example:
def a():
return 1
a.command_name = 'get_one'
Then, in your module you could build a mapping like this:
import inspect #from standard library
import plugin
mapping = {}
for v in plugin.__dict__.itervalues():
if inspect.isfunction(v) and v.hasattr('command_name'):
mapping[v.command_name] = v
To read about arbitrary attributes for user-defined functions see the docs
There are two parts in a plugin system:
Discover plugins
Trigger some code execution in a plugin
The proposed solutions in your question address only the second part.
There many ways to implement both depending on your requirements e.g., to enable plugins, they could be specified in a configuration file for your application:
plugins = some_package.plugin_for_your_app
another_plugin_module
# ...
To implement loading of the plugin modules:
plugins = [importlib.import_module(name) for name in config.get("plugins")]
To get a dictionary: command name -> function:
commands = {name: func
for plugin in plugins
for name, func in plugin.get_commands().items()}
Plugin author can use any method to implement get_commands() e.g., using prefixes or decorators — your main application shouldn't care as long as get_commands() returns the command dictionary for each plugin.
For example, some_plugin.py (full source):
def f(a, b):
return a + b
def get_commands():
return {"add": f, "multiply": lambda x,y: x*y}
It defines two commands add, multiply.
I manage a fairly large python-based quantum chemistry suite, PyQuante. I'm currently struggling with how to set various defaults so that users can choose among different options at runtime.
For example, I have three different methods for computing electron repulsion integrals. Let's call them a,b,c. I used to simply pick the one I liked best (say, c), and have that hard-wired into the module that computes these integrals.
I have now modified this to use a module, Defaults.py, that contains all such hard-wires. But this is set at compile/install time. I would now like users to be able to override these options at runtime, say, using a .pyquanterc.py file.
In my integral routines, I currently have something like
from Defaults import integral_method
I know about dictionaries, and the .update() method. But I don't know how I would use this in real life. My defaults module looks like
integral_method = c
should I modify the end of Defaults.py to look for a .pythonrc.py file and override these values? E.g.
if os.path.exists('$HOME/.pythonrc.py'): do_something
If so, what should do_something look like?
With your current setup, the user can change the default functions in his scripts quite easily:
import Defaults
Defaults.integral_method = somefunc
If the user adds this to his script, all your modules that use integral_method from Defaults will use somefunc to calculate integrals.
I might do this via a factory class.
class IntegralSolver:
"""
Factory class containing methods for solving integrals.
>>> solver = IntegralSolver("method1")
>>> solver(x)
# solution via method1
Can also be used directly:
>>> IntegralSolver.method2(x)
# solution via method2
"""
def __init__(self, method):
self.__call__ = getattr(self, method)
#staticmethod
def method1(x):
return method1_solution
#staticmethod
def method2(x):
return method2_solution
It really depends on how your user runs the toolset. If they twiddle the python code each time, just setting a block at the top labeled OPTIONS should be good. If they run it off the command line, use the argparse library to allow them to switch options on the command line. Perhaps have it read the options out of a file with configParser to read a default file with your options, and if the user sets it, an additional file with their options.
My Python script (for todo lists) is started from the command line like this:
todo [options] <command> [command-options]
Some options can not be used together, for example
todo add --pos=3 --end "Ask Stackoverflow"
would specify both the third position and the end of the list. Likewise
todo list --brief --informative
would confuse my program about being brief or informative. Since I want to have quite a powerful option control, cases like these will be a bunch, and new ones will surely arise in the future. If a users passes a bad combination of options, I want to give an informative message, preferably along with the usage help provided by optparse. Currently I handle this with an if-else statement that I find really ugly and poor. My dream is to have something like this in my code:
parser.set_not_allowed(combination=["--pos", "--end"],
message="--pos and --end can not be used together")
and the OptionParser would use this when parsing the options.
Since this doesn't exist as far as I know, I ask the SO community:
How do you handle this?
Possibly by extending optparse.OptionParser:
class Conflict(object):
__slots__ = ("combination", "message", "parser")
def __init__(self, combination, message, parser):
self.combination = combination
self.message = str(message)
self.parser = parser
def accepts(self, options):
count = sum(1 for option in self.combination if hasattr(options, option))
return count <= 1
class ConflictError(Exception):
def __init__(self, conflict):
self.conflict = conflict
def __str__(self):
return self.conflict.message
class MyOptionParser(optparse.OptionParser):
def __init__(self, *args, **kwds):
optparse.OptionParser.__init__(self, *args, **kwds)
self.conflicts = []
def set_not_allowed(self, combination, message):
self.conflicts.append(Conflict(combination, message, self))
def parse_args(self, *args, **kwds):
# Force-ignore the default values and parse the arguments first
kwds2 = dict(kwds)
kwds2["values"] = optparse.Values()
options, _ = optparse.OptionParser.parse_args(self, *args, **kwds2)
# Check for conflicts
for conflict in self.conflicts:
if not conflict.accepts(options):
raise ConflictError(conflict)
# Parse the arguments once again, now with defaults
return optparse.OptionParser.parse_args(self, *args, **kwds)
You can then handle ConflictError where you call parse_args:
try:
options, args = parser.parse_args()
except ConflictError as err:
parser.error(err.message)
Tamás's answer is a good start, but I couldn't get it to work, as it has (or had) a number of bugs, including a broken call to super, "parser" missing in Conflict.__slots__, always raising an error when a conflict is specified because of the use of parser.has_option() in Conflicts.accepts(), etc.
Since I really needed this feature, I rolled my own solution and have made it available from the Python Package Index as ConflictsOptionParser. It works pretty much as a drop in replacement for optparse.OptionParser. (I do know argparse is the new command line parsing hotness, but it is not available in Python 2.6 and below and has less adoption currently than optparse. Send me an email if you'd like to hack up or have hacked up an additional argparse-based solution.) The key is two new methods, register_conflict(), and, to a lesser extent, unregister_conflict():
#/usr/bin/env python
import conflictsparse
parser = conflictsparse.ConflictsOptionParser("python %prog [OPTIONS] ARG")
# You can retain the Option instances for flexibility, in case you change
# option strings later
verbose_opt = parser.add_option('-v', '--verbose', action='store_true')
quiet_opt = parser.add_option('-q', '--quiet', action='store_true')
# Alternatively, you don't need to keep references to the instances;
# we can re-use the option strings later
parser.add_option('--no-output', action='store_true')
# Register the conflict. Specifying an error message is optional; the
# generic one that is generated will usually do.
parser.register_conflict((verbose_opt, quiet_opt, '--no-output'))
# Now we parse the arguments as we would with
# optparse.OptionParser.parse_args()
opts, args = parser.parse_args()
It has a few advantages over the solution begun by Támas:
It works out of the box and is installable through pip (or easy_install, if you must).
Options in a conflict may be specified either by their option strings or by their optparse.Option instances, which helps with the DRY principle; if you use the instances, you can change the actual strings without worrying about breaking conflict code.
It follows normal optparse.OptionParser.parse_args() behavior and automatically calls optparse.OptionParser.error() when it detects conflicting options in the command line arguments, rather than throwing the error directly. (This is both a feature and a bug; kind of a bug in optparse's general design, but a feature for this package in that it is at least consistent with optparse behavior.)