Related
What I need
I need an ArgumentParser, with a conflict handling scheme, that resolves some registered set of duplicate arguments, but raises on all other arguments.
What I tried
My initial approach (see also the code example at the bottom) was to subclass ArgumentParser, add a _handle_conflict_custom method, and then instantiate the subclass with ArgumentParser(conflict_handler='custom'), thinking that the _get_handler method would pick it up.
The Problem
This raises an error, because the ArgumentParser inherits from _ActionsContainer, which provides the _get_handler and the _handle_conflict_{strategy} methods, and then internally instantiates an _ArgumentGroup (that also inherits from _ActionsContainer), which in turn doesn't know about the newly defined method on ArgumentParser and thus fails to get the custom handler.
Overriding the _get_handler method is not feasible for the same reasons.
I have created a (rudimentary) class diagram illustrating the relationships, and therefore hopefully the problem in subclassing ArgumentParser to achieve what I want.
Motivation
I (think, that I) need this, because I have two scripts, that handle distinct parts of a workflow, and I would like to be able to use those separately as scripts, but also have one script, that imports the methods of both of these scripts, and does everything in one go.
This script should support all the options of the two individual scripts, but I don't want to duplicate the (extensive) argument definitions, so that I would have to make changes in multiple places.
This is easily solved by importing the ArgumentParsers of the (part) scripts and using them as parents, like so combined_parser = ArgumentParser(parents=[arg_parser1, arg_parser2]).
In the scripts I have duplicate options, e.g. for the work directory, so I need to resolve those conflicts.
This could also be done, with conflict_handler='resolve'.
But because there are a lot of possible arguments (which is not up to our team, because we have to maintain compatibility), I also want the script to raise an error if something gets defined that causes a conflict, but hasn't been explicitly allowed to do so, instead of quietly overriding the other flag, potentially causing unwanted behavior.
Other suggestions to achieve these goals (keeping both scripts separate, enabling use of one script that wraps both, avoiding code duplication and raising on unexpected duplicates) are welcome.
Example Code
from argparse import ArgumentParser
class CustomParser(ArgumentParser):
def _handle_conflict_custom(self, action, conflicting_actions):
registered = ['-h', '--help', '-f']
conflicts = conflicting_actions[:]
use_error = False
while conflicts:
option_string, action = conflicts.pop()
if option_string in registered:
continue
else:
use_error = True
break
if use_error:
self._handle_conflict_error(action, conflicting_actions)
else:
self._handle_conflict_resolve(action, conflicting_actions)
if __name__ == '__main__':
ap1 = ArgumentParser()
ap2 = ArgumentParser()
ap1.add_argument('-f') # registered, so should be resolved
ap2.add_argument('-f')
ap1.add_argument('-g') # not registered, so should raise
ap2.add_argument('-g')
# this raises before ever resolving anything, for the stated reasons
ap3 = CustomParser(parents=[ap1, ap2], conflict_handler='custom')
Other questions
I am aware of these similar questions:
python argparse subcommand with dependency and conflict
argparse conflict when used with two connected python3 scripts
Handling argparse conflicts
... and others
But even though some of them provide interesting insights into argparse usage and conflicts, they seem to address issues that are not related to mine.
While I agree that FMc's approach is probably the better one in terms of long term viability, I have found a way to override a custom handler into the ArgumentParser.
The key is to override the _ActionsContainer class which actually defines the handler functions. Then to override the base classes that the ArgumentParser and _ArgumentGroup inherit from.
In the case below, I've simply added a handler that ignores any conflicts, but you could add any custom logic you want.
import argparse
class IgnorantActionsContainer(argparse._ActionsContainer):
def _handle_conflict_ignore(self, action, conflicting_actions):
pass
argparse.ArgumentParser.__bases__ = (argparse._AttributeHolder, IgnorantActionsContainer)
argparse._ArgumentGroup.__bases__ = (IgnorantActionsContainer,)
parser = argparse.ArgumentParser(conflict_handler="ignore")
parser.add_argument("-a", type=int, default=1)
parser.add_argument("-a", type=int, default=2)
parser.add_argument("-a", type=int, default=3)
parser.add_argument("-a", type=int, default=4)
print(parser.parse_args())
Running python custom_conflict_handler.py -h prints:
usage: custom_conflict_handler.py [-h] [-a A] [-a A] [-a A] [-a A]
optional arguments:
-h, --help show this help message and exit
-a A
-a A
-a A
-a A
Running python custom_conflict_handler.py prints:
Namespace(a=1)
Running python custom_conflict_handler.py -a 5 prints:
Namespace(a=5)
For a various reasons -- notably the needs of testing -- I have adopted the
habit of always defining argparse configuration in the form of a data
structure, typically a sequence of dicts. The actual creation of the
ArgumentParser is done in a reusable function that simply builds the parser
from the dicts. This approach has many benefits, especially for more complex
projects.
If each of your scripts were to shift to that model, I would think that you
might be able to detect any configuration conflicts in that function and raise
accordingly, thus avoiding the need to inherit from ArgumentParser and mess
around with understanding its internals.
I'm not certain I understand your conflict-handling needs very well, so the
demo below simply hunts for duplicate options and raises if it sees one, but I
think you should be able to understand the approach and assess whether it might
work for your case. The basic idea is to solve your problem in the realm
of ordinary data structures rather than in the byzantine world of argparse.
import sys
import argparse
from collections import Counter
OPTS_CONFIG1 = (
{
'names': 'path',
'metavar': 'PATH',
},
{
'names': '--nums',
'nargs': '+',
'type': int,
},
{
'names': '--dryrun',
'action': 'store_true',
},
)
OPTS_CONFIG2 = (
{
'names': '--foo',
'metavar': 'FOO',
},
{
'names': '--bar',
'metavar': 'BAR',
},
{
'names': '--dryrun',
'action': 'store_true',
},
)
def main(args):
ap = define_parser(OPTS_CONFIG1, OPTS_CONFIG2)
opts = ap.parse_args(args)
print(opts)
def define_parser(*configs):
# Validation: adjust as needed.
tally = Counter(
nm
for config in configs
for d in config
for nm in d['names'].split()
)
for k, n in tally.items():
if n > 1:
raise Exception(f'Duplicate argument configurations: {k}')
# Define and return parser.
ap = argparse.ArgumentParser()
for config in configs:
for d in config:
kws = dict(d)
xs = kws.pop('names').split()
ap.add_argument(*xs, **kws)
return ap
if __name__ == '__main__':
main(sys.argv[1:])
There is an answer that's marginally less hacky than Hans's approach. You can simply subclass argparse.ActionsContainer and argparse.ArgumentGroup and make sure you inherit from ActionsContainer after argparse.ArgumentParser, this way it'll be later in the MRO and will take precedence. Here's an example:
import argparse
from typing import Iterable, Any
class ArgumentGroup(argparse._ArgumentGroup):
def _handle_conflict_custom(
self,
action: argparse.Action,
conflicting_actions: Iterable[tuple[str, argparse.Action]],
) -> None:
...
class ActionsContainer(argparse._ActionsContainer):
def _handle_conflict_custom(
self,
action: argparse.Action,
conflicting_actions: Iterable[tuple[str, argparse.Action]],
) -> None:
...
def add_argument_group(self, *args: Any, **kwargs: Any) -> ArgumentGroup:
group = ArgumentGroup(self, *args, **kwargs)
self._action_groups.append(group)
return group
class ArgumentParser(argparse.ArgumentParser, ActionsContainer):
...
Based on FMcs approach I have created something a little more elaborate, I know this isn't code review, but feedback is still welcome. Also, maybe it helps someone to see this fleshed out a bit more.
import argparse
from collections import Counter, OrderedDict
from typing import List, Dict, Any
from copy import deepcopy
class OptionConf:
def __init__(self):
self._conf = OrderedDict() # type: Dict[str, List[Dict[str, Any]]]
self._allowed_dupes = list() # type: List[str]
def add_conf(self, command, *conf_args, **conf_kwargs):
if command not in self._conf:
self._conf[command] = []
conf_kwargs['*'] = conf_args
self._conf[command].append(conf_kwargs)
def add_argument(self, *conf_args, **conf_kwargs):
self.add_conf('add_argument', *conf_args, **conf_kwargs)
def register_allowed_duplicate(self, flag):
self._allowed_dupes.append(flag)
def generate_parser(self, **kwargs):
argument_parser = argparse.ArgumentParser(**kwargs)
for command, conf_kwargs_list in self._conf.items():
command_func = getattr(argument_parser, command)
for conf_kwargs in conf_kwargs_list:
list_args = conf_kwargs.pop('*', [])
command_func(*list_args, **conf_kwargs)
conf_kwargs['*'] = list_args
return argument_parser
def _get_add_argument_conf_args(self):
for command, kwargs_list in self._conf.items():
if command != 'add_argument':
continue
return kwargs_list
return []
def resolve_registered(self, other):
if self.__class__ == other.__class__:
conf_args_list = self._get_add_argument_conf_args() # type: List[Dict[str, Any]]
other_conf_args_list = other._get_add_argument_conf_args() # type: List[Dict[str, Any]]
# find all argument names of both parsers
all_names = []
for conf_args in conf_args_list:
all_names += conf_args.get('*', [])
all_other_names = []
for other_conf_args in other_conf_args_list:
all_other_names += other_conf_args.get('*', [])
# check for dupes and throw if appropriate
found_allowed_dupes = []
tally = Counter(all_names + all_other_names)
for name, count in tally.items():
if count > 1 and name not in self._allowed_dupes:
raise Exception(f'Duplicate argument configurations: {name}')
elif count > 1:
found_allowed_dupes.append(name)
# merge them in a new OptionConf, preferring the args of self (AS OPPOSED TO ORIGINAL RESOLVE)
new_opt_conf = OptionConf()
for command, kwargs_list in self._conf.items():
for kwargs in kwargs_list:
list_args = kwargs.get('*', [])
new_opt_conf.add_conf(command, *list_args, **kwargs)
for command, kwargs_list in other._conf.items():
for kwargs in kwargs_list:
# if it's another argument, we remove dupe names
if command == 'add_argument':
all_names = kwargs.pop('*', [])
names = [name for name in all_names if name not in found_allowed_dupes]
# and only add if there are names left
if names:
new_opt_conf.add_argument(*deepcopy(names), **deepcopy(kwargs))
# put names back
kwargs['*'] = all_names
else:
# if not, we just add it
list_args = kwargs.pop('*', [])
new_opt_conf.add_conf(command, *deepcopy(list_args), **deepcopy(kwargs))
# put list args back
kwargs['*'] = list_args
return new_opt_conf
raise NotImplementedError()
if __name__ == '__main__':
opts_conf = OptionConf()
opts_conf.add_argument('pos_arg')
opts_conf.add_argument('-n', '--number', metavar='N', type=int)
opts_conf.add_argument('-i', '--index')
opts_conf.add_argument('-v', '--verbose', action='store_true')
opts_conf2 = OptionConf()
opts_conf2.add_argument('-n', '--number', metavar='N', type=int)
opts_conf2.add_argument('-v', action='store_true')
opts_conf.register_allowed_duplicate('-n')
opts_conf.register_allowed_duplicate('--number')
try:
resolved_opts = opts_conf.resolve_registered(opts_conf2)
except Exception as e:
print(e) # raises on -v
opts_conf.register_allowed_duplicate('-v')
resolved_opts = opts_conf.resolve_registered(opts_conf2)
ap = resolved_opts.generate_parser(description='does it work?')
ap.parse_args(['-h'])
import argparse
import pickle
parser = argparse.ArgumentParser(description='Process some integers.')
_ = pickle.dumps(parser)
In my code, the ArgumentParser object is serialized. But in runtime I get the error Can't pickle local object 'ArgumentParser.__init__.<locals>.identity.
In Lib/argparse.py the identity is function defined locally inside __init__ method and this prevents serialization. If convert this function to a method, then serialization is successful. But I think that this way is not the best solution, since the python library file is being changed. How serialize parser object best way?
I had a program that used sub-parsers and initializing the ArgumentParser noticeably delayed startup, even for —help. I experimented with several things encountered this. I found this to work.
from argparse import ArgumentParser
from pickle import dumps
def identity(string):
return string
p = ArgumentParser()
p.register('type', None, identity)
x = dumps(p)
I created the heir class class ArgumentParserSerializable(ArgumentParser) and definied identity method as static. It also works.
class ArgumentParserSerializable(ArgumentParser):
def __init__(self,
prog=None,
usage=None,
description=None,
epilog=None,
parents=[],
formatter_class=HelpFormatter,
prefix_chars='-',
fromfile_prefix_chars=None,
argument_default=None,
conflict_handler='error',
add_help=True,
allow_abbrev=True):
superinit = super(ArgumentParser, self).__init__
superinit(description=description,
prefix_chars=prefix_chars,
argument_default=argument_default,
conflict_handler=conflict_handler)
# default setting for prog
if prog is None:
prog = _os.path.basename(_sys.argv[0])
self.prog = prog
self.usage = usage
self.epilog = epilog
self.formatter_class = formatter_class
self.fromfile_prefix_chars = fromfile_prefix_chars
self.add_help = add_help
self.allow_abbrev = allow_abbrev
add_group = self.add_argument_group
self._positionals = add_group(_('positional arguments'))
self._optionals = add_group(_('optional arguments'))
self._subparsers = None
self.register('type', None, self.identity)
# add help argument if necessary
# (using explicit default to override global argument_default)
default_prefix = '-' if '-' in prefix_chars else prefix_chars[0]
if self.add_help:
self.add_argument(
default_prefix + 'h', default_prefix * 2 + 'help',
action='help', default=SUPPRESS,
help=_('show this help message and exit'))
# add parent arguments and defaults
for parent in parents:
self._add_container_actions(parent)
try:
defaults = parent._defaults
except AttributeError:
pass
else:
self._defaults.update(defaults)
# register types
#staticmethod
def identity(string):
return string
I write lots of little utility functions which I would like to make available both directly through the command line but also importable as python functions to be used by other utilities. Currently what I do is write my function in a file, and in the same file under if __name__ == "__main__": I use argparse to interface with the function on the command line. For example, let's say I have the file math.py:
import argparse
def add_or_subtract(a: float, b: float, c: float = 1., add: bool = True) -> float:
"""
Do some random math
Parameters
----------
a : float
A number
b : float
Another number
c : float, optional
Another number
add : bool, optional
Whether to add or subtract c
Returns
-------
answer : float
The answer
"""
if add:
return a+b+c
else:
return a+b-c
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("a", type=float, help="a number")
parser.add_argument("b", type=float, help="another number")
parser.add_argument("-c", "--c", type=float, help="another number", default=1.)
parser.add_argument("-a", "--add", action=store_true)
parser.parse_args()
print(add_or_subtract(parser.a, parser.b, parser.c, parser.add))
Basically I have the feeling that I am doing a lot of duplication defining arguments, their acceptable types, and their explanations. If I change some arguments on the function I have to remember to update it in three places. I'm wondering if there is an easier way.
I've been playing a bit with inspect to add CLI arguments based on the arguments in the function, but I want something a bit "smarter" that knows the difference between mandatory and optional arguments, acceptable types, boolean flags etc. It would be even greater if the docstrings could also be parsed for the help. The ideal scenario would be a kind of decorator that "command-lineifies" the function.
Does something like I'm describing exist? Or are there better ways of doing what I want.
Why don't you try implementing commands as classes and later just inherit those?
Something like this might work:
class CommandArgument:
def __init__(self, *args, **kwargs):
self.args = args
self.kwargs = kwargs
def add_to_parser(self, parser: argparse.ArgumentParser):
parser.add_argument(*self.args, **self.kwargs)
class BaseAddOrSubCommand:
a = CommandArgument('a', type=float, help='a number')
b = CommandArgument('b', type=float, help='another number')
c = CommandArgument('b', type=float, help='another number')
args = [a, b, c]
def __init__(self):
self.parser = argparse.ArgumentParser
for arg in self.args:
arg.add_to_parser(self.parser)
self.parser.parse_args()
def execute(self):
if self.parser.add:
return self.parser.a + self.parser.b + self.parser.c
else:
return self.parser.a + self.parser.b - self.parser.c
class MultiplySumOrSubCommand(BaseAddOrSubCommand):
mult = CommandArgument('mult', type=float, help='multiply result with me')
args = BaseAddOrSubCommand.args + [mult]
def execute(self):
return super().execute() * self.parser.mult
if __name__ == '__main__':
command = MultiplySumOrSubCommand()
print(command.execute())
I want to create a command line flag that can be used as
./prog.py --myarg=abcd,e,fg
and inside the parser have this be turned into ['abcd', 'e', 'fg'] (a tuple would be fine too).
I have done this successfully using action and type, but I feel like one is likely an abuse of the system or missing corner cases, while the other is right. However, I don't know which is which.
With action:
import argparse
class SplitArgs(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
setattr(namespace, self.dest, values.split(','))
parser = argparse.ArgumentParser()
parser.add_argument('--myarg', action=SplitArgs)
args = parser.parse_args()
print(args.myarg)
Instead with type:
import argparse
def list_str(values):
return values.split(',')
parser = argparse.ArgumentParser()
parser.add_argument('--myarg', type=list_str)
args = parser.parse_args()
print(args.myarg)
The simplest solution is to consider your argument as a string and split.
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--myarg", type=str)
d = vars(parser.parse_args())
if "myarg" in d.keys():
d["myarg"] = [s.strip() for s in d["myarg"].split(",")]
print(d)
Result:
$ ./toto.py --myarg=abcd,e,fg
{'myarg': ['abcd', 'e', 'fg']}
$ ./toto.py --myarg="abcd, e, fg"
{'myarg': ['abcd', 'e', 'fg']}
I find your first solution to be the right one. The reason is that it allows you to better handle defaults:
names: List[str] = ['Jane', 'Dave', 'John']
parser = argparse.ArumentParser()
parser.add_argument('--names', default=names, action=SplitArgs)
args = parser.parse_args()
names = args.names
This doesn't work with list_str because the default would have to be a string.
Your custom action is the closest way to how it is done internally for other argument types. IMHO there should be a _StoreCommaSeperatedAction added to argparse in the stdlib since it is a somewhat common and useful argument type,
It can be used with an added default as well.
Here is an example without using an action (no SplitArgs class):
class Test:
def __init__(self):
self._names: List[str] = ["Jane", "Dave", "John"]
#property
def names(self):
return self._names
#names.setter
def names(self, value):
self._names = [name.strip() for name in value.split(",")]
test_object = Test()
parser = ArgumentParser()
parser.add_argument(
"-n",
"--names",
dest="names",
default=",".join(test_object.names), # Joining the default here is important.
help="a comma separated list of names as an argument",
)
print(test_object.names)
parser.parse_args(namespace=test_object)
print(test_object.names)
Here is another example using SplitArgs class inside a class completely
"""MyClass
Demonstrates how to split and use a comma separated argument in a class with defaults
"""
import sys
from typing import List
from argparse import ArgumentParser, Action
class SplitArgs(Action):
def __call__(self, parser, namespace, values, option_string=None):
# Be sure to strip, maybe they have spaces where they don't belong and wrapped the arg value in quotes
setattr(namespace, self.dest, [value.strip() for value in values.split(",")])
class MyClass:
def __init__(self):
self.names: List[str] = ["Jane", "Dave", "John"]
self.parser = ArgumentParser(description=__doc__)
self.parser.add_argument(
"-n",
"--names",
dest="names",
default=",".join(self.names), # Joining the default here is important.
action=SplitArgs,
help="a comma separated list of names as an argument",
)
self.parser.parse_args(namespace=self)
if __name__ == "__main__":
print(sys.argv)
my_class = MyClass()
print(my_class.names)
sys.argv = [sys.argv[0], "--names", "miigotu, sickchill,github"]
my_class = MyClass()
print(my_class.names)
And here is how to do it in a function based situation, with a default included
class SplitArgs(Action):
def __call__(self, parser, namespace, values, option_string=None):
# Be sure to strip, maybe they have spaces where they don't belong and wrapped the arg value in quotes
setattr(namespace, self.dest, [value.strip() for value in values.split(",")])
names: List[str] = ["Jane", "Dave", "John"]
parser = ArgumentParser(description=__doc__)
parser.add_argument(
"-n",
"--names",
dest="names",
default=",".join(names), # Joining the default here is important.
action=SplitArgs,
help="a comma separated list of names as an argument",
)
parser.parse_args()
I know this post is old but I recently found myself solving this exact problem. I used functools.partial for a lightweight solution:
import argparse
from functools import partial
csv_ = partial(str.split, sep=',')
p = argparse.ArgumentParser()
p.add_argument('--stuff', type=csv_)
p.parse_args(['--stuff', 'a,b,c'])
# Namespace(stuff=['a', 'b', 'c'])
If you're not familiar with functools.partial, it allows you to create a partially "frozen" function/method. In the above example, I created a new function (csv_) that is essentially a copy of str.split() except that the sep argument has been "frozen" to the comma character.
I am using argparse's action to add various data to a class. I would like to use that action on the default value if that arg is not provided at the command line. Is this possible?
Thanks!
argparse does not use the action when applying the default. It just uses setattr. It may use the type if the default is a string. But you can invoke the action directly.
Here I use a custom action class borrowed from the documentation. In the first parse_args nothing happens. Then I create a new namespace, and invoke the action on the default. Then I pass that namespace to parse_args. To understand this, you many need to import it into an interactive shell, and examine the attributes of the namespace and action.
# sample custom action from docs
class FooAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
print('Setting: %r %r %r' % (namespace, values, option_string))
setattr(namespace, self.dest, 'action:'+values)
p = argparse.ArgumentParser()
a1 = p.add_argument('--foo', action=FooAction, default='default')
print 'action:',a1
print p.parse_args([])
ns = argparse.Namespace()
a1(p, ns, a1.default, 'no string') # call action
print p.parse_args([],ns)
print p.parse_args(['--foo','1'],ns)
which produces:
action: FooAction(option_strings=['--foo'], dest='foo', nargs=None, const=None, default='default', type=None, choices=None, help=None, metavar=None)
Namespace(foo='default')
Setting: Namespace() 'default' 'no string'
Namespace(foo='action:default')
Setting: Namespace(foo='action:default') '1' '--foo'
Namespace(foo='action:1')
I tailored the output to highlight when the action is being used.
Here's a way of performing a special action on an argument that isn't given on the command line (or given with a value == to the default). It's a simplification of the class given in https://stackoverflow.com/a/24638908/901925.
class Parser1:
def __init__(self, desc):
self.parser = argparse.ArgumentParser(description=desc)
self.actions = []
def milestone(self, help_='milestone for latest release.', default=None):
action = self.parser.add_argument('-m', '--milestone', help=help_, default=default)
self.actions.append(action)
return self
def parse(self):
args = self.parser.parse_args()
for a in self.actions:
if getattr(args, a.dest) == a.default:
print 'Please specify', a.dest
values = raw_input('>')
setattr(args, a.dest, values)
return args
print Parser1('desc').milestone(default='PROMPT').parse()
The prompting is done after parse_args. I don't see any reason to call parse_args again.
I needed to prompt the user if an option was not specified - that's how I did it:
class _PromptUserAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if values == self.default:
print 'Please specify', self.dest
values = raw_input('>')
setattr(namespace, self.dest, values)
class Parser:
def __init__(self, desc, add_h=True):
self.parser = argparse.ArgumentParser(description=desc, add_help=add_h,
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
#actions to be run on options if not specified (using default to check)
self.actions = []
#staticmethod
def new(description, add_help=True):
return Parser(description, add_help)
# ...
def milestone(self, help_='Specify the milestone for latest release.'):
action = self.parser.add_argument('-m', '--milestone',
dest='milestone',
action=_PromptUserAction,
default='PROMPT', # needed I think
type=str,
help=help_)
self.actions.append(action)
return self
def parse(self):
"""
Return an object which can be used to get the arguments as in:
parser_instance.parse().milestone
:return: ArgumentParser
"""
args = self.parser.parse_args()
# see: http://stackoverflow.com/a/21588198/281545
dic = vars(args)
ns = argparse.Namespace()
for a in self.actions:
if dic[a.dest] == a.default:
a(self.parser, ns, a.default) # call action
# duh - can I avoid it ?
import sys
return self.parser.parse_args(sys.argv[1:],ns)
I am interested if this can somehow be done without having to reparse the args (the import sys part). Maybe some constructor options for argparse.Action ?