We Keep Coding

Parse multiple subcommands in python simultaneously or other way to group parsed arguments

I am converting Bash shell installer utility to Python 2.7 and need to implement complex CLI so I am able to parse tens of parameters (potentially up to ~150). These are names of Puppet class variables in addition to a dozen of generic deployment options, which where available in shell version.
However after I have started to add more variables I faced are several challenges:
1. I need to group parameters into separate dictionaries so deployment options are separated from Puppet variables. If they are thrown into the same bucket, then I will have to write some logic to sort them, potentially renaming parameters and then dictionary merges will not be trivial.
2. There might be variables with the same name but belonging to different Puppet class, so I thought subcommands would allow me to filter what goes where and avoiding name collisions.
At the momment I have implemented parameter parsing via simply adding multiple parsers:
parser = argparse.ArgumentParser(description='deployment parameters.')
env_select = parser.add_argument_group(None, 'Environment selection')
env_select.add_argument('-c', '--client_id', help='Client name to use.')
env_select.add_argument('-e', '--environment', help='Environment name to use.')
setup_type = parser.add_argument_group(None, 'What kind of setup should be done:')
setup_type.add_argument('-i', '--install', choices=ANSWERS, metavar='', action=StoreBool, help='Yy/Nn Do normal install and configuration')
# MORE setup options
...
args, unk = parser.parse_known_args()
config['deploy_cfg'].update(args.__dict__)
pup_class1_parser = argparse.ArgumentParser(description=None)
pup_class1 = pup_class1_parser.add_argument_group(None, 'Puppet variables')
pup_class1.add_argument('--ad_domain', help='AD/LDAP domain name.')
pup_class1.add_argument('--ad_host', help='AD/LDAP server name.')
# Rest of the parameters
args, unk = pup_class1_parser.parse_known_args()
config['pup_class1'] = dict({})
config['pup_class1'].update(args.__dict__)
# Same for class2, class3 and so on.
The problem with this approach that it does not solve issue 2. Also first parser consumes "-h" option and rest of parameters are not shown in help.
I have tried to use example selected as an answer but I was not able to use both commands at once.
## This function takes the 'extra' attribute from global namespace and re-parses it to create separate namespaces for all other chained commands.
def parse_extra (parser, namespace):
namespaces = []
extra = namespace.extra
while extra:
n = parser.parse_args(extra)
extra = n.extra
namespaces.append(n)
return namespaces
pp = pprint.PrettyPrinter(indent=4)
argparser=argparse.ArgumentParser()
subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name')
parser_a = subparsers.add_parser('command_a', help = "command_a help")
## Setup options for parser_a
parser_a.add_argument('--opt_a1', help='option a1')
parser_a.add_argument('--opt_a2', help='option a2')
parser_b = subparsers.add_parser('command_b', help = "command_b help")
## Setup options for parser_a
parser_b.add_argument('--opt_b1', help='option b1')
parser_b.add_argument('--opt_b2', help='option b2')
## Add nargs="*" for zero or more other commands
argparser.add_argument('extra', nargs = "*", help = 'Other commands')
namespace = argparser.parse_args()
pp.pprint(namespace)
extra_namespaces = parse_extra( argparser, namespace )
pp.pprint(extra_namespaces)
Results me in:
$ python argtest.py command_b --opt_b1 b1 --opt_b2 b2 command_a --opt_a1 a1
usage: argtest.py [-h] {command_a,command_b} ... [extra [extra ...]]
argtest.py: error: unrecognized arguments: command_a --opt_a1 a1
The same result was when I tried to define parent with two child parsers.
QUESTIONS
Can I somehow use parser.add_argument_group for argument parsing or is it just for the grouping in help print out? It would solve issue 1 without missing help side effect. Passing it as parse_known_args(namespace=argument_group) (if I correctly recall my experiments) gets all the variables (thats ok) but also gets all Python object stuff in resulting dict (that's bad for hieradata YAML)
What I am missing in the second example to allow to use multiple subcommands? Or is that impossible with argparse?
Any other suggestion to group command line variables? I have looked at Click, but did not find any advantages over standard argparse for my task.
Note: I am sysadmin, not a programmer so be gently on me for the non object style coding. :)
Thank you
RESOLVED
Argument grouping solved via the answer suggested by hpaulj.
import argparse
import pprint
parser = argparse.ArgumentParser()
group_list = ['group1', 'group2']
group1 = parser.add_argument_group('group1')
group1.add_argument('--test11', help="test11")
group1.add_argument('--test12', help="test12")
group2 = parser.add_argument_group('group2')
group2.add_argument('--test21', help="test21")
group2.add_argument('--test22', help="test22")
args = parser.parse_args()
pp = pprint.PrettyPrinter(indent=4)
d = dict({})
for group in parser._action_groups:
if group.title in group_list:
d[group.title]={a.dest:getattr(args,a.dest,None) for a in group._group_actions}
print "Parsed arguments"
pp.pprint(d)
This gets me desired result for the issue No.1. until I will have multiple parameters with the same name. Solution may look ugly, but at least it works as expected.
python argtest4.py --test22 aa --test11 yy11 --test21 aaa21
Parsed arguments
{ 'group1': { 'test11': 'yy11', 'test12': None},
'group2': { 'test21': 'aaa21', 'test22': 'aa'}}

Your question is too complicated to understand and respond to in one try. But I'll throw out some preliminary ideas.
Yes, argument_groups are just a way of grouping arguments in the help. They have no effect on parsing.
Another recent SO asked about parsing groups of arguments:
Is it possible to only parse one argument group's parameters with argparse?
That poster initially wanted to use a group as a parser, but the argparse class structure does not allow that. argparse is written in object style. parser=ArguementParser... creates one class of object, parser.add_arguement... creates another, add_argument_group... yet another. You customize it by subclassing ArgumentParser or HelpFormatter or Action classes, etc.
I mentioned a parents mechanism. You define one or more parent parsers, and use those to populate your 'main' parser. They could be run indepdently (with parse_known_args), while the 'main' is used to handle help.
We also discussed grouping the arguments after parsing. A namespace is a simple object, in which each argument is an attribute. It can also be converted to a dictionary. It is easy to pull groups of items from a dictionary.
There have SO questions about using multiple subparsers. That's an awkward proposition. Possible, but not easy. Subparsers are like issueing a command to a system program. You generally issue one command per call. You don't nest them or issue sequences. You let shell piping and scripts handle multiple actions.
IPython uses argparse to parse its inputs. It traps help first, and issues its own message. Most arguments come from config files, so it is possible to set values with default configs, custom configs and in the commandline. It's an example of naming a very large set of arguments.
Subparsers let you use the same argument name, but without being able to invoke multiple subparsers in one call that doesn't help much. And even if you could invoke several subparsers, they would still put the arguments in the same namespace. Also argparse tries to handle flaged arguments in an order independent manner. So a --foo at the end of the command line gets parsed the same as though it were at the start.
There was SO question where we discussed using argument names ('dest') like 'group1.argument1', and I've even discussed using nested namespaces. I could look those up if it would help.
Another thought - load sys.argv and partition it before passing it to one or more parsers. You could split it on some key word, or on prefixes etc.

If you have so many arguments this seems like a design issue. It seems very unmanageable. Can you not implement this using a configuration file that has reasonable set of defaults? Or defaults in the code with a reasonable (i.e. SMALL) number of arguments in the command line, and allow everything or everything else to be overridden with parameters in a 'key:value' configuration file? I can't imagine having to use a CLI with the number of variables you are proposing.

ArgumentParser epilog and description formatting in conjunction with ArgumentDefaultsHelpFormatter

I'm using argparse to take in command line input and also to produce help text. I want to use ArgumentDefaultsHelpFormatter as the formatter_class, however this prevents me from also using RawDescriptionHelpFormatter which would allow me to add custom formatting to my description or epilog.
Is there a sensible method of achieving this aside from writing code to produce text for default values myself? According to the argparse docs, all internals of ArgumentParser are considered implementation details, not public API, so sub-classing isn't an attractive option.

I just tried a multiple inheritance approach, and it works:
class CustomFormatter(argparse.ArgumentDefaultsHelpFormatter, argparse.RawDescriptionHelpFormatter):
pass
parser = argparse.ArgumentParser(description='test\ntest\ntest.',
epilog='test\ntest\ntest.',
formatter_class=CustomFormatter)
This may break if the internals of these classes change though.

I don't see why subclassing a HelpFormatter should be a problem. That isn't messing with the internals of ArgumentParser. The documentation has examples of custom Action and Type classes (or functions). I take the 'there are four such classes' line to be an invitation to write my own HelpFormatter if needed.
The provided HelpFormatter subclasses make quite simple changes, changing just one function. So they can be easily copied or altered.
RawDescription just changes:
def _fill_text(self, text, width, indent):
return ''.join(indent + line for line in text.splitlines(keepends=True))
In theory it could be changed without altering the API, but it's unlikely.
The defaults formatter just changes:
def _get_help_string(self, action):
help = action.help
if '%(default)' not in action.help:
if action.default is not SUPPRESS:
defaulting_nargs = [OPTIONAL, ZERO_OR_MORE]
if action.option_strings or action.nargs in defaulting_nargs:
help += ' (default: %(default)s)'
return help
You could get the same effect by just including %(default)s in all of your argument help lines. In contrast to the Raw subclasses, this is just a convenience class. It doesn't give you more control over the formatting.

Text wrapping from optparse help string on multiple lines

I am trying to make my help strings helpful. To do this I have a function Function() with a doc string something like
def Function(x):
""" First line describing what Function does
Keyword Arguments
x = float -- A description of what x does that may be long
"""
Having done this I think have something like this at the end
def parse_command_line(argvs):
parser = optparse.OptionParser()
parser.add_option("-f","--Function", help=Function.__doc__,metavar="Bs" )
(options,arguments) = parser.parse_args(argvs)
return options, arguments
options, arguments = parse_command_line(sys.argv)
The trouble occurs when calling the program with -h or --help The output is line wrapped by OptParse, this means the KeyWord arguments are not started on a new line, is it possible to stop OptParse from wrapping the output or is there a better way to do this?

In case of interest, I have written two such formatter classes for argparse. The first supports much of the mediaWiki, MarkDown, and POD syntaxes (even intermixed):
import MarkupHelpFormatter
MarkupHelpFormatter.InputOptions["mediawiki"] = True
parser = argparse.ArgumentParser(
description="""...your text, with mediawiki markup...""",
epilog='...',
formatter_class=MarkupHelpFormatter.MarkupHelpFormatter
)
The other is called "ParagraphHelpFormatter". It merely wraps text like the default argparse formatter, except that it respects blank lines.
Both are at http://derose.net/steve/utilities/PY/MarkupHelpFormatter.py, and
licensed as CCLI Attribution-Share-alike. They format for ANSI terminal
interfaces. Not highly polished (for example, auto-numbering is unfinished), but you may find them helpful.

argparse provides you with a formatter for raw formatting, ie your lines wont get wrapped
parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
optparse also lets you set the formatter.. I suppose you can write your own formatter, but theres nothing provided that i know of

argparse change definition of argument

I set up my argument parser as follows:
parser=argparse.ArgumentParser()
parser.add_argument('--point',help='enter a point (e.g. 2,3,4)')
parser.parse_args('--point=-2,5,6'.split()) #works
parser.parse_args('--point -2,5,6'.split()) #doesn't work :(
Is there any way to tell argparse that strings which match the regular expression r"-\d+.*" are not options but an argument of an option?
Also note that I could do something like this:
parser.add_argument('--point',nargs='*')
parser.parse_args('--point -2 5 6'.split())
but that's not really how I want it to work.

I think preprocessing sys.argv is the most straightforward way here. Consider for example:
import argparse, re
parser=argparse.ArgumentParser()
parser.add_argument('--point',help='enter a point (e.g. 2,3,4)')
args = '--point -2,5,6'.split() # or sys.argv
is_list = re.compile(r'^-?[\d,.]+$')
args = ['"%s"' % x if is_list.match(x) else x for x in args]
print parser.parse_args(args)
This returns Namespace(point='"-2,5,6"') which should be easy to parse.

You could change the prefix char so - is no longer recognized as indicating the start of an argument. It does look a bit weird but it is useful when negative numbers may appear in the arguments.
import argparse
parser=argparse.ArgumentParser(prefix_chars = '#')
parser.add_argument('##point',help='enter a point (e.g. 2,3,4)')
args = parser.parse_args('##point=-2,5,6'.split()) #works
print(args)
# Namespace(point='-2,5,6')
args = parser.parse_args('##point -2,5,6'.split()) #work also
print(args)
# Namespace(point='-2,5,6')

If you don't mind messing around with argparse internals, argparse already does something very similar to what I want to do. One of the classes that ArgumentParser inherits from has this line in __init__
import re as _re
...
self._negative_number_matcher = _re.compile(r'^-\d+$|^-\d*\.\d+$')
So to get my example to work, we just need to swap out an appropriate regex...
parser._negative_number_matcher = re.compile(r'^-\d+|^-\d*\.\d+')
Usually I'm not very much in favor of messing around with the internals of a class when they're prefixed by an underscore (as it is implementation dependent and liable to change) -- However, in this case, I think it's probably ok because:
If argparse changes that variable name, it doesn't hurt, we're just back to the case where "--print -2,3,4" doesn't work again.
I can't think of any better way to determine if something is a number than regex (I suppose they could try to cast to a float and catch the exception, but if they did that, they wouldn't have a variable named _negative_number_matcher anymore and again, argparse would still work fine except in this corner case where it doesn't do what I want)

Disable abbreviation in argparse

argparse uses per default abbreviation in unambiguous cases.
I don't want abbreviation and I'd like to disable it.
But didn't find it in the documentation.
Is it possible?
Example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--send', action='store_true')
parser.parse_args(['--se']) # returns Namespace(send=True)
But I want it only to be true when the full parameter is supplied. To prevent user errors.
UPDATE:
I created a ticket at python bugtracker after Vikas answer. And it already has been processed.

As of Python 3.5.0 you can disable abbreviations by initiating the ArgumentParser with the following:
parser = argparse.ArgumentParser(allow_abbrev=False)
Also see the documentation.

No, apparently this is not possible. At least in Python 2.7.2.
First, I took a look into the documentation - to no avail.
Then I opened the Lib\argparse.py and looked through the source code. Omitting a lot of details, it seems that each argument is parsed by a regular expression like this (argparse:2152):
# allow one or more arguments
elif nargs == ONE_OR_MORE:
nargs_pattern = '(-*A[A-]*)'
This regex will successfully parse both '-' and '--', so we have no control over the short and long arguments. Other regexes use the -* construct too, so it does not depend on the type of the parameter (no sub-arguments, 1 sub-argument etc).
Later in the code double dashes are converted to one dash (only for non-optional args), again, without any flags to control by user:
# if this is an optional action, -- is not allowed
if action.option_strings:
nargs_pattern = nargs_pattern.replace('-*', '')
nargs_pattern = nargs_pattern.replace('-', '')

No, well not without ugly hacks.
The code snippet #Vladimir posted, i suppose that is not what you are looking for. The actual code that is doing this is:
def _get_option_tuples(self, option_string):
...
if option_string.startswith(option_prefix):
...
See the check is startswith not ==.
And you can always extend argparse.ArgumentParser to provide your own _get_option_tuples(self, option_string) to change this behavior. I just did by replacing two occurrence of option_string.startswith(option_prefix) to option_string == option_prefix and:
>>> parser = my_argparse.MyArgparse
>>> parser = my_argparse.MyArgparse()
>>> parser.add_argument('--send', action='store_true')
_StoreTrueAction(option_strings=['--send'], dest='send', nargs=0, const=True, default=False, type=None, choices=None, help=None, metavar=None)
>>> parser.parse_args(['--se'])
usage: [-h] [--send]
: error: unrecognized arguments: --se
A word of caution
The method _get_option_tuples is prefixed with _, which typically means a private method in python. And it is not a good idea to override a private.

Another way for Python 2.7. Let's get clunky! Say you want to recognize --dog without abbreviation.
p = argparse.ArgumentParser()
p.add_argument('--dog')
p.add_argument('--dox', help=argparse.SUPPRESS, metavar='IGNORE')
By adding a second argument --dox that differs from the argument you want only in the third letter, --d and --do become ambiguous. Therefore, the parser will refuse to recognize them. You would need to add code to catch the resulting exception and process it according to the context in which you are calling parse_args. You might also need to suppress/tweak the help text.
The help=... keeps the argument out of the option list on the default help message (per this), and metavar='IGNORE' is just to make it clear you really aren't doing anything with this option :) .