We Keep Coding

Different levels of help verbosity in argparse

I have an application with several dozens of CLI parameters. I use python argparse to parse arguments. Most of them are rarely used, only 5 or 6 are frequent. I don't want to clutter up the output of --help for simple cases, however there still should be a possibility to see description for all parameters somehow.
It is possible to have several verbosity levels for help in argparse? Here's how I expect it to be.
$ myapp -h
optional arguments:
--foo Do foo stuff
--bar Do bar stuff
$ myapp -hh # or myapp --expert-mode -h
optional arguments:
--foo Do foo stuff
--bar Do bar stuff
expert arguments:
--use-warp-drive
--no-fasten-seatbelts
... 50 more arguments
My first idea was to create two ArgumentParser-s, the basic one and the verbose one. First, parse_args of the basic was called. If --expert flag is present, the verbose parser is created and called. The approach has two downsides. First, -h is unconditionally handled by the first parser, so I should implement my own analogous help flag. Second, expert options are not even parsed without --expert flag, and I want them to be parsed unconditionally (but not to pop up in help).
Note: the solution should be python2 compliant.
Update: using the ideas from Lior Cohen's answer and this question, I created a working example. In my case it was easier to make a whitelist of "simple" options, so it was even unnecessary to use tagged action types.
Illustrating code follows.
class ExpertHelpFormatter(argparse.HelpFormatter):
skip_expert_section = True
whitelist = set(("foo",))
def add_usage(self, usage, actions, groups, prefix=None):
if self.skip_expert_section:
actions = [action for action in actions if action.dest in self.whitelist]
ret = super(ExpertHelpFormatter, self).add_usage(usage, actions, groups, prefix)
if self.skip_expert_section:
self.add_text("Use -hh for detailed help.")
return ret
def add_argument(self, action):
if self.skip_expert_section and action.dest not in self.whitelist:
return
super(ExpertHelpFormatter, self).add_argument(action)
def main():
parser = argparse.ArgumentParser(add_help=False, formatter_class=ExpertHelpFormatter)
parser.add_argument("-h", "--help", action="count", default=0)
parser.add_argument("--foo")
parser.add_argument("--use-warp-drive", action="store_true")
args = parser.parse_args()
if args.help == 1:
print parser.format_help()
return
elif args.help > 1:
ExpertHelpFormatter.skip_expert_section = False
print parser.format_help()
return

Here is a layout that should give you what you need, though it is not "free of work".
Disable default help by add_help = False (see here). this will let you still use -h and --help with action you want.
You need to "tag" your expert arguments by type that will store somehow the fact that foo is "'simple" and use-warp-drive is "expert". You can add an attribute to the Action or save a global dict in the parser itself.
Write an Help Formatter (you can see example). Then, according to the above "tag", you can suppress the "expert" help by returning "" in case of "simple" mode.
Hope this is helping.

How to make conditional arguments using argparse? [duplicate]

I have done as much research as possible but I haven't found the best way to make certain cmdline arguments necessary only under certain conditions, in this case only if other arguments have been given. Here's what I want to do at a very basic level:
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required=False) # only required if --argument is given
p.add_argument('-b', required=False) # only required if --argument is given
From what I have seen, other people seem to just add their own check at the end:
if args.argument and (args.a is None or args.b is None):
# raise argparse error here
Is there a way to do this natively within the argparse package?

I've been searching for a simple answer to this kind of question for some time. All you need to do is check if '--argument' is in sys.argv, so basically for your code sample you could just do:
import argparse
import sys
if __name__ == '__main__':
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required='--argument' in sys.argv) #only required if --argument is given
p.add_argument('-b', required='--argument' in sys.argv) #only required if --argument is given
args = p.parse_args()
This way required receives either True or False depending on whether the user as used --argument. Already tested it, seems to work and guarantees that -a and -b have an independent behavior between each other.

You can implement a check by providing a custom action for --argument, which will take an additional keyword argument to specify which other action(s) should become required if --argument is used.
import argparse
class CondAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
x = kwargs.pop('to_be_required', [])
super(CondAction, self).__init__(option_strings, dest, **kwargs)
self.make_required = x
def __call__(self, parser, namespace, values, option_string=None):
for x in self.make_required:
x.required = True
try:
return super(CondAction, self).__call__(parser, namespace, values, option_string)
except NotImplementedError:
pass
p = argparse.ArgumentParser()
x = p.add_argument("--a")
p.add_argument("--argument", action=CondAction, to_be_required=[x])
The exact definition of CondAction will depend on what, exactly, --argument should do. But, for example, if --argument is a regular, take-one-argument-and-save-it type of action, then just inheriting from argparse._StoreAction should be sufficient.
In the example parser, we save a reference to the --a option inside the --argument option, and when --argument is seen on the command line, it sets the required flag on --a to True. Once all the options are processed, argparse verifies that any option marked as required has been set.

Your post parsing test is fine, especially if testing for defaults with is None suits your needs.
http://bugs.python.org/issue11588 'Add "necessarily inclusive" groups to argparse' looks into implementing tests like this using the groups mechanism (a generalization of mutuall_exclusive_groups).
I've written a set of UsageGroups that implement tests like xor (mutually exclusive), and, or, and not. I thought those where comprehensive, but I haven't been able to express your case in terms of those operations. (looks like I need nand - not and, see below)
This script uses a custom Test class, that essentially implements your post-parsing test. seen_actions is a list of Actions that the parse has seen.
class Test(argparse.UsageGroup):
def _add_test(self):
self.usage = '(if --argument then -a and -b are required)'
def testfn(parser, seen_actions, *vargs, **kwargs):
"custom error"
actions = self._group_actions
if actions[0] in seen_actions:
if actions[1] not in seen_actions or actions[2] not in seen_actions:
msg = '%s - 2nd and 3rd required with 1st'
self.raise_error(parser, msg)
return True
self.testfn = testfn
self.dest = 'Test'
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind=Test)
g1.add_argument('--argument')
g1.add_argument('-a')
g1.add_argument('-b')
print(p.parse_args())
Sample output is:
1646:~/mypy/argdev/usage_groups$ python3 issue25626109.py --arg=1 -a1
usage: issue25626109.py [-h] [--argument ARGUMENT] [-a A] [-b B]
(if --argument then -a and -b are required)
issue25626109.py: error: group Test: argument, a, b - 2nd and 3rd required with 1st
usage and error messages still need work. And it doesn't do anything that post-parsing test can't.
Your test raises an error if (argument & (!a or !b)). Conversely, what is allowed is !(argument & (!a or !b)) = !(argument & !(a and b)). By adding a nand test to my UsageGroup classes, I can implement your case as:
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind='nand', dest='nand1')
arg = g1.add_argument('--arg', metavar='C')
g11 = g1.add_usage_group(kind='nand', dest='nand2')
g11.add_argument('-a')
g11.add_argument('-b')
The usage is (using !() to mark a 'nand' test):
usage: issue25626109.py [-h] !(--arg C & !(-a A & -b B))
I think this is the shortest and clearest way of expressing this problem using general purpose usage groups.
In my tests, inputs that parse successfully are:
''
'-a1'
'-a1 -b2'
'--arg=3 -a1 -b2'
Ones that are supposed to raise errors are:
'--arg=3'
'--arg=3 -a1'
'--arg=3 -b2'

For arguments I've come up with a quick-n-dirty solution like this.
Assumptions: (1) '--help' should display help and not complain about required argument and (2) we're parsing sys.argv
p = argparse.ArgumentParser(...)
p.add_argument('-required', ..., required = '--help' not in sys.argv )
This can easily be modified to match a specific setting.
For required positionals (which will become unrequired if e.g. '--help' is given on the command line) I've come up with the following: [positionals do not allow for a required=... keyword arg!]
p.add_argument('pattern', ..., narg = '+' if '--help' not in sys.argv else '*' )
basically this turns the number of required occurrences of 'pattern' on the command line from one-or-more into zero-or-more in case '--help' is specified.

Here is a simple and clean solution with these advantages:
No ambiguity and loss of functionality caused by oversimplified parsing using the in sys.argv test.
No need to implement a special argparse.Action or argparse.UsageGroup class.
Simple usage even for multiple and complex deciding arguments.
I noticed just one considerable drawback (which some may find desirable): The help text changes according to the state of the deciding arguments.
The idea is to use argparse twice:
Parse the deciding arguments instead of the oversimplified use of the in sys.argv test. For this we use a short parser not showing help and the method .parse_known_args() which ignores unknown arguments.
Parse everything normally while reusing the parser from the first step as a parent and having the results from the first parser available.
import argparse
# First parse the deciding arguments.
deciding_args_parser = argparse.ArgumentParser(add_help=False)
deciding_args_parser.add_argument(
'--argument', required=False, action='store_true')
deciding_args, _ = deciding_args_parser.parse_known_args()
# Create the main parser with the knowledge of the deciding arguments.
parser = argparse.ArgumentParser(
description='...', parents=[deciding_args_parser])
parser.add_argument('-a', required=deciding_args.argument)
parser.add_argument('-b', required=deciding_args.argument)
arguments = parser.parse_args()
print(arguments)

Until http://bugs.python.org/issue11588 is solved, I'd just use nargs:
p = argparse.ArgumentParser(description='...')
p.add_argument('--arguments', required=False, nargs=2, metavar=('A', 'B'))
This way, if anybody supplies --arguments, it will have 2 values.
Maybe its CLI result is less readable, but code is much smaller. You can fix that with good docs/help.

This is really the same as #Mira 's answer but I wanted to show it for the case where when an option is given that an extra arg is required:
For instance, if --option foo is given then some args are also required that are not required if --option bar is given:
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--option', required=True,
help='foo and bar need different args')
if 'foo' in sys.argv:
parser.add_argument('--foo_opt1', required=True,
help='--option foo requires "--foo_opt1"')
parser.add_argument('--foo_opt2', required=True,
help='--option foo requires "--foo_opt2"')
...
if 'bar' in sys.argv:
parser.add_argument('--bar_opt', required=True,
help='--option bar requires "--bar_opt"')
...
It's not perfect - for instance proggy --option foo --foo_opt1 bar is ambiguous but for what I needed to do its ok.

Add additional simple "pre"parser to check --argument, but use parse_known_args() .
pre = argparse.ArgumentParser()
pre.add_argument('--argument', required=False, action='store_true', default=False)
args_pre=pre.parse_known_args()
p = argparse.ArgumentParser()
p.add_argument('--argument', required=False)
p.add_argument('-a', required=args_pre.argument)
p.add_argument('-b', required=not args_pre.argument)

Specify default subcommand with positional arguments

I have read a similar question asked on SO, which doesn't address my own problem. For illustration purpose, say I have a Python program using argpase that provides two subcommands: copy and resume:
prog copy src dest # src, dest positional, required
prog resume id # id positional, required
However, the most natural way to invoke the "copy" command is NOT explicitly give the copy subcommand, that is, I was hoping for:
prog src dest
will do the default copy action, while still keep the benefits of have two subparsers, each handles a different set of arguments. Is it possible with argparse package?

Formally there isn't. The subcommand argument is a required positional argument, where the 'choices' are the subparser names (and their aliases).
That's evident in the help message, were {cmd1,cmd} are shown as choices.
usage: ipython3 [-h] {cmd1,cmd2} ...
positional arguments:
{cmd1,cmd2}
optional arguments:
-h, --help show this help message and exit
The latest Python has a 'bug' that actually lets it be optional. In other words, it does not raise an error if you don't give any positionals. It's a bug because it changes previous behavior, and most people what it to be required. But even when it is optional, and cmd1 defined as the default, it won't run the cmd1 parser on the remaining arguments. And any 'positional' argument will be treated as a wrong command string.
I think the best you can do is define one or more positionals. One might have choices and default, and be optional (nargs='?'). The rest (or other) might has nargs='+'. By playing around with those options you could approximate the desired behavior, but it won't make use of the subparsers mechanism.
Another way to think about the issue, is to consider an input like
prog one two
Should it interpret that as prog src dest or prog cmd id. The only basis for saying it should be the former is the fact that one is not one of copy or resume. But it isn't that smart when it comes to handling 'choices'. It assigns a value based on position, and then tests whether it meets criteria like 'choices' or 'type' (ie. integer, float v string).

Allowing same option in different argparser group

Is it possible to allow the same argparse option to be in 2 different argparser groups?
This is actually what I want to achieve:-
#!/usr/bin/env python
import argparse
... ... ...
g1 = parser.add_argument_group('g1')
g2 = parser.add_argument_group('g2')
g1.add_argument('--aa')
g1.add_argument('--common')
g2.add_argument('--bb')
g2.add_argument('--common')
... and the printed out help looks like this:-
usage: ...
... ... ...
g1:
--aa [aa]
--common [common]
g2:
-bb [bb]
--common [common]
But that is not possible, as argparse complains 'conflicting option string'

http://bugs.python.org/issue10984 discusses adding an argument to two different mutually_exclusive_groups. Doing the same with argument_groups is similar, though simpler.
import argparse
parser=argparse.ArgumentParser()
g1=parser.add_argument_group('group1')
g1.add_argument('-a')
caction=g1.add_argument('-c')
g2=parser.add_argument_group('group2')
g2.add_argument('-b')
g2._group_actions.append(caction)
parser.print_help()
It is a kludge, in the sense that it is modifying a 'private' attribute of the group.
The result:
usage: ipython [-h] [-a A] [-c C] [-b B]
optional arguments:
-h, --help show this help message and exit
group1:
-a A
-c C
group2:
-b B
-c C
Here's what's going on. add_argument creates an Action, registers it with the parser, and returns it. That's what caction captures. If added to a group, it also registers the action with the group - by adding it to a _group_actions list.
If you do g2.add_argument('-c') you get an error because the parser already has an action with that option string. The fact that you are 'adding' it to a different group is incidental. The kludge gets around that by adding it to the group's list, without creating a new action.
In case it isn't obvious from the documentation, argument_groups are basically a 'help' convenience. They do not affect parsing at all. There are other ways you could customize the help. For example, add --common to the parser, possibly with a SUPPRESS help line. Then include a mention of it in the description for each group.

In this special case, you could make --aa and --bb mutually exclusive.
import argparse
parser = argparse.ArgumentParser()
mutex = parser.add_mutually_exclusive_group()
mutex.add_argument('--aa', action = 'store_true')
mutex.add_argument('--bb', action = 'store_true')
parser.add_argument('--common', action = 'store_true')
args = parser.parse_args()
which results in
usage: a.py [-h] [--aa | --bb] [--common]
Generally, argparse has the problem that you must create an option when calling add_argument.
Here's a somewhat related topic: Does argparse (python) support mutually exclusive groups of arguments?
There's a patch that allows you to have one argument in more than one mutually exclusive group:
http://bugs.python.org/issue10984
Another thing you can do is fiddle around with the argparse subparsers, or use another commandline parser altogether.

argparse optional argument before positional argument

I was wondering if it is possible to have a positional argument follow an argument with an optional parameter. Ideally the last argument entered into the command line would always apply toward 'testname'.
import argparse
parser = argparse.ArgumentParser(description='TAF')
parser.add_argument('-r','--release',nargs='?',dest='release',default='trunk')
parser.add_argument('testname',nargs='+')
args = parser.parse_args()
I would like both of these calls to have smoketest apply to testname, but the second one results in an error.
>> python TAF.py -r 1.0 smoketest
>> python TAF.py -r smoketest
TAF.py: error: too few arguments
I realize that moving the positional argument to the front would result in the correct behavior of the optional parameter, however this is not quite the format I am looking for. The choices flag looks like an attractive alternative, however it throws an error instead of ignoring the unmatched item.
EDIT:
I've found a hacky way around this. If anyone has a nicer solution I would appreciate it.
import argparse
parser = argparse.ArgumentParser(description='TAF')
parser.add_argument('-r','--release',nargs='?',dest='release',default='trunk')
parser.add_argument('testname',nargs=argparse.REMAINDER)
args = parser.parse_args()
if not args.testname:
args.testname = args.release
args.release = ''

As stated in the documentation:
'?'. One argument will be consumed from the command line if possible,
and produced as a single item. If no command-line argument is present,
the value from default will be produced. Note that for optional
arguments, there is an additional case - the option string is present
but not followed by a command-line argument. In this case the value
from const will be produced.
So, the behaviour you want is not obtainable using '?'. Probably you could write some hack using argparse.Action and meddling with the previous results.(1)
I think the better solution is to split the functionality of that option. Make it an option that requires an argument(but the option itself is optional) and add an option without argument that sets the release to 'trunk'. In this way you can obtain the same results without any hack. Also I think the interface is simpler.
In your example:
python TAF.py -r smoketest
It's quite clear that smoketest will be interpreted as an argument to -r. At least following unix conventions. If you want to keep nargs='?' then the user must use --:
$ python TAF.py -r -- sometest
Namespace(release=None, testname=['sometest']) #parsed result
(1) An idea on how to do this: check if the option has an argument. If it has one check if it is a valid test name. If so put into by hand into testname and set release to the default value. You'll also have to set a "flag" that tells you that this thing happened.
Now, before parsing sys.argv you must redirect sys.stderr. When doing the parsing you must catch SystemExit, check the stderr and see if the error was "too few arguments", check if the flag was set, if so ignore the error and continue running, otherwise you should reprint to the original stderr the error message and exit.
This approach does not look robust, and it's probably buggy.