I am trying to write a function wo which you can parse a variable amount of arguments via argparse - I know I can do this via nargs="+". Sadly, the way argparse help works (and the way people generally write arguments in the CLI) puts the positional arguments last. This leads to my positional argument being caught as part of the optional arguments.
#!/usr/bin/python
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("positional", help="my positional arg", type=int)
parser.add_argument("-o", "--optional", help="my optional arg", nargs='+', type=float)
args = parser.parse_args()
print args.positional, args.optional
running this as ./test.py -h shows the following usage instruction:
usage: test.py [-h] [-o OPTIONAL [OPTIONAL ...]] positional
but if I run ./test.py -o 0.21 0.11 0.33 0.13 100 gives me
test.py: error: too few arguments
to get a correct parsing of args, I have to run ./test.py 100 -o 0.21 0.11 0.33 0.13
So how do I:
make argparse reformat the usage output so that it is less misleading, OR, even better:
tell argparse to not catch the last element for the optional argument -o if it is the last in the list
?
There is a bug report on this: http://bugs.python.org/issue9338
argparse optionals with nargs='?', '*' or '+' can't be followed by positionals
A simple (user) fix is to use -- to separate postionals from optionals:
./test.py -o 0.21 0.11 0.33 0.13 -- 100
I wrote a patch that reserves some of the arguments for use by the positional. But it isn't a trivial one.
As for changing the usage line - the simplest thing is to write your own, e.g.:
usage: test.py [-h] positional [-o OPTIONAL [OPTIONAL ...]]
usage: test.py [-h] [-o OPTIONAL [OPTIONAL ...]] -- positional
I wouldn't recommend adding logic to the usage formatter to make this sort of change. I think it would get too complex.
Another quick fix is to turn this positional into an (required) optional. It gives the user complete freedom regarding their order, and might reduce confusion. If you don't want to confusion of a 'required optional' just give it a logical default.
usage: test.py [-h] [-o OPTIONAL [OPTIONAL ...]] -p POSITIONAL
usage: test.py [-h] [-o OPTIONAL [OPTIONAL ...]] [-p POS_WITH_DEFAULT]
One easy change to the Help_Formatter is to simply list the arguments in the order that they are defined. The normal way of modifying formatter behavior is to subclass it, and change one or two methods. Most of these methods are 'private' (_ prefix), so you do so with the realization that future code might change (slowly).
In this method, actions is the list of arguments, in the order in which they were defined. The default behavior is to split 'optionals' from 'positionals', and reassemble the list with positionals at the end. There's additional code that handles long lines that need wrapping. Normally it puts positionals on a separate line. I've omitted that.
class Formatter(argparse.HelpFormatter):
# use defined argument order to display usage
def _format_usage(self, usage, actions, groups, prefix):
if prefix is None:
prefix = 'usage: '
# if usage is specified, use that
if usage is not None:
usage = usage % dict(prog=self._prog)
# if no optionals or positionals are available, usage is just prog
elif usage is None and not actions:
usage = '%(prog)s' % dict(prog=self._prog)
elif usage is None:
prog = '%(prog)s' % dict(prog=self._prog)
# build full usage string
action_usage = self._format_actions_usage(actions, groups) # NEW
usage = ' '.join([s for s in [prog, action_usage] if s])
# omit the long line wrapping code
# prefix with 'usage:'
return '%s%s\n\n' % (prefix, usage)
parser = argparse.ArgumentParser(formatter_class=Formatter)
Which produces a usage line like:
usage: stack26985650.py [-h] positional [-o OPTIONAL [OPTIONAL ...]]
Instead of using nargs="+", consider using action="append". This requires passing -o in front of each number, but it will not consume arguments unless you actually want it to.
Related
Having a bit of problem with argparse in Python...
import argparse
parser = argparse.ArgumentParser()
parser.add_argument ("-o", "--optional", help="this is an optional argument")
args = parser.parse_args()
print ( args.optional )
Calling test.py -h will output...
usage: test.py [-h] [-o OPTIONAL]
optional arguments:
-h, --help show this help message and exit
-o OPTIONAL, --optional OPTIONAL
this is an optional argument
Is there any way I can get rid of the extra OPTIONALs in the help menu? I know I could do this with parser.add_argument ("-o", "--optional", help="this is an optional argument", action=store_true), except I can't because I need to call args.optional later on.
Again, this isn't so much about the functionality of the program as the aesthetics because test.py -o hello would print hello.
Usually an option with no arguments has an action, which will suppress that metavar:
parser.add_argument ("-o", "--optional", action='store_true')
Otherwise, you could amend the argument like this:
parser.add_argument ("-o", "--optional", metavar='', help="the help text")
First is this about the parsing of this argument or the display in the help?
parser.add_argument ("-o", "--optional", help="this is an optional argument")
has the default store_true action, and thus takes one argument. As indicated in the usage:
usage: test.py [-h] [-o OPTIONAL]
where 'OPTIONAL' is a standin for the string that you will include after the -o or --optional. And args.optional will have the value of that string.
action='store_true turns this argument into a boolean, False if not given, True if -o is provided. It takes to no added value.
-o OPTIONAL, --optional OPTIONAL
is the normal way an Action like this is displayed in the help. Again OPTIONAL is the place marker for the string that follows -o or --optional. A metavar parameter can be used to customize that place marker. It can be as short as "".
Some people don't like that repeated pattern, preferring something like
-o , --optional OPTIONAL
That has been discussed in previous questions. It requires a change to the HelpFormatter class (ie. a subclassing).
python argparse help message, disable metavar for short options?
Another thing is to simplify the definition
parser.add_argument ("-o", dest="optional", help="this is an optional argument")
I'm using argparse and I have a custom argument group required arguments. Is there any way to change the order of the argument groups in the help message? I think it is more logical to have the required arguments before optional arguments, but haven't found any documentation or questions to help.
For example, changing this:
usage: foo.py [-h] -i INPUT [-o OUTPUT]
Foo
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file name
required arguments:
-i INPUT, --input INPUT
Input file name
to this:
usage: foo.py [-h] -i INPUT [-o OUTPUT]
Foo
required arguments:
-i INPUT, --input INPUT
Input file name
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file name
(example taken from this question)
You might consider adding an explicit optional arguments group:
import argparse
parser = argparse.ArgumentParser(description='Foo', add_help=False)
required = parser.add_argument_group('required arguments')
required.add_argument('-i', '--input', help='Input file name', required=True)
optional = parser.add_argument_group('optional arguments')
optional.add_argument("-h", "--help", action="help", help="show this help message and exit")
optional.add_argument('-o', '--output', help='Output file name', default='stdout')
parser.parse_args(['-h'])
You can move the help action to your optional group as
described here:
Move "help" to a different Argument Group in python argparse
As you can see, the code produces the required output:
usage: code.py -i INPUT [-h] [-o OUTPUT]
Foo
required arguments:
-i INPUT, --input INPUT
Input file name
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file name
This is admittedly a hack, and is reliant on the changeable internal implementation, but after adding the arguments, you can simply do:
parser._action_groups.reverse()
This will effectively make the required arguments group display above the optional arguments group. Note that this answer is only meant to be descriptive, not prescriptive.
Credit: answer by hpaulj
The parser starts out with 2 argument groups, the usual positional and optionals. The -h help is added to optionals. When you do add_argument_group, a group is created (and returned to you). It is also appended to the parser._action_groups list.
When you ask for help (-h) parser.format_help() is called (you can do that as well in testing). Look for that method in argparse.py. That sets up the help message, and one step is:
# positionals, optionals and user-defined groups
for action_group in self._action_groups:
formatter.start_section(action_group.title)
formatter.add_text(action_group.description)
formatter.add_arguments(action_group._group_actions)
formatter.end_section()
So if we reorder the items in the parser._action_groups list, we will reorder the groups in the display. Since this is the only use of _action_groups it should be safe and easy. But some people aren't allowed to peak under the covers (look or change ._ attributes).
The proposed solution(s) is to make your own groups in the order you want to see them, and make sure that the default groups are empty (the add_help=False parameter). That's the only way to do this if you stick with the public API.
Demo:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('foo')
g1 = parser.add_argument_group('REQUIRED')
g1.add_argument('--bar', required=True)
g1.add_argument('baz', nargs=2)
print(parser._action_groups)
print([group.title for group in parser._action_groups])
print(parser.format_help())
parser._action_groups.reverse() # easy inplace change
parser.print_help()
Run result:
1504:~/mypy$ python stack39047075.py
_actions_group list and titles:
[<argparse._ArgumentGroup object at 0xb7247fac>,
<argparse._ArgumentGroup object at 0xb7247f6c>,
<argparse._ArgumentGroup object at 0xb721de0c>]
['positional arguments', 'optional arguments', 'REQUIRED']
default help:
usage: stack39047075.py [-h] --bar BAR foo baz baz
positional arguments:
foo
optional arguments:
-h, --help show this help message and exit
REQUIRED:
--bar BAR
baz
after reverse:
usage: stack39047075.py [-h] --bar BAR foo baz baz
REQUIRED:
--bar BAR
baz
optional arguments:
-h, --help show this help message and exit
positional arguments:
foo
1504:~/mypy$
Another way to implement this is to define a ArgumentParser subclass with a new format_help method. In that method reorder the list used in that for action_group... loop.
This program:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('files', metavar='INPUT', nargs='*',
help='File(s) containing words to include. If none given, stdin will be used.')
parser.add_argument('-x', '--exclude', nargs='*',
help='File(s) containing words to exclude.')
args = parser.parse_args()
print args.files
print args.exclude
produces this output when run in Python 2.7.9:
$ python prog.py --help
usage: prog.py [-h] [-x [EXCLUDE [EXCLUDE ...]]] [INPUT [INPUT ...]]
positional arguments:
INPUT File(s) containing words to include. If
none given, stdin will be used.
optional arguments:
-h, --help show this help message and exit
-x [EXCLUDE [EXCLUDE ...]], --exclude [EXCLUDE [EXCLUDE ...]]
File(s) containing words to exclude.
However, that "help" output instructs the user to use a nonsensical ordering for the arguments. It is nonsensical because if the -x option is used, then no INPUT arguments will be detected.
Argparse ought instead to advise the user to use this ordering:
usage: prog.py [-h] [INPUT [INPUT ...]] [-x [EXCLUDE [EXCLUDE ...]]]
Two questions:
Is this a bug in argparse? (I think it is.)
Regardless of whether it is a bug, how can I fix it so that $ python prog.py --help will output the help text I desire (see above), preferably in as DRY a way as possible?
When generating the usage line, flagged arguments are placed first, positional after. That fits with common commandline usage. It makes no effort to evaluate whether that is the best choice or not.
The simplest way around this is to provide a custom usage parameter.
(there have be SO questions about changing the order of arguments in the usage line. The solution requires customization of the HelpFormatter class. Change argparse usage message argument order
)
As you note, when the * positional is placed after the * optional, all arguments are assigned to the optional. You could use a '--' to separate the two lists of arguments.
There is a bug/issue with patch that should improve handling when the positional takes a known number of arguments (e.g. the default one). It does so by noting that the positional requires an argument, so it reserves one for it. But in the * * case, the positional is satisfied with none, so it has no way of knowing how to split the arguments between the 2.
I like the idea of turning that postional into a flagged argument. That should reduce the ambiguity inherent in lots of * arguments.
Add '-f', '--files' to the input option:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-f', '--files', metavar='INPUT', nargs='*', required=True,
help='File(s) containing words to include. If none given, stdin will be used.')
parser.add_argument('-x', '--exclude', nargs='*',
help='File(s) containing words to exclude.')
args = parser.parse_args()
print args.files
shows:
usage: argparse_test.py [-h] [-f [INPUT [INPUT ...]]]
[-x [EXCLUDE [EXCLUDE ...]]]
optional arguments:
-h, --help show this help message and exit
-f [INPUT [INPUT ...]], --files [INPUT [INPUT ...]]
File(s) containing words to include. If none given,
stdin will be used.
-x [EXCLUDE [EXCLUDE ...]], --exclude [EXCLUDE [EXCLUDE ...]]
File(s) containing words to exclude.
print args.exclude
You can make 'files' required. From the docs:
In general, the argparse module assumes that flags like -f and --bar indicate optional arguments, which can always be omitted at the command line. To make an option required, True can be specified for the required= keyword argument to add_argument():
The simplest way is to add usage="..." to argparse.ArgumentParser().
By viewing the source of argparse, I found a way to resort arguments which might be a little bit dirty:
class MyHelpFormatter(argparse.HelpFormatter):
def _format_actions_usage(self, actions, groups):
actions.sort(key=lambda a: bool(a.option_strings and a.nargs != 0))
return super(MyHelpFormatter, self)._format_actions_usage(actions, groups)
parser = argparse.ArgumentParser(formatter_class = MyHelpFormatter)
Is it possible to allow the same argparse option to be in 2 different argparser groups?
This is actually what I want to achieve:-
#!/usr/bin/env python
import argparse
... ... ...
g1 = parser.add_argument_group('g1')
g2 = parser.add_argument_group('g2')
g1.add_argument('--aa')
g1.add_argument('--common')
g2.add_argument('--bb')
g2.add_argument('--common')
... and the printed out help looks like this:-
usage: ...
... ... ...
g1:
--aa [aa]
--common [common]
g2:
-bb [bb]
--common [common]
But that is not possible, as argparse complains 'conflicting option string'
http://bugs.python.org/issue10984 discusses adding an argument to two different mutually_exclusive_groups. Doing the same with argument_groups is similar, though simpler.
import argparse
parser=argparse.ArgumentParser()
g1=parser.add_argument_group('group1')
g1.add_argument('-a')
caction=g1.add_argument('-c')
g2=parser.add_argument_group('group2')
g2.add_argument('-b')
g2._group_actions.append(caction)
parser.print_help()
It is a kludge, in the sense that it is modifying a 'private' attribute of the group.
The result:
usage: ipython [-h] [-a A] [-c C] [-b B]
optional arguments:
-h, --help show this help message and exit
group1:
-a A
-c C
group2:
-b B
-c C
Here's what's going on. add_argument creates an Action, registers it with the parser, and returns it. That's what caction captures. If added to a group, it also registers the action with the group - by adding it to a _group_actions list.
If you do g2.add_argument('-c') you get an error because the parser already has an action with that option string. The fact that you are 'adding' it to a different group is incidental. The kludge gets around that by adding it to the group's list, without creating a new action.
In case it isn't obvious from the documentation, argument_groups are basically a 'help' convenience. They do not affect parsing at all. There are other ways you could customize the help. For example, add --common to the parser, possibly with a SUPPRESS help line. Then include a mention of it in the description for each group.
In this special case, you could make --aa and --bb mutually exclusive.
import argparse
parser = argparse.ArgumentParser()
mutex = parser.add_mutually_exclusive_group()
mutex.add_argument('--aa', action = 'store_true')
mutex.add_argument('--bb', action = 'store_true')
parser.add_argument('--common', action = 'store_true')
args = parser.parse_args()
which results in
usage: a.py [-h] [--aa | --bb] [--common]
Generally, argparse has the problem that you must create an option when calling add_argument.
Here's a somewhat related topic: Does argparse (python) support mutually exclusive groups of arguments?
There's a patch that allows you to have one argument in more than one mutually exclusive group:
http://bugs.python.org/issue10984
Another thing you can do is fiddle around with the argparse subparsers, or use another commandline parser altogether.
I am trying to use argparse with subparser to switch between 3 fonctionnalities whereas one positional argument should be common to all subparser. Moreover, and it is the key point, i want to put the positional argument as the last argument provided as this one is an output file path. It makes no sense to me to put it at the beginning (as first argument)
import sys,argparse,os
files = argparse.ArgumentParser(add_help=False)
files.add_argument('outfile', help='output mesh file name')
parser = argparse.ArgumentParser(description="A data interpolation program.",prog='data_interpolate.py', parents=[files])
subparsers = parser.add_subparsers(help='Mode command.')
command_parser = subparsers.add_parser('cmd',help='Pass all argument in command line.',parents=[files])
command_parser.add_argument('-min', dest='MINFILE',help='Input file with min values', required=True)
command_parser.add_argument('-max', dest='MAXFILE',help='Input file with min values', required=True)
command_parser.add_argument('u', help='Interpolation parameter. Float between 0 and 1. Out of bound values are limited to 0 or 1.')
subparsers.add_parser('py',help='Pass all argument in python file.',parents=[files])
subparsers.add_parser('json',help='Pass all argument in json file.',parents=[files])
Which gives:
data_interpolation.py -h
usage: data_interpolation.py [-h] outfile {cmd,py,json}
But, to my opinion, the outfile should be given at the end following:
data_interpolation.py [-h] {cmd,py,json} outfile
This has even more sense when using the cmd command as I need to pass other parameter values. For intance:
data_interpolation.py cmd -min minfile.txt -max maxfile.txt 0.6 outfile.txt
How can I set up argparse to have such behaviour?
(note - this is an old question).
The order of positionals is determined by the order in which they are defined. That includes the subparsers argument (which is a positional with choices and a special action).
Defining outfile as an argument to both the main parser and the subparsers is redundant.
Positionals defined via parents will be placed first. So if 'outfile' must be last, it has to be defined separately for each subparser.
It could also be specified last as postional for the main parser (after the subparser definitions).
In [2]: p=argparse.ArgumentParser()
In [5]: sp=p.add_subparsers(dest='cmd')
In [6]: spp=sp.add_parser('cmd1')
In [7]: spp.add_argument('test')
In [8]: p.add_argument('out')
In [9]: p.print_help()
usage: ipython [-h] {cmd1} ... out
...
In [11]: spp.print_help()
usage: ipython cmd1 [-h] test
...
In [15]: p.parse_args('cmd1 test out'.split())
Out[15]: Namespace(cmd='cmd1', out='out', test='test')
cmd1 is interpreted as a subparser choice. test is interpreted by the subparser as a positional. out is left over, and returned to the main parser to use as it sees fit. This parsing could be messed up if the subparser does not return any extras. So I'd be wary of specifying a final positional like this.
You don't need to specify files as a parent of parser, just for each of the subparsers.