Working with Python and argparse, trying to accomplish the following:
$> my_app
Namespace(positional=None)
$> my_app file.txt somedir
Namespace(positional=['file.txt', 'somedir'])
i.e., a positional argument of type list, whose default is None. I would expect the following code to accomplish this:
p = argparse.ArgumentParser()
p.add_argument("positional", nargs='*', default=None)
print(p.parse_args())
But I get:
$> my_app
Namespace(positional=[])
$> my_app file.txt somedir
Namespace(positional=['file.txt', 'somedir'])
The rest of my code uses None as defaults for lists. If None is provided, the code select a sensible default. Thus passing [] is not an option.
The behavior of argparse does rather feel like a bug, but maybe I'm missing something. Any thoughts?
Answer (thx hpaulj) seems to be "as intended" to which I would add "completely unintuitively".
You can achieve the desired behavior like this
import argparse
p = argparse.ArgumentParser()
p.add_argument("positional", nargs='*', default=argparse.SUPPRESS)
print(p.parse_args(namespace=argparse.Namespace(positional=None)))
This prevents the arguments from appearing at all in the namespace which on turn causes the default namespace to take over.
Your case is handled in
def _get_values(self, action, arg_strings):
...
# when nargs='*' on a positional, if there were no command-line
# args, use the default if it is anything other than None
elif (not arg_strings and action.nargs == ZERO_OR_MORE and
not action.option_strings):
if action.default is not None:
value = action.default
self._check_value(action, value)
else:
# since arg_strings is always [] at this point
# there is no need to use self._check_value(action, value)
value = arg_strings
Normally the default is placed in the Namespace at the start of parsing. During parsing that default is overwritten with value(s) provided by the user. optionals are 'seen' when the right flag is used. positionals are 'seen' when the right number of strings are present. Since '*' and '?' accept 0 strings, they will always be 'seen'. Thus their defaults require special handling, if at all.
Related
I am attempting to get my script working, but argparse keeps overwriting my positional arguments from the parent parser. How can I get argparse to honor the parent's value for these? It does keep values from optional args.
Here is a very simplified version of what I need. If you run this, you will see that the args are overwritten.
testargs.py
#! /usr/bin/env python3
import argparse
import sys
def main():
preparser = argparse.ArgumentParser(add_help=False)
preparser.add_argument('first',
nargs='?')
preparser.add_argument('outfile',
nargs='?',
type=argparse.FileType('w', encoding='utf-8'),
default=sys.stdout,
help='Output file')
preparser.add_argument(
'--do-something','-d',
action='store_true')
# Parse args with preparser, and find config file
args, remaining_argv = preparser.parse_known_args()
print(args)
parser = argparse.ArgumentParser(
parents=[preparser],
description=__doc__)
parser.add_argument(
'--clear-screen', '-c',
action='store_true')
args = parser.parse_args(args=remaining_argv,namespace=args )
print(args)
if __name__ == '__main__':
main()
And call it with testargs.py something /tmp/test.txt -d -c
You will see it keeps the -d but drops both the positional args and reverts them to defaults.
EDIT: see additional comments in the accepted answer for some caveats.
When you specify parents=[preparser] it means that parser is an extension of preparser, and will parse all arguments relevent to preparser which it is never given.
Lets say the preparser only has one positional argument first and the parser only has one positional argument second, when you make parser a child of preparser it expects both arguments:
import argparse
parser1 = argparse.ArgumentParser(add_help=False)
parser1.add_argument("first")
parser2 = argparse.ArgumentParser(parents=[parser1])
parser2.add_argument("second")
args2 = parser2.parse_args(["arg1","arg2"])
assert args2.first == "arg1" and args2.second == "arg2"
However passing only the remaining arguments that are left over from parser1 would just be ['second'] which is not the correct arguments to parser2:
parser1 = argparse.ArgumentParser(add_help=False)
parser1.add_argument("first")
args1, remaining_args = parser1.parse_known_args(["arg1","arg2"])
parser2 = argparse.ArgumentParser(parents=[parser1])
parser2.add_argument("second")
>>> args1
Namespace(first='arg1')
>>> remaining_args
['arg2']
>>> parser2.parse_args(remaining_args)
usage: test.py [-h] first second
test.py: error: the following arguments are required: second
To only process the arguments that were not handled by the first pass, do not specify it as the parent to the second parser:
parser1 = argparse.ArgumentParser(add_help=False)
parser1.add_argument("first")
args1, remaining_args = parser1.parse_known_args(["arg1","arg2"])
parser2 = argparse.ArgumentParser() #parents=[parser1]) #NO PARENT!
parser2.add_argument("second")
args2 = parser2.parse_args(remaining_args,args1)
assert args2.first == "arg1" and args2.second == "arg2"
The 2 positionals are nargs='?'. A positional like that is always 'seen', since an empty list matches that nargs.
First time through 'text.txt' matches with first and is put in the Namespace. Second time through there isn't any string to match, so the default is used - same as if you had not given that string the first time.
If I change first to have the default nargs, I get
error: the following arguments are required: first
from the 2nd parser. Even though there's a value in the Namespace it still tries to get a value from the argv. (it's like a default, but not quite).
Defaults for positionals with nargs='?' (or *) are tricky. They are optional, but not in quite the same way as optionals. The positional Actions are still called, but with a empty list of values.
I don't think the parents feature does anything for you. preparser already handles that set of arguments; there's no need to handle them again in parser, especially since all the relevant argument strings have been stripped out.
Another option is to leave the parents in, but use the default sys.argv[1:] in the 2nd parser. (but beware of side effects like opening files)
args = parser.parse_args(namespace=args )
A third option is to parse the arguments independently and merge them with a dictionary update.
adict = vars(preparse_args)
adict.update(vars(parser_args))
# taking some care in who overrides who
For more details look in argparse.py file at ArgumentParser._get_values, specifically the not arg_strings cases.
A note about the FileType. That type works nicely for small scripts where you will use the files right away and exit. It isn't so good on large programs where you might want to close the file after use (close stdout???), or use files in a with context.
edit - note on parents
add_argument creates an Action object, and adds it to the parser's list of actions. parse_args basically matches input strings with these actions.
parents just copies those Action objects (by reference) from parent to child. To the child parser it is just as though the actions were created with add_argument directly.
parents is most useful when you are importing a parser and don't have direct access to its definition. If you are defining both parent and child, then parents just saves you some typing/cut-n-paste.
This and other SO questions (mostly triggered the by-reference copy) show that the developers did not intend you to use both the parent and child to do parsing. It can be done, but there are glitches that the they did not consider.
===================
I can imagine defining a custom Action class that would 'behave' in a situation like this. It might, for example, check the namespace for some not default value before adding its own (possibly default) value.
Consider, for example if I changed the action of first to 'append':
preparser.add_argument('first', action='append', nargs='?')
The result is:
1840:~/mypy$ python3 stack37147683.py /tmp/test.txt -d -c
Namespace(do_something=True, first=['/tmp/test.txt'], outfile=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)
Namespace(clear_screen=True, do_something=True, first=['/tmp/test.txt', None], outfile=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)
From the first parser, first=['/tmp/test.txt']; from the second, first=['/tmp/test.txt', None].
Because of the append, the item from the first is preserved, and a new default has been added by the second parser.
Here is my argparse sample say sample.py
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-p", nargs="+", help="Stuff")
args = parser.parse_args()
print args
Python - 2.7.3
I expect that the user supplies a list of arguments separated by spaces after the -p option. For example, if you run
$ sample.py -p x y
Namespace(p=['x', 'y'])
But my problem is that when you run
$ sample.py -p x -p y
Namespace(p=['y'])
Which is neither here nor there. I would like one of the following
Throw an exception to the user asking him to not use -p twice instead just supply them as one argument
Just assume it is the same option and produce a list of ['x','y'].
I can see that python 2.7 is doing neither of them which confuses me. Can I get python to do one of the two behaviours documented above?
Note: python 3.8 adds an action="extend" which will create the desired list of ['x','y']
To produce a list of ['x','y'] use action='append'. Actually it gives
Namespace(p=[['x'], ['y']])
For each -p it gives a list ['x'] as dictated by nargs='+', but append means, add that value to what the Namespace already has. The default action just sets the value, e.g. NS['p']=['x']. I'd suggest reviewing the action paragraph in the docs.
optionals allow repeated use by design. It enables actions like append and count. Usually users don't expect to use them repeatedly, or are happy with the last value. positionals (without the -flag) cannot be repeated (except as allowed by nargs).
How to add optional or once arguments? has some suggestions on how to create a 'no repeats' argument. One is to create a custom action class.
I ran into the same issue. I decided to go with the custom action route as suggested by mgilson.
import argparse
class ExtendAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if getattr(namespace, self.dest, None) is None:
setattr(namespace, self.dest, [])
getattr(namespace, self.dest).extend(values)
parser = argparse.ArgumentParser()
parser.add_argument("-p", nargs="+", help="Stuff", action=ExtendAction)
args = parser.parse_args()
print args
This results in
$ ./sample.py -p x -p y -p z w
Namespace(p=['x', 'y', 'z', 'w'])
Still, it would have been much neater if there was an action='extend' option in the library by default.
With Python's argparse, I would like to add an optional argument that, if not given, gets the value of another (mandatory) argument.
parser.add_argument('filename',
metavar = 'FILE',
type = str,
help = 'input file'
)
parser.add_argument('--extra-file', '-f',
metavar = 'ANOTHER_FILE',
type = str,
default = ,
help = 'complementary file (default: FILE)'
)
I could of course manually check for None after the arguments are parsed, but isn't there a more pythonic way of doing this?
As far as I know, there is no way to do this that is more clean than:
ns = parser.parse_args()
ns.extra_file = ns.extra_file if ns.extra_file else ns.filename
(just like you propose in your question).
You could probably do some custom action gymnastics similar to this, but I really don't think that would be worthwhile (or "pythonic").
This has slightly different semantics than your original set of options, but may work for you:
parse.add_argument('filename', action='append', dest='filenames')
parse.add_argument('--extra-file', '-f', action='append', dest='filenames')
args = parse.parse_args()
This would replace args.filename with a list args.filenames of at least one file, with -f appending its argument to that list. Since it's possible to specify -f on the command line before the positional argument, you couldn't on any particular order of the input files in args.filenames.
Another option would be to dispense with the -f option and make filename a multi-valued positional argument:
parse.add_argument('filenames', nargs='+')
Again, args.filenames would be a list of at least one filename. This would also interfere if you have other positional arguments for your script.
I want to use argparse to parse command lines of form "arg=val"
For example, the usage would be:
script.py conf_dir=/tmp/good_conf
To achieve it, I am doing this:
desc = "details"
parser = argparse.ArgumentParser(description=desc, add_help=False)
args = parser.add_argument("conf_dir")
args = parser.parse_args("conf_dir=FOO".split())
args = parser.parse_args()
print args.conf_dir
But, the problem is that, on invocation of the script with:
python script.py conf_dir=/tmp/good_conf
I get:
conf_dir=/tmp/good_conf
Where as I expect
/tmp/good_conf
So, the question is: Can I use argparse to parse cmd line, which contains name value pairs?
Any hints?
Edit: The reason I want to do this and not some thing like --conf_dir=/tmp/good_dir is because there are other tools (written in other language), which uses conf_dir=/tmp/good_dir style of arguments. To maintain consistency, I was to parse args in this way.
You need a custom action
class StoreNameValuePair(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
n, v = values.split('=', 1)
setattr(namespace, n, v)
args = parser.add_argument("conf_dir", action=StoreNameValuePair)
As per the documentation, argparse doesn't natively let you have unprefixed options like that. If you omit the leading -, it assumes you are describing a positional argument and expects it to be provided as:
python script.py /tmp/good_conf
If you want it to be optional, it needs to be correctly marked as a flag by calling it --conf_dir, and invoking the script like:
python script.py --conf_dir=/tmp/good_conf
However, to accept name-value pairs, you can implement a custom action. In combination with nargs, such an action could accept an arbitrary number of name-value pairs and store them on the argument parsing result object.
#chepner This is great. I improved this to support multiple args as well and store the result as dict:
class StoreDict(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
kv={}
if not isinstance(values, (list,)):
values=(values,)
for value in values:
n, v = value.split('=')
kv[n]=v
setattr(namespace, self.dest, kv)
The usual way to put name value pairs on the command line is with options. I.e. you would use
python script.py --confdir=/tmp/good_conf
argparse can certainly handle that case. See the docs at:
http://docs.python.org/library/argparse.html#option-value-syntax
Using python optparse.py, is there a way to work out whether a specific option value was set from the command line or from the default value.
Ideally I would like to have a dict just like defaults, but containing the options actually supplied from command line
I know that you could compare the value for each option with defaults, but this wouldn't distinguish a value was passed through command line which matched the default.
Thanks!
EDIT
Sorry my original phrasing wasn't very clear.
I have a large number of scripts which are called from batch files. For audit purposes, I would like to report on the options being passed, and whether they are passed from command line, default, or some other means, to a log file.
Using defaults you can tell whether an option matches a default value, but that still doesn't tell you whether it was actually supplied from command line. This can be relevant: if an option is passed from command line and agrees with the default, if you then change the default in the code the script will still get the same value.
To me it would feel quite natural to have an equivalent to defaults, containing the values actually supplied.
To make the question concrete, in this example:
>>> sys.argv = ['myscript.py','-a','xxx']
>>> import optparse
>>> parser = optparse.OptionParser()
>>> parser.add_option('-a', default = 'xxx')
>>> parser.add_option('-b', default = 'yyy')
How do I know that option a was passed from command line. Is the only way to parse the command line manually?
(I know this is a fairly minor point, but I thought it would be worth asking in case I'm missing smthing on optparse)
Thanks again
Instead of the below boilerplate code:
opts, args = parser.parse_args()
if you use the below code, you will have an additional values object opts_no_defaults with options that were explicitly specified by the user:
opts_no_defaults = optparse.Values()
__, args = parser.parse_args(values=opts_no_defaults)
opts = Values(parser.get_default_values().__dict__)
opts._update_careful(opts_no_defaults.__dict__)
At the end, opts should be identical as in the initial boilerplate code.
print opts_no_defaults.__dict__
print opts.__dict__
for opt in parser._get_all_options():
if opt.dest:
print "Value of %s: %s" % (opt._long_opts[0], getattr(opts, opt.dest))
print "Is %s specified by user? %s" % (opt._long_opts[0], hasattr(opt_no_defaults, opt.dest))
Not knowing you code is impossible to give the better answer, but...
simply don't pass defaults to the parser and check for None values. A None value is a default for optparse lib so you can retrieve your own default and act as usually;
extend optparse to specialize it.
I don't know your program but usually it is not a good design changing behavior when the configuration is the same.
def is_opt_provided (parser, dest):
if any (opt.dest == dest and (opt._long_opts[0] in sys.argv[1:] or opt._short_opts[0] in sys.argv[1:]) for opt in parser._get_all_options()):
return True
return False
Usage:
parser = OptionsParser()
parser.add_option('-o', '--opt', dest='opt_var', ...)
if is_opt_provided(parser, 'opt_var'):
print "Option -o or --opt has been provided"
It would be great if Python maintainers included the suggested function to OptionParser class.