I have the following script:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--arg1', dest='arg1')
parse_results = parser.parse_known_args(['--arg1 = value'])
However I get the result:
(Namespace(arg1=None), ['--arg1 = value'])
Is there a way using argparse to accept this kind of input where there are spaces between argument, = and the value?
The shell processes your argument list before your program ever runs, and that processing is very simple: treat each white-space separated word following the command name as a separate argument. By that logic, --foo = bar is 3 separate arguments, so sys.argv contains ["--foo", "=", "bar"]. There's no avoiding that, since it happens before argparse ever runs.
Given this fact, there's no benefit to writing --foo = bar, since it would be equivalent to --foo=bar or --foo bar, just with extraneous characters (spaces in the first case, = in the second).
As to why --foo=bar is supported at all when it is equivalent to --foo bar, I don't know. It might just be for compatibility with existing practice; it might be useful with different argument parsers, but argparse doesn't seem to need it. At some point in the past, I thought I had come up with a reason why being able to specify the argument to --foo using one shell word instead of two was useful, but I seem to have forgotten it.
It's about how the shell parses arguments:
bruno#bigb:~/Work/playground$ cat argz.py
import sys
print "args:", sys.argv
bruno#bigb:~/Work/playground$ python argz.py --foo = bar --bar=baaz
args: ['argz.py', '--foo', '=', 'bar', '--bar=baaz']
To start, a problem with your code: A program will see space-separated components as separate arguments. Your example should have read
parse_results = parser.parse_known_args(['--arg1', '=', 'value'])
or you can use split() to generate the array on the fly, as the documentation does it:
parse_results = parser.parse_known_args('--arg1 = value'.split())
The above still doesn't give you want you want:
>>> parser.parse_known_args('--arg1 = value'.split())
(Namespace(arg1='='), ['value'])
The equals-sign is parsed as the argument, because long argument syntax is either --option=value or --option value. The = sign is only there to separate the option name from its argument; there is no point in including it if arguments are separated by spaces, so the syntax --option = value is not supported.
Recommendation: Use only standard long-argument syntax, i.e. --arg1 value or --arg=value.
But if you absolutely must support the form --arg1 = value, you can filter the argument list to throw out elements that are a single = sign.:
>>> cleanup = lambda args: (a for a in args if a != '=')
>>> parser.parse_known_args(cleanup('--arg1 = value'.split()))
(Namespace(arg1='value'), [])
When you use
parser.parse_known_args(['--arg1 = value'])
it's equivalent to the commandline:
$myfun.py '--arg1 = value'
In both cases --arg1 = value is treated as one string. Since it does not match '--arg1' or '--arg1=' it goes into the unrecognized strings list.
As others have pointed out, without the quotes $myfun.py --arg1 = value passes to the parser as ['--arg1', '=', 'value']. Which would parse to:
(Namespace(arg1='='), ['value'])
If --arg is defined with nargs='+', then you'd get
(Namespace(arg1=['=', 'value']), [])
and you could ignore the '='.
But for sake of consistency with other commandline practices, I'd suggest sticking with the --arg1=value, and --arg1 value syntax.
Related
I have done as much research as possible but I haven't found the best way to make certain cmdline arguments necessary only under certain conditions, in this case only if other arguments have been given. Here's what I want to do at a very basic level:
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required=False) # only required if --argument is given
p.add_argument('-b', required=False) # only required if --argument is given
From what I have seen, other people seem to just add their own check at the end:
if args.argument and (args.a is None or args.b is None):
# raise argparse error here
Is there a way to do this natively within the argparse package?
I've been searching for a simple answer to this kind of question for some time. All you need to do is check if '--argument' is in sys.argv, so basically for your code sample you could just do:
import argparse
import sys
if __name__ == '__main__':
p = argparse.ArgumentParser(description='...')
p.add_argument('--argument', required=False)
p.add_argument('-a', required='--argument' in sys.argv) #only required if --argument is given
p.add_argument('-b', required='--argument' in sys.argv) #only required if --argument is given
args = p.parse_args()
This way required receives either True or False depending on whether the user as used --argument. Already tested it, seems to work and guarantees that -a and -b have an independent behavior between each other.
You can implement a check by providing a custom action for --argument, which will take an additional keyword argument to specify which other action(s) should become required if --argument is used.
import argparse
class CondAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, **kwargs):
x = kwargs.pop('to_be_required', [])
super(CondAction, self).__init__(option_strings, dest, **kwargs)
self.make_required = x
def __call__(self, parser, namespace, values, option_string=None):
for x in self.make_required:
x.required = True
try:
return super(CondAction, self).__call__(parser, namespace, values, option_string)
except NotImplementedError:
pass
p = argparse.ArgumentParser()
x = p.add_argument("--a")
p.add_argument("--argument", action=CondAction, to_be_required=[x])
The exact definition of CondAction will depend on what, exactly, --argument should do. But, for example, if --argument is a regular, take-one-argument-and-save-it type of action, then just inheriting from argparse._StoreAction should be sufficient.
In the example parser, we save a reference to the --a option inside the --argument option, and when --argument is seen on the command line, it sets the required flag on --a to True. Once all the options are processed, argparse verifies that any option marked as required has been set.
Your post parsing test is fine, especially if testing for defaults with is None suits your needs.
http://bugs.python.org/issue11588 'Add "necessarily inclusive" groups to argparse' looks into implementing tests like this using the groups mechanism (a generalization of mutuall_exclusive_groups).
I've written a set of UsageGroups that implement tests like xor (mutually exclusive), and, or, and not. I thought those where comprehensive, but I haven't been able to express your case in terms of those operations. (looks like I need nand - not and, see below)
This script uses a custom Test class, that essentially implements your post-parsing test. seen_actions is a list of Actions that the parse has seen.
class Test(argparse.UsageGroup):
def _add_test(self):
self.usage = '(if --argument then -a and -b are required)'
def testfn(parser, seen_actions, *vargs, **kwargs):
"custom error"
actions = self._group_actions
if actions[0] in seen_actions:
if actions[1] not in seen_actions or actions[2] not in seen_actions:
msg = '%s - 2nd and 3rd required with 1st'
self.raise_error(parser, msg)
return True
self.testfn = testfn
self.dest = 'Test'
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind=Test)
g1.add_argument('--argument')
g1.add_argument('-a')
g1.add_argument('-b')
print(p.parse_args())
Sample output is:
1646:~/mypy/argdev/usage_groups$ python3 issue25626109.py --arg=1 -a1
usage: issue25626109.py [-h] [--argument ARGUMENT] [-a A] [-b B]
(if --argument then -a and -b are required)
issue25626109.py: error: group Test: argument, a, b - 2nd and 3rd required with 1st
usage and error messages still need work. And it doesn't do anything that post-parsing test can't.
Your test raises an error if (argument & (!a or !b)). Conversely, what is allowed is !(argument & (!a or !b)) = !(argument & !(a and b)). By adding a nand test to my UsageGroup classes, I can implement your case as:
p = argparse.ArgumentParser(formatter_class=argparse.UsageGroupHelpFormatter)
g1 = p.add_usage_group(kind='nand', dest='nand1')
arg = g1.add_argument('--arg', metavar='C')
g11 = g1.add_usage_group(kind='nand', dest='nand2')
g11.add_argument('-a')
g11.add_argument('-b')
The usage is (using !() to mark a 'nand' test):
usage: issue25626109.py [-h] !(--arg C & !(-a A & -b B))
I think this is the shortest and clearest way of expressing this problem using general purpose usage groups.
In my tests, inputs that parse successfully are:
''
'-a1'
'-a1 -b2'
'--arg=3 -a1 -b2'
Ones that are supposed to raise errors are:
'--arg=3'
'--arg=3 -a1'
'--arg=3 -b2'
For arguments I've come up with a quick-n-dirty solution like this.
Assumptions: (1) '--help' should display help and not complain about required argument and (2) we're parsing sys.argv
p = argparse.ArgumentParser(...)
p.add_argument('-required', ..., required = '--help' not in sys.argv )
This can easily be modified to match a specific setting.
For required positionals (which will become unrequired if e.g. '--help' is given on the command line) I've come up with the following: [positionals do not allow for a required=... keyword arg!]
p.add_argument('pattern', ..., narg = '+' if '--help' not in sys.argv else '*' )
basically this turns the number of required occurrences of 'pattern' on the command line from one-or-more into zero-or-more in case '--help' is specified.
Here is a simple and clean solution with these advantages:
No ambiguity and loss of functionality caused by oversimplified parsing using the in sys.argv test.
No need to implement a special argparse.Action or argparse.UsageGroup class.
Simple usage even for multiple and complex deciding arguments.
I noticed just one considerable drawback (which some may find desirable): The help text changes according to the state of the deciding arguments.
The idea is to use argparse twice:
Parse the deciding arguments instead of the oversimplified use of the in sys.argv test. For this we use a short parser not showing help and the method .parse_known_args() which ignores unknown arguments.
Parse everything normally while reusing the parser from the first step as a parent and having the results from the first parser available.
import argparse
# First parse the deciding arguments.
deciding_args_parser = argparse.ArgumentParser(add_help=False)
deciding_args_parser.add_argument(
'--argument', required=False, action='store_true')
deciding_args, _ = deciding_args_parser.parse_known_args()
# Create the main parser with the knowledge of the deciding arguments.
parser = argparse.ArgumentParser(
description='...', parents=[deciding_args_parser])
parser.add_argument('-a', required=deciding_args.argument)
parser.add_argument('-b', required=deciding_args.argument)
arguments = parser.parse_args()
print(arguments)
Until http://bugs.python.org/issue11588 is solved, I'd just use nargs:
p = argparse.ArgumentParser(description='...')
p.add_argument('--arguments', required=False, nargs=2, metavar=('A', 'B'))
This way, if anybody supplies --arguments, it will have 2 values.
Maybe its CLI result is less readable, but code is much smaller. You can fix that with good docs/help.
This is really the same as #Mira 's answer but I wanted to show it for the case where when an option is given that an extra arg is required:
For instance, if --option foo is given then some args are also required that are not required if --option bar is given:
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--option', required=True,
help='foo and bar need different args')
if 'foo' in sys.argv:
parser.add_argument('--foo_opt1', required=True,
help='--option foo requires "--foo_opt1"')
parser.add_argument('--foo_opt2', required=True,
help='--option foo requires "--foo_opt2"')
...
if 'bar' in sys.argv:
parser.add_argument('--bar_opt', required=True,
help='--option bar requires "--bar_opt"')
...
It's not perfect - for instance proggy --option foo --foo_opt1 bar is ambiguous but for what I needed to do its ok.
Add additional simple "pre"parser to check --argument, but use parse_known_args() .
pre = argparse.ArgumentParser()
pre.add_argument('--argument', required=False, action='store_true', default=False)
args_pre=pre.parse_known_args()
p = argparse.ArgumentParser()
p.add_argument('--argument', required=False)
p.add_argument('-a', required=args_pre.argument)
p.add_argument('-b', required=not args_pre.argument)
Is it possible to use argparse to capture an arbitrary set of optional arguments?
For example both the following should be accepted as inputs:
python script.py required_arg1 --var1 value1 --var2 value2 --var3 value3
python script.py required_arg1 --varA valueA --var2 value2 --varB valueB
a priori I don't know what optional arguments would be specified receive but would handle them accordingly.
This is kind of a hackish way, but it works well:
Check, which arguments are not added and add them
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("foo")
parser.add_argument("-bar", type=int)
# parser can have any arguments, whatever you want!
parsed, unknown = parser.parse_known_args() # this is an 'internal' method
# which returns 'parsed', the same as what parse_args() would return
# and 'unknown', the remainder of that
# the difference to parse_args() is that it does not exit when it finds redundant arguments
for arg in unknown:
if arg.startswith(("-", "--")):
# you can pass any arguments to add_argument
parser.add_argument(arg.split('=')[0], type=<your type>, ...)
args = parser.parse_args()
For example:
python3 arbitrary_parser.py ha -bar 12 -extra1 value1 -extra2 value2
Then the result would be
args = Namespace(bar=12, foo='ha', extra1='value1' extra2='value2')
Possible? possibly, but I wouldn't recommend it. argparse is the not best tool for parsing this kind of input, or conversely, this a poor argument specification from an argparse perspective.
Have you thought about what the usage line should look like? How would explain this to your users?
How would you parse this working from sys.argv directly? It looks like you could collect 3 pieces:
prog = sys.argv[0]
arg1 = sys.argv[1]
keys = sys.argv[2::2]
# maybe strip -- off each
values = sys.argv[3::2]
kvdict = {k:v for k, v in zip(keys, values)}
There are other SO questions asking about generic key:value pairs. Things like:
--args key1:value1 key2:value2
This can be handled with nargs='*' and an action that splits each input string on : (or =) and stores things by key.
Your requirement is least amenable to argparse use because it requires bypassing the whole idea of matching argument flags with strings in argv. It requires, some how, turning off all the normal argparse parsing.
Looks like I suggested the same thing a couple of years ago
Parse non-pre-defined argument
or earlier
Using argparse to parse arguments of form "arg= val"
My script is accepting --full, --last and --check using ArgParse. If no option is provided, it just show the help message. But in that message, the parameters appear as optional.
usage: script.py [-h] [--full] [--last] [--check log_file]
If I use the keyword required, then the script will always expect the parameter, which is not correct.
usage: script.py [-h] --full --last --check log_file
So, how can I show something like:
usage: script.py [-h] (--full |--last |--check log_file)
Indicating that the help is optional but that at least one of those parameters is required.
On the question of customizing the usage:
The parser constructor takes a usage parameter. The immediate effect is to set an attribute:
parser = argparse.ArgumentParser( ... usage=custom_usage...)
print(parser.usage)
# should show None or the custom_usage string
Being a normal Python object attribute, you can change it after the parser was created.
usage_str = parser.format_usage()
The format_usage method ask the parser for create the usage that will be shown in the help (and error messages). If the parser.usage value is None, it formats it from the arguments. If a string, it is used as is (I think it fills in values like %(prog)s).
So you could write a usage string from scratch. Or you could set up the parser, get the current usage string, and edit that to suit your needs. Editing is most likely something you'd do during development, while testing the parser in an IDE. But it could be done on the fly.
A crude example:
In [441]: parser=argparse.ArgumentParser()
In [442]: g=parser.add_mutually_exclusive_group()
In [443]: g.add_argument('--foo')
In [444]: g.add_argument('--bar')
In [445]: ustr = parser.format_usage()
# 'usage: ipython3 [-h] [--foo FOO | --bar BAR]\n'
In [450]: parser.usage = ustr.replace('[','(').replace(']',')')
In [451]: parser.format_usage()
# 'usage: usage: ipython3 (-h) (--foo FOO | --bar BAR)\n'
I've replaced the [] with () (even on the -h :( ).
For now testing logical combinations of the args attributes is the best choice. Inside the parse_args functions the parser maintains a list (set actually) of arguments that it has seen. That is used to test for required arguments, and for mutually_exclusive_arguments, but it is not available outside that code.
For store_true (or false) arguments, just check their truth value. For others I like to test for the default None. If you use other default values, test accordingly. A nice thing about None is that the user cannot give you that value.
Perhaps the most general way to test for arguments is to count the number of attributes which are not None:
In [461]: args=argparse.Namespace(one=None, tow=2, three=None)
In [462]: ll = ['one','tow','three']
In [463]: sum([getattr(args,l,None) is not None for l in ll])
Out[463]: 1
0 means none are found; >0 at least one present; ==len(ll) all found; >1 violates mutually exclusivity; '==1' for required mutually exclusive.
As #doublep explained in his answer, if you want to use more than one option at a time:
Change the usage message manually to the one you want.
Add the following code from Python argparse: Make at least one argument required:
if not (args.full or args.last or args.check):
parse.error('[-] Error: DISPLAY_ERROR_MESSAGE')
You can use add_mutually_exclusive_group():
parser = argparse.ArgumentParser ()
group = parser.add_mutually_exclusive_group (required = True)
group.add_argument ('--foo')
group.add_argument ('--bar')
However, the main effect is that you won't be able to use more than one option at a time.
argparse uses per default abbreviation in unambiguous cases.
I don't want abbreviation and I'd like to disable it.
But didn't find it in the documentation.
Is it possible?
Example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--send', action='store_true')
parser.parse_args(['--se']) # returns Namespace(send=True)
But I want it only to be true when the full parameter is supplied. To prevent user errors.
UPDATE:
I created a ticket at python bugtracker after Vikas answer. And it already has been processed.
As of Python 3.5.0 you can disable abbreviations by initiating the ArgumentParser with the following:
parser = argparse.ArgumentParser(allow_abbrev=False)
Also see the documentation.
No, apparently this is not possible. At least in Python 2.7.2.
First, I took a look into the documentation - to no avail.
Then I opened the Lib\argparse.py and looked through the source code. Omitting a lot of details, it seems that each argument is parsed by a regular expression like this (argparse:2152):
# allow one or more arguments
elif nargs == ONE_OR_MORE:
nargs_pattern = '(-*A[A-]*)'
This regex will successfully parse both '-' and '--', so we have no control over the short and long arguments. Other regexes use the -* construct too, so it does not depend on the type of the parameter (no sub-arguments, 1 sub-argument etc).
Later in the code double dashes are converted to one dash (only for non-optional args), again, without any flags to control by user:
# if this is an optional action, -- is not allowed
if action.option_strings:
nargs_pattern = nargs_pattern.replace('-*', '')
nargs_pattern = nargs_pattern.replace('-', '')
No, well not without ugly hacks.
The code snippet #Vladimir posted, i suppose that is not what you are looking for. The actual code that is doing this is:
def _get_option_tuples(self, option_string):
...
if option_string.startswith(option_prefix):
...
See the check is startswith not ==.
And you can always extend argparse.ArgumentParser to provide your own _get_option_tuples(self, option_string) to change this behavior. I just did by replacing two occurrence of option_string.startswith(option_prefix) to option_string == option_prefix and:
>>> parser = my_argparse.MyArgparse
>>> parser = my_argparse.MyArgparse()
>>> parser.add_argument('--send', action='store_true')
_StoreTrueAction(option_strings=['--send'], dest='send', nargs=0, const=True, default=False, type=None, choices=None, help=None, metavar=None)
>>> parser.parse_args(['--se'])
usage: [-h] [--send]
: error: unrecognized arguments: --se
A word of caution
The method _get_option_tuples is prefixed with _, which typically means a private method in python. And it is not a good idea to override a private.
Another way for Python 2.7. Let's get clunky! Say you want to recognize --dog without abbreviation.
p = argparse.ArgumentParser()
p.add_argument('--dog')
p.add_argument('--dox', help=argparse.SUPPRESS, metavar='IGNORE')
By adding a second argument --dox that differs from the argument you want only in the third letter, --d and --do become ambiguous. Therefore, the parser will refuse to recognize them. You would need to add code to catch the resulting exception and process it according to the context in which you are calling parse_args. You might also need to suppress/tweak the help text.
The help=... keeps the argument out of the option list on the default help message (per this), and metavar='IGNORE' is just to make it clear you really aren't doing anything with this option :) .
Using python optparse.py, is there a way to work out whether a specific option value was set from the command line or from the default value.
Ideally I would like to have a dict just like defaults, but containing the options actually supplied from command line
I know that you could compare the value for each option with defaults, but this wouldn't distinguish a value was passed through command line which matched the default.
Thanks!
EDIT
Sorry my original phrasing wasn't very clear.
I have a large number of scripts which are called from batch files. For audit purposes, I would like to report on the options being passed, and whether they are passed from command line, default, or some other means, to a log file.
Using defaults you can tell whether an option matches a default value, but that still doesn't tell you whether it was actually supplied from command line. This can be relevant: if an option is passed from command line and agrees with the default, if you then change the default in the code the script will still get the same value.
To me it would feel quite natural to have an equivalent to defaults, containing the values actually supplied.
To make the question concrete, in this example:
>>> sys.argv = ['myscript.py','-a','xxx']
>>> import optparse
>>> parser = optparse.OptionParser()
>>> parser.add_option('-a', default = 'xxx')
>>> parser.add_option('-b', default = 'yyy')
How do I know that option a was passed from command line. Is the only way to parse the command line manually?
(I know this is a fairly minor point, but I thought it would be worth asking in case I'm missing smthing on optparse)
Thanks again
Instead of the below boilerplate code:
opts, args = parser.parse_args()
if you use the below code, you will have an additional values object opts_no_defaults with options that were explicitly specified by the user:
opts_no_defaults = optparse.Values()
__, args = parser.parse_args(values=opts_no_defaults)
opts = Values(parser.get_default_values().__dict__)
opts._update_careful(opts_no_defaults.__dict__)
At the end, opts should be identical as in the initial boilerplate code.
print opts_no_defaults.__dict__
print opts.__dict__
for opt in parser._get_all_options():
if opt.dest:
print "Value of %s: %s" % (opt._long_opts[0], getattr(opts, opt.dest))
print "Is %s specified by user? %s" % (opt._long_opts[0], hasattr(opt_no_defaults, opt.dest))
Not knowing you code is impossible to give the better answer, but...
simply don't pass defaults to the parser and check for None values. A None value is a default for optparse lib so you can retrieve your own default and act as usually;
extend optparse to specialize it.
I don't know your program but usually it is not a good design changing behavior when the configuration is the same.
def is_opt_provided (parser, dest):
if any (opt.dest == dest and (opt._long_opts[0] in sys.argv[1:] or opt._short_opts[0] in sys.argv[1:]) for opt in parser._get_all_options()):
return True
return False
Usage:
parser = OptionsParser()
parser.add_option('-o', '--opt', dest='opt_var', ...)
if is_opt_provided(parser, 'opt_var'):
print "Option -o or --opt has been provided"
It would be great if Python maintainers included the suggested function to OptionParser class.