We Keep Coding

Argparse positional multiple choices default subset: Invalid Choice

I'm using argparse to obtain a subset of elements using choices with nargs='*'. If no element is introduced, I want to return by default a subset of the elements, but I get an Invalid Choice error.
Here is my sample code:
from argparse import ArgumentParser
parser = ArgumentParser()
parser.add_argument(
"choices",
nargs="*",
type=int,
choices=[1,2,3,4,5],
default=[1,3,5]
)
if __name__ == "__main__":
print(parser.parse_args().choices)
The output I get is:
$ ./thing.py 3
[3]
$ ./thing.py 1 2
[1, 2]
$ ./thing.py
usage: thing.py [-h] [{1,2,3,4,5} ...]
thing.py: error: argument choices: invalid choice: [1, 3, 5] (choose from 1, 2, 3, 4, 5)
I can set default to an int, but not a list. Is there any workaround to this?
Edit: This only seems to happen with positional arguments. There aren't any problems if using nargs="+" and required=False in a non-positional argument.

To elaborate on my comments.
Normal processing of defaults, as #Tomerikoo quotes, is to process string defaults through both the type and choices check. The defaults are placed in the args namespace at the start of parsing. At the end, if the defaults have not be overwritten by using inputs, they will conditionally converted. Non-string defaults are left as is, without type or choices checking.
But a positional with nargs='*' is different. It is "always seen", since an empty list (i.e. none) of strings satisfies its nargs. With usual handling this would overwrite the default, resulting in Namespace(choices=[]).
To get around that the _get_values method replaces the [] with the default. This new value is passed through the choices test, but not through type. This results in the behavior that the OP encountered.
The relevant parts of _get_values are quoted below. The first handles nargs='?', the second '*', and the third normal cases. _get_value does the type test. _check_value does the choices test.
def _get_values(self, action, arg_strings):
....
# optional argument produces a default when not present
if not arg_strings and action.nargs == OPTIONAL:
if action.option_strings:
value = action.const
else:
value = action.default
if isinstance(value, str):
value = self._get_value(action, value)
self._check_value(action, value)
# when nargs='*' on a positional, if there were no command-line
# args, use the default if it is anything other than None
elif (not arg_strings and action.nargs == ZERO_OR_MORE and
not action.option_strings):
if action.default is not None:
value = action.default
else:
value = arg_strings
self._check_value(action, value)
# single argument or optional argument produces a single value
elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
arg_string, = arg_strings
value = self._get_value(action, arg_string)
self._check_value(action, value)
I should add another block, which is used for '*' and non-empty user input:
# all other types of nargs produce a list
else:
value = [self._get_value(action, v) for v in arg_strings]
for v in value:
self._check_value(action, v)
Note that here, individual strings are type and checked, where as in the first case, it's the whole default that is checked.
There have been bug/issues related to this, but no one has some up with a good reason to change it.

Make argparse treat dashes and underscore identically

argparse replaces dashes in optional arguments by underscores to determine their destination:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--use-unicorns', action='store_true')
args = parser.parse_args(['--use-unicorns'])
print(args) # returns: Namespace(use_unicorns=True)
However the user has to remember whether the option is --use-unicorns or --use_unicorns; using the wrong variant raises an error.
This can cause some frustration as the variable args.use_unicorns in the code does not make it clear which variant was defined.
How can I make argparse accept both --use-unicorns and --use_unicorns as valid ways to define this optional argument?

parser.add_argument('--use-unicorns', action='store_true')
args = parser.parse_args(['--use-unicorns'])
print(args) # returns: Namespace(use_unicorns=True)
argparse translates the '-' to '_' because the use of '-' in flags is well established POSIX practice. But args.use-unicones is not acceptable Python. In other words, it does the translation so the dest will be a valid Python variable or attribute name.
Note that argparse does not perform this translation with positionals. In that case the programmer has full control over the dest parameter, and can choose anything that's convenient. Since the argparse only uses getattr and setattr when accessing the Namespace, the constraints on a valid dest are minimal.
There are two users. There's you, the programmer, and there's your end user. What's convenient for you might not be optimal for the other.
You can also specify a dest with defining an optional. And metavar gives you further control over the help display.
It's parser._get_optional_kwargs that performs the '-' replace:
if dest is None:
....
dest = dest.replace('-', '_')

parser.add_argument accepts more than one flag for an argument (link to documentation). One easy way to make the parser accept both variants is to declare the argument as
parser.add_argument('--use-unicorns', '--use_unicorns', action='store_true')
However both options will show up in the help, and it is not very elegant as it forces one to write the variants manually.
An alternative is to subclass argparse.ArgumentParser to make the matching invariant to replacing dashes by underscore. This requires a little bit of fiddling, as both argparse_ActionsContainer._parse_optional and argparse_ActionsContainer._get_option_tuples have to be modified to handle this matching and abbrevations, e.g. --use_unic.
I ended up with the following subclassed method, where the matching to abbrevations is delegated from _parse_optional to _get_option_tuples:
from gettext import gettext as _
import argparse
class ArgumentParser(argparse.ArgumentParser):
def _parse_optional(self, arg_string):
# if it's an empty string, it was meant to be a positional
if not arg_string:
return None
# if it doesn't start with a prefix, it was meant to be positional
if not arg_string[0] in self.prefix_chars:
return None
# if it's just a single character, it was meant to be positional
if len(arg_string) == 1:
return None
option_tuples = self._get_option_tuples(arg_string)
# if multiple actions match, the option string was ambiguous
if len(option_tuples) > 1:
options = ', '.join([option_string
for action, option_string, explicit_arg in option_tuples])
args = {'option': arg_string, 'matches': options}
msg = _('ambiguous option: %(option)s could match %(matches)s')
self.error(msg % args)
# if exactly one action matched, this segmentation is good,
# so return the parsed action
elif len(option_tuples) == 1:
option_tuple, = option_tuples
return option_tuple
# if it was not found as an option, but it looks like a negative
# number, it was meant to be positional
# unless there are negative-number-like options
if self._negative_number_matcher.match(arg_string):
if not self._has_negative_number_optionals:
return None
# if it contains a space, it was meant to be a positional
if ' ' in arg_string:
return None
# it was meant to be an optional but there is no such option
# in this parser (though it might be a valid option in a subparser)
return None, arg_string, None
def _get_option_tuples(self, option_string):
result = []
if '=' in option_string:
option_prefix, explicit_arg = option_string.split('=', 1)
else:
option_prefix = option_string
explicit_arg = None
if option_prefix in self._option_string_actions:
action = self._option_string_actions[option_prefix]
tup = action, option_prefix, explicit_arg
result.append(tup)
else: # imperfect match
chars = self.prefix_chars
if option_string[0] in chars and option_string[1] not in chars:
# short option: if single character, can be concatenated with arguments
short_option_prefix = option_string[:2]
short_explicit_arg = option_string[2:]
if short_option_prefix in self._option_string_actions:
action = self._option_string_actions[short_option_prefix]
tup = action, short_option_prefix, short_explicit_arg
result.append(tup)
underscored = {k.replace('-', '_'): k for k in self._option_string_actions}
option_prefix = option_prefix.replace('-', '_')
if option_prefix in underscored:
action = self._option_string_actions[underscored[option_prefix]]
tup = action, underscored[option_prefix], explicit_arg
result.append(tup)
elif self.allow_abbrev:
for option_string in underscored:
if option_string.startswith(option_prefix):
action = self._option_string_actions[underscored[option_string]]
tup = action, underscored[option_string], explicit_arg
result.append(tup)
# return the collected option tuples
return result
A lot of this code is directly derived from the corresponding methods in argparse (from the CPython implementation here). Using this subclass should make the matching of optional arguments invariant to using dashes - or underscores _.

Argparse default as function is always called

//edit 1: Changed slightly get_variable - forgot to add another argument that is passed to it (was writing it from memory, sorry for that).
I have a problem with default values from argparser.
Some values if not present in command line, are taken from environment using os.env, and if there is none, get it from DEFAULT_FOR_VARIABLE:
def get_variable(name, DEFAULT_FOR_VARIABLE = ''):
if name in os.environ:
return os.environ[name]
print("no default value")
return DEFAULT_FOR_VARIABLE
This is how it's parsed in main():
parser = argparse.ArgumentParser(description=MODULE_NAME)
parser.add_argument('--test_arg', default=get_variable(VAR_NAME, DEFAULT_FOR_TEST_ARG))
args = parser.parse_args()
print(args.test_arg)
No matter if I pass down arguments or not, get_variable function is called and if there is no value in os.environ, print gets executed (to let me know there is missing argument), even when there is a value passed:
λ python Parser_Test.py --test_arg test_arg
no default value
test_arg
It's working as expected when arguments are not passed:
λ python Parser_Test.py
No default value
But when for DEFAULT_FOR_TEST_ARG is set:
λ python Parser_Test.py
No default value
DEFAULT_VALUE_FOR_TEST_ARG
Also checking each parsed argument would be hard, since there is no way of iterating them the way it's provided by argparse - I have quite few of them to check for from the user.
Is there a way to change this behavior? Or should I use non-standard module for parsing arguments in such a case?

Not sure if I fully understand, but can you not do this?
def get_variable(name):
if name in os.environ:
return os.environ[name]
else:
print("no default value")
return 'empty'
Or:
parser = argparse.ArgumentParser(description=MODULE_NAME)
parser.add_argument('--test_arg',dest='test',nargs='?', default="empty")
args = parser.parse_args()
if args.test == "empty":
if name in os.environ:
newGlobalVar = os.environ["name"]
print("no default value")
else:
newGlobalVar = args.test

get_variable(VAR_NAME) is evaluated by the interpreter when the add_argument method is used. In python function arguments are evaluated before being passed to the function.
argparse does defer evaluating the default if it is a string:
In [271]: p = argparse.ArgumentParser()
In [272]: p.add_argument('-f', type=int, default='12');
In [273]: p.parse_args('-f 23'.split())
Out[273]: Namespace(f=23)
In [274]: p.parse_args([])
Out[274]: Namespace(f=12)
Here, if no -f is provided, the '12' will be passed to the type function:
int('12')
or with a custom type:
In [275]: def mytype(astr):
...: print('eval',astr)
...: return int(astr)
In [276]: p.add_argument('-g', type=mytype, default='12');
In [277]: p.parse_args([])
eval 12
Out[277]: Namespace(f=12, g=12)
In [278]: p.parse_args(['-g','3'])
eval 3
Out[278]: Namespace(f=12, g=3)
But in your case the code that you want to conditionally evaluate probably can't be handled by a type function. That is, you aren't evaluating the default in the same way as you would an user provided string.
So a post parsing test probably makes most sense. The default default is None, which is easily tested:
if args.test is None:
args.test = 'the proper default'
The user can't provide any string that will produce None, so it is a safe default.
Just out of curiosity I wrote a type that looks up a name in os.environ:
In [282]: def get_environ(name):
...: if name in os.environ:
...: return os.environ[name]
...: raise argparse.ArgumentTypeError('%s not in environ'%name)
In [283]: p.add_argument('-e', type=get_environ, default='DISPLAY');
Without arguments it looks up the default os.environ['DISPLAY']
In [284]: p.parse_args([])
eval 12
Out[284]: Namespace(e=':0', f=12, g=12)
with a valid name:
In [289]: p.parse_args(['-e','EDITOR'])
eval 12
Out[289]: Namespace(e='nano', f=12, g=12)
and raises an error when the name isn't valid:
In [290]: p.parse_args(['-e','FOO'])
usage: ipython3 [-h] [-f F] [-g G] [-e E]
ipython3: error: argument -e: FOO not in environ
An exception has occurred, use %tb to see the full traceback.
I know it's not what you are aiming for, but it gives an idea of what is possible if you want to delay evaluation of a default.

Here is an approach using a lambda as the default in python 3.6. I think this is on target with what the OP wanted to do. The default doesn't get evaluated immediately. You can easily find them and call them in a for loop to resolve the values. I included the t2 argument with a string default just to show that a normal default still works fine in this context.
import argparse
import os
def get_value(var, dflt):
if var in os.environ:
return os.environ[var]
return dflt
parser = argparse.ArgumentParser(description=os.path.splitext(os.path.basename(__file__))[0])
parser.add_argument('--t1', default=lambda: get_value('t1_value', 't1 default'))
parser.add_argument('--t2', default='t2 default')
args = parser.parse_args()
print("Arguments have been parsed")
print(f"--t1: {args.t1}")
print(f"--t2: {args.t2}")
print("Lazily getting defaults")
for key in vars(args):
f = args.__dict__[key]
if callable(f):
print(f'Getting default value for {key}')
args.__dict__[key] = f()
print(f"--t1: {args.t1}")
print(f"--t2: {args.t2}")
Results:
Connected to pydev debugger (build 202.7660.27)
Arguments have been parsed
--t1: <function <lambda> at 0x000002425DD3FAE8>
--t2: t2 default
Lazily getting defaults
Getting default value for t1
--t1: t1_default
--t2: t2 default
Process finished with exit code 0
You can do a similar thing with a specialized class, but I think the lambda is more concise and essentially the same.

For a one-liner you could also try this:
os.environ.get(VAR_NAME, DEFAULT_FOR_TEST_ARG)

Comma separated inputs instead of space separated inputs for argparse

I'm using argparse to receive inputs from the command line to run my script.
My current input string looks like this:
path> python <\filename\\> -t T1 T2 T3 -f F1 F2
Is there a parameter in argparse such that instead of separating inputs by space, I can separate them by commas?
In other words:
path> python <\filename\\> -t T1,T2,T3 -f F1,F2

There is no such feature in argparse.
Alternatives:
post-process the args namespace and split/parse the values manually
define a custom action and split/parse the values manually
define a custom type and split/parse the values manually
subclass ArgumentParser and customise ArgumentParser.convert_arg_line_to_args

There are some useful answers here already, but I wanted a bit more: splitting on commas, validating values with choices, and getting useful error messages, so below I offer a solution.
Simple Version
As a first pass, we can just pass in an appropriate function to the type parameter:
>>> import argparse
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
But this doesn't work with choices because it checks if the whole list is in choices and not each value:
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','), choices=('a', 'b', 'c'))
>>> parser.parse_args(['--foo', 'a,b,c'])
usage: cmd [-h] [--foo {a,b,c}]
cmd: error: argument --foo: invalid choice: ['a', 'b', 'c'] (choose from 'a', 'b', 'c')
Setting nargs to something like * or + checks each value, but only for space-separated arguments (e.g., --foo a b) and not the comma-separated ones. It seems like there is no supported way to check if each value is in the choices if we produce the list ourselves. Therefore we need to raise errors ourselves via the type parameter (as Shiplu Mokaddim partially implemented). Creating a custom Action class sounds promising as actions have access to the choices, but the action happens after the type function applies and values are checked, so we still couldn't use the choices parameter on add_argument() for this purpose.
Better Version
Here is a solution using a custom type function. We this function to take a list of valid choices, but since the function for type conversion can only take an argument string, we need to wrap it in a class (and define the special __call__() method) or a function closure. This solution uses the latter.
>>> def csvtype(choices):
... """Return a function that splits and checks comma-separated values."""
... def splitarg(arg):
... values = arg.split(',')
... for value in values:
... if value not in choices:
... raise argparse.ArgumentTypeError(
... 'invalid choice: {!r} (choose from {})'
... .format(value, ', '.join(map(repr, choices))))
... return values
... return splitarg
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=csvtype(('a', 'b', 'c')))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
>>> parser.parse_args(['--foo', 'a,b,d'])
usage: cmd [-h] [-f F]
cmd: error: argument -f: invalid choice: 'd' (choose from 'a', 'b', 'c')
Notice that we get an appropriate error as well. For this, be sure to use argparse.ArgumentTypeError and not argparse.ArgumentError inside the function.
Other Options
User wim suggested some other options not discussed above. I don't find those attractive for the following reasons:
Post-processing the argument after parsing means you have to do more work to get the error messages to be consistent with those from argparse. Just raising argparse.ArgumentError will lead to a stacktrace. Also, argparse catches errors raised during parsing and alters them to specify the option that was used, which you'd otherwise need to do manually.
Subclassing ArgumentParser is more work, and convert_arg_line_to_args() is for reading arguments from a file, not the command line.

You can use module shlex to extract the parameters, then replace commas with spaces, and pass the results to argparse for further processing:
comma_args = shlex.split("-t T1,T2,T3 -f F1,F2")
# ['-t', 'T1,T2,T3', '-f', 'F1,F2']
args = [x.replace(","," ") for x in comma_args]
# ['-t', 'T1 T2 T3', '-f', 'F1 F2']
parse_args(args)

To ensure compatibility with choices, I use two parsers:
#!/usr/bin/env python3
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--arg", type=str, metavar="ARG[,ARG,...]")
# Pass the usage to the subparser, remove the usage: prefix.
subparser_usage = parser.format_usage().replace("usage: ", "", 1)
subparser = argparse.ArgumentParser(usage=subparser_usage)
subparser.add_argument("arg", nargs="*", type=int, choices=[1, 2, 3])
args = parser.parse_args()
if args.arg:
subargs = subparser.parse_args(args.arg.split(","))
args.arg = subargs.arg
print(args)
if __name__ == "__main__":
main()
This fulfills all possible requirements:
easy and understandable
involves only argparse
choices are ensured with a correct usage and help message
this allows to have also positional arguments (thanks to the comma separated list)
See the following examples:
$ ./example --arg 1,2,3
Namespace(arg=[1, 2, 3])
$ ./example --arg 1,2,4
usage: example [-h] [--arg ARG[,ARG,...]]
example: error: argument arg: invalid choice: 4 (choose from 1, 2, 3)
$ ./example --arg 1
Namespace(arg=[1])

Here this comma-separated input is actually a different type. All you have to is to define the type.
Here I define a custom type that does it.
class DelimiterSeperatedInput:
def __init__(self, item_type, separator=','):
self.item_type = item_type
self.separator = separator
def __call__(self, value):
values = []
try:
for val in value.split(self.separator):
typed_value = self.item_type(val)
values.append(typed_value)
except Exception:
raise ArgumentError("%s is not a valid argument" % value)
return values
parser.add_argument('-t', type=DelimiterSeperatedInput(str),
help='comma separated string values')
parser.add_argument('-f', type=DelimiterSeperatedInput(float, ":"),
help="colon separated floats')
This code may not work as-is, you might have to fix. But it was to give an idea.
Note: I could reduce the __call__ function body by using list, map etc. But then it wouldn't be very readable. Once you get the idea, you can do kinds of stuff with it.

If you're okay with space not comma separated, then it's builtin to argparse:
In [1]: from argparse import ArgumentParser
In [2]: parser = ArgumentParser()
In [3]: parser.add_argument('-a', nargs='+')
Out[3]: _StoreAction(option_strings=['-a'], dest='a', nargs='+', const=None,
default=None, type=None, choices=None, help=None, metavar=None)
In [4]: parser.parse_args(['-a', 'foo', 'bar'])
Out[4]: Namespace(a=['foo', 'bar'])

Parse Args that aren't declared

I'm writing a utility for running bash commands that essentially takes as input a string and a list optional argument and uses the optional arguments to interpolate string.
I'd like it to work like this:
interpolate.py Hello {user_arg} my name is {computer_arg} %% --user_arg=john --computer_arg=hal
The %% is a separator, it separates the string to be interpolated from the arguments. What follows is the arguments used to interpolate the string. In this case the user has chosen user_arg and computer_arg as arguments. The program can't know in advance which argument names the user will choose.
My problem is how to parse the arguments? I can trivially split the input arguments on the separator but I can't figure out how to get optparse to just give the list of optional args as a dictionary, without specifying them in advance. Does anyone know how to do this without writing a lot of regex?

Well, if you use '--' to separate options from arguments instead of %%, optparse/argparse will just give you the arguments as a plain list (treating them as positional arguments instead of switched). After that it's not 'a lot of' regex, it's just a mere split:
for argument in args:
if not argument.startswith("--"):
# decide what to do in this case...
continue
arg_name, arg_value = argument.split("=", 1)
arg_name = arg_name[2:]
# use the argument any way you like

With argparse you could use the parse_known_args method to consume predefined arguments and any additional arguments. For example, using the following script
import sys
import argparse
def main(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument('string', type=str, nargs='*',
help="""String to process. Optionally with interpolation
(explain this here...)""")
args, opt_args = parser.parse_known_args(argv)
print args
print opt_args
return 0
if __name__=='__main__':
sys.exit(main(sys.argv[1:]))
and calling with
python script.py Hello, my name is {name} --name=chris
yields the following output:
Namespace(string=['Hello,' 'my', 'name', 'is', '{name}'])
['--name=chris']
All that is left to do is to loop through the args namespace looking for strings of the form {...} and replacing them with the corresponding element in opt_args, if present. (I'm not sure if argparse can do argument interpolation automatically, the above example is the only immediate solution which comes to mind).

For something like this, you really don't need optparse or argparse - the benefit of such libraries are of little use in this circumstance (things like lone -v type arguments, checking for invalid options, value validation and so on)
def partition_list(lst, sep):
"""Slices a list in two, cutting on index matching "sep"
>>> partition_list(['a', 'b', 'c'], sep='b')
(['a'], ['c'])
"""
if sep in lst:
idx = lst.index(sep)
return (lst[:idx], lst[idx+1:])
else:
return (lst[:], )
def args_to_dict(args):
"""Crudely parses "--blah=123" type arguments into dict like
{'blah': '123'}
"""
ret = {}
for a in args:
key, _, value = a.partition("=")
key = key.replace("--", "", 1)
ret[key] = value
return ret
if __name__ == '__main__':
import sys
# Get stuff before/after the "%%" separator
string, args = partition_list(sys.argv[1:], "%%")
# Join input string
string_joined = " ".join(string)
# Parse --args=stuff
d = args_to_dict(args)
# Do string-interpolation
print string_joined.format(**d)