We Keep Coding

Why does argparse not accept "--" as argument?

My script takes -d, --delimiter as argument:
parser.add_argument('-d', '--delimiter')
but when I pass it -- as delimiter, it is empty
script.py --delimiter='--'
I know -- is special in argument/parameter parsing, but I am using it in the form --option='--' and quoted.
Why does it not work?
I am using Python 3.7.3
Here is test code:
#!/bin/python3
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--delimiter')
parser.add_argument('pattern')
args = parser.parse_args()
print(args.delimiter)
When I run it as script --delimiter=-- AAA it prints empty args.delimiter.

This looks like a bug. You should report it.
This code in argparse.py is the start of _get_values, one of the primary helper functions for parsing values:
if action.nargs not in [PARSER, REMAINDER]:
try:
arg_strings.remove('--')
except ValueError:
pass
The code receives the -- argument as the single element of a list ['--']. It tries to remove '--' from the list, because when using -- as an end-of-options marker, the '--' string will end up in arg_strings for one of the _get_values calls. However, when '--' is the actual argument value, the code still removes it anyway, so arg_strings ends up being an empty list instead of a single-element list.
The code then goes through an else-if chain for handling different kinds of argument (branch bodies omitted to save space here):
# optional argument produces a default when not present
if not arg_strings and action.nargs == OPTIONAL:
...
# when nargs='*' on a positional, if there were no command-line
# args, use the default if it is anything other than None
elif (not arg_strings and action.nargs == ZERO_OR_MORE and
not action.option_strings):
...
# single argument or optional argument produces a single value
elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
...
# REMAINDER arguments convert all values, checking none
elif action.nargs == REMAINDER:
...
# PARSER arguments convert all values, but check only the first
elif action.nargs == PARSER:
...
# SUPPRESS argument does not put anything in the namespace
elif action.nargs == SUPPRESS:
...
# all other types of nargs produce a list
else:
...
This code should go through the 3rd branch,
# single argument or optional argument produces a single value
elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
but because the argument is missing from arg_strings, len(arg_strings) is 0. It instead hits the final case, which is supposed to handle a completely different kind of argument. That branch ends up returning an empty list instead of the '--' string that should have been returned, which is why args.delimiter ends up being an empty list instead of a '--' string.
This bug manifests with positional arguments too. For example,
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('a')
parser.add_argument('b')
args = parser.parse_args(["--", "--", "--"])
print(args)
prints
Namespace(a='--', b=[])
because when _get_values handles the b argument, it receives ['--'] as arg_strings and removes the '--'. When handling the a argument, it receives ['--', '--'], representing one end-of-options marker and one actual -- argument value, and it successfully removes the end-of-options marker, but when handling b, it removes the actual argument value.

Existing bug report
Patches have been suggested, but it hasn't been applied. Argparse incorrectly handles '--' as argument to option
Some simple examples:
In [1]: import argparse
In [2]: p = argparse.ArgumentParser()
In [3]: a = p.add_argument('--foo')
In [4]: p.parse_args(['--foo=123'])
Out[4]: Namespace(foo='123')
The unexpected case:
In [5]: p.parse_args(['--foo=--'])
Out[5]: Namespace(foo=[])
Fully quote passes through - but I won't get into how you might achieve this via shell call:
In [6]: p.parse_args(['--foo="--"'])
Out[6]: Namespace(foo='"--"')
'--' as separate string:
In [7]: p.parse_args(['--foo','--'])
usage: ipython3 [-h] [--foo FOO]
ipython3: error: argument --foo: expected one argument
...
another example of the double quote:
In [8]: p.parse_args(['--foo','"--"'])
Out[8]: Namespace(foo='"--"')
In _parse_known_args, the input is scanned and classified as "O" or "A". The '--' is handled as
# all args after -- are non-options
if arg_string == '--':
arg_string_pattern_parts.append('-')
for arg_string in arg_strings_iter:
arg_string_pattern_parts.append('A')
I think the '--' are stripped out after that, but I haven't found that part of the code yet. I'm also not finding were the '--foo=...' version is handled.
I vaguely recall some bug/issues over handling of multiple occurances of '--'. With the migration to github, I'm not following argparse developements as much as I used to.
edit
get_values starts with:
def _get_values(self, action, arg_strings):
# for everything but PARSER, REMAINDER args, strip out first '--'
if action.nargs not in [PARSER, REMAINDER]:
try:
arg_strings.remove('--')
except ValueError:
pass
Why that results in a empty list will require more thought and testing.
The '=' is handled in _parse_optional, which is used during the first scan:
# if the option string before the "=" is present, return the action
if '=' in arg_string:
option_string, explicit_arg = arg_string.split('=', 1)
if option_string in self._option_string_actions:
action = self._option_string_actions[option_string]
return action, option_string, explicit_arg
old bug issues
argparse handling multiple "--" in args improperly
argparse: Allow the use of -- to break out of nargs and into subparser

It calls parse_args which calls parse_known_args which calls _parse_known_args.
Then, on line 2078 (or something similar), it does this (inside a while loop going through the string):
start_index = consume_optional(start_index)
which calls the consume_optional (which makes sense, because this is an optional argument it is parsing right now) defined earlier in the method _parse_known_args. When given --delimiter='--', it will make this action_tuples:
# if the action expect exactly one argument, we've
# successfully matched the option; exit the loop
elif arg_count == 1:
stop = start_index + 1
args = [explicit_arg]
action_tuples.append((action, args, option_string))
break
##
## The above code gives you the following:
##
action_tuples=[(_StoreAction(option_strings=['-d', '--delimiter'], dest='delimiter', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None), ['--'], '--delimiter')]
That is then iterated to, and is then fed to take_action on line 2009:
assert action_tuples
for action, args, option_string in action_tuples:
take_action(action, args, option_string)
return stop
The take_action function will then call self._get_values(action, argument_strings) on line 1918, which, as mentioned in the answer by #hpaulj, removes the --. Then, you're left with the empty list.

Passing multiple sets of arguments [duplicate]

I need to let the end user of my python script types something like:
script.py -sizes <2,2> <3,3> <6,6>
where each element of the -sizes option is a pair of two positive integers. How can I achieve this with argparse ?

Define a custom type:
def pair(arg):
# For simplity, assume arg is a pair of integers
# separated by a comma. If you want to do more
# validation, raise argparse.ArgumentError if you
# encounter a problem.
return [int(x) for x in arg.split(',')]
then use this as the type for a regular argument:
p.add_argument('--sizes', type=pair, nargs='+')
Then
>>> p.parse_args('--sizes 1,3 4,6'.split())
Namespace(sizes=[[1, 3], [4, 6]])

Argparse doesn't try to cover all possible formats of input data. You can always get sizes as string and parse them with few lines of code:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--sizes', nargs='+')
args = parser.parse_args()
try:
sizes = [tuple(map(int, s.split(',', maxsplit=1))) for s in args.sizes]
except Exception:
print('sizes cannot be parsed')
print(sizes)
"Special cases aren't special enough to break the rules."

Make argparse treat dashes and underscore identically

argparse replaces dashes in optional arguments by underscores to determine their destination:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--use-unicorns', action='store_true')
args = parser.parse_args(['--use-unicorns'])
print(args) # returns: Namespace(use_unicorns=True)
However the user has to remember whether the option is --use-unicorns or --use_unicorns; using the wrong variant raises an error.
This can cause some frustration as the variable args.use_unicorns in the code does not make it clear which variant was defined.
How can I make argparse accept both --use-unicorns and --use_unicorns as valid ways to define this optional argument?

parser.add_argument('--use-unicorns', action='store_true')
args = parser.parse_args(['--use-unicorns'])
print(args) # returns: Namespace(use_unicorns=True)
argparse translates the '-' to '_' because the use of '-' in flags is well established POSIX practice. But args.use-unicones is not acceptable Python. In other words, it does the translation so the dest will be a valid Python variable or attribute name.
Note that argparse does not perform this translation with positionals. In that case the programmer has full control over the dest parameter, and can choose anything that's convenient. Since the argparse only uses getattr and setattr when accessing the Namespace, the constraints on a valid dest are minimal.
There are two users. There's you, the programmer, and there's your end user. What's convenient for you might not be optimal for the other.
You can also specify a dest with defining an optional. And metavar gives you further control over the help display.
It's parser._get_optional_kwargs that performs the '-' replace:
if dest is None:
....
dest = dest.replace('-', '_')

parser.add_argument accepts more than one flag for an argument (link to documentation). One easy way to make the parser accept both variants is to declare the argument as
parser.add_argument('--use-unicorns', '--use_unicorns', action='store_true')
However both options will show up in the help, and it is not very elegant as it forces one to write the variants manually.
An alternative is to subclass argparse.ArgumentParser to make the matching invariant to replacing dashes by underscore. This requires a little bit of fiddling, as both argparse_ActionsContainer._parse_optional and argparse_ActionsContainer._get_option_tuples have to be modified to handle this matching and abbrevations, e.g. --use_unic.
I ended up with the following subclassed method, where the matching to abbrevations is delegated from _parse_optional to _get_option_tuples:
from gettext import gettext as _
import argparse
class ArgumentParser(argparse.ArgumentParser):
def _parse_optional(self, arg_string):
# if it's an empty string, it was meant to be a positional
if not arg_string:
return None
# if it doesn't start with a prefix, it was meant to be positional
if not arg_string[0] in self.prefix_chars:
return None
# if it's just a single character, it was meant to be positional
if len(arg_string) == 1:
return None
option_tuples = self._get_option_tuples(arg_string)
# if multiple actions match, the option string was ambiguous
if len(option_tuples) > 1:
options = ', '.join([option_string
for action, option_string, explicit_arg in option_tuples])
args = {'option': arg_string, 'matches': options}
msg = _('ambiguous option: %(option)s could match %(matches)s')
self.error(msg % args)
# if exactly one action matched, this segmentation is good,
# so return the parsed action
elif len(option_tuples) == 1:
option_tuple, = option_tuples
return option_tuple
# if it was not found as an option, but it looks like a negative
# number, it was meant to be positional
# unless there are negative-number-like options
if self._negative_number_matcher.match(arg_string):
if not self._has_negative_number_optionals:
return None
# if it contains a space, it was meant to be a positional
if ' ' in arg_string:
return None
# it was meant to be an optional but there is no such option
# in this parser (though it might be a valid option in a subparser)
return None, arg_string, None
def _get_option_tuples(self, option_string):
result = []
if '=' in option_string:
option_prefix, explicit_arg = option_string.split('=', 1)
else:
option_prefix = option_string
explicit_arg = None
if option_prefix in self._option_string_actions:
action = self._option_string_actions[option_prefix]
tup = action, option_prefix, explicit_arg
result.append(tup)
else: # imperfect match
chars = self.prefix_chars
if option_string[0] in chars and option_string[1] not in chars:
# short option: if single character, can be concatenated with arguments
short_option_prefix = option_string[:2]
short_explicit_arg = option_string[2:]
if short_option_prefix in self._option_string_actions:
action = self._option_string_actions[short_option_prefix]
tup = action, short_option_prefix, short_explicit_arg
result.append(tup)
underscored = {k.replace('-', '_'): k for k in self._option_string_actions}
option_prefix = option_prefix.replace('-', '_')
if option_prefix in underscored:
action = self._option_string_actions[underscored[option_prefix]]
tup = action, underscored[option_prefix], explicit_arg
result.append(tup)
elif self.allow_abbrev:
for option_string in underscored:
if option_string.startswith(option_prefix):
action = self._option_string_actions[underscored[option_string]]
tup = action, underscored[option_string], explicit_arg
result.append(tup)
# return the collected option tuples
return result
A lot of this code is directly derived from the corresponding methods in argparse (from the CPython implementation here). Using this subclass should make the matching of optional arguments invariant to using dashes - or underscores _.

Comma separated inputs instead of space separated inputs for argparse

I'm using argparse to receive inputs from the command line to run my script.
My current input string looks like this:
path> python <\filename\\> -t T1 T2 T3 -f F1 F2
Is there a parameter in argparse such that instead of separating inputs by space, I can separate them by commas?
In other words:
path> python <\filename\\> -t T1,T2,T3 -f F1,F2

There is no such feature in argparse.
Alternatives:
post-process the args namespace and split/parse the values manually
define a custom action and split/parse the values manually
define a custom type and split/parse the values manually
subclass ArgumentParser and customise ArgumentParser.convert_arg_line_to_args

There are some useful answers here already, but I wanted a bit more: splitting on commas, validating values with choices, and getting useful error messages, so below I offer a solution.
Simple Version
As a first pass, we can just pass in an appropriate function to the type parameter:
>>> import argparse
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
But this doesn't work with choices because it checks if the whole list is in choices and not each value:
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','), choices=('a', 'b', 'c'))
>>> parser.parse_args(['--foo', 'a,b,c'])
usage: cmd [-h] [--foo {a,b,c}]
cmd: error: argument --foo: invalid choice: ['a', 'b', 'c'] (choose from 'a', 'b', 'c')
Setting nargs to something like * or + checks each value, but only for space-separated arguments (e.g., --foo a b) and not the comma-separated ones. It seems like there is no supported way to check if each value is in the choices if we produce the list ourselves. Therefore we need to raise errors ourselves via the type parameter (as Shiplu Mokaddim partially implemented). Creating a custom Action class sounds promising as actions have access to the choices, but the action happens after the type function applies and values are checked, so we still couldn't use the choices parameter on add_argument() for this purpose.
Better Version
Here is a solution using a custom type function. We this function to take a list of valid choices, but since the function for type conversion can only take an argument string, we need to wrap it in a class (and define the special __call__() method) or a function closure. This solution uses the latter.
>>> def csvtype(choices):
... """Return a function that splits and checks comma-separated values."""
... def splitarg(arg):
... values = arg.split(',')
... for value in values:
... if value not in choices:
... raise argparse.ArgumentTypeError(
... 'invalid choice: {!r} (choose from {})'
... .format(value, ', '.join(map(repr, choices))))
... return values
... return splitarg
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=csvtype(('a', 'b', 'c')))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
>>> parser.parse_args(['--foo', 'a,b,d'])
usage: cmd [-h] [-f F]
cmd: error: argument -f: invalid choice: 'd' (choose from 'a', 'b', 'c')
Notice that we get an appropriate error as well. For this, be sure to use argparse.ArgumentTypeError and not argparse.ArgumentError inside the function.
Other Options
User wim suggested some other options not discussed above. I don't find those attractive for the following reasons:
Post-processing the argument after parsing means you have to do more work to get the error messages to be consistent with those from argparse. Just raising argparse.ArgumentError will lead to a stacktrace. Also, argparse catches errors raised during parsing and alters them to specify the option that was used, which you'd otherwise need to do manually.
Subclassing ArgumentParser is more work, and convert_arg_line_to_args() is for reading arguments from a file, not the command line.

You can use module shlex to extract the parameters, then replace commas with spaces, and pass the results to argparse for further processing:
comma_args = shlex.split("-t T1,T2,T3 -f F1,F2")
# ['-t', 'T1,T2,T3', '-f', 'F1,F2']
args = [x.replace(","," ") for x in comma_args]
# ['-t', 'T1 T2 T3', '-f', 'F1 F2']
parse_args(args)

To ensure compatibility with choices, I use two parsers:
#!/usr/bin/env python3
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--arg", type=str, metavar="ARG[,ARG,...]")
# Pass the usage to the subparser, remove the usage: prefix.
subparser_usage = parser.format_usage().replace("usage: ", "", 1)
subparser = argparse.ArgumentParser(usage=subparser_usage)
subparser.add_argument("arg", nargs="*", type=int, choices=[1, 2, 3])
args = parser.parse_args()
if args.arg:
subargs = subparser.parse_args(args.arg.split(","))
args.arg = subargs.arg
print(args)
if __name__ == "__main__":
main()
This fulfills all possible requirements:
easy and understandable
involves only argparse
choices are ensured with a correct usage and help message
this allows to have also positional arguments (thanks to the comma separated list)
See the following examples:
$ ./example --arg 1,2,3
Namespace(arg=[1, 2, 3])
$ ./example --arg 1,2,4
usage: example [-h] [--arg ARG[,ARG,...]]
example: error: argument arg: invalid choice: 4 (choose from 1, 2, 3)
$ ./example --arg 1
Namespace(arg=[1])

Here this comma-separated input is actually a different type. All you have to is to define the type.
Here I define a custom type that does it.
class DelimiterSeperatedInput:
def __init__(self, item_type, separator=','):
self.item_type = item_type
self.separator = separator
def __call__(self, value):
values = []
try:
for val in value.split(self.separator):
typed_value = self.item_type(val)
values.append(typed_value)
except Exception:
raise ArgumentError("%s is not a valid argument" % value)
return values
parser.add_argument('-t', type=DelimiterSeperatedInput(str),
help='comma separated string values')
parser.add_argument('-f', type=DelimiterSeperatedInput(float, ":"),
help="colon separated floats')
This code may not work as-is, you might have to fix. But it was to give an idea.
Note: I could reduce the __call__ function body by using list, map etc. But then it wouldn't be very readable. Once you get the idea, you can do kinds of stuff with it.

If you're okay with space not comma separated, then it's builtin to argparse:
In [1]: from argparse import ArgumentParser
In [2]: parser = ArgumentParser()
In [3]: parser.add_argument('-a', nargs='+')
Out[3]: _StoreAction(option_strings=['-a'], dest='a', nargs='+', const=None,
default=None, type=None, choices=None, help=None, metavar=None)
In [4]: parser.parse_args(['-a', 'foo', 'bar'])
Out[4]: Namespace(a=['foo', 'bar'])

Python Argparse: Issue with optional arguments which are negative numbers

I'm having a small issue with argparse. I have an option xlim which is the xrange of a plot. I want to be able to pass numbers like -2e-5. However this does not work - argparse interprets this is a positional argument. If I do -0.00002 it works: argparse reads it as a negative number. Is it possible to have able to read in -2e-3?
The code is below, and an example of how I would run it is:
./blaa.py --xlim -2.e-3 1e4
If I do the following it works:
./blaa.py --xlim -0.002 1e4
The code:
parser.add_argument('--xlim', nargs = 2,
help = 'X axis limits',
action = 'store', type = float,
default = [-1.e-3, 1.e-3])
Whilst I can get it to work this way I would really rather be able to use scientific notation. Anyone have any ideas?
Cheers

One workaround I've found is to quote the value, but adding a space. That is,
./blaa.py --xlim " -2.e-3" 1e4
This way argparse won't think -2.e-3 is an option name because the first character is not a hyphen-dash, but it will still be converted properly to a float because float(string) ignores spaces on the left.

As already pointed out by the comments, the problem is that a - prefix is parsed as an option instead of as an argument. One way to workaround this is change the prefix used for options with prefix_chars argument:
#!/usr/bin/python
import argparse
parser = argparse.ArgumentParser(prefix_chars='#')
parser.add_argument('##xlim', nargs = 2,
help = 'X axis limits',
action = 'store', type = float,
default = [-1.e-3, 1.e-3])
print parser.parse_args()
Example output:
$ ./blaa.py ##xlim -2.e-3 1e4
Namespace(xlim=[-0.002, 10000.0])
Edit: Alternatively, you can keep using - as separator, pass xlim as a single value and use a function in type to implement your own parsing:
#!/usr/bin/python
import argparse
def two_floats(value):
values = value.split()
if len(values) != 2:
raise argparse.ArgumentError
values = map(float, values)
return values
parser = argparse.ArgumentParser()
parser.add_argument('--xlim',
help = 'X axis limits',
action = 'store', type=two_floats,
default = [-1.e-3, 1.e-3])
print parser.parse_args()
Example output:
$ ./blaa.py --xlim "-2e-3 1e4"
Namespace(xlim=[-0.002, 10000.0])

If you specify the value for your option with an equals sign, argparse will not treat it as a separate option, even if it starts with -:
./blaa.py --xlim='-0.002 1e4'
# As opposed to --xlim '-0.002 1e4'
And if the value does not have spaces in it (or other special characters given your shell), you can drop the quotes:
./blaa.py --xlim=-0.002
See: https://www.gnu.org/software/guile/manual/html_node/Command-Line-Format.html
With this, there is no need to write your own type= parser or redefine the prefix character from - to # as the accepted answer suggests.

Here is the code that I use. (It is similar to jeremiahbuddha's but it answers the question more directly since it deals with negative numbers.)
Put this before calling argparse.ArgumentParser()
for i, arg in enumerate(sys.argv):
if (arg[0] == '-') and arg[1].isdigit(): sys.argv[i] = ' ' + arg

Another workaround is to pass in the argument using '=' symbol in addition to quoting the argument - i.e., --xlim="-2.3e14"

If you are up to modifying argparse.py itself, you could change the negative number matcher to handle scientific notation:
In class _ActionsContainer.__init__()
self._negative_number_matcher = _re.compile(r'^-(\d+\.?|\d*\.\d+)([eE][+\-]?\d+)?$')
Or after creating the parser, you could set parser._negative_number_matcher to this value. This approach might have problems if you are creating groups or subparsers, but should work with a simple parser.

Inspired by andrewfn's approach, I created a separate helper function to do the sys.argv fiddling:
def _tweak_neg_scinot():
import re
import sys
p = re.compile('-\\d*\\.?\\d*e', re.I)
sys.argv = [' ' + a if p.match(a) else a for a in sys.argv]
The regex looks for:
- : a negative sign
\\d* : zero or more digits (for oddly formatted values like -.5e-2 or -4354.5e-6)
\\.? : an optional period (e.g., -2e-5 is reasonable)
\\d* : another set of zero or more digits (for things like -2e-5 and -7.e-3)
e : to match the exponent marker
re.I makes it match both -2e-5 and -2E-5. Using p.match means that it only searches from the start of each string.