We Keep Coding

Why does argparse not accept "--" as argument?

My script takes -d, --delimiter as argument:
parser.add_argument('-d', '--delimiter')
but when I pass it -- as delimiter, it is empty
script.py --delimiter='--'
I know -- is special in argument/parameter parsing, but I am using it in the form --option='--' and quoted.
Why does it not work?
I am using Python 3.7.3
Here is test code:
#!/bin/python3
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--delimiter')
parser.add_argument('pattern')
args = parser.parse_args()
print(args.delimiter)
When I run it as script --delimiter=-- AAA it prints empty args.delimiter.

This looks like a bug. You should report it.
This code in argparse.py is the start of _get_values, one of the primary helper functions for parsing values:
if action.nargs not in [PARSER, REMAINDER]:
try:
arg_strings.remove('--')
except ValueError:
pass
The code receives the -- argument as the single element of a list ['--']. It tries to remove '--' from the list, because when using -- as an end-of-options marker, the '--' string will end up in arg_strings for one of the _get_values calls. However, when '--' is the actual argument value, the code still removes it anyway, so arg_strings ends up being an empty list instead of a single-element list.
The code then goes through an else-if chain for handling different kinds of argument (branch bodies omitted to save space here):
# optional argument produces a default when not present
if not arg_strings and action.nargs == OPTIONAL:
...
# when nargs='*' on a positional, if there were no command-line
# args, use the default if it is anything other than None
elif (not arg_strings and action.nargs == ZERO_OR_MORE and
not action.option_strings):
...
# single argument or optional argument produces a single value
elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
...
# REMAINDER arguments convert all values, checking none
elif action.nargs == REMAINDER:
...
# PARSER arguments convert all values, but check only the first
elif action.nargs == PARSER:
...
# SUPPRESS argument does not put anything in the namespace
elif action.nargs == SUPPRESS:
...
# all other types of nargs produce a list
else:
...
This code should go through the 3rd branch,
# single argument or optional argument produces a single value
elif len(arg_strings) == 1 and action.nargs in [None, OPTIONAL]:
but because the argument is missing from arg_strings, len(arg_strings) is 0. It instead hits the final case, which is supposed to handle a completely different kind of argument. That branch ends up returning an empty list instead of the '--' string that should have been returned, which is why args.delimiter ends up being an empty list instead of a '--' string.
This bug manifests with positional arguments too. For example,
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('a')
parser.add_argument('b')
args = parser.parse_args(["--", "--", "--"])
print(args)
prints
Namespace(a='--', b=[])
because when _get_values handles the b argument, it receives ['--'] as arg_strings and removes the '--'. When handling the a argument, it receives ['--', '--'], representing one end-of-options marker and one actual -- argument value, and it successfully removes the end-of-options marker, but when handling b, it removes the actual argument value.

Existing bug report
Patches have been suggested, but it hasn't been applied. Argparse incorrectly handles '--' as argument to option
Some simple examples:
In [1]: import argparse
In [2]: p = argparse.ArgumentParser()
In [3]: a = p.add_argument('--foo')
In [4]: p.parse_args(['--foo=123'])
Out[4]: Namespace(foo='123')
The unexpected case:
In [5]: p.parse_args(['--foo=--'])
Out[5]: Namespace(foo=[])
Fully quote passes through - but I won't get into how you might achieve this via shell call:
In [6]: p.parse_args(['--foo="--"'])
Out[6]: Namespace(foo='"--"')
'--' as separate string:
In [7]: p.parse_args(['--foo','--'])
usage: ipython3 [-h] [--foo FOO]
ipython3: error: argument --foo: expected one argument
...
another example of the double quote:
In [8]: p.parse_args(['--foo','"--"'])
Out[8]: Namespace(foo='"--"')
In _parse_known_args, the input is scanned and classified as "O" or "A". The '--' is handled as
# all args after -- are non-options
if arg_string == '--':
arg_string_pattern_parts.append('-')
for arg_string in arg_strings_iter:
arg_string_pattern_parts.append('A')
I think the '--' are stripped out after that, but I haven't found that part of the code yet. I'm also not finding were the '--foo=...' version is handled.
I vaguely recall some bug/issues over handling of multiple occurances of '--'. With the migration to github, I'm not following argparse developements as much as I used to.
edit
get_values starts with:
def _get_values(self, action, arg_strings):
# for everything but PARSER, REMAINDER args, strip out first '--'
if action.nargs not in [PARSER, REMAINDER]:
try:
arg_strings.remove('--')
except ValueError:
pass
Why that results in a empty list will require more thought and testing.
The '=' is handled in _parse_optional, which is used during the first scan:
# if the option string before the "=" is present, return the action
if '=' in arg_string:
option_string, explicit_arg = arg_string.split('=', 1)
if option_string in self._option_string_actions:
action = self._option_string_actions[option_string]
return action, option_string, explicit_arg
old bug issues
argparse handling multiple "--" in args improperly
argparse: Allow the use of -- to break out of nargs and into subparser

It calls parse_args which calls parse_known_args which calls _parse_known_args.
Then, on line 2078 (or something similar), it does this (inside a while loop going through the string):
start_index = consume_optional(start_index)
which calls the consume_optional (which makes sense, because this is an optional argument it is parsing right now) defined earlier in the method _parse_known_args. When given --delimiter='--', it will make this action_tuples:
# if the action expect exactly one argument, we've
# successfully matched the option; exit the loop
elif arg_count == 1:
stop = start_index + 1
args = [explicit_arg]
action_tuples.append((action, args, option_string))
break
##
## The above code gives you the following:
##
action_tuples=[(_StoreAction(option_strings=['-d', '--delimiter'], dest='delimiter', nargs=None, const=None, default=None, type=None, choices=None, help=None, metavar=None), ['--'], '--delimiter')]
That is then iterated to, and is then fed to take_action on line 2009:
assert action_tuples
for action, args, option_string in action_tuples:
take_action(action, args, option_string)
return stop
The take_action function will then call self._get_values(action, argument_strings) on line 1918, which, as mentioned in the answer by #hpaulj, removes the --. Then, you're left with the empty list.

Comma separated inputs instead of space separated inputs for argparse

I'm using argparse to receive inputs from the command line to run my script.
My current input string looks like this:
path> python <\filename\\> -t T1 T2 T3 -f F1 F2
Is there a parameter in argparse such that instead of separating inputs by space, I can separate them by commas?
In other words:
path> python <\filename\\> -t T1,T2,T3 -f F1,F2

There is no such feature in argparse.
Alternatives:
post-process the args namespace and split/parse the values manually
define a custom action and split/parse the values manually
define a custom type and split/parse the values manually
subclass ArgumentParser and customise ArgumentParser.convert_arg_line_to_args

There are some useful answers here already, but I wanted a bit more: splitting on commas, validating values with choices, and getting useful error messages, so below I offer a solution.
Simple Version
As a first pass, we can just pass in an appropriate function to the type parameter:
>>> import argparse
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
But this doesn't work with choices because it checks if the whole list is in choices and not each value:
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','), choices=('a', 'b', 'c'))
>>> parser.parse_args(['--foo', 'a,b,c'])
usage: cmd [-h] [--foo {a,b,c}]
cmd: error: argument --foo: invalid choice: ['a', 'b', 'c'] (choose from 'a', 'b', 'c')
Setting nargs to something like * or + checks each value, but only for space-separated arguments (e.g., --foo a b) and not the comma-separated ones. It seems like there is no supported way to check if each value is in the choices if we produce the list ourselves. Therefore we need to raise errors ourselves via the type parameter (as Shiplu Mokaddim partially implemented). Creating a custom Action class sounds promising as actions have access to the choices, but the action happens after the type function applies and values are checked, so we still couldn't use the choices parameter on add_argument() for this purpose.
Better Version
Here is a solution using a custom type function. We this function to take a list of valid choices, but since the function for type conversion can only take an argument string, we need to wrap it in a class (and define the special __call__() method) or a function closure. This solution uses the latter.
>>> def csvtype(choices):
... """Return a function that splits and checks comma-separated values."""
... def splitarg(arg):
... values = arg.split(',')
... for value in values:
... if value not in choices:
... raise argparse.ArgumentTypeError(
... 'invalid choice: {!r} (choose from {})'
... .format(value, ', '.join(map(repr, choices))))
... return values
... return splitarg
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=csvtype(('a', 'b', 'c')))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
>>> parser.parse_args(['--foo', 'a,b,d'])
usage: cmd [-h] [-f F]
cmd: error: argument -f: invalid choice: 'd' (choose from 'a', 'b', 'c')
Notice that we get an appropriate error as well. For this, be sure to use argparse.ArgumentTypeError and not argparse.ArgumentError inside the function.
Other Options
User wim suggested some other options not discussed above. I don't find those attractive for the following reasons:
Post-processing the argument after parsing means you have to do more work to get the error messages to be consistent with those from argparse. Just raising argparse.ArgumentError will lead to a stacktrace. Also, argparse catches errors raised during parsing and alters them to specify the option that was used, which you'd otherwise need to do manually.
Subclassing ArgumentParser is more work, and convert_arg_line_to_args() is for reading arguments from a file, not the command line.

You can use module shlex to extract the parameters, then replace commas with spaces, and pass the results to argparse for further processing:
comma_args = shlex.split("-t T1,T2,T3 -f F1,F2")
# ['-t', 'T1,T2,T3', '-f', 'F1,F2']
args = [x.replace(","," ") for x in comma_args]
# ['-t', 'T1 T2 T3', '-f', 'F1 F2']
parse_args(args)

To ensure compatibility with choices, I use two parsers:
#!/usr/bin/env python3
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--arg", type=str, metavar="ARG[,ARG,...]")
# Pass the usage to the subparser, remove the usage: prefix.
subparser_usage = parser.format_usage().replace("usage: ", "", 1)
subparser = argparse.ArgumentParser(usage=subparser_usage)
subparser.add_argument("arg", nargs="*", type=int, choices=[1, 2, 3])
args = parser.parse_args()
if args.arg:
subargs = subparser.parse_args(args.arg.split(","))
args.arg = subargs.arg
print(args)
if __name__ == "__main__":
main()
This fulfills all possible requirements:
easy and understandable
involves only argparse
choices are ensured with a correct usage and help message
this allows to have also positional arguments (thanks to the comma separated list)
See the following examples:
$ ./example --arg 1,2,3
Namespace(arg=[1, 2, 3])
$ ./example --arg 1,2,4
usage: example [-h] [--arg ARG[,ARG,...]]
example: error: argument arg: invalid choice: 4 (choose from 1, 2, 3)
$ ./example --arg 1
Namespace(arg=[1])

Here this comma-separated input is actually a different type. All you have to is to define the type.
Here I define a custom type that does it.
class DelimiterSeperatedInput:
def __init__(self, item_type, separator=','):
self.item_type = item_type
self.separator = separator
def __call__(self, value):
values = []
try:
for val in value.split(self.separator):
typed_value = self.item_type(val)
values.append(typed_value)
except Exception:
raise ArgumentError("%s is not a valid argument" % value)
return values
parser.add_argument('-t', type=DelimiterSeperatedInput(str),
help='comma separated string values')
parser.add_argument('-f', type=DelimiterSeperatedInput(float, ":"),
help="colon separated floats')
This code may not work as-is, you might have to fix. But it was to give an idea.
Note: I could reduce the __call__ function body by using list, map etc. But then it wouldn't be very readable. Once you get the idea, you can do kinds of stuff with it.

If you're okay with space not comma separated, then it's builtin to argparse:
In [1]: from argparse import ArgumentParser
In [2]: parser = ArgumentParser()
In [3]: parser.add_argument('-a', nargs='+')
Out[3]: _StoreAction(option_strings=['-a'], dest='a', nargs='+', const=None,
default=None, type=None, choices=None, help=None, metavar=None)
In [4]: parser.parse_args(['-a', 'foo', 'bar'])
Out[4]: Namespace(a=['foo', 'bar'])

Is there a "missing" function in Python?

In R, there is a missing() function to test, quote : "whether a value was specified as an argument to a function" :
my_function <- function(argn1){
if(missing(argn1)){
print("argn1 has not been supplied")
} else {
print("argn1 has been supplied")
}
}
Then when calling :
my_function("hello")
[1] "argn1 has been supplied"
my_function()
[1] "argn1 has not been supplied"
Is there such a thing in Python ?

Well usually arguments without a default value are mandatory. So you can provide a default object missing for instance to check whether the attribute was given explicitly. Like:
missing = object()
def foo(arg1 = missing):
if arg1 is missing:
print('Arg1 is missing')
else:
print('Arg1 is not missing')
Using the is over == can be of vital importance, since is checks reference equality.
Sometimes one uses None, like:
def foo(arg1 = None):
if arg1 is None:
# ...
But note that here Python cannot make a difference between an implicit argument, like foo() or an explicit call with None, like foo(None).
Furthermore there is also the option to use *args:
def foo(*args):
# ...
If you call foo(None,1) then all the arguments will be put into a tuple an that tuple is named args (here args will be args = (None,1)). So then we can check if the tuple contains at least one element:
def foo(*args):
if args:
print('At least one element provided')
else:
print('No element provided')

No, because Python does not support calling a function with the wrong number of arguments. If you define a function to take one argument (without a default):
def my_func(arg1):
pass
and then call it just via my_func(), Python will raise a TypeError.

No, there is no missing function.
However, you can do the same thing by using function arguments with default values:
def foo(arg=None):
if arg is None:
print('arg is missing.')
else:
print('arg is present.')

In python, you will often find this kind of function declaration:
def my_function(arg1=None):
if arg1:
# Do something
else:
# Do something else
to achieve what you are looking for

There is no obvious equivalent. But you can use default parameters.
def my_function(str=None):
if str is None:
print("argn1 has not been supplied")
else:
print("argn1 has been supplied")

Checking if variable exists in Namespace

I'm trying to use the output of my argparse (simple argparse with just 4 positional arguments that each kick of a function depending on the variable that is set to True)
Namespace(battery=False, cache=True, health=False, hotspare=False)
At the moment I'm trying to figure out how to best ask python to see when one of those variables is set to True; without having to hardcode like I do now:
if args.battery is True:
do_something()
elif args.cache is True:
do_something_else()
etc ...
I'd rather just use one command to check if a variable exists within the namespace and if it's set to True; but I can't for the life of me figure out how to do this in an efficient manner.

You can use hasattr(ns, "battery") (assume ns = Namespace(battery=False, cache=True, health=False, hotspare=False)).
Much cleaner than vars(ns).get("battery") I would think.

Use vars() to convert your namespace to a dictionary, then use dict.get('your key') which will return your object if it exists, or a None when it doesn't.
Example
my_args.py
import argparse
_parser = argparse.ArgumentParser("A description of your program")
_parser.add_argument("--some-arg", help="foo", action="store_true")
_parser.add_argument("--another-one", type=str)
your_args_dict = vars(_parser.parse_args())
print(your_args_dict.get('some_arg'))
print(your_args_dict.get('another_one'))
print(your_args_dict.get('foo_bar'))
command line
$ python3 my_args.py --some-arg --another-one test
True
test
None

If solved my problem with a list comprehension (after using the tip to use vars() ) :
l = [ k for (k,v) in args.items() if v ]
l is a list of keys in the dict that have a value of 'True'

Answering the title (not the OP detailed question), to check that variable a is defined in the ns namespace:
'a' in vars(ns)

Parse Args that aren't declared

I'm writing a utility for running bash commands that essentially takes as input a string and a list optional argument and uses the optional arguments to interpolate string.
I'd like it to work like this:
interpolate.py Hello {user_arg} my name is {computer_arg} %% --user_arg=john --computer_arg=hal
The %% is a separator, it separates the string to be interpolated from the arguments. What follows is the arguments used to interpolate the string. In this case the user has chosen user_arg and computer_arg as arguments. The program can't know in advance which argument names the user will choose.
My problem is how to parse the arguments? I can trivially split the input arguments on the separator but I can't figure out how to get optparse to just give the list of optional args as a dictionary, without specifying them in advance. Does anyone know how to do this without writing a lot of regex?

Well, if you use '--' to separate options from arguments instead of %%, optparse/argparse will just give you the arguments as a plain list (treating them as positional arguments instead of switched). After that it's not 'a lot of' regex, it's just a mere split:
for argument in args:
if not argument.startswith("--"):
# decide what to do in this case...
continue
arg_name, arg_value = argument.split("=", 1)
arg_name = arg_name[2:]
# use the argument any way you like

With argparse you could use the parse_known_args method to consume predefined arguments and any additional arguments. For example, using the following script
import sys
import argparse
def main(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument('string', type=str, nargs='*',
help="""String to process. Optionally with interpolation
(explain this here...)""")
args, opt_args = parser.parse_known_args(argv)
print args
print opt_args
return 0
if __name__=='__main__':
sys.exit(main(sys.argv[1:]))
and calling with
python script.py Hello, my name is {name} --name=chris
yields the following output:
Namespace(string=['Hello,' 'my', 'name', 'is', '{name}'])
['--name=chris']
All that is left to do is to loop through the args namespace looking for strings of the form {...} and replacing them with the corresponding element in opt_args, if present. (I'm not sure if argparse can do argument interpolation automatically, the above example is the only immediate solution which comes to mind).

For something like this, you really don't need optparse or argparse - the benefit of such libraries are of little use in this circumstance (things like lone -v type arguments, checking for invalid options, value validation and so on)
def partition_list(lst, sep):
"""Slices a list in two, cutting on index matching "sep"
>>> partition_list(['a', 'b', 'c'], sep='b')
(['a'], ['c'])
"""
if sep in lst:
idx = lst.index(sep)
return (lst[:idx], lst[idx+1:])
else:
return (lst[:], )
def args_to_dict(args):
"""Crudely parses "--blah=123" type arguments into dict like
{'blah': '123'}
"""
ret = {}
for a in args:
key, _, value = a.partition("=")
key = key.replace("--", "", 1)
ret[key] = value
return ret
if __name__ == '__main__':
import sys
# Get stuff before/after the "%%" separator
string, args = partition_list(sys.argv[1:], "%%")
# Join input string
string_joined = " ".join(string)
# Parse --args=stuff
d = args_to_dict(args)
# Do string-interpolation
print string_joined.format(**d)