Some lines of code are worth a thousand words. Create a python file test.py with the following:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--G', type=float, default=1, nargs='?')
args = parser.parse_args()
print (args.G / 3)
Then, in a terminal:
python test.py
gives 0, while:
python test.py --G 1
gives 0.333333333. I think this is because argparse doesn't seem to cast default arguments to their proper type (so 1 remains an int in default=1, in spite of the type=float), but it casts the argument if it is given explicitly.
I think this is inconsistent and prone to errors. Is there a reason why argparse behaves this way?
I think the 1 in default=1 is evaluated when you call parser.add_argument, where as the non-default value you pass as argument is evaluated at runtime, and therefore can be converted to float by argparse. It's not how argparse behaves; it's how python behaves in general. Consider this
def foo(x=1):
# there is no way to tell the function body that you want 1.0
# except for explicitly conversion
# because 1 is passed by value, which has already been evaluated
# and determined to be an int
print (x/3)
HUGE EDIT: Ha, I understand your point now and I think it's reasonable. So I dig into the source code and looks what I found:
https://hg.python.org/cpython/file/3.5/Lib/argparse.py#l1984
So argparse DOES appear to make sure your default value is type compliant, so long as it's a string. Try:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--G', type=float, default='1', nargs='?')
args = parser.parse_args()
print (args.G / 3)
Not sure if that's a bug or a compromise...
The other answers are right - only string defaults are passed through the type function. But there seems to be some reluctance to accept that logic.
Maybe this example will help:
import argparse
def mytype(astring):
return '['+astring+']'
parser = argparse.ArgumentParser()
parser.add_argument('--foo', type=mytype, default=1)
parser.add_argument('--bar', type=mytype, default='bar')
print parser.parse_args([])
print mytype(1)
produces:
0923:~/mypy$ python stack35429336.py
Namespace(bar='[bar]', foo=1)
Traceback (most recent call last):
File "stack35429336.py", line 8, in <module>
print mytype(1)
File "stack35429336.py", line 3, in mytype
return '['+astring+']'
TypeError: cannot concatenate 'str' and 'int' objects
I define a type function - it takes a string input, and returns something - anything that I want. And raises an error if it can't return that. Here I just append some characters to the string.
When the default is a string, it gets modified. But when it is a number (not a string) it is inserted without change. In fact as written mytype raises an error if given a number.
The argparse type is often confused with the function type(a). The latter returns values like int,str,bool. Plus the most common examples are int and float. But in argparse float is used as a function
float(x) -> floating point number
Convert a string or number to a floating point number, if possible.
type=bool is a common error. Parsing boolean values with argparse. bool() does not convert the string 'False' to the boolean False.
In [50]: bool('False')
Out[50]: True
If argparse passed every default through the type function, it would be difficult to place values like None or False in the namespace. No builtin function converts a string to None.
The key point is that the type parameter is a function (callable), not a casting operation or target.
For further clarification - or confusion - explore the default and type with nargs=2 or action='append'.
The purpose of type is to sanitise and validate the arbitrary input you receive from the command line. It's a first line of defence against bogus input. There is no need to "defend" yourself against the default value, because that's the value you have decided on yourself and you are in power and have full control over your application. type does not mean that the resulting value after parse_args() must be that type; it means that any input should be that type/should be the result of that function.
What the default value is is completely independent of that and entirely up to you. It's completely conceivable to have a default value of False, and if a user inputs a number for this optional argument, then it'll be a number instead.
Upgrading a comment below to be included here: Python's philosophy includes "everyone is a responsible adult and Python won't overly question your decisions" (paraphrased). This very much applies here I think.
The fact that argparse does cast default strings could be explained by the fact that CLI input is always a string, and it's possible that you're chaining several argparsers, and the default value is set by a previous input. Anything that's not a string type however must have been explicitly chosen by you, so it won't be further mangled.
If the default value is a string, the parser parses the value as if it were a command-line argument. In particular, the parser applies any type conversion argument, if provided, before setting the attribute on the Namespace return value. Otherwise, the parser uses the value as is:
parser = argparse.ArgumentParser()
parser.add_argument('--length', default='10', type=int)
parser.add_argument('--width', default=10.5, type=int)
args = parser.parse_args()
Here, args.length will be int 10 and args.width will be float 10.5, just as the default value.
The example is from https://docs.python.org/3.8/library/argparse.html#default.
At first, I also have the same question. Then I notice this example, it explains the question pretty clearly.
Related
Is it possible to tell argparse to give the same errors on default argument values as it would on user-specified argument values?
For example, the following will not result in any error:
parser = argparse.ArgumentParser()
parser.add_argument('--choice', choices=['a', 'b', 'c'], default='invalid')
args = vars(parser.parse_args()) # args = {'choice': 'invalid'}
whereas omitting the default, and having the user specify --choice=invalid on the command-line will result in an error (as expected).
Reason for asking is that I would like to have the user to be able to specify default command-line options in a JSON file which are then set using ArgumentParser.set_defaults(), but unfortunately the behaviour demonstrated above prevents these user-specified defaults from being validated.
Update: argparse is inconsistent and I now consider the behavior above to be a bug. The following does trigger an error:
parser = argparse.ArgumentParser()
parser.add_argument('--num', type=int, default='foo')
args = parser.parse_args() # triggers exception in case --num is not
# specified on the command-line
I have opened a bug report for this: https://github.com/python/cpython/issues/100949
I took the time to dig into the source code, and what is happening is that a check is only happening for arguments you gave on the command line. The only way to enforce a check, in my opinion, is to subclass ArgumentParser and have it do the check when you add the argument:
class ValidatingArgumentParser(argparse.ArgumentParser):
def add_argument(self, *args, **kwargs):
super().add_argument(*args, **kwargs)
self._check_value(self._actions[-1],kwargs['default'])
No. Explicit arguments need to be validated because they originate from outside the source code. Default values originate in the source code, so it's the job of the programmer, not the argument parser, to ensure they are valid.
(This is the difference between validation and debugging.)
(Using set_defaults on unvalidated user input still falls under the purview of debugging, as it's not the argument parser itself adding the default values, but the programmer.)
In a project I see the following use of add_argument():
parser = argparse.ArgumentParser()
parser.add_argument("--somearg",type=make_arg_instance,help='blahblahblah')
args = parser.parse_args()
i = args.somearg.somefunction()
in which make_arg_instanceis a function defined as:
def make_arg_instance():
somearg = SomeClassName()
return somearg
It seems the argument 'somearg' is an class object instead of int and that's why later a method is called as in i = args.somearg.somefunction().
In most tutorials that I've seen the type of argument is always int therefore could someone explain a bit about how to add argument of type class or any type other than int? Thanks!
From the documentation for the type argument:
By default, ArgumentParser objects read command-line arguments in as simple strings. However, quite often the command-line string should instead be interpreted as another type, like a float or int. The type keyword argument of add_argument() allows any necessary type-checking and type conversions to be performed.
[...]
type= can take any callable that takes a single string argument and returns the converted value
You can pass in any callable, provided it takes a single argument to be converted. int() is such a callable, but that doesn't mean you have to use that. If you want some other type or want to apply extra constraints to the allowed values, you can specify your own.
I want to manually handle the situation where parse_args() throws an error in case of a unknown value for an argument. For example:
If I have the following python file called script.py:
argp = argparse.ArgumentParser(description='example')
argp.add_argument('--compiler', choices=['default', 'clang3.4', 'clang3.5'])
args = argp.parse_args()
and I run the script with the following args python script.py --compiler=foo it throws the following error:
error: argument --compiler: invalid choice: 'foo' (choose from 'default', 'clang3.4', 'clang3.5')
SystemExit: 2
What do I need to do in order to handle this behaviour myself instead of the script quitting itself? One idea is to subclass argparse.ArgumentParser and override parse_args() or just monkey patch the method but I was wondering if there's a better way that does not require overriding the standard library behaviour?
The whole point to defining choices is to make the parser complain about values that are not in the list. But there are some alternatives:
omit choices (include them in the help text if you want), and do your own testing after parsing. argparse doesn't have to do everything for you. It's main purpose is to figure out what your user wants.
redefine the parser.error method (via subclassing is best) to redirect the error from sys.exit. But you'll have to parse the error message to distinguish between this error and other ones that the parser might raise.
define a type function that checks for choices, and makes the default substitution.
The parsing of the '--compiler' option goes something like this:
grab the string argument after the --compiler flag
pass it through the type function. Default is lambda x:x. int converts it to integer, etc. Raise ValueError is value is bad.
check the returned value against the choices list (if any)
use the action to add the value to the Namespace (default simply stores it).
Error in any of these steps produces an ArgumentError which is trapped by the parser.error method and passed to a parser.exit method.
Since the store_action occurs after type and choices checking, a custom action won't bypass their errors.
Here's a possible type solution (not tested)
def compile_choices(astr):
if astr in ['default', 'clang3.4', 'clang3.5']:
return astr
else:
return 'default'
# could raise ValueError('bad value') if there are some strings you don't like
argp.add_argument('--compiler', type=compile_choices)
=================
If compile_choices takes other arguments, such as the list of choices or the default, you'll need to wrap in some why that defines those values before parsing.
An example accepting a binary string representation:
parser.add_argument('--binary', type=lambda x: int(x, base=2),
help='integer in binary format', default='1010')
or
parser.add_argument('--binary', type=functools.partial(int, base=2), default='1010')
This question already has answers here:
type=dict in argparse.add_argument()
(13 answers)
Closed 9 years ago.
I'm trying to accept an argument of type=dict with argparse but no matter the input it gives an error of invalid dict value.
#!/usr/bin/env python
import argparse
MYDICT = {'key': 'value'}
parser = argparse.ArgumentParser()
parser.add_argument("-m", "--mydict", action="store",
required=False, type=dict,
default=MYDICT)
args = parser.parse_args()
print args.mydict
This is what happens when I try and pass a dictionary to the script
./argp.py -m "{'key1': 'value1'}"
usage: argp.py [-h] [-m MYDICT]
argp.py: error: argument -m/--mydict: invalid dict value: "{'key1': 'value1'}"
Looking at the documents I would think that this would be possible.
http://docs.python.org/dev/library/argparse.html
“Any object that supports the in operator can be passed as the choices value, so dict objects, set objects, custom containers, etc. are all supported.”
I do not think it is possible to pass a dictionary as an argument in the command line because there doesn't exist a conversion function from string to dict (EDIT: A hack is possible which gives similar behaviour, see below). What you are essentially telling python to do is:
dict("{'key1': 'value1'}")
Which if you try it out in the python console, does not work.
What the phrase:
"Any object that supports the in operator can be passed as the choices value, so dict objects, set objects, custom containers, etc. are all supported."
refers to is the choices argument that can be passed with the add_argument function - not to the type argument.
Your best bet is to probably accept your argument as a string and then convert it using the json capabilities of python:
parser.add_argument('-m', '--my-dict', type=str)
args = parser.parse_args()
import json
my_dictionary = json.loads(args.my_dict)
You can then pass a dictionary in the form of a string. You can try the json encoder/decoder out for yourself in the python console to see how it works:
>>>json.loads('{"value1":"key1"}')
{u'value1': u'key1'}
EDIT: hpaulj has pointed out to me that you can "hack" the type parameter by passing it json.loads which allows you to pass JSON that is similar looking to a dictionary.
import json
parser.add_argument('-d', '--my-dict', type=json.loads)
args = parse.parse_args()
mydict = args.my_dict # Will return a dictionary
NOTE: The input format you pass is not the same as python dictionary but is probably similar enough for your use case.
The reason this works is actually quite interesting because internally argparse will just use the parameter value as a function to convert the argument. i.e. if type=int then it will use int(arg) or if type=json.loads then json.loads(arg)
This also means that you can pass any function which takes a single parameter in as the argument to type and perform custom conversions if you need to :)
argparse uses per default abbreviation in unambiguous cases.
I don't want abbreviation and I'd like to disable it.
But didn't find it in the documentation.
Is it possible?
Example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--send', action='store_true')
parser.parse_args(['--se']) # returns Namespace(send=True)
But I want it only to be true when the full parameter is supplied. To prevent user errors.
UPDATE:
I created a ticket at python bugtracker after Vikas answer. And it already has been processed.
As of Python 3.5.0 you can disable abbreviations by initiating the ArgumentParser with the following:
parser = argparse.ArgumentParser(allow_abbrev=False)
Also see the documentation.
No, apparently this is not possible. At least in Python 2.7.2.
First, I took a look into the documentation - to no avail.
Then I opened the Lib\argparse.py and looked through the source code. Omitting a lot of details, it seems that each argument is parsed by a regular expression like this (argparse:2152):
# allow one or more arguments
elif nargs == ONE_OR_MORE:
nargs_pattern = '(-*A[A-]*)'
This regex will successfully parse both '-' and '--', so we have no control over the short and long arguments. Other regexes use the -* construct too, so it does not depend on the type of the parameter (no sub-arguments, 1 sub-argument etc).
Later in the code double dashes are converted to one dash (only for non-optional args), again, without any flags to control by user:
# if this is an optional action, -- is not allowed
if action.option_strings:
nargs_pattern = nargs_pattern.replace('-*', '')
nargs_pattern = nargs_pattern.replace('-', '')
No, well not without ugly hacks.
The code snippet #Vladimir posted, i suppose that is not what you are looking for. The actual code that is doing this is:
def _get_option_tuples(self, option_string):
...
if option_string.startswith(option_prefix):
...
See the check is startswith not ==.
And you can always extend argparse.ArgumentParser to provide your own _get_option_tuples(self, option_string) to change this behavior. I just did by replacing two occurrence of option_string.startswith(option_prefix) to option_string == option_prefix and:
>>> parser = my_argparse.MyArgparse
>>> parser = my_argparse.MyArgparse()
>>> parser.add_argument('--send', action='store_true')
_StoreTrueAction(option_strings=['--send'], dest='send', nargs=0, const=True, default=False, type=None, choices=None, help=None, metavar=None)
>>> parser.parse_args(['--se'])
usage: [-h] [--send]
: error: unrecognized arguments: --se
A word of caution
The method _get_option_tuples is prefixed with _, which typically means a private method in python. And it is not a good idea to override a private.
Another way for Python 2.7. Let's get clunky! Say you want to recognize --dog without abbreviation.
p = argparse.ArgumentParser()
p.add_argument('--dog')
p.add_argument('--dox', help=argparse.SUPPRESS, metavar='IGNORE')
By adding a second argument --dox that differs from the argument you want only in the third letter, --d and --do become ambiguous. Therefore, the parser will refuse to recognize them. You would need to add code to catch the resulting exception and process it according to the context in which you are calling parse_args. You might also need to suppress/tweak the help text.
The help=... keeps the argument out of the option list on the default help message (per this), and metavar='IGNORE' is just to make it clear you really aren't doing anything with this option :) .