This question already has answers here:
type=dict in argparse.add_argument()
(13 answers)
Closed 9 years ago.
I'm trying to accept an argument of type=dict with argparse but no matter the input it gives an error of invalid dict value.
#!/usr/bin/env python
import argparse
MYDICT = {'key': 'value'}
parser = argparse.ArgumentParser()
parser.add_argument("-m", "--mydict", action="store",
required=False, type=dict,
default=MYDICT)
args = parser.parse_args()
print args.mydict
This is what happens when I try and pass a dictionary to the script
./argp.py -m "{'key1': 'value1'}"
usage: argp.py [-h] [-m MYDICT]
argp.py: error: argument -m/--mydict: invalid dict value: "{'key1': 'value1'}"
Looking at the documents I would think that this would be possible.
http://docs.python.org/dev/library/argparse.html
“Any object that supports the in operator can be passed as the choices value, so dict objects, set objects, custom containers, etc. are all supported.”
I do not think it is possible to pass a dictionary as an argument in the command line because there doesn't exist a conversion function from string to dict (EDIT: A hack is possible which gives similar behaviour, see below). What you are essentially telling python to do is:
dict("{'key1': 'value1'}")
Which if you try it out in the python console, does not work.
What the phrase:
"Any object that supports the in operator can be passed as the choices value, so dict objects, set objects, custom containers, etc. are all supported."
refers to is the choices argument that can be passed with the add_argument function - not to the type argument.
Your best bet is to probably accept your argument as a string and then convert it using the json capabilities of python:
parser.add_argument('-m', '--my-dict', type=str)
args = parser.parse_args()
import json
my_dictionary = json.loads(args.my_dict)
You can then pass a dictionary in the form of a string. You can try the json encoder/decoder out for yourself in the python console to see how it works:
>>>json.loads('{"value1":"key1"}')
{u'value1': u'key1'}
EDIT: hpaulj has pointed out to me that you can "hack" the type parameter by passing it json.loads which allows you to pass JSON that is similar looking to a dictionary.
import json
parser.add_argument('-d', '--my-dict', type=json.loads)
args = parse.parse_args()
mydict = args.my_dict # Will return a dictionary
NOTE: The input format you pass is not the same as python dictionary but is probably similar enough for your use case.
The reason this works is actually quite interesting because internally argparse will just use the parameter value as a function to convert the argument. i.e. if type=int then it will use int(arg) or if type=json.loads then json.loads(arg)
This also means that you can pass any function which takes a single parameter in as the argument to type and perform custom conversions if you need to :)
Related
I am working on a project with many command line arguments and I'd like to be able to specify all of the arguments in the form of a file (i.e. JSON), and load this file into an argparse object, instead of pasting them into the terminal every time. I also need to enforce the presence of required arguments. I've found a couple options but none of them do exactly what I need.
If I have something like this
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--image_path", type=str, required=True)
parser.add_argument("--count", type=int, required=True)
args = parser.parse_args()
One approach might be to add something like this to update the dictionary with the JSON contents
args_dict = vars(args)
with open('path/to/args.json', 'rb') as f:
args_dict.update(json.load(f))
But an error occurs at the parse_args() line, since it didn't see the required arguments passed in.
The other approach I considered is to use
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
and just pass the file name, prefixed with #, to the command. The problem here is that I have to convert the dictionary-like format (JSON) to a line-separated list of arguments, as noted in the documentation. This is not trivial in my case.
I also considered just saving the arguments as a line-separated list to begin with, but the way that I know how to do that involves iterating over sys.argv and printing to a text file, which is problematic because I have some args that are assigned default values, which are not represented in sys.argv. I need to record the default values that were used for later reference.
It seems like there should be a simple way to do this, but I'm stumped at the moment.
I need to pass a dictionary to a python module via a command line. I have not found a way to get Python to cooperate. The following is a very simplified example of what I'd like to accomplish. This produces the following error TypeError: _get_positional_kwargs() missing 1 required positional argument: 'dest'
. Adding a dest argument to the add_argument line results in TypeError: __init__() got an unexpected keyword argument 'kwargs', indicating the sad possibility that argparse simply may not take a dictionary as an argument. Is a pickle file truly the only option for passing dictionaries back and forth from command line calls?
def run(**kwargs):
for i in kwargs.items():
print(f'{i}: {kwargs[i]}')
# ...some code to manipulate the dictionary
return kwargs
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Test script")
parser.add_argument(kwargs='**', description='A dictionary to be manipulated and returned')
args = parser.parse_args()
x = run(args)
pass
Additional info - I know that I can pass a list to a *nargs argument with a command line that includes space-delimited arguments, but I need to do something similar with a dictionary.
UPDATE - WHAT WORKED:
Based on the helpful input from #chepner, I was able to get it to work with the json solution he mentioned in his answer. On my Windows platform, I did have to remove all spaces from the dictionary string and escape characters are required in the final command line string.
{\"this\":1,\"is\":2,\"a\":5,\"dictionary\":7}
Arguments are strings, period. What you want to pass is a string that contains an encoded dict, for example using JSON.
import argparse
import json
parser = argparse.ArgumentParser()
parser.add_argument('data', type=json.loads)
args = parser.parse_args()
If you run your script with something like
myscript '{"foo": 3, "bar": "hello"}'
then args.data["foo"] == 3 and args.data["bar"] == "hello".
In a project I see the following use of add_argument():
parser = argparse.ArgumentParser()
parser.add_argument("--somearg",type=make_arg_instance,help='blahblahblah')
args = parser.parse_args()
i = args.somearg.somefunction()
in which make_arg_instanceis a function defined as:
def make_arg_instance():
somearg = SomeClassName()
return somearg
It seems the argument 'somearg' is an class object instead of int and that's why later a method is called as in i = args.somearg.somefunction().
In most tutorials that I've seen the type of argument is always int therefore could someone explain a bit about how to add argument of type class or any type other than int? Thanks!
From the documentation for the type argument:
By default, ArgumentParser objects read command-line arguments in as simple strings. However, quite often the command-line string should instead be interpreted as another type, like a float or int. The type keyword argument of add_argument() allows any necessary type-checking and type conversions to be performed.
[...]
type= can take any callable that takes a single string argument and returns the converted value
You can pass in any callable, provided it takes a single argument to be converted. int() is such a callable, but that doesn't mean you have to use that. If you want some other type or want to apply extra constraints to the allowed values, you can specify your own.
Some lines of code are worth a thousand words. Create a python file test.py with the following:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--G', type=float, default=1, nargs='?')
args = parser.parse_args()
print (args.G / 3)
Then, in a terminal:
python test.py
gives 0, while:
python test.py --G 1
gives 0.333333333. I think this is because argparse doesn't seem to cast default arguments to their proper type (so 1 remains an int in default=1, in spite of the type=float), but it casts the argument if it is given explicitly.
I think this is inconsistent and prone to errors. Is there a reason why argparse behaves this way?
I think the 1 in default=1 is evaluated when you call parser.add_argument, where as the non-default value you pass as argument is evaluated at runtime, and therefore can be converted to float by argparse. It's not how argparse behaves; it's how python behaves in general. Consider this
def foo(x=1):
# there is no way to tell the function body that you want 1.0
# except for explicitly conversion
# because 1 is passed by value, which has already been evaluated
# and determined to be an int
print (x/3)
HUGE EDIT: Ha, I understand your point now and I think it's reasonable. So I dig into the source code and looks what I found:
https://hg.python.org/cpython/file/3.5/Lib/argparse.py#l1984
So argparse DOES appear to make sure your default value is type compliant, so long as it's a string. Try:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--G', type=float, default='1', nargs='?')
args = parser.parse_args()
print (args.G / 3)
Not sure if that's a bug or a compromise...
The other answers are right - only string defaults are passed through the type function. But there seems to be some reluctance to accept that logic.
Maybe this example will help:
import argparse
def mytype(astring):
return '['+astring+']'
parser = argparse.ArgumentParser()
parser.add_argument('--foo', type=mytype, default=1)
parser.add_argument('--bar', type=mytype, default='bar')
print parser.parse_args([])
print mytype(1)
produces:
0923:~/mypy$ python stack35429336.py
Namespace(bar='[bar]', foo=1)
Traceback (most recent call last):
File "stack35429336.py", line 8, in <module>
print mytype(1)
File "stack35429336.py", line 3, in mytype
return '['+astring+']'
TypeError: cannot concatenate 'str' and 'int' objects
I define a type function - it takes a string input, and returns something - anything that I want. And raises an error if it can't return that. Here I just append some characters to the string.
When the default is a string, it gets modified. But when it is a number (not a string) it is inserted without change. In fact as written mytype raises an error if given a number.
The argparse type is often confused with the function type(a). The latter returns values like int,str,bool. Plus the most common examples are int and float. But in argparse float is used as a function
float(x) -> floating point number
Convert a string or number to a floating point number, if possible.
type=bool is a common error. Parsing boolean values with argparse. bool() does not convert the string 'False' to the boolean False.
In [50]: bool('False')
Out[50]: True
If argparse passed every default through the type function, it would be difficult to place values like None or False in the namespace. No builtin function converts a string to None.
The key point is that the type parameter is a function (callable), not a casting operation or target.
For further clarification - or confusion - explore the default and type with nargs=2 or action='append'.
The purpose of type is to sanitise and validate the arbitrary input you receive from the command line. It's a first line of defence against bogus input. There is no need to "defend" yourself against the default value, because that's the value you have decided on yourself and you are in power and have full control over your application. type does not mean that the resulting value after parse_args() must be that type; it means that any input should be that type/should be the result of that function.
What the default value is is completely independent of that and entirely up to you. It's completely conceivable to have a default value of False, and if a user inputs a number for this optional argument, then it'll be a number instead.
Upgrading a comment below to be included here: Python's philosophy includes "everyone is a responsible adult and Python won't overly question your decisions" (paraphrased). This very much applies here I think.
The fact that argparse does cast default strings could be explained by the fact that CLI input is always a string, and it's possible that you're chaining several argparsers, and the default value is set by a previous input. Anything that's not a string type however must have been explicitly chosen by you, so it won't be further mangled.
If the default value is a string, the parser parses the value as if it were a command-line argument. In particular, the parser applies any type conversion argument, if provided, before setting the attribute on the Namespace return value. Otherwise, the parser uses the value as is:
parser = argparse.ArgumentParser()
parser.add_argument('--length', default='10', type=int)
parser.add_argument('--width', default=10.5, type=int)
args = parser.parse_args()
Here, args.length will be int 10 and args.width will be float 10.5, just as the default value.
The example is from https://docs.python.org/3.8/library/argparse.html#default.
At first, I also have the same question. Then I notice this example, it explains the question pretty clearly.
This first link has the same question in the first section, but it is unanswered
(python argparse: parameter=value). And this second question is similar, but I can't seem to get it working for my particular case
( Using argparse to parse arguments of form "arg= val").
So my situation is this -- I am re-writing a Python wrapper which is used by many other scripts (I would prefer not to modify these other scripts). Currently, the Python wrapper is called with command line arguments of the form --key=value for a number of different arguments, but was parsed manually. I would like to parse them with argparse.
N.B. The argument names are unwieldy, so I am renaming using the dest option in add_argument.
parser = argparse.ArgumentParser(description='Wrappin Ronnie Reagan')
parser.add_argument("--veryLongArgName1", nargs=1, dest="arg1", required=True)
parser.add_argument("--veryLongArgName2", nargs=1, dest="arg2")
parser.add_argument("--veryLongArgName3", nargs=1, dest="arg3")
userOpts = vars(parser.parse_args())
Which, while apparently parsing the passed command lines correctly, displays this as the help:
usage: testing_argsparse.py [-h] --veryLongArgName1 ARG1
[--veryLongArgName2 ARG2]
[--veryLongArgName3 ARG3]
testing_argsparse.py: error: argument --veryLongArgName1 is required
But what I want is that all parameters are specified with the --key=value format, not --key value. i.e.
usage: testing_argsparse.py [-h] --veryLongArgName1=ARG1
[--veryLongArgName2=ARG2]
[--veryLongArgName3=ARG3]
testing_argsparse.py: error: argument --veryLongArgName1 is required
testing_argsparse.py --veryLongArgName1=foo
works. argparse module accepts both --veryLongArgName1=foo and --veryLongArgName1 foo formats.
What exact command line arguments are you trying to pass to argparse that's causing it to not work?
A little late but for anyone with a similar request as the OP you could use a custom HelpFormatter.
class ArgFormatter(argparse.HelpFormatter):
def _format_args(self, *args):
result = super(ArgFormatter, self)._format_args(*args)
return result and '%%%' + result
def _format_actions_usage(self, *args):
result = super(ArgFormatter, self)._format_actions_usage(*args)
return result and result.replace(' %%%', '=')
This can then be passed to ArgumentParser to give the wanted behavior.
parser = argparse.ArgumentParser(
description='Wrappin Ronnie Reagan',
formatter_class=ArgFormatter)
This intercepts the args (ARG1, ARG2, ...) and adds a custom prefix which is later replaced (along with the unwanted space) for an = symbol. The and in the return statements makes sure to only modify the result if it's non-empty.