Related
I am trying to add command line options to program. The program should accept both a list of numbers, and parse options. For instance a call to the program could look like:
python3 ./parsetest.py 4 3 v=100
The 4 and the 3 will be processed in one way whereas the v=100 command will set the internal variable v to 100. I found the getopt library is supposed to have this functionality: http://stackabuse.com/command-line-arguments-in-python/ Is there a better/simpler way to do this?
When I try to parse the input like the above, I believe getopt should place arguments which trigger its keywords into a special list, and those which do not trigger are placed elsewhere. However, when I try to use getopt following the example above, no matter what I do, the v=100 option does not appear to trigger the special arguments list. When passed to getopt, I expected the 'v=100' string to be split up into argument v and value 100. For instance, the print(arguments,values) command below results in: [] ['4', '3', 'v=100']. There are values, but there are no arguments.
Here is my code:
import sys,getopt
v = 1e100
fullCmdArguments = sys.argv
argumentList = fullCmdArguments[1:]
unixOptions = "v:"
gnuOptions = ["v="]
print(argumentList)
try:
arguments, values = getopt.getopt(argumentList, unixOptions, gnuOptions)
except getopt.error as err:
# output error, and return with an error code
print (str(err))
sys.exit(2)
print(arguments, values)
for currentArgument, currentValue in arguments:
print (currentArgument, currentValue)
if currentArgument in ("-v", "v"):
v = currentValue
else:
#Its an integer then, do something with it
print()
If you can use argparse the following will work:
Example
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('numbers', type=int, nargs='+')
parser.add_argument('-v', type=int, default=-1)
args = parser.parse_args()
print(args.numbers)
print(args.v)
# due to the default set above, `args.v` will always have an int value
v = args.v
Using the nargs keyword argument you can set it to '+' requiring one or more numbers. The v with a hyphen is by default optional. If this is not wanted, you can add the required=True option.
Usage
python my_file.py 1 2 3 5 -v=12
[1, 2, 3, 5]
12
python my_file.py 1 2 3 5
[1, 2, 3, 5]
-1
Further reading
For more information about the argparse module you can have a look at:
The official tutorial: https://docs.python.org/3/howto/argparse.html
The module doc: https://docs.python.org/3/library/argparse.html#module-argparse
I'm using argparse to receive inputs from the command line to run my script.
My current input string looks like this:
path> python <\filename\\> -t T1 T2 T3 -f F1 F2
Is there a parameter in argparse such that instead of separating inputs by space, I can separate them by commas?
In other words:
path> python <\filename\\> -t T1,T2,T3 -f F1,F2
There is no such feature in argparse.
Alternatives:
post-process the args namespace and split/parse the values manually
define a custom action and split/parse the values manually
define a custom type and split/parse the values manually
subclass ArgumentParser and customise ArgumentParser.convert_arg_line_to_args
There are some useful answers here already, but I wanted a bit more: splitting on commas, validating values with choices, and getting useful error messages, so below I offer a solution.
Simple Version
As a first pass, we can just pass in an appropriate function to the type parameter:
>>> import argparse
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
But this doesn't work with choices because it checks if the whole list is in choices and not each value:
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=lambda arg: arg.split(','), choices=('a', 'b', 'c'))
>>> parser.parse_args(['--foo', 'a,b,c'])
usage: cmd [-h] [--foo {a,b,c}]
cmd: error: argument --foo: invalid choice: ['a', 'b', 'c'] (choose from 'a', 'b', 'c')
Setting nargs to something like * or + checks each value, but only for space-separated arguments (e.g., --foo a b) and not the comma-separated ones. It seems like there is no supported way to check if each value is in the choices if we produce the list ourselves. Therefore we need to raise errors ourselves via the type parameter (as Shiplu Mokaddim partially implemented). Creating a custom Action class sounds promising as actions have access to the choices, but the action happens after the type function applies and values are checked, so we still couldn't use the choices parameter on add_argument() for this purpose.
Better Version
Here is a solution using a custom type function. We this function to take a list of valid choices, but since the function for type conversion can only take an argument string, we need to wrap it in a class (and define the special __call__() method) or a function closure. This solution uses the latter.
>>> def csvtype(choices):
... """Return a function that splits and checks comma-separated values."""
... def splitarg(arg):
... values = arg.split(',')
... for value in values:
... if value not in choices:
... raise argparse.ArgumentTypeError(
... 'invalid choice: {!r} (choose from {})'
... .format(value, ', '.join(map(repr, choices))))
... return values
... return splitarg
>>> parser = argparse.ArgumentParser(prog='cmd')
>>> parser.add_argument('--foo', type=csvtype(('a', 'b', 'c')))
>>> parser.parse_args(['--foo', 'a,b,c'])
Namespace(foo=['a', 'b', 'c'])
>>> parser.parse_args(['--foo', 'a,b,d'])
usage: cmd [-h] [-f F]
cmd: error: argument -f: invalid choice: 'd' (choose from 'a', 'b', 'c')
Notice that we get an appropriate error as well. For this, be sure to use argparse.ArgumentTypeError and not argparse.ArgumentError inside the function.
Other Options
User wim suggested some other options not discussed above. I don't find those attractive for the following reasons:
Post-processing the argument after parsing means you have to do more work to get the error messages to be consistent with those from argparse. Just raising argparse.ArgumentError will lead to a stacktrace. Also, argparse catches errors raised during parsing and alters them to specify the option that was used, which you'd otherwise need to do manually.
Subclassing ArgumentParser is more work, and convert_arg_line_to_args() is for reading arguments from a file, not the command line.
You can use module shlex to extract the parameters, then replace commas with spaces, and pass the results to argparse for further processing:
comma_args = shlex.split("-t T1,T2,T3 -f F1,F2")
# ['-t', 'T1,T2,T3', '-f', 'F1,F2']
args = [x.replace(","," ") for x in comma_args]
# ['-t', 'T1 T2 T3', '-f', 'F1 F2']
parse_args(args)
To ensure compatibility with choices, I use two parsers:
#!/usr/bin/env python3
import argparse
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--arg", type=str, metavar="ARG[,ARG,...]")
# Pass the usage to the subparser, remove the usage: prefix.
subparser_usage = parser.format_usage().replace("usage: ", "", 1)
subparser = argparse.ArgumentParser(usage=subparser_usage)
subparser.add_argument("arg", nargs="*", type=int, choices=[1, 2, 3])
args = parser.parse_args()
if args.arg:
subargs = subparser.parse_args(args.arg.split(","))
args.arg = subargs.arg
print(args)
if __name__ == "__main__":
main()
This fulfills all possible requirements:
easy and understandable
involves only argparse
choices are ensured with a correct usage and help message
this allows to have also positional arguments (thanks to the comma separated list)
See the following examples:
$ ./example --arg 1,2,3
Namespace(arg=[1, 2, 3])
$ ./example --arg 1,2,4
usage: example [-h] [--arg ARG[,ARG,...]]
example: error: argument arg: invalid choice: 4 (choose from 1, 2, 3)
$ ./example --arg 1
Namespace(arg=[1])
Here this comma-separated input is actually a different type. All you have to is to define the type.
Here I define a custom type that does it.
class DelimiterSeperatedInput:
def __init__(self, item_type, separator=','):
self.item_type = item_type
self.separator = separator
def __call__(self, value):
values = []
try:
for val in value.split(self.separator):
typed_value = self.item_type(val)
values.append(typed_value)
except Exception:
raise ArgumentError("%s is not a valid argument" % value)
return values
parser.add_argument('-t', type=DelimiterSeperatedInput(str),
help='comma separated string values')
parser.add_argument('-f', type=DelimiterSeperatedInput(float, ":"),
help="colon separated floats')
This code may not work as-is, you might have to fix. But it was to give an idea.
Note: I could reduce the __call__ function body by using list, map etc. But then it wouldn't be very readable. Once you get the idea, you can do kinds of stuff with it.
If you're okay with space not comma separated, then it's builtin to argparse:
In [1]: from argparse import ArgumentParser
In [2]: parser = ArgumentParser()
In [3]: parser.add_argument('-a', nargs='+')
Out[3]: _StoreAction(option_strings=['-a'], dest='a', nargs='+', const=None,
default=None, type=None, choices=None, help=None, metavar=None)
In [4]: parser.parse_args(['-a', 'foo', 'bar'])
Out[4]: Namespace(a=['foo', 'bar'])
I spent some times on the argparse documentation, but I'm still struggling with this module for one option in my program:
parser.add_argument("-r", "--rmsd", dest="rmsd", nargs=2,
help="extract the poses that are close from a ref according RMSD",
metavar=("ref","rmsd"))
I'd like to the first argument to be a string (type str) and mandatory, while the second argument should have type int, and if no value is given have a default one (let's say default=50). I know how to do that when there is only one argument expected, but I have no idea how to proceed when nargs=2... Is that even possible?
You can do the following. The required keyword sets the field mandatory and the default=50 sets the default value of the option to 50 if not specified:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-s", "--string", type=str, required=True)
parser.add_argument("-i", "--integer", type=int, default=50)
args = parser.parse_args()
print args.string
print args.integer
Output:
$ python arg_parser.py -s test_string
test_string
50
$ python arg_parser.py -s test_string -i 100
test_string
100
$ python arg_parser.py -i 100
usage: arg_parser.py [-h] -s STRING [-i INTEGER]
arg_parser.py: error: argument -s/--string is required
I tend to agree with Mike's solution, but here's another way. It's not ideal, since the usage/help string tells the user to use 1 or more arguments.
import argparse
def string_integer(int_default):
"""Action for argparse that allows a mandatory and optional
argument, a string and integer, with a default for the integer.
This factory function returns an Action subclass that is
configured with the integer default.
"""
class StringInteger(argparse.Action):
"""Action to assign a string and optional integer"""
def __call__(self, parser, namespace, values, option_string=None):
message = ''
if len(values) not in [1, 2]:
message = 'argument "{}" requires 1 or 2 arguments'.format(
self.dest)
if len(values) == 2:
try:
values[1] = int(values[1])
except ValueError:
message = ('second argument to "{}" requires '
'an integer'.format(self.dest))
else:
values.append(int_default)
if message:
raise argparse.ArgumentError(self, message)
setattr(namespace, self.dest, values)
return StringInteger
And with that, you get:
>>> import argparse
>>> parser = argparse.ArgumentParser(description="")
parser.add_argument('-r', '--rmsd', dest='rmsd', nargs='+',
... action=string_integer(50),
... help="extract the poses that are close from a ref "
... "according RMSD")
>>> parser.parse_args('-r reference'.split())
Namespace(rmsd=['reference', 50])
>>> parser.parse_args('-r reference 30'.split())
Namespace(rmsd=['reference', 30])
>>> parser.parse_args('-r reference 30 3'.split())
usage: [-h] [-r RMSD [RMSD ...]]
: error: argument -r/--rmsd: argument "rmsd" requires 1 or 2 arguments
>>> parser.parse_args('-r reference 30.3'.split())
usage: [-h] [-r RMSD [RMSD ...]]
: error: argument -r/--rmsd: second argument to "rmsd" requires an integer
Sorry for jumping in way late. I'd use a function for type to call.
def two_args_str_int(x):
try:
return int(x)
except:
return x
parser.add_argument("-r", "--rmsd", dest="rmsd", nargs=2, type=two_args_str_int
help="extract the poses that are close from a ref according RMSD",
metavar=("ref","rmsd"))
I would recommend using two arguments:
import argparse
parser = argparse.ArgumentParser(description='Example with to arguments.')
parser.add_argument('-r', '--ref', dest='reference', required=True,
help='be helpful')
parser.add_argument('-m', '--rmsd', type=int, dest='reference_msd',
default=50, help='be helpful')
args = parser.parse_args()
print args.reference
print args.reference_msd
I had a similar problem, but "use two arguments" approach didn't work for me because I need a list of pairs: parser.add_argument('--replace', nargs=2, action='append') and if I use separate arguments then I would have to validate lengths of lists etc.
Here is what I did:
Use tuple for metavar to properly show help: tuple=('OLD', 'NEW') results in the help string being displayed as --replace OLD NEW. It is documented but I could not find it until tried different options.
Use custom validation: after parse_args, validate the resulting list's items and call parser.error() if something is wrong. That's because they have different data types.
I'm writing a utility for running bash commands that essentially takes as input a string and a list optional argument and uses the optional arguments to interpolate string.
I'd like it to work like this:
interpolate.py Hello {user_arg} my name is {computer_arg} %% --user_arg=john --computer_arg=hal
The %% is a separator, it separates the string to be interpolated from the arguments. What follows is the arguments used to interpolate the string. In this case the user has chosen user_arg and computer_arg as arguments. The program can't know in advance which argument names the user will choose.
My problem is how to parse the arguments? I can trivially split the input arguments on the separator but I can't figure out how to get optparse to just give the list of optional args as a dictionary, without specifying them in advance. Does anyone know how to do this without writing a lot of regex?
Well, if you use '--' to separate options from arguments instead of %%, optparse/argparse will just give you the arguments as a plain list (treating them as positional arguments instead of switched). After that it's not 'a lot of' regex, it's just a mere split:
for argument in args:
if not argument.startswith("--"):
# decide what to do in this case...
continue
arg_name, arg_value = argument.split("=", 1)
arg_name = arg_name[2:]
# use the argument any way you like
With argparse you could use the parse_known_args method to consume predefined arguments and any additional arguments. For example, using the following script
import sys
import argparse
def main(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument('string', type=str, nargs='*',
help="""String to process. Optionally with interpolation
(explain this here...)""")
args, opt_args = parser.parse_known_args(argv)
print args
print opt_args
return 0
if __name__=='__main__':
sys.exit(main(sys.argv[1:]))
and calling with
python script.py Hello, my name is {name} --name=chris
yields the following output:
Namespace(string=['Hello,' 'my', 'name', 'is', '{name}'])
['--name=chris']
All that is left to do is to loop through the args namespace looking for strings of the form {...} and replacing them with the corresponding element in opt_args, if present. (I'm not sure if argparse can do argument interpolation automatically, the above example is the only immediate solution which comes to mind).
For something like this, you really don't need optparse or argparse - the benefit of such libraries are of little use in this circumstance (things like lone -v type arguments, checking for invalid options, value validation and so on)
def partition_list(lst, sep):
"""Slices a list in two, cutting on index matching "sep"
>>> partition_list(['a', 'b', 'c'], sep='b')
(['a'], ['c'])
"""
if sep in lst:
idx = lst.index(sep)
return (lst[:idx], lst[idx+1:])
else:
return (lst[:], )
def args_to_dict(args):
"""Crudely parses "--blah=123" type arguments into dict like
{'blah': '123'}
"""
ret = {}
for a in args:
key, _, value = a.partition("=")
key = key.replace("--", "", 1)
ret[key] = value
return ret
if __name__ == '__main__':
import sys
# Get stuff before/after the "%%" separator
string, args = partition_list(sys.argv[1:], "%%")
# Join input string
string_joined = " ".join(string)
# Parse --args=stuff
d = args_to_dict(args)
# Do string-interpolation
print string_joined.format(**d)
I'm having a small issue with argparse. I have an option xlim which is the xrange of a plot. I want to be able to pass numbers like -2e-5. However this does not work - argparse interprets this is a positional argument. If I do -0.00002 it works: argparse reads it as a negative number. Is it possible to have able to read in -2e-3?
The code is below, and an example of how I would run it is:
./blaa.py --xlim -2.e-3 1e4
If I do the following it works:
./blaa.py --xlim -0.002 1e4
The code:
parser.add_argument('--xlim', nargs = 2,
help = 'X axis limits',
action = 'store', type = float,
default = [-1.e-3, 1.e-3])
Whilst I can get it to work this way I would really rather be able to use scientific notation. Anyone have any ideas?
Cheers
One workaround I've found is to quote the value, but adding a space. That is,
./blaa.py --xlim " -2.e-3" 1e4
This way argparse won't think -2.e-3 is an option name because the first character is not a hyphen-dash, but it will still be converted properly to a float because float(string) ignores spaces on the left.
As already pointed out by the comments, the problem is that a - prefix is parsed as an option instead of as an argument. One way to workaround this is change the prefix used for options with prefix_chars argument:
#!/usr/bin/python
import argparse
parser = argparse.ArgumentParser(prefix_chars='#')
parser.add_argument('##xlim', nargs = 2,
help = 'X axis limits',
action = 'store', type = float,
default = [-1.e-3, 1.e-3])
print parser.parse_args()
Example output:
$ ./blaa.py ##xlim -2.e-3 1e4
Namespace(xlim=[-0.002, 10000.0])
Edit: Alternatively, you can keep using - as separator, pass xlim as a single value and use a function in type to implement your own parsing:
#!/usr/bin/python
import argparse
def two_floats(value):
values = value.split()
if len(values) != 2:
raise argparse.ArgumentError
values = map(float, values)
return values
parser = argparse.ArgumentParser()
parser.add_argument('--xlim',
help = 'X axis limits',
action = 'store', type=two_floats,
default = [-1.e-3, 1.e-3])
print parser.parse_args()
Example output:
$ ./blaa.py --xlim "-2e-3 1e4"
Namespace(xlim=[-0.002, 10000.0])
If you specify the value for your option with an equals sign, argparse will not treat it as a separate option, even if it starts with -:
./blaa.py --xlim='-0.002 1e4'
# As opposed to --xlim '-0.002 1e4'
And if the value does not have spaces in it (or other special characters given your shell), you can drop the quotes:
./blaa.py --xlim=-0.002
See: https://www.gnu.org/software/guile/manual/html_node/Command-Line-Format.html
With this, there is no need to write your own type= parser or redefine the prefix character from - to # as the accepted answer suggests.
Here is the code that I use. (It is similar to jeremiahbuddha's but it answers the question more directly since it deals with negative numbers.)
Put this before calling argparse.ArgumentParser()
for i, arg in enumerate(sys.argv):
if (arg[0] == '-') and arg[1].isdigit(): sys.argv[i] = ' ' + arg
Another workaround is to pass in the argument using '=' symbol in addition to quoting the argument - i.e., --xlim="-2.3e14"
If you are up to modifying argparse.py itself, you could change the negative number matcher to handle scientific notation:
In class _ActionsContainer.__init__()
self._negative_number_matcher = _re.compile(r'^-(\d+\.?|\d*\.\d+)([eE][+\-]?\d+)?$')
Or after creating the parser, you could set parser._negative_number_matcher to this value. This approach might have problems if you are creating groups or subparsers, but should work with a simple parser.
Inspired by andrewfn's approach, I created a separate helper function to do the sys.argv fiddling:
def _tweak_neg_scinot():
import re
import sys
p = re.compile('-\\d*\\.?\\d*e', re.I)
sys.argv = [' ' + a if p.match(a) else a for a in sys.argv]
The regex looks for:
- : a negative sign
\\d* : zero or more digits (for oddly formatted values like -.5e-2 or -4354.5e-6)
\\.? : an optional period (e.g., -2e-5 is reasonable)
\\d* : another set of zero or more digits (for things like -2e-5 and -7.e-3)
e : to match the exponent marker
re.I makes it match both -2e-5 and -2E-5. Using p.match means that it only searches from the start of each string.