We Keep Coding

Python argparse : How can I get Namespace objects for argument groups separately?

I have some command line arguments categorized in groups as follows:
cmdParser = argparse.ArgumentParser()
cmdParser.add_argument('mainArg')
groupOne = cmdParser.add_argument_group('group one')
groupOne.add_argument('-optA')
groupOne.add_argument('-optB')
groupTwo = cmdParser.add_argument_group('group two')
groupTwo.add_argument('-optC')
groupTwo.add_argument('-optD')
How can I parse the above, such that I end up with three different Namespace objects?
global_args - containing all the arguments not part of any group
groupOne_args - containing all the arguments in groupOne
groupTwo_args - containing all the arguments in groupTwo
Thank you!

you can do it in this way:
import argparse
parser = argparse.ArgumentParser()
group1 = parser.add_argument_group('group1')
group1.add_argument('--test1', help="test1")
group2 = parser.add_argument_group('group2')
group2.add_argument('--test2', help="test2")
args = parser.parse_args('--test1 one --test2 two'.split())
arg_groups={}
for group in parser._action_groups:
group_dict={a.dest:getattr(args,a.dest,None) for a in group._group_actions}
arg_groups[group.title]=argparse.Namespace(**group_dict)
This will give you the normal args, plus dictionary arg_groups containing namespaces for each of the added groups.
(Adapted from this answer)

Nothing in argparse is designed to do that.
For what it's worth, the parser starts off with two argument groups, one that displays as positionals and the other as optionals (I forget the exact titles). So in your example there will actually be 4 groups.
The parser only uses argument groups when formatting the help. For parsing all arguments are put in a master parser._actions list. And during parsing the parser only passes around one namespace object.
You could define separate parsers, with different sets of arguments, and call each with parse_known_args. That works better with optionals (flagged) arguments than with positionals. And it fragments your help.
I have explored in other SO questions a novel Namespace class that could nest values based on some sort of dotted dest (names like group1.optA, group2.optC, etc). I don't recall whether I had to customize the Action classes or not.
The basic point is that when saving a value to the namespace, a parser, or actually a Action (argument) object does:
setattr(namespace, dest, value)
That (and getattr/hasattr) is all that the parser expects of the namespace. The default Namespace class is simple, little more than a plain object subclass. But it could be more elaborate.

I was looking for a solution for this for a very long time,
And I think I finally got it.
So I will just put it here...
from argparse import ArgumentParser
def _parse_args():
parser = ArgumentParser()
parser.add_argument('-1', '--flag-1', action='store_true', default=False)
parser.add_argument('-2', '--flag-2', action='store_true', default=False)
parser.add_argument('-3', '--flag-3', action='store_true', default=False)
args, unknown = parser.parse_known_args()
print(f"args : {args}")
print(f"unknown : {unknown}")
hidden = ArgumentParser(add_help=False)
hidden.add_argument('-d', '--debug', action='store_true', default=False)
hidden_args = hidden.parse_args(unknown)
print(f"hidden_args : {hidden_args}")
if __name__ == "__main__":
_parse_args()
as a result:
show help:
ubuntu → playAround $ ./test.py -h
usage: test.py [-h] [-1] [-2] [-3]
optional arguments:
-h, --help show this help message and exit
-1, --flag-1
-2, --flag-2
-3, --flag-3
With debug flag:
ubuntu → playAround $ ./test.py -d
args : Namespace(flag_1=False, flag_2=False, flag_3=False)
unknown : ['-d']
hidden_args : Namespace(debug=True)
with flags 1 and 2:
ubuntu → playAround $ ./test.py -12
args : Namespace(flag_1=True, flag_2=True, flag_3=False)
unknown : []
hidden_args : Namespace(debug=False)
with flags 1 and 2 and debug:
ubuntu → playAround $ ./test.py -12 -d
args : Namespace(flag_1=True, flag_2=True, flag_3=False)
unknown : ['-d']
hidden_args : Namespace(debug=True)
The only thing you can't do with this approch is to pass the debug short flag along side to the other short flags:
ubuntu → playAround $ ./test.py -12d
usage: test.py [-h] [-1] [-2] [-3]
test.py: error: argument -2/--flag-2: ignored explicit argument 'd'

Here's a simple method that calls the parse_known_args() method after each group is defined to get the names for those arguments separately.
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('mainArg')
global_names = set(vars(parser.parse_known_args()[0]).keys())
group1 = parser.add_argument_group('group 1')
group1.add_argument('-optA')
group1.add_argument('-optB')
group1_names = set(vars(parser.parse_known_args()[0]).keys()) - global_names
group2 = parser.add_argument_group('group 2')
group2.add_argument('-optC')
group2.add_argument('-optD')
group2_names = set(vars(parser.parse_known_args()[0]).keys()) - global_names - group1_names
args = parser.parse_args()
global_args = argparse.Namespace(**dict((k, v) for k, v in vars(args).items() if k in global_names))
group1_args = argparse.Namespace(**dict((k, v) for k, v in vars(args).items() if k in group1_names))
group2_args = argparse.Namespace(**dict((k, v) for k, v in vars(args).items() if k in group2_names))
print(global_args)
print(group1_args)
print(group2_args)
E.g. python args.py hi -optA fooA -optB fooB -optC fooC -optD fooD will output:
Namespace(mainArg='hi')
Namespace(optA='fooA', optB='fooB')
Namespace(optC='fooC', optD='fooD')

Argparse: how to handle variable number of arguments (nargs='*')

I thought that nargs='*' was enough to handle a variable number of arguments. Apparently it's not, and I don't understand the cause of this error.
The code:
p = argparse.ArgumentParser()
p.add_argument('pos')
p.add_argument('foo')
p.add_argument('--spam', default=24, type=int, dest='spam')
p.add_argument('vars', nargs='*')
p.parse_args('1 2 --spam 8 8 9'.split())
I think the resulting namespace should be Namespace(pos='1', foo='2', spam='8', vars=['8', '9']). Instead, argparse gives this error:
usage: prog.py [-h] [--spam SPAM] pos foo [vars [vars ...]]
error: unrecognized arguments: 9 8
Basically, argparse doesn't know where to put those additional arguments... Why is that?

For anyone who doesn't know what is nargs:
nargs stands for Number Of Arguments
3: 3 values, can be any number you want
?: a single value, which can be optional
*: a flexible number of values, which will be gathered into a list
+: like *, but requiring at least one value
argparse.REMAINDER: all the values that are remaining in the command
line
Example:
Python
import argparse
my_parser = argparse.ArgumentParser()
my_parser.add_argument('--input', action='store', type=int, nargs=3)
args = my_parser.parse_args()
print(args.input)
Console
$ python nargs_example.py --input 42
usage: nargs_example.py [-h] [--input INPUT INPUT INPUT]
nargs_example.py: error: argument --input: expected 3 arguments
$ python nargs_example.py --input 42 42 42
[42, 42, 42]
See more

The relevant Python bug is Issue 15112.
argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional
When argparse parses ['1', '2', '--spam', '8', '8', '9'] it first tries to match ['1','2'] with as many of the positional arguments as possible. With your arguments the pattern matching string is AAA*: 1 argument each for pos and foo, and zero arguments for vars (remember * means ZERO_OR_MORE).
['--spam','8'] are handled by your --spam argument. Since vars has already been set to [], there is nothing left to handle ['8','9'].
The programming change to argparse checks for the case where 0 argument strings is satisfying the pattern, but there are still optionals to be parsed. It then defers the handling of that * argument.
You might be able to get around this by first parsing the input with parse_known_args, and then handling the remainder with another call to parse_args.
To have complete freedom in interspersing optionals among positionals, in issue 14191, I propose using parse_known_args with just the optionals, followed by a parse_args that only knows about the positionals. The parse_intermixed_args function that I posted there could be implemented in an ArgumentParser subclass, without modifying the argparse.py code itself.
Here's a way of handling subparsers. I've taken the parse_known_intermixed_args function, simplified it for presentation sake, and then made it the parse_known_args function of a Parser subclass. I had to take an extra step to avoid recursion.
Finally I changed the _parser_class of the subparsers Action, so each subparser uses this alternative parse_known_args. An alternative would be to subclass _SubParsersAction, possibly modifying its __call__.
from argparse import ArgumentParser
def parse_known_intermixed_args(self, args=None, namespace=None):
# self - argparse parser
# simplified from http://bugs.python.org/file30204/test_intermixed.py
parsefn = super(SubParser, self).parse_known_args # avoid recursion
positionals = self._get_positional_actions()
for action in positionals:
# deactivate positionals
action.save_nargs = action.nargs
action.nargs = 0
namespace, remaining_args = parsefn(args, namespace)
for action in positionals:
# remove the empty positional values from namespace
if hasattr(namespace, action.dest):
delattr(namespace, action.dest)
for action in positionals:
action.nargs = action.save_nargs
# parse positionals
namespace, extras = parsefn(remaining_args, namespace)
return namespace, extras
class SubParser(ArgumentParser):
parse_known_args = parse_known_intermixed_args
parser = ArgumentParser()
parser.add_argument('foo')
sp = parser.add_subparsers(dest='cmd')
sp._parser_class = SubParser # use different parser class for subparsers
spp1 = sp.add_parser('cmd1')
spp1.add_argument('-x')
spp1.add_argument('bar')
spp1.add_argument('vars',nargs='*')
print parser.parse_args('foo cmd1 bar -x one 8 9'.split())
# Namespace(bar='bar', cmd='cmd1', foo='foo', vars=['8', '9'], x='one')

Simple solution: Specify the --spam flag before specifying pos and foo:
p = argparse.ArgumentParser()
p.add_argument('pos')
p.add_argument('foo')
p.add_argument('--spam', default=24, type=int, dest='spam')
p.add_argument('vars', nargs='*')
p.parse_args('--spam 8 1 2 8 9'.split())
The same works if you place the --spam flag after specifying your variable arguments.
p = argparse.ArgumentParser()
p.add_argument('pos')
p.add_argument('foo')
p.add_argument('--spam', default=24, type=int, dest='spam')
p.add_argument('vars', nargs='*')
p.parse_args('1 2 8 9 --spam 8'.split())
EDIT: For what it's worth, it seems that changing the * to a + will also fix the error.
p = argparse.ArgumentParser()
p.add_argument('pos')
p.add_argument('foo')
p.add_argument('--spam', default=24, type=int, dest='spam')
p.add_argument('vars', nargs='+')
p.parse_args('1 2 --spam 8 8 9'.split())

If you expect to have at least one optional argument then p.add_argument('vars', nargs='+') will work in your specific case

Python,argparse: how to have nargs=2 with type=str and type=int

I spent some times on the argparse documentation, but I'm still struggling with this module for one option in my program:
parser.add_argument("-r", "--rmsd", dest="rmsd", nargs=2,
help="extract the poses that are close from a ref according RMSD",
metavar=("ref","rmsd"))
I'd like to the first argument to be a string (type str) and mandatory, while the second argument should have type int, and if no value is given have a default one (let's say default=50). I know how to do that when there is only one argument expected, but I have no idea how to proceed when nargs=2... Is that even possible?

You can do the following. The required keyword sets the field mandatory and the default=50 sets the default value of the option to 50 if not specified:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-s", "--string", type=str, required=True)
parser.add_argument("-i", "--integer", type=int, default=50)
args = parser.parse_args()
print args.string
print args.integer
Output:
$ python arg_parser.py -s test_string
test_string
50
$ python arg_parser.py -s test_string -i 100
test_string
100
$ python arg_parser.py -i 100
usage: arg_parser.py [-h] -s STRING [-i INTEGER]
arg_parser.py: error: argument -s/--string is required

I tend to agree with Mike's solution, but here's another way. It's not ideal, since the usage/help string tells the user to use 1 or more arguments.
import argparse
def string_integer(int_default):
"""Action for argparse that allows a mandatory and optional
argument, a string and integer, with a default for the integer.
This factory function returns an Action subclass that is
configured with the integer default.
"""
class StringInteger(argparse.Action):
"""Action to assign a string and optional integer"""
def __call__(self, parser, namespace, values, option_string=None):
message = ''
if len(values) not in [1, 2]:
message = 'argument "{}" requires 1 or 2 arguments'.format(
self.dest)
if len(values) == 2:
try:
values[1] = int(values[1])
except ValueError:
message = ('second argument to "{}" requires '
'an integer'.format(self.dest))
else:
values.append(int_default)
if message:
raise argparse.ArgumentError(self, message)
setattr(namespace, self.dest, values)
return StringInteger
And with that, you get:
>>> import argparse
>>> parser = argparse.ArgumentParser(description="")
parser.add_argument('-r', '--rmsd', dest='rmsd', nargs='+',
... action=string_integer(50),
... help="extract the poses that are close from a ref "
... "according RMSD")
>>> parser.parse_args('-r reference'.split())
Namespace(rmsd=['reference', 50])
>>> parser.parse_args('-r reference 30'.split())
Namespace(rmsd=['reference', 30])
>>> parser.parse_args('-r reference 30 3'.split())
usage: [-h] [-r RMSD [RMSD ...]]
: error: argument -r/--rmsd: argument "rmsd" requires 1 or 2 arguments
>>> parser.parse_args('-r reference 30.3'.split())
usage: [-h] [-r RMSD [RMSD ...]]
: error: argument -r/--rmsd: second argument to "rmsd" requires an integer

Sorry for jumping in way late. I'd use a function for type to call.
def two_args_str_int(x):
try:
return int(x)
except:
return x
parser.add_argument("-r", "--rmsd", dest="rmsd", nargs=2, type=two_args_str_int
help="extract the poses that are close from a ref according RMSD",
metavar=("ref","rmsd"))

I would recommend using two arguments:
import argparse
parser = argparse.ArgumentParser(description='Example with to arguments.')
parser.add_argument('-r', '--ref', dest='reference', required=True,
help='be helpful')
parser.add_argument('-m', '--rmsd', type=int, dest='reference_msd',
default=50, help='be helpful')
args = parser.parse_args()
print args.reference
print args.reference_msd

I had a similar problem, but "use two arguments" approach didn't work for me because I need a list of pairs: parser.add_argument('--replace', nargs=2, action='append') and if I use separate arguments then I would have to validate lengths of lists etc.
Here is what I did:
Use tuple for metavar to properly show help: tuple=('OLD', 'NEW') results in the help string being displayed as --replace OLD NEW. It is documented but I could not find it until tried different options.
Use custom validation: after parse_args, validate the resulting list's items and call parser.error() if something is wrong. That's because they have different data types.

Python: Using run time inputs to change key values in a function

import sys
def fun( a=7, b=1 ):
print a, b
fun()
fun( a=5 )
fun( sys.argv[1:] )
The first fun() prints out '7,1', the second prints out '5,1', but the third prints out '['a=8', 'b=6'] 1]'. I would like to be able to call my program with
python my_program.py a=5 b=6
to change the value of the printed a and b. But this doesn't work since sys.argv[1] is a list of strings.
Is there any way to convert the the string list to a form the function can understand?

Use ** for kwarg unpacking:
d = {}
for a in sys.argv[1:]:
k, v = a.split('=')
d[k] = int(v)
func(**d)
Another way, using the csv module:
import csv
func(**{k: int(v) for k, v in csv.reader(sys.argv[1:], delimiter='=')})
As #MartijnPieters noted you may use the argparse module
parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('-a', type=int)
parser.add_argument('-b', type=int)
args = parser.parse_args()
fun(**dict(args._get_kwargs()))
but then you have to adjust the way you input your arguments

Use * to unpack a list into arguments:
fun(*sys.argv[1:])
Note that this will result in string arguments; sys.argv command line entries are always strings.
You may want to look into the argparse module to handle command line arguments for you, including automatic conversion:
import argparse
parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('-a', type=int, default=7)
parser.add_argument('-b', type=int, default=1)
args = parser.parse_args()
func(args.a, args.b)
Then use:
$ python my_program.py -a=5 -b=6
The added advantage here is that you get proper help feedback, and you follow standard command-line conventions:
$ python my_program.py --help
usage: [-h] [-a A] [-b B]
Process some integers.
optional arguments:
-h, --help show this help message and exit
-a A
-b B

Python argparse type and choice restrictions with nargs > 1

The title pretty much says it all. If I have nargs greater than 1, is there any way I can set restrictions (such as choice/type) on the individual args parsed?
This is some example code:
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--credits', nargs=2,
help='number of credits required for a subject')
For the -c argument I need to specify a subject and how many credits are required. The subject should be limited to a predefined list of subjects, and the number of credits required should be a float.
I could probably do this with a subparser, but as it is this is already part of a sub-command so I don't really want things to get any more complicated.

You can validate it with a custom action:
import argparse
import collections
class ValidateCredits(argparse.Action):
def __call__(self, parser, args, values, option_string=None):
# print '{n} {v} {o}'.format(n=args, v=values, o=option_string)
valid_subjects = ('foo', 'bar')
subject, credits = values
if subject not in valid_subjects:
raise ValueError('invalid subject {s!r}'.format(s=subject))
credits = float(credits)
Credits = collections.namedtuple('Credits', 'subject required')
setattr(args, self.dest, Credits(subject, credits))
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--credits', nargs=2, action=ValidateCredits,
help='subject followed by number of credits required',
metavar=('SUBJECT', 'CREDITS')
)
args = parser.parse_args()
print(args)
print(args.credits.subject)
print(args.credits.required)
For example,
% test.py -c foo 2
Namespace(credits=Credits(subject='foo', required=2.0))
foo
2.0
% test.py -c baz 2
ValueError: invalid subject 'baz'
% test.py -c foo bar
ValueError: could not convert string to float: bar

Side note, because this question turns up when searching for "argparse nargs choices":
A custom action is only needed if the nargs arguments require a heterogenous type validation, i.e., the argument at index 0 should be a different type (here: limited type of subjects) than the argument at index 1 (here: float) etc.
If a homogenous type validation is desired, it is sufficient to combine nargs with choices directly. For instance:
parser.add_argument(
"--list-of-xs-or-ys",
nargs="*",
choices=["x", "y"],
)
would allow anything like --list-of-xs-or-ys x y x y, but would complain if the user specifies anything else than x or y.

A caller of a Action class only catch a ArgumentError.
https://github.com/python/cpython/blob/3.8/Lib/argparse.py#L1805
For expecting to catch an exception by a caller, You should raise as following in your custom action.
raise ArgumentError(self, 'invalid subject {s!r}'.format(s=subject))

i suppose you could try this - in add_argument(), you can specify a limited set of inputs with choice='xyz' or choice=[this, that]
as described here:
http://docs.python.org/library/argparse.html#choices
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--credits', choice='abcde', nargs=2,
help='number of credits required for a subject')