We Keep Coding

Disable argparse arguments from being overwritten

I have a legacy Python application which uses some options in its CLI, using argparse, like:
parser = argparse.ArgumentParser()
parser.add_argument('-f', default='foo')
Now I need to remove this option, since now its value cannot be overwritten by users but it has to assume its default value (say 'foo' in the example). Is there a way to keep the option but prevent it to show up and be overwritten by users (so that I can keep the rest of the code as it is)?

It's not entirely clear what you can do, not with the parser. Can you edit the setup? Or just modify results of parsing?
If you can edit the setup, you could replace the add_argument line with a
parser.setdefaults(f='foo')
https://docs.python.org/3/library/argparse.html#parser-defaults
The -f won't appear in the usage or help, but it will appear in the args
Or you could leave it in, but suppress the help display
parser.add_argument('-f', default='foo', help=argparse.SUPPRESS)
https://docs.python.org/3/library/argparse.html#help
Setting the value after parsing is also fine.

Yes you can do that. After the parser is parsed (args = parser.parse_args()) it is a NameSpace so you can do this:
parser = argparse.ArgumentParser()
args = parser.parse_args()
args.foo = 'foo value'
print(args)
>>> Namespace(OTHER_OPTIONS, foo='foo value')
I assumed that you wanted to add test to your parser, so your original code will still work, but you do not want it as an option for the user.

I think it doesn't make sense for the argparse module to provide this as a standard option, but there are several easy ways to achieve what you want.
The most obvious way is to just overwrite the value after having called parse_args() (as already mentioned in comments and in another answer):
args.f = 'foo'
However, the user may not be aware that the option is not supported anymore and that the application is now assuming the value "foo". Depending on the use case, it might be better to warn the user about this. The argparse module has several options to do this.
Another possibility is to use an Action class to do a little magic. For example, you could print a warning if the user provided an option that is not supported anymore, or even use the built-in error handling.
import argparse
class FooAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if values != 'foo':
print('Warning: option `-f` has been removed, assuming `-f foo` now')
# Or use the built-in error handling like this:
# parser.error('Option "-f" is not supported anymore.')
# You could override the argument value like this:
# setattr(namespace, self.dest, 'foo')
parser = argparse.ArgumentParser()
parser.add_argument('-f', default='foo', action=FooAction)
args = parser.parse_args()
print('option=%s' % args.f)
You could also just limit the choices to only "foo" and let argparse create an error for other values:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-f', default='foo', choices=['foo'])
args = parser.parse_args()
print('option=%s' % args.f)
Calling python test.py -f bar would then result in:
usage: test.py [-h] [-f {foo}]
test.py: error: argument -f: invalid choice: 'bar' (choose from 'foo')

Set default for all subparsers on top level parser

I have an argparse parser with several subcommands some of which share an option (via a parent parser). Now I want to set a default value for such an option regardless of which subparser will be executed in the end. My non working code looks like this:
from argparse import ArgumentParser
base = ArgumentParser(add_help=False)
base.add_argument('--foo', action='store_true')
parser = ArgumentParser()
subparsers = parser.add_subparsers(dest='action')
s1 = subparsers.add_parser('a', parents=[base])
s2 = subparsers.add_parser('b', parents=[base])
parser.set_defaults(foo=42)
print(parser.parse_args(['a']))
s1.set_defaults(foo=43)
print(parser.parse_args(['a']))
This prints
Namespace(action='a', foo=False)
Namespace(action='a', foo=43)
I have many subparsers and many options so I want to avoid saving every subparser by name and calling set_defauls on it. Can that be done?
I will know the value I want to set there only after creating all the parsers so I can not specify the defaults in the call to add_argument.
Background: what I am actually working on
The defaults I want to set come from a config file. I actually have two parsers, one to find the config file first and one to parse the subcommands. But I need to define both parsers up front in order to overwrite the help method of the first parser with the help method of the second parser in order to display the full --help text before parsing the config (because that might fail and I could not display the help text). A reduced version of my code looks like this:
import argparse
base = argparse.ArgumentParser(add_help=False)
base.add_argument("--config", help="config file to use")
p1 = argparse.ArgumentParser(parents=[base])
p1.add_argument('remainder', nargs=argparse.REMAINDER)
p2 = argparse.ArgumentParser(parents=[base])
s = p2.add_subparsers(dest='action')
s1 = s.add_parser('a') # add some options
s2 = s.add_parser('b') # add some options
# and so on
p1.print_help = p2.print_help
a1 = p1.parse_args()
config = load_my_config(a1.config)
p2.set_defaults(**config.get_my_defaults())
a2 = p2.parse_args(a1.remainder)

I found a solution to listing all the subparsers. The solution is not to remember all the variables for the different sub-parsers but only the _SubParsersAction object that was used to create them:
import argparse
p = argparse.ArgumentParser()
s = p.add_subparsers()
a = s.add_parser('a')
a.add_argument(...)
b = s.add_parser('b')
c = s.add_parser('c')
...
# now I don't need to remember all the variables a, b, c, ...
# in order to set the defaults on all of theses sub-parsers
config = load_my_config_file()
defaults = config.get_defaults()
for name, sparser in s.choises:
print("Setting defaults on sub parser for '{}'").format(name)
sparser.set_defaults(**defaults)

How ca I get Python ArgParse to stop overwritting positional arguments in child parser

I am attempting to get my script working, but argparse keeps overwriting my positional arguments from the parent parser. How can I get argparse to honor the parent's value for these? It does keep values from optional args.
Here is a very simplified version of what I need. If you run this, you will see that the args are overwritten.
testargs.py
#! /usr/bin/env python3
import argparse
import sys
def main():
preparser = argparse.ArgumentParser(add_help=False)
preparser.add_argument('first',
nargs='?')
preparser.add_argument('outfile',
nargs='?',
type=argparse.FileType('w', encoding='utf-8'),
default=sys.stdout,
help='Output file')
preparser.add_argument(
'--do-something','-d',
action='store_true')
# Parse args with preparser, and find config file
args, remaining_argv = preparser.parse_known_args()
print(args)
parser = argparse.ArgumentParser(
parents=[preparser],
description=__doc__)
parser.add_argument(
'--clear-screen', '-c',
action='store_true')
args = parser.parse_args(args=remaining_argv,namespace=args )
print(args)
if __name__ == '__main__':
main()
And call it with testargs.py something /tmp/test.txt -d -c
You will see it keeps the -d but drops both the positional args and reverts them to defaults.
EDIT: see additional comments in the accepted answer for some caveats.

When you specify parents=[preparser] it means that parser is an extension of preparser, and will parse all arguments relevent to preparser which it is never given.
Lets say the preparser only has one positional argument first and the parser only has one positional argument second, when you make parser a child of preparser it expects both arguments:
import argparse
parser1 = argparse.ArgumentParser(add_help=False)
parser1.add_argument("first")
parser2 = argparse.ArgumentParser(parents=[parser1])
parser2.add_argument("second")
args2 = parser2.parse_args(["arg1","arg2"])
assert args2.first == "arg1" and args2.second == "arg2"
However passing only the remaining arguments that are left over from parser1 would just be ['second'] which is not the correct arguments to parser2:
parser1 = argparse.ArgumentParser(add_help=False)
parser1.add_argument("first")
args1, remaining_args = parser1.parse_known_args(["arg1","arg2"])
parser2 = argparse.ArgumentParser(parents=[parser1])
parser2.add_argument("second")
>>> args1
Namespace(first='arg1')
>>> remaining_args
['arg2']
>>> parser2.parse_args(remaining_args)
usage: test.py [-h] first second
test.py: error: the following arguments are required: second
To only process the arguments that were not handled by the first pass, do not specify it as the parent to the second parser:
parser1 = argparse.ArgumentParser(add_help=False)
parser1.add_argument("first")
args1, remaining_args = parser1.parse_known_args(["arg1","arg2"])
parser2 = argparse.ArgumentParser() #parents=[parser1]) #NO PARENT!
parser2.add_argument("second")
args2 = parser2.parse_args(remaining_args,args1)
assert args2.first == "arg1" and args2.second == "arg2"

The 2 positionals are nargs='?'. A positional like that is always 'seen', since an empty list matches that nargs.
First time through 'text.txt' matches with first and is put in the Namespace. Second time through there isn't any string to match, so the default is used - same as if you had not given that string the first time.
If I change first to have the default nargs, I get
error: the following arguments are required: first
from the 2nd parser. Even though there's a value in the Namespace it still tries to get a value from the argv. (it's like a default, but not quite).
Defaults for positionals with nargs='?' (or *) are tricky. They are optional, but not in quite the same way as optionals. The positional Actions are still called, but with a empty list of values.
I don't think the parents feature does anything for you. preparser already handles that set of arguments; there's no need to handle them again in parser, especially since all the relevant argument strings have been stripped out.
Another option is to leave the parents in, but use the default sys.argv[1:] in the 2nd parser. (but beware of side effects like opening files)
args = parser.parse_args(namespace=args )
A third option is to parse the arguments independently and merge them with a dictionary update.
adict = vars(preparse_args)
adict.update(vars(parser_args))
# taking some care in who overrides who
For more details look in argparse.py file at ArgumentParser._get_values, specifically the not arg_strings cases.
A note about the FileType. That type works nicely for small scripts where you will use the files right away and exit. It isn't so good on large programs where you might want to close the file after use (close stdout???), or use files in a with context.
edit - note on parents
add_argument creates an Action object, and adds it to the parser's list of actions. parse_args basically matches input strings with these actions.
parents just copies those Action objects (by reference) from parent to child. To the child parser it is just as though the actions were created with add_argument directly.
parents is most useful when you are importing a parser and don't have direct access to its definition. If you are defining both parent and child, then parents just saves you some typing/cut-n-paste.
This and other SO questions (mostly triggered the by-reference copy) show that the developers did not intend you to use both the parent and child to do parsing. It can be done, but there are glitches that the they did not consider.
===================
I can imagine defining a custom Action class that would 'behave' in a situation like this. It might, for example, check the namespace for some not default value before adding its own (possibly default) value.
Consider, for example if I changed the action of first to 'append':
preparser.add_argument('first', action='append', nargs='?')
The result is:
1840:~/mypy$ python3 stack37147683.py /tmp/test.txt -d -c
Namespace(do_something=True, first=['/tmp/test.txt'], outfile=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)
Namespace(clear_screen=True, do_something=True, first=['/tmp/test.txt', None], outfile=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)
From the first parser, first=['/tmp/test.txt']; from the second, first=['/tmp/test.txt', None].
Because of the append, the item from the first is preserved, and a new default has been added by the second parser.

Using argparse to parse arguments of form "arg= val"

I want to use argparse to parse command lines of form "arg=val"
For example, the usage would be:
script.py conf_dir=/tmp/good_conf
To achieve it, I am doing this:
desc = "details"
parser = argparse.ArgumentParser(description=desc, add_help=False)
args = parser.add_argument("conf_dir")
args = parser.parse_args("conf_dir=FOO".split())
args = parser.parse_args()
print args.conf_dir
But, the problem is that, on invocation of the script with:
python script.py conf_dir=/tmp/good_conf
I get:
conf_dir=/tmp/good_conf
Where as I expect
/tmp/good_conf
So, the question is: Can I use argparse to parse cmd line, which contains name value pairs?
Any hints?
Edit: The reason I want to do this and not some thing like --conf_dir=/tmp/good_dir is because there are other tools (written in other language), which uses conf_dir=/tmp/good_dir style of arguments. To maintain consistency, I was to parse args in this way.

You need a custom action
class StoreNameValuePair(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
n, v = values.split('=', 1)
setattr(namespace, n, v)
args = parser.add_argument("conf_dir", action=StoreNameValuePair)

As per the documentation, argparse doesn't natively let you have unprefixed options like that. If you omit the leading -, it assumes you are describing a positional argument and expects it to be provided as:
python script.py /tmp/good_conf
If you want it to be optional, it needs to be correctly marked as a flag by calling it --conf_dir, and invoking the script like:
python script.py --conf_dir=/tmp/good_conf
However, to accept name-value pairs, you can implement a custom action. In combination with nargs, such an action could accept an arbitrary number of name-value pairs and store them on the argument parsing result object.

#chepner This is great. I improved this to support multiple args as well and store the result as dict:
class StoreDict(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
kv={}
if not isinstance(values, (list,)):
values=(values,)
for value in values:
n, v = value.split('=')
kv[n]=v
setattr(namespace, self.dest, kv)

The usual way to put name value pairs on the command line is with options. I.e. you would use
python script.py --confdir=/tmp/good_conf
argparse can certainly handle that case. See the docs at:
http://docs.python.org/library/argparse.html#option-value-syntax

Python Optparse list

I'm using the python optparse module in my program, and I'm having trouble finding an easy way to parse an option that contains a list of values.
For example:
--groups one,two,three.
I'd like to be able to access these values in a list format as options.groups[]. Is there an optparse option to convert comma separated values into a list? Or do I have to do this manually?

S.Lott's answer has already been accepted, but here's a code sample for the archives:
def foo_callback(option, opt, value, parser):
setattr(parser.values, option.dest, value.split(','))
parser = OptionParser()
parser.add_option('-f', '--foo',
type='string',
action='callback',
callback=foo_callback)

Look at option callbacks. Your callback function can parse the value into a list using a basic optarg.split(',')

Again, just for the sake of archive completeness, expanding the example above:
You can still use "dest" to specify the option name for later access
Default values cannot be used in such cases (see explanation in Triggering callback on default value in optparse)
If you'd like to validate the input, OptionValueError should be thrown from foo_callback
The code (with tiny changes) would then be:
def get_comma_separated_args(option, opt, value, parser):
setattr(parser.values, option.dest, value.split(','))
parser = OptionParser()
parser.add_option('-f', '--foo',
type='string',
action='callback',
callback=get_comma_separated_args,
dest = foo_args_list)

With optparse, to get a list value you can use action 'append':
from optparse import OptionParser
parser = OptionParser()
parser.add_option("--group",
action="append",
dest="my_groups")
(options, args) = parser.parse_args()
print options.my_groups
Then call your program like this:
$ python demo.py --group one --group two --group three
['one', 'two', 'three']