We Keep Coding

Named arguments with Python argparse

I'm trying to create a terminal application a bit similar to cutechess-cli, which has options like the following:
-epdout FILE Save the end position of the games to FILE in FEN format.
-recover Restart crashed engines instead of stopping the match
-repeat [N] Play each opening twice (or N times). Unless the -noswap...
which can all be done with argparse in Python.
However it also has "named arguments" like the following:
-resign movecount=COUNT score=SCORE [twosided=VALUE]
Adjudicate the game as a loss if an engine's score is
at least SCORE centipawns below zero for at least COUNT
consecutive moves.
-sprt elo0=ELO0 elo1=ELO1 alpha=ALPHA beta=BETA
Use a Sequential Probability Ratio Test as a termination
criterion for the match. This option should only be used...
I can implement that with argparse using nargs='*' and then writing my own parser (maybe just regex). However that doesn't give nice documentation, and if argparse can already do something like this, I would rarther use the builtin approach.
Summary: Does argparse have a concept of named arguments similar to resign and sprt above? And if not, would the best approach be to do this manyally using nargs='*'?

You can use a custom type to split the values and use the metavar argument to give a better description for the value:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--arg', nargs='*', type=lambda text: text.split('=', maxsplit=1), metavar='PARAM-NAME=PARAM-VALUE', help='Some other parameters')
args = parser.parse_args()
args.arg = {k: v for k,v in args.arg}
Which produces:
usage: [-h] [--arg [PARAM-NAME=PARAM-VALUE [PARAM-NAME=PARAM-VALUE ...]]]
optional arguments:
-h, --help show this help message and exit
--arg [PARAM-NAME=PARAM-VALUE [PARAM-NAME=PARAM-VALUE ...]]
Some other parameters
If you wanted to you could avoid the "postprocessing" step to build the dictionary by using a custom Action type. But it seems an overkill to me in this case.

Python argparse - creating subcommands with no labels

I want to create a command parser mycommand, using argparse, with two subcommands read and write: read should have just one argument which is some path, and write should have two arguments one of which is a path and the other a value. It should be possible to execute the command in the following way:
mycommand read <path>
mycommand write <path> <value>
without using labels for the <path>, <value> arguments, i.e. without requiring --path. How can I do this?

This is pretty straight forward following the docs:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
read = subparsers.add_parser('read')
read.add_argument('path')
write = subparsers.add_parser('write')
write.add_argument('path')
write.add_argument('value')
print(parser.parse_args(['read', 'foo']))
print(parser.parse_args(['write', 'foo', 'bar']))
Note that this doesn't tell you what parser parsed the arguments. If you want that, you can simply add a dest to the add_subparsers command:
subparsers = parser.add_subparsers(dest='subparser')
Finally, you can add a default attribute for each subparser that you can use to perform the actions specific to that subparser. This is spelled out in the docs too, but for completeness, in our example, it might look something like this1:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest='subparser')
def handle_read(args):
print('Handling read')
print(args)
read = subparsers.add_parser('read')
read.add_argument('path')
read.set_defaults(handler=handle_read)
def handle_write(args):
print('Handling write')
print(args)
write = subparsers.add_parser('write')
write.add_argument('path')
write.add_argument('value')
write.set_defaults(handler=handle_write)
args = parser.parse_args()
args.handler(args)
1I added the dest to subparsers in this example too for illustrative purposes -- Using argparse with handler functions probably makes that attribute on args obsolete.

argparse - Build back command line

In Python, how can I parse the command line, edit the resulting parsed arguments object and generate a valid command line back with the updated values?
For instance, I would like python cmd.py --foo=bar --step=0 call python cmd.py --foo=bar --step=1 with all the original --foo=bar arguments, potentially without extra arguments added when default value is used.
Is it possible with argparse?

You can use argparse to parse the command-line arguments, and then modify those as desired. At the moment however, argparse lacks the functionality to work in reverse and convert those values back into a command-line string. There is however a package for doing precisely that, called argunparse. For example, the following code in cmd.py
import sys
import argparse
import argunparse
parser = argparse.ArgumentParser()
unparser = argunparse.ArgumentUnparser()
parser.add_argument('--foo')
parser.add_argument('--step', type=int)
kwargs = vars(parser.parse_args())
kwargs['step'] += 1
prefix = f'python {sys.argv[0]} '
arg_string = unparser.unparse(**kwargs)
print(prefix + arg_string)
will print the desired command line:
python cmd.py --foo=bar --step=1

argparse is clearly designed to go one way, from sys.argv to the args namespace. No thought has been given to preserving information that would let you map things back the other way, much less do the mapping itself.
In general, multiple sys.argv could produce the same args. You could, for example, have several arguments that have the same dest. Or you can repeat 'optionals'. But for a restricted 'parser' setup there may be enough information to recreate a usable argv.
Try something like:
parser = argparser.ArgumentParser()
arg1 = parser.add_argument('--foo', default='default')
arg2 = parser.add_argument('bar', nargs=2)
and then examine the arg1 and arg2 objects. They contain all the information that you supplied to the add_argument method. Of course you could have defined those values in your own data structures before hand, e.g.
{'option_string':'--foo', 'default':'default'}
{'dest':'bar', 'nargs':2}
and used those as input to add_argument.
While the parser may have enough information to recreate a useable sys.argv, you have to figure out how to do that yourself.
default=argparse.SUPPRESS may be handy. It keeps the parser from adding a default entry to the namespace. So if the option isn't used, it won't appear in the namespace.

This isn't possible in any easy way that I know of, then again I've never needed to do this.
But with the lack of information in the question in regards to how you call your script, I'll assume the following:
python test.py cmd --foo=bar --step=0
And what you could do is do:
from sys import argv
for index in range(1, len(argv)): # the first object is the script itself
if '=' in argv[index]:
param, value = argv[index].split('=', 1)
if param == '--step':
value = '1'
argv[index] = param + '=' + value
print(argv)
Note that this is very specific to --step and may be what you've already thought of and just wanted a "better way", but again, I don't think there is.

depending on the scope, this works at the same module at least:
pprint(argparse._sys.argv)

Per the other answers, rebuilding is imperfect, but if you aren't doing anything too fancy and are okay with imperfect, something like this could work for you as a starting point:
def unparse_args(parser, parsed_args):
"""Unparse argparsed args"""
positional_args = [action.dest
for action in parser._actions
if not action.option_strings]
optionals = []
positionals = []
for key, value in vars(parsed_args).items():
if not value:
# none and false flags go away
continue
elif key in positional_args:
positionals.append(value)
elif value is True:
optionals.append(f"--{key}")
else:
optionals.append(f"--{key}={value}")
return " ".join(optionals + positionals)
Here's an example using this with a git clone clone:
parser = argparse.ArgumentParser(description='A sample git clone wrapper')
# options
parser.add_argument("-v", "--verbose", action="store_true",
help="be more verbose")
parser.add_argument("-q", "--quiet", action="store_true",
help="be more quiet")
parser.add_argument("--recurse-submodules", nargs='?',
help="initialize submodules in the clone")
parser.add_argument("--recursive", nargs='?',
help="alias of --recurse-submodules")
parser.add_argument("-b", "--branch",
help=" checkout <branch> instead of the remote's HEAD")
parser.add_argument("--depth", type=int,
help="create a shallow clone of that depth")
parser.add_argument("--shallow-submodules", action="store_true",
help="any cloned submodules will be shallow")
# positional
parser.add_argument("repo", help="The git repo to clone")
parser.add_argument("dir", nargs='?',help="The location to clone the repo")
# make a fake call to your git clone clone and parse the args
cmdargs = ["--depth=1", "-q", "ohmyzsh/ohmyzsh"]
parsedargs = parser.parse_args(cmdargs)
# now unparse them
unparsed = unparse_args(parser, parsedargs)
print(unparsed)

Python - Difference between docopt and argparse

I have to write a command-line interface and I've seen I can use docopt and argparse.
I would like to know what are the main differences between the two so that I can make an enlightened choice.
Please stick to the facts. I don't want Wow. docopt. So beautiful. Very useful.

Docopt parses a doc string, whereas argparse constructs its parsing by creating an object instance and adding behaviour to it by function calls.
Example for argparse:
parser = argparse.ArgumentParser()
parser.add_argument("operation", help="mathematical operation that will be performed",
choices=['add', 'subtract', 'multiply', 'divide'])
parser.add_argument("num1", help="the first number", type=int)
parser.add_argument("num2", help="the second number", type=int)
args = parser.parse_args()
Example for docopt:
"""Calculator using docopt
Usage:
calc_docopt.py <operation> <num1> <num2>
calc_docopt.py (-h | --help)
Arguments:
<operation> Math Operation
<num1> First Number
<num2> Second Number
Options:
-h, --help Show this screen.
"""
from docopt import docopt
if __name__ == '__main__':
arguments = docopt(__doc__, version='Calculator with docopt')
print(arguments)
Note, that docopt uses Usage: and Options: sections for parsing. Here Arguments: is provided only for end-user convenience.

The Why page for Click:
https://click.palletsprojects.com/en/7.x/why/
has a nice comparison between argparse, docopt and click itself.
Click is another command-line parsing utility for Python.

argparse is in the python default library, so this doesn't add any extra dependencies to your program. The differences are mainly the way of writing your code. Using argparse it is possible to add hooks for plugins so they can add their own argumnets to your program. For example flake8 uses this.
docopt is a third party module provides a simple way of parsing arguments. I personally like docopt because of its simplicity, but I'm not saying it is the best to use in all cases. In their documentation they mention that using docopt it is possible to use more combinations of argument passing than when using argparse.

Python: argument parser that handles global options to sub-commands properly

argparse fails at dealing with sub-commands receiving global options:
import argparse
p = argparse.ArgumentParser()
p.add_argument('--arg', action='store_true')
s = p.add_subparsers()
s.add_parser('test')
will have p.parse_args('--arg test'.split()) work,
but fails on p.parse_args('test --arg'.split()).
Anyone aware of a python argument parser that handles global options to sub-commands properly?

You can easily add this argument to both parsers (main parser and subcommand parser):
import argparse
main = argparse.ArgumentParser()
subparser = main.add_subparsers().add_parser('test')
for p in [main,subparser]:
p.add_argument('--arg', action='store_true')
print main.parse_args('--arg test'.split()).arg
print main.parse_args('test --arg'.split()).arg
Edit: As #hpaulj pointed in comment, there is also parents argument which you can pass to ArgumentParser constructor or to add_parser method. You can list in this value parsers which are bases for new one.
import argparse
base = argparse.ArgumentParser(add_help=False)
base.add_argument('--arg', action='store_true')
main = argparse.ArgumentParser(parents=[base])
subparser = main.add_subparsers().add_parser('test', parents=[base])
print main.parse_args('--arg test'.split()).arg
print main.parse_args('test --arg'.split()).arg
More examples/docs:
looking for best way of giving command line arguments in python, where some params are req for some option and some params are req for other options
Python argparse - Add argument to multiple subparsers (I'm not sure if this question is not overlaping with this one too much)
http://docs.python.org/dev/library/argparse.html#parents

Give docopt a try:
>>> from docopt import docopt
>>> usage = """
... usage: prog.py command [--test]
... prog.py another [--test]
...
... --test Perform the test."""
>>> docopt(usage, argv='command --test')
{'--test': True,
'another': False,
'command': True}
>>> docopt(usage, argv='--test command')
{'--test': True,
'another': False,
'command': True}

There's a ton of argument-parsing libs in the Python world. Here are a few that I've seen, all of which should be able to handle address the problem you're trying to solve (based on my fuzzy recollection of them when I played with them last):
opster—I think this is what mercurial uses, IIRC
docopt—This one is new, but uses an interesting approach
cliff—This is a relatively new project by Doug Hellmann (PSF member, virtualenvwrapper author, general hacker extraordinaire) is a bit more than just an argument parser, but is designed from the ground up to handle multi-level commands
clint—Another project that aims to be "argument parsing and more", this one by Kenneth Reitz (of Requests fame).

Here's a dirty workaround --
import argparse
p = argparse.ArgumentParser()
p.add_argument('--arg', action='store_true')
s = p.add_subparsers()
s.add_parser('test')
def my_parse_args(ss):
#parse the info the subparser knows about; don't issue an error on unknown stuff
namespace,leftover=p.parse_known_args(ss)
#reparse the unknown as global options and add it to the namespace.
if(leftover):
s.add_parser('null',add_help=False)
p.parse_args(leftover+['null'],namespace=namespace)
return namespace
#print my_parse_args('-h'.split()) #This works too, but causes the script to stop.
print my_parse_args('--arg test'.split())
print my_parse_args('test --arg'.split())
This works -- And you could modify it pretty easily to work with sys.argv (just remove the split string "ss"). You could even subclass argparse.ArgumentParser and replace the parse_args method with my_parse_args and then you'd never know the difference -- Although subclassing to replace a single method seems overkill to me.
I think however, that this is a lit bit of a non-standard way to use subparsers. In general, global options are expected to come before subparser options, not after.

The parser has a specific syntax: command <global options> subcommand <subcommand ptions>, you are trying to feed the subcommand with an option and but you didn't define one.