We Keep Coding

Argparse outputting help text twice

After an hour googling, I can't find anybody who has had anything resembling this issue besides myself. I created a command line interface with argparse. Originally I had tried to leverage argparse's built in help text behavior. But my boss isn't satisfied with the default help text, so he is having me write up the full usage/help text in a text file and just display the entire file.
For some reason, in a certain case, its outputting the text twice.
Here is the basics of how my program is broken down:
I have a top level parser. I read in my help text file, set it to a string help_text, and then set "usage=help_text" on the parser. Then I create subparsers (4 of them and then a base case) to create subcommands. Only one of those subparsers has any additional arguments (one positional, one optional). Before I reworked the help text, I had help text for each individual subcommand by using "help=" but now those are all blank. Lastly, I have set up a base case to display the help text whenever no subcommands are given.
Here is the behavior I'm getting:
When I call the main function with no subcommands and no arguments, my help_text from the text file outputs, and then like 2-3 additional lines of boiler plate I can't seem to get rid of. Also because the word usage appears in my text file, it says "usage: usage"
When I call the main command and then type --help, the exact same thing happens as above.
When I call the one subcommand that has a required positional argument and I don't include that argument... it spits out the entire help text twice. Right above the second time it prints, it prints the default usage line for that subcommand.
Lastly, when I use a different subcommand that has no arguments and give it an argument (one too many) it spits out everything completely correctly without even the extra couple lines at the end.
I don't know how to make heads or tales about this. Here is the main function of the script (I can verify that this problem occurs only in the main function where argparse is used, not the other functions that the main function calls):
def main():
# Import help text from file
p = Path(__file__).with_name("help_text.txt")
with p.open() as file:
help_text = file.read()
# Configure the top level Parser
parser = argparse.ArgumentParser(prog='hubmap-clt', description='Name of cli', usage=help_text)
subparsers = parser.add_subparsers()
# Create Subparsers to give subcommands
parser_transfer = subparsers.add_parser('subcommandone')
parser_transfer.add_argument('argument1', type=str)
parser_transfer.add_argument('--optionalargument', default='mydefault')
parser_login = subparsers.add_parser('subcommandtwo')
parser_whoami = subparsers.add_parser('subcommandthree')
parser_logout = subparsers.add_parser('subcommandfour')
# Assign subparsers to their respective functions
parser_subcommandone.set_defaults(func=subcommandone)
parser_subcommandtwo.set_defaults(func=subcommandtwo)
parser_subcommandthree.set_defaults(func=subcommandthree)
parser_subcommandfour.set_defaults(func=subcommandfour)
parser.set_defaults(func=base_case)
# Parse the arguments and call appropriate functions
args = parser.parse_args()
if len(sys.argv) == 1:
args.func(args, parser)
else:
args.func(args)
So to clarify:
Why does the extra couple lines of boiler-plat help text appear sometimes which looks like this:
name of cli
positional arguments:
{subcommandone,subcommandtwo,subcommandthree,subcommandfour}
optional arguments:
-h, --help show this help message and exit
Why does using subcommandone with too few arguments print out the help text twice (but NOT the extra lines of boiler-plate help text.
why does using subcommandtwo with one too MANY arguments print everything perfectly without any extra lines?

With a modification of your main:
def foo():
# Import help text from file
# p = Path(__file__).with_name("help_text.txt")
# with p.open() as file:
# help_text = file.read()
help_text = "cli usage: foobar\n morebar"
# Configure the top level Parser
parser = argparse.ArgumentParser(
prog="hubmap-clt", description="Name of cli", usage=help_text
)
subparsers = parser.add_subparsers()
# Create Subparsers to give subcommands
parser_transfer = subparsers.add_parser("subcommandone")
parser_transfer.add_argument("argument1", type=str)
parser_transfer.add_argument("--optionalargument", default="mydefault")
parser_login = subparsers.add_parser("subcommandtwo")
# parser_whoami = subparsers.add_parser("subcommandthree")
# parser_logout = subparsers.add_parser("subcommandfour")
# Assign subparsers to their respective functions
parser_transfer.set_defaults(func="subcommandone")
parser_login.set_defaults(func="subcommandtwo")
# parser_subcommandthree.set_defaults(func="subcommandthree")
# parser_subcommandfour.set_defaults(func="subcommandfour")
parser.set_defaults(func="base_case")
return parser
in an iteractive ipython session:
In [8]: p = foo()
In [9]: p.print_usage()
usage: cli usage: foobar
morebar
Usage is exactly as I specified. And the help for the main parser:
In [10]: p.print_help()
usage: cli usage: foobar
morebar
Name of cli
positional arguments:
{subcommandone,subcommandtwo}
optional arguments:
-h, --help show this help message and exit
That's what I expect given the arguments.
Help for a subparser:
In [11]: p.parse_args(["subcommandone", "-h"])
usage: cli usage: foobar
morebar subcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
positional arguments:
argument1
optional arguments:
-h, --help show this help message and exit
--optionalargument OPTIONALARGUMENT
Usage is like the main's but with some added info on how to call this subparser and its arguments.
Error when calling the subparsers without enough values:
In [15]: p.parse_args(["subcommandone"])
usage: cli usage: foobar
morebar subcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
cli usage: foobar
morebar subcommandone: error: the following arguments are required: argument1
Is this repeat of cli usage that bothering you? This error is raised by the subparser, and I suspect the extra comes from the prog of that subparser. I think I saw something like this on the Python bug/issues for argparse.
error with too much:
In [17]: p.parse_args(["subcommandone", "test", "extra"])
usage: cli usage: foobar
morebar
hubmap-clt: error: unrecognized arguments: extra
In this case error is produced by the main parser, hence the "hubmat-clt" prog.
change prog:
...: parser_transfer = subparsers.add_parser(
...: "subcommandone", prog="hubmap-clt sobcommandone"
...: )
In [21]: p.parse_args(["subcommandone", "test", "extra"])
usage: cli usage: foobar
morebar
hubmap-clt: error: unrecognized arguments: extra
In [22]: p.parse_args(["subcommandone"])
usage: hubmap-clt sobcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
hubmap-clt sobcommandone: error: the following arguments are required: argument1
[21] is as before [17]. But [22] is now showing the prog that I set. I could also have specified a custom usage for the subparser.
If I modify the function to use default usage and prog, but also display the subparser's prog. And I gave the main an "main_foo" positional argument:
In [30]: p = foo()
hubmap-clt main_foo subcommandone
In [31]: p.parse_args(["subcommandone"])
Out[31]: Namespace(main_foo='subcommandone')
In [32]: p.parse_args(["foo", "subcommandone"])
usage: hubmap-clt main_foo subcommandone [-h] [--optionalargument OPTIONALARGUMENT] argument1
hubmap-clt main_foo subcommandone: error: the following arguments are required: argument1
Notice how the main's usage has been incorporated into the 'prog' for the subparser.
In the bug/issue I found the main parser's usage gets incorporated into the prog of the subparser. That's why you see the duplicate.
https://bugs.python.org/issue42297
[argparse] Bad error message formatting when using custom usage text
The relatively recent date of this bug issue indicates that custom usage is not that common, and even less so when used with subparsers. As my post on this issue indicates, the relation between the main parser, the "subparsers" command, and individual subparsers gets complicated.

How to read with python, input from keyboard in the terminal, by calling the script.py through an alias [duplicate]

In Python, how can we find out the command line arguments that were provided for a script, and process them?
For some more specific examples, see Implementing a "[command] [action] [parameter]" style command-line interfaces? and How do I format positional argument help using Python's optparse?.

import sys
print("\n".join(sys.argv))
sys.argv is a list that contains all the arguments passed to the script on the command line. sys.argv[0] is the script name.
Basically,
import sys
print(sys.argv[1:])

The canonical solution in the standard library is argparse (docs):
Here is an example:
from argparse import ArgumentParser
parser = ArgumentParser()
parser.add_argument("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")
parser.add_argument("-q", "--quiet",
action="store_false", dest="verbose", default=True,
help="don't print status messages to stdout")
args = parser.parse_args()
argparse supports (among other things):
Multiple options in any order.
Short and long options.
Default values.
Generation of a usage help message.

Just going around evangelizing for argparse which is better for these reasons.. essentially:
(copied from the link)
argparse module can handle positional
and optional arguments, while
optparse can handle only optional
arguments
argparse isn’t dogmatic about
what your command line interface
should look like - options like -file
or /file are supported, as are
required options. Optparse refuses to
support these features, preferring
purity over practicality
argparse produces more
informative usage messages, including
command-line usage determined from
your arguments, and help messages for
both positional and optional
arguments. The optparse module
requires you to write your own usage
string, and has no way to display
help for positional arguments.
argparse supports action that
consume a variable number of
command-line args, while optparse
requires that the exact number of
arguments (e.g. 1, 2, or 3) be known
in advance
argparse supports parsers that
dispatch to sub-commands, while
optparse requires setting
allow_interspersed_args and doing the
parser dispatch manually
And my personal favorite:
argparse allows the type and
action parameters to add_argument()
to be specified with simple
callables, while optparse requires
hacking class attributes like
STORE_ACTIONS or CHECK_METHODS to get
proper argument checking

There is also argparse stdlib module (an "impovement" on stdlib's optparse module). Example from the introduction to argparse:
# script.py
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument(
'integers', metavar='int', type=int, choices=range(10),
nargs='+', help='an integer in the range 0..9')
parser.add_argument(
'--sum', dest='accumulate', action='store_const', const=sum,
default=max, help='sum the integers (default: find the max)')
args = parser.parse_args()
print(args.accumulate(args.integers))
Usage:
$ script.py 1 2 3 4
4
$ script.py --sum 1 2 3 4
10

If you need something fast and not very flexible
main.py:
import sys
first_name = sys.argv[1]
last_name = sys.argv[2]
print("Hello " + first_name + " " + last_name)
Then run python main.py James Smith
to produce the following output:
Hello James Smith

The docopt library is really slick. It builds an argument dict from the usage string for your app.
Eg from the docopt readme:
"""Naval Fate.
Usage:
naval_fate.py ship new <name>...
naval_fate.py ship <name> move <x> <y> [--speed=<kn>]
naval_fate.py ship shoot <x> <y>
naval_fate.py mine (set|remove) <x> <y> [--moored | --drifting]
naval_fate.py (-h | --help)
naval_fate.py --version
Options:
-h --help Show this screen.
--version Show version.
--speed=<kn> Speed in knots [default: 10].
--moored Moored (anchored) mine.
--drifting Drifting mine.
"""
from docopt import docopt
if __name__ == '__main__':
arguments = docopt(__doc__, version='Naval Fate 2.0')
print(arguments)

One way to do it is using sys.argv. This will print the script name as the first argument and all the other parameters that you pass to it.
import sys
for arg in sys.argv:
print arg

#set default args as -h , if no args:
if len(sys.argv) == 1: sys.argv[1:] = ["-h"]

I use optparse myself, but really like the direction Simon Willison is taking with his recently introduced optfunc library. It works by:
"introspecting a function
definition (including its arguments
and their default values) and using
that to construct a command line
argument parser."
So, for example, this function definition:
def geocode(s, api_key='', geocoder='google', list_geocoders=False):
is turned into this optparse help text:
Options:
-h, --help show this help message and exit
-l, --list-geocoders
-a API_KEY, --api-key=API_KEY
-g GEOCODER, --geocoder=GEOCODER

I like getopt from stdlib, eg:
try:
opts, args = getopt.getopt(sys.argv[1:], 'h', ['help'])
except getopt.GetoptError, err:
usage(err)
for opt, arg in opts:
if opt in ('-h', '--help'):
usage()
if len(args) != 1:
usage("specify thing...")
Lately I have been wrapping something similiar to this to make things less verbose (eg; making "-h" implicit).

As you can see optparse "The optparse module is deprecated with and will not be developed further; development will continue with the argparse module."

Pocoo's click is more intuitive, requires less boilerplate, and is at least as powerful as argparse.
The only weakness I've encountered so far is that you can't do much customization to help pages, but that usually isn't a requirement and docopt seems like the clear choice when it is.

import argparse
parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
const=sum, default=max,
help='sum the integers (default: find the max)')
args = parser.parse_args()
print(args.accumulate(args.integers))
Assuming the Python code above is saved into a file called prog.py
$ python prog.py -h
Ref-link: https://docs.python.org/3.3/library/argparse.html

You may be interested in a little Python module I wrote to make handling of command line arguments even easier (open source and free to use) - Commando

Yet another option is argh. It builds on argparse, and lets you write things like:
import argh
# declaring:
def echo(text):
"Returns given word as is."
return text
def greet(name, greeting='Hello'):
"Greets the user with given name. The greeting is customizable."
return greeting + ', ' + name
# assembling:
parser = argh.ArghParser()
parser.add_commands([echo, greet])
# dispatching:
if __name__ == '__main__':
parser.dispatch()
It will automatically generate help and so on, and you can use decorators to provide extra guidance on how the arg-parsing should work.

I recommend looking at docopt as a simple alternative to these others.
docopt is a new project that works by parsing your --help usage message rather than requiring you to implement everything yourself. You just have to put your usage message in the POSIX format.

Also with python3 you might find convenient to use Extended Iterable Unpacking to handle optional positional arguments without additional dependencies:
try:
_, arg1, arg2, arg3, *_ = sys.argv + [None] * 2
except ValueError:
print("Not enough arguments", file=sys.stderr) # unhandled exception traceback is meaningful enough also
exit(-1)
The above argv unpack makes arg2 and arg3 "optional" - if they are not specified in argv, they will be None, while if the first is not specified, ValueError will be thouwn:
Traceback (most recent call last):
File "test.py", line 3, in <module>
_, arg1, arg2, arg3, *_ = sys.argv + [None] * 2
ValueError: not enough values to unpack (expected at least 4, got 3)

My solution is entrypoint2. Example:
from entrypoint2 import entrypoint
#entrypoint
def add(file, quiet=True):
''' This function writes report.
:param file: write report to FILE
:param quiet: don't print status messages to stdout
'''
print file,quiet
help text:
usage: report.py [-h] [-q] [--debug] file
This function writes report.
positional arguments:
file write report to FILE
optional arguments:
-h, --help show this help message and exit
-q, --quiet don't print status messages to stdout
--debug set logging level to DEBUG

import sys
# Command line arguments are stored into sys.argv
# print(sys.argv[1:])
# I used the slice [1:] to print all the elements except the first
# This because the first element of sys.argv is the program name
# So the first argument is sys.argv[1], the second is sys.argv[2] ecc
print("File name: " + sys.argv[0])
print("Arguments:")
for i in sys.argv[1:]:
print(i)
Let's name this file command_line.py and let's run it:
C:\Users\simone> python command_line.py arg1 arg2 arg3 ecc
File name: command_line.py
Arguments:
arg1
arg2
arg3
ecc
Now let's write a simple program, sum.py:
import sys
try:
print(sum(map(float, sys.argv[1:])))
except:
print("An error has occurred")
Result:
C:\Users\simone> python sum.py 10 4 6 3
23

This handles simple switches, value switches with optional alternative flags.
import sys
# [IN] argv - array of args
# [IN] switch - switch to seek
# [IN] val - expecting value
# [IN] alt - switch alternative
# returns value or True if val not expected
def parse_cmd(argv,switch,val=None,alt=None):
for idx, x in enumerate(argv):
if x == switch or x == alt:
if val:
if len(argv) > (idx+1):
if not argv[idx+1].startswith('-'):
return argv[idx+1]
else:
return True
//expecting a value for -i
i = parse_cmd(sys.argv[1:],"-i", True, "--input")
//no value needed for -p
p = parse_cmd(sys.argv[1:],"-p")

Several of our biotechnology clients have posed these two questions recently:
How can we execute a Python script as a command?
How can we pass input values to a Python script when it is executed as a command?
I have included a Python script below which I believe answers both questions. Let's assume the following Python script is saved in the file test.py:
#
#----------------------------------------------------------------------
#
# file name: test.py
#
# input values: data - location of data to be processed
# date - date data were delivered for processing
# study - name of the study where data originated
# logs - location where log files should be written
#
# macOS usage:
#
# python3 test.py "/Users/lawrence/data" "20220518" "XYZ123" "/Users/lawrence/logs"
#
# Windows usage:
#
# python test.py "D:\data" "20220518" "XYZ123" "D:\logs"
#
#----------------------------------------------------------------------
#
# import needed modules...
#
import sys
import datetime
def main(argv):
#
# print message that process is starting...
#
print("test process starting at", datetime.datetime.now().strftime("%Y%m%d %H:%M"))
#
# set local values from input values...
#
data = sys.argv[1]
date = sys.argv[2]
study = sys.argv[3]
logs = sys.argv[4]
#
# print input arguments...
#
print("data value is", data)
print("date value is", date)
print("study value is", study)
print("logs value is", logs)
#
# print message that process is ending...
#
print("test process ending at", datetime.datetime.now().strftime("%Y%m%d %H:%M"))
#
# call main() to begin processing...
#
if __name__ == '__main__':
main(sys.argv)
The script can be executed on a macOS computer in a Terminal shell as shown below and the results will be printed to standard output (be sure the current directory includes the test.py file):
$ python3 test.py "/Users/lawrence/data" "20220518" "XYZ123" "/Users/lawrence/logs"
test process starting at 20220518 16:51
data value is /Users/lawrence/data
date value is 20220518
study value is XYZ123
logs value is /Users/lawrence/logs
test process ending at 20220518 16:51
The script can also be executed on a Windows computer in a Command Prompt as shown below and the results will be printed to standard output (be sure the current directory includes the test.py file):
D:\scripts>python test.py "D:\data" "20220518" "XYZ123" "D:\logs"
test process starting at 20220518 17:20
data value is D:\data
date value is 20220518
study value is XYZ123
logs value is D:\logs
test process ending at 20220518 17:20
This script answers both questions posed above and is a good starting point for developing scripts that will be executed as commands with input values.

Reason for the new answer:
Existing answers specify multiple options.
Standard option is to use argparse, a few answers provided examples from the documentation, and one answer suggested the advantage of it. But all fail to explain the answer adequately/clearly to the actual question by OP, at least for newbies.
An example of argparse:
import argparse
def load_config(conf_file):
pass
if __name__ == '__main__':
parser = argparse.ArgumentParser()
//Specifies one argument from the command line
//You can have any number of arguments like this
parser.add_argument("conf_file", help="configuration file for the application")
args = parser.parse_args()
config = load_config(args.conf_file)
Above program expects a config file as an argument. If you provide it, it will execute happily. If not, it will print the following
usage: test.py [-h] conf_file
test.py: error: the following arguments are required: conf_file
You can have the option to specify if the argument is optional.
You can specify the expected type for the argument using type key
parser.add_argument("age", type=int, help="age of the person")
You can specify default value for the arguments by specifying default key
This document will help you to understand it to an extent.

argparse - Build back command line

In Python, how can I parse the command line, edit the resulting parsed arguments object and generate a valid command line back with the updated values?
For instance, I would like python cmd.py --foo=bar --step=0 call python cmd.py --foo=bar --step=1 with all the original --foo=bar arguments, potentially without extra arguments added when default value is used.
Is it possible with argparse?

You can use argparse to parse the command-line arguments, and then modify those as desired. At the moment however, argparse lacks the functionality to work in reverse and convert those values back into a command-line string. There is however a package for doing precisely that, called argunparse. For example, the following code in cmd.py
import sys
import argparse
import argunparse
parser = argparse.ArgumentParser()
unparser = argunparse.ArgumentUnparser()
parser.add_argument('--foo')
parser.add_argument('--step', type=int)
kwargs = vars(parser.parse_args())
kwargs['step'] += 1
prefix = f'python {sys.argv[0]} '
arg_string = unparser.unparse(**kwargs)
print(prefix + arg_string)
will print the desired command line:
python cmd.py --foo=bar --step=1

argparse is clearly designed to go one way, from sys.argv to the args namespace. No thought has been given to preserving information that would let you map things back the other way, much less do the mapping itself.
In general, multiple sys.argv could produce the same args. You could, for example, have several arguments that have the same dest. Or you can repeat 'optionals'. But for a restricted 'parser' setup there may be enough information to recreate a usable argv.
Try something like:
parser = argparser.ArgumentParser()
arg1 = parser.add_argument('--foo', default='default')
arg2 = parser.add_argument('bar', nargs=2)
and then examine the arg1 and arg2 objects. They contain all the information that you supplied to the add_argument method. Of course you could have defined those values in your own data structures before hand, e.g.
{'option_string':'--foo', 'default':'default'}
{'dest':'bar', 'nargs':2}
and used those as input to add_argument.
While the parser may have enough information to recreate a useable sys.argv, you have to figure out how to do that yourself.
default=argparse.SUPPRESS may be handy. It keeps the parser from adding a default entry to the namespace. So if the option isn't used, it won't appear in the namespace.

This isn't possible in any easy way that I know of, then again I've never needed to do this.
But with the lack of information in the question in regards to how you call your script, I'll assume the following:
python test.py cmd --foo=bar --step=0
And what you could do is do:
from sys import argv
for index in range(1, len(argv)): # the first object is the script itself
if '=' in argv[index]:
param, value = argv[index].split('=', 1)
if param == '--step':
value = '1'
argv[index] = param + '=' + value
print(argv)
Note that this is very specific to --step and may be what you've already thought of and just wanted a "better way", but again, I don't think there is.

depending on the scope, this works at the same module at least:
pprint(argparse._sys.argv)

Per the other answers, rebuilding is imperfect, but if you aren't doing anything too fancy and are okay with imperfect, something like this could work for you as a starting point:
def unparse_args(parser, parsed_args):
"""Unparse argparsed args"""
positional_args = [action.dest
for action in parser._actions
if not action.option_strings]
optionals = []
positionals = []
for key, value in vars(parsed_args).items():
if not value:
# none and false flags go away
continue
elif key in positional_args:
positionals.append(value)
elif value is True:
optionals.append(f"--{key}")
else:
optionals.append(f"--{key}={value}")
return " ".join(optionals + positionals)
Here's an example using this with a git clone clone:
parser = argparse.ArgumentParser(description='A sample git clone wrapper')
# options
parser.add_argument("-v", "--verbose", action="store_true",
help="be more verbose")
parser.add_argument("-q", "--quiet", action="store_true",
help="be more quiet")
parser.add_argument("--recurse-submodules", nargs='?',
help="initialize submodules in the clone")
parser.add_argument("--recursive", nargs='?',
help="alias of --recurse-submodules")
parser.add_argument("-b", "--branch",
help=" checkout <branch> instead of the remote's HEAD")
parser.add_argument("--depth", type=int,
help="create a shallow clone of that depth")
parser.add_argument("--shallow-submodules", action="store_true",
help="any cloned submodules will be shallow")
# positional
parser.add_argument("repo", help="The git repo to clone")
parser.add_argument("dir", nargs='?',help="The location to clone the repo")
# make a fake call to your git clone clone and parse the args
cmdargs = ["--depth=1", "-q", "ohmyzsh/ohmyzsh"]
parsedargs = parser.parse_args(cmdargs)
# now unparse them
unparsed = unparse_args(parser, parsedargs)
print(unparsed)

how to modify nargs( of optparse-add_option) from User input(raw_input)?

This Question is continuation of old question #: how to access nargs of optparse-add_action?
As that question was answered for what it was in-tented.
Brief:
Suppose if I am using add_option utility like below:
parser.add_option('-c','--categories', dest='Categories', nargs=4 )
Is there a way to modify nargs of add_option() from user input using raw_input.?
EDIT:
I will give a clear difference between my "previous question need" and "this question need".
First question case:
My script will ask for user inputs if user has provided no inputs, i.e.,He has just run
#./commandparser.py
Second Question case requirement is:
when i run my script ./commandparser.py -c abc bac cad
it throws error: commandparser.py: error: -c option requires 4 arguments and exit the script.
Instead of throwing error and exit the script. I want some mechanism so that it asks user to input remaining arguments i.e., 4th argument without exiting the script.

Are you, by any chance, trying to accept a variable number of values for this option? That is, use the 'rawinput' to set nargs, which is then used to parse the command line?
The optparse documentation has an example of using a custom callback to handle a variable number of values:
https://docs.python.org/2/library/optparse.html#callback-example-6-variable-arguments
argparse, on the other hand, does allow a variable number of values, with nargs values like '?' (0 or 1), '+' (1 or more), '*' (0 or more).
Since I'm more conversant with argparse I'll sketch out an interactive script to handle your revised requirement:
import argparse
parser = argparse.ArgumentParser(prog='PROG')
parser.add_argument('-c', '--categories', nargs='+', help='4 categories', default=[])
args = parser.parse_args()
print(args)
categories = args.categories
while len(categories)<4:
print(parser.format_usage())
x = raw_input('enter %s categories: '%(4-len(categories))).split()
categories.extend(x)
print('categories', categories)
If 'categories' are the only arguments, you could replace all of the argparse stuff (or optparse) with categories = sys.argv[1:], or [2:] if you still expect the '-c' flag.
Or using optparse (adapted from the docs example for variable length callback):
def vararg_callback(option, opt_str, value, parser):
value = []
for arg in parser.rargs:
# stop on --foo like options
if arg[:2] == "--" and len(arg) > 2:
break
# stop on -a (ignore the floats issue)
if arg[:1] == "-" and len(arg) > 1:
break
value.append(arg)
del parser.rargs[:len(value)]
setattr(parser.values, option.dest, value)
def use_opt():
import optparse
parser = optparse.OptionParser()
parser.add_option('-c','--categories', dest='categories', action="callback", callback=vararg_callback)
(options, args) = parser.parse_args()
print options, args
return options, args, parser
args, rest, parser = use_opt()

How to define a mutually exclusive group of two positional arguments?

I would like to use argparse to make some code to be used in the following two ways:
./tester.py all
./tester.py name someprocess
i.e. either all is specified OR name with some additional string.
I have tried to implement as follows:
import argparse
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group()
group.add_argument('all', action='store_true', \
help = "Stops all processes")
group.add_argument('name', \
help = "Stops the named process")
print parser.parse_args()
which gives me an error
ValueError: mutually exclusive arguments must be optional
Any idea how to do it right? I also would like to avoid sub parsers in this case.

The question is a year old, but since all the answers suggest a different syntax, I'll give something closer to the OP.
First, the problems with the OP code:
A positional store_true does not make sense (even if it is allowed). It requires no arguments, so it is always True. Giving an 'all' will produce error: unrecognized arguments: all.
The other argument takes one value and assigns it to the name attribute. It does not accept an additional process value.
Regarding the mutually_exclusive_group. That error message is raised even before parse_args. For such a group to make sense, all the alternatives have to be optional. That means either having a -- flag, or be a postional with nargs equal to ? or *. And doesn't make sense to have more than one such positional in the group.
The simplest alternative to using --all and --name, would be something like this:
p=argparse.ArgumentParser()
p.add_argument('mode', choices=['all','name'])
p.add_argument('process',nargs='?')
def foo(args):
if args.mode == 'all' and args.process:
pass # can ignore the process value or raise a error
if args.mode == 'name' and args.process is None:
p.error('name mode requires a process')
args = p.parse_args()
foo(args) # now test the namespace for correct `process` argument.
Accepted namespaces would look like:
Namespace(mode='name', process='process1')
Namespace(mode='all', process=None)
choices imitates the behavior of a subparsers argument. Doing your own tests after parse_args is often simpler than making argparse do something special.

import argparse
parser = argparse.ArgumentParser()
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument('-a','--all', action='store_true', \
help = "Stops all processes")
group.add_argument('-n','--name', \
help = "Stops the named process")
print parser.parse_args()
./tester.py -h
usage: zx.py [-h] (-a | -n NAME)
optional arguments:
-h, --help show this help message and exit
-a, --all Stops all processes
-n NAME, --name NAME Stops the named process

"OR name with some additional string."
positional argument cannot take additional string
I think the best solution for you is (named test.py):
import argparse
p = argparse.ArgumentParser()
meg = p.add_mutually_exclusive_group()
meg.add_argument('-a', '--all', action='store_true', default=None)
meg.add_argument('-n', '--name', nargs='+')
print p.parse_args([])
print p.parse_args(['-a'])
print p.parse_args('--name process'.split())
print p.parse_args('--name process1 process2'.split())
print p.parse_args('--all --name process1'.split())
$ python test.py
Namespace(all=None, name=None)
Namespace(all=True, name=None)
Namespace(all=None, name=['process'])
Namespace(all=None, name=['process1', 'process2'])
usage: t2.py [-h] [-a | -n NAME [NAME ...]]
t2.py: error: argument -n/--name: not allowed with argument -a/--all

I would agree that this looks exactly like a sub-parser problem, and that if you don't want to make it an optional argument by using --all and --name, one suggestion from me would be just to ignore the all and name altogether, and use the following semantics:
If tester.py is called without any arguments, stop all process.
If tester.py is called with some arguments, stop only those processes.
Which can be done using:
import argparse, sys
parser = argparse.ArgumentParser()
parser.add_argument('processes', nargs='*')
parsed = parser.parse(sys.argv[1:])
print parsed
which will behave as follows:
$ python tester.py
Namespace(processes=[])
$ python tester.py proc1
Namespace(processes=['proc1'])
Or, if you insist on your own syntax, you can create a custom class. And actually you're not having a "mutually exclusive group" case, since I assume if all is specified, you will ignore the rest of the arguments (even when name is one of the other arguments), and when name is specified, anything else after that will be regarded as processes' name.
import argparse
import sys
class AllOrName(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
if len(values)==0:
raise argparse.ArgumentError(self, 'too few arguments')
if values[0]=='all':
setattr(namespace, 'all', True)
elif values[0]=='name':
if len(values)==1:
raise argparse.ArgumentError(self, 'please specify at least one process name')
setattr(namespace, 'name', values[1:])
else:
raise argparse.ArgumentError(self, 'only "all" or "name" should be specified')
parser = argparse.ArgumentParser()
parser.add_argument('processes', nargs='*', action=AllOrName)
parsed = parser.parse_args(sys.argv[1:])
print parsed
with the following behaviour:
$ python argparse_test.py name
usage: argparse_test.py [-h] [processes [processes ...]]
argparse_test.py: error: argument processes: please specify at least one process name
$ python argparse_test.py name proc1
Namespace(name=['proc1'], processes=None)
$ python argparse_test.py all
Namespace(all=True, processes=None)
$ python argparse_test.py host
usage: argparse_test.py [-h] [processes [processes ...]]
argparse_test.py: error: argument processes: only "all" or "name" should be specified
$ python argparse_test.py
usage: argparse_test.py [-h] [processes [processes ...]]
argparse_test.py: error: argument processes: too few arguments

This is probably what you're looking for:
group.add_argument('--all', dest=is_all, action='store_true')
group.add_argument('--name', dest=names, nargs='+')
Passing --name will then require at list one value and store them as a list.