We Keep Coding

argparse telling me a store_true arg needs an argument all of a sudden

I'm using the argparse library and have a few boolean arguments created like this:
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Generate a batch of Line Protocol points of a specified shape")
parser.add_argument('--loop', action='store_true', help="If True, script runs in infinit loop; used with Telegraf `execd` input plugin")
args = parser.parse_args()
print(args)
This has worked for weeks with no issue and now all of a sudden I'm getting the error:
ipython3: error: argument --loop: expected one argument

If you have the following file (example.py):
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--loop', action='store_true', help="<help stuff")
args = parser.parse_args()
print(args)
you can execute it either with "plain" Python:
$ python example.py
Namespace(loop=False)
$ python example.py --loop
Namespace(loop=True)
or with ipython, where you need to separate ipython arguments with script arguments by --:
$ ipython3 example.py -- --loop
Namespace(loop=True)

Is this the full error message:
0814:~/mypy$ ipython3 --loop
usage: ipython3 [-h] [--debug] [--quiet] [--init] [--autoindent] [--no-autoindent] [--automagic]
[--no-automagic] [--pdb] [--no-pdb] [--pprint] [--no-pprint] [--color-info]
[--no-color-info] [--ignore-cwd] [--no-ignore-cwd] [--nosep] [--autoedit-syntax]
[--no-autoedit-syntax] [--simple-prompt] [--no-simple-prompt] [--banner] [--no-banner]
[--confirm-exit] [--no-confirm-exit] [--term-title] [--no-term-title] [--classic]
[--quick] [-i] [--profile-dir ProfileDir.location]
[--profile TerminalIPythonApp.profile] [--ipython-dir TerminalIPythonApp.ipython_dir]
[--log-level TerminalIPythonApp.log_level]
[--config TerminalIPythonApp.extra_config_file]
[--autocall TerminalInteractiveShell.autocall]
[--colors TerminalInteractiveShell.colors] [--logfile TerminalInteractiveShell.logfile]
[--logappend TerminalInteractiveShell.logappend] [-c TerminalIPythonApp.code_to_run]
[-m TerminalIPythonApp.module_to_run] [--ext TerminalIPythonApp.extra_extensions]
[--gui TerminalIPythonApp.gui] [--pylab [TerminalIPythonApp.pylab]]
[--matplotlib [TerminalIPythonApp.matplotlib]]
[--cache-size TerminalInteractiveShell.cache_size]
[extra_args [extra_args ...]]
ipython3: error: argument --loop: expected one argument
1240:~/mypy$

Parse variable number of commands with python argparse

I'm developing a command line tool with Python whose functionality is broken down into a number of sub-commands, and basically each one takes as arguments input and output files. The tricky part is that each command requires different number of parameters (some require no output file, some require several input files, etc).
Ideally, the interface would be called as:
./test.py ncinfo inputfile
Then, the parser would realise that the ncinfo command requires a single argument (if this does not fit the input command, it complains), and then it calls the function:
ncinfo(inputfile)
that does the actual job.
When the command requires more options, for instance
./test.py timmean inputfile outputfile
the parser would realise it, check that indeed the two arguments are given, and then then it calls:
timmean(inputfile, outputfile)
This scheme is ideally generalised for an arbitrary list of 1-argument commands, 2-argument commands, and so on.
However I'm struggling to get this behaviour with Python argparse. This is what I have so far:
#! /home/navarro/SOFTWARE/anadonda3/bin/python
import argparse
# create the top-level parser
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# create the parser for the "ncinfo" command
parser_1 = subparsers.add_parser('ncinfo', help='prints out basic netCDF strcuture')
parser_1.add_argument('filein', help='the input file')
# create the parser for the "timmean" command
parser_2 = subparsers.add_parser('timmean', help='calculates temporal mean and stores it in output file')
parser_2.add_argument('filein', help='the input file')
parser_2.add_argument('fileout', help='the output file')
# parse the argument lists
parser.parse_args()
print(parser.filein)
print(parser.fileout)
But this doesn't work as expected. First, when I call the script without arguments, I get no error message telling me which options I have. Second, when I try to run the program to use ncinfo, I get an error
./test.py ncinfo testfile
Traceback (most recent call last):
File "./test.py", line 21, in <module>
print(parser.filein)
AttributeError: 'ArgumentParser' object has no attribute 'filein'
What am I doing wrong that precludes me achieving the desired behaviour? Is the use of subparsers sensible in this context?
Bonus point: is there a way to generalise the definition of the commands, so that I do not need to add manually every single command? For instance, grouping all 1-argument commands into a list, and then define the parser within a loop. This sounds reasonable, but I don't know if it is possible. Otherwise, as the number of tools grows, the parser itself is going to become hard to maintain.

import argparse
import sys
SUB_COMMANDS = [
"ncinfo",
"timmean"
]
def ncinfo(args):
print("executing: ncinfo")
print(" inputfile: %s" % args.inputfile)
def timmean(args):
print("executing: timmean")
print(" inputfile: %s" % args.inputfile)
print(" outputfile: %s" % args.outputfile)
def add_parser(subcmd, subparsers):
if subcmd == "ncinfo":
parser = subparsers.add_parser("ncinfo")
parser.add_argument("inputfile", metavar="INPUT")
parser.set_defaults(func=ncinfo)
elif subcmd == "timmean":
parser = subparsers.add_parser("timmean")
parser.add_argument("inputfile", metavar="INPUT")
parser.add_argument("outputfile", metavar="OUTPUT")
parser.set_defaults(func=timmean)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('-o', '--common-option', action='store_true')
subparsers = parser.add_subparsers(help="sub-commands")
for cmd in SUB_COMMANDS:
add_parser(cmd, subparsers)
args = parser.parse_args(sys.argv[1:])
if args.common_option:
print("common option is active")
try:
args.func(args)
except AttributeError:
parser.error("too few arguments")
Some usage examples:
$ python test.py --help
usage: test.py [-h] [-o] {ncinfo,timmean} ...
positional arguments:
{ncinfo,timmean} sub-commands
optional arguments:
-h, --help show this help message and exit
-o, --common-option
$ python test.py ncinfo --help
usage: test.py ncinfo [-h] INPUT
positional arguments:
INPUT
optional arguments:
-h, --help show this help message and exit
$ python test.py timmean --help
usage: test.py timmean [-h] INPUT OUTPUT
positional arguments:
INPUT
OUTPUT
optional arguments:
-h, --help show this help message and exit
$ python test.py -o ncinfo foo
common option is active
executing: ncinfo
inputfile: foo
$ python test.py -o timmean foo bar
common option is active
executing: timmean
inputfile: foo
outputfile: bar

Python : extract parameters created with argparse

I have a file called simple_example.py, which consists of 2 functions:
# import the necessary packages
import argparse
class simple:
#staticmethod
def func1():
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--name", help="name of the user", default='host')
ap.add_argument('-num', '--number', required=True, help='choose a number')
args = vars(ap.parse_args())
# display a friendly message to the user
print("Hi there {}, it's nice to meet you! you chose {}".format(args['name'], args['age']))
#staticmethod
def func2():
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--name", help="name of the user", default='host')
ap.add_argument('-num', '--number', required=True, help='choose a number')
ap.add_argument("-g", "--greet", help="say greetings", default='hello')
args = vars(ap.parse_args())
# display a friendly message to the user
print("{} there {}, it's nice to meet you! you chose {}".format(args['greet'], args['name'], args['age']))
I'd like to be able to call either func1() or func2() from the command line, so, I created another file called pyrun.py from this link
# !/usr/bin/env python
# make executable in bash chmod +x PyRun
import sys
import inspect
import importlib
import os
if __name__ == "__main__":
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile(inspect.currentframe()))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
# get the second argument from the command line
methodname = sys.argv[1]
# split this into module, class and function name
modulename, classname, funcname = methodname.split(".")
# get pointers to the objects based on the string names
themodule = importlib.import_module(modulename)
theclass = getattr(themodule, classname)
thefunc = getattr(theclass, funcname)
# pass all the parameters from the third until the end of what the function needs & ignore the rest
args = inspect.getargspec(thefunc)
print(args)
However, args in ArgSpec(args=[], varargs=None, keywords=None, defaults=None) shows an empty list.
How can I extract the parameters from either func1 or func2?
Is there a better way to run either func1 or func2 from the command line?

You probably want to use sub-commands. Here is an implementation of your example using sub-commands.
import argparse
def func1(args):
print("Hi there {}, it is nice to meet you! You chose {}.".format(args.name, args.number))
def func2(args):
print("{} there {}, it is nice to meet you! You chose {}.".format(args.greet, args.name, args.number))
#
# The top-level parser
#
parser = argparse.ArgumentParser('top.py', description='An example sub-command implementation')
#
# General sub-command parser object
#
subparsers = parser.add_subparsers(help='sub-command help')
#
# Specific sub-command parsers
#
cmd1_parser = subparsers.add_parser('cmd1', help='The first sub-command')
cmd2_parser = subparsers.add_parser('cmd2', help='The second sub-command')
#
# Assign the execution functions
#
cmd1_parser.set_defaults(func=func1)
cmd2_parser.set_defaults(func=func2)
#
# Add the common options
#
for cmd_parser in [cmd1_parser, cmd2_parser]:
cmd_parser.add_argument('-n', '--name', default='host', help='Name of the user')
cmd_parser.add_argument('-num', '--number', required=True, help='Number to report')
#
# Add command-specific options
#
cmd2_parser.add_argument('-g', '--greet', default='hello', help='Greeting to use')
#
# Parse the arguments
#
args = parser.parse_args()
#
# Invoke the function
#
args.func(args)
Example output:
$ python ./top.py cmd1 -n Mark -num 3
Hi there Mark, it is nice to meet you! You chose 3.
$ python ./top.py cmd2 -n Bob -num 7 -g Hello
Hello there Bob, it is nice to meet you! You chose 7.
And, of course, the help functions work for each of the sub-commands.
$ python ./top.py cmd2 -h
usage: top.py cmd2 [-h] [-n NAME] -num NUMBER [-g GREET]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME Name of the user
-num NUMBER, --number NUMBER
Number to report
-g GREET, --greet GREET
Greeting to use

If I put your first block of code in a file, I can import it into a ipython session and run your 2 functions:
In [2]: import stack49311085 as app
In [3]: app.simple
Out[3]: stack49311085.simple
ipython tab expansion (which uses some form of inspect) shows me that the module has a simple class, and the class itself has two static functions.
I can call func1, and get an argparse error message:
In [4]: app.simple.func1()
usage: ipython3 [-h] [-n NAME] -num NUMBER
ipython3: error: the following arguments are required: -num/--number
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
Similarly for func2:
In [7]: app.simple.func2()
usage: ipython3 [-h] [-n NAME] -num NUMBER [-g GREET]
ipython3: error: the following arguments are required: -num/--number
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
parse_args as a default parses the sys.argv[1:] list, which obviouslly is not tailored to its requirements.
def foo(argv=None):
parser = ....
....
args = parse.parse_args(argv=argv)
return args
is a more useful wrapper. With this I can pass a test argv list, and get back the parsed Namespace. If I don't give it such a list, it will used the sys.argv default. When testing a parser I like to return and/or display the whole Namespace.
I haven't used inspect enough to try to figure out what you are trying to do with it, or how to correct it. You don't need inspect to run code in an imported module like this.
I can test your imported parser by modifying the sys.argv
In [8]: import sys
In [9]: sys.argv
Out[9]:
['/usr/local/bin/ipython3',
'--pylab',
'--nosep',
'--term-title',
'--InteractiveShellApp.pylab_import_all=False']
In [10]: sys.argv[1:] = ['-h']
In [11]: app.simple.func2()
usage: ipython3 [-h] [-n NAME] -num NUMBER [-g GREET]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME name of the user
-num NUMBER, --number NUMBER
choose a number
-g GREET, --greet GREET
say greetings
An exception has occurred, use %tb to see the full traceback.
SystemExit: 0
Or following the help:
In [12]: sys.argv[1:] = ['-num=42', '-nPaul', '-gHI']
In [13]: app.simple.func2()
...
---> 30 print("{} there {}, it's nice to meet you! you chose {}".format(args['greet'], args['name'], args['age']))
KeyError: 'age'
Oops, there's an error in your code. You ask for args['age'], but didn't define a parser argument with that name. That's part of why I like to print the args Namespace` - to make sure it is setting all the attributes that I expect.
Normally we don't use different parsers for different inputs. It's possible to do that based on your own test of sys.avgv[1], but keep in mind that that string will still be on sys.argv[1:] list that your parser(s) read. Instead write one parser that can handle the various styles of input. The subparser mentioned in the other answer is one option. Another is to base your action on the value of the args.greet attribute. If not used it will be the default value.

Flexible UNIX command line interface with Python

I was wondering how to create a flexible CLI interface with Python. So far I have come up with the following:
$ cat cat.py
#!/usr/bin/env python
from sys import stdin
from fileinput import input
from argparse import ArgumentParser, FileType
def main(args):
for line in input():
print line.strip()
if __name__ == "__main__":
parser = ArgumentParser()
parser.add_argument('FILE', nargs='?', type=FileType('r'), default=stdin)
main(parser.parse_args())
This handles both stdin and file input:
$ echo 'stdin test' | ./cat.py
stdin test
$ ./cat.py file
file test
The problem is it doesn't handle multiple input or no input the way I would like:
$ ./cat.py file file
usage: cat.py [-h] [FILE]
cat.py: error: unrecognized arguments: file
$ ./cat.py
For multiple inputs it should cat the file multiple times and for no input input should ideally have same the behaviour as -h:
$ ./cat.py -h
usage: cat.py [-h] [FILE]
positional arguments:
FILE
optional arguments:
-h, --help show this help message and exit
Any ideas on creating a flexible CLI interface with Python?

Use nargs='*' to allow for 0 or more arguments:
if __name__ == "__main__":
parser = ArgumentParser()
parser.add_argument('FILE', nargs='*', type=FileType('r'), default=stdin)
main(parser.parse_args())
The help output now is:
$ bin/python cat.py -h
usage: cat.py [-h] [FILE [FILE ...]]
positional arguments:
FILE
optional arguments:
-h, --help show this help message and exit
and when no arguments are given, stdout is used.
If you want to require at least one FILE argument, use nargs='+' instead, but then the default is ignored, so you may as well drop that:
if __name__ == "__main__":
parser = ArgumentParser()
parser.add_argument('FILE', nargs='+', type=FileType('r'))
main(parser.parse_args())
Now not specifying a command-line argument gives:
$ bin/python cat.py
usage: cat.py [-h] FILE [FILE ...]
cat.py: error: too few arguments
You can always specify stdin still by passing in - as an argument:
$ echo 'hello world!' | bin/python cat.py -
hello world!

A pretty good CLI interface the handles file input, standard input, no input, file output and inplace editing:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
def main(args, help):
'''
Simple line numbering program to demonstrate CLI interface
'''
if not (select.select([sys.stdin,],[],[],0.0)[0] or args.files):
help()
return
if args.output and args.output != '-':
sys.stdout = open(args.output, 'w')
try:
for i, line in enumerate(fileinput.input(args.files, inplace=args.inplace)):
print i + 1, line.strip()
except IOError:
sys.stderr.write("%s: No such file %s\n" %
(os.path.basename(__file__), fileinput.filename()))
if __name__ == "__main__":
import os, sys, select, argparse, fileinput
parser = argparse.ArgumentParser()
parser.add_argument('files', nargs='*', help='input files')
group = parser.add_mutually_exclusive_group()
group.add_argument('-i', '--inplace', action='store_true',
help='modify files inplace')
group.add_argument('-o', '--output',
help='output file. The default is stdout')
main(parser.parse_args(), parser.print_help)
The code simply emulates nl and numbers the lines but should serve as a good skeleton for many applications.

Argparse subparser: hide metavar in command listing

I'm using the Python argparse module for command line subcommands in my program. My code basically looks like this:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(title="subcommands", metavar="<command>")
subparser = subparsers.add_parser("this", help="do this")
subparser = subparsers.add_parser("that", help="do that")
parser.parse_args()
When running "python test.py --help" I would like to list the available subcommands. Currently I get this output:
usage: test.py [-h] <command> ...
optional arguments:
-h, --help show this help message and exit
subcommands:
<command>
this do this
that do that
Can I somehow remove the <command> line in the subcommands listing and still keep it in the usage line? I have tried to give help=argparse.SUPPRESS as argument to add_subparsers, but that just hides all the subcommands in the help output.

I solved it by adding a new HelpFormatter that just removes the line if formatting a PARSER action:
class SubcommandHelpFormatter(argparse.RawDescriptionHelpFormatter):
def _format_action(self, action):
parts = super(argparse.RawDescriptionHelpFormatter, self)._format_action(action)
if action.nargs == argparse.PARSER:
parts = "\n".join(parts.split("\n")[1:])
return parts