We Keep Coding

Parse variable number of commands with python argparse

I'm developing a command line tool with Python whose functionality is broken down into a number of sub-commands, and basically each one takes as arguments input and output files. The tricky part is that each command requires different number of parameters (some require no output file, some require several input files, etc).
Ideally, the interface would be called as:
./test.py ncinfo inputfile
Then, the parser would realise that the ncinfo command requires a single argument (if this does not fit the input command, it complains), and then it calls the function:
ncinfo(inputfile)
that does the actual job.
When the command requires more options, for instance
./test.py timmean inputfile outputfile
the parser would realise it, check that indeed the two arguments are given, and then then it calls:
timmean(inputfile, outputfile)
This scheme is ideally generalised for an arbitrary list of 1-argument commands, 2-argument commands, and so on.
However I'm struggling to get this behaviour with Python argparse. This is what I have so far:
#! /home/navarro/SOFTWARE/anadonda3/bin/python
import argparse
# create the top-level parser
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# create the parser for the "ncinfo" command
parser_1 = subparsers.add_parser('ncinfo', help='prints out basic netCDF strcuture')
parser_1.add_argument('filein', help='the input file')
# create the parser for the "timmean" command
parser_2 = subparsers.add_parser('timmean', help='calculates temporal mean and stores it in output file')
parser_2.add_argument('filein', help='the input file')
parser_2.add_argument('fileout', help='the output file')
# parse the argument lists
parser.parse_args()
print(parser.filein)
print(parser.fileout)
But this doesn't work as expected. First, when I call the script without arguments, I get no error message telling me which options I have. Second, when I try to run the program to use ncinfo, I get an error
./test.py ncinfo testfile
Traceback (most recent call last):
File "./test.py", line 21, in <module>
print(parser.filein)
AttributeError: 'ArgumentParser' object has no attribute 'filein'
What am I doing wrong that precludes me achieving the desired behaviour? Is the use of subparsers sensible in this context?
Bonus point: is there a way to generalise the definition of the commands, so that I do not need to add manually every single command? For instance, grouping all 1-argument commands into a list, and then define the parser within a loop. This sounds reasonable, but I don't know if it is possible. Otherwise, as the number of tools grows, the parser itself is going to become hard to maintain.

import argparse
import sys
SUB_COMMANDS = [
"ncinfo",
"timmean"
]
def ncinfo(args):
print("executing: ncinfo")
print(" inputfile: %s" % args.inputfile)
def timmean(args):
print("executing: timmean")
print(" inputfile: %s" % args.inputfile)
print(" outputfile: %s" % args.outputfile)
def add_parser(subcmd, subparsers):
if subcmd == "ncinfo":
parser = subparsers.add_parser("ncinfo")
parser.add_argument("inputfile", metavar="INPUT")
parser.set_defaults(func=ncinfo)
elif subcmd == "timmean":
parser = subparsers.add_parser("timmean")
parser.add_argument("inputfile", metavar="INPUT")
parser.add_argument("outputfile", metavar="OUTPUT")
parser.set_defaults(func=timmean)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('-o', '--common-option', action='store_true')
subparsers = parser.add_subparsers(help="sub-commands")
for cmd in SUB_COMMANDS:
add_parser(cmd, subparsers)
args = parser.parse_args(sys.argv[1:])
if args.common_option:
print("common option is active")
try:
args.func(args)
except AttributeError:
parser.error("too few arguments")
Some usage examples:
$ python test.py --help
usage: test.py [-h] [-o] {ncinfo,timmean} ...
positional arguments:
{ncinfo,timmean} sub-commands
optional arguments:
-h, --help show this help message and exit
-o, --common-option
$ python test.py ncinfo --help
usage: test.py ncinfo [-h] INPUT
positional arguments:
INPUT
optional arguments:
-h, --help show this help message and exit
$ python test.py timmean --help
usage: test.py timmean [-h] INPUT OUTPUT
positional arguments:
INPUT
OUTPUT
optional arguments:
-h, --help show this help message and exit
$ python test.py -o ncinfo foo
common option is active
executing: ncinfo
inputfile: foo
$ python test.py -o timmean foo bar
common option is active
executing: timmean
inputfile: foo
outputfile: bar

Displaying help through command line argument

I have a python program that I am running through command line arguments. I have used sys module.
Below is my test.py Python file where I am taking all the args:
if len(sys.argv) > 1:
files = sys.argv
get_input(files)
The get_input method is in another Python file where I have the options defined.
options = {
'--case1': case1,
'--case2': case2,
}
def get_input(arguments):
for file in arguments[1:]:
if file in options:
options[file]()
else:
invalid_input(file)
To run:
python test.py --case1 --case2
My intentions are that I want to show the user all the commands in case they want to read the docs for that.
They should be able to read all the commands like they usually are in all the package for reading help, python test.py --help . With this they should be able to look into all the commands they can run.
How do I do this?

One of the best quality a Python developer can be proud of is to use built-in libraries instead of custom ones. So let's use argparse:
import argparse
# define your command line arguments
parser = argparse.ArgumentParser(description='My application description')
parser.add_argument('--case1', help='It does something', action='store_true')
parser.add_argument('--case2', help='It does something else, I guess', action='store_true')
# parse command line arguments
args = parser.parse_args()
# Accessing arguments values
print('case1 ', args.case1)
print('case2 ', args.case2)
You can now use your cmd arguments like python myscript.py --case1
This comes with a default --help argument you can now use like: python myscript.py --help which will output:
usage: myscript.py [-h] [--case1] [--case2]
My application description
optional arguments:
-h, --help show this help message and exit
--case1 It does something
--case2 It does something else, I guess

Hi you can use option parser and add your options and related help information.
It has by default help option which shows all the available options which you have added.
The detailed document is here. And below is the example.
from optparse import OptionParser
parser = OptionParser()
parser.add_option("-f", "--file", dest="filename",
help="write report to FILE", metavar="FILE")
parser.add_option("-q", "--quiet",
action="store_false", dest="verbose", default=True,
help="don't print status messages to stdout")
(options, args) = parser.parse_args()

Python : extract parameters created with argparse

I have a file called simple_example.py, which consists of 2 functions:
# import the necessary packages
import argparse
class simple:
#staticmethod
def func1():
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--name", help="name of the user", default='host')
ap.add_argument('-num', '--number', required=True, help='choose a number')
args = vars(ap.parse_args())
# display a friendly message to the user
print("Hi there {}, it's nice to meet you! you chose {}".format(args['name'], args['age']))
#staticmethod
def func2():
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--name", help="name of the user", default='host')
ap.add_argument('-num', '--number', required=True, help='choose a number')
ap.add_argument("-g", "--greet", help="say greetings", default='hello')
args = vars(ap.parse_args())
# display a friendly message to the user
print("{} there {}, it's nice to meet you! you chose {}".format(args['greet'], args['name'], args['age']))
I'd like to be able to call either func1() or func2() from the command line, so, I created another file called pyrun.py from this link
# !/usr/bin/env python
# make executable in bash chmod +x PyRun
import sys
import inspect
import importlib
import os
if __name__ == "__main__":
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile(inspect.currentframe()))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
# get the second argument from the command line
methodname = sys.argv[1]
# split this into module, class and function name
modulename, classname, funcname = methodname.split(".")
# get pointers to the objects based on the string names
themodule = importlib.import_module(modulename)
theclass = getattr(themodule, classname)
thefunc = getattr(theclass, funcname)
# pass all the parameters from the third until the end of what the function needs & ignore the rest
args = inspect.getargspec(thefunc)
print(args)
However, args in ArgSpec(args=[], varargs=None, keywords=None, defaults=None) shows an empty list.
How can I extract the parameters from either func1 or func2?
Is there a better way to run either func1 or func2 from the command line?

You probably want to use sub-commands. Here is an implementation of your example using sub-commands.
import argparse
def func1(args):
print("Hi there {}, it is nice to meet you! You chose {}.".format(args.name, args.number))
def func2(args):
print("{} there {}, it is nice to meet you! You chose {}.".format(args.greet, args.name, args.number))
#
# The top-level parser
#
parser = argparse.ArgumentParser('top.py', description='An example sub-command implementation')
#
# General sub-command parser object
#
subparsers = parser.add_subparsers(help='sub-command help')
#
# Specific sub-command parsers
#
cmd1_parser = subparsers.add_parser('cmd1', help='The first sub-command')
cmd2_parser = subparsers.add_parser('cmd2', help='The second sub-command')
#
# Assign the execution functions
#
cmd1_parser.set_defaults(func=func1)
cmd2_parser.set_defaults(func=func2)
#
# Add the common options
#
for cmd_parser in [cmd1_parser, cmd2_parser]:
cmd_parser.add_argument('-n', '--name', default='host', help='Name of the user')
cmd_parser.add_argument('-num', '--number', required=True, help='Number to report')
#
# Add command-specific options
#
cmd2_parser.add_argument('-g', '--greet', default='hello', help='Greeting to use')
#
# Parse the arguments
#
args = parser.parse_args()
#
# Invoke the function
#
args.func(args)
Example output:
$ python ./top.py cmd1 -n Mark -num 3
Hi there Mark, it is nice to meet you! You chose 3.
$ python ./top.py cmd2 -n Bob -num 7 -g Hello
Hello there Bob, it is nice to meet you! You chose 7.
And, of course, the help functions work for each of the sub-commands.
$ python ./top.py cmd2 -h
usage: top.py cmd2 [-h] [-n NAME] -num NUMBER [-g GREET]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME Name of the user
-num NUMBER, --number NUMBER
Number to report
-g GREET, --greet GREET
Greeting to use

If I put your first block of code in a file, I can import it into a ipython session and run your 2 functions:
In [2]: import stack49311085 as app
In [3]: app.simple
Out[3]: stack49311085.simple
ipython tab expansion (which uses some form of inspect) shows me that the module has a simple class, and the class itself has two static functions.
I can call func1, and get an argparse error message:
In [4]: app.simple.func1()
usage: ipython3 [-h] [-n NAME] -num NUMBER
ipython3: error: the following arguments are required: -num/--number
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
Similarly for func2:
In [7]: app.simple.func2()
usage: ipython3 [-h] [-n NAME] -num NUMBER [-g GREET]
ipython3: error: the following arguments are required: -num/--number
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
parse_args as a default parses the sys.argv[1:] list, which obviouslly is not tailored to its requirements.
def foo(argv=None):
parser = ....
....
args = parse.parse_args(argv=argv)
return args
is a more useful wrapper. With this I can pass a test argv list, and get back the parsed Namespace. If I don't give it such a list, it will used the sys.argv default. When testing a parser I like to return and/or display the whole Namespace.
I haven't used inspect enough to try to figure out what you are trying to do with it, or how to correct it. You don't need inspect to run code in an imported module like this.
I can test your imported parser by modifying the sys.argv
In [8]: import sys
In [9]: sys.argv
Out[9]:
['/usr/local/bin/ipython3',
'--pylab',
'--nosep',
'--term-title',
'--InteractiveShellApp.pylab_import_all=False']
In [10]: sys.argv[1:] = ['-h']
In [11]: app.simple.func2()
usage: ipython3 [-h] [-n NAME] -num NUMBER [-g GREET]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME name of the user
-num NUMBER, --number NUMBER
choose a number
-g GREET, --greet GREET
say greetings
An exception has occurred, use %tb to see the full traceback.
SystemExit: 0
Or following the help:
In [12]: sys.argv[1:] = ['-num=42', '-nPaul', '-gHI']
In [13]: app.simple.func2()
...
---> 30 print("{} there {}, it's nice to meet you! you chose {}".format(args['greet'], args['name'], args['age']))
KeyError: 'age'
Oops, there's an error in your code. You ask for args['age'], but didn't define a parser argument with that name. That's part of why I like to print the args Namespace` - to make sure it is setting all the attributes that I expect.
Normally we don't use different parsers for different inputs. It's possible to do that based on your own test of sys.avgv[1], but keep in mind that that string will still be on sys.argv[1:] list that your parser(s) read. Instead write one parser that can handle the various styles of input. The subparser mentioned in the other answer is one option. Another is to base your action on the value of the args.greet attribute. If not used it will be the default value.

parse the cmdline option to be -option key=value type in argparse

I want to parse the cmdline option to be -option key=value type in argparse.
For example:
script.py -project prj1=rev1
Generally:
script.py -project prj1 --> OK
script.py -project=prj1 --> OK
script.py -project prj1=rev1 --> How to flag that argument value should be in str=str format.
script.py -project=prj1,prj2 --> How to flag that we need comma separated strings.
In the above, -project is the option. proj1=rev1 in this way I want the argument to be present. It should flag an error if it is not in the proper format and print the help message. I can use regular expression once I collect the project value using (\w)=(\w). If not in the above format can flag an error. But is there a way to filter out this and flag an error at parsing the cmdline arguments itself?

You can take advantage of this fact from the argparse documentation:
type= can take any callable that takes a single string argument and returns the converted value:
For example, to support the first format (--project prj1=rev1) you could do something like:
import os
import sys
import argparse
def handle_kv_string(val):
if '=' in val:
return val.split('=')
else:
raise argparse.ArgumentTypeError('Must specify k=v')
def parse_args():
p = argparse.ArgumentParser()
p.add_argument('--project',
type=handle_kv_string)
return p.parse_args()
def main():
args = parse_args()
print args
if __name__ == '__main__':
main()
This gets you, with valid arguments:
$ ./argtest --project foo=bar
Namespace(project=['foo', 'bar'])
And with invalid arguments:
$ ./argtest --project foo
usage: argtest.py [-h] [--project PROJECT]
argtest.py: error: argument --project: Must specify k=v
You could apply a similar solution to your second example.

Python optparse not seeing argument

I am trying to pass '-f nameoffile' to the program when I call it from the command line. I got this from the python sites documentation but when I pass '-f filename' or '--file=filename' it throws the error that I didnt pass enough arguments. If i pass -h the programs responds how it should and gives me the help. Any ideas? I imagine its something simple that I am overlooking. Any and all help is great, thanks, Justin.
[justin87#el-beasto-loco python]$ python openall.py -f chords.tar
Usage: openall.py [options] arg
openall.py: error: incorrect number of arguments
[justin87#el-beasto-loco python]$
#!/usr/bin/python
import tarfile
import os
import zipfile
from optparse import OptionParser
def check_tar(file):
if tarfile.is_tarfile(file):
return True
def open_tar(file):
try:
tar = tarfile.open(file)
tar.extractall()
tar.close()
except tarfile.ReadError:
print "File is somehow invalid or can not be handled by tarfile"
except tarfile.CompressionError:
print "Compression method is not supported or data cannot be decoded"
except tarfile.StreamError:
print "Is raised for the limitations that are typical for stream-like TarFile objects."
except tarfile.ExtractError:
print "Is raised for non-fatal errors when using TarFile.extract(), but only if TarFile.errorlevel== 2."
def check_zip(file):
if zipfile.is_zipfile(file):
return True
def open_zip(file):
try:
zip = zipfile.ZipFile(file)
zip.extractall()
zip.close()
#open the zip
print "GOT TO OPENING"
except zipfile.BadZipfile:
print "The error raised for bad ZIP files (old name: zipfile.error)."
except zipfile.LargeZipFile:
print "The error raised when a ZIP file would require ZIP64 functionality but that has not been enabled."
rules = ((check_tar, open_tar),
(check_zip, open_zip)
)
def checkall(file):
for checks, extracts in rules:
if checks(file):
return extracts(file)
def main():
usage = "usage: %prog [options] arg"
parser = OptionParser(usage)
parser.add_option("-f", "--file", dest="filename",
help="read data from FILENAME")
(options, args) = parser.parse_args()
if len(args) != 1:
parser.error("incorrect number of arguments")
file = options.filename
checkall(file)
if __name__ == '__main__':
main()

Your problem is probably the if len(args) != 1:. That is looking for an additional argument (i.e. not an option). If you remove that check and look at your options dictionary you should see {'filename': 'blah'}.

Your input filename isn't an option to the program, it's an argument:
def main():
usage = "Usage: %prog [options] FILE"
description = "Read data from FILE."
parser = OptionParser(usage, description=description)
(options, args) = parser.parse_args()
if len(args) != 1:
parser.error("incorrect number of arguments")
file = args[0]
checkall(file)
You can usually tell the difference because options generally have sensible defaults while arguments don't.

After parsing the options out of the argument list, you check that you were passed an argument. This is independent of the argument to -f. It sounds like you're just not passing this argument. Since you also don't actually use this argument, you should probably just remove the check on len(args).

You should set the 'action' attribute in the 'add_option()' method to 'store', this tells the optparse object to store the argument immediately following the option flag, though this is the default behavior. The value following the flag will then be stored in 'options.filename' and not in args. I also think that the
if len(args) != 1:
is also an issue, you will get the same message if len(args) is greater than or less than 1.