I am using Python 3.7 and Cement 3.0.4.
What I have is a base app with a controller and the controller has a command that takes a single optional argument. What I'm seeing is if I pass the command an invalid argument I get an invalid argument error as I expect but I am getting the "usage" output for the app itself rather than the command on the controller. Here is an example:
from cement import App, Controller, ex
class Base(Controller):
class Meta:
label = 'base'
#ex(help='example sub-command')
def cmd1(self):
print('Inside Base.cmd1()')
#ex(
arguments=[
(['-n', '--name'],
{'help': ''': The name you want printed out''',
'dest': 'name',
'required': False}),
],
help=' the help for cmd2.')
def cmd2(self):
print(self.app.pargs.name)
This app has a command called cmd2 and it takes an optional argument of -n which, as the help states, will be printed out. So if I do this:
with MyApp(argv=['cmd2', '-n', 'bob']) as app:
app.run()
I will the output:
bob
as expected. However if I pass an invalid argument to cmd2:
with MyApp(argv=['cmd2', '-a', 'bob']) as app:
app.run()
I get:
usage: myapp [-h] [-d] [-q] {cmd1,cmd2} ...
myapp: error: unrecognized arguments: -a bob
What I would like to see, instead of the usage for myapp, would be the usage for the cmd2 command, similar to if I did -h on the command:
with MyApp(argv=['cmd2', '-h']) as app:
app.run()
outputs
usage: myapp cmd2 [-h] [-n NAME]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME : The name you want printed out
I realize much of this is delegated to Argparse and is not handled by Cement. I've done some debugging and I'm seeing there are multiple ArgparseArgumentHandler classes nested. So in the case above there is an ArgparseArgumentHandler for myapp and it has in it's actions a SubParsersAction that has a choices field that has a map containing my two commands on the controller, cmd1 and cmd2 mapped to their own ArgparseArgumentHandler.
When the invalid argument is detected it is within the ArgparseArgumentHandler for myapp and thus it calls print_usage() on myapp rather than on the ArgparseArgumentHandler for the invoked command, cmd2.
My knowledge of Argparse is limited and I do find it a bit complex to navigate. The only workaround I can think of right now is subclassing ArgparseArgumentHandler, and overriding error() and trying to determine if the error is due to recognized arguments and if so try to find the parser for it.. something like this pseudocode:
class ArgparseArgumentOverride(ext_argparse.ArgparseArgumentHandler):
def error(self, message):
# determine if there are unknown args
args, argv = self.parse_known_args(self.original_arguments, self.original_namespace)
# we are in an error state and have unrecognized args
if argv:
controller_namespace = args.__controller_namespace__
for action in self._actions:
if action.choices is not None:
# we found an choice with our namespace
if action.choices[controller_namespace]:
command_parser= action.choices[controller_namespace]
# this should be the show_usage for the command
complete_command.print_usage(sys.stderr)
Again above is pseudocode and actually doing something like that would feel very fragile, error prone, and unpredictable. I know there has to be an better way to do this, I'm just not finding it. I've been digging through the docs and source for hours and still haven't found what I'm looking for. Could anyone tell me what I'm missing? Any advice on how to proceed here would be really appreciated. Thanks much!
I'm not familiar cement, but as you deduce the usage is generated by argparse:
In [235]: parser = argparse.ArgumentParser(prog='myapp')
In [236]: parser.add_argument('-d');
In [237]: sp = parser.add_subparsers(dest='cmd')
In [238]: sp1 = sp.add_parser('cmd1')
In [239]: sp2 = sp.add_parser('cmd2')
In [240]: sp2.add_argument('-n','--name');
In [241]: parser.parse_args('cmd2 -a'.split())
usage: myapp [-h] [-d D] {cmd1,cmd2} ...
myapp: error: unrecognized arguments: -a
If the error is tied to a sp2 argument, then the usage reflects that:
In [242]: parser.parse_args('cmd2 -n'.split())
usage: myapp cmd2 [-h] [-n NAME]
myapp cmd2: error: argument -n/--name: expected one argument
But unknown args are handled by the main parser. For example if we use parse_known_args instead:
In [245]: parser.parse_known_args('cmd2 foobar'.split())
Out[245]: (Namespace(cmd='cmd2', d=None, name=None), ['foobar'])
In [246]: parser.parse_known_args('cmd2 -a'.split())
Out[246]: (Namespace(cmd='cmd2', d=None, name=None), ['-a'])
The unknown args are returned as the extras list. parse_args returns the error instead of the extras.
In _SubParsers_Action.__call__, the relevant code is:
# parse all the remaining options into the namespace
# store any unrecognized options on the object, so that the top
# level parser can decide what to do with them
# In case this subparser defines new defaults, we parse them
# in a new namespace object and then update the original
# namespace for the relevant parts.
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)
if arg_strings:
vars(namespace).setdefault(_UNRECOGNIZED_ARGS_ATTR, [])
getattr(namespace, _UNRECOGNIZED_ARGS_ATTR).extend(arg_strings)
In theory you could construct an alternate _SubParsersAction class (or subclass), that handles that arg_strings differently. Changing the parse_known_args call to parse_args might be enough:
subnamespace = parser.parse_args(arg_strings, None)
Note that parse_args calls parse_known_args, and raises the error if there are unknowns:
def parse_args(self, args=None, namespace=None):
args, argv = self.parse_known_args(args, namespace)
if argv:
msg = _('unrecognized arguments: %s')
self.error(msg % ' '.join(argv))
return args
Related
I have the following code:
# Get parsed arguments
args = argparse.ArgumentParser(description=Messages().Get(112))
# Get the arguments for sinit
args.add_argument('init', help=Messages().Get(100), action="store_true")
args.add_argument('--url', default=None, help=Messages().Get(101))
# Get the arguments for schema import
args.add_argument('schema-import', help=Messages().Get(104), action="store_true")
args.add_argument('--file', default=None, help=Messages().Get(104))
The --url argument should only be used with init. For example: script.py schema-import --url should not be accepted but script.py schema-import --file should.
How to set arguments as child arguments?
As mentioned there might be a way to do this with argparse, I'm not sure, but in any event I find it more transparent to explicitly handle argument dependencies in application logic. This should achieve what I think you want:
import argparse
import sys
args = argparse.ArgumentParser(description="please only use the '--url' argument if you also use the 'init' argument")
# Going to use aliases here it's more conventional. So user can use, eg,
# -i or --init for the first argument.
args.add_argument('-i', '--init', help='init help', action="store_true")
args.add_argument('-u', '--url', default=None, help='init help')
args.add_argument('-s', '--schema-import', help='schema-import help', action="store_true")
args.add_argument('-f', '--file', help='file help')
def main():
arguments = args.parse_args()
if arguments.url and not arguments.init:
# You can log an output or raise an exception if you want
# But most likely a print statment is most appropriate
# Since you are interacting with the CLI.
print("You can only use the URL option with init. Exiting")
sys.exit(0)
print("gaurd clauses passed. Here is my code...")
...
if __name__ == "__main__":
main()
Test results (my file called temp.py):
$python temp.py -u https://www.google.com
You can only use the URL option with init. Exiting
$
$python temp.py -i -u https://www.google.com
Gaurd clauses passed. Here is my code...
Why bother with doing all the logic when you can let argparse do all the work for you?
Simply use Sub-commands to define different "branches" of execution:
args = argparse.ArgumentParser(description=Messages().Get(112))
subparsers = args.add_subparsers()
parser_init = subparsers.add_parser('init', help=Messages().Get(100))
parser_init.add_argument('--url', default=None, help=Messages().Get(101))
parser_schema = subparsers.add_parser('schema-import', help=Messages().Get(104))
parser_schema.add_argument('--file', default=None, help=Messages().Get(104))
And this will give you what you want without any logic added:
>>> print(args.parse_args(['schema-import', '--url', "some.url"]))
usage: args.py [-h] {init,schema-import} ...
args.py: error: unrecognized arguments: --url some.url
>>> print(args.parse_args(['schema-import', '--file', "some.file"]))
Namespace(file='some.file')
I'm developing a command line tool with Python whose functionality is broken down into a number of sub-commands, and basically each one takes as arguments input and output files. The tricky part is that each command requires different number of parameters (some require no output file, some require several input files, etc).
Ideally, the interface would be called as:
./test.py ncinfo inputfile
Then, the parser would realise that the ncinfo command requires a single argument (if this does not fit the input command, it complains), and then it calls the function:
ncinfo(inputfile)
that does the actual job.
When the command requires more options, for instance
./test.py timmean inputfile outputfile
the parser would realise it, check that indeed the two arguments are given, and then then it calls:
timmean(inputfile, outputfile)
This scheme is ideally generalised for an arbitrary list of 1-argument commands, 2-argument commands, and so on.
However I'm struggling to get this behaviour with Python argparse. This is what I have so far:
#! /home/navarro/SOFTWARE/anadonda3/bin/python
import argparse
# create the top-level parser
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# create the parser for the "ncinfo" command
parser_1 = subparsers.add_parser('ncinfo', help='prints out basic netCDF strcuture')
parser_1.add_argument('filein', help='the input file')
# create the parser for the "timmean" command
parser_2 = subparsers.add_parser('timmean', help='calculates temporal mean and stores it in output file')
parser_2.add_argument('filein', help='the input file')
parser_2.add_argument('fileout', help='the output file')
# parse the argument lists
parser.parse_args()
print(parser.filein)
print(parser.fileout)
But this doesn't work as expected. First, when I call the script without arguments, I get no error message telling me which options I have. Second, when I try to run the program to use ncinfo, I get an error
./test.py ncinfo testfile
Traceback (most recent call last):
File "./test.py", line 21, in <module>
print(parser.filein)
AttributeError: 'ArgumentParser' object has no attribute 'filein'
What am I doing wrong that precludes me achieving the desired behaviour? Is the use of subparsers sensible in this context?
Bonus point: is there a way to generalise the definition of the commands, so that I do not need to add manually every single command? For instance, grouping all 1-argument commands into a list, and then define the parser within a loop. This sounds reasonable, but I don't know if it is possible. Otherwise, as the number of tools grows, the parser itself is going to become hard to maintain.
import argparse
import sys
SUB_COMMANDS = [
"ncinfo",
"timmean"
]
def ncinfo(args):
print("executing: ncinfo")
print(" inputfile: %s" % args.inputfile)
def timmean(args):
print("executing: timmean")
print(" inputfile: %s" % args.inputfile)
print(" outputfile: %s" % args.outputfile)
def add_parser(subcmd, subparsers):
if subcmd == "ncinfo":
parser = subparsers.add_parser("ncinfo")
parser.add_argument("inputfile", metavar="INPUT")
parser.set_defaults(func=ncinfo)
elif subcmd == "timmean":
parser = subparsers.add_parser("timmean")
parser.add_argument("inputfile", metavar="INPUT")
parser.add_argument("outputfile", metavar="OUTPUT")
parser.set_defaults(func=timmean)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('-o', '--common-option', action='store_true')
subparsers = parser.add_subparsers(help="sub-commands")
for cmd in SUB_COMMANDS:
add_parser(cmd, subparsers)
args = parser.parse_args(sys.argv[1:])
if args.common_option:
print("common option is active")
try:
args.func(args)
except AttributeError:
parser.error("too few arguments")
Some usage examples:
$ python test.py --help
usage: test.py [-h] [-o] {ncinfo,timmean} ...
positional arguments:
{ncinfo,timmean} sub-commands
optional arguments:
-h, --help show this help message and exit
-o, --common-option
$ python test.py ncinfo --help
usage: test.py ncinfo [-h] INPUT
positional arguments:
INPUT
optional arguments:
-h, --help show this help message and exit
$ python test.py timmean --help
usage: test.py timmean [-h] INPUT OUTPUT
positional arguments:
INPUT
OUTPUT
optional arguments:
-h, --help show this help message and exit
$ python test.py -o ncinfo foo
common option is active
executing: ncinfo
inputfile: foo
$ python test.py -o timmean foo bar
common option is active
executing: timmean
inputfile: foo
outputfile: bar
I'd like to fetch all parameters passed to sys.argv that have the format
someprogram.py --someparameter 23 -p 42 -anotherparam somevalue.
Result I'm looking for is a namespace containing all the variables, already parsed.
To my understanding, argparse is expecting the user to define what are the parameters he is expecting.
Any way to do that with argparse ?
Thanks !
If you know that the parameters will always be given in the format --name value or -name value you can do it easily
class ArgHolder(object):
pass
name = None
for x in sys.argv[1:]:
if name:
setattr(ArgHolder, curname, x)
name = None
elif x.startswith('-'):
name = x.lstrip('-')
Now you will have collected all arguments in the class ArgHolder which is a namespace. You may also collect the values in an instance of ArgHolder
Using Click we can build such a command:
import click
#click.command(help="Your description here")
#click.option("--someparameter", type=int, help="Description of someparameter")
#click.option("--p", type=int, help="Description of p")
#click.option("--anotherparam", type=str, help="Description of anotherparam")
def command(someparameter, p, anotherparam):
pass
if __name__ == '__main__':
command()
And you will have a help option automatically:
$ python command.py --help
Usage: command.py [OPTIONS]
Your description here.
Options:
--someparameter INTEGER Description of someparameter.
...
--help Show this message and exit.
If you need to get all unknown arguments, you can get them from a context in such way:
#click.command(context_settings=dict(
ignore_unknown_options=True,
allow_extra_args=True,
), add_help_option=False)
#click.pass_context
def command(ctx):
click.echo(ctx.args)
I have a file called simple_example.py, which consists of 2 functions:
# import the necessary packages
import argparse
class simple:
#staticmethod
def func1():
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--name", help="name of the user", default='host')
ap.add_argument('-num', '--number', required=True, help='choose a number')
args = vars(ap.parse_args())
# display a friendly message to the user
print("Hi there {}, it's nice to meet you! you chose {}".format(args['name'], args['age']))
#staticmethod
def func2():
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-n", "--name", help="name of the user", default='host')
ap.add_argument('-num', '--number', required=True, help='choose a number')
ap.add_argument("-g", "--greet", help="say greetings", default='hello')
args = vars(ap.parse_args())
# display a friendly message to the user
print("{} there {}, it's nice to meet you! you chose {}".format(args['greet'], args['name'], args['age']))
I'd like to be able to call either func1() or func2() from the command line, so, I created another file called pyrun.py from this link
# !/usr/bin/env python
# make executable in bash chmod +x PyRun
import sys
import inspect
import importlib
import os
if __name__ == "__main__":
cmd_folder = os.path.realpath(os.path.abspath(os.path.split(inspect.getfile(inspect.currentframe()))[0]))
if cmd_folder not in sys.path:
sys.path.insert(0, cmd_folder)
# get the second argument from the command line
methodname = sys.argv[1]
# split this into module, class and function name
modulename, classname, funcname = methodname.split(".")
# get pointers to the objects based on the string names
themodule = importlib.import_module(modulename)
theclass = getattr(themodule, classname)
thefunc = getattr(theclass, funcname)
# pass all the parameters from the third until the end of what the function needs & ignore the rest
args = inspect.getargspec(thefunc)
print(args)
However, args in ArgSpec(args=[], varargs=None, keywords=None, defaults=None) shows an empty list.
How can I extract the parameters from either func1 or func2?
Is there a better way to run either func1 or func2 from the command line?
You probably want to use sub-commands. Here is an implementation of your example using sub-commands.
import argparse
def func1(args):
print("Hi there {}, it is nice to meet you! You chose {}.".format(args.name, args.number))
def func2(args):
print("{} there {}, it is nice to meet you! You chose {}.".format(args.greet, args.name, args.number))
#
# The top-level parser
#
parser = argparse.ArgumentParser('top.py', description='An example sub-command implementation')
#
# General sub-command parser object
#
subparsers = parser.add_subparsers(help='sub-command help')
#
# Specific sub-command parsers
#
cmd1_parser = subparsers.add_parser('cmd1', help='The first sub-command')
cmd2_parser = subparsers.add_parser('cmd2', help='The second sub-command')
#
# Assign the execution functions
#
cmd1_parser.set_defaults(func=func1)
cmd2_parser.set_defaults(func=func2)
#
# Add the common options
#
for cmd_parser in [cmd1_parser, cmd2_parser]:
cmd_parser.add_argument('-n', '--name', default='host', help='Name of the user')
cmd_parser.add_argument('-num', '--number', required=True, help='Number to report')
#
# Add command-specific options
#
cmd2_parser.add_argument('-g', '--greet', default='hello', help='Greeting to use')
#
# Parse the arguments
#
args = parser.parse_args()
#
# Invoke the function
#
args.func(args)
Example output:
$ python ./top.py cmd1 -n Mark -num 3
Hi there Mark, it is nice to meet you! You chose 3.
$ python ./top.py cmd2 -n Bob -num 7 -g Hello
Hello there Bob, it is nice to meet you! You chose 7.
And, of course, the help functions work for each of the sub-commands.
$ python ./top.py cmd2 -h
usage: top.py cmd2 [-h] [-n NAME] -num NUMBER [-g GREET]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME Name of the user
-num NUMBER, --number NUMBER
Number to report
-g GREET, --greet GREET
Greeting to use
If I put your first block of code in a file, I can import it into a ipython session and run your 2 functions:
In [2]: import stack49311085 as app
In [3]: app.simple
Out[3]: stack49311085.simple
ipython tab expansion (which uses some form of inspect) shows me that the module has a simple class, and the class itself has two static functions.
I can call func1, and get an argparse error message:
In [4]: app.simple.func1()
usage: ipython3 [-h] [-n NAME] -num NUMBER
ipython3: error: the following arguments are required: -num/--number
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
Similarly for func2:
In [7]: app.simple.func2()
usage: ipython3 [-h] [-n NAME] -num NUMBER [-g GREET]
ipython3: error: the following arguments are required: -num/--number
An exception has occurred, use %tb to see the full traceback.
SystemExit: 2
parse_args as a default parses the sys.argv[1:] list, which obviouslly is not tailored to its requirements.
def foo(argv=None):
parser = ....
....
args = parse.parse_args(argv=argv)
return args
is a more useful wrapper. With this I can pass a test argv list, and get back the parsed Namespace. If I don't give it such a list, it will used the sys.argv default. When testing a parser I like to return and/or display the whole Namespace.
I haven't used inspect enough to try to figure out what you are trying to do with it, or how to correct it. You don't need inspect to run code in an imported module like this.
I can test your imported parser by modifying the sys.argv
In [8]: import sys
In [9]: sys.argv
Out[9]:
['/usr/local/bin/ipython3',
'--pylab',
'--nosep',
'--term-title',
'--InteractiveShellApp.pylab_import_all=False']
In [10]: sys.argv[1:] = ['-h']
In [11]: app.simple.func2()
usage: ipython3 [-h] [-n NAME] -num NUMBER [-g GREET]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME name of the user
-num NUMBER, --number NUMBER
choose a number
-g GREET, --greet GREET
say greetings
An exception has occurred, use %tb to see the full traceback.
SystemExit: 0
Or following the help:
In [12]: sys.argv[1:] = ['-num=42', '-nPaul', '-gHI']
In [13]: app.simple.func2()
...
---> 30 print("{} there {}, it's nice to meet you! you chose {}".format(args['greet'], args['name'], args['age']))
KeyError: 'age'
Oops, there's an error in your code. You ask for args['age'], but didn't define a parser argument with that name. That's part of why I like to print the args Namespace` - to make sure it is setting all the attributes that I expect.
Normally we don't use different parsers for different inputs. It's possible to do that based on your own test of sys.avgv[1], but keep in mind that that string will still be on sys.argv[1:] list that your parser(s) read. Instead write one parser that can handle the various styles of input. The subparser mentioned in the other answer is one option. Another is to base your action on the value of the args.greet attribute. If not used it will be the default value.
I would like to be able to use the global options from an argparse.ArgumentParser object to override/augment the defaults values for a sub-command.
An example would be that displayed help reflects the global updates, i.e., for the following toy example:
import argparse
import os
import sys
# Global parser and options.
parser = argparse.ArgumentParser(
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument("--user",
dest="user",
default=os.environ.get("USER"),
help="Override the setting of the $USER variable.")
# Sub-command parser and options.
subparsers = parser.add_subparsers()
command = subparsers.add_parser(
"command",
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
command.add_argument("--config",
dest="config",
default="~%s/config" % os.environ.get("USER"),
help="The config file.")
options = parser.parse_args()
Ideally when when I run this in help mode I would get,
> python example.py --user thing command --help
usage: example.py command [-h] [--config CONFIG]
optional arguments:
-h, --help show this help message and exit
--config CONFIG The config file. (default: ~thing/config)
i.e., the config file path is user specific (thing). I realize that I could change the default config to be "~%(user)s/config" and then resolve this at run-time with the options namespace, however I would like the help to be more explicit.
I gather an alternative solution would be to try to parse the arguments once to obtain the global options, i.e.,
if "--help" in sys.argv:
# Parse the command minus the help to obtain the global options.
args = sys.argv[1:]
args.remove("--help")
# Update the defaults with the global options.
options = parser.parse_args(args)
command.set_defaults(config="~%s/config" % options.user)
# Re-parse the options.
parser.parse_args()
Though this seems somewhat hacky. Is there a better approach/solution?
After defining the global options, but before defining the subcommands, call parse_known_args to find out what the value of --user is. Then, finish defining the subparser commands, using the value of --user to define the default of --config, before
calling parse_args to parse all options on the command line.
This is a little different from your alternative, but it keeps all the command-line processing inside the argparse object.
(I trimmed down your code a little just to shorten it for this answer.)
import os
import argparse
# Global preparser and options.
preparser = argparse.ArgumentParser(add_help=False)
preparser.add_argument("--user", dest="user", default=os.environ.get("USER"))
# ****** NEW *******
options, _ = preparser.parse_known_args() # Ignore what we haven't defined yet
user = options.user # Use this to define the default of --config
parser = argparse.ArgumentParser(parents=[ preparser ], add_help=True,
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
# Sub-command parser and options.
subparsers = parser.add_subparsers()
command = subparsers.add_parser("command")
# ****** MODIFIED *******
command.add_argument("--config", dest="config", default="~%s/config" % (user,))
options = parser.parse_args()
Here's a solution that modifies the help line directly, at runtime. It could also modify os.environ or some other global as well, but I'll keep it simple. The key is assigning the action created by add_argument('--config'...) to a variable. help is just an attribute of that variable. You are free to modify that.
class FooAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
config.help = 'new user (%s)'% values
setattr(namespace, self.dest, values)
parser = argparse.ArgumentParser()
parser.add_argument('--user',action=FooAction)
sub = parser.add_subparsers()
cmd = sub.add_parser('cmd')
config = cmd.add_argument('--config',help='initial help')
config.help = 'default help' # alt way of setting 'help'
# print config # to see other attributes of the config action
args = parser.parse_args()
print args
Invoked with just cmd -h we get the default help
$ python stack12167228.py cmd -h
usage: stack12167228.py cmd [-h] [--config CONFIG]
optional arguments:
-h, --help show this help message and exit
--config CONFIG default help
Invoked with --user xxx cmd -h, the help is customized
$ python stack12167228.py --user xxx cmd -h
usage: stack12167228.py cmd [-h] [--config CONFIG]
optional arguments:
-h, --help show this help message and exit
--config CONFIG new user (xxx)