I'm in the works of transferring an open source project's CLI from argparse to click. Currently the library allows for the following CLI usage patterns:
manim file.py Scene1 Scene2 -p -ql
but also for subcommands (some with subcommands of their own), like so:
manim plugins --list
manim cfg write --level cwd
... and more.
Thus, the manim keyword is both a way to access subcommands, and more importantly, a command itself with a positional arg (the python file). In argparse, determining if the text following the manim keyword is a subcommand, or positional arg, is a matter of reading sys.argv[2] and determining whether this matches one of the available subcommands, else it's a positional argument. The result is a lot of effort to add subcommands and lots of boilerplate code... hence the motion to transition to click; however, this particular usage pattern doesn't seem to come out of the box in click.
In other words, the following code:
#click.group()
#click.argument('file')
def main(file):
print('main',file)
#main.command()
#click.option('--opt')
def subcommand1(opt):
print("sub1",opt)
presents two issues:
requires the main Group's positional argument to be satisfied in order to run any subcommands (i.e. main file.py subcommand1 and not main subcommand1)
it won't execute the Group + argument without a subcommand (i.e. main file.py is not possible without specifying subommand1 to the Group).
After further research, I've solved the second issue using the click Group parameter, invoke_without_command
#click.group(invoke_without_command=True)
#click.argument('file')
def main(file):
print('main',file)
#main.command()
#click.option('--opt')
def subcommand1(opt):
print("sub1",opt)
but I still face issue with the 1st point -- the general ability to opt out of using the positional arg and instead invoke the subcommand. This StackOverflow article answered by Stephen Raunch seems relevant; however, it's only for the help option and not the the more general use case of arguments and options.
How can I allow for arbitrary nesting of subcommands, like a Group, but still provide the capability to execute that Group like a #click.command() with positional arguments? If this isn't a recommended pattern, what would you recommend instead?
In doing some more research on the topic, I came across this stackoverflow article which led to the creation of this custom group:
import click
from click.decorators import pass_context
class SkipArg(click.Group):
def parse_args(self, ctx, args):
if args[0] in self.commands:
if len(args) == 1 or args[1] not in self.commands:
# This condition needs updating for multiple positional arguments
args.insert(0, '')
super(SkipArg, self).parse_args(ctx, args)
#click.group(cls=SkipArg, invoke_without_command=True)
#click.argument('file')
#pass_context
def main(ctx, file):
print('main',ctx,file)
#main.command()
#click.option('--opt')
def subcommand1(opt):
print("sub1",opt)
This code meets the criteria of using click to have a command with positional arguments nested with more subcommands. The logic of the SkipArg will have to be updated for multiple positional arguments in the case of where I'll be using click, but is enough as a demonstration.
How it meets the criteria:
The main Group functions as a command since it is executable using invoke_without_command.
The positional argument is not required to execute subcommand1 because the class SkipArg inserts '' into the front of the arguments list. This is a bit of hard coded logic that applies empty filler text for the [FILE] positional argument when the zeroth argument matches one of the subcommands.
Related
Is there a way to group multiple commands, each with their own different parameters under a single function.
At first glance, a MultiCommand or Group might seem like a natural way of doing what I'd like, i.e. have a single main command act as the Group (and pass the invoke_without_command=True flag) then nest auxiliary Commands beneath it as subcommands. However this doesn't quite have the behavior that I'd want, since I'd like all the options from all commands to be able to be specified without explicitly invoking a subcommand. Additionally, using a Group would also not display the help text of the subcommands without invoking the subcommand on the command line as well.
I guess what I'd ideally like to have is a way to group multiple commands together without the nesting inherent to Click's Group API.
Sorry if this question might be somewhat general. Any help, ideas or tips that can point me in the right direction would be much appreciated.
Here's an outline of what I'd like (file name: cli_test.py):
import click
#click.command()
#click.option('--db-file', type=click.File(mode='r'))
def db_reader(db_file):
click.echo(db_file)
#click.command()
#click.option('--xval', type=float)
#click.option('--yval', type=float)
def get_vals(xval, yval):
return xval, yval
#click.command()
#click.option('--absolutize/--no-absolutize')
def flagger(absolutize):
click.echo(absolutize)
#click.command()
def cli_runner():
db = db_reader.main(standalone_mode=False)
vals = flagger.main(standalone_mode=False)
flag = flagger.main(standalone_mode=False)
if __name__ == '__main__':
cli_runner()
Essentially, I'd like a single command that can be run on the CLI (cli_runner in the above example), which would be able to take the parameters of all Click commands called within it, and then dispatch the processing of them to the appropriate Command instance. However as it stands right now, if I were to invoke on the CLI: $ python cli_test.py --xval 4 I'd get the error Error: no such option: --xval. I have also tried playing around with the pass_context and then ctx.invoke() approach, but the same problem exists.
I suppose I could pass parameters from all contained commands into cli_runner, but that would defeat the purpose of what I want to do, which is to ultimately have 3-4 modular "subcommands", that can then be grouped together in different combinations into larger interfaces that serve slightly different use cases.
I'm trying to create a terminal application a bit similar to cutechess-cli, which has options like the following:
-epdout FILE Save the end position of the games to FILE in FEN format.
-recover Restart crashed engines instead of stopping the match
-repeat [N] Play each opening twice (or N times). Unless the -noswap...
which can all be done with argparse in Python.
However it also has "named arguments" like the following:
-resign movecount=COUNT score=SCORE [twosided=VALUE]
Adjudicate the game as a loss if an engine's score is
at least SCORE centipawns below zero for at least COUNT
consecutive moves.
-sprt elo0=ELO0 elo1=ELO1 alpha=ALPHA beta=BETA
Use a Sequential Probability Ratio Test as a termination
criterion for the match. This option should only be used...
I can implement that with argparse using nargs='*' and then writing my own parser (maybe just regex). However that doesn't give nice documentation, and if argparse can already do something like this, I would rarther use the builtin approach.
Summary: Does argparse have a concept of named arguments similar to resign and sprt above? And if not, would the best approach be to do this manyally using nargs='*'?
You can use a custom type to split the values and use the metavar argument to give a better description for the value:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--arg', nargs='*', type=lambda text: text.split('=', maxsplit=1), metavar='PARAM-NAME=PARAM-VALUE', help='Some other parameters')
args = parser.parse_args()
args.arg = {k: v for k,v in args.arg}
Which produces:
usage: [-h] [--arg [PARAM-NAME=PARAM-VALUE [PARAM-NAME=PARAM-VALUE ...]]]
optional arguments:
-h, --help show this help message and exit
--arg [PARAM-NAME=PARAM-VALUE [PARAM-NAME=PARAM-VALUE ...]]
Some other parameters
If you wanted to you could avoid the "postprocessing" step to build the dictionary by using a custom Action type. But it seems an overkill to me in this case.
I am writing a python script.
It takes two arguments, a file, and the optional argument of a set of rules. The rules need to be formatted as a dictionary.
I am new to argparse and want to make this command line friendly.
If I leave the optional argument out and just type in the file, the script runs perfectly. If I add in a test dictionairy, it returns --
har_parser.py: error: unrecognized arguments:
I am not sure if it my misuse of the command line, which would be an easy fix if I need to change how I am passing in arguments.
Currently I am running the script as so...:
$ python myScript.py arg1 arg2
The other more likely scenario is that I wrote my function incorrectly, due to my novice experience with argparse.
Any direction would be appreciated.
def main():
parser = argparse.ArgumentParser()
parser.add_argument("file", nargs=1)
parser.add_argument("--rules")
args = parser.parse_args()
if args.rules:
print(parseHar(str(args.file[0]), args.rules[0]))
else:
print(parseHar(str(args.file[0])))
Question
When using the python click library to create command-line tools, is it possible to pass an unknown number of arguments to a function? I am thinking of something similar to the *args command.
Usecase
I am trying to build a wrapper for catkin and would like to use click for all the nice utilities it comes with. This application should perform some tasks, like changing into the root of the workspace, before calling catkin with the specified command. E.g. catkin build to compile the code.
The problem with this is, that I do not want to explicitly declare every possible compile flag that you can pass to catkin but rather want to only look out for the actual command and pass all the arguments directly to catkin.
Example
What I have found so far, is the possibility to define a last argument with the option nargs=-1 which will collect all succeeding arguments in this variable. This way you can collect for example a couple of file names. This is almost what I am looking for, except that it wont take flags beginning with a dash -. It will from an error saying Error: no such option: -a
#!/usr/bin/python
import click
import subprocess
#click.command()
#click.argument('action', type=click.STRING)
#click.option('--debug', is_flag=True, help='Build in debug mode.')
#click.argument('catkin_args', nargs=-1, type=click.STRING)
def main(action, debug, catkin_args):
""" A wrapper for catkin """
# Do something here ...
if debug:
# Do some more special things...
subprocess.call(["catkin"] + catkin_args)
According to the docs it's possible in click 4.0+; you just need to set the type of your catkin_args to click.UNPROCESSED.
Documentation has an example wrapping timeit like you describe you want to do with catkin.
I'm developing a management script that does a fairly large amount of work via a plethora of command-line options. The first few iterations of the script have used optparse to collect user input and then just run down the page, testing the value of each option in the appropriate order, and doing the action if necessary. This has resulted in a jungle of code that's really hard to read and maintain.
I'm looking for something better.
My hope is to have a system where I can write functions in more or less normal python fashion, and then when the script is run, have options (and help text) generated from my functions, parsed, and executed in the appropriate order. Additionally, I'd REALLY like to be able to build django-style sub-command interfaces, where myscript.py install works completely separately from myscript.py remove (separate options, help, etc.)
I've found simon willison's optfunc and it does a lot of this, but seems to just miss the mark — I want to write each OPTION as a function, rather than try to compress the whole option set into a huge string of options.
I imagine an architecture involving a set of classes for major functions, and each defined method of the class corresponding to a particular option in the command line. This structure provides the advantage of having each option reside near the functional code it modifies, easing maintenance. The thing I don't know quite how to deal with is the ordering of the commands, since the ordering of class methods is not deterministic.
Before I go reinventing the wheel: Are there any other existing bits of code that behave similarly? Other things that would be easy to modify? Asking the question has clarified my own thinking on what would be nice, but feedback on why this is a terrible idea, or how it should work would be welcome.
Don't waste time on "introspection".
Each "Command" or "Option" is an object with two sets of method functions or attributes.
Provide setup information to optparse.
Actually do the work.
Here's the superclass for all commands
class Command( object ):
name= "name"
def setup_opts( self, parser ):
"""Add any options to the parser that this command needs."""
pass
def execute( self, context, options, args ):
"""Execute the command in some application context with some options and args."""
raise NotImplemented
You create sublcasses for Install and Remove and every other command you need.
Your overall application looks something like this.
commands = [
Install(),
Remove(),
]
def main():
parser= optparse.OptionParser()
for c in commands:
c.setup_opts( parser )
options, args = parser.parse()
command= None
for c in commands:
if c.name.startswith(args[0].lower()):
command= c
break
if command:
status= command.execute( context, options, args[1:] )
else:
logger.error( "Command %r is unknown", args[0] )
status= 2
sys.exit( status )
The WSGI library werkzeug provides Management Script Utilities which may do what you want, or at least give you a hint how to do the introspection yourself.
from werkzeug import script
# actions go here
def action_test():
"sample with no args"
pass
def action_foo(name=2, value="test"):
"do some foo"
pass
if __name__ == '__main__':
script.run()
Which will generate the following help message:
$ python /tmp/test.py --help
usage: test.py <action> [<options>]
test.py --help
actions:
foo:
do some foo
--name integer 2
--value string test
test:
sample with no args
An action is a function in the same module starting with "action_" which takes a number of arguments where every argument has a default. The type of the default value specifies the type of the argument.
Arguments can then be passed by position or using --name=value from the shell.