We Keep Coding

Categorize help output in Python Click

I am trying to figure out how to categorize commands in Click to resemble something close to the structure that kubectl uses in the way it separate commands out.
For instance, in a vanilla Click help output we have:
Usage: cli.py [OPTIONS] COMMAND [ARGS]...
A CLI tool
Options:
-h, --help Show this message and exit.
Commands:
command1 This is command1
command2 This is command2
command3 This is command3
command4 This is command4
Instead, what would be ideal for my usage is to have a separation to better categorize the command structure.
For instance:
Usage: cli.py [OPTIONS] COMMAND [ARGS]...
A CLI tool
Options:
-h, --help Show this message and exit.
Specific Commands for X:
command1 This is command1
command2 This is command2
Specific Commands for Y:
command3 This is command3
command4 This is command4
Global Commands:
version Shows version
I am using the latest Python and latest version of Click also for this.
I have tried looking into hooking into various Click classes to change this behaviour but have been unsuccessful in doing so.
Closest I have gotten is being able to structure commands based on priority but I am not able to logically separate them out as in the example above.
Any help would be greatly appreciated.

I've achieved this by creating my own click.Group:
class OrderedGroup(click.Group):
def __init__(self, name=None, commands=None, **attrs):
super(OrderedGroup, self).__init__(name, commands, **attrs)
self.commands = commands or collections.OrderedDict()
def list_commands(self, ctx):
return self.commands
def format_commands(self, ctx, formatter):
super().get_usage(ctx)
formatter.write_paragraph()
with formatter.section("Specific Commands for X:"):
formatter.write_text(
f'{self.commands.get("command1").name}\t\t{self.commands.get("command1").get_short_help_str()}')
formatter.write_text(
f"{self.commands.get('command2').name}\t\t{self.commands.get('command2').get_short_help_str()}")
with formatter.section("Specific Commands for Y:"):
formatter.write_text(
f'{self.commands.get("command3").name}\t\t{self.commands.get("command3").get_short_help_str()}')
formatter.write_text(
f'{self.commands.get("command4").name}\t\t{self.commands.get("command4").get_short_help_str()}')
with formatter.section("Global Commands"):
formatter.write_text(
f'{self.commands.get("version").name}\t\t{self.commands.get("version").get_short_help_str()}')
And created the cli group as such:
#click.group(cls=OrderedGroup)
def cli():
pass
Does this help?

Python click and --help option callback isn't eager

I have the following click code:
#click.group(invoke_without_command=True)
def cli():
click.echo("Starting CallFlow....")
setup_logging()
# ##################----GEN---##################
#cli.command(help="a sub command")
#click.option(
"--folder", help="Tests folder path", type=str, nargs=1,
)
def sub1(folder):
# run some code here
Running my prog name that uses the above cli like this:
prog-name --help
shows me the correct help text:
Usage: prog-name [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
sub1 Help text
But running
prog-name sub1 --help --folder
I get an error that folder requires an argument like so:
Error: --folder option requires an argument
I thought that --help was an eager parameter and gets evaluated first. shouldn't that produce a help text?

From the documentation, the concept of eagerness refers only to the order of execution. Usually, the command line options will be processed in the order they are defined; making options like --help and --version eager means that they will be evaluated first.
If --help were not eager, your example would require --folder to always be passed first, like:
prog-name sub1 --folder test_folder --help

How to group arguments into sections for a program's help message

Is it possible to add some sub sections on optional arguments? So it would be easier for user to understand which argument is related with which argument?
I mean for example psql --help, outputs this (I don't know which arguments parse library psql uses, but giving it just as a good example of desired output):
Usage:
psql [OPTION]... [DBNAME [USERNAME]]
General options:
-c, --command=COMMAND run only single command (SQL or internal) and exit
-d, --dbname=DBNAME database name to connect to (default: "oerp")
-f, --file=FILENAME execute commands from file, then exit
-l, --list list available databases, then exit
-v, --set=, --variable=NAME=VALUE
set psql variable NAME to VALUE
(e.g., -v ON_ERROR_STOP=1)
-V, --version output version information, then exit
-X, --no-psqlrc do not read startup file (~/.psqlrc)
-1 ("one"), --single-transaction
execute as a single transaction (if non-interactive)
-?, --help[=options] show this help, then exit
--help=commands list backslash commands, then exit
--help=variables list special variables, then exit
Input and output options:
-a, --echo-all echo all input from script
-b, --echo-errors echo failed commands
-e, --echo-queries echo commands sent to server
-E, --echo-hidden display queries that internal commands generate
-L, --log-file=FILENAME send session log to file
-n, --no-readline disable enhanced command line editing (readline)
-o, --output=FILENAME send query results to file (or |pipe)
-q, --quiet run quietly (no messages, only query output)
-s, --single-step single-step mode (confirm each query)
-S, --single-line single-line mode (end of line terminates SQL command)
Output format options:
-A, --no-align unaligned table output mode
-F, --field-separator=STRING
field separator for unaligned output (default: "|")
-H, --html HTML table output mode
-P, --pset=VAR[=ARG] set printing option VAR to ARG (see \pset command)
-R, --record-separator=STRING
record separator for unaligned output (default: newline)
-t, --tuples-only print rows only
-T, --table-attr=TEXT set HTML table tag attributes (e.g., width, border)
-x, --expanded turn on expanded table output
-z, --field-separator-zero
set field separator for unaligned output to zero byte
-0, --record-separator-zero
set record separator for unaligned output to zero byte
Connection options:
-h, --host=HOSTNAME database server host or socket directory (default: "/var/run/postgresql")
-p, --port=PORT database server port (default: "5432")
-U, --username=USERNAME database user name (default: "oerp")
-w, --no-password never prompt for password
-W, --password force password prompt (should happen automatically)
For more information, type "\?" (for internal commands) or "\help" (for SQL
commands) from within psql, or consult the psql section in the PostgreSQL
documentation.
Report bugs to <pgsql-bugs#postgresql.org>.
And there are sections like General options, Input and output options and so on.
With argparse, I only get positional arguments and optional arguments options. Is it possible to make some subsections for optional arguments?

As BoarGules mentioned in a comment, you want ArgumentParser.add_argument_group().
Example code copied here, modified:
import argparse
parser = argparse.ArgumentParser(prog='PROG')
group = parser.add_argument_group('group')
group.add_argument('--foo', help='foo help')
group.add_argument('bar', help='bar help')
other_group = parser.add_argument_group('other group')
other_group.add_argument('baz', help='baz help')
parser.print_help()
Output:
usage: PROG [-h] [--foo FOO] bar baz
optional arguments:
-h, --help show this help message and exit
group:
--foo FOO foo help
bar bar help
other group:
baz baz help

Python scripts enabled to run w/out the CLI do not take command line arguments

In windows 7, if a python (2.7.10) script has been associated with the Python interpreter and the extension has been registered in the PATHEXT (windows) list, then when you kick off a script, one can simply type:
MyPythonScript
instead of the traditional:
python MyPythonScript.py
Which is great - but it no longer (appears) to take command line arguments. For example, consider the two below examples of a script that takes command line arguments. First the traditional way:
>>>> python echo_input.py --help
usage: echo_input.py [-h] [-s SIMPLE_VALUE] [-c] [-t] [-f] [-a COLLECTION]
[-A] [-B] [--version]
optional arguments:
-h, --help show this help message and exit
-s SIMPLE_VALUE Store a simple value
-c Store a constant value
-t Set a switch to true
-f Set a switch to false
-a COLLECTION Add repeated values to a list
-A Add different values to list
-B Add different values to list
--version show program's version number and exit
works just fine, but if it is invoked the alternative way:
echo_input --help
simple_value = None
constant_value = None
boolean_switch = False
collection = []
const_collection = []
It appears to no longer recognizes the command line arguments. FYI: The above script (by default) prints out those 5 lines if the it is run w/out any parameters as shown below for contrast:
>python echo_input.py
simple_value = None
constant_value = None
boolean_switch = False
collection = []
const_collection = []
So it appears as though it has lost it's ability to take command line arguments such as a simple flag for help.
I'm stuck for both an answer to this and a work around and would greatly appreciate any suggestions or experience.
Thank you in advance for your time ... :-)
Sources of inspiration:
Registering Python Scripts to run on Windows:
Python FAQ2:

Does this work (thanks #eryksun for suggested fixes -- I don't have a Windows machine any more)?
C:\>ftype Python.File="C:\Python27\python.exe" "%1" %*
C:\>assoc .py=Python.File
This adds/modifies the ProgId in HKLM\SOFTWARE\Classes. Depending on the current configuration, you may also have to select this ProgId (Python.File) in Explorer's "open with" dialog.

importing a python script from another script and running it with arguments

I have a python script which has been packaged up as a command line script (dbtoyaml.py in Pyrseas since you ask).
I am running another python script from which I want to call this script. Is there no way to import the module and artificially populate the required arguments from my second script to avoid changing any of the pyrseas code at all?
from pyrseas import dbtoyaml
-- My initial script, which also takes arguments
dbtoyaml.main(['-m','-H MYHOSTNAME' .... other options])
Hasn't yet worked for me.
I get a strange error:
usage: checkSchemaChanges.py [-h] [-H HOST] [-p PORT] [-U USERNAME] [-W]
[-c CONFIG] [-r REPOSITORY] [-o OUTPUT]
[--version] [-m] [-O] [-x] [-n SCHEMA]
[-N SCHEMA] [-t TABLE] [-T TABLE]
dbname
checkSchemaChanges.py: error: unrecognized arguments: MYHOSTNAME mydatabaseuser
Which is a mixture of my new script (checkSchemaChanges.py, and MYHOSTNAME and mydatabaseuser at the bottom) and the parameters from dbtoyaml, which are all correct.
Could it be the double set of parameters which is confusing argparse?

When I'm writing a command line script I oftentimes will specifically design my script so this is possible. The key is to parse the args separate from the main function.
For example the main function might look like this:
def main(**kwargs):
# the body of the script goes here
Then elsewhere in the module I will configure the arg parser, parse the args and pass the result into the main script:
def run():
parser = ... # configure parser here
configs = parse_args(parser)
main(**configs)
That way, if someone wants to call the script from within Python, they can do so (it also makes testing much easier):
import somescript
somescript.main(option='value', option2='value2')
Unfortunately, it appears that the authors of the script you are using did not do anything like that. As stated in another answer you can overwrite sys.argv, then import the script. While that may feel hacky, it should be less resource intensive than opening a new process and calling the command separately.

this seems like not the best way to do it but you can probably set sys.argv
import sys
sys.argv += ['-m','-H MYHOSTNAME' .... other options]
from pyrseas import dbtoyaml
dbtoyaml.main()
but really I have no idea what dbtoyaml.py lookslike or is