We Keep Coding

Put subparser command at the beginning of arguments

I wanted to create an interface with argparse for my script with subcommands; so, if my script is script.py, I want to call it like python script.py command --foo bar, where command is one of the N possible custom commands.
The problem is, I already tried looking for a solution here on StackOverflow, but it seems like everything I tried is useless.
What I have currently is this:
parser = argparse.ArgumentParser()
parser.add_argument("-x", required=True)
parser.add_argument("-y", required=True)
parser.add_argument("-f", "--files", nargs="+", required=True)
# subparsers for commands
subparsers = parser.add_subparsers(title="Commands", dest="command")
subparsers.required = True
summary_parser = subparsers.add_parser("summary", help="help of summary command")
If I try to run it with:
args = parser.parse_args("-x 1 -y 2 -f a/path another/path".split())
I got this error, as it should be: script.py: error: the following arguments are required: command.
If, however, I run this command:
args = parser.parse_args("summary -x 1 -y 2 -f a/path another/path".split())
I got this error, that I shouldn't have: script.py: error: the following arguments are required: -x, -y, -f/--files.
If I put the command at the end, changing also the order of arguments because of -f, it works.
args = parser.parse_args("-x 1 -f a/path another/path -y 2 summary".split())
If I add the parents keyword in subparser, so substitute the summary_parser line with summary_parser = subparsers.add_parser("summary", help=HELP_CMD_SUMMARY, parents=[parser], add_help=False), then I got:
script.py summary: error: the following arguments are required: command when summary is in front of every other argument;
script.py summary: error: the following arguments are required: -x, -y, -f/--files, command when summary is at the end of the args.
My question is, how I have to setup the parsers to have the behaviour script.py <command> <args>? Every command shares the same args, because they are needed to create certain objects, but at the same time every command can needs other arguments too.

Creating another parser helped me getting what I wanted.
The root parser should add all the optional arguments - and also have add_help=False, to avoid an help message conflict -, then another parser - parser2, with a lot of fantasy - will be created.
The second parser will have subparsers, and they all needs to specify as parents the root parser.
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument() ...
parser2 = argparse.ArgumentParser()
subparsers = parser2.add_subparsers(title="Commands", dest="command")
subparsers.required = True
summary_parser = subparsers.add_parser("summary", parents=[parser])
summary_parser.add_argument("v", "--verbose", action="store_true")
# parse args
args = parser2.parse_args()
Now the output will be this:
usage: script.py [-h] {summary} ...
optional arguments:
-h, --help show this help message and exit
Commands:
{summary}
summary For each report print its summary, then exit
usage: script.py summary [-h] -x X -y Y -f FILES [FILES ...] [-v]
optional arguments:
-h, --help show this help message and exit
-x X
-y Y
-f FILES [FILES ...], --files FILES [FILES ...]
-v, --verbose

Python Cement CLI - print help for subparser when invalid argument given

I am using Python 3.7 and Cement 3.0.4.
What I have is a base app with a controller and the controller has a command that takes a single optional argument. What I'm seeing is if I pass the command an invalid argument I get an invalid argument error as I expect but I am getting the "usage" output for the app itself rather than the command on the controller. Here is an example:
from cement import App, Controller, ex
class Base(Controller):
class Meta:
label = 'base'
#ex(help='example sub-command')
def cmd1(self):
print('Inside Base.cmd1()')
#ex(
arguments=[
(['-n', '--name'],
{'help': ''': The name you want printed out''',
'dest': 'name',
'required': False}),
],
help=' the help for cmd2.')
def cmd2(self):
print(self.app.pargs.name)
This app has a command called cmd2 and it takes an optional argument of -n which, as the help states, will be printed out. So if I do this:
with MyApp(argv=['cmd2', '-n', 'bob']) as app:
app.run()
I will the output:
bob
as expected. However if I pass an invalid argument to cmd2:
with MyApp(argv=['cmd2', '-a', 'bob']) as app:
app.run()
I get:
usage: myapp [-h] [-d] [-q] {cmd1,cmd2} ...
myapp: error: unrecognized arguments: -a bob
What I would like to see, instead of the usage for myapp, would be the usage for the cmd2 command, similar to if I did -h on the command:
with MyApp(argv=['cmd2', '-h']) as app:
app.run()
outputs
usage: myapp cmd2 [-h] [-n NAME]
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME : The name you want printed out
I realize much of this is delegated to Argparse and is not handled by Cement. I've done some debugging and I'm seeing there are multiple ArgparseArgumentHandler classes nested. So in the case above there is an ArgparseArgumentHandler for myapp and it has in it's actions a SubParsersAction that has a choices field that has a map containing my two commands on the controller, cmd1 and cmd2 mapped to their own ArgparseArgumentHandler.
When the invalid argument is detected it is within the ArgparseArgumentHandler for myapp and thus it calls print_usage() on myapp rather than on the ArgparseArgumentHandler for the invoked command, cmd2.
My knowledge of Argparse is limited and I do find it a bit complex to navigate. The only workaround I can think of right now is subclassing ArgparseArgumentHandler, and overriding error() and trying to determine if the error is due to recognized arguments and if so try to find the parser for it.. something like this pseudocode:
class ArgparseArgumentOverride(ext_argparse.ArgparseArgumentHandler):
def error(self, message):
# determine if there are unknown args
args, argv = self.parse_known_args(self.original_arguments, self.original_namespace)
# we are in an error state and have unrecognized args
if argv:
controller_namespace = args.__controller_namespace__
for action in self._actions:
if action.choices is not None:
# we found an choice with our namespace
if action.choices[controller_namespace]:
command_parser= action.choices[controller_namespace]
# this should be the show_usage for the command
complete_command.print_usage(sys.stderr)
Again above is pseudocode and actually doing something like that would feel very fragile, error prone, and unpredictable. I know there has to be an better way to do this, I'm just not finding it. I've been digging through the docs and source for hours and still haven't found what I'm looking for. Could anyone tell me what I'm missing? Any advice on how to proceed here would be really appreciated. Thanks much!

I'm not familiar cement, but as you deduce the usage is generated by argparse:
In [235]: parser = argparse.ArgumentParser(prog='myapp')
In [236]: parser.add_argument('-d');
In [237]: sp = parser.add_subparsers(dest='cmd')
In [238]: sp1 = sp.add_parser('cmd1')
In [239]: sp2 = sp.add_parser('cmd2')
In [240]: sp2.add_argument('-n','--name');
In [241]: parser.parse_args('cmd2 -a'.split())
usage: myapp [-h] [-d D] {cmd1,cmd2} ...
myapp: error: unrecognized arguments: -a
If the error is tied to a sp2 argument, then the usage reflects that:
In [242]: parser.parse_args('cmd2 -n'.split())
usage: myapp cmd2 [-h] [-n NAME]
myapp cmd2: error: argument -n/--name: expected one argument
But unknown args are handled by the main parser. For example if we use parse_known_args instead:
In [245]: parser.parse_known_args('cmd2 foobar'.split())
Out[245]: (Namespace(cmd='cmd2', d=None, name=None), ['foobar'])
In [246]: parser.parse_known_args('cmd2 -a'.split())
Out[246]: (Namespace(cmd='cmd2', d=None, name=None), ['-a'])
The unknown args are returned as the extras list. parse_args returns the error instead of the extras.
In _SubParsers_Action.__call__, the relevant code is:
# parse all the remaining options into the namespace
# store any unrecognized options on the object, so that the top
# level parser can decide what to do with them
# In case this subparser defines new defaults, we parse them
# in a new namespace object and then update the original
# namespace for the relevant parts.
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)
if arg_strings:
vars(namespace).setdefault(_UNRECOGNIZED_ARGS_ATTR, [])
getattr(namespace, _UNRECOGNIZED_ARGS_ATTR).extend(arg_strings)
In theory you could construct an alternate _SubParsersAction class (or subclass), that handles that arg_strings differently. Changing the parse_known_args call to parse_args might be enough:
subnamespace = parser.parse_args(arg_strings, None)
Note that parse_args calls parse_known_args, and raises the error if there are unknowns:
def parse_args(self, args=None, namespace=None):
args, argv = self.parse_known_args(args, namespace)
if argv:
msg = _('unrecognized arguments: %s')
self.error(msg % ' '.join(argv))
return args

Python argparse, Value after positional argument

So I'm writing this very small program to do http get and post requests. The requests are as follows:
requestApp.py help
requestApp.py help get
requestApp.py help post
requestApp.py get [-v] [-h key:value] URL
requestApp.py post [-v] [-h key:value] [-d inline-data] [-f file] URL
As you can see, the -v, -h, -d, -f, URL arguments are optional. The get and post arguments are non-optional. I'll show you the snippet of my program that is relevant to this situation:
parser = argparse.ArgumentParser(description='httpc is a curl-like application but supports HTTP protocol only.')
parser.add_argument('command', type=str, help=help_output())
parser.add_argument('url', action='store_true', help='The URL that will be provided to perform the requested command.')
parser.add_argument('-v', '--verbose', action='store_true')
The command argument will be help, get, or post, and the url argument is self explanatory. My question is related to the second and third commands above, namely:
requestApp.py help get
requestApp.py help post
How can I make sure that when typing help get, the get will not be registered in the URL (same for help post). In addition, when I do include a URL, I want it to be stored inside of the URL argument. Would I have to manually evaluate the arguments passed through if statements? Or there is a better way to do it?

Perhaps the closest an argparse solution can come, at least without going the subparser route, is:
import argparse
import sys
print(sys.argv)
parser = argparse.ArgumentParser()
parser.add_argument('-k', '--keyvalue')
parser.add_argument('-v', '--verbose', action='store_true')
parser.add_argument('-d', '--data')
parser.add_argument('-f', '--file')
parser.add_argument('pos1', choices = ['help', 'get', 'post'])
parser.add_argument('pos2')
args = parser.parse_args()
print(args)
The resulting help is:
1744:~/mypy$ python stack54383659.py get aurl -h
['stack54383659.py', 'get', 'aurl', '-h']
usage: stack54383659.py [-h] [-k KEYVALUE] [-v] [-d DATA] [-f FILE]
{help,get,post} pos2
positional arguments:
{help,get,post}
pos2
optional arguments:
-h, --help show this help message and exit
-k KEYVALUE, --keyvalue KEYVALUE
-v, --verbose
-d DATA, --data DATA
-f FILE, --file FILE
The fit isn't perfect. For example you can give just help, but you can provide just -h. The 2nd positional value can be any string, 'get', a valid url or something else. Your own code will have to valid that. The key:value bit requires your own parsing.
In the argparse way of parsing the optionals can occur in any order. The two positionals have to occur in the given order (relative to each other).
In newer Pythons I can change the last positional to be 'optional', and use the new intermixed parser. That would allow me to give just 'help' (or just 'get'):
parser.add_argument('pos2', nargs='?')
args = parser.parse_intermixed_args()
intermixed is needed if the two positional values are separated by flags. For some complex reasons, the regular parsing may consume the '?' argument prematurely leaving you with an extra unrecognized string.
Another approach is to define all the flagged arguments, and use parse_known_args. The non-flag values will be in the extras list, which you can parse as you like. Older parsers like optparse did essentially that. argparse added a limited ability to handle positional arguments as well, but strictly by position, not by value.

It is quite complicated to do this using argparse here is how to do it using docopt, docopt parses the usage pattern and returns a dictionary :
"""
Usage:
requestApp help [get|post]
requestApp get [-v] [-k=key:value] <URL>
requestApp post [-v] [-k=key:value] [-d=data] [-f file] <URL>
Options:
-v --verbose This is verbose mode
-d=data This option does this
-k=key:value This one does that
-f file This one is magic
"""
from docopt import docopt
ARGS = docopt(__doc__)
For example with requestApp.py post -k hello:world -f myfile.txt google.com docopt will return:
{
"--verbose": false,
"-d": None,
"-f": "myfile.txt",
"-k": "hello:world",
"<URL>": "google.com",
"get": false,
"help": false,
"post": true
}
Then you can do:
if ARGS['help']:
if ARGS['get']: pass # requestApp help get
else if ARGS['post']: pass # requestApp help post
else: pass # requestApp help
exit()
if ARGS['get']: pass # requestApp get
else if ARGS['post']: pass # requestApp post
if ARGS['--verbose']: print("this is just the beginning")
-h is a reserved option by default (for help) that makes docopt returns the usage pattern and exit.
docopt will return the usage pattern to stdout and exit if you try illegal commands such as requestApp help unicorn

argparse command line -option after given path

I'm new to python and currently experimenting using argparse to add command line options. However my code is not working, despite looking at various online tutorials and reading up on argparse I still don't fully understand it. My problem is whenever I try to call my -option it gives me a find.py error: argument regex:
Here is my call:
./find.py ../Python -name '[0-9]*\.txt'
../Python is one directory behind my current one and has a list of files/directories. Without the -name option I print out the files with their path (this works fine) but with the -name option I want to print out files matching the regex but it won't work. Here is what I currently have:
#!/usr/bin/python2.7
import os, sys, argparse,re
from stat import *
def regex_type(s, pattern=re.compile(r"[a-f0-9A-F]")):
if not pattern.match(s):
raise argparse.ArgumentTypeError
return s
def main():
direc = sys.argv[1]
for f in os.listdir(direc):
pathname = os.path.join(direc, f)
mode = os.stat(pathname).st_mode
if S_ISREG(mode):
print pathname
parser = argparse.ArgumentParser()
parser.add_argument(
'-name', default=[sys.stdin], nargs="*")
parser.add_argument('regex', type=regex_type)
args = parser.parse_args()
if __name__ == '__main__':
main()

I tweaked your type function to be more informative:
def regex_type(s, pattern=re.compile(r"[a-f0-9A-F]")):
print('regex string', s)
if not pattern.match(s):
raise argparse.ArgumentTypeError('pattern not match')
return s
Called with
2104:~/mypy$ python2 stack50072557.py .
I get:
<director list>
('regex string', '.')
usage: stack50072557.py [-h] [-name [NAME [NAME ...]]] regex
stack50072557.py: error: argument regex: pattern not match
So it tries to pass sys.argv[1], the first string after the script name, to the regex_type function. If it fails it issues the error message and usage.
OK, the problem was the ..; I'll make a directory:
2108:~/mypy$ mkdir foo
2136:~/mypy$ python2 stack50072557.py foo
('regex string', 'foo')
Namespace(name=[<open file '<stdin>', mode 'r' at 0x7f3bea2370c0>], regex='foo')
2138:~/mypy$ python2 stack50072557.py foo -name a b c
('regex string', 'foo')
Namespace(name=['a', 'b', 'c'], regex='foo')
The strings following '-name' are allocated to that attribute. There's nothing in your code that will test them or pass them through the regex_type function. Only the first non-flag string does that.
Reading sys.argv[1] initially does not remove it from the list. It's still there for use by the parser.
I would set up a parser that uses a store_true --name argument, and 2 positionals - one for the dir and the other for regex.
After parsing check args.name. If false print the contents of args.dir. If true, perform your args.regex filter on those contents. glob might be useful.
The parser finds out what your user wants. Your own code acts on it. Especially as a beginner, it is easier and cleaner to separate the two steps.
With:
def parse(argv=None):
parser = argparse.ArgumentParser()
parser.add_argument('-n', '--name', action='store_true')
parser.add_argument('--dir', default='.')
parser.add_argument('--regex', default=r"[a-f0-9A-F]")
args = parser.parse_args(argv)
print(args)
return args
def main(argv=None):
args = parse(argv)
dirls = os.listdir(args.dir)
if args.name:
dirls = [f for f in dirls if re.match(args.regex, f)]
print(dirls)
else:
print(dirls)
I get runs like:
1005:~/mypy$ python stack50072557.py
Namespace(dir='.', name=False, regex='[a-f0-9A-F]')
['test.npz', 'stack49909128.txt', 'stack49969840.txt', 'stack49824248.py', 'test.h5', 'stack50072557.py', 'stack49963862.npy', 'Mcoo.npz', 'test_attribute.h5', 'stack49969861.py', 'stack49969605.py', 'stack49454474.py', 'Mcsr.npz', 'Mdense.npy', 'stack49859957.txt', 'stack49408644.py', 'Mdok', 'test.mat5', 'stack50012754.py', 'foo', 'test']
1007:~/mypy$ python stack50072557.py -n
Namespace(dir='.', name=True, regex='[a-f0-9A-F]')
['foo']
1007:~/mypy$ python stack50072557.py -n --regex='.*\.txt'
Namespace(dir='.', name=True, regex='.*\\.txt')
['stack49909128.txt', 'stack49969840.txt', 'stack49859957.txt']
and help:
1007:~/mypy$ python stack50072557.py -h
usage: stack50072557.py [-h] [-n] [--dir DIR] [--regex REGEX]
optional arguments:
-h, --help show this help message and exit
-n, --name
--dir DIR
--regex REGEX
If I change the dir line to:
parser.add_argument('dir', default='.')
help is now
1553:~/mypy$ python stack50072557.py -h
usage: stack50072557.py [-h] [-n] [--regex REGEX] dir
positional arguments:
dir
optional arguments:
-h, --help show this help message and exit
-n, --name
--regex REGEX
and runs are:
1704:~/mypy$ python stack50072557.py -n
usage: stack50072557.py [-h] [-n] [--regex REGEX] dir
stack50072557.py: error: too few arguments
1705:~/mypy$ python stack50072557.py . -n
Namespace(dir='.', name=True, regex='[a-f0-9A-F]')
['foo']
1705:~/mypy$ python stack50072557.py ../mypy -n --regex='.*\.txt'
Namespace(dir='../mypy', name=True, regex='.*\\.txt')
['stack49909128.txt', 'stack49969840.txt', 'stack49859957.txt']
I get the error because it now requires a directory, even it is '.'.
Note that the script still uses:
if __name__ == '__main__':
main()
My main loads the dir, and applies the regex filter to that list of names. My args.dir replaces your direc.

Defining Argument values with docopt

Im working on my first python "app" and after some advice from the participants on Stackoverflow. Ive decided to scrap what I had and start from scratch.
It seems to be parsing the arguments nicely for usage etc but im not sure how I am meant to assign the values to the args?
Do I have to create a nest of ifs? if so how do i do that for the args in docopt?
maybe like this?
if opt in ("-f", "--file"):
FWORD = arg
CODE
#!/usr/bin/python
"""
Basic domain bruteforcer
Usage:
your_script.py (-f <file>) (-d <domain>) [-t 10] [-v]
your_script.py -h | --help
Options:
-h --help Show this screen.
-f --file File to read potential Sub-domains from. (Required argument)
-p --proxy Proxy address and port. [default: http://127.0.0.1:8080] (Optional)
-d --domain Domain to bruteforce.(Required argument)
-t --thread Thread count. (Optional)
-v --verbose Turn debug on. (Optional)
"""
from docopt import docopt
def fread(FWORD, *args):
flist = open(FWORD).readlines()
return flist
if __name__ == "__main__":
arguments = docopt(__doc__, version='0.1a')
print fread(fword)

You almost got it. Your arguments variable contains the argument and you look them up as you would in a dict. So if you want to call the fread function with the file argument your main would look like this:
if __name__ == "__main__":
arguments = docopt(__doc__, version='0.1a')
fread(arguments['<file>'])
If you call the script like this:
> python your_script.py -f myfiles/file.txt -d google.com
Then your arguments will look like this:
>>> print arguments
{'--domain': True,
'--file': True,
'--help': False,
'--thread': False,
'--verbose': False,
'10': False,
'<domain>': 'google.com',
'<file>': 'myfiles/file.txt'}

You should take a look at argparse from the python standard library.