I have been struggling with this for a few days now and still dont have a good solution. Instead of providing code this time which with this problem has lately been leading to unhelpful tangents, let me just give you an idea of exactly what I am trying to accomplish and perhaps this will streamline the solution.
All I am trying to do run a python program while inputting a few variables to control what the program does. Allow me to give a specific example.
Example Syntax Structure
program_name function_to_run variable_1 variable_2 variable_n
Generic Syntax Example
parrot add "Mr Fluffy" "Red" "15oz"
Another Example
datamine search "Chris"
So to expand on these examples. The first program "parrot" has an add function. When the program is run and the add function is used from the command line, the program expects three variables (Name, color, weight). In the second example, the program named "datamine" has a function named "search" that expects a single string (the search term). The idea is, the program (datamine) for example will have several functions that could be used. Perhaps "add", "search", "delete" are all examples and each will have different expected variables. Using datamine help would list out each function and the required and or optional components.
Using argparse, I have not been able to figure out a working implementation of this yet. From past experience, I think the solution will involved using custom actions. Can anyone please help with some example code? I am using Python 3 by the way.
Thanks for the help!
Use subparsers. The docs give a good example of how to use set_defaults to specify the function that should be called for each subparser:
One particularly effective way of handling sub-commands is to combine the use of the add_subparsers() method with calls to set_defaults() so that each subparser knows which Python function it should execute.
In your examples, parrot and datamine would be separate parsers in separate modules, and add and search would be subparsers under them respectively. For example, the datamine module would look something like this:
#!/usr/bin/env python
# datamine
def add(a, b):
print(a + b)
def search(query, search_all=True):
run_my_search_app(query, search_all=search_all)
if __name__ == '__main__':
# create the top-level parser
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# create the parser for the "add" command
parser_add = subparsers.add_parser('add')
parser_add.add_argument('-a', type=int, default=1)
parser_add.add_argument('-b', type=int, default=2)
parser_add.set_defaults(func=add)
# create the parser for the "search" command
parser_search = subparsers.add_parser('search')
parser_search.add_argument('query')
parser_search.add_argument('--search-all', action='store_true')
parser_search.set_defaults(func=search)
args = parser.parse_args()
args = vars(args)
func = args.pop("func")
func(**args)
If this file is executable in your shell as datamine, you can do:
datamine add -a 11 -b 5
datamine search foo --search-all
Without optional flags you don't need anything fancy - just look at sys.argv directly:
import sys
def my_add(*args):
print( ','.join(args))
def my_search(*args):
print(args)
fn_map = {"add": my_add, "search": my_search}
if sys.argv[1:]:
fn = fn_map[sys.argv[1]]
rest = sys.argv[2:]
fn(*rest)
sample runs
1951:~/mypy$ python stack43990444.py
1951:~/mypy$ python stack43990444.py add "Mr Fluffy" "Red" "15oz"
Mr Fluffy,Red,15oz
1951:~/mypy$ python stack43990444.py search "Chris"
('Chris',)
Fully functional extrapolation of code from your parrot example using subparsers. Data set (created by this code) and usage examples at the bottom. Beware, example set does not consist strictly of parrots
#!/usr/bin/env python3
import argparse
import json
def add_parrot(name, weight, kind, **kwargs):
print("Adding {} of type {} and size {}".format(name, kind, weight))
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
parrots.append({'name': name, 'weight': weight, 'type': kind})
with open('parrots.json', 'w') as parrotdb:
json.dump(parrots, parrotdb)
def delete_parrot(name, **kwargs):
print("Uh oh! What happened to {}?".format(name))
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
parrots[:] = [p for p in parrots if p.get('name') != name]
with open('parrots.json', 'w') as parrotdb:
json.dump(parrots, parrotdb)
def show_parrots(name=None, weight=0, kind=None, **kwargs):
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
for p in parrots:
if (name or weight or kind):
if name in p['name'] or weight == p['weight'] or kind == p['type']:
print("{}\t{}\t{}".format(
p['name'], p['weight'], p['type']))
else:
print("{}\t{}\t{}".format(p['name'], p['weight'], p['type']))
parser = argparse.ArgumentParser(description="Manage Parrots")
subparsers = parser.add_subparsers()
add_parser = subparsers.add_parser('insert', aliases=['add', 'a'])
add_parser.add_argument('name')
add_parser.add_argument('weight', type=int)
add_parser.add_argument('kind')
add_parser.set_defaults(func=add_parrot)
del_parser = subparsers.add_parser("delete", aliases=['del', 'd'])
del_parser.add_argument('name')
del_parser.set_defaults(func=delete_parrot)
ls_parser = subparsers.add_parser('list', aliases=['show', 'ls'])
ls_parser.add_argument('--name')
ls_parser.add_argument('--size', type=int)
ls_parser.add_argument('--type', dest='kind')
ls_parser.set_defaults(func=show_parrots)
args = parser.parse_args()
args.func(**vars(args))
Dataset and usage examples:
➜ ~ cat parrots.json
[{"name": "tweety", "weight": 4, "type": "yellow"}, {"name": "donald", "weight": 18, "type": "white"}, {"name": "daffy", "weight": 12, "type": "black"}]
➜ ~ ./parrot.py ls
tweety 4 yellow
donald 18 white
daffy 12 black
➜ ~ ./parrot.py ls --name tweety
tweety 4 yellow
➜ ~ ./parrot.py delete tweety
Uh oh! What happened to tweety?
➜ ~ ./parrot.py ls --name tweety
➜ ~
Related
I have 2 subparsers:
from argparse import ArgumentParser
parser = ArgumentParser()
parser.add_argument('--say')
subparsers = parser.add_subparsers(dest='action', help='what should I do')
a = subparsers.add_parser('eat')
a.add_argument('food')
b = subparsers.add_parser('drink')
b.add_argument('beverage')
print(parser.parse_args(['eat', 'apple']))
print(parser.parse_args(['drink', 'water']))
print(parser.parse_args(['--say', 'thanks', 'drink', 'water'])) # Works, but I don't like it
print(parser.parse_args(['drink', 'water', '--say', 'thanks'])) # I want this to work
Namespace(action='eat', food='apple', say=None)
Namespace(action='drink', beverage='water', say=None)
Namespace(action='drink', beverage='water', say='thanks')
usage: test.py [-h] [--say SAY] {eat,drink} ...
test.py: error: unrecognized arguments: --say thanks
Adding parents gets everything complicated...
from argparse import ArgumentParser
parser = ArgumentParser(add_help=False)
parser.add_argument('--say')
subparsers = parser.add_subparsers(dest='action', help='what should I do')
a = subparsers.add_parser('eat', parents=[parser])
a.add_argument('food')
b = subparsers.add_parser('drink', parents=[parser])
b.add_argument('beverage')
print(parser.parse_args(['eat', 'apple']))
usage: test.py eat [-h] [--say SAY] {eat,drink} ... food
test.py eat: error: invalid choice: 'apple' (choose from 'eat', 'drink')
Suddenly nothing works because I have to input "eat" TWICE??? (the 2nd time is for the action, but why do I need the 1st?)
All I want is to have a shared argument, that I can add at the end, or the start of my argument list (to my choosing), for multiple subparsers, i.e. I want all of the following to work:
test.py eat apple --say thanks
test.py --say grace drink coffee
test.py --say grace eat bread --clap hands (if I add another shared argument)
Once parse_args has recognized the subcommand, all further arguments are passed to the subparser, not the main parser. Using parents= inherits all arguments, including the subcommands, not just the other options.
Instead, you want yet another parser that does nothing except provide --say; you can then use that as a parent for each of the main parser and its two subparsers.
from argparse import ArgumentParser
say_parser = ArgumentParser(add_help=False)
say_parser.add_argument('--say')
parser = ArgumentParser(parents=[say_parser])
subparsers = parser.add_subparsers(dest='action', help='what should I do')
a = subparsers.add_parser('eat', parents=[say_parser])
a.add_argument('food')
b = subparsers.add_parser('drink', parents=[say_parser])
b.add_argument('beverage')
print(parser.parse_args(['eat', 'apple']))
print(parser.parse_args(['drink', 'water']))
print(parser.parse_args(['--say', 'thanks', 'drink', 'water']))
print(parser.parse_args(['drink', 'water', '--say', 'thanks']))
If you don't want the third one to work, you can omit the say_parser from the parent list of the main parser.
I have to invoke my script in this way.
script.py multiple --ways aa bb -as abab -bs bebe
In this case -as abab reffers to "aa" option from --ways parameter and -bs bebe to "bb" option.
And all chosed options should affect what "fulfill" method will be used.
If we chose 'aa' and 'bb' there should only be options '-as' and '-bs' not '-cs'.
import sys
from argparse import ArgumentParser
def fulfill_aa_parser(aa_parser):
aa_parser.add_argument('--ass', '-as', type=str, required=True, choices=['abababa'])
def fulfill_bb_parser(aa_parser):
aa_parser.add_argument('--bass', '-bs', type=str, required=True, choices=['bebebe'])
def fulfill_cc_parser(aa_parser):
aa_parser.add_argument('--cass', '-cs', type=str, required=True, choices=['cycycyc'])
def fulfill_multiple_parser(multiple_parser):
multiple_parser.add_argument('--ways', '-w', type=str, choices=['aa','bb', 'cc'], nargs='+', required=True)
def main(argv):
parser = ArgumentParser(description='TEST CASE')
subparsers = parser.add_subparsers(dest='type')
multiple_parser = subparsers.add_parser(
'multiple'
)
aabbparsers = multiple_parser.add_subparsers()
aa_parser = aabbparsers.add_parser('aa')
bb_parser = aabbparsers.add_parser('bb')
cc_parser = aabbparsers.add_parser('cc')
fulfill_multiple_parser(multiple_parser)
fulfill_aa_parser(aa_parser)
fulfill_bb_parser(bb_parser)
fulfill_cc_parser(cc_parser)
args = parser.parse_args(argv)
if args.type is None:
parser.print_help()
return
if __name__ == '__main__':
main(sys.argv[1:])
Parsing this in this way:
fulfill_aa_parser(multiple_parser)
fulfill_bb_parser(multiple_parser)
fulfill_cc_parser(multiple_parser)
will lead to parser always asking for '-as', '-bs' ,'-cs' and options in '--ways' will not affect this
EDIT : \
This is it looks when there is some thought put to it.
Just simply pass parser to this function
def fulfill_apple_argparser(parser):
parser.add_argument("--apple_argument")
def fulfill_banana_argparser(parser):
parser.add_argument("--banana_argument")
def fulfill_peach_argparser(parser):
parser.add_argument("--peach_argument")
def many_fruits_parse(parser, progs=None, list_of_fruits=('apple', 'banana', 'peach')):
progs = progs or []
if len(list_of_fruits) == 0 or parser in progs:
return
fulfill = {'apple': fulfill_apple_argparser, 'banana': fulfill_banana_argparser,
'peach': fulfill_peach_argparser}
subparsers = parser.add_subparsers(title='subparser', dest=parser.prog)
progs.append(parser)
for fruit in list_of_fruits:
secondary = [x for x in list_of_fruits if x != fruit]
fruit_parser = subparsers.add_parser(fruit, help=fruit)
fulfill[fruit](fruit_parser)
many_fruits_parse(fruit_parser, progs, secondary)
add_subparsers creates a special kind of positional argument, one that uses the add_parser command to create choices. Once a valid choice is provided, parsing is passed to that parser.
With
script.py multiple --ways aa bb -as abab -bs bebe
parser passes the task to multiple_parser. The --ways optional then gets 2 values
Namespace(ways=['aa','bb'])
Neither of those strings is used as a value for aabbparsers, and multiple_parser doesn't know what to do with '-as` or '-bs', and (I expect) will raise an error.
With:
script.py multiple aa -as abab
parsing is passed from parser to multiple_parser to aa_parser, which in turn handles '-as abab', producing (I think)
Namespace(as='abab')
Nesting as you do with multiple and aa is the only way to use multiple subparsers. You can't have two subparses 'in-parallel' (e.g. 'aa' and 'bb').
Especially when testing it's a good idea to provide a dest to the add_subparsers command. It gives information on which subparsers is being invoked.
I simplifying my problem here. I need to do this:
python test.py --arg1 '--no-route53'
I added the parser like this:
parser.add_argument("--arg1", nargs="?", type=str, help="option")
args = parser.parse_args()
I wanted to get the '--no-route53' as a whole string and use it later in my script. But I keep getting this error:
test.py: error: unrecognized arguments: --no-route53
How can I work around it?
UPDATE1: if i give extra space after '--no-route53', like this and it worked:
python test.py --arg1 '--no-route53 '
I had the exact same issue a while ago. I ended up pre-parsing the sys.argvlist:
def bug_workaround(argv):
# arg needs to be prepended by space in order to not be interpreted as
# an option
add_space = False
args = []
for arg in argv[1:]: # Skip argv[0] as that should not be passed to parse_args
if add_space:
arg = " " + arg
add_space = True if arg == "--args" else False
args.append(arg)
return args
parser = argparse.ArgumentParser(description='My progrm.')
.
.
.
parser.add_argument('--args', type=str,
help='Extra args (in "quotes")')
args = parser.parse_args(bug_workaround(sys.argv))
# Prune added whitespace (for bug workaround)
if args.args:
args.args = args.args[1:]
import argparse
parser = argparse.ArgumentParser(prog='tool')
args = [('-u', '--upf', 'ref. upf', dict(required='True')),
('-s', '--skew', 'ref. skew', {}),
('-m', '--model', 'ref. model', {})]
for args1, args2, desc, options in args:
parser.add_argument(args1, args2, help=desc, **options)
parser.print_help()
Output:
usage: capcheck [-h] -u UPF [-s SKEW] [-m MODEL]
optional arguments:
-h, --help show this help message and exit
-u UPF, --upf UPF ref. upf
-s SKEW, --skew SKEW ref. skew
-m MODEL, --model MODEL
ref. model
How do I print ref. model in the same line as -m MODEL, --model MODEL instead of that appearing on a separate line when I run the script with -h option?
You could supply formatter_class argument:
parser = argparse.ArgumentParser(prog='tool',
formatter_class=lambda prog: argparse.HelpFormatter(prog,max_help_position=27))
args = [('-u', '--upf', 'ref. upf', dict(required='True')),
('-s', '--skew', 'ref. skew', {}),
('-m', '--model', 'ref. model', {})]
for args1, args2, desc, options in args:
parser.add_argument(args1, args2, help=desc, **options)
parser.print_help()
Note: Implementation of argparse.HelpFormatter is private only the name is public. Therefore the code might stop working in future versions of argparse. File a feature request to provide a public interface for the customization of max_help_position on http://bugs.python.org/
Output
usage: tool [-h] -u UPF [-s SKEW] [-m MODEL]
optional arguments:
-h, --help show this help message and exit
-u UPF, --upf UPF ref. upf
-s SKEW, --skew SKEW ref. skew
-m MODEL, --model MODEL ref. model
Inspired by #jfs's answer, I have come up with this solution:
def make_wide(formatter, w=120, h=36):
"""Return a wider HelpFormatter, if possible."""
try:
# https://stackoverflow.com/a/5464440
# beware: "Only the name of this class is considered a public API."
kwargs = {'width': w, 'max_help_position': h}
formatter(None, **kwargs)
return lambda prog: formatter(prog, **kwargs)
except TypeError:
warnings.warn("argparse help formatter failed, falling back.")
return formatter
Having that, you can call it with any HelpFormatter that you like:
parser = argparse.ArgumentParser(
formatter_class=make_wide(argparse.ArgumentDefaultsHelpFormatter)
)
or
parser = argparse.ArgumentParser(
formatter_class=make_wide(argparse.HelpFormatter, w=140, h=20)
)
What this does is make sure that the wider formatter can actually be created using the width and max_help_position arguments. If the private API changes, that is noted by make_wide by a TypeError and the formatter is returned unchanged. That should make the code more reliable for deployed applications.
I'd welcome any suggestions to make this more pythonic.
If you are providing a custom formatter_class to your ArgumentParser
parser = argparse.ArgumentParser(formatter_class=help_formatter)
and then use subparsers, the formatter will only apply to the top-level help message. In order to use the same (or some other) formatter for all subparsers, you need to provide formatter_class argument for each add_parser call:
subparsers = parser.add_subparsers(metavar="ACTION", dest="action")
child_parser = subparsers.add_parser(
action_name, formatter_class=help_formatter
)
As the argparse library tries to use the COLUMNS environment variable to get the terminal width, we also can set this variable, and let argparse do its job.
import os
import argparse
rows, columns = os.popen('stty size', 'r').read().split()
os.environ["COLUMNS"] = str(columns)
parser = argparse.ArgumentParser(etc...
Tested and approved on RHEL/Python 2.7.5
Credits to https://stackoverflow.com/a/943921 for getting the real terminal width
Another approach: hijack sys.argv, check it for --help and -h, if found extract help text using argparse.format_help, massage it, print it, and exit.
import sys, re, argparse
RGX_MID_WS = re.compile(r'(\S)\s{2,}')
def main(argv):
# note add_help = False
parser = argparse.ArgumentParser(description = '%(prog)s: testing help mods', formatter_class= argparse.RawTextHelpFormatter, add_help = False)
parser.add_argument('bar', nargs='+', help='two bars that need to be frobbled')
parser.add_argument('--foo', action='store_true', help='foo the bars before frobbling\nfoo the bars before frobbling')
parser.add_argument('--xxxxx', nargs=2, help='many xes')
parser.add_argument('--bacon', help ='a striped food')
parser.add_argument('--badger', help='in a striped pyjamas')
parser.add_argument('--animal', dest='animal', choices=('zabra', 'donkey', 'bat') ,help ='could be one of these')
# may exit
lArgs = help_manage(parser)
args = parser.parse_args() # args = lArgs
print('bars are: ', args.bar)
def help_manage(parser):
"""
check for -h, --help, -h in a single-letter cluster;
if none found, return, otherwise clean up help text and exit
"""
lArgs = sys.argv[1:]
lArgsNoHelp = [sOpt for sOpt in lArgs if (not sOpt in ('--help', '-h')) and not (sOpt[0] == '-' and sOpt[1] != '-' and 'h' in sOpt)]
# no change? then no --help params
if len(lArgsNoHelp) == len(lArgs): return
sHelp = parser.format_help()
# to see help as formated by argparse, uncomment:
# print(sHelp)
# exit()
for sLine in sHelp.split('\n'): print(clean_line(sLine))
exit()
def clean_line(sLine):
"""
this is just an example, and goes nowhere near covering all possible
argument properties
"""
# avoid messing with usage: lines
if 'usage' in sLine: return sLine
if sLine.startswith(' ') and '[' in sLine: return sLine
if sLine.endswith(' arguments:'): return sLine + '\n'
sLine = sLine.lstrip()
sLine = RGX_MID_WS.sub(r'\1\n', sLine)
if sLine.startswith('-'): sLine = '\n' + sLine
return sLine.replace('{', '\n(can be: ').replace('}', ')').replace('\n\n', '\n')
if __name__ == '__main__':
bRes = main(sys.argv[1:])
sys.exit(bRes)
Help without formatting:
usage: argparse_fix_min2.py [--foo] [--xxxxx XXXXX XXXXX] [--bacon BACON]
[--badger BADGER] [--animal {zabra,donkey,bat}]
bar [bar ...]
argparse_fix_min2.py: testing help mods
positional arguments:
bar two bars that need to be frobbled
optional arguments:
--foo foo the bars before frobbling
foo the bars before frobbling
--xxxxx XXXXX XXXXX many xes
--bacon BACON a striped food
--badger BADGER in a striped pyjamas
--animal {zabra,donkey,bat}
could be one of these
with formatting:
usage: argparse_fix_min2.py [--foo] [--xxxxx XXXXX XXXXX] [--bacon BACON]
[--badger BADGER] [--animal {zabra,donkey,bat}]
bar [bar ...]
argparse_fix_min2.py: testing help mods
positional arguments:
bar
two bars that need to be frobbled
optional arguments:
--foo
foo the bars before frobbling
foo the bars before frobbling
--xxxxx XXXXX XXXXX
many xes
--bacon BACON
a striped food
--badger BADGER
in a striped pyjamas
--animal
(can be: zabra,donkey,bat)
could be one of these
"""
I'm writing a server querying tool, and I have a little bit of code to parse arguments at the very top:
# Parse arguments
p = argparse.ArgumentParser()
g = p.add_mutually_exclusive_group(required=True)
g.add_argument('--odam', dest='query_type', action='store_const',
const='odam', help="Odamex Master query.")
g.add_argument('--odas', dest='query_type', action='store_const',
const='odas', help="Odamex Server query.")
p.add_argument('address', nargs='*')
args = p.parse_args()
# Default master server arguments.
if args.query_type == 'odam' and not args.address:
args.address = [
'master1.odamex.net:15000',
'master2.odamex.net:15000',
]
# If we don't have any addresses by now, we can't go on.
if not args.address:
print "If you are making a server query, you must pass an address."
sys.exit(1)
Is there a nicer way to do this, preferably all within the parser? That last error looks a little out of place, and it would be nice if I could make nargs for address depend on if --odam or ---odas is passed. I could create a subparser, but that would make help look a little odd since it would leave off the addresses part of the command.
You can do this with an custom argparse.Action:
import argparse
import sys
class AddressAction(argparse.Action):
def __call__(self, parser, args, values, option = None):
args.address=values
if args.query_type=='odam' and not args.address:
args.address=[
'master1.odamex.net:15000',
'master2.odamex.net:15000',
]
if not args.address:
parser.error("If you are making a server query, you must pass an address.")
p = argparse.ArgumentParser()
g = p.add_mutually_exclusive_group(required=True)
g.add_argument('--odam', dest='query_type', action='store_const',
const='odam', help="Odamex Master query.")
g.add_argument('--odas', dest='query_type', action='store_const',
const='odas', help="Odamex Server query.")
p.add_argument('address', nargs='*', action=AddressAction)
args = p.parse_args()
yields
% test.py --odas
If you are making a server query, you must pass an address.
% test.py --odam
Namespace(address=['master1.odamex.net:15000', 'master2.odamex.net:15000'], query_type='odam')
% test.py --odam 1 2 3
Namespace(address=['1', '2', '3'], query_type='odam')