I am thinking about using argparse as my source of truth for command line settings and reduce a lot of code this way.
For example, at the moment I am using separate variables to store the argparse arguments and also do some basic operations on them.
# Store product sets for upload, if argument is "all", take all options from available choices
args = parser.parse_args()
product_sets = args.product_sets
if "all" in product_sets:
product_sets = PRODUCT_SET_CHOICES[:-1]
Is it possible to do such a logical operation also directly on argparse?
Is it frowned upon to use argparse across the script rather than saving everything in a separate settings dict or variables?
Related
I'm using Python's argparse module to create a CLI for my application. I've made a subparsers variable to store the parsers for each command, but when I can't find a way to change the title of the subparsers without modifying parser's (the main ArgumentParser's) internal variables.
Original Code
parser = ArgumentParser(prog="pacstall", formatter_class=CustomHelpFormatter)
subparsers = parser.add_subparsers(dest="command")
parser._subparsers.title = "commands" # type: ignore[union-attr]
parser._optionals.title = "options"
Result
Edited Code
parser = ArgumentParser(prog="pacstall", formatter_class=CustomHelpFormatter)
subparsers = parser.add_subparsers(title="commands", dest="command")
parser._optionals.title = "options"
Result
As you can see the order of the options and commands are switched if I make that change. Also I have no idea how to modify the title of the _optionals to "options" without modifying parser._optionals.title.
Here is my full parser file.
The parser gets 2 initial argument_groups, with titles 'positional arguments' and 'optional arguments'. There's has been discussion about changing those, or giving users more control. But as you found, you can change them directly.
I won't tell anyone if you modify parser._options :)
You can also create your own argument_group - that's documented. You'll still get the default --help in the original (unless you disable that).
subparsers puts that argument in the 'positionals' by default. If you specify a 'title', it makes a new group.
Groups are displayed in the order that they were created, or rather the order that they appear in the parser._action_groups list. Look at parser.format_help to see how that's done.
I am working on a project with many command line arguments and I'd like to be able to specify all of the arguments in the form of a file (i.e. JSON), and load this file into an argparse object, instead of pasting them into the terminal every time. I also need to enforce the presence of required arguments. I've found a couple options but none of them do exactly what I need.
If I have something like this
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--image_path", type=str, required=True)
parser.add_argument("--count", type=int, required=True)
args = parser.parse_args()
One approach might be to add something like this to update the dictionary with the JSON contents
args_dict = vars(args)
with open('path/to/args.json', 'rb') as f:
args_dict.update(json.load(f))
But an error occurs at the parse_args() line, since it didn't see the required arguments passed in.
The other approach I considered is to use
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
and just pass the file name, prefixed with #, to the command. The problem here is that I have to convert the dictionary-like format (JSON) to a line-separated list of arguments, as noted in the documentation. This is not trivial in my case.
I also considered just saving the arguments as a line-separated list to begin with, but the way that I know how to do that involves iterating over sys.argv and printing to a text file, which is problematic because I have some args that are assigned default values, which are not represented in sys.argv. I need to record the default values that were used for later reference.
It seems like there should be a simple way to do this, but I'm stumped at the moment.
I currently have a process executor which executes a script along with parameters. The script and parameters are user provided.
execute_script(script_name: str, parameters: str) -> None :
// Some logic
subprocess.Popen( ....)
The user provides both script name and parameters as a string. The execute_script(...) is a backend function which gets eventually invoked by an API exposed to the user. So, even though the parameters are executed as command-line arguments, they are not received as command-line arguments, but rather string variables.
For example:
script_name="sendEmail", parameters="--subject 'URGENT' --to 'xyz#redif.com, abc#hotmail.com'"
script_name="calculatePerformance", parameters="--revenue 193456343 --tax_per 32543 --operating_cost 3452564.256 --includeVacationDays"
script_name="generate_report", parameters"--duration 'quarterly' --dept 'Finance'"
I do not have control over the parameters being sent by yer user. We have a lot of such script and they keep increasing. So keeping a track of potential parameters per script is not feasible.
I need to ensure that only a permitted values are used for certain parameters when used in conjunction with particular scripts.
For Example:
the duration can ONLY be quarterly or monthly if script is generate_report. For any other script, any value can be provided.
I want to convert the parameters provided to the function from a string to a map, so that it is much more easier to extract and validate values for a script parameter.
Using space as delimiter to split and parse the string does not seem to be cleanest approach. Is there a more compact way to convert the parameters from string to map ?
Alternatively, any other suggestions for parsing the string?
I think you could get away with using a combination of shlex and argparse.
Use shlex.split as a way to tokenize the arguments, and then use argparse.ArgumentParser with parse_known_args to figure out the values you care about inspecting.
import argparse
import shlex
script_name = "generate_report"
parameters = "--duration 'quarterly' --dept 'Finance'"
parser_argv = [script_name, *shlex.split(parameters)]
parser = argparse.ArgumentParser()
parser.add_argument("--duration")
args, unknown = parser.parse_known_args(parser_argv)
if script_name == "generate_report":
if args.duration not in {"quarterly", "monthly"}:
raise ValueError(
f"Incorrect duration for generate_report: {args.duration}"
)
A nice thing about this design, is that if the scripts you end up invoking are already in python, you could expose the argparse setup in these tools and use them directly your method.
This would mean that you don't duplicate the parser code, allowing you to do validation as close as possible to the actual scripts inside this validator.
I am using Argparse to parse shell input to my Python function.
The tricky part is that this script first reads in a file that partially determines what kind of arguments are available to Argparse (it's a JSON file containing criteria by which the user can specify what data to output).
But before these arguments are added to my parser, I would like to read in some arguments relating to the file reading itself. (e.g. whether to fix the formatting of the input file). Kinda like this:
test.py (fix_formatting=True, **more arguments added later)
When I try to run args = parser.parse_args() twice, after the initial input and after adding more keys, things fall apart: Argparse quite predictably complains that some of the user input are unrecognized arguments:. I thought I might use subparsers to that end.
So I tried variations of (following the example in the docs as best as I could):
def main():
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='sub-command help')
settingsparser = subparsers.add_parser('settings') #i want a subparser called 'settings'
settingsparser.add_argument('--fix_formatting', action='store_true') #this subparser shall have a --fix_formatting
Then I try to parse only the "settings" part like so:
settings=parser.parse_args(['settings'])
This seems to work. But then I add my other keys and things break:
keys=['alpha','beta','gamma','delta']
for key in keys:
parser.add_argument("--"+key, type=str, help="X")
args = parser.parse_args()
If I parse any input for any of the arguments from keys, Argparse complains that I make an invalid choice: [...] (choose from 'settings'). Now I don't understand why I have to choose from "settings"; the docs say that the parse
will only contain attributes for the main parser and the subparser that was selected by the command line (and not any other subparsers)
what is my error of understanding here?
and if this is the wrong approach, how would one go about parsing one bit of input before another bit?
Any help is much appreciated!
parse_args calls parse_known_args. This returns the args namesparse along with a list of strings (from sys.argv) that it could not process (extras). parse_args raises this error if this list is not empty.
https://docs.python.org/3/library/argparse.html#partial-parsing
Thus parse_known_args is useful if you want to parse some of the input.
sys.argv remains unchanged. Subsequent calls to a parser (whether it was the original one or not) use that again, unless you pass the extras.
I don't think subparsers help you here. They aren't meant for delayed or two stage parsing. I'd suggest playing with the documentation examples for subparsers first.
To the main parser, the subparsers look like
subparsers = parser.add_argument('cmd', choices=['select',...])
In other words, it adds a positional argument where the choices are the subparser names that you define. That may help you see why it expects you to name select. Positionals are normally required.
(there's an exception to this in recent versions, https://stackoverflow.com/a/22994500/901925)
I am converting Bash shell installer utility to Python 2.7 and need to implement complex CLI so I am able to parse tens of parameters (potentially up to ~150). These are names of Puppet class variables in addition to a dozen of generic deployment options, which where available in shell version.
However after I have started to add more variables I faced are several challenges:
1. I need to group parameters into separate dictionaries so deployment options are separated from Puppet variables. If they are thrown into the same bucket, then I will have to write some logic to sort them, potentially renaming parameters and then dictionary merges will not be trivial.
2. There might be variables with the same name but belonging to different Puppet class, so I thought subcommands would allow me to filter what goes where and avoiding name collisions.
At the momment I have implemented parameter parsing via simply adding multiple parsers:
parser = argparse.ArgumentParser(description='deployment parameters.')
env_select = parser.add_argument_group(None, 'Environment selection')
env_select.add_argument('-c', '--client_id', help='Client name to use.')
env_select.add_argument('-e', '--environment', help='Environment name to use.')
setup_type = parser.add_argument_group(None, 'What kind of setup should be done:')
setup_type.add_argument('-i', '--install', choices=ANSWERS, metavar='', action=StoreBool, help='Yy/Nn Do normal install and configuration')
# MORE setup options
...
args, unk = parser.parse_known_args()
config['deploy_cfg'].update(args.__dict__)
pup_class1_parser = argparse.ArgumentParser(description=None)
pup_class1 = pup_class1_parser.add_argument_group(None, 'Puppet variables')
pup_class1.add_argument('--ad_domain', help='AD/LDAP domain name.')
pup_class1.add_argument('--ad_host', help='AD/LDAP server name.')
# Rest of the parameters
args, unk = pup_class1_parser.parse_known_args()
config['pup_class1'] = dict({})
config['pup_class1'].update(args.__dict__)
# Same for class2, class3 and so on.
The problem with this approach that it does not solve issue 2. Also first parser consumes "-h" option and rest of parameters are not shown in help.
I have tried to use example selected as an answer but I was not able to use both commands at once.
## This function takes the 'extra' attribute from global namespace and re-parses it to create separate namespaces for all other chained commands.
def parse_extra (parser, namespace):
namespaces = []
extra = namespace.extra
while extra:
n = parser.parse_args(extra)
extra = n.extra
namespaces.append(n)
return namespaces
pp = pprint.PrettyPrinter(indent=4)
argparser=argparse.ArgumentParser()
subparsers = argparser.add_subparsers(help='sub-command help', dest='subparser_name')
parser_a = subparsers.add_parser('command_a', help = "command_a help")
## Setup options for parser_a
parser_a.add_argument('--opt_a1', help='option a1')
parser_a.add_argument('--opt_a2', help='option a2')
parser_b = subparsers.add_parser('command_b', help = "command_b help")
## Setup options for parser_a
parser_b.add_argument('--opt_b1', help='option b1')
parser_b.add_argument('--opt_b2', help='option b2')
## Add nargs="*" for zero or more other commands
argparser.add_argument('extra', nargs = "*", help = 'Other commands')
namespace = argparser.parse_args()
pp.pprint(namespace)
extra_namespaces = parse_extra( argparser, namespace )
pp.pprint(extra_namespaces)
Results me in:
$ python argtest.py command_b --opt_b1 b1 --opt_b2 b2 command_a --opt_a1 a1
usage: argtest.py [-h] {command_a,command_b} ... [extra [extra ...]]
argtest.py: error: unrecognized arguments: command_a --opt_a1 a1
The same result was when I tried to define parent with two child parsers.
QUESTIONS
Can I somehow use parser.add_argument_group for argument parsing or is it just for the grouping in help print out? It would solve issue 1 without missing help side effect. Passing it as parse_known_args(namespace=argument_group) (if I correctly recall my experiments) gets all the variables (thats ok) but also gets all Python object stuff in resulting dict (that's bad for hieradata YAML)
What I am missing in the second example to allow to use multiple subcommands? Or is that impossible with argparse?
Any other suggestion to group command line variables? I have looked at Click, but did not find any advantages over standard argparse for my task.
Note: I am sysadmin, not a programmer so be gently on me for the non object style coding. :)
Thank you
RESOLVED
Argument grouping solved via the answer suggested by hpaulj.
import argparse
import pprint
parser = argparse.ArgumentParser()
group_list = ['group1', 'group2']
group1 = parser.add_argument_group('group1')
group1.add_argument('--test11', help="test11")
group1.add_argument('--test12', help="test12")
group2 = parser.add_argument_group('group2')
group2.add_argument('--test21', help="test21")
group2.add_argument('--test22', help="test22")
args = parser.parse_args()
pp = pprint.PrettyPrinter(indent=4)
d = dict({})
for group in parser._action_groups:
if group.title in group_list:
d[group.title]={a.dest:getattr(args,a.dest,None) for a in group._group_actions}
print "Parsed arguments"
pp.pprint(d)
This gets me desired result for the issue No.1. until I will have multiple parameters with the same name. Solution may look ugly, but at least it works as expected.
python argtest4.py --test22 aa --test11 yy11 --test21 aaa21
Parsed arguments
{ 'group1': { 'test11': 'yy11', 'test12': None},
'group2': { 'test21': 'aaa21', 'test22': 'aa'}}
Your question is too complicated to understand and respond to in one try. But I'll throw out some preliminary ideas.
Yes, argument_groups are just a way of grouping arguments in the help. They have no effect on parsing.
Another recent SO asked about parsing groups of arguments:
Is it possible to only parse one argument group's parameters with argparse?
That poster initially wanted to use a group as a parser, but the argparse class structure does not allow that. argparse is written in object style. parser=ArguementParser... creates one class of object, parser.add_arguement... creates another, add_argument_group... yet another. You customize it by subclassing ArgumentParser or HelpFormatter or Action classes, etc.
I mentioned a parents mechanism. You define one or more parent parsers, and use those to populate your 'main' parser. They could be run indepdently (with parse_known_args), while the 'main' is used to handle help.
We also discussed grouping the arguments after parsing. A namespace is a simple object, in which each argument is an attribute. It can also be converted to a dictionary. It is easy to pull groups of items from a dictionary.
There have SO questions about using multiple subparsers. That's an awkward proposition. Possible, but not easy. Subparsers are like issueing a command to a system program. You generally issue one command per call. You don't nest them or issue sequences. You let shell piping and scripts handle multiple actions.
IPython uses argparse to parse its inputs. It traps help first, and issues its own message. Most arguments come from config files, so it is possible to set values with default configs, custom configs and in the commandline. It's an example of naming a very large set of arguments.
Subparsers let you use the same argument name, but without being able to invoke multiple subparsers in one call that doesn't help much. And even if you could invoke several subparsers, they would still put the arguments in the same namespace. Also argparse tries to handle flaged arguments in an order independent manner. So a --foo at the end of the command line gets parsed the same as though it were at the start.
There was SO question where we discussed using argument names ('dest') like 'group1.argument1', and I've even discussed using nested namespaces. I could look those up if it would help.
Another thought - load sys.argv and partition it before passing it to one or more parsers. You could split it on some key word, or on prefixes etc.
If you have so many arguments this seems like a design issue. It seems very unmanageable. Can you not implement this using a configuration file that has reasonable set of defaults? Or defaults in the code with a reasonable (i.e. SMALL) number of arguments in the command line, and allow everything or everything else to be overridden with parameters in a 'key:value' configuration file? I can't imagine having to use a CLI with the number of variables you are proposing.