We Keep Coding

Parsing argparse input bit by bit

I am using Argparse to parse shell input to my Python function.
The tricky part is that this script first reads in a file that partially determines what kind of arguments are available to Argparse (it's a JSON file containing criteria by which the user can specify what data to output).
But before these arguments are added to my parser, I would like to read in some arguments relating to the file reading itself. (e.g. whether to fix the formatting of the input file). Kinda like this:
test.py (fix_formatting=True, **more arguments added later)
When I try to run args = parser.parse_args() twice, after the initial input and after adding more keys, things fall apart: Argparse quite predictably complains that some of the user input are unrecognized arguments:. I thought I might use subparsers to that end.
So I tried variations of (following the example in the docs as best as I could):
def main():
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(help='sub-command help')
settingsparser = subparsers.add_parser('settings') #i want a subparser called 'settings'
settingsparser.add_argument('--fix_formatting', action='store_true') #this subparser shall have a --fix_formatting
Then I try to parse only the "settings" part like so:
settings=parser.parse_args(['settings'])
This seems to work. But then I add my other keys and things break:
keys=['alpha','beta','gamma','delta']
for key in keys:
parser.add_argument("--"+key, type=str, help="X")
args = parser.parse_args()
If I parse any input for any of the arguments from keys, Argparse complains that I make an invalid choice: [...] (choose from 'settings'). Now I don't understand why I have to choose from "settings"; the docs say that the parse
will only contain attributes for the main parser and the subparser that was selected by the command line (and not any other subparsers)
what is my error of understanding here?
and if this is the wrong approach, how would one go about parsing one bit of input before another bit?
Any help is much appreciated!

parse_args calls parse_known_args. This returns the args namesparse along with a list of strings (from sys.argv) that it could not process (extras). parse_args raises this error if this list is not empty.
https://docs.python.org/3/library/argparse.html#partial-parsing
Thus parse_known_args is useful if you want to parse some of the input.
sys.argv remains unchanged. Subsequent calls to a parser (whether it was the original one or not) use that again, unless you pass the extras.
I don't think subparsers help you here. They aren't meant for delayed or two stage parsing. I'd suggest playing with the documentation examples for subparsers first.
To the main parser, the subparsers look like
subparsers = parser.add_argument('cmd', choices=['select',...])
In other words, it adds a positional argument where the choices are the subparser names that you define. That may help you see why it expects you to name select. Positionals are normally required.
(there's an exception to this in recent versions, https://stackoverflow.com/a/22994500/901925)

Python argparse : how to detect duplicated optional argument?

I'm using argparse with optional parameter, but I want to avoid having something like this : script.py -a 1 -b -a 2
Here we have twice the optional parameter 'a', and only the second parameter is returned. I want either to get both values or get an error message.
How should I define the argument ?
[Edit]
This is the code:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', dest='alpha', action='store', nargs='?')
parser.add_argument('-b', dest='beta', action='store', nargs='?')
params, undefParams = self.parser.parse_known_args()

append action will collect the values from repeated use in a list
parser.add_argument('-a', '--alpha', action='append')
producing an args namespace like:
namespace(alpha=['1','3'], b='4')
After parsing you can check args.alpha, and accept or complain about the number of values. parser.error('repeated -a') can be used to issue an argparse style error message.
You could implement similar functionality in a custom Action class, but that requires understanding the basic structure and operation of such a class. I can't think anything that can be done in an Action that can't just as well be done in the appended list after.
https://stackoverflow.com/a/23032953/901925 is an answer with a no-repeats custom Action.
Why are you using nargs='?' with flagged arguments like this? Without a const parameter this is nearly useless (see the nargs=? section in the docs).
Another similar SO: Python argparse with nargs behaviour incorrect

How to use nosetests in python while also passing/accepting arguments for argparse?

I want to use nose and coverage in my project. When I run nose with --with-coverage argument, my programs argument-parsing module goes nuts because "--with-coverage" isn't a real argument according to it.
How do I turn the argparse off, but during testing only? Nose says all my tests fail because of the bad argument.

I actually just ran into this issue myself the other day. You don't need to "disable" your parsing module or anything. What you can do is change the module that uses argparse to ignore those arguments it receives that it doesn't recognize. That way they can still be used by other scripts (for example if your command-line call passes secondary arguments to another program execution).
Without your code, I'll assume you're using the standard parse_args() method on your argparse.ArgumentParser instance. Replace it with parse_known_args() instead.
Then, whenever you subsequently reference the parsed-arguments Namespace object, you'll need to specify and element, specifically 0. While parse_args() returns the args object alone, parse_known_args() returns tuple: the first element is the parsed known arguments, and the latter element contains the ignored unrecognized arguments (which you can later use/pass in your Python code, if necessary).
Here's the example change from my own project:
class RunArgs(object):
'''
A placeholder for processing arguments passed to program execution.
'''
def __init__(self):
self.getArgs()
#self.pause = self.args.pause # old assignment
self.pause = self.args[0].pause # new assignment
#...
def __repr__(self):
return "<RunArgs(t=%s, #=%s, v=%s)>" % (str(x) for x in (self.pause,self.numreads,self.verbose))
def getArgs(self):
global PAUSE_TIME
global NUM_READS
parser = argparse.ArgumentParser()
parser.add_argument('-p', '--pause', required=False,
type=self.checkPauseArg, action='store', default=PAUSE_TIME)
parser.add_argument('-n', '--numreads', required=False,
type=self.checkNumArg, action='store', default=NUM_READS)
parser.add_argument('-v', '--verbose', required=False,
action='store_true')
#self.args = parser.parse_args() # old parse call
self.args = parser.parse_known_args() # new parse call
#...

I've read that you can use nose-testconfig, or otherwise use mock to replace the call (not test it). Though I'd agree with #Ned Batchelder, it begs questioning the structure of the problem.

As a workaround, instead of running nose with command-line arguments, you can have a .noserc or nose.cfg in the current working directory:
[nosetests]
verbosity=3
with-coverage=1
Though, I agree that parse_known_args() is a better solution.

It sounds like you have tests that run your code, and then your code uses argparse which implicitly pulls arguments from sys.argv. This is a bad way to structure your code. Your code under test should be getting arguments passed to it some other way so that you can control what arguments it sees.
This is an example of why global variables are bad. sys.argv is a global, shared by the entire process. You've limited the modularity, and therefore the testability, of your code by relying on that global.

argparse key=value parameters

This first link has the same question in the first section, but it is unanswered
(python argparse: parameter=value). And this second question is similar, but I can't seem to get it working for my particular case
( Using argparse to parse arguments of form "arg= val").
So my situation is this -- I am re-writing a Python wrapper which is used by many other scripts (I would prefer not to modify these other scripts). Currently, the Python wrapper is called with command line arguments of the form --key=value for a number of different arguments, but was parsed manually. I would like to parse them with argparse.
N.B. The argument names are unwieldy, so I am renaming using the dest option in add_argument.
parser = argparse.ArgumentParser(description='Wrappin Ronnie Reagan')
parser.add_argument("--veryLongArgName1", nargs=1, dest="arg1", required=True)
parser.add_argument("--veryLongArgName2", nargs=1, dest="arg2")
parser.add_argument("--veryLongArgName3", nargs=1, dest="arg3")
userOpts = vars(parser.parse_args())
Which, while apparently parsing the passed command lines correctly, displays this as the help:
usage: testing_argsparse.py [-h] --veryLongArgName1 ARG1
[--veryLongArgName2 ARG2]
[--veryLongArgName3 ARG3]
testing_argsparse.py: error: argument --veryLongArgName1 is required
But what I want is that all parameters are specified with the --key=value format, not --key value. i.e.
usage: testing_argsparse.py [-h] --veryLongArgName1=ARG1
[--veryLongArgName2=ARG2]
[--veryLongArgName3=ARG3]
testing_argsparse.py: error: argument --veryLongArgName1 is required

testing_argsparse.py --veryLongArgName1=foo
works. argparse module accepts both --veryLongArgName1=foo and --veryLongArgName1 foo formats.
What exact command line arguments are you trying to pass to argparse that's causing it to not work?

A little late but for anyone with a similar request as the OP you could use a custom HelpFormatter.
class ArgFormatter(argparse.HelpFormatter):
def _format_args(self, *args):
result = super(ArgFormatter, self)._format_args(*args)
return result and '%%%' + result
def _format_actions_usage(self, *args):
result = super(ArgFormatter, self)._format_actions_usage(*args)
return result and result.replace(' %%%', '=')
This can then be passed to ArgumentParser to give the wanted behavior.
parser = argparse.ArgumentParser(
description='Wrappin Ronnie Reagan',
formatter_class=ArgFormatter)
This intercepts the args (ARG1, ARG2, ...) and adds a custom prefix which is later replaced (along with the unwanted space) for an = symbol. The and in the return statements makes sure to only modify the result if it's non-empty.

Argparse: ignore multiple positional arguments when optional argument is specified

I'm trying to make argparse ignore the fact that two normally required positional arguments shouldn't be evaluated when an optional argument (-l) is specified.
Basically I'm trying to replicate the behavior of --help: when you specify the -h, all missing required arguments are ignored.
Example code:
parser = argparse.ArgumentParser(description="Foo bar baz")
parser.add_argument('arg1', help='arg1 is a positional argument that does this')
parser.add_argument('arg2', help='arg2 is a positional argument that does this')
parser.add_argument('-l', '--list', dest='list', help='this is an optional argument that prints stuff')
options, args = parser.parse_args()
if options.list:
print "I list stuff"
And of course, if I run it now, I get :
error: too few arguments
I tried different things like nargs='?', but couldn't get anything working.
This question is quite similar but wasn't answered.

Unfortunately, argparse isn't quite flexible enough for this. The best you can do is to make arg1 and arg2 optional using nargs="?" and check yourself whether they are given if needed.
The internal help action is implemented by printing the help message and exiting the program as soon as -h or --help are encountered on the command line. You could write a similar action yourself, something like
class MyAction(argparse.Action):
def __call__(self, parser, values, namespace, option_string):
print "Whatever"
parser.exit()
(Warning: untested code!)
There are definite downsides to the latter approac, though. The help message will unconditionally show arg1 and arg2 as compulsory arguments. And parsing the command line simply stops when encountering -l or --list, ignoring any further arguments. This behaviour is quite acceptable for --help, but is less than desirable for other options.

I ran into this issue and decided to use subcommands. Subcommands might be overkill, but if you find your program not using some of the positional arguments in many instances (as I did), then subcommands might be a good solution.
For your given example, I'd use something like the following:
parser = argparse.ArgumentParser(description="Foo bar baz")
subparsers = parser.add_subparsers(description='available subcommands')
parser_main = subparsers.add_parser('<main_command_name>')
parser_main.add_argument('arg1', help='arg1 is a positional argument that does this')
parser_main.add_argument('arg2', help='arg2 is a positional argument that does this')
parser_list = subparsers.add_parser('list', help='this is a subcommand that prints stuff')
options, args = parser.parse_args()
I left out some details that you might want to include (like set_defaults(func=list)), which are mentioned in the argparse documentation.

I think the nargs='*' is helpful.
Positional arguments is ignorable, then you can use if to check the positional arguments is true or false.
http://docs.python.org/library/argparse.html#nargs

The cleanest approach I've been able to find so far is to split the parsing into two stages. First check for -l/--list:
parser = argparse.ArgumentParser(description="Foo bar baz")
parser.add_argument('-l', '--list', dest='list', action='store_true',
help='this is an optional argument that prints stuff')
options, remainder = parser.parse_known_args()
Now, since you used parse_known_args, you won't get an error up to here, and you can decide what to do with the remainder of the arguments:
if options.list:
print "I list stuff"
else:
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument('arg1', help='arg1 is a positional argument that does this')
parser.add_argument('arg2', help='arg2 is a positional argument that does this')
options = parser.parse_args(remainder)
You may want to set the usage option in the first parser to make the help string a bit nicer.

I may have found a solution here. True, it is a dirty hack, but it works.
Note: all the following applies to Python 3.3.2.
As per the answer here, parse_args checks which actions are required and throws an error if any of them are missing. I propose to override this behavior.
By subclassing ArgumentParser we can define a new ArgumentParser.error method (original here) that will check whether the error was thrown because some arguments are missing and take necessary action. Code follows:
import argparse
import sys
from gettext import gettext as _
class ArgumentParser(argparse.ArgumentParser):
skip_list = []
def error(self, message):
# Let's see if we are missing arguments
if message.startswith('the following arguments are required:'):
missingArgs = message.split('required: ')[1].split(', ')
newArgs = [] # Creating a list of whatever we should not skip but is missing
for arg in missingArgs:
if arg not in self.skip_list:
newArgs.append(arg)
else:
self.skip_list.remove(arg) # No need for it anymore
if len(newArgs) == 0:
return # WARNING! UNTESTED! MAY LEAD TO SPACETIME MELTDOWN!
else: # Some required stuff is still missing, so we show a corrected error message
message = _('the following arguments are required: %s') % ', '.join(newArgs)
self.print_usage(sys.stderr) # Original method behavior
args = {'prog': self.prog, 'message': message}
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
The new method first checks whether the error is because arguments are missing from the command line (see here for the code that generates the error). If so, the method gets the names of the arguments from the error message and puts them into a list (missingArgs).
Then, we iterate over this list and check which arguments should be skipped, and which are still required. To determine which arguments to skip, we compare them against skip_list. It is a field in our ArgumentParser subclass that should contain the names of the arguments to skip even when they are required by the parser. Please note that arguments that happen to be in skip_list are removed from it when they are found.
If there are still required arguments that are missing from the command line, the method throws a corrected error message. If all the missing arguments should be skipped, however, the method returns.
WARNING! The original definition of ArgumentParser.error states that if it is overridden in a subclass it should not return, but rather exit or raise an exception. Therefore, what is shown here is potentially unsafe and may cause your program to crash, your computer to catch fire or worse - IT MAY EVAPORATE ALL YOUR TEA. However, it seems like in this particular case (missing required arguments) it is safe to return from the method. But it might not be. You have been warned.
In order to fill skip_list we could use code like this:
class SpecialHelp(argparse._HelpAction):
def __call__(self, parser, namespace, values, option_string=None):
parser.print_help()
print()
for action in parser._actions:
if action != self and action.required:
parser.skip_list.append(argparse._get_action_name(action))
This particular class imitates the built-in help action, but instead of exiting it inserts all the remaining required arguments into skip_list.
Hope my answer helps and best of luck.

It may be ugly, but that is what I normally do.
def print_list():
the_list = ["name1", "name2"]
return "{0}".format(the_list)
...
parser.add_argument("-l", "--list", action='version',
version=print_list(), help="print the list")