We Keep Coding

How to deal with different arguments that may have similarly named dest?

Let's say you want to use subcommands and at its core the subcommands want the same object data points to be stored in Namespace but perhaps grouped by subcommands. How can one extend argparse but not lose any of its standard behavior while achieving this?
For example:
import argparse
parser = argparse.ArgumentParser()
subparser = parser.add_subparsers()
fooparser = subparser.add_parser('foo')
fooparser.add_argument('rawr', dest='rawr')
barparser = subparser.add_parser('bar')
barparser.add_argument('rawr', dest='rawr')
# It would be nice that in the Namespace object this shows up as the following:
# args: foo 0
# Namespace(foo.rawr=0)
# args: bar 1
# Namespace(bar.rawr=1)
The above example just tries to explain my point but the main issue is that what happens is that, when the above code executes parse_args() returns a Namespace that just has rawr=N but what if my code distinguishes behavior based on the subcommand so its important that there be an object that has an attribute rawr within the Namespace object. For example:
if args.foo.rawr:
# do action 1
pass
if args.bar.rawr:
# do action 2
pass
If args only has args.rawr, then you cannot discriminate action 1 or action 2, they both are legal actions without the additional nested layer.

To save the subcommand name, use .add_subparsers(dest=), like this:
subparser = parser.add_subparsers(dest='command')
fooparser = subparser.add_parser('foo')
fooparser.add_argument('rawr')
barparser = subparser.add_parser('bar')
barparser.add_argument('rawr')
for a in ['foo', '0'], ['bar', '1']:
args = parser.parse_args(a)
print(args)
if args.command == 'foo':
print('doing foo!')
elif args.command == 'bar':
print('doing bar!')
Output:
Namespace(command='foo', rawr='0')
doing foo!
Namespace(command='bar', rawr='1')
doing bar!
Thanks to George Shuklin for pointing this out on Medium

Supplying a dest for subparser is desirable, though not required. But it may be enough to further identify the arguments.
Positionals can take any name you want to supply; you can't supply an extra dest. That name will be used in the args Namespace. Use metavar to control the string used in the help.
For flagged arguments (optionals), use the dest.
subparser = parser.add_subparsers(dest='cmd')
fooparser = subparser.add_parser('foo')
fooparser.add_argument('-b','--baz', dest='foo_baz')
fooparser.add_argument('foo_rawr', metavar='rawr')
barparser = subparser.add_parser('bar')
barparser.add_argument('-b','--baz', dest='bar_baz')
barparser.add_argument('bar_rawr', metavar='rawr')
Include a print(args) during debugging to get a clear idea of what the parser does.
In previous SO we have discussed using custom Namespace class and custom Action subclasses to create some sort of nesting or dict like behavior, but I think that's more work than most people need.
Docs also illustrate the use of
parser_foo.set_defaults(func=foo)
to set an extra argument based on the subparser. In this example the value may be an actual function object. The use of the dest is also mentioned in the docs, though perhaps as too much of an afterthought.

arparse python argument from directory

I have a file structure like this:
project/
main_prog.py
tools/
script.py
md_script/
__init__.py
md_script.py
I search in tools for local python modules. In this example it's md_script. And i want to use it as positional argument like install in my code, but when i use it, I'v got an error:
./jsh.py md_script
usage: jsh.py [-h] {install,call,list,log,restore} ... [md_script]
jsh.py: error: invalid choice: 'md_script' (choose from 'install', 'call', 'list', 'log', 'restore')
python3.4 on ubuntu14.10
Here is my code:
parser = argparse.ArgumentParser(prog='jsh.py',
description='Some help.', epilog='Example of usage: some help')
subparsers = parser.add_subparsers()
parser_install = subparsers.add_parser('install', help = 'Install new project.')
parser_install.add_argument('install', nargs='?', help = 'Name of project to be installed')
if os.path.isdir(full/path/to/tools/):
name_arg = next(os.walk(full/path/to/tools))[1]
tools_arg = parser.add_argument_group('Tools', 'Modules from tools')
for element in name_arg:
tools_arg.add_argument(element, nargs='?', help='md_script description')
args = parser.parse_args()
try:
if not len(sys.argv) > 1:
parser.print_help()
elif 'install' in args:
do_some_stuff
elif element in args:
do_some_md_script_stuff
else:
parser.print_help()

The usage line shows what's wrong:
usage: jsh.py [-h] {install,call,list,log,restore} ... [md_script]
You need to use something like
jsh.py install md_script
You specified subparsers, so you have to give it a subparser name.
From the usage it also looks like you created other subparsers, call, list, etc that you don't show in the code.
You also define positional arguments after creating subparser. That's where the [md_script] comes from. Be careful about making a lot of nargs='?' positionals (including the argument for the install subparser). This could make things confusing for your users. In fact it seems to confusing you. Remember that subparser is in effect a positional argument (one that requires 1 string).
I'd suggest experimenting with a simplier parser before creating one this complicated.
So from your comments and examples I see that you goal is let the user name a module, so your script can invoke it in some way or other. For that populating the subparsers with these names makes sense.
I wonder why you also create an optional positional argument with the same name:
module_pars = subparsers.add_parser(element, help = 'Modules from tools')
module_pars.add_argument(element, nargs='?', help=element+' description')
Is that because you are using the presence of the attribute as evidence that this subparser was invoked?
elif element in args:
do_some_md_script_stuff
The argparse documentation has a couple of other ideas.
One particularly effective way of handling sub-commands is to combine the use of the add_subparsers() method with calls to set_defaults() so that each subparser knows which Python function it should execute.
and
However, if it is necessary to check the name of the subparser that was invoked, the dest keyword argument to the add_subparsers() call will work:
These avoid the messiness of a '?' positional argument, freeing you to use subparser arguments for real information.
subparsers = parser.add_subparsers(dest='module')
....
for element in name_arg:
# module_pars = 'parser_'+element # this does nothing
module_pars = subparsers.add_parser(element, help = 'Modules from tools')
module_pars.set_defaults(func = do_some_md_script_stuff)
# or module_pars.set_default(element='I am here')
module_pars.add_argument('real_argument')
Now you can check:
if args.module='md_script':
do_some_md_script_stuff(args)
or
if hasattr(args, 'func'):
func(args)
With the alternative set_defaults, your original test should still work:
if element in args:
do_some_md_script_stuff

I did it like this. It's exactly what I want to.
if os.path.isdir(TOOLS_PATH):
name_arg = next(os.walk(TOOLS_PATH))[1]
for element in name_arg:
module_pars = 'parser_'+element
module_pars = subparsers.add_parser(element, help = 'Modules from tools')
module_pars.add_argument(element, nargs='?', help=element+' description')
I didn't test it, because i dont have a test module, but ./jsh.py md_script goes into elif element in args: print('md_script') and print string. So it looks like it works.
Thanks for all replies.
Edit: I tested it. In add_argument i must change nargs='?' for nargs='*' to catch more than one argument.
And to catch arguments from command line I used this:
elif args:
for element in name_arg:
if element in args:
element_arg = sys.argv[2:]
done_cmd,msg = opt_exec_module(element,*element_arg)
my_logger(done_cmd,msg)
Not very elegant but it works.

How to use nosetests in python while also passing/accepting arguments for argparse?

I want to use nose and coverage in my project. When I run nose with --with-coverage argument, my programs argument-parsing module goes nuts because "--with-coverage" isn't a real argument according to it.
How do I turn the argparse off, but during testing only? Nose says all my tests fail because of the bad argument.

I actually just ran into this issue myself the other day. You don't need to "disable" your parsing module or anything. What you can do is change the module that uses argparse to ignore those arguments it receives that it doesn't recognize. That way they can still be used by other scripts (for example if your command-line call passes secondary arguments to another program execution).
Without your code, I'll assume you're using the standard parse_args() method on your argparse.ArgumentParser instance. Replace it with parse_known_args() instead.
Then, whenever you subsequently reference the parsed-arguments Namespace object, you'll need to specify and element, specifically 0. While parse_args() returns the args object alone, parse_known_args() returns tuple: the first element is the parsed known arguments, and the latter element contains the ignored unrecognized arguments (which you can later use/pass in your Python code, if necessary).
Here's the example change from my own project:
class RunArgs(object):
'''
A placeholder for processing arguments passed to program execution.
'''
def __init__(self):
self.getArgs()
#self.pause = self.args.pause # old assignment
self.pause = self.args[0].pause # new assignment
#...
def __repr__(self):
return "<RunArgs(t=%s, #=%s, v=%s)>" % (str(x) for x in (self.pause,self.numreads,self.verbose))
def getArgs(self):
global PAUSE_TIME
global NUM_READS
parser = argparse.ArgumentParser()
parser.add_argument('-p', '--pause', required=False,
type=self.checkPauseArg, action='store', default=PAUSE_TIME)
parser.add_argument('-n', '--numreads', required=False,
type=self.checkNumArg, action='store', default=NUM_READS)
parser.add_argument('-v', '--verbose', required=False,
action='store_true')
#self.args = parser.parse_args() # old parse call
self.args = parser.parse_known_args() # new parse call
#...

I've read that you can use nose-testconfig, or otherwise use mock to replace the call (not test it). Though I'd agree with #Ned Batchelder, it begs questioning the structure of the problem.

As a workaround, instead of running nose with command-line arguments, you can have a .noserc or nose.cfg in the current working directory:
[nosetests]
verbosity=3
with-coverage=1
Though, I agree that parse_known_args() is a better solution.

It sounds like you have tests that run your code, and then your code uses argparse which implicitly pulls arguments from sys.argv. This is a bad way to structure your code. Your code under test should be getting arguments passed to it some other way so that you can control what arguments it sees.
This is an example of why global variables are bad. sys.argv is a global, shared by the entire process. You've limited the modularity, and therefore the testability, of your code by relying on that global.

Not recognizing command line argument in Python

I am trying to add a command line argument depending on some values returned by the functions. When I am not giving that argument it says:
main.py: error: argument -opp/--operator is required
When I am giving the argument it says:
main.py: error: unrecognized arguments: -opp +
Following is the piece of EDITED code (as told in one of the answers):
parser.add_argument('-z', help='Help msg', required=True)
args, unknown = parser.parse_known_args()
value = some_functions(args.z)
if value == some_particular_value:
parser.add_argument('-opp','--operator',help='Some help msg',required=True)
args = parser.parse_args()
Please help me in adding this argument. Thanks!

There are, though, a couple of mistakes in your code. Here's the corrected version:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-z', help = 'Help msg', required = True)
args, _ = parser.parse_known_args()
# value = some_functions(args.z)
if value == some_particular_value:
parser.add_argument('-opp', '--operator', help = 'Some help msg', required = True)
# args2, _ = parser.parse_known_args()
# some_function2(args2.operator)
So, let's analyse your mistakes:
Assigning instead of comparing
That's typical newbie mistake. Within a conditional operator (if, case...) you set the value instead of checking it. The difference is in amount of the = sign.
If you assign value, the condition in the operator will be always True and test will always succeed (in most of programming languages and cases).
Check this out:
a = 1
if a = 2:
print a
This may print 2 in some languages (like C or Java; using the correct syntax). Why? You've just set it! Yet, Python is smart enough to tell you about your mistake:
File "<stdin>", line 1
if a = 2:
^
SyntaxError: invalid syntax
And compare it to this:
a = 1
if a == 2:
print a
This will not print anything. Because the if test did not pass.
Assigning value instead of calling method
You want to be using the method add_argument instead of re-defining the parser variable, right?
parser = add_argument(...)
That's something like I've described above. You should be calling a method of a parser variable, not defining its new value:
parser.add_argument(...)
Re-parsing arguments is missing?
You did not show the part of the code where you check for the operator argument. Note: you should parse your arguments again, when defined a new argument:
parser.add_argument(...)
args, _ = parser.parse_known_arguments()
Then you will get a new argument in the args variable.
Using the wrong name of argument?
Again, you are missing part of code, where you check for the operator argument' value. If you are trying to access it with
args.opp # whoops...
Then you'd just get an error saying There's no argument 'opp'!, because it has its full name and should be accessed with it:
args.operator # aaah, here it is!

Argparse: ignore multiple positional arguments when optional argument is specified

I'm trying to make argparse ignore the fact that two normally required positional arguments shouldn't be evaluated when an optional argument (-l) is specified.
Basically I'm trying to replicate the behavior of --help: when you specify the -h, all missing required arguments are ignored.
Example code:
parser = argparse.ArgumentParser(description="Foo bar baz")
parser.add_argument('arg1', help='arg1 is a positional argument that does this')
parser.add_argument('arg2', help='arg2 is a positional argument that does this')
parser.add_argument('-l', '--list', dest='list', help='this is an optional argument that prints stuff')
options, args = parser.parse_args()
if options.list:
print "I list stuff"
And of course, if I run it now, I get :
error: too few arguments
I tried different things like nargs='?', but couldn't get anything working.
This question is quite similar but wasn't answered.

Unfortunately, argparse isn't quite flexible enough for this. The best you can do is to make arg1 and arg2 optional using nargs="?" and check yourself whether they are given if needed.
The internal help action is implemented by printing the help message and exiting the program as soon as -h or --help are encountered on the command line. You could write a similar action yourself, something like
class MyAction(argparse.Action):
def __call__(self, parser, values, namespace, option_string):
print "Whatever"
parser.exit()
(Warning: untested code!)
There are definite downsides to the latter approac, though. The help message will unconditionally show arg1 and arg2 as compulsory arguments. And parsing the command line simply stops when encountering -l or --list, ignoring any further arguments. This behaviour is quite acceptable for --help, but is less than desirable for other options.

I ran into this issue and decided to use subcommands. Subcommands might be overkill, but if you find your program not using some of the positional arguments in many instances (as I did), then subcommands might be a good solution.
For your given example, I'd use something like the following:
parser = argparse.ArgumentParser(description="Foo bar baz")
subparsers = parser.add_subparsers(description='available subcommands')
parser_main = subparsers.add_parser('<main_command_name>')
parser_main.add_argument('arg1', help='arg1 is a positional argument that does this')
parser_main.add_argument('arg2', help='arg2 is a positional argument that does this')
parser_list = subparsers.add_parser('list', help='this is a subcommand that prints stuff')
options, args = parser.parse_args()
I left out some details that you might want to include (like set_defaults(func=list)), which are mentioned in the argparse documentation.

I think the nargs='*' is helpful.
Positional arguments is ignorable, then you can use if to check the positional arguments is true or false.
http://docs.python.org/library/argparse.html#nargs

The cleanest approach I've been able to find so far is to split the parsing into two stages. First check for -l/--list:
parser = argparse.ArgumentParser(description="Foo bar baz")
parser.add_argument('-l', '--list', dest='list', action='store_true',
help='this is an optional argument that prints stuff')
options, remainder = parser.parse_known_args()
Now, since you used parse_known_args, you won't get an error up to here, and you can decide what to do with the remainder of the arguments:
if options.list:
print "I list stuff"
else:
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument('arg1', help='arg1 is a positional argument that does this')
parser.add_argument('arg2', help='arg2 is a positional argument that does this')
options = parser.parse_args(remainder)
You may want to set the usage option in the first parser to make the help string a bit nicer.

I may have found a solution here. True, it is a dirty hack, but it works.
Note: all the following applies to Python 3.3.2.
As per the answer here, parse_args checks which actions are required and throws an error if any of them are missing. I propose to override this behavior.
By subclassing ArgumentParser we can define a new ArgumentParser.error method (original here) that will check whether the error was thrown because some arguments are missing and take necessary action. Code follows:
import argparse
import sys
from gettext import gettext as _
class ArgumentParser(argparse.ArgumentParser):
skip_list = []
def error(self, message):
# Let's see if we are missing arguments
if message.startswith('the following arguments are required:'):
missingArgs = message.split('required: ')[1].split(', ')
newArgs = [] # Creating a list of whatever we should not skip but is missing
for arg in missingArgs:
if arg not in self.skip_list:
newArgs.append(arg)
else:
self.skip_list.remove(arg) # No need for it anymore
if len(newArgs) == 0:
return # WARNING! UNTESTED! MAY LEAD TO SPACETIME MELTDOWN!
else: # Some required stuff is still missing, so we show a corrected error message
message = _('the following arguments are required: %s') % ', '.join(newArgs)
self.print_usage(sys.stderr) # Original method behavior
args = {'prog': self.prog, 'message': message}
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
The new method first checks whether the error is because arguments are missing from the command line (see here for the code that generates the error). If so, the method gets the names of the arguments from the error message and puts them into a list (missingArgs).
Then, we iterate over this list and check which arguments should be skipped, and which are still required. To determine which arguments to skip, we compare them against skip_list. It is a field in our ArgumentParser subclass that should contain the names of the arguments to skip even when they are required by the parser. Please note that arguments that happen to be in skip_list are removed from it when they are found.
If there are still required arguments that are missing from the command line, the method throws a corrected error message. If all the missing arguments should be skipped, however, the method returns.
WARNING! The original definition of ArgumentParser.error states that if it is overridden in a subclass it should not return, but rather exit or raise an exception. Therefore, what is shown here is potentially unsafe and may cause your program to crash, your computer to catch fire or worse - IT MAY EVAPORATE ALL YOUR TEA. However, it seems like in this particular case (missing required arguments) it is safe to return from the method. But it might not be. You have been warned.
In order to fill skip_list we could use code like this:
class SpecialHelp(argparse._HelpAction):
def __call__(self, parser, namespace, values, option_string=None):
parser.print_help()
print()
for action in parser._actions:
if action != self and action.required:
parser.skip_list.append(argparse._get_action_name(action))
This particular class imitates the built-in help action, but instead of exiting it inserts all the remaining required arguments into skip_list.
Hope my answer helps and best of luck.

It may be ugly, but that is what I normally do.
def print_list():
the_list = ["name1", "name2"]
return "{0}".format(the_list)
...
parser.add_argument("-l", "--list", action='version',
version=print_list(), help="print the list")