How can I get an argparse subparser to only parse arguments for it's own arguments? It seems that calling parse_known_args on the subparser parses all the parent's arguments.
Given this simplified version, and the inputs foo bar:
main_parser = argparse.ArgumentParser()
main_parser.add_argument("command")
args, subc_args = main_parser.parse_known_args()
print("main parser = {}".format(args))
sub_parser = main_parser.add_subparsers()
sub_command_parser = sub_parser.add_parser("sub_command")
sub_command_parser.add_argument("hostname")
sub_args = sub_command_parser.parse_known_args()
print("sub parser = {}".format(sub_args))
The first print returns what I expect:
main parser = Namespace(command='foo')
Whereas the penultimate line returns the same thing, only with the extra argument as part of the "unknown" list:
sub parser = (Namespace(hostname='foo'), ['bar'])
How can I get something like sub_command_parser.parse_known_args() to ignore the arguments that were given before it? What I need is for sub_args to only contain those arguments that were added with sub_command_parser.add_argument(). I can call them directly afterwards like this; sub_args[1], but that seems hacky and unreliable.
Background: I have a package split up into a main file as the entry point which handles the top level arguments, and the modules that do the actual work. I want to add a top level "main parser" in the main file, and sub-parsers in the modules.
Normal subcommands usage, all in one file:
main_parser = argparse.ArgumentParser()
sub_parser = main_parser.add_subparsers(dest='command')
sub_command_parser = sub_parser.add_parser("sub_command")
sub_command_parser.add_argument("hostname")
args = main_parser.parse_args()
print(args)
args.command # name of the subcommand
args.hostname # value from subcommand argument
This automatically passes the remaining strings to the subparser, the named in command. This usage should be clear in the Python documentation.
If you are going to split the parsers between files, and call them separately, skip add_subparsers bit. It doesn't do anything for you.
# Main file
#
parser = argparse.ArgumentParser()
parser.add_argument("command")
args, subc_args = parser.parse_known_args()
print("main parser = {}".format(args.command))
# Subcommand file
#
parser = argparse.ArgumentParser()
parser.add_argument("hostname")
sub_args, unknown_args = parser.parse_known_args(subc_args)
print("sub parser = {}".format(sub_args.hostname))
That helped, thanks #hpaulj. Looks like there isn't a good way to handle subcommands with argparse. The full working example for people stumbling on this is:
# Main file
#
main_parser = argparse.ArgumentParser()
main_parser.add_argument("command")
args, subc_args = main_parser.parse_known_args()
print("main parser = {}".format(args.command))
# Subcommand file
#
sub_parser = main_parser.add_subparsers()
sub_command_parser = sub_parser.add_parser("sub_command")
sub_command_parser.add_argument("hostname")
sub_args, unknown_args = sub_command_parser.parse_known_args(subc_args)
print("sub parser = {}".format(sub_args.hostname))
Note that when calling the class (or however you've implemented the subparser creation in the subcommand file), you'll need to supply the subc_args too, as optional.
EDIT: Specifying add_help=False in the main parser gets around this :)
I need to use sys.argv to check for an argument from the command line, which would be the filename in my case. My code is as follows. I'm not allowed to import argparse, only allowed to use sys. I know I'm doing something wrong here. Appreciate any help.
def get_inputfile_object( ):
'''
Check the command line for an argument. If one was there, use it as the
filename. Otherwise, use DEFAULT_INPUT_FILENAME. Open the file.
If file is successfully opened:
print MSG_OPENING_FILE
Return: a file object for that file
If the file cannot be opened:
print MSG_ERROR_OPENNING_FILE
Return: True
'''
if sys.argv > 1:
pass
else:
input_filename = DEFAULT_INPUT_FILENAME
input_filename = DEFAULT_INPUT_FILENAME
if os.path.isfile(input_filename) and os.access(input_filename,os.R_OK):
#Prints the opening file message, and the name of the file
print (MSG_OPENING_FILE,input_filename)
return open(input_filename,'r')
else:
print (MSG_ERROR_OPENING_FILE)
return True
sys.argv is a list of arguments.
You need to check the length of the list:
if len(sys.argv) > 1:
You should check out argparse.
The argparse module also automatically generates help and usage
messages and issues errors when users give the program invalid
arguments.
Haven't tested it, but you can try something similar to this:
import argparse
# setup the parser
parser = argparse.ArgumentParser(description='Describe script')
# add positional argument
parser.add_argument('filename', type=str, help='filename description')
# parse the args
args = parser.parse_args()
print(args.filename)
I want to parse the cmdline option to be -option key=value type in argparse.
For example:
script.py -project prj1=rev1
Generally:
script.py -project prj1 --> OK
script.py -project=prj1 --> OK
script.py -project prj1=rev1 --> How to flag that argument value should be in str=str format.
script.py -project=prj1,prj2 --> How to flag that we need comma separated strings.
In the above, -project is the option. proj1=rev1 in this way I want the argument to be present. It should flag an error if it is not in the proper format and print the help message. I can use regular expression once I collect the project value using (\w)=(\w). If not in the above format can flag an error. But is there a way to filter out this and flag an error at parsing the cmdline arguments itself?
You can take advantage of this fact from the argparse documentation:
type= can take any callable that takes a single string argument and returns the converted value:
For example, to support the first format (--project prj1=rev1) you could do something like:
import os
import sys
import argparse
def handle_kv_string(val):
if '=' in val:
return val.split('=')
else:
raise argparse.ArgumentTypeError('Must specify k=v')
def parse_args():
p = argparse.ArgumentParser()
p.add_argument('--project',
type=handle_kv_string)
return p.parse_args()
def main():
args = parse_args()
print args
if __name__ == '__main__':
main()
This gets you, with valid arguments:
$ ./argtest --project foo=bar
Namespace(project=['foo', 'bar'])
And with invalid arguments:
$ ./argtest --project foo
usage: argtest.py [-h] [--project PROJECT]
argtest.py: error: argument --project: Must specify k=v
You could apply a similar solution to your second example.
I spent few hours reading tutorials about argparse and managed to learn to use normal parameters. The official documentation is not very readable to me. I'm new to Python. I'm trying to write a program that could be invoked in following ways:
cat inFile | program [options] > outFile -- If no inFile or outfile is specified, read from stdin and output to stdout.
program [options] inFile outFile
program [options] inFile > outFile -- If only one file is specified it is input and output should go to stdout.
cat inFile | program [options] - outFile -- If '-' is given in place of inFlie read from stdin.
program [options] /path/to/folder outFile -- Process all files from /path/to/folder and it subdirectories.
I want it to behave like regular cli program under GNU/Linux.
It would be also nice if the program would be able to be invoked:
program [options] inFile0 inFile1 ... inFileN outFile -- first path/file always interpreted as input, last one always interpreted as output. Any additional ones interpreted as inputs.
I could probably write dirty code that would accomplish this but this is going to be used, so someone will end up maintaining it (and he will know where I live...).
Any help/suggestions are much appreciated.
Combining answers and some more knowledge from the Internet I've managed to write this(it does not accept multiple inputs but this is enough):
import sys, argparse, os.path, glob
def inputFile(path):
if path == "-":
return [sys.stdin]
elif os.path.exists(path):
if os.path.isfile(path):
return [path]
else:
return [y for x in os.walk(path) for y in glob.glob(os.path.join(x[0], '*.dat'))]
else:
exit(2)
def main(argv):
cmdArgsParser = argparse.ArgumentParser()
cmdArgsParser.add_argument('inFile', nargs='?', default='-', type=inputFile)
cmdArgsParser.add_argument('outFile', nargs='?', default='-', type=argparse.FileType('w'))
cmdArgs = cmdArgsParser.parse_args()
print cmdArgs.inFile
print cmdArgs.outFile
if __name__ == "__main__":
main(sys.argv[1:])
Thank you!
You need a positional argument (name not starting with a dash), optional arguments (nargs='?'), a default argument (default='-'). Additionally, argparse.FileType is a convenience factory to return sys.stdin or sys.stdout if - is passed (depending on the mode).
All together:
#!/usr/bin/env python
import argparse
# default argument is sys.argv[0]
parser = argparse.ArgumentParser('foo')
parser.add_argument('in_file', nargs='?', default='-', type=argparse.FileType('r'))
parser.add_argument('out_file', nargs='?', default='-', type=argparse.FileType('w'))
def main():
# default argument is is sys.argv[1:]
args = parser.parse_args(['bar', 'baz'])
print(args)
args = parser.parse_args(['bar', '-'])
print(args)
args = parser.parse_args(['bar'])
print(args)
args = parser.parse_args(['-', 'baz'])
print(args)
args = parser.parse_args(['-', '-'])
print(args)
args = parser.parse_args(['-'])
print(args)
args = parser.parse_args([])
print(args)
if __name__ == '__main__':
main()
I'll give you a start script to play with. It uses optionals rather than positionals. and only one input file. But it should give a taste of what you can do.
import argparse
parser = argparse.ArgumentParser()
inarg = parser.add_argument('-i','--infile', type=argparse.FileType('r'), default='-')
outarg = parser.add_argument('-o','--outfile', type=argparse.FileType('w'), default='-')
args = parser.parse_args()
print(args)
cnt = 0
for line in args.infile:
print(cnt, line)
args.outfile.write(line)
cnt += 1
When called without arguments, it just echos your input (after ^D). I'm a little bothered that it doesn't exit until I issue another ^D.
FileType is convenient, but has the major fault - it opens the files, but you have to close them yourself, or let Python do so when exiting. There's also the complication that you don't want to close stdin/out.
The best argparse questions include a basic script, and specific questions on how to correct or improve it. Your specs are reasonably clear. but it would be nice if you gave us more to work with.
To handle the subdirectories option, I would skip the FileType bit. Use argparse to get 2 lists of strings (or a list and an name), and then do the necessary chgdir and or glob to find and iterate over files. Don't expect argparse to do the actual work. Use it to parse the commandline strings. Here a sketch of such a script, leaving most details for you to fill in.
import argparse
import os
import sys # of stdin/out
....
def open_output(outfile):
# function to open a file for writing
# should handle '-'
# return a file object
def glob_dir(adir):
# function to glob a dir
# return a list of files ready to open
def open_forread(afilename):
# function to open file for reading
# be sensitive to '-'
def walkdirs(alist):
outlist = []
for name in alist:
if <name is file>;
outlist.append(name)
else <name is a dir>:
glist = glob(dir)
outlist.extend(glist)
else:
<error>
return outlist
def cat(infile, outfile):
<do your thing here>
def main(args):
# handle args options
filelist = walkdirs(args.inlist)
fout = open_outdir(args.outfile)
for name in filelist:
fin = open_forread(name)
cat(fin,fout)
if <fin not stdin>: fin.close()
if <fout not stdout>: fout.close()
if '__name__' == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('inlist', nargs='*')
parser.add_argument('outfile')
# add options
args = parser.parse_args()
main(args)
The parser here requires you to give it an outfile name, even if it is '-'. I could define its nargs='?' to make it optional. But that does not play nicely with the 'inlist` '*'.
Consider
myprog one two three
Is that
namespace(inlist=['one','two','three'], outfile=default)
or
namespace(inlist=['one','two'], outfile='three')
With both a * and ? positional, the identity of the last string is ambiguous - is it the last entry for inlist, or the optional entry for outfile? argparse chooses the former, and never assigns the value to outfile.
With --infile, --outfile definitions, the allocation of these strings is clear.
In sense this problem is too complex for argparse - there's nothing in it to handle things like directories. In another sense it is too simple. You could just as easily split sys.argv[1:] between inlist and outfile without the help of argparse.
I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.
Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.
I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.
Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values...
I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:
Default with no user input:
$ ./argparse-partial.py
Option is "default"
Default from configuration file:
$ cat argparse-partial.config
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config
Option is "Hello world!"
Default from configuration file, overridden by commandline:
$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"
argprase-partial.py follows. It is slightly complicated to handle -h for help properly.
import argparse
import ConfigParser
import sys
def main(argv=None):
# Do argv default this way, as doing it in the functional
# declaration sets it at compile time.
if argv is None:
argv = sys.argv
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
description=__doc__, # printed with -h/--help
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file", metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args()
defaults = { "option":"default" }
if args.conf_file:
config = ConfigParser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument("--option")
args = parser.parse_args(remaining_argv)
print "Option is \"{}\"".format(args.option)
return(0)
if __name__ == "__main__":
sys.exit(main())
Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.
I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:
subp.set_defaults(**dict(conffile.items(subn)))
This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.
A more complete example is below:
####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost
import ConfigParser
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')
parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')
conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')
for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
subp.set_defaults(**dict(conffile.items(subn)))
print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
I can't say it's the best way, but I have an OptionParser class that I made that does just that - acts like optparse.OptionParser with defaults coming from a config file section. You can have it...
class OptionParser(optparse.OptionParser):
def __init__(self, **kwargs):
import sys
import os
config_file = kwargs.pop('config_file',
os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
self.config_section = kwargs.pop('config_section', 'OPTIONS')
self.configParser = ConfigParser()
self.configParser.read(config_file)
optparse.OptionParser.__init__(self, **kwargs)
def add_option(self, *args, **kwargs):
option = optparse.OptionParser.add_option(self, *args, **kwargs)
name = option.get_opt_string()
if name.startswith('--'):
name = name[2:]
if self.configParser.has_option(self.config_section, name):
self.set_default(name, self.configParser.get(self.config_section, name))
Feel free to browse the source. Tests are in a sibling directory.
Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.
One issue with the current is that it will not error if the arguments in the config file are invalid. Here's a version with a different downside: you'll need to include the -- or - prefix in the keys.
Here's the python code (Gist link with MIT license):
# Filename: main.py
import argparse
import configparser
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--config_file', help='config file')
args, left_argv = parser.parse_known_args()
if args.config_file:
with open(args.config_file, 'r') as f:
config = configparser.SafeConfigParser()
config.read([args.config_file])
parser.add_argument('--arg1', help='argument 1')
parser.add_argument('--arg2', type=int, help='argument 2')
for k, v in config.items("Defaults"):
parser.parse_args([str(k), str(v)], args)
parser.parse_args(left_argv, args)
print(args)
Here's an example of a config file:
# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3
Now, running
> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')
However, if our config file has an error:
# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'
Running the script will produce an error, as desired:
> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
[--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'
The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.
You can use ChainMap
A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.
You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.
import os
from collections import ChainMap, defaultdict
options = ChainMap(command_line_options, os.environ, config_file_options,
defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']
print(value, value2)
'optvalue', 'default-value'
fromfile_prefix_chars
Maybe not the perfect API, but worth knowing about.
main.py
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())
Then:
$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py #opts.txt
Namespace(a='1', b='2')
$ ./main.py #opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 #opts.txt
Namespace(a='1', b='2')
Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars
Tested on Python 3.6.5, Ubuntu 18.04.
Try to this way
# encoding: utf-8
import imp
import argparse
class LoadConfigAction(argparse._StoreAction):
NIL = object()
def __init__(self, option_strings, dest, **kwargs):
super(self.__class__, self).__init__(option_strings, dest)
self.help = "Load configuration from file"
def __call__(self, parser, namespace, values, option_string=None):
super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)
config = imp.load_source('config', values)
for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
setattr(namespace, key, getattr(config, key))
Use it:
parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")
And create example config:
# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
parse_args() can take an existing Namespace and merge the existing Namespace with args/options it's currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:
foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')
ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')
args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')
# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')
I've mocked it up for a real config file option. I'm parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.
I have this very simple JSON config file, config.ini:
[DEFAULT]
delimiter = |
and when I run this:
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')
# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()
# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()
if pre_args.config_file:
config = configparser.ConfigParser()
config.read(pre_args.config_file)
for name, val in config['DEFAULT'].items():
setattr(ConfigNamespace, name, val)
# Parse cmd-line args again, merging with ConfigNamespace,
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)
print(args)
with various cmd-line settings, I get:
./main.py
Namespace(config_file=None, delimiter=',')
./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')
./main.py -c config.ini -d \;
Namespace(config_file='config.ini', delimiter=';')