May I know what is the best practice to debug an argpars function.
Say I have a py file test_file.py with the following lines
# Script start
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument(“–output_dir”, type=str, default=”/data/xx”)
args = parser.parse_args()
os.makedirs(args.output_dir)
# Script stop
The above script can be executed from terminal by:
python test_file.py –output_dir data/xx
However, for debugging process, I would like to avoid using terminal. Thus the workaround would be
# other line were commented for debugging process
# Thus, active line are
# Script start
import os
args = {“output_dir”:”data/xx”}
os.makedirs(args.output_dir)
#Script stop
However, I am unable to execute the modified script. May I know what have I miss?
When used as a script, parse_args will produce a Namespace object, which displays as:
argparse.Namespace(output_dir='data/xx')
then
args.output_dir
will be the value of that attribute
In the test you could do one several things:
args = parser.parse_args([....]) # a 'fake' sys.argv[1:] list
args = argparse.Namespace(output_dir= 'mydata')
and use args as before. Or simply call the
os.makedirs('data/xx')
I would recommend organizing the script as:
# Script start
import argparse
import os
# this parser definition could be in a function
parser = argparse.ArgumentParser()
parser.add_argument(“–output_dir”, type=str, default=”/data/xx”)
def main(args):
os.makedirs(args.output_dir)
if __name__=='__main__':
args = parser.parse_args()
main(args)
That way the parse_args step isn't run when the file is imported. Whether you pass the args Namespace to main or pass values like args.output_dir, or a dictionary, etc. is your choice.
You can write it in a shell script to do what you want
bash:
#!/usr/bin/
cd /path/to/my/script.py
python script.py --output_dir data/xx
If that is insufficient, you can store your args in a json config file
configs.json
{"output_dir": "data/xx"}
To grab them:
import json
with open('configs.json', 'rb') as fh:
args = json.loads(fh.read())
output_dir = args.get('output_dir')
# 'data/xx'
Do take note of the double quotes around your keys and values in the json file
I spent few hours reading tutorials about argparse and managed to learn to use normal parameters. The official documentation is not very readable to me. I'm new to Python. I'm trying to write a program that could be invoked in following ways:
cat inFile | program [options] > outFile -- If no inFile or outfile is specified, read from stdin and output to stdout.
program [options] inFile outFile
program [options] inFile > outFile -- If only one file is specified it is input and output should go to stdout.
cat inFile | program [options] - outFile -- If '-' is given in place of inFlie read from stdin.
program [options] /path/to/folder outFile -- Process all files from /path/to/folder and it subdirectories.
I want it to behave like regular cli program under GNU/Linux.
It would be also nice if the program would be able to be invoked:
program [options] inFile0 inFile1 ... inFileN outFile -- first path/file always interpreted as input, last one always interpreted as output. Any additional ones interpreted as inputs.
I could probably write dirty code that would accomplish this but this is going to be used, so someone will end up maintaining it (and he will know where I live...).
Any help/suggestions are much appreciated.
Combining answers and some more knowledge from the Internet I've managed to write this(it does not accept multiple inputs but this is enough):
import sys, argparse, os.path, glob
def inputFile(path):
if path == "-":
return [sys.stdin]
elif os.path.exists(path):
if os.path.isfile(path):
return [path]
else:
return [y for x in os.walk(path) for y in glob.glob(os.path.join(x[0], '*.dat'))]
else:
exit(2)
def main(argv):
cmdArgsParser = argparse.ArgumentParser()
cmdArgsParser.add_argument('inFile', nargs='?', default='-', type=inputFile)
cmdArgsParser.add_argument('outFile', nargs='?', default='-', type=argparse.FileType('w'))
cmdArgs = cmdArgsParser.parse_args()
print cmdArgs.inFile
print cmdArgs.outFile
if __name__ == "__main__":
main(sys.argv[1:])
Thank you!
You need a positional argument (name not starting with a dash), optional arguments (nargs='?'), a default argument (default='-'). Additionally, argparse.FileType is a convenience factory to return sys.stdin or sys.stdout if - is passed (depending on the mode).
All together:
#!/usr/bin/env python
import argparse
# default argument is sys.argv[0]
parser = argparse.ArgumentParser('foo')
parser.add_argument('in_file', nargs='?', default='-', type=argparse.FileType('r'))
parser.add_argument('out_file', nargs='?', default='-', type=argparse.FileType('w'))
def main():
# default argument is is sys.argv[1:]
args = parser.parse_args(['bar', 'baz'])
print(args)
args = parser.parse_args(['bar', '-'])
print(args)
args = parser.parse_args(['bar'])
print(args)
args = parser.parse_args(['-', 'baz'])
print(args)
args = parser.parse_args(['-', '-'])
print(args)
args = parser.parse_args(['-'])
print(args)
args = parser.parse_args([])
print(args)
if __name__ == '__main__':
main()
I'll give you a start script to play with. It uses optionals rather than positionals. and only one input file. But it should give a taste of what you can do.
import argparse
parser = argparse.ArgumentParser()
inarg = parser.add_argument('-i','--infile', type=argparse.FileType('r'), default='-')
outarg = parser.add_argument('-o','--outfile', type=argparse.FileType('w'), default='-')
args = parser.parse_args()
print(args)
cnt = 0
for line in args.infile:
print(cnt, line)
args.outfile.write(line)
cnt += 1
When called without arguments, it just echos your input (after ^D). I'm a little bothered that it doesn't exit until I issue another ^D.
FileType is convenient, but has the major fault - it opens the files, but you have to close them yourself, or let Python do so when exiting. There's also the complication that you don't want to close stdin/out.
The best argparse questions include a basic script, and specific questions on how to correct or improve it. Your specs are reasonably clear. but it would be nice if you gave us more to work with.
To handle the subdirectories option, I would skip the FileType bit. Use argparse to get 2 lists of strings (or a list and an name), and then do the necessary chgdir and or glob to find and iterate over files. Don't expect argparse to do the actual work. Use it to parse the commandline strings. Here a sketch of such a script, leaving most details for you to fill in.
import argparse
import os
import sys # of stdin/out
....
def open_output(outfile):
# function to open a file for writing
# should handle '-'
# return a file object
def glob_dir(adir):
# function to glob a dir
# return a list of files ready to open
def open_forread(afilename):
# function to open file for reading
# be sensitive to '-'
def walkdirs(alist):
outlist = []
for name in alist:
if <name is file>;
outlist.append(name)
else <name is a dir>:
glist = glob(dir)
outlist.extend(glist)
else:
<error>
return outlist
def cat(infile, outfile):
<do your thing here>
def main(args):
# handle args options
filelist = walkdirs(args.inlist)
fout = open_outdir(args.outfile)
for name in filelist:
fin = open_forread(name)
cat(fin,fout)
if <fin not stdin>: fin.close()
if <fout not stdout>: fout.close()
if '__name__' == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('inlist', nargs='*')
parser.add_argument('outfile')
# add options
args = parser.parse_args()
main(args)
The parser here requires you to give it an outfile name, even if it is '-'. I could define its nargs='?' to make it optional. But that does not play nicely with the 'inlist` '*'.
Consider
myprog one two three
Is that
namespace(inlist=['one','two','three'], outfile=default)
or
namespace(inlist=['one','two'], outfile='three')
With both a * and ? positional, the identity of the last string is ambiguous - is it the last entry for inlist, or the optional entry for outfile? argparse chooses the former, and never assigns the value to outfile.
With --infile, --outfile definitions, the allocation of these strings is clear.
In sense this problem is too complex for argparse - there's nothing in it to handle things like directories. In another sense it is too simple. You could just as easily split sys.argv[1:] between inlist and outfile without the help of argparse.
I need to take an optional argument when running my Python script:
python3 myprogram.py afile.json
or
python3 myprogram.py
This is what I've been trying:
filename = 0
parser = argparse.ArgumentParser(description='Create Configuration')
parser.add_argument('filename', type=str,
help='optional filename')
if filename is not 0:
json_data = open(filename).read()
else:
json_data = open('defaultFile.json').read()
I was hoping to have the filename stored in my variable called "filename" if it was provided. Obviously this isn't working. Any advice?
Please read the tutorial carefully. http://docs.python.org/howto/argparse.html
i believe you need to actually parse the arguments:
parser = argparse.ArgumentParser()
args = parser.parse_args()
then filename will be come available args.filename
Check sys.argv. It gives a list with the name of the script and any command line arguments.
Example script:
import sys
print sys.argv
Calling it:
> python script.py foobar baz
['script.py', 'foobar', 'baz']
If you are looking for the first parameter sys.argv[1] does the trick. More info here.
Try argparse's default parameter, its well documented.
import argparse
parser = argparse.ArgumentParser(description='Create Configuration')
parser.add_argument('--file-name', type=str, help='optional filename',
default="defaultFile.json")
args = parser.parse_args()
print(args.file_name)
Output:
$ python open.py --file-name option1
option1
$ python open.py
defaultFile.json
Alternative library:
click library for arg parsing.
I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.
Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.
I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.
Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values...
I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:
Default with no user input:
$ ./argparse-partial.py
Option is "default"
Default from configuration file:
$ cat argparse-partial.config
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config
Option is "Hello world!"
Default from configuration file, overridden by commandline:
$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"
argprase-partial.py follows. It is slightly complicated to handle -h for help properly.
import argparse
import ConfigParser
import sys
def main(argv=None):
# Do argv default this way, as doing it in the functional
# declaration sets it at compile time.
if argv is None:
argv = sys.argv
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
description=__doc__, # printed with -h/--help
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file", metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args()
defaults = { "option":"default" }
if args.conf_file:
config = ConfigParser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument("--option")
args = parser.parse_args(remaining_argv)
print "Option is \"{}\"".format(args.option)
return(0)
if __name__ == "__main__":
sys.exit(main())
Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.
I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:
subp.set_defaults(**dict(conffile.items(subn)))
This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.
A more complete example is below:
####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost
import ConfigParser
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')
parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')
conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')
for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
subp.set_defaults(**dict(conffile.items(subn)))
print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
I can't say it's the best way, but I have an OptionParser class that I made that does just that - acts like optparse.OptionParser with defaults coming from a config file section. You can have it...
class OptionParser(optparse.OptionParser):
def __init__(self, **kwargs):
import sys
import os
config_file = kwargs.pop('config_file',
os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
self.config_section = kwargs.pop('config_section', 'OPTIONS')
self.configParser = ConfigParser()
self.configParser.read(config_file)
optparse.OptionParser.__init__(self, **kwargs)
def add_option(self, *args, **kwargs):
option = optparse.OptionParser.add_option(self, *args, **kwargs)
name = option.get_opt_string()
if name.startswith('--'):
name = name[2:]
if self.configParser.has_option(self.config_section, name):
self.set_default(name, self.configParser.get(self.config_section, name))
Feel free to browse the source. Tests are in a sibling directory.
Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.
One issue with the current is that it will not error if the arguments in the config file are invalid. Here's a version with a different downside: you'll need to include the -- or - prefix in the keys.
Here's the python code (Gist link with MIT license):
# Filename: main.py
import argparse
import configparser
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--config_file', help='config file')
args, left_argv = parser.parse_known_args()
if args.config_file:
with open(args.config_file, 'r') as f:
config = configparser.SafeConfigParser()
config.read([args.config_file])
parser.add_argument('--arg1', help='argument 1')
parser.add_argument('--arg2', type=int, help='argument 2')
for k, v in config.items("Defaults"):
parser.parse_args([str(k), str(v)], args)
parser.parse_args(left_argv, args)
print(args)
Here's an example of a config file:
# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3
Now, running
> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')
However, if our config file has an error:
# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'
Running the script will produce an error, as desired:
> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
[--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'
The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.
You can use ChainMap
A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.
You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.
import os
from collections import ChainMap, defaultdict
options = ChainMap(command_line_options, os.environ, config_file_options,
defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']
print(value, value2)
'optvalue', 'default-value'
fromfile_prefix_chars
Maybe not the perfect API, but worth knowing about.
main.py
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())
Then:
$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py #opts.txt
Namespace(a='1', b='2')
$ ./main.py #opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 #opts.txt
Namespace(a='1', b='2')
Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars
Tested on Python 3.6.5, Ubuntu 18.04.
Try to this way
# encoding: utf-8
import imp
import argparse
class LoadConfigAction(argparse._StoreAction):
NIL = object()
def __init__(self, option_strings, dest, **kwargs):
super(self.__class__, self).__init__(option_strings, dest)
self.help = "Load configuration from file"
def __call__(self, parser, namespace, values, option_string=None):
super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)
config = imp.load_source('config', values)
for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
setattr(namespace, key, getattr(config, key))
Use it:
parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")
And create example config:
# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
parse_args() can take an existing Namespace and merge the existing Namespace with args/options it's currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:
foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')
ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')
args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')
# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')
I've mocked it up for a real config file option. I'm parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.
I have this very simple JSON config file, config.ini:
[DEFAULT]
delimiter = |
and when I run this:
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')
# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()
# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()
if pre_args.config_file:
config = configparser.ConfigParser()
config.read(pre_args.config_file)
for name, val in config['DEFAULT'].items():
setattr(ConfigNamespace, name, val)
# Parse cmd-line args again, merging with ConfigNamespace,
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)
print(args)
with various cmd-line settings, I get:
./main.py
Namespace(config_file=None, delimiter=',')
./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')
./main.py -c config.ini -d \;
Namespace(config_file='config.ini', delimiter=';')