I realize the title might be confusing, but I didn't know how to word my problem.
My program's command line syntax looks like this:
conv.py val from to, where to is optional, but I don't think that should matter.
I'm trying to add a flag that forces my program to ignore cached data and update its database. It should work like this:
conv.py -f val from to
but also like this:
conv.py -f
I know it should be possible to do this because the inbuilt -h flag in argparse works in a similar manner where you can say conv.py val from to or conv.py -h or conv.py -h val. However, I am at a loss as to how to achieve this.
My current code just has the -f flag as an optional argument:
def parse_args():
parser = argparse.ArgumentParser(prog='conv')
parser.add_argument('-f', action='store_true')
parser.add_argument('value')
parser.add_argument('from_', metavar='from' )
parser.add_argument('to', nargs='?', default='neg')
args = parser.parse_args()
return args.from_, args.to, args.value, args.f
I would like to make it so the presence of the -f flag is acceptable by itself or with all the other arguments. Any help is appreciated.
To do that you would create a custom action which exits the parsing:
import argparse
class MyAction(argparse.Action):
def do_the_thing(self):
print("hello from my action")
def __call__(self, parser, namespace, values, option_string=None):
self.do_the_thing()
parser.exit()
parser = argparse.ArgumentParser()
parser.add_argument('value')
parser.add_argument('from_', metavar='from' )
parser.add_argument('to', nargs='?', default='neg')
parser.add_argument("-f", nargs=0, action=MyAction)
args = parser.parse_args()
print("after args parsed")
Now if -f was passed, then print("after args parsed") will not be reached regardless of whether required arguments were sent or not. You may access the parser namespace from within the action instance.
May I know what is the best practice to debug an argpars function.
Say I have a py file test_file.py with the following lines
# Script start
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument(“–output_dir”, type=str, default=”/data/xx”)
args = parser.parse_args()
os.makedirs(args.output_dir)
# Script stop
The above script can be executed from terminal by:
python test_file.py –output_dir data/xx
However, for debugging process, I would like to avoid using terminal. Thus the workaround would be
# other line were commented for debugging process
# Thus, active line are
# Script start
import os
args = {“output_dir”:”data/xx”}
os.makedirs(args.output_dir)
#Script stop
However, I am unable to execute the modified script. May I know what have I miss?
When used as a script, parse_args will produce a Namespace object, which displays as:
argparse.Namespace(output_dir='data/xx')
then
args.output_dir
will be the value of that attribute
In the test you could do one several things:
args = parser.parse_args([....]) # a 'fake' sys.argv[1:] list
args = argparse.Namespace(output_dir= 'mydata')
and use args as before. Or simply call the
os.makedirs('data/xx')
I would recommend organizing the script as:
# Script start
import argparse
import os
# this parser definition could be in a function
parser = argparse.ArgumentParser()
parser.add_argument(“–output_dir”, type=str, default=”/data/xx”)
def main(args):
os.makedirs(args.output_dir)
if __name__=='__main__':
args = parser.parse_args()
main(args)
That way the parse_args step isn't run when the file is imported. Whether you pass the args Namespace to main or pass values like args.output_dir, or a dictionary, etc. is your choice.
You can write it in a shell script to do what you want
bash:
#!/usr/bin/
cd /path/to/my/script.py
python script.py --output_dir data/xx
If that is insufficient, you can store your args in a json config file
configs.json
{"output_dir": "data/xx"}
To grab them:
import json
with open('configs.json', 'rb') as fh:
args = json.loads(fh.read())
output_dir = args.get('output_dir')
# 'data/xx'
Do take note of the double quotes around your keys and values in the json file
I am creating a python script and for parsing the arguments I would need this:
the script will accept three parameters, only one always mandatory, the second one will only be mandatory depending on certain values of the first one and the third one may or may not appear.
This is my try:
class pathAction(argparse.Action):
folder = {'remote':'/path1', 'projects':'/path2'}
def __call__(self, parser, args, values, option = None):
args.path = values
print "ferw %s " % args.component
if args.component=='hos' or args.component=='hcr':
print "rte %s" % args.path
if args.path and pathAction.folder.get(args.path):
args.path = pathAction.folder[args.path]
else:
parser.error("You must enter the folder you want to clean: available choices[remote, projects]")
def main():
try:
# Arguments parsing
parser = argparse.ArgumentParser(description="""This script will clean the old component files.""")
parser.add_argument("-c", "--component", help="component to clean", type=lowerit, choices=["hos", "hcr", "mdw", "gui"], required=True)
parser.add_argument("-p", "--path", help="path to clean", action = pathAction, choices = ["remote", "projects"])
parser.add_argument("-d", "--delete", help="parameter for deleting the files from the filesystem", nargs='*', default=True)
args = parser.parse_args()
if works well except one case: if i have -c it should complain because there is no -p however it does not
Can you help me please?
Thanks
You can add some custom validation like this:
if args.component and not args.path:
parser.error('Your error message!')
Your special action will be used only if there is a -p argument. If you just give it a -c the cross check is never used.
Generally checking for interactions after parse_args (as Gohn67 suggested) is more reliable, and simpler than with custom actions.
What happens if your commandline was '-p remote -c ...'? pathAction would be called before the -c value is parsed and set. Is that what you want? Your special action only works if -p is given, and is the last argument.
Another option is to make 'component' a subparser positional. By default positionals are required. path and delete can be added to those subparsers that need them.
import argparse
parser = argparse.ArgumentParser(description="""This script will clean the old component files.""")
p1 = argparse.ArgumentParser(add_help=False)
p1.add_argument("path", help="path to clean", choices = ["remote", "projects"])
p2 = argparse.ArgumentParser(add_help=False)
p2.add_argument("-d", "--delete", help="parameter for deleting the files from the filesystem", nargs='*', default=True)
sp = parser.add_subparsers(dest='component',description="component to clean")
sp.add_parser('hos', parents=[p1,p2])
sp.add_parser('hcr', parents=[p1,p2])
sp.add_parser('mdw', parents=[p2])
sp.add_parser('gui', parents=[p2])
print parser.parse_args()
sample use:
1848:~/mypy$ python2.7 stack21625446.py hos remote -d 1 2 3
Namespace(component='hos', delete=['1', '2', '3'], path='remote')
I used parents to simplify adding arguments to multiple subparsers. I made path a positional, since it is required (for 2 of the subparsers). In those cases --path just makes the user type more. With nargs='*', --delete has to belong to the subparsers so it can occur last. If it's nargs was fixed (None or number) it could be an argument of parser.
I have this Command(BaseCommand) class with this defined:
make_option('--username', action='store_true', dest='username', default=None),
Then I try to run it like this:
python manage.py thescript --username=something
The output is:
manage.py: error: --username option does not take a value
Why?
EDIT
I am always getting None:
class Command(BaseCommand):
args = '--username=username ...'
help = '...'
option_list = BaseCommand.option_list + (
make_option('--username', action='store', default=None, help='...'),
)
def handle(self, *args, **options):
print options['username']
The line store_true will cause the command to store the boolean value True if the argument --username is given, rather then storing the next value.
I suspect that doing something along the lines of action='store' would do what you expect.
I think you're currently using optparse, correct? If so, you can find a full listing of the different kinds of actions available in the optparse documentation.
I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.
Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.
I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.
Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values...
I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:
Default with no user input:
$ ./argparse-partial.py
Option is "default"
Default from configuration file:
$ cat argparse-partial.config
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config
Option is "Hello world!"
Default from configuration file, overridden by commandline:
$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"
argprase-partial.py follows. It is slightly complicated to handle -h for help properly.
import argparse
import ConfigParser
import sys
def main(argv=None):
# Do argv default this way, as doing it in the functional
# declaration sets it at compile time.
if argv is None:
argv = sys.argv
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
description=__doc__, # printed with -h/--help
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file", metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args()
defaults = { "option":"default" }
if args.conf_file:
config = ConfigParser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument("--option")
args = parser.parse_args(remaining_argv)
print "Option is \"{}\"".format(args.option)
return(0)
if __name__ == "__main__":
sys.exit(main())
Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.
I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:
subp.set_defaults(**dict(conffile.items(subn)))
This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.
A more complete example is below:
####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost
import ConfigParser
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')
parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')
conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')
for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
subp.set_defaults(**dict(conffile.items(subn)))
print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
I can't say it's the best way, but I have an OptionParser class that I made that does just that - acts like optparse.OptionParser with defaults coming from a config file section. You can have it...
class OptionParser(optparse.OptionParser):
def __init__(self, **kwargs):
import sys
import os
config_file = kwargs.pop('config_file',
os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
self.config_section = kwargs.pop('config_section', 'OPTIONS')
self.configParser = ConfigParser()
self.configParser.read(config_file)
optparse.OptionParser.__init__(self, **kwargs)
def add_option(self, *args, **kwargs):
option = optparse.OptionParser.add_option(self, *args, **kwargs)
name = option.get_opt_string()
if name.startswith('--'):
name = name[2:]
if self.configParser.has_option(self.config_section, name):
self.set_default(name, self.configParser.get(self.config_section, name))
Feel free to browse the source. Tests are in a sibling directory.
Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.
One issue with the current is that it will not error if the arguments in the config file are invalid. Here's a version with a different downside: you'll need to include the -- or - prefix in the keys.
Here's the python code (Gist link with MIT license):
# Filename: main.py
import argparse
import configparser
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--config_file', help='config file')
args, left_argv = parser.parse_known_args()
if args.config_file:
with open(args.config_file, 'r') as f:
config = configparser.SafeConfigParser()
config.read([args.config_file])
parser.add_argument('--arg1', help='argument 1')
parser.add_argument('--arg2', type=int, help='argument 2')
for k, v in config.items("Defaults"):
parser.parse_args([str(k), str(v)], args)
parser.parse_args(left_argv, args)
print(args)
Here's an example of a config file:
# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3
Now, running
> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')
However, if our config file has an error:
# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'
Running the script will produce an error, as desired:
> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
[--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'
The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.
You can use ChainMap
A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.
You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.
import os
from collections import ChainMap, defaultdict
options = ChainMap(command_line_options, os.environ, config_file_options,
defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']
print(value, value2)
'optvalue', 'default-value'
fromfile_prefix_chars
Maybe not the perfect API, but worth knowing about.
main.py
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())
Then:
$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py #opts.txt
Namespace(a='1', b='2')
$ ./main.py #opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 #opts.txt
Namespace(a='1', b='2')
Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars
Tested on Python 3.6.5, Ubuntu 18.04.
Try to this way
# encoding: utf-8
import imp
import argparse
class LoadConfigAction(argparse._StoreAction):
NIL = object()
def __init__(self, option_strings, dest, **kwargs):
super(self.__class__, self).__init__(option_strings, dest)
self.help = "Load configuration from file"
def __call__(self, parser, namespace, values, option_string=None):
super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)
config = imp.load_source('config', values)
for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
setattr(namespace, key, getattr(config, key))
Use it:
parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")
And create example config:
# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
parse_args() can take an existing Namespace and merge the existing Namespace with args/options it's currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:
foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')
ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')
args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')
# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')
I've mocked it up for a real config file option. I'm parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.
I have this very simple JSON config file, config.ini:
[DEFAULT]
delimiter = |
and when I run this:
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')
# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()
# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()
if pre_args.config_file:
config = configparser.ConfigParser()
config.read(pre_args.config_file)
for name, val in config['DEFAULT'].items():
setattr(ConfigNamespace, name, val)
# Parse cmd-line args again, merging with ConfigNamespace,
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)
print(args)
with various cmd-line settings, I get:
./main.py
Namespace(config_file=None, delimiter=',')
./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')
./main.py -c config.ini -d \;
Namespace(config_file='config.ini', delimiter=';')