I have written a code function.py in python which has input file path and a output file path and some flags . currently I have hardcoded everything.I want to use command line arguments to provide these inputs so that anyone can run my script by providing input to cmd.how can I do it in python?
In CMD
function.py "input file path" "output file path"
A very rudimentary example would be:
import sys
input_file_path = sys.argv[1]
output_file_path = sys.argv[2]
Note that sys.argv[0] would be your filename. You should also do the relevant checks to make sure there are the correct number of arguments, whether they are valid, etc.
As an alternative to sys.argv, I prefer argparse.
As an example:
# Import the argparse module
import argparse
# Define a function to use argparse to parse your command-line arguments
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
"-i",
"--input-file",
dest="input_file",
help="File to use as input",
type=str
)
parser.add_argument(
"-o",
"--output-file",
dest="output_file",
help="File to output to",
type=str
)
return parser.parse_args()
# If calling this module from the command line, this `if` statement will evaluate to True
if __name__ == "__main__":
# Parse your command-line arguments
args = parse_args()
# Get the parsed value of the "-i" argument:
infile = args.input_file
# Get the parsed value of the "-o" argument:
outfile = args.output_file
May I know what is the best practice to debug an argpars function.
Say I have a py file test_file.py with the following lines
# Script start
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument(“–output_dir”, type=str, default=”/data/xx”)
args = parser.parse_args()
os.makedirs(args.output_dir)
# Script stop
The above script can be executed from terminal by:
python test_file.py –output_dir data/xx
However, for debugging process, I would like to avoid using terminal. Thus the workaround would be
# other line were commented for debugging process
# Thus, active line are
# Script start
import os
args = {“output_dir”:”data/xx”}
os.makedirs(args.output_dir)
#Script stop
However, I am unable to execute the modified script. May I know what have I miss?
When used as a script, parse_args will produce a Namespace object, which displays as:
argparse.Namespace(output_dir='data/xx')
then
args.output_dir
will be the value of that attribute
In the test you could do one several things:
args = parser.parse_args([....]) # a 'fake' sys.argv[1:] list
args = argparse.Namespace(output_dir= 'mydata')
and use args as before. Or simply call the
os.makedirs('data/xx')
I would recommend organizing the script as:
# Script start
import argparse
import os
# this parser definition could be in a function
parser = argparse.ArgumentParser()
parser.add_argument(“–output_dir”, type=str, default=”/data/xx”)
def main(args):
os.makedirs(args.output_dir)
if __name__=='__main__':
args = parser.parse_args()
main(args)
That way the parse_args step isn't run when the file is imported. Whether you pass the args Namespace to main or pass values like args.output_dir, or a dictionary, etc. is your choice.
You can write it in a shell script to do what you want
bash:
#!/usr/bin/
cd /path/to/my/script.py
python script.py --output_dir data/xx
If that is insufficient, you can store your args in a json config file
configs.json
{"output_dir": "data/xx"}
To grab them:
import json
with open('configs.json', 'rb') as fh:
args = json.loads(fh.read())
output_dir = args.get('output_dir')
# 'data/xx'
Do take note of the double quotes around your keys and values in the json file
I am writing an automation script in python using argparse module in which I want to use the -s as an option which takes file/file path as an argument. Can somebody help me to do this?
Example: ./argtest.py -s /home/test/hello.txt
Just do this:
import argparse
parser = argparse.ArgumentParser(description="My program!", formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument("-s", type=argparse.FileType('r'), help="Filename to be passed")
args = vars(parser.parse_args())
open_file = args.s
If you want to open the file for writing, just change r to w in type=argparse.FileType('r'). You could also change it to a, r+, w+, etc.
You can use
import argparse
parse = argparse.ArgumentParser()
parse.add_argument("-s")
args = parse.parse_args()
# print argument of -s
print('argument: ',args.s)
Suppose the above code is stored in the file example.py
$ python example.py -s /home/test/hello.txt
argument: /home/test/hello.txt
You can click here(Python3.x) or here(Python2.x) to learn more.
I need to take an optional argument when running my Python script:
python3 myprogram.py afile.json
or
python3 myprogram.py
This is what I've been trying:
filename = 0
parser = argparse.ArgumentParser(description='Create Configuration')
parser.add_argument('filename', type=str,
help='optional filename')
if filename is not 0:
json_data = open(filename).read()
else:
json_data = open('defaultFile.json').read()
I was hoping to have the filename stored in my variable called "filename" if it was provided. Obviously this isn't working. Any advice?
Please read the tutorial carefully. http://docs.python.org/howto/argparse.html
i believe you need to actually parse the arguments:
parser = argparse.ArgumentParser()
args = parser.parse_args()
then filename will be come available args.filename
Check sys.argv. It gives a list with the name of the script and any command line arguments.
Example script:
import sys
print sys.argv
Calling it:
> python script.py foobar baz
['script.py', 'foobar', 'baz']
If you are looking for the first parameter sys.argv[1] does the trick. More info here.
Try argparse's default parameter, its well documented.
import argparse
parser = argparse.ArgumentParser(description='Create Configuration')
parser.add_argument('--file-name', type=str, help='optional filename',
default="defaultFile.json")
args = parser.parse_args()
print(args.file_name)
Output:
$ python open.py --file-name option1
option1
$ python open.py
defaultFile.json
Alternative library:
click library for arg parsing.
I have a Python application which needs quite a few (~30) configuration parameters. Up to now, I used the OptionParser class to define default values in the app itself, with the possibility to change individual parameters at the command line when invoking the application.
Now I would like to use 'proper' configuration files, for example from the ConfigParser class. At the same time, users should still be able to change individual parameters at the command line.
I was wondering if there is any way to combine the two steps, e.g. use optparse (or the newer argparse) to handle command line options, but reading the default values from a config file in ConfigParse syntax.
Any ideas how to do this in an easy way? I don't really fancy manually invoking ConfigParse, and then manually setting all defaults of all the options to the appropriate values...
I just discovered you can do this with argparse.ArgumentParser.parse_known_args(). Start by using parse_known_args() to parse a configuration file from the commandline, then read it with ConfigParser and set the defaults, and then parse the rest of the options with parse_args(). This will allow you to have a default value, override that with a configuration file and then override that with a commandline option. E.g.:
Default with no user input:
$ ./argparse-partial.py
Option is "default"
Default from configuration file:
$ cat argparse-partial.config
[Defaults]
option=Hello world!
$ ./argparse-partial.py -c argparse-partial.config
Option is "Hello world!"
Default from configuration file, overridden by commandline:
$ ./argparse-partial.py -c argparse-partial.config --option override
Option is "override"
argprase-partial.py follows. It is slightly complicated to handle -h for help properly.
import argparse
import ConfigParser
import sys
def main(argv=None):
# Do argv default this way, as doing it in the functional
# declaration sets it at compile time.
if argv is None:
argv = sys.argv
# Parse any conf_file specification
# We make this parser with add_help=False so that
# it doesn't parse -h and print help.
conf_parser = argparse.ArgumentParser(
description=__doc__, # printed with -h/--help
# Don't mess with format of description
formatter_class=argparse.RawDescriptionHelpFormatter,
# Turn off help, so we print all options in response to -h
add_help=False
)
conf_parser.add_argument("-c", "--conf_file",
help="Specify config file", metavar="FILE")
args, remaining_argv = conf_parser.parse_known_args()
defaults = { "option":"default" }
if args.conf_file:
config = ConfigParser.SafeConfigParser()
config.read([args.conf_file])
defaults.update(dict(config.items("Defaults")))
# Parse rest of arguments
# Don't suppress add_help here so it will handle -h
parser = argparse.ArgumentParser(
# Inherit options from config_parser
parents=[conf_parser]
)
parser.set_defaults(**defaults)
parser.add_argument("--option")
args = parser.parse_args(remaining_argv)
print "Option is \"{}\"".format(args.option)
return(0)
if __name__ == "__main__":
sys.exit(main())
Check out ConfigArgParse - its a new PyPI package (open source) that serves as a drop in replacement for argparse with added support for config files and environment variables.
I'm using ConfigParser and argparse with subcommands to handle such tasks. The important line in the code below is:
subp.set_defaults(**dict(conffile.items(subn)))
This will set the defaults of the subcommand (from argparse) to the values in the section of the config file.
A more complete example is below:
####### content of example.cfg:
# [sub1]
# verbosity=10
# gggg=3.5
# [sub2]
# host=localhost
import ConfigParser
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
parser_sub1 = subparsers.add_parser('sub1')
parser_sub1.add_argument('-V','--verbosity', type=int, dest='verbosity')
parser_sub1.add_argument('-G', type=float, dest='gggg')
parser_sub2 = subparsers.add_parser('sub2')
parser_sub2.add_argument('-H','--host', dest='host')
conffile = ConfigParser.SafeConfigParser()
conffile.read('example.cfg')
for subp, subn in ((parser_sub1, "sub1"), (parser_sub2, "sub2")):
subp.set_defaults(**dict(conffile.items(subn)))
print parser.parse_args(['sub1',])
# Namespace(gggg=3.5, verbosity=10)
print parser.parse_args(['sub1', '-V', '20'])
# Namespace(gggg=3.5, verbosity=20)
print parser.parse_args(['sub1', '-V', '20', '-G','42'])
# Namespace(gggg=42.0, verbosity=20)
print parser.parse_args(['sub2', '-H', 'www.example.com'])
# Namespace(host='www.example.com')
print parser.parse_args(['sub2',])
# Namespace(host='localhost')
I can't say it's the best way, but I have an OptionParser class that I made that does just that - acts like optparse.OptionParser with defaults coming from a config file section. You can have it...
class OptionParser(optparse.OptionParser):
def __init__(self, **kwargs):
import sys
import os
config_file = kwargs.pop('config_file',
os.path.splitext(os.path.basename(sys.argv[0]))[0] + '.config')
self.config_section = kwargs.pop('config_section', 'OPTIONS')
self.configParser = ConfigParser()
self.configParser.read(config_file)
optparse.OptionParser.__init__(self, **kwargs)
def add_option(self, *args, **kwargs):
option = optparse.OptionParser.add_option(self, *args, **kwargs)
name = option.get_opt_string()
if name.startswith('--'):
name = name[2:]
if self.configParser.has_option(self.config_section, name):
self.set_default(name, self.configParser.get(self.config_section, name))
Feel free to browse the source. Tests are in a sibling directory.
Update: This answer still has issues; for example, it cannot handle required arguments, and requires an awkward config syntax. Instead, ConfigArgParse seems to be exactly what this question asks for, and is a transparent, drop-in replacement.
One issue with the current is that it will not error if the arguments in the config file are invalid. Here's a version with a different downside: you'll need to include the -- or - prefix in the keys.
Here's the python code (Gist link with MIT license):
# Filename: main.py
import argparse
import configparser
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--config_file', help='config file')
args, left_argv = parser.parse_known_args()
if args.config_file:
with open(args.config_file, 'r') as f:
config = configparser.SafeConfigParser()
config.read([args.config_file])
parser.add_argument('--arg1', help='argument 1')
parser.add_argument('--arg2', type=int, help='argument 2')
for k, v in config.items("Defaults"):
parser.parse_args([str(k), str(v)], args)
parser.parse_args(left_argv, args)
print(args)
Here's an example of a config file:
# Filename: config_correct.conf
[Defaults]
--arg1=Hello!
--arg2=3
Now, running
> python main.py --config_file config_correct.conf --arg1 override
Namespace(arg1='override', arg2=3, config_file='test_argparse.conf')
However, if our config file has an error:
# config_invalid.conf
--arg1=Hello!
--arg2='not an integer!'
Running the script will produce an error, as desired:
> python main.py --config_file config_invalid.conf --arg1 override
usage: test_argparse_conf.py [-h] [--config_file CONFIG_FILE] [--arg1 ARG1]
[--arg2 ARG2]
main.py: error: argument --arg2: invalid int value: 'not an integer!'
The main downside is that this uses parser.parse_args somewhat hackily in order to obtain the error checking from ArgumentParser, but I am not aware of any alternatives to this.
You can use ChainMap
A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.
You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.
import os
from collections import ChainMap, defaultdict
options = ChainMap(command_line_options, os.environ, config_file_options,
defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']
print(value, value2)
'optvalue', 'default-value'
fromfile_prefix_chars
Maybe not the perfect API, but worth knowing about.
main.py
#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='#')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())
Then:
$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py #opts.txt
Namespace(a='1', b='2')
$ ./main.py #opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 #opts.txt
Namespace(a='1', b='2')
Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars
Tested on Python 3.6.5, Ubuntu 18.04.
Try to this way
# encoding: utf-8
import imp
import argparse
class LoadConfigAction(argparse._StoreAction):
NIL = object()
def __init__(self, option_strings, dest, **kwargs):
super(self.__class__, self).__init__(option_strings, dest)
self.help = "Load configuration from file"
def __call__(self, parser, namespace, values, option_string=None):
super(LoadConfigAction, self).__call__(parser, namespace, values, option_string)
config = imp.load_source('config', values)
for key in (set(map(lambda x: x.dest, parser._actions)) & set(dir(config))):
setattr(namespace, key, getattr(config, key))
Use it:
parser.add_argument("-C", "--config", action=LoadConfigAction)
parser.add_argument("-H", "--host", dest="host")
And create example config:
# Example config: /etc/myservice.conf
import os
host = os.getenv("HOST_NAME", "localhost")
parse_args() can take an existing Namespace and merge the existing Namespace with args/options it's currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:
foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')
ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')
args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')
# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')
I've mocked it up for a real config file option. I'm parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.
I have this very simple JSON config file, config.ini:
[DEFAULT]
delimiter = |
and when I run this:
import argparse
import configparser
parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')
# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()
# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()
if pre_args.config_file:
config = configparser.ConfigParser()
config.read(pre_args.config_file)
for name, val in config['DEFAULT'].items():
setattr(ConfigNamespace, name, val)
# Parse cmd-line args again, merging with ConfigNamespace,
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)
print(args)
with various cmd-line settings, I get:
./main.py
Namespace(config_file=None, delimiter=',')
./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')
./main.py -c config.ini -d \;
Namespace(config_file='config.ini', delimiter=';')