In Python, how can I parse the command line, edit the resulting parsed arguments object and generate a valid command line back with the updated values?
For instance, I would like python cmd.py --foo=bar --step=0 call python cmd.py --foo=bar --step=1 with all the original --foo=bar arguments, potentially without extra arguments added when default value is used.
Is it possible with argparse?
You can use argparse to parse the command-line arguments, and then modify those as desired. At the moment however, argparse lacks the functionality to work in reverse and convert those values back into a command-line string. There is however a package for doing precisely that, called argunparse. For example, the following code in cmd.py
import sys
import argparse
import argunparse
parser = argparse.ArgumentParser()
unparser = argunparse.ArgumentUnparser()
parser.add_argument('--foo')
parser.add_argument('--step', type=int)
kwargs = vars(parser.parse_args())
kwargs['step'] += 1
prefix = f'python {sys.argv[0]} '
arg_string = unparser.unparse(**kwargs)
print(prefix + arg_string)
will print the desired command line:
python cmd.py --foo=bar --step=1
argparse is clearly designed to go one way, from sys.argv to the args namespace. No thought has been given to preserving information that would let you map things back the other way, much less do the mapping itself.
In general, multiple sys.argv could produce the same args. You could, for example, have several arguments that have the same dest. Or you can repeat 'optionals'. But for a restricted 'parser' setup there may be enough information to recreate a usable argv.
Try something like:
parser = argparser.ArgumentParser()
arg1 = parser.add_argument('--foo', default='default')
arg2 = parser.add_argument('bar', nargs=2)
and then examine the arg1 and arg2 objects. They contain all the information that you supplied to the add_argument method. Of course you could have defined those values in your own data structures before hand, e.g.
{'option_string':'--foo', 'default':'default'}
{'dest':'bar', 'nargs':2}
and used those as input to add_argument.
While the parser may have enough information to recreate a useable sys.argv, you have to figure out how to do that yourself.
default=argparse.SUPPRESS may be handy. It keeps the parser from adding a default entry to the namespace. So if the option isn't used, it won't appear in the namespace.
This isn't possible in any easy way that I know of, then again I've never needed to do this.
But with the lack of information in the question in regards to how you call your script, I'll assume the following:
python test.py cmd --foo=bar --step=0
And what you could do is do:
from sys import argv
for index in range(1, len(argv)): # the first object is the script itself
if '=' in argv[index]:
param, value = argv[index].split('=', 1)
if param == '--step':
value = '1'
argv[index] = param + '=' + value
print(argv)
Note that this is very specific to --step and may be what you've already thought of and just wanted a "better way", but again, I don't think there is.
depending on the scope, this works at the same module at least:
pprint(argparse._sys.argv)
Per the other answers, rebuilding is imperfect, but if you aren't doing anything too fancy and are okay with imperfect, something like this could work for you as a starting point:
def unparse_args(parser, parsed_args):
"""Unparse argparsed args"""
positional_args = [action.dest
for action in parser._actions
if not action.option_strings]
optionals = []
positionals = []
for key, value in vars(parsed_args).items():
if not value:
# none and false flags go away
continue
elif key in positional_args:
positionals.append(value)
elif value is True:
optionals.append(f"--{key}")
else:
optionals.append(f"--{key}={value}")
return " ".join(optionals + positionals)
Here's an example using this with a git clone clone:
parser = argparse.ArgumentParser(description='A sample git clone wrapper')
# options
parser.add_argument("-v", "--verbose", action="store_true",
help="be more verbose")
parser.add_argument("-q", "--quiet", action="store_true",
help="be more quiet")
parser.add_argument("--recurse-submodules", nargs='?',
help="initialize submodules in the clone")
parser.add_argument("--recursive", nargs='?',
help="alias of --recurse-submodules")
parser.add_argument("-b", "--branch",
help=" checkout <branch> instead of the remote's HEAD")
parser.add_argument("--depth", type=int,
help="create a shallow clone of that depth")
parser.add_argument("--shallow-submodules", action="store_true",
help="any cloned submodules will be shallow")
# positional
parser.add_argument("repo", help="The git repo to clone")
parser.add_argument("dir", nargs='?',help="The location to clone the repo")
# make a fake call to your git clone clone and parse the args
cmdargs = ["--depth=1", "-q", "ohmyzsh/ohmyzsh"]
parsedargs = parser.parse_args(cmdargs)
# now unparse them
unparsed = unparse_args(parser, parsedargs)
print(unparsed)
Related
I'm trying to write a little app that can do multiple things depending on arguments specified in argparse.
I use a positional argument (soundfiledir) for file directory which must always be specified, but after that I want to specify arguments based on what I want the app to perform. For example, the -A flag will run a specific set of jobs (python main.py [soundfile path] -A)
parser = argparse.ArgumentParser()
parser.add_argument('soundfiledir', type=soundfiledir_format, help = "Specify soundfile directory") #positional argument. must always be provided.
parser.add_argument('-A', '--all', action = "store_true", help = "If this flag is specified, the program will transcribe all the sound files in the sound file directory (with timestamps), and will automatically concatenate files recorded close in time")
if args.all:
does stuff
On top of this, I also use subparsers. For example a subparser called fileconcatenator (python main.py [soundfile path] fileconcatenator) can be specified with some options (python main.py [soundfile path] fileconcatenator -a 15)
subparser = parser.add_subparsers(dest = 'command')
fileconcatenator_parser = subparser.add_parser('fileconcatenator', help = "Concatenates sound files together")
group1 = fileconcatenator_parser.add_mutually_exclusive_group(required=True)
group1.add_argument('-a','--autoconcat',type = positive_int, nargs = "?", const = 3, default = None, \
help="Concatenate audio files recorded close in time. By default any file recorded within 3mns of each other.")
group1.add_argument('-m', '--manconcat', type = list, default = [], \
help = "Concatenate audio files specified as a list.")
fileconverter_parser = subparser.add_parser('fileconverter',help = "Converts files to 16kHz mono wav")
fileconverter_parser.add_argument('-f', '--filestoconvert', type = list, required=True, default = ["All"], \
help = "Specify which files to convert.")
note: You might notice I have type set as positive_int, this is a user specified type programmed using
def positive_int(s):
try:
value = int(s)
return int(s)
except ValueError:
raise argparse.ArgumentTypeError(f"Expected positive integer got {s!r}")
if value <= 0:
raise argparse.ArgumentTypeError(f"Expected positive integer got {s!r}")
In main, I have things set up as following:
def main():
if args.all:
do stuff
if args.autoconcat is None:
pass
else:
do stuff
The problem is, when I run python main.py [soundfile path] -A , I get AttributeError: 'Namespace' object has no attribute 'autoconcat'
The program still runs (because the if args.autoconcat comes after the if args.all block), but I'd like to know what I'm doing wrong.
Any help greatly appreciated. I will amend the question if you find it unclear.
A quote from the Python argparse docs:
Note that the object returned by parse_args() will only contain attributes for
the main parser and the subparser that was selected by the command line (and not
any other subparsers). So in the example above, when the a command is specified,
only the foo and bar attributes are present, and when the b command is specified,
only the foo and baz attributes are present.
This is exactly your case : You are not invoking the programs subcommand fileconcatenator, so the args object will not contain the arguments of that subcommand, e.g. autoconcat. You have to check first which subcommand was called. This can be done by having an option common for all subcommands, which is not modifiable by the command-line user. It will be set for each subcommand separately, and when subcommand a is called, this argument will have value a, and when subcommand b is called the argument will have value b. This can be achieve by calling set_defaults on each subparser like this:
fileconcatenator_parser = subparser.add_parser('fileconcatenator', help = "Concatenates sound files together")
fileconcatenator_parser.set_defaults(parser_name="fileconcatenator")
# adding some arguments here
fileconverter_parser = subparser.add_parser('fileconverter',help = "Converts files to 16kHz mono wav")
fileconverter_parser.set_defaults(parser_name="fileconverter")
#adding some arguments here
and then in main, check first if parser_name is fileconverter or fileconcatenator, and check for the arguments based on which subcommand was called.
def main():
args = parser.parse_args()
if args.parser_name == "fileconverter":
# do something with args.filestoconvert
elif args.parser_name == "fileconcatenator":
if args.autoconcat is None:
pass
else:
# do something
You may have to call set_defaults(parser_name="main") on the main parser
to make it work.
So I'm trying to get my program to do take the command line arguments and use it in my script. I read argparse and the optparse documentation and I'm still lost.
What I'm trying to do is have my code execute this on the command line:
./program <-p port> <-s> [required1] [required2]
The -p is optional, and I want to make the port a variable in my script, like so:
server_address = ('some server name', **port**)
I thought that that's what store and dest would do... as in store would take the port argument and dest would be the variable name and I could call it like program.port. It doesn't work this way, however, and I can't find or decipher explanations for what exactly store and dest do.
I'm new to Python, so this might not be a well-formed question.
so, following the documentation:
You create a parser
import argparse
parser = argparse.ArgumentParser(description='Some helpful text about what your function does')
You add arguments, optional ones have '-'s before hand, see below
parser.add_argument('-p', '--port', type=int, default=0, help='port')
parser.add_argument('-s', help='I don\'t know what this is')
parser.add_argument('required_1') # Note the lack of dashes
parser.add_argument('required_2')
You need to parse the arguments with a function call
args = parser.parse_args()
This creates a namespace object which you can then access your variables from, see below
port = args.port
or
port = vars(args)['port']
req1 = args.required_1
req2 = args.required_2
etc...
For more information on namespace objects, checkout this question
Hopefully that helps.
I have a program which can be used in the following way:
program install -a arg -b arg
program list
program update
There can only ever be one of the positional arguments specified (install, list or update). And there can only be other arguments in the install scenario.
The argparse documentation is a little dense and I'm having a hard time figuring out how to do this correctly. What should my add_arguments look like?
This seems like you want to use subparsers.
from argparse import ArgumentParser
parser = ArgumentParser()
subparsers = parser.add_subparsers()
install = subparsers.add_parser('install')
install.add_argument('-b')
install.add_argument('-a')
install.set_defaults(subparser='install')
lst = subparsers.add_parser('list')
lst.set_defaults(subparser='list')
update = subparsers.add_parser('update')
update.set_defaults(subparser='update')
print parser.parse_args()
As stated in the docs, I have combined with set_defaults so that you can know which subparser was invoked.
Let's say I have a script that does some work on a file. It takes this file's name on the command line, but if it's not provided, it defaults to a known filename (content.txt, say). With python's argparse, I use the following:
parser = argparse.ArgumentParser(description='my illustrative example')
parser.add_argument('--content', metavar='file',
default='content.txt', type=argparse.FileType('r'),
help='file to process (defaults to content.txt)')
args = parser.parse_args()
# do some work on args.content, which is a file-like object
This works great. The only problem is that if I run python myscript --help, I get an ArgumentError if the file isn't there (which I guess makes sense), and the help text is not shown. I'd rather it not try to open the file if the user just wants --help. Is there any way to do this? I know I could make the argument a string and take care of opening the file myself later (and I've been doing that), but it would be convenient to have argparse take care of it.
You could subclass argparse.FileType:
import argparse
import warnings
class ForgivingFileType(argparse.FileType):
def __call__(self, string):
try:
super(ForgivingFileType,self).__call__(string)
except IOError as err:
warnings.warn(err)
parser = argparse.ArgumentParser(description='my illustrative example')
parser.add_argument('--content', metavar='file',
default='content.txt', type=ForgivingFileType('r'),
help='file to process (defaults to content.txt)')
args = parser.parse_args()
This works without having to touch private methods like ArgumentParser._parse_known_args.
Looking at the argparse code, I see:
ArgumentParser.parse_args calls parse_known_args and makes sure that there isn't any pending argument to be parsed.
ArgumentParser.parse_known_args sets default values and calls ArgumentParser._parse_known_args
Hence, the workaround would be to use ArgumentParser._parse_known_args directly to detect -h and, after that, use ArgumentParser.parse_args as usual.
import sys, argparse
parser = argparse.ArgumentParser(description='my illustrative example', argument_default=argparse.SUPPRESS)
parser.add_argument('--content', metavar='file',
default='content.txt', type=argparse.FileType('r'),
help='file to process (defaults to content.txt)')
parser._parse_known_args(sys.argv[1:], argparse.Namespace())
args = parser.parse_args()
Note that ArgumentParser._parse_known_args needs a couple of parameters: the arguments from the command line and the namespace.
Of course, I wouldn't recommend this approach since it takes advantage of the internal argparse implementation and that might change in the future. However, I don't find it too messy, so you still might want to use it if you think maintenance risks pay off.
Use stdin as default:
parser.add_argument('file', default='-', nargs='?', type=argparse.FileType('r'))
Perhaps you could define your own type or action in the add_argument call that checks if the file exists, and returns a file handle if it does and None (or something else) otherwise.
This would require you to write some code of yourself as well though, but if the default value can not always be used you probably have to do some checking sooner or later. Like Manny D argues you might want to reconsider your default value.
I have an application that allows you to send event data to a custom script. You simply lay out the command line arguments and assign what event data goes with what argument. The problem is that there is no real flexibility here. Every option you map out is going to be used, but not every option will necessarily have data. So when the application builds the string to send to the script, some of the arguments are blank and python's OptionParser errors out with "error: --someargument option requires an argument"
Being that there are over 200 points of data, it's not like I can write separate scripts to handle each combination of possible arguments (it would take 2^200 scripts). Is there a way to handle empty arguments in python's optionparser?
Sorry, misunderstood the question with my first answer. You can accomplish the ability to have optional arguments to command line flags use the callback action type when you define an option. Use the following function as a call back (you will likely wish to tailor to your needs) and configure it for each of the flags that can optionally receive an argument:
import optparse
def optional_arg(arg_default):
def func(option,opt_str,value,parser):
if parser.rargs and not parser.rargs[0].startswith('-'):
val=parser.rargs[0]
parser.rargs.pop(0)
else:
val=arg_default
setattr(parser.values,option.dest,val)
return func
def main(args):
parser=optparse.OptionParser()
parser.add_option('--foo',action='callback',callback=optional_arg('empty'),dest='foo')
parser.add_option('--file',action='store_true',default=False)
return parser.parse_args(args)
if __name__=='__main__':
import sys
print main(sys.argv)
Running from the command line you'll see this:
# python parser.py
(<Values at 0x8e42d8: {'foo': None, 'file': False}>, [])
# python parser.py --foo
(<Values at 0x8e42d8: {'foo': 'empty', 'file': False}>, [])
# python parser.py --foo bar
(<Values at 0x8e42d8: {'foo': 'bar', 'file': False}>, [])
Yes, there is an argument to do so when you add the option:
from optparse import OptionParser
parser = OptionParser()
parser.add_option("--SomeData",action="store", dest="TheData", default='')
Give the default argument the value you want the option to have it is to be specified but optionally have an argument.
I don't think optparse can do this. argparse is a different (non-standard) module that can handle situations like this where the options have optional values.
With optparse you have to either have to specify the option including it's value or leave out both.
Optparse already allows you to pass the empty string as an option argument. So if possible, treat the empty string as "no value". For long options, any of the following work:
my_script --opt= --anotheroption
my_script --opt='' --anotheroption
my_script --opt="" --anotheroption
my_script --opt '' --anotheroption
my_script --opt "" --anotheroption
For short-style options, you can use either of:
my_script -o '' --anotheroption
my_script -o "" --anotheroption
Caveat: this has been tested under Linux and should work the same under other Unixlike systems; Windows handles command line quoting differently and might not accept all of the variants listed above.
Mark Roddy's solution would work, but it requires attribute modification of a parser object during runtime, and has no support for alternative option formattings other than - or --.
A slightly less involved solution is to modify the sys.argv array before running optparse and insert an empty string ("") after a switch which doesn't need to have arguments.
The only constraint of this method is that you have your options default to a predictable value other than the one you are inserting into sys.argv (I chose None for the example below, but it really doesn't matter).
The following code creates an example parser and set of options, extracts an array of allowed switches from the parser (using a little bit of instance variable magic), and then iterates through sys.argv, and every time it finds an
allowed switch, it checks to see if it was given without any arguments following it . If there is no argument after a switch, the empty string will be inserted on the command
line. After altering sys.argv, the parser is invoked, and you can check for options whose values are "", and act accordingly.
#Instantiate the parser, and add some options; set the options' default values to None, or something predictable that
#can be checked later.
PARSER_DEFAULTVAL = None
parser = OptionParser(usage="%prog -[MODE] INPUT [options]")
#This method doesn't work if interspersed switches and arguments are allowed.
parser.allow_interspersed_args = False
parser.add_option("-d", "--delete", action="store", type="string", dest="to_delete", default=PARSER_DEFAULTVAL)
parser.add_option("-a", "--add", action="store", type="string", dest="to_add", default=PARSER_DEFAULTVAL)
#Build a list of allowed switches, in this case ['-d', '--delete', '-a', '--add'] so that you can check if something
#found on sys.argv is indeed a valid switch. This is trivial to make by hand in a short example, but if a program has
#a lot of options, or if you want an idiot-proof way of getting all added options without modifying a list yourself,
#this way is durable. If you are using OptionGroups, simply run the loop below with each group's option_list field.
allowed_switches = []
for opt in parser.option_list:
#Add the short (-a) and long (--add) form of each switch to the list.
allowed_switches.extend(opt._short_opts + opt._long_opts)
#Insert empty-string values into sys.argv whenever a switch without arguments is found.
for a in range(len(sys.argv)):
arg = sys.argv[a]
#Check if the sys.argv value is a switch
if arg in allowed_switches:
#Check if it doesn't have an accompanying argument (i.e. if it is followed by another switch, or if it is last
#on the command line)
if a == len(sys.argv) - 1 or argv[a + 1] in allowed_switches:
sys.argv.insert(a + 1, "")
options, args = parser.parse_args()
#If the option is present (i.e. wasn't set to the default value)
if not (options.to_delete == PARSER_DEFAULTVAL):
if options.droptables_ids_csv == "":
#The switch was not used with any arguments.
...
else:
#The switch had arguments.
...
After checking that the cp command understands e.g. --backup=simple but not --backup simple, I answered the problem like this:
import sys
from optparse import OptionParser
def add_optval_option(pog, *args, **kwargs):
if 'empty' in kwargs:
empty_val = kwargs.pop('empty')
for i in range(1, len(sys.argv)):
a = sys.argv[i]
if a in args:
sys.argv.insert(i+1, empty_val)
break
pog.add_option(*args, **kwargs)
def main(args):
parser = OptionParser()
add_optval_option(parser,
'--foo', '-f',
default='MISSING',
empty='EMPTY',
help='"EMPTY" if given without a value. Note: '
'--foo=VALUE will work; --foo VALUE will *not*!')
o, a = parser.parse_args(args)
print 'Options:'
print ' --foo/-f:', o.foo
if a[1:]:
print 'Positional arguments:'
for arg in a[1:]:
print ' ', arg
else:
print 'No positional arguments'
if __name__=='__main__':
import sys
main(sys.argv)
Self-advertisement: This is part of the opo module of my thebops package ... ;-)