We Keep Coding

Using argparse, how can I process a "chdir" argument before fromfile expansion?

I want to support a sub-command CLI model, like used by git The particular bit I'm having trouble with is the "change directory" option. Like git, I want a -C DIR option which will have the program change to the specified directory before doing the sub-command. Not really a problem, using sub-parsers, BUT I also want to use the argparse.ArgumentParser(fromfile_prefix_chars='#') mechanism after the -C DIR argument is applied during parsing.
Here's the rub: fromfile argument expansion is performed by argparse before all other argument processing. Thus, any such fromfile arguments must either use absolute paths, or paths relative to the CWD at time the parser is invoked. I don't want absolute paths; I "need" to use fromfile paths that are relative to the -C DIR option. I wrote my own class ChdirAction(argparse.Action) to do the obvious. It worked fine, but since fromfile arguments were already expanded, it didn't give me what I want. (After discovering this not-what-I-want behavior, I looked at python3.5/argparse.py and found the same frustration embedded in cold, hard, unforgiving code.)
Here's a directory diagram that might help explain what I want:
/ foo / aaa / iii / arg.txt
| |
| + jjj / arg.txt
| |
| + arg.txt
|
+ bbb / iii / arg.txt
|
+ jjj / arg.txt
Consider when the CWD is either aaa or bbb at the time command line arguments are parsed. If I run with something like prog -C ./iii #arg.txt
I want the parser to expand #arg.txt with arguments from /foo/aaa/iii/arg.txt. What actually happens is that fromfile expands from the contents of /foo/aaa/arg.txt. When CWD is /foo/aaa this is the "wrong" file; when /foo/bbb it raises "error: [Errno 2] No such file or directory: 'arg.txt'"
More generally, prog -C ./DIR #arg.txt should expand from /foo/aaa/DIR/arg.txt which should work even the fromfile has "up-directory" parts, e.g. prog -C ./iii #../arg.txt should expand from /foo/aaa/arg.txt.
If this behavior can be made to happen, then I could -C DIR to any of {aaa,bbb}/{iii,jjj} and obtain consitent behaviour from a common command line construction.
As described, my problem isn't much of a problem. If can provide the -C DIR, to be realized by an os.chdir(DIR) after argument parsing, then I can also construct appropriate fromfile arguments. They could be either absolute or relative to the CWD at parsing (prior to any -C DIR taking effect). This might look like:
cd /foo/aaa; prog -C ./DIR #arg.txt #./DIR/arg.txt
I don't like it, but it would be okay. The REAL problem is that the actual change-directory argument I'm using is more like -C PATTERN. In my real problem case, PATTERN could be a simple path (absolute or relative). Or, it might be a glob pattern, or a partial name that has "non-trivial" resolution logic to find the actual directory for os.chdir(DIR). In this case (which I am struggling with), I can't have the invoker of the program resolve the actual location of the fromfile path.
Actually, I could, but that would put an inappropriate burden on the invoker. AND, when that invoker is an Eclipse launcher, I don't really have the control-flow power necessary to do it. So, it's back to having the program take care of it's own needs; a nicer abstraction, but how do I implement it?

Even as I was fleshing out the question, I came up with an idea. So I tried it out and it's kinda, sorta, okay(ish). I can get a constrained version of what I really want, but it's good enough for me (for now), so I thought I might as well share. It might be good enough for you, too. Even better, it might elicit a true solution from somewhere, maybe S.Bethard?
My hack is to do parsing in two phases: the first, is just enough to get the -C PATTERN argument by way of ArgumentParser.parse_known_args(...) without enabling the fromfile mechanism. If the result of that first (minimal) parsing yields a directory change argument, then I process it. The program aborts if more than a single -C PATTERN was specified, or the PATTERN can't be unambiguously resolved.
Then, I use a completely separate ArgumentParser object, configured with the full set of argument specifications that I actually want and parse it with the fromfile mechanism enabled.
There is some monkey business to get the --help argument to work (setting the proper conflict resolution policy, then merely accepting the arg in the first parser just to pass along to the second, which actually has all the "real" argument specs). Also, the first parser should support the same verbose/quiet options that the second one does, honoring their setting and also passing along from first to second parser.
Here's a simplified version of my application-level arg parser method. It doens't support verbose/quiet options at the first parser stage. I've elided the complexity of how a -C PATTERN is resolved to an actual directory. Also, I cut out the majority of the second parser's argument specification, leaving just the second parser's -C PATTERN argument (needed for --help output).
NOTE: Both parsers have a -C PATTERN argument. In the chdirParser it is meaningful; in the argParser it's present only so it will show up in the help output. Something similar should be done for verbose/quiet options - probably not that trixy, but it's not (yet) important to me so I don't mind always reporting a change of directory, even in quiet mode.
def cli_args_from_argv():
import argparse
import glob
import os
import sys
chdirParser = argparse.ArgumentParser(conflict_handler='resolve')
chdirParser.add_argument("-C", dest="chdir_pattern", action="append" , default=None)
chdirParser.add_argument("--help", "-h", dest="help", action="store_true", default=False)
(partial, remainder) = chdirParser.parse_known_args()
if partial.help:
remainder = ['--help']
elif partial.chdir_pattern:
if len(partial.chdir_pattern) > 1:
print(r'Too many -C options - at most one may be given, but received: {!r}'.format(partial.chdir_pattern), file=sys.stderr)
sys.exit(1)
pattern = partial.chdir_pattern[0]
resolved_dir = pattern
if os.path.exists(resolved_dir):
resolved_dir = pattern
else:
### ELIDED: resolution of pattern into an unambiguous and existing directory
if not resolved_dir:
print("Failed to resolve -C {!r}".format(pattern), file=sys.stderr)
sys.exit(1)
print("Changing to directory: {!r}".format(resolved_dir))
print("");
os.chdir(target_dir)
argParser = argparse.ArgumentParser(usage="usage: PROG [common-args] SUBCMD [subcmd-args]", fromfile_prefix_chars=':')
### ELIDED: a bunches of add_argument(...)
argParser.add_argument("-C", dest="chdir_spec", action="store", default=None, help="Before anything else, chdir to SPEC", metavar="SPEC")
return argParser.parse_args(args=remainder)
I have a feeling that there's probably a better way... Do you know?

I think the resolve bit can be replaced with
chdirParser = argparse.ArgumentParser(add_help=False)
and omit the -h definition and save. Let the second parser handle sys.argv unchanged (since you are including (but ignoring) the -C argument).
That append and test for len(partial.chdir_pattern) > 1 should work if you expect the user to use several -C dir1 ... -C dir2... commands. The alternative to use the default store action, which ends up saving the last of those repetitions. Why might the user repeat the -C, and why should you care? Usually we just ignore repetitions.
You might replace
print("Failed to resolve -C {!r}".format(pattern), file=sys.stderr)
sys.exit(1)
with
parser.error("Failed to resolve -C {!r}".format(pattern)')
It prints the usage (with only -C) and does ansys.exit(2)`. Not quite the same, but may be close enough.
For the second parser, the -C might be simplified (using defaults):
argParser.add_argument("-C", "--chdir-spec", help="Before anything else, chdir to SPEC", metavar="SPEC")
And use the full sys.argv.
return argParser.parse_args()
Otherwise, using 2 parsers makes sense, since the fromfile is present in the changed directory (and you want to ignore any such file in the initial directory).
I thought maybe a :arg.txt string the commandline would give problems in the first parser. But with parse_known_args it will just treat it as an unknown positional. But the proof's in the testing.

For the life of me I cannot make a file-set argument the next to last one

I want to run something like:
python command.py -i -c "f*.xxx" "search"
This fine since "file-set" is not expanded" but:
python command.py -i -c f*.xxx "search"
This is expanded so
sys.argv = ['command.py','-i', '-c', 'f1.xxx','f2.xxx','search']
Why couldn't it be ['command.py','-i', '-c', ['f1.xxx','f2.xxx'],'search'] ?
Since the "f*.xxx" is not accessible I have no way to know if 'search' is a file name and the real 'search' missing. I want to print an error message that
says "Please use quotes". And I must keep position this way and turning off
globbing is not an option. getopt and argparse do not solve this.
Thanks

It's your shell, not Python, that's doing the expansion. Python always sees a single flat list of arguments.
In case you want to parse arguments, which basically turns this flat list into a more complex data structure, you can use the argparse module, or use more extensive third party projects like click.

Python script argument conditional

Is anyone able to tell me how to write a conditional for an argument on a python script? I want it to print "Argument2 Entered" if it is run with a second command line arguments such as:
python script.py argument1 argument2
And print "No second argument" if it is run without command line arguments, like this:
python script.py argument1
Is this possible?

import sys
if len(sys.argv)==2: # first entry in sys.argv is script itself...
print "No second argument"
elif len(sys.argv)==3:
print "Second argument"

There are many answers to this, depending on what exactly you want to do and how much flexibility you are likely to need.
The simplest solution is to examine the variable sys.argv, which is a list containing all of the command-line arguments. (It also contains the name of the script as the first element.) To do this, simply look at len(sys.argv) and change behaviour based on its value.
However, this is often not flexible enough for what people expect command-line programs to do. For example, if you want a flag (-i, --no-defaults, ...) then it's not obvious how to write one with just sys.argv. Likewise for arguments (--dest-dir="downloads"). There are therefore many modules people have written to simplify this sort of argument parsing.
The built-in solution is argparse, which is powerful and pretty easy-to-use but not particularly concise.
A clever solution is plac, which inspects the signature of the main function to try to deduce what the command-line arguments should be.

There are many ways to do this simple thing in Python. If you are interested to know more than I recommend to read this article. BTW I am giving you one solution below:
import click
'''
Prerequisite: # python -m pip install click
run: python main.py ttt yyy
'''
#click.command(context_settings=dict(ignore_unknown_options=True))
#click.argument("argument1")
#click.argument("argument2")
def main(argument1, argument2):
print(f"argument1={argument1} and argument2={argument2}")
if __name__ == '__main__':
main()

Following block should be self explanatory
$ ./first.py second third 4th 5th
5
$ cat first.py
#!/usr/bin/env python
import sys
print (len(sys.argv))

This is related to many other posts depending upon where you are going with this, so I'll put four here:
What's the best way to grab/parse command line arguments passed to a Python script?
Implementing a "[command] [action] [parameter]" style command-line interfaces?
How can I process command line arguments in Python?
How do I format positional argument help using Python's optparse?
But the direct answer to your question from the Python docs:
sys.argv -
The list of command line arguments passed to a Python script. argv[0] is the script name (it is operating system dependent whether this is a full pathname or not). If the command was executed using the -c command line option to the interpreter, argv[0] is set to the string '-c'. If no script name was passed to the Python interpreter, argv[0] is the empty string.
To loop over the standard input, or the list of files given on the command line, see the fileinput module.

kwargs in python executables

I'm trying to create a program that can be called from the command line and use keyword arguments in python 2.6. So far I've tried:
#!/usr/bin/python
def read(foo = 5):
print foo
return 0
if __name__ == '__main__'
read()
When I try to run this from the command line: ./test.py the program prints 5 as expected. Is there a way to use ./test.py foo=6? I want to preserve the keyword arguments.
It seems like a simple question, but I haven't found a good source for this.

python has built in library to help you achieve passing command line arguments to a script
argparse. THe usage is a little different then what you are describing in your question though...
On a basic level you can access all command line arguments by sys.argv, which will be a list of arguments
Sorry should have mentioned the python 2.6 library is called optparse

Something like this?
if __name__ == '__main__':
kwargs = dict(x.split('=', 1) for x in sys.argv[1:])
read(**kwargs)
That said, argparse and optparse are probably going to give you something more robust and more natural for someone used to the commandline. (Not to mention, supporting arguments of types other than string.)
Oh, and if what you're really after is just interactive use of your function, use the interactive interpreter; either python or ipython. You'd need to put the code into a file ending in .py and import it, then you could just call it.

A less usual, but very interesting alternative is docopt: a library that generates an argument parser from the help message that you write for your program (on github).

How to make a custom command line interface using OptionParser?

I am using the OptionParser from optparse module to parse my command that I get using the raw_input().
I have these questions.
1.) I use OptionParser to parse this input, say for eg. (getting multiple args)
my prompt> -a foo -b bar -c spam eggs
I did this with setting the action='store_true' in add_option() for '-c',now if there is another option with multiple argument say -d x y z then how to know which arguments come from which option? also if one of the arguments has to be parsed again like
my prompt> -a foo -b bar -c spam '-f anotheroption'
2.) if i wanted to do something like this..
my prompt> -a foo -b bar
my prompt> -c spam eggs
my prompt> -d x y z
now each entry must not affect the other options set by the previous command. how to accomplish these?

For part 2: you want a new OptionParser instance for each line you process. And look at the cmd module for writing a command loop like this.

You can also solve #1 using the nargs option attribute as follows:
parser = OptionParser()
parser.add_option("-c", "", nargs=2)
parser.add_option("-d", "", nargs=3)

optparse solves #1 by requiring that an argument always have the same number of parameters (even if that number is 0), variable-parameter arguments are not allowed:
Typically, a given option either takes
an argument or it doesn’t. Lots of
people want an “optional option
arguments” feature, meaning that some
options will take an argument if they
see it, and won’t if they don’t. This
is somewhat controversial, because it
makes parsing ambiguous: if "-a" takes
an optional argument and "-b" is
another option entirely, how do we
interpret "-ab"? Because of this
ambiguity, optparse does not support
this feature.
You would solve #2 by not reusing the previous values to parse_args, so it would create a new values object rather than update.