How to get shell tab completion cooperating with argparse in a Python script?
#!/usr/bin/env python
import argparse
def main(**args):
pass
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('positional', choices=['spam', 'eggs'])
parser.add_argument('--optional', choices=['foo1', 'foo2', 'bar'])
args = parser.parse_args()
main(**vars(args))
With an executable flag set on the .py file, the expected results should be something like:
$ ./example.py sp<tab>
-> completes to "./example.py spam"
$ ./example.py --op<tab>
-> completes to "./example.py --optional"
$ ./example.py --optional b<tab>
-> completes to "./example.py --optional bar"
$ ./example.py --optional f<tab>
-> completes to "./example.py --optional foo"
and, additionally, prints "foo1 foo2" choices on stdout on a new line
Have a look at argcomplete by Andrey Kislyuk.
Install it with:
pip install argcomplete
Import the module and add one line in your source before calling parser.parse_args():
#!/usr/bin/env python
import argparse as ap
import argcomplete
def main(**args):
pass
if __name__ == '__main__':
parser = ap.ArgumentParser()
parser.add_argument('positional', choices=['spam', 'eggs'])
parser.add_argument('--optional', choices=['foo1', 'foo2', 'bar'])
argcomplete.autocomplete(parser)
args = parser.parse_args()
main(**vars(args))
and to make sure that bash knows about this script, you use
eval "$(register-python-argcomplete your_script)"
you should put that line in your ~/.bashrc or follow argcomplete's docs and activate 'global' completion.
After that you completion works as requested.
The way this works is that the eval line creates a function _python_argcomlete which is registered using complete. (Run register-python-argcomplete your_script to just have a look at what gets eval-ed into bash). The autocomplete function looks for environment variables set by the bash completion mechanism to see if it needs to act. If it acts, it exits the program. If it doesn't act, this is a normal call to the program that function does nothing and the normal flow of the program continues.
For auto-complete to work you need a bash function to generate the possible options, and then you need to run complete -F <function_name> <program_name>
The best way of doing this is to have the program generate the completion function based on it's own parsing algorithm to avoid duplication. However, at a quick glance on argparse, I could not find a way to access it's internal structure, but I suggest you look for it.
Here is a bash function that will do for the above program:
function _example_auto() {
local cur=${COMP_WORDS[COMP_CWORD]}
local prev=${COMP_WORDS[COMP_CWORD-1]}
case "$prev" in
--optional )
COMPREPLY=( $(compgen -W "foo1 foo2 bar" -- $cur) )
return 0
;;
*)
COMPREPLY=( $(compgen -W "--optional spam eggs" -- $cur) )
return 0
;;
esac
}
Related
I want to log the parent command which ultimately executed my Python script. I want to captured it within the Python subprocess so I may include it in existing logs.
I have a shell script run.sh containing:
#!/bin/bash
python runme.py "$#"
And runme.py contains:
import sys
import subprocess
print("Doing stuff...")
# What command called me?
psoutput = subprocess.check_output(['ps', '--no-headers','-o','cmd'], text=True)
cmds = psoutput.strip().split('\n')
print(cmds[0])
I execute this from my company's shell (on RHEL) like this, and it outputs...
$ ./run.sh arg1 arg2 arg3
Doing stuff...
run.sh arg1 arg2 arg3
Looks good! But if another process is running first, it shows up first in ps and this doesn't work:
$ sleep 10s &
$ ./run.sh arg1 arg2 arg3
Doing stuff...
sleep 10s
I'm really trying to select my parent process from ps. This solution uses -C <process-name> but I don't know what my process name is. Following this solution I've tried this with several similar ideas for process name:
subprocess.check_output(['ps', '-o', 'ppid=', '-C', 'python runme.py'], text=True)
But that outputs a lot of numbers that look like PID's, but none seem related to my bash call.
How can I reliably get my parent process from within Python?
Pure python solution.
It works under any shell.
The parent cmd line result is a list with the command and the arguments.
import psutil
import os
ppid = psutil.Process(os.getppid())
print(ppid.cmdline())
One (okay) solution I realized while writing my question:
run.sh contains
command_run=$(echo "$BASH_SOURCE $#")
runme.py --command-run "$command_run" "$#"
runme.py contains
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
'-c', '--command-run', required=False,
default=None, dest='command_run',
help='Command text passed from bash'
)
args = parser.parse_args()
print("Doing stuff...")
print(args.command_run)
Additional arguments can be passed after adding more parsing. As above, the following works now:
$ sleep 5s &
$ ./run.sh
Doing stuff...
run.sh
I'll mark this answer as correct unless someone has a nicer solution than passing $BASH_SOURCE this way.
I am doing some practice Python exercises and I found one which is asking to create a Python module with various functions. I created a Python package and implemented the functions. So far so good, but a request is that if you call the module with the argument "-h", the message "Help" will be displayed, if the module is being imported, nothing is being displayed. How can we do this, is there any default function that needs to be overwritten? I'm not sure on how can we call a module, I thought we just use a package to better encapsulate our methods
Many thanks and sorry for being noob
Python is an interpreted language, that just starts with the top-level source code no main function, you might have seen in other languages. Files to be imported are written exactly the same way. All of the code, that is outside of functions is executed.
e.g.
myscript.py
def fn(a, b):
return a+b
print(fn(1, 1))
This as a fully working program, printing out the answer to, how much is 1+1. But what if you would like to import it to use the fn function inside another script? Doing import myscript would print 2 (and then finally provide you the fn function). The workaround is checking for __name__ == '__main__': inside myscript.py, which will evaluate to true, when being executed only (e.g. python myscript.py). It will be false otherwise (import myscript).
See the related Q&A.
Reference: https://docs.python.org/2/howto/argparse.html
$ cat sample.py
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--foo", help="sample argument", action="store", dest='foo')
args = parser.parse_args()
if args.foo:
print("foo = {}".format(args.foo))
$ python sample.py --help
$ python sample.py --help
usage: sample.py [-h] [--foo FOO]
optional arguments:
-h, --help show this help message and exit
--foo FOO sample argument
$
$ python sample.py --foo argument_value
$ python sample.py --foo bar
foo = bar
$ python sample.py --foo=10
foo = 10
$
Is it possible to run a python script with parameters in command line like this:
./hello(var=True)
or is it mandatory to do like this:
python -c "from hello import *;hello(var=True)"
The first way is shorter and simpler.
Most shells use parentheses for grouping or sub-shells. So you can't call any commands like command(arg) from a normal shell ...but you can write a python script (./hello.py) that takes an argument.
import optparse
parser = optparse.OptionParser()
parser.add_option('-f', dest="f", action="store_true", default=False)
options, remainder = parser.parse_args()
print ("Flag={}".format(options.f))
And the call it with python hello.py -f
./hello(var=True) would be impossible from REPL shell. In some case it could be useful to have python function available in your current shell session. Here a workaround to make your python functions available in your shell environment.
# python-tools.sh
#!/usr/bin/env bash
set -a # make all available export all variable)
function hello(){
cd "/app/python/commands"
python "test.py" $#
}
Content of the python script
#! /usr/bin/env python
# /app/python/commands/test.py script
import sys
def test(*args):
print(args)
if __name__ == '__main__':
if sys.argv[1] in globals().keys():
print(sys.argv[1])
globals()[sys.argv[1]](sys.argv[2:])
else:
print("%s Not known function" % sys.argv[1])
Then source python-tools.sh
source python-tools.sh
After the hello function is available
$ hello test arg2 arg2
test
(['arg2', 'arg2'],)
I need to start a python script with bash using nohup passing an arg that aids in defining a constant in a script I import. There are lots of questions about passing args but I haven't found a successful way using nohup.
a simplified version of my bash script:
#!/bin/bash
BUCKET=$1
echo $BUCKET
script='/home/path/to/script/script.py'
echo "starting $script with nohup"
nohup /usr/bin/python $script $BUCKET &
the relevant part of my config script i'm importing:
FLAG = sys.argv[0]
if FLAG == "b1":
AWS_ACCESS_KEY_ID = "key"
BUCKET = "bucket1"
AWS_SECRET_ACCESS_KEY = "secret"
elif FLAG == "b2":
AWS_ACCESS_KEY_ID = "key"
BUCKET = "bucket2"
AWS_SECRET_ACCESS_KEY = "secret"
else:
AWS_ACCESS_KEY_ID = "key"
BUCKET = "bucket3"
AWS_SECRET_ACCESS_KEY = "secret"
the script thats using it:
from config import BUCKET, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
#do stuff with the values.
Frankly, since I'm passing the args to script.py, I'm not confident that they'll be in scope for the import script. That said, when I take a similar approach without using nohup, it works.
In general, the argument vector for any program starts with the program itself, and then all of its arguments and options. Depending on the language, the program may be sys.argv[0], argv[0], $0, or something else, but it's basically always argument #0.
Each program whose job is to run another program—like nohup, and like the Python interpreter itself—generally drops itself and all of its own options, and gives the target program the rest of the command line.
So, nohup takes a COMMAND and zero or more ARGS. Inside that COMMAND, argv[0] will be COMMAND itself (in this case, '/usr/bin/python'), and argv[1] and later will be the additional arguments ('/home/path/to/script/script.py' and whatever $BUCKET resolves to).
Next, Python takes zero or more options, a script, and zero or more args to that script, and exposes the script and its args as sys.argv. So, in your script, sys.argv[0] will be '/home/path/to/script/script.py', and sys.argv[1] will be whatever $BUCKET resolves to.
And bash works similarly to Python; $1 will be the first argument to the bash wrapper script ($0 will be the script itself), and so on. So, sys.argv[1] in the inner Python script will end up getting the first argument passed to the bash wrapper script.
Importing doesn't affect sys.argv at all. So, in both your config module and your top-level script, if you import sys, sys.argv[1] will hold the $1 passed to the bash wrapper script.
(On some platforms, in some circumstances argv[0] may not have the complete path, or may even be empty. But that isn't relevant here. What you care about is the eventual sys.argv[1], and bash, nohup, and python are all guaranteed to pass that through untouched.)
nohup python3 -u ./train.py --dataset dataset_directory/ --model model_output_directory > output.log &
Here Im executing train.py file with python3, Then -u is used to ignore buffering and show the logs on the go without storing, specifying my dataset_directory with argument style and model_output_directory then Greater than symbol(>)
then the logs is stored in output.log and them atlast and(&) symbol is used
To terminate this process
ps ax | grep train
then note the process_ID
sudo kill -9 Process_ID
I have tested optcomplete working with the optparse module. Its example is a simple file so I could get that working. I also tested it using the argparse module as the prior one is deprecated. But I really do not understand how and by whom the python program gets called on tab presses. I suspect bash together with the shebang line and the argparse (or optparse) module are involved in some way. I have been trying to figure this out (now gonna read the source code).
I have a little more complex program structure, which includes a wrapper around the piece of code which handles the arguments. Its argparse.ArgumentParser() instantiation and calls to add_argument() - which are superclassed into another intermediate module to avoid duplicating code, and wrapper around that is being called - are inside a function.
I want to understand the way this tab completion works between bash and python (or for that matter any other interpretor like perl).
NOTE: I have a fair understanding of bash completion (which I learned just now), and I think I understand the bash(only) custom completion.
NOTE: I have read other similar SO questions, and none really answer this Q.
Edit: Here is the bash function.
I already understood how the python module gets to know about words typed in the command line, by reading os.environ values of variables
$COMP_WORDS
$COMP_CWORD
$COMP_LINE
$COMP_POINT
$COMPREPLY
These variables have values only on tab press.
My question is how does the python module gets triggered?
To understand what's happening here, let's check what that bash function actually does:
COMPREPLY=( $( \
COMP_LINE=$COMP_LINE COMP_POINT=$COMP_POINT \
COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD \
OPTPARSE_AUTO_COMPLETE=1 $1 ) )
See the $1 at the end? That means that it actually calls the Python file we want to execute with special environment variables set! To trace what's happening, let's prepare a little script to intercept what optcomplete.autocomplete does:
#!/usr/bin/env python2
import os, sys
import optparse, optcomplete
from cStringIO import StringIO
if __name__ == '__main__':
parser = optparse.OptionParser()
parser.add_option('-s', '--simple', action='store_true',
help="Simple really simple option without argument.")
parser.add_option('-o', '--output', action='store',
help="Option that requires an argument.")
opt = parser.add_option('-p', '--script', action='store',
help="Option that takes python scripts args only.")
opt.completer = optcomplete.RegexCompleter('.*\.py')
# debug env variables
sys.stderr.write("\ncalled with args: %s\n" % repr(sys.argv))
for k, v in sorted(os.environ.iteritems()):
sys.stderr.write(" %s: %s\n" % (k, v))
# setup capturing the actions of `optcomplete.autocomplete`
def fake_exit(i):
sys.stderr.write("autocomplete tried to exit with status %d\n" % i)
sys.stdout = StringIO()
sys.exit = fake_exit
# Support completion for the command-line of this script.
optcomplete.autocomplete(parser, ['.*\.tar.*'])
sys.stderr.write("autocomplete tried to write to STDOUT:\n")
sys.stderr.write(sys.stdout.getvalue())
sys.stderr.write("\n")
opts, args = parser.parse_args()
This gives us the following when we try to autocomplete it:
$ ./test.py [tab]
called with args: ['./test.py']
...
COMP_CWORD: 1
COMP_LINE: ./test.py
COMP_POINT: 10
COMP_WORDS: ./test.py
...
OPTPARSE_AUTO_COMPLETE: 1
...
autocomplete tried to exit with status 1
autocomplete tried to write to STDOUT:
-o -h -s -p --script --simple --help --output
So optcomplete.autocomplete just reads the environment, prepares the matches, writes them to STDOUT and exits. The result -o -h -s -p --script --simple --help --output is then put into a bash array (COMPREPLY=( ... )) and returned to bash to present the choices to the user. No magic involved :)