I want to log the parent command which ultimately executed my Python script. I want to captured it within the Python subprocess so I may include it in existing logs.
I have a shell script run.sh containing:
#!/bin/bash
python runme.py "$#"
And runme.py contains:
import sys
import subprocess
print("Doing stuff...")
# What command called me?
psoutput = subprocess.check_output(['ps', '--no-headers','-o','cmd'], text=True)
cmds = psoutput.strip().split('\n')
print(cmds[0])
I execute this from my company's shell (on RHEL) like this, and it outputs...
$ ./run.sh arg1 arg2 arg3
Doing stuff...
run.sh arg1 arg2 arg3
Looks good! But if another process is running first, it shows up first in ps and this doesn't work:
$ sleep 10s &
$ ./run.sh arg1 arg2 arg3
Doing stuff...
sleep 10s
I'm really trying to select my parent process from ps. This solution uses -C <process-name> but I don't know what my process name is. Following this solution I've tried this with several similar ideas for process name:
subprocess.check_output(['ps', '-o', 'ppid=', '-C', 'python runme.py'], text=True)
But that outputs a lot of numbers that look like PID's, but none seem related to my bash call.
How can I reliably get my parent process from within Python?
Pure python solution.
It works under any shell.
The parent cmd line result is a list with the command and the arguments.
import psutil
import os
ppid = psutil.Process(os.getppid())
print(ppid.cmdline())
One (okay) solution I realized while writing my question:
run.sh contains
command_run=$(echo "$BASH_SOURCE $#")
runme.py --command-run "$command_run" "$#"
runme.py contains
import argparse
parser = argparse.ArgumentParser()
parser.add_argument(
'-c', '--command-run', required=False,
default=None, dest='command_run',
help='Command text passed from bash'
)
args = parser.parse_args()
print("Doing stuff...")
print(args.command_run)
Additional arguments can be passed after adding more parsing. As above, the following works now:
$ sleep 5s &
$ ./run.sh
Doing stuff...
run.sh
I'll mark this answer as correct unless someone has a nicer solution than passing $BASH_SOURCE this way.
Related
So to be more precise, what I am trying to do is :
read a full shell command as argument of my python script like : python myPythonScript.py ls -Fl
Call that command within my python script when I'd like to (Make some loops on some folders and apply the command etc ...)
I tried this :
import subprocess
from optparse import OptionParser
from subprocess import call
def execCommand(cmd):
call(cmd)
if __name__ == '__main__':
parser = OptionParser()
(options,args) = parser.parse_args()
print args
execCommand(args)
The result is that now I can do python myPythonScript.py ls , but I don't know how to add options. I know I can use parser.add_option , but don't know how to make it work for all options as I don't want to make only specific options available, but all possible options depending on the command I am running.
Can I use something like parser.add_option('-*') ? How can I parse the options then and call the command with its options ?
EDIT
I need my program to parse all type of commands passed as argument : python myScript.py ls -Fl , python myScript.py git pull, python myScript rm -rf * etc ...
OptionParser is useful when your own program wants to process the arguments: it helps you turn string arguments into booleans or integers or list items or whatever. In your case, you just want to pass the arguments on to the program you're invoking, so don't bother with OptionParser. Just pass the arguments as given in sys.argv.
subprocess.call(sys.argv[1:])
Depending on how much your program depends on command line arguments, you can go with simple route.
Simple way of reading command line arguments
Use sys to obtain all the arguments to python command line.
import sys
print sys.argv[1:]
Then you can use subprocess to execute it.
from subprocess import call
# e.g. call(["ls", "-l"])
call(sys.argv[1:])
This sample below works fine for me.
import sys
from subprocess import call
print(sys.argv[1:])
call(sys.argv[1:])
Is it possible to run a python script with parameters in command line like this:
./hello(var=True)
or is it mandatory to do like this:
python -c "from hello import *;hello(var=True)"
The first way is shorter and simpler.
Most shells use parentheses for grouping or sub-shells. So you can't call any commands like command(arg) from a normal shell ...but you can write a python script (./hello.py) that takes an argument.
import optparse
parser = optparse.OptionParser()
parser.add_option('-f', dest="f", action="store_true", default=False)
options, remainder = parser.parse_args()
print ("Flag={}".format(options.f))
And the call it with python hello.py -f
./hello(var=True) would be impossible from REPL shell. In some case it could be useful to have python function available in your current shell session. Here a workaround to make your python functions available in your shell environment.
# python-tools.sh
#!/usr/bin/env bash
set -a # make all available export all variable)
function hello(){
cd "/app/python/commands"
python "test.py" $#
}
Content of the python script
#! /usr/bin/env python
# /app/python/commands/test.py script
import sys
def test(*args):
print(args)
if __name__ == '__main__':
if sys.argv[1] in globals().keys():
print(sys.argv[1])
globals()[sys.argv[1]](sys.argv[2:])
else:
print("%s Not known function" % sys.argv[1])
Then source python-tools.sh
source python-tools.sh
After the hello function is available
$ hello test arg2 arg2
test
(['arg2', 'arg2'],)
I am writing a python script which uses os.system command to call a shell script.I need help understanding how I can pass the arguments to the shell script? Below is what I am trying..but it wouldn't work.
os.system('./script.sh arg1 arg2 arg3')
I do not want to use subprocess for calling the shell script. Any help is appreciated.
Place your script and it's args into a string, see example below.
HTH
#!/usr/bin/env python
import os
arg3 = 'arg3'
cmd = '/bin/echo arg1 arg2 %s' % arg3
print 'running "%s"' % cmd
os.system(cmd)
If you insert the following line before os.system (...), you will likely see your problem.
print './script.sh arg1 arg2 arg3'
When developing this type of thing, it usually is useful to make sure the command really is what you expect, before you actually try it.
example:
def Cmd():
return "something"
print Cmd()
when you are satisfied, comment out the print Cmd() line and use os.system (Cmd ()) or subprocess version.
How to get shell tab completion cooperating with argparse in a Python script?
#!/usr/bin/env python
import argparse
def main(**args):
pass
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('positional', choices=['spam', 'eggs'])
parser.add_argument('--optional', choices=['foo1', 'foo2', 'bar'])
args = parser.parse_args()
main(**vars(args))
With an executable flag set on the .py file, the expected results should be something like:
$ ./example.py sp<tab>
-> completes to "./example.py spam"
$ ./example.py --op<tab>
-> completes to "./example.py --optional"
$ ./example.py --optional b<tab>
-> completes to "./example.py --optional bar"
$ ./example.py --optional f<tab>
-> completes to "./example.py --optional foo"
and, additionally, prints "foo1 foo2" choices on stdout on a new line
Have a look at argcomplete by Andrey Kislyuk.
Install it with:
pip install argcomplete
Import the module and add one line in your source before calling parser.parse_args():
#!/usr/bin/env python
import argparse as ap
import argcomplete
def main(**args):
pass
if __name__ == '__main__':
parser = ap.ArgumentParser()
parser.add_argument('positional', choices=['spam', 'eggs'])
parser.add_argument('--optional', choices=['foo1', 'foo2', 'bar'])
argcomplete.autocomplete(parser)
args = parser.parse_args()
main(**vars(args))
and to make sure that bash knows about this script, you use
eval "$(register-python-argcomplete your_script)"
you should put that line in your ~/.bashrc or follow argcomplete's docs and activate 'global' completion.
After that you completion works as requested.
The way this works is that the eval line creates a function _python_argcomlete which is registered using complete. (Run register-python-argcomplete your_script to just have a look at what gets eval-ed into bash). The autocomplete function looks for environment variables set by the bash completion mechanism to see if it needs to act. If it acts, it exits the program. If it doesn't act, this is a normal call to the program that function does nothing and the normal flow of the program continues.
For auto-complete to work you need a bash function to generate the possible options, and then you need to run complete -F <function_name> <program_name>
The best way of doing this is to have the program generate the completion function based on it's own parsing algorithm to avoid duplication. However, at a quick glance on argparse, I could not find a way to access it's internal structure, but I suggest you look for it.
Here is a bash function that will do for the above program:
function _example_auto() {
local cur=${COMP_WORDS[COMP_CWORD]}
local prev=${COMP_WORDS[COMP_CWORD-1]}
case "$prev" in
--optional )
COMPREPLY=( $(compgen -W "foo1 foo2 bar" -- $cur) )
return 0
;;
*)
COMPREPLY=( $(compgen -W "--optional spam eggs" -- $cur) )
return 0
;;
esac
}
I have tested optcomplete working with the optparse module. Its example is a simple file so I could get that working. I also tested it using the argparse module as the prior one is deprecated. But I really do not understand how and by whom the python program gets called on tab presses. I suspect bash together with the shebang line and the argparse (or optparse) module are involved in some way. I have been trying to figure this out (now gonna read the source code).
I have a little more complex program structure, which includes a wrapper around the piece of code which handles the arguments. Its argparse.ArgumentParser() instantiation and calls to add_argument() - which are superclassed into another intermediate module to avoid duplicating code, and wrapper around that is being called - are inside a function.
I want to understand the way this tab completion works between bash and python (or for that matter any other interpretor like perl).
NOTE: I have a fair understanding of bash completion (which I learned just now), and I think I understand the bash(only) custom completion.
NOTE: I have read other similar SO questions, and none really answer this Q.
Edit: Here is the bash function.
I already understood how the python module gets to know about words typed in the command line, by reading os.environ values of variables
$COMP_WORDS
$COMP_CWORD
$COMP_LINE
$COMP_POINT
$COMPREPLY
These variables have values only on tab press.
My question is how does the python module gets triggered?
To understand what's happening here, let's check what that bash function actually does:
COMPREPLY=( $( \
COMP_LINE=$COMP_LINE COMP_POINT=$COMP_POINT \
COMP_WORDS="${COMP_WORDS[*]}" COMP_CWORD=$COMP_CWORD \
OPTPARSE_AUTO_COMPLETE=1 $1 ) )
See the $1 at the end? That means that it actually calls the Python file we want to execute with special environment variables set! To trace what's happening, let's prepare a little script to intercept what optcomplete.autocomplete does:
#!/usr/bin/env python2
import os, sys
import optparse, optcomplete
from cStringIO import StringIO
if __name__ == '__main__':
parser = optparse.OptionParser()
parser.add_option('-s', '--simple', action='store_true',
help="Simple really simple option without argument.")
parser.add_option('-o', '--output', action='store',
help="Option that requires an argument.")
opt = parser.add_option('-p', '--script', action='store',
help="Option that takes python scripts args only.")
opt.completer = optcomplete.RegexCompleter('.*\.py')
# debug env variables
sys.stderr.write("\ncalled with args: %s\n" % repr(sys.argv))
for k, v in sorted(os.environ.iteritems()):
sys.stderr.write(" %s: %s\n" % (k, v))
# setup capturing the actions of `optcomplete.autocomplete`
def fake_exit(i):
sys.stderr.write("autocomplete tried to exit with status %d\n" % i)
sys.stdout = StringIO()
sys.exit = fake_exit
# Support completion for the command-line of this script.
optcomplete.autocomplete(parser, ['.*\.tar.*'])
sys.stderr.write("autocomplete tried to write to STDOUT:\n")
sys.stderr.write(sys.stdout.getvalue())
sys.stderr.write("\n")
opts, args = parser.parse_args()
This gives us the following when we try to autocomplete it:
$ ./test.py [tab]
called with args: ['./test.py']
...
COMP_CWORD: 1
COMP_LINE: ./test.py
COMP_POINT: 10
COMP_WORDS: ./test.py
...
OPTPARSE_AUTO_COMPLETE: 1
...
autocomplete tried to exit with status 1
autocomplete tried to write to STDOUT:
-o -h -s -p --script --simple --help --output
So optcomplete.autocomplete just reads the environment, prepares the matches, writes them to STDOUT and exits. The result -o -h -s -p --script --simple --help --output is then put into a bash array (COMPREPLY=( ... )) and returned to bash to present the choices to the user. No magic involved :)