We Keep Coding

Returning status of grep command executed remotely via ssh embedded in expect command inside python

I have a below python script where in I am executing a remote SSH command using expect. Here, even the target file contains the string "error" or not, the exit code is returned as success always as the ssh connectivity is what only checked. How can I get the status of the grep command?
#! /usr/bin/python
import subprocess
def execute(cmd):
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
_output, _error = proc.communicate()
_code = proc.returncode
return _output, _error, _code
host = "localhost"
passwd = "kube"
cmd="/usr/bin/expect -c 'spawn ssh "+host+" \"cat /home/kube/f1 | grep -qi error\"; expect \"password:\" { send \""+passwd+"\r\"} ;interact' "
_output, _error, return_code = execute(cmd)
print cmd + "\n" + _output + "\n" + _error
if (return_code == 0):
print "no error"
else:
print "contains error"

Option 1:
Let the remote command output something which indicates success/failure for you. E.g.:
ssh user#host "/some/command | grep -q some-string && echo MAGIC-SUCCESS || echo MAGIC-FAILURE"
And in Python you can get the output and parse it.
Option 2:
According to man expect:
wait [args]
[...] wait normally returns a list of four integers. The first integer is the pid of the process that was waited upon. The second integer is the corresponding spawn id. The third integer is -1 if an operating system error occurred, or 0 otherwise. If the third integer was 0, the fourth integer is the status returned by the spawned process. If the third integer was -1, the fourth integer is the value of errno set by the operating system. [...]
So your Expect code can check the wait result and then exit with different values and Python code can get the exit status.
For Expect part it's like this:
spawn ...
...
expect eof
set result [wait]
exit [lindex $result 3]

run cmake command via subprocess.Popen or subprocess.run [duplicate]

If I run echo a; echo b in bash the result will be that both commands are run. However if I use subprocess then the first command is run, printing out the whole of the rest of the line.
The code below echos a; echo b instead of a b, how do I get it to run both commands?
import subprocess, shlex
def subprocess_cmd(command):
process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
proc_stdout = process.communicate()[0].strip()
print proc_stdout
subprocess_cmd("echo a; echo b")

You have to use shell=True in subprocess and no shlex.split:
import subprocess
command = "echo a; echo b"
ret = subprocess.run(command, capture_output=True, shell=True)
# before Python 3.7:
# ret = subprocess.run(command, stdout=subprocess.PIPE, shell=True)
print(ret.stdout.decode())
returns:
a
b

I just stumbled on a situation where I needed to run a bunch of lines of bash code (not separated with semicolons) from within python. In this scenario the proposed solutions do not help. One approach would be to save a file and then run it with Popen, but it wasn't possible in my situation.
What I ended up doing is something like:
commands = '''
echo "a"
echo "b"
echo "c"
echo "d"
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands)
print out
So I first create the child bash process and after I tell it what to execute. This approach removes the limitations of passing the command directly to the Popen constructor.

Join commands with "&&".
os.system('echo a > outputa.txt && echo b > outputb.txt')

If you're only running the commands in one shot then you can just use subprocess.check_output convenience function:
def subprocess_cmd(command):
output = subprocess.check_output(command, shell=True)
print output

>>> command = "echo a; echo b"
>>> shlex.split(command);
['echo', 'a; echo', 'b']
so, the problem is shlex module do not handle ";"

Got errors like when I used capture_output=True
TypeError: __init__() got an unexpected keyword argument 'capture_output'
After made changes like as below and its works fine
import subprocess
command = '''ls'''
result = subprocess.run(command, stdout=subprocess.PIPE,shell=True)
print(result.stdout.splitlines())

import subprocess
cmd = "vsish -e ls /vmkModules/lsom/disks/ | cut -d '/' -f 1 | while read diskID ; do echo $diskID; vsish -e cat /vmkModules/lsom/disks/$diskID/virstoStats | grep -iE 'Delete pending |trims currently queued' ; echo '====================' ;done ;"
def subprocess_cmd(command):
process = subprocess.Popen(command,stdout=subprocess.PIPE, shell=True)
proc_stdout = process.communicate()[0].strip()
for line in proc_stdout.decode().split('\n'):
print (line)
subprocess_cmd(cmd)

Popen Subprocess Python

I am using Popen to remote call script
s = Popen(['ssh' , ssh_argument0 , ssh_argument1 , '/tmp/remote/dude_rc_admsvr.sh %s %s' %(DomainHome , ACTIVITY)])
stdout = s.communicate()
print stdout
The script is not exiting with the status mentioned under shell script , instead it only prints success or failure status only ..
i want to exit with the status codes as per shell script.
here is shell script
tail -Fn0 ${ADM_DOMAIN_LOG} | \
while read LOG_LINE;
do
echo ${LOG_LINE} | grep -q "${PASS_MSG}"
if [ $? = 0 ]
then
echo "${STATUS_SUCCESS}"
exit 0
elif echo ${LOG_LINE} | grep -q "${FAIL_MSG}"
then
echo "${STATUS_FAILURE}"
exit 1
elif echo ${LOG_LINE} | grep -q "${FAIL_MSG2}"
then
echo "${STATUS_FAILURE}"
exit 1
fi
done
exit
How to get the status code returned ?

From subprocess.check_call's documentation:
If the return code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the return
code in the returncode attribute.
Therefore, to get the status code, you need to catch the exception and retrieve the returncode attribute.
try:
check_call(['ssh' , ssh_argument0 , ssh_argument1 , '/tmp/remote/dude_rc_admsvr.sh %s %s' %(DomainHome , ACTIVITY)])
except subprocess.CalledProcessError as e:
print('status code = %s' % e.returncode)

Adding a new answer since you changed your code in question from using check_call to using Popen and communicate.
From subprocess.communicate's documentation:
Note that if you want to send data to the process’s stdin, you need to
create the Popen object with stdin=PIPE. Similarly, to get anything
other than None in the result tuple, you need to give stdout=PIPE
and/or stderr=PIPE too.
In other words, you need to add stdout=PIPE to your call to Popen's constructor in order to use communicate.
s = Popen(['ssh' , ssh_argument0 , ssh_argument1 , '/tmp/remote/dude_rc_admsvr.sh %s %s' %(DomainHome , ACTIVITY)], stdout=PIPE)
stdout = s.communicate()
returncode = s.returncode

Found Issue :
The tail statement in shell script was holding the SSH session which is why Popen couldn't able to exit the subshell , Adding kill -9 ps -eaf |grep tail resolved the issue.

How to avoid passing shell constructs to executable using Popen

I am trying to call an executable called foo, and pass it some command line arguments. An external script calls into the executable and uses the following command:
./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log
The script uses Popen to execute this command as follows:
from subprocess import Popen
from subprocess import PIPE
def run_command(command, returnObject=False):
cmd = command.split(' ')
print('%s' % cmd)
p = None
print('command : %s' % command)
if returnObject:
p = Popen(cmd)
else:
p = Popen(cmd)
p.communicate()
print('returncode: %s' % p.returncode)
return p.returncode
return p
command = "./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log
"
run_command(command)
However, this passes extra arguments ['2>&1', '|', '/usr/bin/tee', 'temp.log'] to the foo executable.
How can I get rid of these extra arguments getting passed to foo while maintaining the functionality?
I have tried shell=True but read about avoiding it for security purposes (shell injection attack). Looking for a neat solution.
Thanks
UPDATE:
- Updated the file following the tee command

The string
./main/foo --config config_file 2>&1 | /usr/bin/tee >temp.log
...is full of shell constructs. These have no meaning to anything without a shell in play. Thus, you have two options:
Set shell=True
Replace them with native Python code.
For instance, 2>&1 is the same thing as passing stderr=subprocess.STDOUT to Popen, and your tee -- since its output is redirected and it's passed no arguments -- could just be replaced with stdout=open('temp.log', 'w').
Thus:
p = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=open('temp.log', 'w'))
...or, if you really did want the tee command, but were just using it incorrectly (that is, if you wanted tee temp.log, not tee >temp.log):
p1 = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['tee', 'temp.log'], stdin=p1.stdout)
p1.stdout.close() # drop our own handle so p2's stdin is the only handle on p1.stdout
stdout, _ = p2.communicate()
Wrapping this in a function, and checking success for both ends might look like:
def run():
p1 = subprocess.Popen(['./main/foo', '--config', 'config_file'],
stderr=subprocess.STDOUT,
stdout=subprocess.PIPE)
p2 = subprocess.Popen(['tee', 'temp.log'], stdin=p1.stdout)
p1.stdout.close() # drop our own handle so p2's stdin is the only handle on p1.stdout
# True if both processes were successful, False otherwise
return (p2.wait() == 0 && p1.wait() == 0)
By the way -- if you want to use shell=True and return the exit status of foo, rather than tee, things get a bit more interesting. Consider the following:
p = subprocess.Popen(['bash', '-c', 'set -o pipefail; ' + command_str])
...the pipefail bash extension will force the shell to exit with the status of the first pipeline component to fail (and 0 if no components fail), rather than using only the exit status of the final component.

Here's a couple of "neat" code examples in addition to the explanation from #Charles Duffy answer.
To run the shell command in Python:
#!/usr/bin/env python
from subprocess import check_call
check_call("./main/foo --config config_file 2>&1 | /usr/bin/tee temp.log",
shell=True)
without the shell:
#!/usr/bin/env python
from subprocess import Popen, PIPE, STDOUT
tee = Popen(["/usr/bin/tee", "temp.log"], stdin=PIPE)
foo = Popen("./main/foo --config config_file".split(),
stdout=tee.stdin, stderr=STDOUT)
pipestatus = [foo.wait(), tee.wait()]
Note: don't use "command arg".split() with non-literal strings.
See How do I use subprocess.Popen to connect multiple processes by pipes?

You may combine answers to two StackOverflow questions:
1. piping together several subprocesses
x | y problem
2. Merging a Python script's subprocess' stdout and stderr (while keeping them distinguishable)
2>&1 problem

Subprocess.Popen behaves differently in interpreter, executable scripts

Let's say you have the following:
command = shlex.split("mcf -o -q -e -w %s %s" % (SOLFILE, NETFILE))
task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print "stdout: %s" % stdout #debugging
print "stderr: %s" % stderr #debugging
if stderr:
sys.exit("MCF crashed on %s" % NETFILE)
It's not necessary to know what mcf is, except that it's a C program which will overflow if it's not given a satisfiable netfile. (Why can't I just ensure that all the netfiles are satisfiable? Well, because the easiest way to check that is to feed it to mcf and see if it overflows...)
Anyway, when I run this in an executable script, task.communicate() doesn't seem to store anything in stdout and stderr. (To be precise, I get stdout == stderr == ''.) Instead, the stderr stream from mcf seems to be "leaking" to the terminal rather than getting captured by the subprocess pipe. Here's some sample output to illustrate:
Netfile: facility3cat_nat5000_wholesaler_capacitation_test_.net
Solfile: facility3cat_nat5000_wholesaler_capacitation_test_.sol
*** buffer overflow detected ***: mcf terminated
======= Backtrace: =========
...
...[fifty lines of Linda Blair-esque output]...
...
stdout: None
stderr:
...[program continues, since stderr did not evaluate to True]...
This only fails when running the script from the command line. When I step through it line by line in the interpreter, stdout and stderr are correctly assigned:
>>> task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = task.communicate()
>>> stderr
'*** buffer overflow detected ***: mcf terminated\n======= Backtrace: =========\n'
...[more headspinning and vomit]...
Could anyone help me to understand why this works in the interpreter, but not when executed? Thanks in advance!

I wrote a little test script to test the subprocess module with.
#!/bin/bash
echo echo to stderr 1>&2
echo echo to stdout
Then I wrote a small Python script that calls it:
#!/usr/bin/python
import subprocess
command = ('./joe.sh',)
task = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print 'stdout == %r\nstderr == %r' % (stdout, stderr)
The output of running it looks just like this:
$ python joe.py
stdout == 'echo to stdout\n'
stderr == 'echo to stderr\n'
The output of running that same sequence in ipython is the same.
So the subprocess module is behaving in the manner you expect, and not how it's behaving for you in your question. I think something other than the subprocess module must be at fault here because what you're doing works for me.
I'm running Python 2.7, so another possibility is that maybe there is some kind of weird bug in older versions of the subprocess module.