We Keep Coding

python subprocess module hangs for spark-submit command when writing STDOUT

I have a python script that is used to submit spark jobs using the spark-submit tool. I want to execute the command and write the output both to STDOUT and a logfile in real time. i'm using python 2.7 on a ubuntu server.
This is what I have so far in my SubmitJob.py script
#!/usr/bin/python
# Submit the command
def submitJob(cmd, log_file):
with open(log_file, 'w') as fh:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
fh.write(output)
rc = process.poll()
return rc
if __name__ == "__main__":
cmdList = ["dse", "spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
log_file = "/tmp/out.log"
exist_status = submitJob(cmdList, log_file)
print "job finished with status ",exist_status
The strange thing is, when I execute the same command direcly in the shell it works fine and produces output on screen as the proggram proceeds.
So it looks like something is wrong in the way I'm using the subprocess.PIPE for stdout and writing the file.
What's the current recommended way to use subprocess module for writing to stdout and log file in real time line by line? I see bunch of options on the internet but not sure which is correct or latest.
thanks

Figured out what the problem was.
I was trying to redirect both stdout n stderr to pipe to display on screen. This seems to block the stdout when stderr is present. If I remove the stderr=stdout argument from Popen, it works fine. So for spark-submit it looks like you don't need to redirect stderr explicitly as it already does this implicitly

To print the Spark log
One can call the commandList given by user330612
cmdList = ["spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
Then it can be printed by using subprocess, remember to use communicate() to prevent deadlocks https://docs.python.org/2/library/subprocess.html
Warning Deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that. Here below is the code to print the log.
import subprocess
p = subprocess.Popen(cmdList,stdout=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
stderr=stderr.splitlines()
stdout=stdout.splitlines()
for line in stderr:
print line #now it can be printed line by line to a file or something else, for the log
for line in stdout:
print line #for the output
More information about subprocess and printing lines can be found at:
https://pymotw.com/2/subprocess/

How to redirect command output using os.execvp() in python

I am invoking shell script using os.execvp() in python. my shell script has some echo statements whcih I want to redirect in file.
Here is what I am trying:
cmd = "/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
cmdline = ["/bin/sh", cmd]
os.execvp(cmdline[0], cmdline)
Below is the error I am getting:
Error: /bin/sh: /opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log: No such file or directory
Can any one help?

This is happening because you are passing this entire string as if it were the program name to execute:
"/opt/rpm/rpm_upgrade.sh >& /opt/rpm/upgrader.log"
The easy way to fix this is:
cmdline = ["/bin/sh", "/opt/rpm/rpm_upgrade.sh",
">&", "/opt/rpm/upgrader.log"]
os.execvp(cmdline[0], cmdline)
Now sh will receive three arguments rather than one.
Or you can switch to the more full-featured subprocess module, which lets you redirect output in Python:
import subprocess
with open("/opt/rpm/upgrader.log", "wb") as outfile:
subprocess.check_call(["/opt/rpm/rpm_upgrade.sh"], shell=True,
stdout=outfile, stderr=subprocess.STDOUT)

Python and Shell commands issue

currently I'm developing a small project in which i'd like to execute some shell commands from a local webserver using Apache and Python.
I'd like to execute this command (shows the ip's of the devices connected to the server) and assing its output to a variable in Python:
grep -o '^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' access.log
When i type this in the terminal, it shows the Ip's correctly. But i want to execute this command inside a python script which looks like this:
import os
print "Content-type:text/html\r\n\r\n"
print '<html>'
print '<head>'
print '<title>Connected devices</title>'
print '</head>'
print '<body>'
print '<h2>Conected Devices</h2>'
f = os.popen("grep -o '^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' access.log")
now = f.read()
print now
print '</body>'
print '</html>'
When i call the python script from my server i can only see the title (Connected devices). Do you know which could be my problem?
Thanks in advance

First of all, check in which directory the script is being run.
import os
os.getcwd()
If its running on correct directory, try with below :
from subprocess import Popen, PIPE, STDOUT
cmd = "grep -o '^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' access.log"
p = Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)
output = p.stdout.read()
print output

Catching runtime error for process created by python subprocess

I am writing a script which can take a file name as input, compile it and run it.
I am taking the name of a file as input(input_file_name). I first compile the file from within python:
self.process = subprocess.Popen(['gcc', input_file_name, '-o', 'auto_gen'], stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
Next, I'm executing the executable using the same(Popen) call:
subprocess.Popen('./auto_gen', stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT, shell=False)
In both cases, I'm catching the stdout(and stderr) contents using
(output, _) = self.process.communicate()
Now, if there is an error during compilation, I am able to catch the error because the returncode is 1 and I can get the details of the error because gcc sends them on stderr.
However, the program itself can return a random value even on executing successfully(because there might not be a "return 0" at the end). So I can't catch runtime errors using the returncode. Moreover, the executable does not send the error details on stderr. So I can't use the trick I used for catching compile-time errors.
What is the best way to catch a runtime error OR to print the details of the error? That is, if ./auto_gen throws a segmentation fault, I should be able to print either one of:
'Runtime error'
'Segmentation Fault'
'Program threw a SIGSEGV'

Try this. The code runs a subprocess which fails and prints to stderr. The except block captures the specific error exit code and stdout/stderr, and displays it.
#!/usr/bin/env python
import subprocess
try:
out = subprocess.check_output(
"ls non_existent_file",
stderr=subprocess.STDOUT,
shell=True)
print 'okay:',out
except subprocess.CalledProcessError as exc:
print 'error: code={}, out="{}"'.format(
exc.returncode, exc.output,
)
Example output:
$ python ./subproc.py
error: code=2, out="ls: cannot access non_existent_file: No such file or directory
"

If ./autogen is killed by a signal then self.process.returncode (after .wait() or .communicate()) is less than zero and its absolute value reports the signal e.g., returncode == -11 for SIGSERV.

please check following link for runtime errors or output of subprocess
https://www.endpoint.com/blog/2015/01/28/getting-realtime-output-using-python
def run_command(command):
process = subprocess.Popen(shlex.split(command),
stdout=subprocess.PIPE)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
rc = process.poll()
return rc

Subprocess.Popen behaves differently in interpreter, executable scripts

Let's say you have the following:
command = shlex.split("mcf -o -q -e -w %s %s" % (SOLFILE, NETFILE))
task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print "stdout: %s" % stdout #debugging
print "stderr: %s" % stderr #debugging
if stderr:
sys.exit("MCF crashed on %s" % NETFILE)
It's not necessary to know what mcf is, except that it's a C program which will overflow if it's not given a satisfiable netfile. (Why can't I just ensure that all the netfiles are satisfiable? Well, because the easiest way to check that is to feed it to mcf and see if it overflows...)
Anyway, when I run this in an executable script, task.communicate() doesn't seem to store anything in stdout and stderr. (To be precise, I get stdout == stderr == ''.) Instead, the stderr stream from mcf seems to be "leaking" to the terminal rather than getting captured by the subprocess pipe. Here's some sample output to illustrate:
Netfile: facility3cat_nat5000_wholesaler_capacitation_test_.net
Solfile: facility3cat_nat5000_wholesaler_capacitation_test_.sol
*** buffer overflow detected ***: mcf terminated
======= Backtrace: =========
...
...[fifty lines of Linda Blair-esque output]...
...
stdout: None
stderr:
...[program continues, since stderr did not evaluate to True]...
This only fails when running the script from the command line. When I step through it line by line in the interpreter, stdout and stderr are correctly assigned:
>>> task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = task.communicate()
>>> stderr
'*** buffer overflow detected ***: mcf terminated\n======= Backtrace: =========\n'
...[more headspinning and vomit]...
Could anyone help me to understand why this works in the interpreter, but not when executed? Thanks in advance!

I wrote a little test script to test the subprocess module with.
#!/bin/bash
echo echo to stderr 1>&2
echo echo to stdout
Then I wrote a small Python script that calls it:
#!/usr/bin/python
import subprocess
command = ('./joe.sh',)
task = subprocess.Popen(command, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print 'stdout == %r\nstderr == %r' % (stdout, stderr)
The output of running it looks just like this:
$ python joe.py
stdout == 'echo to stdout\n'
stderr == 'echo to stderr\n'
The output of running that same sequence in ipython is the same.
So the subprocess module is behaving in the manner you expect, and not how it's behaving for you in your question. I think something other than the subprocess module must be at fault here because what you're doing works for me.
I'm running Python 2.7, so another possibility is that maybe there is some kind of weird bug in older versions of the subprocess module.