We Keep Coding

python subprocess module hangs for spark-submit command when writing STDOUT

I have a python script that is used to submit spark jobs using the spark-submit tool. I want to execute the command and write the output both to STDOUT and a logfile in real time. i'm using python 2.7 on a ubuntu server.
This is what I have so far in my SubmitJob.py script
#!/usr/bin/python
# Submit the command
def submitJob(cmd, log_file):
with open(log_file, 'w') as fh:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
fh.write(output)
rc = process.poll()
return rc
if __name__ == "__main__":
cmdList = ["dse", "spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
log_file = "/tmp/out.log"
exist_status = submitJob(cmdList, log_file)
print "job finished with status ",exist_status
The strange thing is, when I execute the same command direcly in the shell it works fine and produces output on screen as the proggram proceeds.
So it looks like something is wrong in the way I'm using the subprocess.PIPE for stdout and writing the file.
What's the current recommended way to use subprocess module for writing to stdout and log file in real time line by line? I see bunch of options on the internet but not sure which is correct or latest.
thanks

Figured out what the problem was.
I was trying to redirect both stdout n stderr to pipe to display on screen. This seems to block the stdout when stderr is present. If I remove the stderr=stdout argument from Popen, it works fine. So for spark-submit it looks like you don't need to redirect stderr explicitly as it already does this implicitly

To print the Spark log
One can call the commandList given by user330612
cmdList = ["spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
Then it can be printed by using subprocess, remember to use communicate() to prevent deadlocks https://docs.python.org/2/library/subprocess.html
Warning Deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that. Here below is the code to print the log.
import subprocess
p = subprocess.Popen(cmdList,stdout=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
stderr=stderr.splitlines()
stdout=stdout.splitlines()
for line in stderr:
print line #now it can be printed line by line to a file or something else, for the log
for line in stdout:
print line #for the output
More information about subprocess and printing lines can be found at:
https://pymotw.com/2/subprocess/

Python: pass to subprocess a temporary generated file not as the last argument

how to use subprocess if my temp-file argument is in the middle of the command? For example a terminal command looks like this:
program subprogram -a -b tmpFILE otherFILE
I tried variations of this:
from subprocess import Popen, PIPE
from tempfile import SpooledTemporaryFile as tempfile
tmpFILE=tempfile()
tmpFILE.write(someList)
tmpFILE.seek(0)
print Popen(['program','subprogram', '-a', '-b', otherFile],stdout=PIPE,stdin=tmpFILE).stdout.read()
f.close()
or
print Popen(['program','subprogram', '-a', '-b', tmpFILE, otherFile],stdout=PIPE,stdin=tmpFILE).stdout.read()
but nothing works... My temporary generated file in python shouldn't be as the last parameter.
Thanks

Is there a reason to use SpooledTemporaryFile instead of other types of temp file? If not, I recommend using NamedTemporaryFile as you can retrieve the name from it. I have tried to retrieve the name from SpooledTemporaryFile and got '<fdopen>' which does not seem to be valid.
Here is the suggested code:
from subprocess import Popen, PIPE
import tempfile
with tempfile.NamedTemporaryFile() as temp_file:
temp_file.write(someList)
temp_file.flush()
process = Popen(['program', 'subprogram', '-a', '-b', temp_file.name, otherFile], stdout=PIPE, stderr=PIPE)
stdout, stderr = process.communicate()
Discussion
Using the with statement, you don't have to worry about closing the file. As soon as the with block is finished, the file is automatically closed.
Instead of calling seek, you should call flush to commit your file buffer to disk before calling program.

python subprocess issue with Nmap

Im trying to get a linux binary to send its standard output to a variable by using subprocess. But just keep getting tracebacks.
>>> import subprocess
>>>nmap -sn -Pn todd.ns.cloudflare.com --script dns-check-zone --script-args='dns-check-zone.domain=www.macmonster.com
Any Ideas (oh and Im using Python2.7).
Ideally I would like to avoid using Shell=true to avoid any security concerns.
Thanks,

shlex to the rescue!
The module shlex will take a string containing the whole shell command and split it up exactly how Popen and check_output expect it. Like this:
import shlex, subprocess
cmd = "/usr/bin/nmap -sn -Pn todd.ns.cloudflare.com --script dns-check-zone --script-args='dns-check-zone.domain=www.macmonster.com'"
args = shlex.split(cmd)
output = subprocess.check_output(args)
When you look at contents of args you'll see:
>>> print args
['/usr/bin/nmap', '-sn', '-Pn', 'todd.ns.cloudflare.com', '--script', 'dns-check-zone', '--script-args=dns-check-zone.domain=www.macmonster.com']
Note that shlex split up the option "--script dns-check-zone" into two tokens. On the other hand it kept "--script-args='dns-check-zone.domain=www.macmonster.com'", but removed the single-quotes.

import subprocess
output = subprocess.check_output(["/usr/bin/nmap", "-sP", "-n", "172.16.1.0/24"])
output = subprocess.check_output(["/usr/bin/nmap", "-sP", "-n", "172.16.1.0/24"], stderr=subprocess.STDOUT)

Have you tried this:
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
var = p.stdout.read()
print var
I would have used communicate, but it returns an odd list-type thing.

Python OS.popen error

I'm trying to get a few filenames by executing this command in os.popen :
ls /etc/tor/statistiekjes/ |egrep dns
But when I run my script I get :
<open file 'ls /etc/tor/statistiekjes/ |egrep dns', mode 'r' at 0xb7786860>
egrep: write error: Broken pipe
Code :
lscmd = "ls /etc/tor/statistiekjes/ |egrep "+FILE
print lscmd
inputList=os.popen(lscmd,'r')
File is an argument past to the script to grep on

For this particular problem, you could use native Python calls:
import os
import re
for name in (name for name in os.listdir('/etc/tor/statistiekjes/')
if re.search(FILE,name)):
print(repr(name))
However, you are probably looking for a more general solution to calling external programs. In that case, use subprocess instead of os.popen, since os.popen is deprecated:
import subprocess
import shlex
proc1 = subprocess.Popen(shlex.split('ls /etc/tor/statistiekjes/'),
stdout=subprocess.PIPE)
proc2 = subprocess.Popen(shlex.split('egrep {pat}'.format(pat=FILE)),
stdin=proc1.stdout,
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out,err=proc2.communicate()
print(out)
See "Replacing shell pipeline".
PS. subprocess.Popen has a shell=True parameter which could also be used. However, it is best to avoid using shell=True if possible. It is a security risk.

You can use subprocess.Popen, with shell=True flags:
from subprocess import Popen, PIPE
lscmd = "ls /etc/tor/statistiekjes/ |egrep "+FILE
inputList = Popen(lscmd, shell=True, stdout=PIPE).communicate()[0]
print inputList
Enjoy.

Retrieving the output of subprocess.call() [duplicate]

This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 4 years ago.
How can I get the output of a process run using subprocess.call()?
Passing a StringIO.StringIO object to stdout gives this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 588, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 945, in _get_handles
c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'
>>>

If you have Python version >= 2.7, you can use subprocess.check_output which basically does exactly what you want (it returns standard output as string).
Simple example (linux version, see note):
import subprocess
print subprocess.check_output(["ping", "-c", "1", "8.8.8.8"])
Note that the ping command is using linux notation (-c for count). If you try this on Windows remember to change it to -n for same result.
As commented below you can find a more detailed explanation in this other answer.

Output from subprocess.call() should only be redirected to files.
You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:
from subprocess import Popen, PIPE
p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.
See here for more info.

For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code.
from subprocess import PIPE, run
command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)

I have the following solution. It captures the exit code, the stdout, and the stderr too of the executed external command:
import shlex
from subprocess import Popen, PIPE
def get_exitcode_stdout_stderr(cmd):
"""
Execute the external command and get its exitcode, stdout and stderr.
"""
args = shlex.split(cmd)
proc = Popen(args, stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
exitcode = proc.returncode
#
return exitcode, out, err
cmd = "..." # arbitrary external command, e.g. "python mytest.py"
exitcode, out, err = get_exitcode_stdout_stderr(cmd)
I also have a blog post on it here.
Edit: the solution was updated to a newer one that doesn't need to write to temp. files.

I recently just figured out how to do this, and here's some example code from a current project of mine:
#Getting the random picture.
#First find all pictures:
import shlex, subprocess
cmd = 'find ../Pictures/ -regex ".*\(JPG\|NEF\|jpg\)" '
#cmd = raw_input("shell:")
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
#Another way to get output
#output = subprocess.Popen(args,stdout = subprocess.PIPE).stdout
ber = raw_input("search complete, display results?")
print output
#... and on to the selection process ...
You now have the output of the command stored in the variable "output". "stdout = subprocess.PIPE" tells the class to create a file object named 'stdout' from within Popen. The communicate() method, from what I can tell, just acts as a convenient way to return a tuple of the output and the errors from the process you've run. Also, the process is run when instantiating Popen.

The key is to use the function subprocess.check_output
For example, the following function captures stdout and stderr of the process and returns that as well as whether or not the call succeeded. It is Python 2 and 3 compatible:
from subprocess import check_output, CalledProcessError, STDOUT
def system_call(command):
"""
params:
command: list of strings, ex. `["ls", "-l"]`
returns: output, success
"""
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call(["ls", "-l"])
If you want to pass commands as strings rather than arrays, use this version:
from subprocess import check_output, CalledProcessError, STDOUT
import shlex
def system_call(command):
"""
params:
command: string, ex. `"ls -l"`
returns: output, success
"""
command = shlex.split(command)
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call("ls -l")

In Ipython shell:
In [8]: import subprocess
In [9]: s=subprocess.check_output(["echo", "Hello World!"])
In [10]: s
Out[10]: 'Hello World!\n'
Based on sargue's answer. Credit to sargue.