We Keep Coding

python Popen print the output of the external application and save it (log it) at the same time

I am trying to run an external application using Popen and print the output in the console or separated console (better) and at the same time save the output to the file. There is no user interaction via console, app.bat just sends (writes) the data and should terminate automatically when the execution is finished.
Running the following command will result only in printing the results in the python console.
p = subprocess.Popen("app.bat --x --y", shell=False)
If I add stdout as file I can redirect the output to the file, but nothing is written in the console, which does not give users any feedback (and the feedback needs to be in real-time, not after the execution because app runs approximately 1-3min).
file_ = open("ouput.txt", "w+")
p = subprocess.Popen("app.bat --x --y", shell=False,stdout=file_)
Therefore, my question is how to run the external app and at the same time write in the console and in the file?

For what you want to do I'd encourage you to use the logging module.
A good starter here is https://docs.python.org/2/howto/logging-cookbook.html
It even describes your usecase almost exactly.

If you want to post-process the output of your Popen() call, you should typically redirect stdout to PIPE and then read the output from there. This will allow you to e.g. both write to file and to screen:
import subprocess
logfile ='output.txt'
command = ['app.bat', '--x', '--y']
p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
with open(logfile, 'w+') as f:
for line in p.stdout:
print(line.rstrip())
f.write(line)
Now, this will block until app.bat finishes, which may be exactly what you want. But, if you want your Python script to continue to run, and have app.bat run in te background, you can start a thread that will handle your subprocess stdout:
import subprocess
import threading
logfile ='output.txt'
command = ['app.bat', '--x', '--y']
def writer(p, logfile):
with open(logfile, 'w+') as f:
for line in p.stdout:
print(line.rstrip())
f.write(line)
p = subprocess.Popen(command, stdout=subprocess.PIPE, universal_newlines=True)
t = threading.Thread(target=writer, args=(p,logfile))
t.start()
# Other commands while app.bat runs
t.join()

python subprocess module hangs for spark-submit command when writing STDOUT

I have a python script that is used to submit spark jobs using the spark-submit tool. I want to execute the command and write the output both to STDOUT and a logfile in real time. i'm using python 2.7 on a ubuntu server.
This is what I have so far in my SubmitJob.py script
#!/usr/bin/python
# Submit the command
def submitJob(cmd, log_file):
with open(log_file, 'w') as fh:
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = process.stdout.readline()
if output == '' and process.poll() is not None:
break
if output:
print output.strip()
fh.write(output)
rc = process.poll()
return rc
if __name__ == "__main__":
cmdList = ["dse", "spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
log_file = "/tmp/out.log"
exist_status = submitJob(cmdList, log_file)
print "job finished with status ",exist_status
The strange thing is, when I execute the same command direcly in the shell it works fine and produces output on screen as the proggram proceeds.
So it looks like something is wrong in the way I'm using the subprocess.PIPE for stdout and writing the file.
What's the current recommended way to use subprocess module for writing to stdout and log file in real time line by line? I see bunch of options on the internet but not sure which is correct or latest.
thanks

Figured out what the problem was.
I was trying to redirect both stdout n stderr to pipe to display on screen. This seems to block the stdout when stderr is present. If I remove the stderr=stdout argument from Popen, it works fine. So for spark-submit it looks like you don't need to redirect stderr explicitly as it already does this implicitly

To print the Spark log
One can call the commandList given by user330612
cmdList = ["spark-submit", "--spark-master", "spark://127.0.0.1:7077", "--class", "com.spark.myapp", "./myapp.jar"]
Then it can be printed by using subprocess, remember to use communicate() to prevent deadlocks https://docs.python.org/2/library/subprocess.html
Warning Deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that. Here below is the code to print the log.
import subprocess
p = subprocess.Popen(cmdList,stdout=subprocess.PIPE,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
stdout, stderr = p.communicate()
stderr=stderr.splitlines()
stdout=stdout.splitlines()
for line in stderr:
print line #now it can be printed line by line to a file or something else, for the log
for line in stdout:
print line #for the output
More information about subprocess and printing lines can be found at:
https://pymotw.com/2/subprocess/

Python: no output from (Perl) subprocess

So I want to get a Perl script to activate from Python and after some time I have gotten to a stage where it executes it, but I get no output, and I don't know what is wrong.
I don't even know if it recognises the script or the input files, because it accepts everything and gives no error message.
script = subprocess.Popen(["perl" , "C:\\Users\\...\\pal2nal.pl" , "C:\\Users\\...\\input_file" , "C:\\Users\\...\\output_file" ], stdout=subprocess.PIPE)
while True:
line = script.stdout.readline()
if line == b'' and script.poll() != None:
break
sys.stdout.write(line.decode('utf-8'))
sys.stdout.flush()
output = script.communicate()[0]
exitCode = script.returncode
This is the script if anyone is interested http://www.bork.embl.de/pal2nal/distribution/pal2nal.v14.tar.gz
This is my first time working with subprocesses and have tried with error checking but was ultimately unsuccessful.

To display the subprocess output line-by-line as soon as it arrives and to capture it in a variable:
#!/usr/bin/env python3
import io
from subprocess import Popen, PIPE
lines = []
with Popen(["perl" , r"C:\Users\...\pal2nal.pl",
r"C:\Users\...\input_file",
r"C:\Users\...\output_file"],
stdout=PIPE, bufsize=1) as p:
for line in io.TextIOWrapper(p.stdout, encoding='utf-8'):
print(line, end='')
lines.append(line)
output = ''.join(lines)
print(len(lines), p.returncode)

Linux, Write output to the file and terminate it using Python

There are various topics available on this very topic, "How to write output to the text file".
But my issue is different because the output to the command in question is continous.
What I want to do is, write the Output of the command cgps -s to the file aaa.txt
here is the code,
import signal
import os
import subprocess
p = subprocess.Popen(["cgps", "-s", ">> aaa.txt"], stdout=subprocess.PIPE,shell=True, preexec_fn=os.setsid)
os.killpg(p.pid, signal.SIGTERM)
The code doesn't work at all and no file is created with the name aaa.txt
When I execute this command through terminal,
cgps -s > aaa.txt
Then I have to press CTRL+C to terminate the output from being written on the output file because the output is continuous.
Is there any way to just capture one output and write it to the file and terminate it using Python or using Command line ?

So you are trying to create a pipe. Try this:
import subprocess
p = subprocess.Popen(["cgps", "-s"], stdout=subprocess.PIPE)
with open("aaa.txt", "w") as f:
while True:
line = p.stdout.readline()
if not line:
break
if some_condition(): # <-- check whether you have enough output
p.terminate()
break
f.writeline(line)
p.wait()

Python: How to read stdout non blocking from another process?

During the runtime of a process I would like to read its stdout and write it to a file. Any attempt of mine however failed because no matter what I tried as soon as I tried reading from the stdout it blocked until the process finished.
Here is a snippet of what I am trying to do. (The first part is simply a python script that writes something to stdout.)
import subprocess
p = subprocess.Popen('python -c \'\
from time import sleep\n\
for i in range(3):\n\
sleep(1)\n\
print "Hello", i\
\'', shell = True, stdout = subprocess.PIPE)
while p.poll() == None:
#read the stdout continuously
pass
print "Done"
I know that there are multiple questions out there that deal with the same subject. However, none of the ones I found was able to answer my question.

What is happening is buffering on the writer side. Since you are writing such small chunks from the little code snippet the underlying FILE object is buffering the output until the end. The following works as you expect.
#!/usr/bin/python
import sys
import subprocess
p = subprocess.Popen("""python -c '
from time import sleep ; import sys
for i in range(3):
sleep(1)
print "Hello", i
sys.stdout.flush()
'""", shell = True, stdout = subprocess.PIPE)
while True:
inline = p.stdout.readline()
if not inline:
break
sys.stdout.write(inline)
sys.stdout.flush()
print "Done"
However, you may not be expecting the right thing. The buffering is there to reduce the number of system calls in order to make the system more efficient. Does it really matter to you that the whole text is buffered until the end before you write it to a file? Don't you still get all the output in the file?

the following code would print stdout line by line as the subprocess runs until the readline() method returns an empty string:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
print line
p.stdout.close()
print 'Done'
update relating to your question better:
import subprocess
p = subprocess.Popen(['python'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
p.stdin.write("""
from time import sleep ; import sys
for i in range(3):
sleep(1)
print "Hello", i
sys.stdout.flush()
""")
p.stdin.close()
for line in iter(p.stdout.readline, ''):
print line
p.stdout.close()
print 'Done'

You can use subprocess.communicate() to get the output from stdout. Something like:
while(p.poll() == None):
#read the stdout continuously
print(p.communicate()[0])
pass
More info available at: http://docs.python.org/library/subprocess.html