We Keep Coding

How to run python subprocess with real-time output?

When I run a shell command in Python it does not show the output until the command is finished. the script that I run takes a few hours to finish and I'd like to see the progress while it is running.
How can I have python run it and show the outputs in real-time?

Use the following function to run your code. In this example, I want to run an R script with two arguments. You can replace cmd with any other shell command.
from subprocess import Popen, PIPE
def run(command):
process = Popen(command, stdout=PIPE, shell=True)
while True:
line = process.stdout.readline().rstrip()
if not line:
break
print(line)
cmd = f"Rscript {script_path} {arg1} {arg2}"
run(cmd)

Python Popen - how to execute commands in nested sub shell using python

"I have an issue executing commands in nested adb sub shell in python. executing "command_two" in adb shell opens a sub console in command line (and the console waits for input). how do i execute commands (give input to the console) in that console using the python.
Code:
R = subprocess.Popen('adb shell', stdin=subprocess.PIPE)
R.communicate('command_one\ncommand_two\n)

Please try this:
import time
R = subprocess.Popen('adb shell', shell=True, stdin=subprocess.PIPE)
R.communicate('command_one\n')
time.sleep(2)
R.communicate('command_two\n')

Python : Redirecting subprocess Popen stdout to log file

I have a python process running, having a logger object configured to print logs in a log file.
Now, I am trying to call a scala script through this python process, by using subprocess module of Python.
subprocess.Popen(scala_run_command, stdout=subprocess.PIPE, shell=True)
The issue is, whenever the python process exits, it hangs the shell, which comes to life only after explicitly running stty sane command. My guess is that it is caused because the scala script outputs to shell and hence the shell hangs, because of its stdout [something in its stdout causes the shell to lose its sanity].
For the same reason, I wanted to try to put the output of scala run script to be captured in my default log file, which does not seem to be happening using multiple ways.
So, the query boils down to, how to the get the stdout output of shell command ran through subprocess module in a log file. Even if there is a better way to achieve this instead of subprocess, run, I would love to know the ideas.
The current state of code looks like this.
__echo_command = 'echo ":load %s"'
__spark_console_command = 'spark;'
def run_scala_script(self, script):
echo_command = self.__echo_command % script
spark_console_command = self.__spark_console_command
echo_result = subprocess.run(echo_command, stdout=subprocess.PIPE, shell=True)
result = subprocess.run(spark_console_command, stdout=subprocess.PIPE, shell=True, input=echo_result.stdout)
logger.info('Scala script %s completed successfully' % script)
logger.info(result.stdout)

Use
p = subprocess.Popen(...)
followed by
stdout, stderr = p.communicate()
and then stdout and stderr will contain the output bytes from the subprocess' output streams. You can then log the stdout value.

Capturing console output in Python

I can capture the output of a command line execution, for example
import subprocess
cmd = ['ipconfig']
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
output = proc.communicate()[0]
print output
In the above example it is easy as ipconfig, when run directly on the command line, will print the output on the same console window. However, there can be situations where the command opens a new console window to display all the output. An example of this is running VLC.exe from the command line. The following VLC command will open a new console window to display messages:
vlc.exe -I rc XYZ.avi
I want to know how can I capture the output displayed on this second console window in Python. The above example for ipconfig does not work in this case.
Regards
SS

I think this is really matter of VLC interface. There may be some option to vlc.exe so it doesn't start the console in new window or provides the output otherwise. In general there is nothing that stops me to run a process which runs another process and does not provide its output to caller of my process.

Note that, in your second command, you have multiple arguments (-I, rc, and the file name - XYZ.avi), in such cases, your cmd variable should be a list of - command you want to run , followed by all the arguments for that command:
Try this:
import subprocess
cmd=['vlc.exe', '-I', 'rc', 'XYZ.avi']
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE)
output = proc.communicate()[0]
print output
If you are not running the script from right directory, you might want to provide absolute paths for both vlc.exe and your XYZ.avi in cmd var

Use bat file in CGI Python on localhost

I'm quite stuck with with a complex mix of files and languages! The problem:
My webform starts a python script, as a cgi script, on localhost(apache). In this python script I want to execute a batchfile. This batchfile executes several commands, which i tested thoroughly.
If i execute the following python file in the python interpreter or in CMD it does execute the bat file.
But when I 'start' the python script from the webform it says it did it, but there are no results, so i guess something is wrong with the cgi part of the problem?!
The process is complicated, so if someone has a better way of doing this...pls reply;). I'm using windows so that makes things even more annoying sometimes.
I think it's not the script, because I try subprocess.call, os.startfile and os.system already!
It either does nothing or the webpage keeps loading(endless loop)
Python script:
import os
from subprocess import Popen, PIPE
import subprocess
print "Content-type:text/html\r\n\r\n"
p = subprocess.Popen(["test.bat"], stdout = subprocess.PIPE, stderr = subprocess.PIPE)
out, error = p.communicate()
print out
print "DONE!"
The bat file:
#echo off
::Preprocess the datasets
CMD /C java weka.filters.unsupervised.attribute.StringToWordVector -b -i data_new.arff -o data_new_std.arff -r tweetin.arff -s tweetin_std.arff
:: Make predictions with incoming tweets
CMD /C java weka.classifiers.functions.SMO -T tweetin_std.arff -t data_new_std.arff -p 2 -c first > result.txt
Thanks for your reply!!

Your bat file is redirecting the second program's output to a file, so p.communicate can only get the output of the first program. I'm assuming you want to return the content of result.txt?
I think you should skip the bat file and just do both java invocations in python. You get more control of the execution and you can check the return codes, there might be problems with java not being in the PATH environment variable when run as CGI. The following is mostly equivalent with respect to you getting the program's output back, you want to capture the second program's output if your webservice is supposed to return the predictions.
import os
import shlex
from subprocess import Popen, PIPE
import subprocess
print "Content-type:text/html\r\n\r\n"
p = subprocess.Popen(shlex.split("java weka.filters.unsupervised.attribute.StringToWordVector -b -i data_new.arff -o data_new_std.arff -r tweetin.arff -s tweetin_std.arff"),
stdout = subprocess.PIPE, stderr = subprocess.PIPE)
out, error = p.communicate()
return_code = subprocess.call(shlex.split("java weka.classifiers.functions.SMO -T tweetin_std.arff -t data_new_std.arff -p 2 -c first > result.txt"))
print out
print "DONE!"

A couple of things come to mind. You might want to try setting your Popen's shell=True. Sometimes I have noticed that's solved my problem.
p = subprocess.Popen(["test.bat"], stdout = subprocess.PIPE, stderr = subprocess.PIPE, shell=True)
You may also want to take a look at Fabric, which is perfect for this kind of automation.