As an add-on to my old question: I have a question. When I run cmd.exe and execute the target program, the output is printed to cmd nicely and it exits with code 0 in the end,
Since the program continuously prints to my cmd.exe stdout, how come I can't mimic that behaviour in Python?
The following code is how I parse lines from my target executable.
res = subprocess.Popen(command, universal_newlines=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1)
with res.stdout:
for line in iter(res.stdout.readline, b''):
print line
res.wait()
The python parsing doesnt even read the same things as cmd.exe does!
It doesnt print the last 5-10 lines (the ones telling me the process is complete).
Do I have to subprocess popen cmd.exe then call the target program? Are there any other alternatives?
Related
My problem is very similar to the problems described here and here with one thing not being covered: Assume, that I have an external longrunning.exe on windows, and I'm calling that with subprocess.Popen(). My exe now prints some commands and after some time, it gets into a loopback-mode and waits for input. When that happens, it outputs a single dot every second in the windows command prompt. It does put the subsequent dots on the same line, whereas all output before that is on its own line. I seem to be able to catch all output before that, but I cannot get this output, probably to some buffering going on(?). How can I get this output in my python console? Relevant code as below:
import subprocess
import sys
cmd = 'long_running.exe'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
sys.stdout.write(line)
I'm new to python, and trying to run a exe software from python in windows.
I wrote the following code:
from subprocess import STDOUT, Popen, PIPE
cmd=r'C:\Users\lenaq\Desktop\sep\WATv16\TLWMA-0.09.exe'
with open('test.log', 'w') as f:
p = subprocess.Popen(cmd, shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
for c in iter(lambda: p.stdout.read(1), ''):
sys.stdout.write(c)
f.write(c)
The exe program have some running errors, and I need to get the output of the program in order to fix the params file in order to prevent the errors.
the problem is that by using the above code I don't get the full output of the exe (when comparing to the os.system() command). the error message window of the exe pops out before the completion of the output writing, and I don't know where is the problem.
can you please help me...
stderr=PIPE redirects the error stream to p.stderr, and you're not reading that (note that using p.communicate allows to get both stream results, but reading them separately can lead to deadlocks).
Anyway, if you don't care about merging both out & err streams, you could change that to:
stderr=STDOUT
so both out & err use the same stream p.stdout
Also: don't use shell=True, you don't need it.
If that doesn't fix it in your case, it means that the underlying program crashed while not flushing its output. Output flush works differently when output is not redirected, which may explain why you get more output when running it without redirection with os.system (more about this issue: forcing a program to flush its standard output when redirected)
One lead yet to be explored would be to use winpty which is an equivalent of unbuffer on Windows: What is the equivalent of unbuffer program on Windows?. Something like:
cmd = ["winpty.exe","-Xallow-non-tty","-Xplain","TLWMA-0.09.exe"]
p = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
I was trying to run and process the stdout of some java program and found that my Python script was eternally waiting. Then I've wrote a new test script to test subprocess and found that, again, I see no output when running this:
$ cat test.py
#!/usr/bin/env python
import subprocess
c = ['/usr/bin/tail', '-f', '/var/log/dmesg']
proc = subprocess.Popen(c,
bufsize=1,
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
for line in proc.stdout:
print line
Why is subprocess ignoring my bufsize argument? Is there some intermediate buffering I'm missing to take into account? I expect to read the first 10 lines of tail and then eternally wait until new lines are appened to the dmesg file. My user do have permissions, running the command on bash gives output.
Changing tail to yes seems to fill some buffers and I can see lots of output.
You can use iter(proc.stdout.readline,''):
for line in iter(proc.stdout.readline,''):
print line
for line in proc.stdout reads all the input before iterating over the content.
My problem is this--I need to get output from a subprocess and I am using the following code to call it-- (Feel free to ignore the long arguments. The importing thing is the stdout= subprocess.PIPE)
(stdout, stderr) = subprocess.Popen([self.ChapterToolPath, "-x", book.xmlPath , "-a", book.aacPath , "-o", book.outputPath+ "/" + fileName + ".m4b"], stdout= subprocess.PIPE).communicate()
print stdout
Thanks to an answer below, I've been able to get the output of the program, but I still end up waiting for the process to terminate before I get anything. The interesting thing is that in my debugger, there is all sorts of text flying by in the console and it is all ignored. But the moment that anything is written to the console in black (I am using pycharm) the program continues without a problem. Could the main program be waiting for some kind of output in order to move on? This would make sense because I am trying to communicate with it.... Is there a difference between text that I can see in the console and actual text that makes it to the stdout? And how would I collect the text written to the console?
Thanks!
The first line of the documentation for subprocess.call() describes it as such:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
Thus, it necessarily waits for the subprocess to exit.
subprocess.Popen(), by contrast, does not do this, returning a handle on a process with which one than then communicate().
To get all output from a program:
from subprocess import check_output as qx
output = qx([program, arg1, arg2, ...])
To get output while the program is running:
from subprocess import Popen, PIPE
p = Popen([program, arg1, ...], stdout=PIPE)
for line in iter(p.stdout.readline, ''):
print line,
There might be a buffering issue on the program' side if it prints line-by-line when run interactively but buffers its output if run as a subprocess. There are various solutions depending on your OS or the program e.g., you could run it using pexpect module.
I trying to start a program (HandBreakCLI) as a subprocess or thread from within python 2.7. I have gotten as far as starting it, but I can't figure out how to monitor it's stderr and stdout.
The program outputs it's status (% done) and info about the encode to stderr and stdout, respectively. I'd like to be able to periodically retrieve the % done from the appropriate stream.
I've tried calling subprocess.Popen with stderr and stdout set to PIPE and using the subprocess.communicate, but it sits and waits till the process is killed or complete then retrieves the output then. Doesn't do me much good.
I've got it up and running as a thread, but as far as I can tell I still have to eventually call subprocess.Popen to execute the program and run into the same wall.
Am I going about this the right way? What other options do I have or how to I get this to work as described?
I have accomplished the same with ffmpeg. This is a stripped down version of the relevant portions. bufsize=1 means line buffering and may not be needed.
def Run(command):
proc = subprocess.Popen(command, bufsize=1,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
return proc
def Trace(proc):
while proc.poll() is None:
line = proc.stdout.readline()
if line:
# Process output here
print 'Read line', line
proc = Run([ handbrakePath ] + allOptions)
Trace(proc)
Edit 1: I noticed that the subprocess (handbrake in this case) needs to flush after lines to use this (ffmpeg does).
Edit 2: Some quick tests reveal that bufsize=1 may not be actually needed.