My problem is very similar to the problems described here and here with one thing not being covered: Assume, that I have an external longrunning.exe on windows, and I'm calling that with subprocess.Popen(). My exe now prints some commands and after some time, it gets into a loopback-mode and waits for input. When that happens, it outputs a single dot every second in the windows command prompt. It does put the subsequent dots on the same line, whereas all output before that is on its own line. I seem to be able to catch all output before that, but I cannot get this output, probably to some buffering going on(?). How can I get this output in my python console? Relevant code as below:
import subprocess
import sys
cmd = 'long_running.exe'
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, b''):
sys.stdout.write(line)
Related
I have an interactive tool that run on console; when I run the tool, it waits for commands, executes them and when I am done, I type the exit command to quit.
I would like to use Python to pass commands to this interactive tool, and read the output; although I am having problems.
This is what I did so far:
proc = Popen(myapp, stdin=PIPE,stdout=PIPE, stderr=PIPE, shell=False, universal_newlines=True)
print(proc.stdout.readline())
This is opening my app and prints out the first line of output, but then even if I pass a write statement to stdin, nothing happens. If I would run this from terminal, I would get 4 lines before the prompt would show, but here I get just the first line.
proc.stdin.write('list\n')
proc.stdin.flush()
print(proc.stdout.readline())
I tried to parse the lines based on the count, but I get a large number, while I should get just 3 lines; and after the counter passes the 3rd line, the app just hangs there doing nothing until I kill it.
total_lines = len(proc.stdout.readline()) # this returns me 300+ lines instead of 3
for lines in range(total_lines):
print(proc.stdout.readline()) # when the counter passes 3, the app hangs
The idea is to pass different commands in sequence; kinda like what you would do in a regular shell, and read the output; although I can't get this to work. I don't get how you interact with this app; since it should work exactly like it does in terminal, but it does not.
On the terminal, I have two programs to run using subprocess
First, I will call ./matrix-odas & so the first program will run in the background and I can then type the second command. The first command will return some messages.
The second command ~/odas/bin/odaslive -vc ~/odas/config/odaslive/matrix_creator.cfg will open the second program and it will keep running and keep printing out text. I'd like to use subprocess to open these programs and capture both outputs.
I have never used subprocess before and following tutorials, I am writing the script on Jupyter notebook (python 3.7) in order to see the output easily.
from subprocess import Popen, PIPE
p = Popen(["./matrix-odas", "&"], stdout=PIPE, stderr=PIPE, cwd=wd, universal_newlines=True)
stdout, stderr = p.communicate()
print(stdout)
This is the code that i tried to open the first program. But Jupyter notebook always gets stuck at p.communicate() and I can't see the messages. Without running the first program in the background, I won't be able to get the command prompt after the messages are printed.
I would like to know what subprocess function should I use to solve this issue and which platform is better to test subprocess code. Any suggestions will be appreciated. Thank you so much!
From this example at the end of this section of the docs
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
it looks like you can access stdout (and I would assume stderr) from the object directly. I am not sure whether you need to use Popen as a context manager to access that property or not.
I'm struggling to get some python script to start a subprocess, wait until it completes and then retrieve the required data. I'm quite new to Python.
The command I wish to run as a subprocess is
./bin.testing/Eva -t --suite="temp0"
Running that command by hand in the Linux terminal produces:
in terminal mode
Evaluation error = 16.7934
I want to run the command as a python sub-process, and receive the output back. However, everything I try seems to skip the second line (ultimately, it's the second line that I want.) At the moment, I have this:
def job(self,fen_file):
from subprocess import Popen, PIPE
from sys import exit
try:
eva=Popen('{0}/Eva -t --suite"{0}"'.format(self.exedir,fen_file),shell=True,stdout=PIPE,stderr=PIPE)
stdout,stderr=eva.communicate()
except:
print ('Error running test suite '+fen_file)
exit("Stopping")
print(stdout)
.
.
.
return 0
All this seems to produce is
in terminal mode
0
with the important line missing. The print statement is just so I can see what I am getting back from the sub-process -- the intention is that it will be replaced with code that processes the number from the second line and returns the output (here I'm just returning 0 just so I can get this particular bit to work first. The caller of this function prints the result, which is why there is a zero at the end of the output.) exedir is just the directory of the executable for the sub-process, and fen-file is just an ascii file that the sub-process needs. I have tried removing the 'in terminal mode' from the source code of the sub-process and re compiling it, but that doesn't work -- it still doesn't return the important second line.
Thanks in advance; I expect what I am doing wrong is really very simple.
Edit: I ought to add that the subprocess Eva can take a second or two to complete.
Since the 2nd line is an error message, it's probably stored in your stderr variable!
To know for sure you can print your stderr in your code, or you can run the program on the command line and see if the output is split into stdout and stderr. One easy way is to do ./bin.testing/Eva -t --suite="temp0" > /dev/null. Any messages you get are stderr since stdout is redirected to /dev/null.
Also, typically with Popen the shell=True option is discouraged unless really needed. Instead pass a list:
[os.path.join(self.exedir, 'Eva'), '-t', '--suite=' + fen_file], shell=False, ...
This can avoid problems down the line if one of your arguments would normally be interpreted by the shell. (Note, I removed the ""'s, because the shell would normally eat those for you!)
Try using subprocess check_output.
output_lines = subprocess.check_output(['./bin.testing/Eva', '-t', '--suite="temp0"'])
for line in output_lines.splitlines():
print(line)
I'm writing a simple wrapper over python debugger (pdb) and I need to parse pdb output. But I have a problem reading text from process pipe.
Example of my code:
import subprocess, threading, time
def readProcessOutput(process):
while not process.poll():
print(process.stdout.readline())
process = subprocess.Popen('python -m pdb script.py', shell=True, universal_newlines=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
read_thread = threading.Thread(target=readProcessOutput, args=(process,))
read_thread.start()
while True:
time.sleep(0.5)
When i execute given command (python -m pdb script.py) in OS shell I get results like this:
> c:\develop\script.py(1)<module>()
-> print('hello, world!')
(Pdb)
But when i run my script i get only two lines, but can't get pdb prompt. Writing commands to stdin after this has no effect. So my question is:
why I cannot read third line? How can I avoid this problem and get correct output?
Platform: Windows XP, Python 3.3
The third line can not be read by readline() because it is not terminated yet by the end of line. You see usually the cursor after "(pdb) " until you write anything + enter.
The communication to processes that have some prompt is usually more complicated. It proved to me to write also an independent thread for data writer first for easier testing the communication in order to be sure that the main thread never freezes if too much is tried to be written or read. Then it can be simplified again.
My problem is this--I need to get output from a subprocess and I am using the following code to call it-- (Feel free to ignore the long arguments. The importing thing is the stdout= subprocess.PIPE)
(stdout, stderr) = subprocess.Popen([self.ChapterToolPath, "-x", book.xmlPath , "-a", book.aacPath , "-o", book.outputPath+ "/" + fileName + ".m4b"], stdout= subprocess.PIPE).communicate()
print stdout
Thanks to an answer below, I've been able to get the output of the program, but I still end up waiting for the process to terminate before I get anything. The interesting thing is that in my debugger, there is all sorts of text flying by in the console and it is all ignored. But the moment that anything is written to the console in black (I am using pycharm) the program continues without a problem. Could the main program be waiting for some kind of output in order to move on? This would make sense because I am trying to communicate with it.... Is there a difference between text that I can see in the console and actual text that makes it to the stdout? And how would I collect the text written to the console?
Thanks!
The first line of the documentation for subprocess.call() describes it as such:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
Thus, it necessarily waits for the subprocess to exit.
subprocess.Popen(), by contrast, does not do this, returning a handle on a process with which one than then communicate().
To get all output from a program:
from subprocess import check_output as qx
output = qx([program, arg1, arg2, ...])
To get output while the program is running:
from subprocess import Popen, PIPE
p = Popen([program, arg1, ...], stdout=PIPE)
for line in iter(p.stdout.readline, ''):
print line,
There might be a buffering issue on the program' side if it prints line-by-line when run interactively but buffers its output if run as a subprocess. There are various solutions depending on your OS or the program e.g., you could run it using pexpect module.