We Keep Coding

How to capture output from continuous process in Python?

I am new to Python and Linux. I have a process running in a terminal window that will go indefinitely. The only way to stop it would be for it to crash or for me to hit ctrl+C. This process outputs text to the terminal window that I wish to capture with Python, so I can do some additional processing of that text.
I know I need to do something with getting stdout, but no matter what I try, I can't seem to capture the stdout correctly. Here is what I have so far.
import subprocess
command = 'echo this is a test. Does it come out as a single line?'
def myrun(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout = []
while True:
line = p.stdout.read()
stdout.append(line)
if line == '' and p.poll() != None:
break
return ''.join(stdout)
result = myrun(command)
print('> ' + result),
This will work when my command is a simple "echo blah blah blah". I am guessing this is because the echo process terminates. If I try running the continuous command, the output is never captured. Is this possible to do?

read() will block on reading until reach EOF, use read(1024) or readline() instead:
read(size=-1)
Read and return up to size bytes. If the argument is omitted, None, or negative, data is read and returned until EOF is reached.
eg:
p = subprocess.Popen('yes', stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
print(line.strip())
see more on the python io doc.

Understanding subprocess module in python when running program from python script

I usually run a program from my OpenSuse linux terminal by typing ./run file_name. This will bring up a series of options that I can choose from by typing a numeric value 0-9 and hitting return on my keyboard. Now I want to do this from a python script automatically. My example below is not working, but I can't understand where I'm failing and how to debug:
import subprocess
p = subprocess.Popen(["/path/to/program/run", file_name], stdin = subprocess.PIPE,stdout=subprocess.PIPE,shell=False)
print "Hello"
out, err = p.communicate(input='0\r\n')
print out
print err
for line in p.stdout.readlines():
print line
The output of this program is just
>> Hello
>>
i.e. then it seems to freeze (I have no idea whats actually happening!) I would have expected to see what I see when I run ./run file_name
and hit 0 and then return directly in my terminal, but I assure you this is not the case.
What can I do to debug my code?
Edit 1: as suggested in comments
import subprocess
fileName = 'test_profile'
p = subprocess.Popen(["/path/to/program/run", fileName], stdin = subprocess.PIPE,stdout=subprocess.PIPE,shell=False)
print "Hello"
for line in iter(p.stdout.readline,""):
print line
will indeed return the stdout of my program!

communicate waits for the completion of the program. For example:
import subprocess
p = subprocess.Popen(["cut", "-c2"], stdin=subprocess.PIPE, stdout=subprocess.PIPE,shell=False)
out, err = p.communicate(input='abc')
print("Result: '{}'".format(out.strip()))
# Result: 'b'
It sounds like you have a more interactive script, in which case you probably should try out pexpect
import pexpect
child = pexpect.spawn('cut -c2')
child.sendline('abc')
child.readline() # repeat what was typed
print(child.readline()) # prints 'b'

Linux, Write output to the file and terminate it using Python

There are various topics available on this very topic, "How to write output to the text file".
But my issue is different because the output to the command in question is continous.
What I want to do is, write the Output of the command cgps -s to the file aaa.txt
here is the code,
import signal
import os
import subprocess
p = subprocess.Popen(["cgps", "-s", ">> aaa.txt"], stdout=subprocess.PIPE,shell=True, preexec_fn=os.setsid)
os.killpg(p.pid, signal.SIGTERM)
The code doesn't work at all and no file is created with the name aaa.txt
When I execute this command through terminal,
cgps -s > aaa.txt
Then I have to press CTRL+C to terminate the output from being written on the output file because the output is continuous.
Is there any way to just capture one output and write it to the file and terminate it using Python or using Command line ?

So you are trying to create a pipe. Try this:
import subprocess
p = subprocess.Popen(["cgps", "-s"], stdout=subprocess.PIPE)
with open("aaa.txt", "w") as f:
while True:
line = p.stdout.readline()
if not line:
break
if some_condition(): # <-- check whether you have enough output
p.terminate()
break
f.writeline(line)
p.wait()

Getting output from and giving commands to a python subprocess

I am trying to get output from a subprocess and then give commands to that process based on the preceding output. I need to do this a variable number of times, when the program needs further input. (I also need to be able to hide the subprocess command prompt if possible).
I figured this would be an easy task given that I have seen this problem being discussed in posts from 2003 and it is nearly 2012 and it appears to be a pretty common need and really seems like it should be a basic part of any programming language. Apparently I was wrong and somehow almost 9 years later there is still no standard way of accomplishing this task in a stable, non-destructive, platform independent way!
I don't really understand much about file i/o and buffering or threading so I would prefer a solution that is as simple as possible. If there is a module that accomplishes this that is compatible with python 3.x, I would be very willing to download it. I realize that there are multiple questions that ask basically the same thing, but I have yet to find an answer that addresses the simple task that I am trying to accomplish.
Here is the code I have so far based on a variety of sources; however I have absolutely no idea what to do next. All my attempts ended in failure and some managed to use 100% of my CPU (to do basically nothing) and would not quit.
import subprocess
from subprocess import Popen, PIPE
p = Popen(r'C:\postgis_testing\shellcomm.bat',stdin=PIPE,stdout=PIPE,stderr=subprocess.STDOUT shell=True)
stdout,stdin = p.communicate(b'command string')
In case my question is unclear I am posting the text of the sample batch file that I demonstrates a situation in which it is necessary to send multiple commands to the subprocess (if you type an incorrect command string the program loops).
#echo off
:looper
set INPUT=
set /P INPUT=Type the correct command string:
if "%INPUT%" == "command string" (echo you are correct) else (goto looper)
If anyone can help me I would very much appreciate it, and I'm sure many others would as well!
EDIT here is the functional code using eryksun's code (next post) :
import subprocess
import threading
import time
import sys
try:
import queue
except ImportError:
import Queue as queue
def read_stdout(stdout, q, p):
it = iter(lambda: stdout.read(1), b'')
for c in it:
q.put(c)
if stdout.closed:
break
_encoding = getattr(sys.stdout, 'encoding', 'latin-1')
def get_stdout(q, encoding=_encoding):
out = []
while 1:
try:
out.append(q.get(timeout=0.2))
except queue.Empty:
break
return b''.join(out).rstrip().decode(encoding)
def printout(q):
outdata = get_stdout(q)
if outdata:
print('Output: %s' % outdata)
if __name__ == '__main__':
#setup
p = subprocess.Popen(['shellcomm.bat'], stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
bufsize=0, shell=True) # I put shell=True to hide prompt
q = queue.Queue()
encoding = getattr(sys.stdin, 'encoding', 'utf-8')
#for reading stdout
t = threading.Thread(target=read_stdout, args=(p.stdout, q, p))
t.daemon = True
t.start()
#command loop
while p.poll() is None:
printout(q)
cmd = input('Input: ')
cmd = (cmd + '\n').encode(encoding)
p.stdin.write(cmd)
time.sleep(0.1) # I added this to give some time to check for closure (otherwise it doesn't work)
#tear down
for n in range(4):
rc = p.poll()
if rc is not None:
break
time.sleep(0.25)
else:
p.terminate()
rc = p.poll()
if rc is None:
rc = 1
printout(q)
print('Return Code: %d' % rc)
However when the script is run from a command prompt the following happens:
C:\Users\username>python C:\postgis_testing\shellcomm7.py
Input: sth
Traceback (most recent call last):
File "C:\postgis_testing\shellcomm7.py", line 51, in <module>
p.stdin.write(cmd)
IOError: [Errno 22] Invalid argument
It seems that the program closes out when run from command prompt. any ideas?

This demo uses a dedicated thread to read from stdout. If you search around, I'm sure you can find a more complete implementation written up in an object oriented interface. At least I can say this is working for me with your provided batch file in both Python 2.7.2 and 3.2.2.
shellcomm.bat:
#echo off
echo Command Loop Test
echo.
:looper
set INPUT=
set /P INPUT=Type the correct command string:
if "%INPUT%" == "command string" (echo you are correct) else (goto looper)
Here's what I get for output based on the sequence of commands "wrong", "still wrong", and "command string":
Output:
Command Loop Test
Type the correct command string:
Input: wrong
Output:
Type the correct command string:
Input: still wrong
Output:
Type the correct command string:
Input: command string
Output:
you are correct
Return Code: 0
For reading the piped output, readline might work sometimes, but set /P INPUT in the batch file naturally isn't writing a line ending. So instead I used lambda: stdout.read(1) to read a byte at a time (not so efficient, but it works). The reading function puts the data on a queue. The main thread gets the output from the queue after it writes a a command. Using a timeout on the get call here makes it wait a small amount of time to ensure the program is waiting for input. Instead you could check the output for prompts to know when the program is expecting input.
All that said, you can't expect a setup like this to work universally because the console program you're trying to interact with might buffer its output when piped. In Unix systems there are some utility commands available that you can insert into a pipe to modify the buffering to be non-buffered, line-buffered, or a given size -- such as stdbuf. There are also ways to trick the program into thinking it's connected to a pty (see pexpect). However, I don't know a way around this problem on Windows if you don't have access to the program's source code to explicitly set the buffering using setvbuf.
import subprocess
import threading
import time
import sys
if sys.version_info.major >= 3:
import queue
else:
import Queue as queue
input = raw_input
def read_stdout(stdout, q):
it = iter(lambda: stdout.read(1), b'')
for c in it:
q.put(c)
if stdout.closed:
break
_encoding = getattr(sys.stdout, 'encoding', 'latin-1')
def get_stdout(q, encoding=_encoding):
out = []
while 1:
try:
out.append(q.get(timeout=0.2))
except queue.Empty:
break
return b''.join(out).rstrip().decode(encoding)
def printout(q):
outdata = get_stdout(q)
if outdata:
print('Output:\n%s' % outdata)
if __name__ == '__main__':
ARGS = ["shellcomm.bat"] ### Modify this
#setup
p = subprocess.Popen(ARGS, bufsize=0, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
q = queue.Queue()
encoding = getattr(sys.stdin, 'encoding', 'utf-8')
#for reading stdout
t = threading.Thread(target=read_stdout, args=(p.stdout, q))
t.daemon = True
t.start()
#command loop
while 1:
printout(q)
if p.poll() is not None or p.stdin.closed:
break
cmd = input('Input: ')
cmd = (cmd + '\n').encode(encoding)
p.stdin.write(cmd)
#tear down
for n in range(4):
rc = p.poll()
if rc is not None:
break
time.sleep(0.25)
else:
p.terminate()
rc = p.poll()
if rc is None:
rc = 1
printout(q)
print('\nReturn Code: %d' % rc)

Python: How to read stdout non blocking from another process?

During the runtime of a process I would like to read its stdout and write it to a file. Any attempt of mine however failed because no matter what I tried as soon as I tried reading from the stdout it blocked until the process finished.
Here is a snippet of what I am trying to do. (The first part is simply a python script that writes something to stdout.)
import subprocess
p = subprocess.Popen('python -c \'\
from time import sleep\n\
for i in range(3):\n\
sleep(1)\n\
print "Hello", i\
\'', shell = True, stdout = subprocess.PIPE)
while p.poll() == None:
#read the stdout continuously
pass
print "Done"
I know that there are multiple questions out there that deal with the same subject. However, none of the ones I found was able to answer my question.

What is happening is buffering on the writer side. Since you are writing such small chunks from the little code snippet the underlying FILE object is buffering the output until the end. The following works as you expect.
#!/usr/bin/python
import sys
import subprocess
p = subprocess.Popen("""python -c '
from time import sleep ; import sys
for i in range(3):
sleep(1)
print "Hello", i
sys.stdout.flush()
'""", shell = True, stdout = subprocess.PIPE)
while True:
inline = p.stdout.readline()
if not inline:
break
sys.stdout.write(inline)
sys.stdout.flush()
print "Done"
However, you may not be expecting the right thing. The buffering is there to reduce the number of system calls in order to make the system more efficient. Does it really matter to you that the whole text is buffered until the end before you write it to a file? Don't you still get all the output in the file?

the following code would print stdout line by line as the subprocess runs until the readline() method returns an empty string:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
for line in iter(p.stdout.readline, ''):
print line
p.stdout.close()
print 'Done'
update relating to your question better:
import subprocess
p = subprocess.Popen(['python'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
p.stdin.write("""
from time import sleep ; import sys
for i in range(3):
sleep(1)
print "Hello", i
sys.stdout.flush()
""")
p.stdin.close()
for line in iter(p.stdout.readline, ''):
print line
p.stdout.close()
print 'Done'

You can use subprocess.communicate() to get the output from stdout. Something like:
while(p.poll() == None):
#read the stdout continuously
print(p.communicate()[0])
pass
More info available at: http://docs.python.org/library/subprocess.html