I have noticed two different behaviors with two approaches that should have result in the same outcome.
The goal - to execute an external program using subprocess module, send some data and read the results.
The external program is PLINK, platform is WindowsXP, Python version 3.3.
The main idea-
execution=["C:\\Pr..\\...\\plink.exe", "-l", username, "-pw", "***", IP]
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False)
con=a.stdout.readline()
if (con.decode("utf-8").count("FATAL ERROR: Network error: Connection timed out")==0):
a.stdin.write(b"con rout 1\n")
print(a.stdout.readline().decode("utf-8"))
a.stdin.write(b"infodf\n")
print(a.stdout.readline().decode("utf-8"))
else:
print("ERROR")
a.kill()
So far so good.
Now, I want to be able to do a loop (after each write to the sub process's stdin), that waits untill EOF of the sub process's stdout, print it, then another stdin command, and so on.
So I first tried what previous discussions about the same topic yield (live output from subprocess command, read subprocess stdout line by line, python, subprocess: reading output from subprocess) .
And it didnt work (it hangs forever) because the PLINK process is remaining alive untill I kill it myself, so there is no use of waiting for the stdout of the sub process to reach EOF or to do a loop while stdout is true because it will always be true until I kill it.
So I decided to read from stdout twice every time I am writing to stdin (good enought for me)-
execution=["C:\\Pr..\\...\\plink.exe", "-l", username, "-pw", "***", IP]
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False)
con=a.stdout.readline()
if (con.decode("utf-8").count("FATAL ERROR: Network error: Connection timed out")==0):
a.stdin.write(b"con rout 1\n")
print(a.stdout.readline().decode("utf-8"))
print(a.stdout.readline().decode("utf-8")) //the extra line [1]
a.stdin.write(b"infodf\n")
print(a.stdout.readline().decode("utf-8"))
print(a.stdout.readline().decode("utf-8")) //the extra line [2]
else:
print("ERROR")
a.kill()
But the first extra readline() hangs forever, as far as I understand, for the same reason I mentioned. The first extra readline() waits forever for output, because the only output was already read in the first readline(), and because PLINK is alive, the function just "sit" there and waits for a new output line to get.
So I tried this code, expecting the same hang because PLINK never dies until i kill it-
execution=["C:\\Pr..\\...\\plink.exe", "-l", username, "-pw", "***", IP]
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False)
con=a.stdout.readline()
if (con.decode("utf-8").count("FATAL ERROR: Network error: Connection timed out")==0):
a.stdin.write(b"con rout 1\n")
print(a.stdout.readline().decode("utf-8"))
a.stdin.write(b"infodf\n")
print(a.stdout.readline().decode("utf-8"))
print(a.communicate()[0].decode("utf-8")) //Popen.communicate() function
else:
print("ERROR")
a.kill()
I tried that because according to the documentation of communicate(), the function wait until the process is ended, and then it finishes. Also, it reads from stdout until EOF. (same as writing and reading stdout and stdin)
But communicate() finishes and does not hang, in opposite of the previous code block.
What am I missing here? why when using communicate() the PLINK ends, but when using readline() it does not?
Your program without communicate() deadlocks because both processes are waiting on each other to write something before they write anything more themselves.
communicate() does not deadlock in your example because it closes the stream, like the command a.stdin.close() would. This sends an EOF to your subprocess, letting it know that there is no more input coming, so it can close itself, which in turn closes its output, so a.stdout.read() eventually returns an EOF (empty string).
There is no special signal that your main process will receive from your subprocess to let you know that it is done writing the results from one command, but is ready for another command.
This means that to communicate back and forth with one subprocess like you're trying to, you must read the exact number of lines that the subprocess sends. Like you saw, if you try to read too many lines, you deadlock. You might be able to use what you know, such as the command you sent it, and the output you have seen so far, to figure out exactly how many lines to read.
You can use threads to write and read at the same time, especially if the output only needs to be printed to the user:
from threading import Thread
def print_remaining(stream):
for line in stream:
print(line.decode("utf-8"))
con = a.stdout.readline()
if "FATAL ERROR" not in con.decode("utf-8"):
Thread(target=print_remaining, args=[a.stdout]).start()
for cmd in LIST_OF_COMMANDS_TO_SEND:
a.stdin.write(cmd)
The thing is that when using subprocess.Popen your code continues to be read even before the process terminates. Try appending .wait() to your Popen call (see documentation),
a=subprocess.Popen(execution, bufsize=0, stdout=PIPE, stdin=PIPE, stderr=STDOUT, shell=False).wait()
This will ensure that the execution finishes before going on with anything else.
Related
We have created a commodity function used in many projects which uses subprocess to start a command. This function is as follows:
def _popen( command_list ):
p = subprocess.Popen( command_list, stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
out, error_msg = p.communicate()
# Some processes (e.g. system_start) print a number of dots in stderr
# even when no error occurs.
if error_msg.strip('.') == '':
error_msg = ''
return out, error_msg
For most processes this works as intended.
But now I have to use it with a background-process which need to keep running as long as my python-script is running as well and thus now the fun starts ;-).
Note: the script also needs to start other non background-processes using this same _popen-function.
I know that by skipping p.communicate I can make the process start in the background, while my python script continues.
But there are 2 problems with this:
I need to check that the background process started correctly
While the main process is running I need to check the stdout and stderr of the background process from time to time without stopping the process / ending hanging in the background process.
Check background process started correctly
For 1 I currently adapted the _popen version to take an extra parameter 'skip_com' (default False) to skip the p.communicate call. And in that case I return the p-object i.s.o. out and error_msg.
This so I can check if the process is running directly after starting it up and if not call communicate on the p-object to check what the error_msg is.
MY_COMMAND_LIST = [ "<command that should go to background>" ]
def _popen( command_list, skip_com=False ):
p = subprocess.Popen( command_list, stdout=subprocess.PIPE,
stderr=subprocess.PIPE )
if not skip_com:
out, error_msg = p.communicate()
# Some processes (e.g. system_start) print a number of dots in stderr
# even when no error occurs.
if error_msg.strip('.') == '':
error_msg = ''
return out, error_msg
else:
return p
...
p = _popen( MY_COMMAND_LIST, True )
error = _get_command_pid( MY_COMMAND_LIST ) # checks if background command is running using _popen and ps -ef
if error:
_, error_msg = p.communicate()
I do not know if there is a better way to do this.
check stdout / stderr
For 2 I have not found a solution which does not cause the script to wait for the end of the background process.
The only ways I know to communicate is using iter on e.g. p.stdout.readline. But that will hang if the process is still running:
for line in iter( p.stdout.readline, "" ): print line
Any one an idea how to do this?
/edit/ I need to check the data I get from stdout and stderr seperately. Especially stderr is important in this case since if the background process encounters an error it will exit and I need to catch that in my main program to be able to prevent errors caused by that exit.
The stdout output is needed in some situations to check the expected behaviour in the background process and to react on that.
Update
The subprocess will actually exit if it encounters an error
If you don't need to read the output to detect an error then redirect it to DEVNULL and call .poll() to check child process' status from time to time without stopping the process.
assuming you have to read the output:
Do not use stdout=PIPE, stderr=PIPE unless you read from the pipes. Otherwise, the child process may hang as soon as any of the corresponding OS pipe buffers fill up.
If you want to start a process and do something else while it is running then you need a non-blocking way to read its output. A simple portable way is to use a thread:
def process_output(process):
with finishing(process): # close pipes, call .wait()
for line in iter(process.stdout.readline, b''):
if detected_error(line):
communicate_error(process, line)
process = Popen(command, stdout=PIPE, stderr=STDOUT, bufsize=1)
Thread(target=process_output, args=[process]).start()
I need to check the data I get from stdout and stderr seperately.
Use two threads:
def read_stdout(process):
with waiting(process), process.stdout: # close pipe, call .wait()
for line in iter(process.stdout.readline, b''):
do_something_with_stdout(line)
def read_stderr(process):
with process.stderr:
for line in iter(process.stderr.readline, b''):
if detected_error(line):
communicate_error(process, line)
process = Popen(command, stdout=PIPE, stderr=PIPE, bufsize=1)
Thread(target=read_stdout, args=[process]).start()
Thread(target=read_stderr, args=[process]).start()
You could put the code into a custom class (to group do_something_with_stdout(), detected_error(), communicate_error() methods).
It may be better or worse than what you imagine...
Anyway, the correct way of reading a pipe line by line is simply:
for line in p.stdout:
#process line is you want of just
print line
Or if you need to process that inside of a higher level loop
line = next(p.stdout)
But a harder problem could come from the commands started from Python. Many programs use the underlying C standard library, and by default stdout is a buffered stream. The system detects whether the standard output is connected to a terminal, and automatically flushes output on a new line (\n) or on a read on same terminal. But if output is connected to a pipe or a file, everything is buffered until the buffer is full, which on current systems requires several kBytes. In that case nothing can be done at Python level. Above code would get a full line as soon as it would written on the pipe, but cannot guess before callee has actually written something...
I'm writing a wrapper for Xfoil and my first command set of commands are:
commands=[]
commands.append('plop\n')
commands.append('g,f\n')
commands.append('\n')
commands.append('load '+ afile+'\n')
commands.append('\n')
#commands.append('ppar\n');
#commands.append('n %g\n',n);
commands.append('\n')
commands.append('\n')
commands.append('oper\n')
commands.append('iter '+ str(iter) + '\n')
commands.append('visc {0:f}\n'.format(Re))
commands.append('m {0:f}\n'.format(M))
I'm interacting with xfoil as below:
xfoil_path=os.getcwd()+'/xfoil.exe'
Xfoil = Popen(xfoil_path, shell=True, stdin=PIPE, stdout=None, stderr=None, creationflags=0)
for i in commands:
print '\nExecuting:', i
#stdin.write returns None if write is blocked and that seems to be the case here
Xfoil.stdin.write(i)
Xfoil.wait()
#print Xfoil.stdin.write(i)
However, Xfoil.stdin.write is being blocked form interacting with the program -- xfoil.exe -- as Xfoil.stdin.write(i) returns a None.
This happens immediately after the first command i.e. plop
How do I resolve this?
Solution is to add Xfoil.stdin.close(); Closing the buffer allows the program to proceed.
Xfoil = Popen(xfoil_path, shell=True, stdin=PIPE, stdout=None, stderr=None, creationflags=0)
for i in commands:
Xfoil.stdin.write(i)
Xfoil.stdin.close()
Xfoil.wait()
Seeking help understand why Xfoil.stdin.close() needs to be added. How does closing the buffer allow xfoil.exe to proceed?
To send multiple commands, you could use Popen.communicate() method that sends commands, closes the pipe, and waits for the child process to finish:
import os
from subprocess import Popen, PIPE
process = Popen(os.path.abspath('xfoil.exe'), stdin=PIPE)
process.communicate(b"".join(commands))
Xfoil.wait() in your code waits for the executable to finish after the first command. Closing the pipe (Xfoil.stdin) indicates EOF otherwise the deadlock may happen if xfoil.exe read until EOF (no command makes it exit otherwise).
I have a script that runs another command, waits for it to finish, logs the stdout and stderr and based the return code does other stuff. Here is the code:
p = subprocess.Popen(command, stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
o, e = p.communicate()
if p.returncode:
# report error
# do other stuff
The problem I'm having is that if command takes a long time to run none of the other actions get done. The possible errors won't get reported and the other stuff that needs to happen if no errors doesn't get done. It essentially doesn't go past p.communicate() if it takes too long. Some times this command can takes hours (or even longer) to run and some times it can take as little as 5 seconds.
Am I missing something or doing something wrong?
As per the documentation located here, it's safe to say that you're code is waiting for the subprocess to finish.
If you need to go do 'other things' while you wait you could create a loop like:
while p.poll():
# 'other things'
time.sleep(0.2)
Pick a sleep time that's reasonable for how often you want python to wake up and check the subprocess as well as doing its 'other things'.
The Popen.communicate waits for the process to finish, before anything is returned. Thus it is not ideal for any long running command; and even less so if the subprocess can hang waiting for input, say prompting for a password.
The stderr=subprocess.PIPE, stdout=subprocess.PIPE are needed only if you want to capture the output of the command into a variable. If you are OK with the output going to your terminal, then you can remove these both; and even use subprocess.call instead of Popen. Also, if you do not provide input to your subprocess, then do not use stdin=subprocess.PIPE at all, but direct that from the null device instead (in Python 3.3+ you can use stdin=subprocess.DEVNULL; in Python <3.3 use stdin=open(os.devnull, 'rb')
If you need the contents too, then instead of calling p.communicate(), you can read p.stdout and p.stderr yourself in chunks and output to the terminal, but it is a bit complicated, as it is easy to deadlock the program - the dummy approach would try to read from the subprocess' stdout while the subprocess would want to write to stderr. For this case there are 2 remedies:
you could use select.select to poll both stdout and stderr to see whichever becomes ready first and read from it then
or, if you do not care for stdout and stderr being combined into one,
you can use STDOUT to redirect the stderr stream into the stdout stream: stdout=subprocess.PIPE, stderr=subprocess.STDOUT; now all the output comes to p.stdout that you can read easily in loop and output the chunks, without worrying about deadlocks:
If the stdout, stderr are going to be huge, you can also spool them to a file right there in Popen; say,
stdout = open('stdout.txt', 'w+b')
stderr = open('stderr.txt', 'w+b')
p = subprocess.Popen(..., stdout=stdout, stderr=stderr)
while p.poll() is None:
# reading at the end of the file will return an empty string
err = stderr.read()
print(err)
out = stdout.read()
print(out)
# if we met the end of the file, then we can sleep a bit
# here to avoid spending excess CPU cycles just to poll;
# another option would be to use `select`
if not err and not out: # no input, sleep a bit
time.sleep(0.01)
I'm trying to get output of another script, using Python's subprocess.Popen like follows
process = Popen(command, stdout=PIPE, shell=True)
exitcode = process.wait()
output = process.stdout.read() # hangs here
It hangs at the third line, only when I run it as a python script and I cannot reproduce this in the python shell.
The other script prints just a few words and I am assuming that it's not a buffer issue.
Does anyone has idea about what I am doing wrong here?
You probably want to use .communicate() rather than .wait() plus .read(). Note the warning about wait() on the subprocess documentation page:
Warning This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that.
http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait
read() waits for EOF before returning.
You can:
wait for the subprocess to die, then read() will return.
use readline() if your output is broken into lines (will still hang if no output lines).
use os.read(F,N) which returns at most N bytes from F, but will still block if the pipe is empty (unless O_NONBLOCK is set on the fd).
You can see how to deal with hanging reading of stdout/stderr in the next sources:
readingproc
I trying to start a program (HandBreakCLI) as a subprocess or thread from within python 2.7. I have gotten as far as starting it, but I can't figure out how to monitor it's stderr and stdout.
The program outputs it's status (% done) and info about the encode to stderr and stdout, respectively. I'd like to be able to periodically retrieve the % done from the appropriate stream.
I've tried calling subprocess.Popen with stderr and stdout set to PIPE and using the subprocess.communicate, but it sits and waits till the process is killed or complete then retrieves the output then. Doesn't do me much good.
I've got it up and running as a thread, but as far as I can tell I still have to eventually call subprocess.Popen to execute the program and run into the same wall.
Am I going about this the right way? What other options do I have or how to I get this to work as described?
I have accomplished the same with ffmpeg. This is a stripped down version of the relevant portions. bufsize=1 means line buffering and may not be needed.
def Run(command):
proc = subprocess.Popen(command, bufsize=1,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
universal_newlines=True)
return proc
def Trace(proc):
while proc.poll() is None:
line = proc.stdout.readline()
if line:
# Process output here
print 'Read line', line
proc = Run([ handbrakePath ] + allOptions)
Trace(proc)
Edit 1: I noticed that the subprocess (handbrake in this case) needs to flush after lines to use this (ffmpeg does).
Edit 2: Some quick tests reveal that bufsize=1 may not be actually needed.