We Keep Coding

Hanging parent process after running a timed-out subprocess and piping results

I wrote some code to run a script (via a subprocess) and kill the child process after a certain timeout. I'm running a script called "runtime_hang_script.sh" that just contains "./runtime_hang," which runs an infinite loop. I'm also redirecting stdout to a pipe -- I plan to write it to both sys.stdout and to a file (aka I'm trying to implement tee). However, my code hangs after the subprocess times out. Note that this ONLY hangs when running "sh runtime_hang_script.sh" and not "./runtime_hang." Also, this doesn't hang when I try piping directly to a file or when I don't read from the pipe.
I've tried other implementations of creating a timed subprocess, but I keep on getting the same issue. I've even tried raising a signal at the end of the problem -- for some reason, the signal is raised earlier than anticipated, so this doesn't work either. Any help would be appreciated. Thanks in advance!
process = None
def run():
global process
timeout_secs = 5
args = ['sh', 'runtime_hang_script.sh']
sys.stdout.flush()
process = subprocess.Popen(args, stdout=subprocess.PIPE, bufsize=1)
with process.stdout:
for line in iter(process.stdout.readline, b''):
sys.stdout.write(line.decode('utf-8'))
sys.stdout.flush()
process.wait()
proc_thread = threading.Thread(target=run)
proc_thread.start()
proc_thread.join(5)
print(proc_thread.is_alive())
if proc_thread.is_alive():
process.kill()

Assuming you are using Python 3.3 or newer, you can use the timeout argument of the subprocess.communicate() method to implement your 5-second timeout:
import subprocess
import sys
timeout_secs = 5
args = ['sh', 'runtime_hang_script.sh']
process = subprocess.Popen(args, stdout=subprocess.PIPE, bufsize=1)
try:
print("Waiting for data from child process...")
(stdoutData, stderrData) = process.communicate(None, timeout_secs)
print("From child process: stdoutData=[%s] stderrData=[%s]" % (stdoutData, stderrData))
except subprocess.TimeoutExpired:
print("Oops, child process took too long! Now it has to die")
process.kill()
print("Waiting for child process to exit...")
process.wait()
print("Child process exited.")
Note that spawning a child thread isn't necessary with this approach, since the timeout can work directly from the main thread.

Terminate a subprocess if string is in output

I'm trying to terminate a subprocess pid if a string is in the output, but it is not working. What is wrong?
import subprocess
import shlex
if "PING" in subprocess.check_call(shlex.split("ping -c 10 gogole.com")):
subprocess.check_call(shlex.split("ping -c 10 gogole.com")).terminate()

Please refere to the documentation for the methods you call. First of all, check_call executes until the process is finished, then returns the return code from the process. I'm not sure how you intend to find "PING" from a return code, which is typically an integer.
If it is there, look at the body of your if statement: you fork a totally new instance of ping, wait for it to complete, and then try to terminate the return code.
I recommend that you work through a tutorial on subprocesses. Learn how to grab a process handle and invoke operations on that. You'll need to get a handle on the output stream, look for "PING" in that, and then call terminate on the process handle you got at invocation.

import subprocess, os
run = "ping -c 10 google.com"
log = ""
process = subprocess.Popen(run, stdout=subprocess.PIPE, shell=True)
while True:
out = process.stdout.read(1)
log +=out
print log
if out == '' and process.poll() != None:
break
if "PING" in log:
print "terminated!"
process.kill()
process.terminate()
break

Python subprocess hangs on interaction

I am trying to write a minecraft server wrapper that allows me to send it commands and receive output. Eventually, I'll attach a socket interface so that I can control my home server remotely to restart / second commands / etc.
To this end, I am attempting to use the python subprocess module to start the server, then send commands and receive the output of the server. Right now, I am running into an issue I can grab the output of the server and reflect it to screen, but the very first command I send to the process freezes the whole thing and I have to kill it. It should be noted that I have attempted to remove the process.communicate line and instead replaced it with a print(command). This also froze the process My very basic current code is as follows:
from subprocess import Popen, PIPE
from threading import Thread
import threading
def listen(process):
while process.poll() is None:
output = process.stdout.readline()
print(str(output))
def talk(process):
command = input("Enter command: ")
while command != "exit_wrapper":
#freezes on first send command
parse_command(process, command)
command = input("Enter command: ")
print("EXITTING! KILLING SERVER!")
process.kill()
def parse_command(process, command):
process.communicate(command.encode())
def main():
process = Popen("C:\\Minecraft Servers\\ServerStart.bat", cwd = "C:\\Minecraft Servers\\", stdout=PIPE, stdin=PIPE)
listener = Thread(None, listen, None, kwargs={'process':process})
listener.start()
talker = Thread(None, talk, None, kwargs={'process':process})
talker.start()
listener.join()
talker.join()
if __name__ == "__main__":
main()
Any help offered would be greatly appreciated!

subprocess.Popen.communicate() documentation clearly states:
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.
And in your case it's doing exactly that. What you want to do instead of waiting for the process to be terminated is to interact with the process so, much like you're reading from the STDOUT stream directly in your listen() function, you should write to the process STDIN in order to send it commands. Something like:
def talk(process):
command = input("Enter command: ")
while command != "exit_wrapper" and process.poll() is None:
process.stdin.write(command) # write the command to the process STDIN
process.stdin.flush() # flush it
command = input("Enter command: ") # get the next command from user
if process.poll() is None:
print("EXITING! KILLING SERVER!")
process.kill()
The problem with this approach, however, is that you'll have a potential of overwriting the server's output with your Enter command: prompt and that the user will end up typing the command over the server's output instead in the 'prompt' you've designated.
What you might want to do instead is to parse your server's output in the listen() function and then based on the collected output determine when the wrapped server expects user input and then, and only then, call the talk() function (of course, remove the while loop from it) to obtain user input.
You should also pipe-out STDERR as well, in case the Minecraft server is trying to tell you something over it.

PYTHON subprocess cmd.exe closes after first command

I am working on a python program which implements the cmd window.
I am using subproccess with PIPE.
If for example i write "dir" (by stdout), I use communicate() in order to get the response from the cmd and it does work.
The problem is that in a while True loop, this doesn't work more than one time, it seems like the subprocess closes itself..
Help me please
import subprocess
process = subprocess.Popen('cmd.exe', shell=False, stdin=subprocess.PIPE,stdout=subprocess.PIPE,stderr=None)
x=""
while x!="x":
x = raw_input("insert a command \n")
process.stdin.write(x+"\n")
o,e=process.communicate()
print o
process.stdin.close()

The main problem is that trying to read subprocess.PIPE deadlocks when the program is still running but there is nothing to read from stdout. communicate() manually terminates the process to stop this.
A solution would be to put the piece of code that reads stdout in another thread, and then access it via Queue, which allows for reliable sharing of data between threads by timing out instead of deadlocking.
The new thread will read standard out continuously, stopping when there is no more data.
Each line will be grabbed from the queue stream until a timeout is reached(no more data in Queue), then the list of lines will be displayed to the screen.
This process will work for non-interactive programs
import subprocess
import threading
import Queue
def read_stdout(stdout, queue):
while True:
queue.put(stdout.readline()) #This hangs when there is no IO
process = subprocess.Popen('cmd.exe', shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
q = Queue.Queue()
t = threading.Thread(target=read_stdout, args=(process.stdout, q))
t.daemon = True # t stops when the main thread stops
t.start()
while True:
x = raw_input("insert a command \n")
if x == "x":
break
process.stdin.write(x + "\n")
o = []
try:
while True:
o.append(q.get(timeout=.1))
except Queue.Empty:
print ''.join(o)

Python, Popen and select - waiting for a process to terminate or a timeout

I run a subprocess using:
p = subprocess.Popen("subprocess",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.
I tried this:
SECONDS_TO_WAIT = 10
select.select([],
[p.stdout, p.stderr],
[p.stdout, p.stderr],
SECONDS_TO_WAIT)
but it just returns:
([],[],[])
on either condition. What can I do?

Have you tried using the Popen.Poll() method. You could just do this:
p = subprocess.Popen("subprocess",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
time.sleep(SECONDS_TO_WAIT)
retcode = p.poll()
if retcode is not None:
# process has terminated
This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.
Edit:
How about:
t_nought = time.time()
seconds_passed = 0
while(p.poll() is not None and seconds_passed < 10):
seconds_passed = time.time() - t_nought
if seconds_passed >= 10:
#TIMED OUT
This has the ugliness of being a busy wait, but I think it accomplishes what you want.
Additionally looking at the select call documentation again I think you may want to change it as follows:
SECONDS_TO_WAIT = 10
select.select([p.stderr],
[],
[p.stdout, p.stderr],
SECONDS_TO_WAIT)
Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).
I hope this helps.

This is what i came up with. Works when you need and don't need to timeout on thep process, but with a semi-busy loop.
def runCmd(cmd, timeout=None):
'''
Will execute a command, read the output and return it back.
#param cmd: command to execute
#param timeout: process timeout in seconds
#return: a tuple of three: first stdout, then stderr, then exit code
#raise OSError: on missing command or if a timeout was reached
'''
ph_out = None # process output
ph_err = None # stderr
ph_ret = None # return code
p = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# if timeout is not set wait for process to complete
if not timeout:
ph_ret = p.wait()
else:
fin_time = time.time() + timeout
while p.poll() == None and fin_time > time.time():
time.sleep(1)
# if timeout reached, raise an exception
if fin_time < time.time():
# starting 2.6 subprocess has a kill() method which is preferable
# p.kill()
os.kill(p.pid, signal.SIGKILL)
raise OSError("Process timeout has been reached")
ph_ret = p.returncode
ph_out, ph_err = p.communicate()
return (ph_out, ph_err, ph_ret)

Here is a nice example:
from threading import Timer
from subprocess import Popen, PIPE
proc = Popen("ping 127.0.0.1", shell=True)
t = Timer(60, proc.kill)
t.start()
proc.wait()

Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:
from twisted.internet.utils import getProcessOutputAndValue
from twisted.internet import reactor
def stop(r):
reactor.stop()
def eb(reason):
reason.printTraceback()
def cb(result):
stdout, stderr, exitcode = result
# do something
getProcessOutputAndValue('/bin/someproc', []
).addCallback(cb).addErrback(eb).addBoth(stop)
reactor.run()
Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:
http://twistedmatrix.com/projects/core/documentation/howto/process.html

Python 3.3
import subprocess as sp
try:
sp.check_call(["/subprocess"], timeout=10,
stdin=sp.DEVNULL, stdout=sp.DEVNULL, stderr=sp.DEVNULL)
except sp.TimeoutError:
# timeout (the subprocess is killed at this point)
except sp.CalledProcessError:
# subprocess failed before timeout
else:
# subprocess ended successfully before timeout
See TimeoutExpired docs.

If, as you said in the comments above, you're just tweaking the output each time and re-running the command, would something like the following work?
from threading import Timer
import subprocess
WAIT_TIME = 10.0
def check_cmd(cmd):
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
def _check():
if p.poll()!=0:
print cmd+" did not quit within the given time period."
# check whether the given process has exited WAIT_TIME
# seconds from now
Timer(WAIT_TIME, _check).start()
check_cmd('echo')
check_cmd('python')
The code above, when run, outputs:
python did not quit within the given time period.
The only downside of the above code that I can think of is the potentially overlapping processes as you keep running check_cmd.

This is a paraphrase on Evan's answer, but it takes into account the following :
Explicitly canceling the Timer object : if the Timer interval would be long and the process will exit by its "own will" , this could hang your script :(
There is an intrinsic race in the Timer approach (the timer attempt killing the process just after the process has died and this on Windows will raise an exception).
DEVNULL = open(os.devnull, "wb")
process = Popen("c:/myExe.exe", stdout=DEVNULL) # no need for stdout
def kill_process():
""" Kill process helper"""
try:
process.kill()
except OSError:
pass # Swallow the error
timer = Timer(timeout_in_sec, kill_process)
timer.start()
process.wait()
timer.cancel()