I have the following snippet of code:
def terminal_command(command, timeout=5*60):
"""Executes a terminal command."""
cmd = command.split(" ")
timer = time.strftime('%Hh %Mm %Ss', time.gmtime(timeout))
proc = None
try:
proc = subprocess.run(cmd, timeout=timeout, capture_output=True)
except subprocess.TimeoutExpired:
print("Timeout")
proc.terminate()
reason = "timeout"
stdout = b'error'
stderr = b'error'
if proc != None:
# Finished!
stdout = proc.stdout
stderr = proc.stderr
reason = "finished"
return stdout.decode('utf-8').strip(), stderr.decode('utf-8').strip(), reason
I ran a command which takes significantly longer than 5 minutes. In this instance, subprocess.run raises an exception, but proc is now None so I cannot use proc.terminate(). When the code terminates, as has been well documented elsewhere, the child process continues to run. I would like to terminate it.
Is there any way to terminate a subprocess on a TimeoutExpired, whilst redirecting output? I am on a Linux system so am open to requiring Popen but ideally I would like this to be cross-platform.
Four months later: I got it.
The core issue appears to be that using os.kill with signal.SIGKILL doesn't properly kill the process.
Modifying my code to the following works.
def custom_terminal_command(self, command, timeout=5*60, cwd=None):
with subprocess.Popen(command.split(" "), preexec_fn=os.setsid) as process:
wd = os.getcwd()
try:
if cwd is not None:
# Man fuck linux
for d in cwd.split("/"):
os.chdir(d)
stdout, stderr = process.communicate(None, timeout=timeout)
except subprocess.TimeoutExpired as exc:
import signal
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
try:
import msvcrt
except ModuleNotFoundError:
_mswindows = False
else:
_mswindows = True
if _mswindows:
# Windows accumulates the output in a single blocking
# read() call run on child threads, with the timeout
# being done in a join() on those threads. communicate()
# _after_ kill() is required to collect that and add it
# to the exception.
exc.stdout, exc.stderr = process.communicate()
else:
# POSIX _communicate already populated the output so
# far into the TimeoutExpired exception.
process.wait()
reason = 'timeout'
stdout, stderr = process.communicate()
except: # Including KeyboardInterrupt, communicate handled that.
process.kill()
# We don't call process.wait() as .__exit__ does that for us.
reason = 'other'
stdout, stderr = process.communicate()
raise
else:
reason = 'finished'
finally:
os.chdir(wd)
try:
return stdout.decode('utf-8').strip(), stderr.decode('utf-8').strip(), reason
except AttributeError:
try:
return stdout.strip(), stderr.strip(), reason
except AttributeError:
return stdout, stderr, reason
See the following SO post for a short discussion: How to terminate a python subprocess launched with shell=True
Related
I am using subprocess to execute a python script. I can successfully run the script.
import subprocess
import time
proc = subprocess.Popen(['python', 'test.py', ''], shell=True)
proc.communicate()
time.sleep(10)
proc.terminate()
test.py
import time
while True:
print("I am inside test.py")
time.sleep(1)
I can see the message I am inside test.py printed every second. However, I am not able to terminate the script while it is still running by proc.terminate().
Could someone kindly help me out?
proc.communicate waits for the process to finish unless you include a timeout. Your process doesn't exit and you don't have a timeout so communicate never returns. You could verify that with a print after the call. Instead, add a timeout and catch its exception
import subprocess
import time
proc = subprocess.Popen(['python', 'test.py', ''], shell=True)
try:
proc.communicate(timeout=10)
except subprocess.TimeoutExpired:
proc.terminate()
First things first, don't use a list to pass in the arguments for the subprocess.Popen() if you're using the shell = True!, change the command to string, "python test.py".
Popen.communicate(input=None, timeout=None) is a blocking class method, it shall Interact with process, and Wait for process to terminate and set the returncode attribute.
since your test.py running infinite while loop, he will never return !
you have 2 options to timeout the process proc that you have spawned:
assign the timeout keyword argument in the,e.g. timing the process for 5 seconds, communicate(timeout=5) method. If the process proc does not terminate after timeout seconds, a TimeoutExpired exception will be raised. Catching this exception and retrying communication will not lose any output (in your case you dont need the child outputs, but i will mention this in the example below). ATTENTION The child process is not killed if the timeout expires, so in order to cleanup properly a well-behaved application should kill the child process (proc) and finish communication.
by using the poll method and do the timing by your calling method.
communicate with timeout
try:
outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
poll with time.sleep
proc = subprocess.Popen('python test.py', shell=True)
t=10
while proc.poll() is None and t >= 0:
print('Still sleeping')
time.sleep(1)
t -= 1
proc.kill()
My goal is to implement a Python 3 method that will support running a system command (using subprocess) following a few requirements:
Running long lasting commands
Live logging of both stdout and stderr
Enforcing a timeout to stop the command if it fails to complete on time
In order to support live logging, I have used 2 threads which handles both stdout and stderr outputs.
My challenge is to enforce the timeout on the threads and the subprocess process.
My attempt to implement the timeout using a signal handler, seems to freeze the interpreter as soon as the handler is called.
What's wrong with my implementation ?
Is there any other way to implement my requirements?
Here is my current implementation attempt:
def run_live_output(cmd, timeout=900, **kwargs):
full_output = StringIO()
def log_popen_pipe(p, log_errors=False):
while p.poll() is None:
output = ''
if log_errors:
output = p.stderr.readline()
log.warning(f"{output}")
else:
output = p.stdout.readline()
log.info(f"{output}")
full_output.write(output)
if p.poll():
log.error(f"{cmd}\n{p.stderr.readline()}")
class MyTimeout(Exception):
pass
def handler(signum, frame):
log.info(f"Signal handler called with signal {signum}")
raise MyTimeout
with subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
universal_newlines=True,
**kwargs
) as sp:
with ThreadPoolExecutor(2) as pool:
try:
signal.signal(signal.SIGALRM, handler)
signal.alarm(timeout)
r1 = pool.submit(log_popen_pipe, sp)
r2 = pool.submit(log_popen_pipe, sp, log_errors=True)
r1.result()
r2.result()
except MyTimeout:
log.info(f"Timed out - Killing the threads and process")
pool.shutdown(wait=True)
sp.kill()
except Exception as e:
log.info(f"{e}")
return full_output.getvalue()
Q-1) My attempt to implement the timeout using a signal handler, seems to freeze the interpreter as soon as the handler is called, What's wrong with my implementation ?
A-1) No your signal handler not freezing, There is freezing but not in the signal handler, signal handler is fine. Your main thread blocked (frozen) when you call pool.shutdown(wait=True). Because your subprocess is still running and you do while p.poll() is None: in the log_popen_pipe func. That's why your main thread will not continue until log_popen_pipe finished.
To solve this issue, we need to remove pool.shutdown(wait=True) and then call the sp.terminate(). I suggest you to use sp.terminate() instead sp.kill() because sp.kill() will send SIGKILL signal which is not preferred until you really need it. In addition that, end of the with ThreadPoolExecutor(2) as pool: statement, pool.shutdown(wait=True) will be called and this will not block you if log_popen_pipe func ended.
In your case log_popen_pipe func will finished if subprocess finished when we do sp.terminate().
Q-2) Is there any other way to implement my requirements?
A-2) Yes there is, you can use Timer class from threading library. Timer class will create 1 thread and this thread will wait for timeout seconds and end of the timeout seconds, this created thread will call sp.terminate func
Here is the code:
from io import StringIO
import signal,subprocess
from concurrent.futures import ThreadPoolExecutor
import logging as log
from threading import Timer
log.root.setLevel(log.INFO)
def run_live_output(cmd, timeout=900, **kwargs):
full_output = StringIO()
def log_popen_pipe(p, log_errors=False):
while p.poll() is None:
output = ''
if log_errors:
output = p.stderr.readline()
log.warning(f"{output}")
else:
output = p.stdout.readline()
log.info(f"{output}")
full_output.write(output)
if p.poll()!=None:
log.error(f"subprocess finished, {cmd}\n{p.stdout.readline()}")
with subprocess.Popen(
cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
universal_newlines=True,
**kwargs
) as sp:
Timer(timeout,sp.terminate).start()
with ThreadPoolExecutor(2) as pool:
try:
r1 = pool.submit(log_popen_pipe, sp)
r2 = pool.submit(log_popen_pipe, sp, log_errors=True)
r1.result()
r2.result()
except Exception as e:
log.info(f"{e}")
return full_output.getvalue()
run_live_output(["python3","...."],timeout=4)
By the way p.poll() will return the returncode of the terminated subprocess. If you want to get output of successfully terminated subprocess, you need to use if p.poll()==0 0 generally means subprocess successfully terminated
I wrote some code to run a script (via a subprocess) and kill the child process after a certain timeout. I'm running a script called "runtime_hang_script.sh" that just contains "./runtime_hang," which runs an infinite loop. I'm also redirecting stdout to a pipe -- I plan to write it to both sys.stdout and to a file (aka I'm trying to implement tee). However, my code hangs after the subprocess times out. Note that this ONLY hangs when running "sh runtime_hang_script.sh" and not "./runtime_hang." Also, this doesn't hang when I try piping directly to a file or when I don't read from the pipe.
I've tried other implementations of creating a timed subprocess, but I keep on getting the same issue. I've even tried raising a signal at the end of the problem -- for some reason, the signal is raised earlier than anticipated, so this doesn't work either. Any help would be appreciated. Thanks in advance!
process = None
def run():
global process
timeout_secs = 5
args = ['sh', 'runtime_hang_script.sh']
sys.stdout.flush()
process = subprocess.Popen(args, stdout=subprocess.PIPE, bufsize=1)
with process.stdout:
for line in iter(process.stdout.readline, b''):
sys.stdout.write(line.decode('utf-8'))
sys.stdout.flush()
process.wait()
proc_thread = threading.Thread(target=run)
proc_thread.start()
proc_thread.join(5)
print(proc_thread.is_alive())
if proc_thread.is_alive():
process.kill()
Assuming you are using Python 3.3 or newer, you can use the timeout argument of the subprocess.communicate() method to implement your 5-second timeout:
import subprocess
import sys
timeout_secs = 5
args = ['sh', 'runtime_hang_script.sh']
process = subprocess.Popen(args, stdout=subprocess.PIPE, bufsize=1)
try:
print("Waiting for data from child process...")
(stdoutData, stderrData) = process.communicate(None, timeout_secs)
print("From child process: stdoutData=[%s] stderrData=[%s]" % (stdoutData, stderrData))
except subprocess.TimeoutExpired:
print("Oops, child process took too long! Now it has to die")
process.kill()
print("Waiting for child process to exit...")
process.wait()
print("Child process exited.")
Note that spawning a child thread isn't necessary with this approach, since the timeout can work directly from the main thread.
I'm running a process with the use of Popen. I need to wait for the process to terminate. I'm checking that the process have terminated through the returncode. When returncode is different from None the process must have terminated. The problem is that when print_output is False the returncode is always None, even when the process have finished running (terminated). This is however not the case when print_output is True. I'm using the following code to run the process:
def run(command, print_output=True):
# code mostly from: http://sharats.me/the-ever-useful-and-neat-subprocess-module.html
from subprocess import Popen, PIPE
from threading import Thread
from queue import Queue, Empty
from time import sleep
io_q = Queue()
def stream_watcher(identifier, stream):
for line in stream:
io_q.put((identifier, line))
if not stream.closed:
stream.close()
with Popen(command, stdout=PIPE, stderr=PIPE, universal_newlines=True) as proc:
if print_output:
Thread(target=stream_watcher, name='stdout-watcher', args=('STDOUT', proc.stdout)).start()
Thread(target=stream_watcher, name='stderr-watcher', args=('STDERR', proc.stderr)).start()
def printer():
while True:
try:
# Block for 1 second.
item = io_q.get(True, 1)
except Empty:
# No output in either streams for a second. Are we done?
if proc.poll() is not None:
break
else:
identifier, line = item
print(identifier + ':', line, end='')
Thread(target=printer, name='printer').start()
while proc.returncode is None:
sleep(2)
proc.poll()
if not proc.returncode == 0:
raise RuntimeError(
'The process call "{}" returned with code {}. The return code is not 0, thus an error '
'occurred.'.format(list(command), proc.returncode))
return proc.stdout, proc.stderr
Any clues to what might cause this problem?
EDIT: Discovered something pretty weird. I'm running the following code:
run(my_command, True)
print('--------done--------')
run(my_command, False)
print('--------done--------')
'--------done--------' is never printed even though run(my_command, False) gets executed.
TL;DR
add popen.wait() after subprocess.Popen()
Explanation Part (sort of)
Python goes too fast and the child process is ended but returncode can't be read
(I don't really know why it does that. Explanations welcome)
Why did I use this:
Shell command execution and get both return code and output (stdout)
def exec_cmd(cmd):
pop = subprocess.Popen(shlex.split(cmd), stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
pop.wait()
return [pop.returncode, pop.communicate()[0]]
Also: please read the .wait warning on the popen page
I'm not sure why it did not work, but I think it has something to do with not closing the streams. The following code works:
def run(command, print_output=True):
from subprocess import Popen, PIPE, STDOUT
from io import StringIO
popen = Popen(command, stdout=PIPE, stderr=STDOUT, universal_newlines=True)
out = StringIO()
for line in popen.stdout:
if print_output:
print(line, end='')
else:
out.write(line)
popen.stdout.close()
return_code = popen.wait()
if not return_code == 0:
raise RuntimeError(
'The process call "{}" returned with code {}. The return code is not 0, thus an error '
'occurred.'.format(list(command), return_code))
stdout_string = out.getvalue()
out.close()
return stdout_string
I run a subprocess using:
p = subprocess.Popen("subprocess",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
This subprocess could either exit immediately with an error on stderr, or keep running. I want to detect either of these conditions - the latter by waiting for several seconds.
I tried this:
SECONDS_TO_WAIT = 10
select.select([],
[p.stdout, p.stderr],
[p.stdout, p.stderr],
SECONDS_TO_WAIT)
but it just returns:
([],[],[])
on either condition. What can I do?
Have you tried using the Popen.Poll() method. You could just do this:
p = subprocess.Popen("subprocess",
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE)
time.sleep(SECONDS_TO_WAIT)
retcode = p.poll()
if retcode is not None:
# process has terminated
This will cause you to always wait 10 seconds, but if the failure case is rare this would be amortized over all the success cases.
Edit:
How about:
t_nought = time.time()
seconds_passed = 0
while(p.poll() is not None and seconds_passed < 10):
seconds_passed = time.time() - t_nought
if seconds_passed >= 10:
#TIMED OUT
This has the ugliness of being a busy wait, but I think it accomplishes what you want.
Additionally looking at the select call documentation again I think you may want to change it as follows:
SECONDS_TO_WAIT = 10
select.select([p.stderr],
[],
[p.stdout, p.stderr],
SECONDS_TO_WAIT)
Since you would typically want to read from stderr, you want to know when it has something available to read (ie the failure case).
I hope this helps.
This is what i came up with. Works when you need and don't need to timeout on thep process, but with a semi-busy loop.
def runCmd(cmd, timeout=None):
'''
Will execute a command, read the output and return it back.
#param cmd: command to execute
#param timeout: process timeout in seconds
#return: a tuple of three: first stdout, then stderr, then exit code
#raise OSError: on missing command or if a timeout was reached
'''
ph_out = None # process output
ph_err = None # stderr
ph_ret = None # return code
p = subprocess.Popen(cmd, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
# if timeout is not set wait for process to complete
if not timeout:
ph_ret = p.wait()
else:
fin_time = time.time() + timeout
while p.poll() == None and fin_time > time.time():
time.sleep(1)
# if timeout reached, raise an exception
if fin_time < time.time():
# starting 2.6 subprocess has a kill() method which is preferable
# p.kill()
os.kill(p.pid, signal.SIGKILL)
raise OSError("Process timeout has been reached")
ph_ret = p.returncode
ph_out, ph_err = p.communicate()
return (ph_out, ph_err, ph_ret)
Here is a nice example:
from threading import Timer
from subprocess import Popen, PIPE
proc = Popen("ping 127.0.0.1", shell=True)
t = Timer(60, proc.kill)
t.start()
proc.wait()
Using select and sleeping doesn't really make much sense. select (or any kernel polling mechanism) is inherently useful for asynchronous programming, but your example is synchronous. So either rewrite your code to use the normal blocking fashion or consider using Twisted:
from twisted.internet.utils import getProcessOutputAndValue
from twisted.internet import reactor
def stop(r):
reactor.stop()
def eb(reason):
reason.printTraceback()
def cb(result):
stdout, stderr, exitcode = result
# do something
getProcessOutputAndValue('/bin/someproc', []
).addCallback(cb).addErrback(eb).addBoth(stop)
reactor.run()
Incidentally, there is a safer way of doing this with Twisted by writing your own ProcessProtocol:
http://twistedmatrix.com/projects/core/documentation/howto/process.html
Python 3.3
import subprocess as sp
try:
sp.check_call(["/subprocess"], timeout=10,
stdin=sp.DEVNULL, stdout=sp.DEVNULL, stderr=sp.DEVNULL)
except sp.TimeoutError:
# timeout (the subprocess is killed at this point)
except sp.CalledProcessError:
# subprocess failed before timeout
else:
# subprocess ended successfully before timeout
See TimeoutExpired docs.
If, as you said in the comments above, you're just tweaking the output each time and re-running the command, would something like the following work?
from threading import Timer
import subprocess
WAIT_TIME = 10.0
def check_cmd(cmd):
p = subprocess.Popen(cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
def _check():
if p.poll()!=0:
print cmd+" did not quit within the given time period."
# check whether the given process has exited WAIT_TIME
# seconds from now
Timer(WAIT_TIME, _check).start()
check_cmd('echo')
check_cmd('python')
The code above, when run, outputs:
python did not quit within the given time period.
The only downside of the above code that I can think of is the potentially overlapping processes as you keep running check_cmd.
This is a paraphrase on Evan's answer, but it takes into account the following :
Explicitly canceling the Timer object : if the Timer interval would be long and the process will exit by its "own will" , this could hang your script :(
There is an intrinsic race in the Timer approach (the timer attempt killing the process just after the process has died and this on Windows will raise an exception).
DEVNULL = open(os.devnull, "wb")
process = Popen("c:/myExe.exe", stdout=DEVNULL) # no need for stdout
def kill_process():
""" Kill process helper"""
try:
process.kill()
except OSError:
pass # Swallow the error
timer = Timer(timeout_in_sec, kill_process)
timer.start()
process.wait()
timer.cancel()