I'm using python 3.6 in Windows, and my aim is to run a cmd command and save the output as a string in a variable.
I'm using subprocess and its objects like check_output, Popen and Communicate, and getoutput. But here is my problem with these:
subprocess.check_output the problem is if the code returns non-zero it raises an exception and I can't read the output, for example, executing the netstat -abcd.
stdout_value = (subprocess.check_output(command, shell=True, stdin=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=self.timeout)).decode()
subprocess.Popen and communicate() the problem is some commands like netstat -abcd returns empty from communicate().
self.process = subprocess.Popen(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
stdin=subprocess.PIPE)
try:
self.process.wait(timeout=5)
stdout_value = self.process.communicate()[0]
except:
self.process.kill()
self.process.wait()
subprocess.getoutput(Command) is ok but there is no timeout so my code would block forever on executing some commands like netstat. I also tried to run it as a thread but the code is blocking and I can't stop the thread itself.
stdout_value = subprocess.getoutput(command)
What I want is to run any cmd commands (blocking like netstat or nonblocking like dir) with timeout for example if the user executes netstat it only shows the lines generated in timeout and then kills it.
Thanks.
EDIT------
According to Jean's answer, I rewrote the code but the timeout doesn't work in running some commands like netstat.
# command = "netstat"
command = "test.exe" # running an executable program can't be killed after timeout
self.process = subprocess.run(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
timeout=3,
universal_newlines=True
)
stdout_value = self.process.stdout
subprocess.run() with timeout doesn't seem to run properly on Windows.
You can try running the subprocess within a Timer-thread or in case of you dont need communicate(), you can do something like this:
import time
import subprocess
#cmd = 'cmd /c "dir c:\\ /s"'
#cmd = ['calc.exe']
cmd = ['ping', '-n', '25', 'www.google.com']
#_stdout = open('C:/temp/stdout.txt', 'w')
#_stderr = open('C:/temp/stderr.txt', 'w')
_stdout = subprocess.PIPE
_stderr = subprocess.PIPE
proc = subprocess.Popen(cmd, bufsize=0, stdout=_stdout, stderr=_stderr)
_startTime = time.time()
while proc.poll() is None and proc.returncode is None:
if (time.time() - _startTime) >= 5:
print ("command ran for %.6f seconds" % (time.time() - _startTime))
print ("timeout - killing process!")
proc.kill()
break
print (proc.stdout.read())
It works for all three commands on Win7/py3.6, but not for the 'killed-netstat' issue!
Related
I'm trying to launch a process with root privileges and kill it later on.
But for some reason, I can't get it to work.
Here is a small script to reproduce my problem (disclaimer: code is a bit dirty its only for bug reproduction):
import os
import time
import subprocess
command = ["sudo", "sleep", "25"]
process = subprocess.Popen(command,
bufsize=1,
stdin=open(os.devnull),
stderr=subprocess.PIPE,
stdout=subprocess.PIPE)
def kill():
pid = process.pid
cmd = "sudo kill %s" % pid
print(cmd)
print(os.system(cmd))
time.sleep(2)
kill()
stdout, stderr = process.communicate()
print("stdout: " + stdout)
print("stderr: " + stderr)
ret = process.wait()
print("ret: " + str(ret))
This code doesn't seem to be able to kill my subprocess, but when I launch os.system("sudo kill <pid>") in another python instance, it does work.
Problem here in your code is, that it does not close thread in kill function
Function kill does kill your subprocess command. But it does not end thread.
Note: Use -9 if you want to force fully kill the process.
Solution to your problem is. use process.wait() (this will close your thread) in your kill function.
def kill():
pid = process.pid
cmd = "sudo kill -9 %s" % pid . # -9 to kill force fully
print(cmd)
print(os.system(cmd))
print(process.wait()) # this will print -9 if killed force fully, else -15.
You may try this one too. Here what I tried to do is setting a session id to the group of processes that may get created during the subprocess call and when you want kill, a signal is sent to the process group leader, it's transmitted to all of the child processes of this group.
import signal
process = subprocess.Popen(command,
stderr=subprocess.PIPE,
stdout=subprocess.PIPE,
preexec_fn=os.setsid) # add session id to group
print(process.pid)
def kill():
pid = process.pid
cmd = "sudo kill %s" % pid
print(cmd)
os.killpg(os.getpgid(process.pid), signal.SIGTERM) # send signal to the group
time.sleep(2)
kill()
I want to interact with a process.
I can start the process and print out the first two lines (something like 'process successfully started').
Now I want to send a new command to the process which should return again something like 'command done' but nothing happens.
Please help me.
import subprocess
def PrintAndPraseOutput(output, p):
print(output)
if 'sucessfully' in output:
p.stdin.write('command')
cmd = ["./programm"]
p = subprocess.Popen(cmd, universal_newlines=True, shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
while p.poll() is None:
output = p.stdout.readline()
PrintAndPraseOutput(output, p)
Update:
same problem, no output after 'process successfully started'
import subprocess
def print_and_parse_output(output, p):
print(output)
if 'successfully' in output:
p.stdin.write('command\n')
with subprocess.Popen(["./programm"], universal_newlines=True, shell=False, stdout=subprocess.PIPE, stdin=subprocess.PIPE) as proc:
while proc.poll() is None:
output = proc.stdout.readline()
print_and_parse_output(output, proc)
Your I/O should be line buffered, so PrintAndPraseOutput should send a '\n' at the end of the string.
BTW, you have a couple of spelling errors. That function should be named print_and_parse_output to conform to PEP-0008, and "successfully" has 2 c's.
def print_and_parse_output(output, p):
print(output)
if 'successfully' in output:
p.stdin.write('command\n')
When using subprocess like this it's a good idea to put it in a with statement. From the subprocess.Popen` docs:
Popen objects are supported as context managers via the with
statement: on exit, standard file descriptors are closed, and the
process is waited for.
with Popen(["ifconfig"], stdout=PIPE) as proc:
log.write(proc.stdout.read())
I'm launching a subprocess with the following command:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
However, when I try to kill using:
p.terminate()
or
p.kill()
The command keeps running in the background, so I was wondering how can I actually terminate the process.
Note that when I run the command with:
p = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
It does terminate successfully when issuing the p.terminate().
Use a process group so as to enable sending a signal to all the process in the groups. For that, you should attach a session id to the parent process of the spawned/child processes, which is a shell in your case. This will make it the group leader of the processes. So now, when a signal is sent to the process group leader, it's transmitted to all of the child processes of this group.
Here's the code:
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(os.getpgid(pro.pid), signal.SIGTERM) # Send the signal to all the process groups
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
p.kill()
p.kill() ends up killing the shell process and cmd is still running.
I found a convenient fix this by:
p = subprocess.Popen("exec " + cmd, stdout=subprocess.PIPE, shell=True)
This will cause cmd to inherit the shell process, instead of having the shell launch a child process, which does not get killed. p.pid will be the id of your cmd process then.
p.kill() should work.
I don't know what effect this will have on your pipe though.
If you can use psutil, then this works perfectly:
import subprocess
import psutil
def kill(proc_pid):
process = psutil.Process(proc_pid)
for proc in process.children(recursive=True):
proc.kill()
process.kill()
proc = subprocess.Popen(["infinite_app", "param"], shell=True)
try:
proc.wait(timeout=3)
except subprocess.TimeoutExpired:
kill(proc.pid)
I could do it using
from subprocess import Popen
process = Popen(command, shell=True)
Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))
it killed the cmd.exe and the program that i gave the command for.
(On Windows)
When shell=True the shell is the child process, and the commands are its children. So any SIGTERM or SIGKILL will kill the shell but not its child processes, and I don't remember a good way to do it.
The best way I can think of is to use shell=False, otherwise when you kill the parent shell process, it will leave a defunct shell process.
None of these answers worked for me so Im leaving the code that did work. In my case even after killing the process with .kill() and getting a .poll() return code the process didn't terminate.
Following the subprocess.Popen documentation:
"...in order to cleanup properly a well-behaved application should kill the child process and finish communication..."
proc = subprocess.Popen(...)
try:
outs, errs = proc.communicate(timeout=15)
except TimeoutExpired:
proc.kill()
outs, errs = proc.communicate()
In my case I was missing the proc.communicate() after calling proc.kill(). This cleans the process stdin, stdout ... and does terminate the process.
As Sai said, the shell is the child, so signals are intercepted by it -- best way I've found is to use shell=False and use shlex to split the command line:
if isinstance(command, unicode):
cmd = command.encode('utf8')
args = shlex.split(cmd)
p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Then p.kill() and p.terminate() should work how you expect.
Send the signal to all the processes in group
self.proc = Popen(commands,
stdout=PIPE,
stderr=STDOUT,
universal_newlines=True,
preexec_fn=os.setsid)
os.killpg(os.getpgid(self.proc.pid), signal.SIGHUP)
os.killpg(os.getpgid(self.proc.pid), signal.SIGTERM)
There is a very simple way for Python 3.5 or + (Actually tested on Python 3.8)
import subprocess, signal, time
p = subprocess.Popen(['cmd'], shell=True)
time.sleep(5) #Wait 5 secs before killing
p.send_signal(signal.CTRL_C_EVENT)
Then, your code may crash at some point if you have a keyboard input detection, or sth like this. In this case, on the line of code/function where the error is given, just use:
try:
FailingCode #here goes the code which is raising KeyboardInterrupt
except KeyboardInterrupt:
pass
What this code is doing is just sending a "CTRL+C" signal to the running process, what will cause the process to get killed.
Solution that worked for me
if os.name == 'nt': # windows
subprocess.Popen("TASKKILL /F /PID {pid} /T".format(pid=process.pid))
else:
os.kill(process.pid, signal.SIGTERM)
Full blown solution that will kill running process (including subtree) on timeout reached or specific conditions via a callback function.
Works both on windows & Linux, from Python 2.7 up to 3.10 as of this writing.
Install with pip install command_runner
Example for timeout:
from command_runner import command_runner
# Kills ping after 2 seconds
exit_code, output = command_runner('ping 127.0.0.1', shell=True, timeout=2)
Example for specific condition:
Here we'll stop ping if current system time seconds digit is > 5
from time import time
from command_runner import command_runner
def my_condition():
# Arbitrary condition for demo
return True if int(str(int(time()))[-1]) > 5
# Calls my_condition() every second (check_interval) and kills ping if my_condition() returns True
exit_code, output = command_runner('ping 127.0.0.1', shell=True, stop_on=my_condition, check_interval=1)
I'm rather puzzled by why the code below doesn't print stdout and exit, instead it hangs (on windows). Any reason why?
import subprocess
from subprocess import Popen
def main():
proc = Popen(
'C:/Python33/python.exe',
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE
)
proc.stdin.write(b'exit()\r\n')
proc.stdin.flush()
print(proc.stdout.read(1))
if __name__=='__main__':
main()
Replace the following:
proc.stdin.flush()
with:
proc.stdin.close()
Otherwise the subprocess python.exe will wait forever stdin to be closed.
Alternative: using communicate()
proc = Popen(...)
out, err = proc.communicate(b'exit()\r\n')
print(out) # OR print(out[:1]) if you want only the first byte to be print.
I'm running into some difficulties getting output from a subprocess stdout pipe. I'm launching some third party code via it, in order to extract log output. Up until a recent update of the third party code, everything worked fine. After the update, python has started blocking indefinitely, and not actually showing any output. I can manually launch the third party app fine and see output.
A basic version of the code I'm using:
import subprocess, time
from threading import Thread
def enqueue_output(out):
print "Hello from enqueue_output"
for line in iter(out.readline,''):
line = line.rstrip("\r\n")
print "Got %s" % line
out.close()
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
thread = Thread(target=enqueue_output, args=(proc.stdout,))
thread.daemon = True
thread.start()
time.sleep(30)
This works perfectly if I substitute third_party.exe for this script:
import time, sys
while True:
print "Test"
sys.stdout.flush()
time.sleep(1)
So I'm unclear as to magic needs to be done to get this working with the original command.
These are all variants of the subprocess.Popen line I've tried with no success:
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=0)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, shell=True)
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, creationflags=subprocess.CREATE_NEW_CONSOLE)
si = subprocess.STARTUPINFO()
si.dwFlags = subprocess.STARTF_USESTDHANDLES | subprocess.STARTF_USESHOWWINDOW
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, startupinfo=si)
Edit 1:
I can't actually use .communicate() in this case. The app I'm launching remains running for long periods of time (days to weeks). The only way I could actually test .communicate() would be to kill the app shortly after it launches, which I don't feel would give me valid results.
Even the non-threaded version of this fails:
import subprocess, time
from threading import Thread
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print "App started, reading output..."
for line in iter(proc.stdout.readline,''):
line = line.rstrip("\r\n")
print "Got: %s" % line
Edit 2:
Thanks to jdi, the following works okay:
import tempfile, time, subprocess
w = "test.txt"
f = open("test.txt","a")
p = subprocess.Popen("third_party.exe", shell=True, stdout=f,
stderr=subprocess.STDOUT, bufsize=0)
time.sleep(30)
with open("test.txt", 'r') as r:
for line in r:
print line
f.close()
First I would recommend that you simplify this example to make sure you can actually read anything. Remove the complication of the thread from the mix:
proc = subprocess.Popen("third_party.exe", stdout=subprocess.PIPE, bufsize=1)
print proc.communicate()
If that works, great. Then you are having problems possibly with how you are reading the stdout directly or possibly in your thread.
If this does not work, have you tried piping stderr to stdout as well?
proc = subprocess.Popen("third_party.exe",
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, bufsize=1)
Update
Since you say communicate() is deadlocking, here is another approach you can try to see if its a problem with the internal buffer of subprocess...
import tempfile
import subprocess
w = tempfile.NamedTemporaryFile()
p = subprocess.Popen('third_party.exe', shell=True, stdout=w,
stderr=subprocess.STDOUT, bufsize=0)
with open(w.name, 'r') as r:
for line in r:
print line
w.close()
args = ['svn','log','-v']
def foo(info=''):
import logging
import subprocess
import tempfile
try:
pipe = subprocess.Popen(args,bufsize = 0,\
stdout = subprocess.PIPE,\
stderr=subprocess.STDOUT)
except Exception as e:
logging.error(str(e))
return False
while 1:
s = pipe.stdout.read()
if s:
print s,
if pipe.returncode is None:
pipe.poll()
else:
break
if not 0 == pipe.returncode:
return False
return True
print foo()
This one should works,not thread,temp file magic.