I have a simple code which works fine on Win 2003:
proc = subprocess.Popen('<some python script which runs another process>', stdout = subprocess.PIPE, stderr = subprocess.PIPE, stdin = subprocess.PIPE)
out = proc.communicate()[0]
But on Windows 8 this part; out = proc.communicate()[0], hangs.
Have anybody seeen this issue?
I've checked that process is really ternimated (PID is absent when child process has been started)
It's also a problem to make proc.stdout.readlines(), it hangs too. How to check that stdout has EOF?
When I stop child process proc.communicate() works fine.
Here is the simplest example:
import subprocess
proc = subprocess.Popen([sys.executable, "D:\\test.py"], stdout = subprocess.PIPE)
print 'PID', proc.pid #When this PID is printed I see in the taskbr that process is already finished
print 'Output', proc.communicate() # but this part is hangs
And code od test.py:
import os, time
from subprocess import Popen, PIPE
CREATE_NEW_PROCESS_GROUP = 0x00000200 # note: could get it from subprocess
DETACHED_PROCESS = 0x00000008 # 0x8 | 0x200 == 0x208
p = Popen("start /B notepad", shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE,
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)
print 'Done'
exit()
Is it valid scenario?
To allow .communicate() to return without waiting for the grandchild (notepad) to exit, you could try in test.py:
import sys
from subprocess import Popen, PIPE
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
p = Popen('grandchild', stdin=PIPE, stdout=PIPE, stderr=PIPE,
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)
See Popen waiting for child process even when the immediate child has terminated.
Related
Am seeing garbled output with below code. Can someone help on how to synchronize the output?
import subprocess
from subprocess import Popen, PIPE
def do_shell(cmd):
p = Popen('/bin/bash', shell=False, universal_newlines=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
o, e = p.communicate(cmd)
p.wait()
print('Output: ' + o.decode('utf8'))
Is it possible to wait for 5 sec after each command execution passed to communicate?
Note:Here cmd is string of commands (cmd=cmd1\ncmd2\ncmd3\n)
from subprocess import Popen, PIPE, STDOUT, TimeoutExpired
cmd = 'for x in $(seq 1 3); do echo stage $x; sleep 1; done'
proc = Popen(cmd, shell=True, close_fds=True,
stdout=PIPE, stderr=STDOUT,
universal_newlines=True, start_new_session=True)
for line in proc.stdout:
print(line.strip())
Default behavior of communicate is to spit out all stdout after the proc. is terminated. However I need access to stdout while proc. is still running, but terminate it, if it's running too long.
EDIT
The following seem to work the way I want, not sure if that's correct approach?
from os import killpg
from signal import SIGKILL
from concurrent import futures
from subprocess import Popen, PIPE, STDOUT, TimeoutExpired
cmd = 'for x in $(seq 1 3); do echo stage $x; sleep 1; done'
proc = Popen(cmd, shell=True, close_fds=True,
stdout=PIPE, stderr=STDOUT,
universal_newlines=True, start_new_session=True)
def handler(proc, timeout=None):
try:
proc.wait(timeout)
except TimeoutExpired as err:
print(err)
killpg(proc.pid, SIGKILL)
exe = futures.ThreadPoolExecutor(max_workers=1)
exe.submit(handler, proc, 2)
for line in proc.stdout:
print(line.strip())
I have created a script which should run a command and kill it after 15 seconds
import logging
import subprocess
import time
import os
import sys
import signal
#cmd = "ping 192.168.1.1 -t"
cmd = "C:\\MyAPP\MyExe.exe -t 80 -I C:\MyApp\Temp -M Documents"
proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,shell=True)
**for line in proc.stdout:
print (line.decode("utf-8"), end='')**
time.sleep(15)
os.kill(proc.pid, signal.SIGTERM)
#proc.kill() #Tried this too but no luck
This doesnot terminate my subprocess. however if I comment out the logging to stdout part, ie
for line in proc.stdout:
print (line.decode("utf-8"), end='')
the subprocess has been killed.
I have tried proc.kill() and CTRL_C_EVENT too but no luck.
Any help would be highly appreciated. Please see me as novice to python
To terminate subprocess in 15 seconds while printing its output line-by-line:
#!/usr/bin/env python
from __future__ import print_function
from threading import Timer
from subprocess import Popen, PIPE, STDOUT
# start process
cmd = r"C:\MyAPP\MyExe.exe -t 80 -I C:\MyApp\Temp -M Documents"
process = Popen(cmd, stdout=PIPE, stderr=STDOUT,
bufsize=1, universal_newlines=True)
# terminate process in 15 seconds
timer = Timer(15, terminate, args=[process])
timer.start()
# print output
for line in iter(process.stdout.readline, ''):
print(line, end='')
process.stdout.close()
process.wait() # wait for the child process to finish
timer.cancel()
Notice, you don't need shell=True here. You could define terminate() as:
def terminate(process):
if process.poll() is None:
try:
process.terminate()
except EnvironmentError:
pass # ignore
If you want to kill the whole process tree then define terminate() as:
from subprocess import call
def terminate(process):
if process.poll() is None:
call('taskkill /F /T /PID ' + str(process.pid))
Use raw-string literals for Windows paths: r"" otherwise you should escape all backslashes in the string literal
Drop shell=True. It creates an additional process for no reason here
universal_newlines=True enables text mode (bytes are decode into Unicode text using the locale preferred encoding automatically on Python 3)
iter(process.stdout.readline, '') is necessary for compatibility with Python 2 (otherwise the data may be printed with a delay due to the read-ahead buffer bug)
Use process.terminate() instead of process.send_signal(signal.SIGTERM) or os.kill(proc.pid, signal.SIGTERM)
taskkill allows to kill a process tree on Windows
The problem is reading from stdout is blocking. You need to either read the subprocess's output or run the timer on a separate thread.
from subprocess import Popen, PIPE
from threading import Thread
from time import sleep
class ProcKiller(Thread):
def __init__(self, proc, time_limit):
super(ProcKiller, self).__init__()
self.proc = proc
self.time_limit = time_limit
def run(self):
sleep(self.time_limit)
self.proc.kill()
p = Popen('while true; do echo hi; sleep 1; done', shell=True)
t = ProcKiller(p, 5)
t.start()
p.communicate()
EDITED to reflect suggested changes in comment
from subprocess import Popen, PIPE
from threading import Thread
from time import sleep
from signal import SIGTERM
import os
class ProcKiller(Thread):
def __init__(self, proc, time_limit):
super(ProcKiller, self).__init__()
self.proc = proc
self.time_limit = time_limit
def run(self):
sleep(self.time_limit)
os.kill(self.proc.pid, SIGTERM)
p = Popen('while true; do echo hi; sleep 1; done', shell=True)
t = ProcKiller(p, 5)
t.start()
p.communicate()
I'm rather puzzled by why the code below doesn't print stdout and exit, instead it hangs (on windows). Any reason why?
import subprocess
from subprocess import Popen
def main():
proc = Popen(
'C:/Python33/python.exe',
stderr=subprocess.STDOUT,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE
)
proc.stdin.write(b'exit()\r\n')
proc.stdin.flush()
print(proc.stdout.read(1))
if __name__=='__main__':
main()
Replace the following:
proc.stdin.flush()
with:
proc.stdin.close()
Otherwise the subprocess python.exe will wait forever stdin to be closed.
Alternative: using communicate()
proc = Popen(...)
out, err = proc.communicate(b'exit()\r\n')
print(out) # OR print(out[:1]) if you want only the first byte to be print.
I have a script that has been working properly for the past 3 months. The Server went down last Monday and since then my script stopped working. The script hangs at coords = p.communicate()[0].split().
Here's a part of the script:
class SelectByLatLon(GridSelector):
def __init__(self, from_lat, to_lat, from_lon, to_lon):
self.from_lat = from_lat
self.to_lat = to_lat
self.from_lon = from_lon
self.to_lon = to_lon
def get_selection(self, file):
p = subprocess.Popen(
[
os.path.join(module_root, 'bin/points_from_latlon.tcl'),
file,
str(self.from_lat), str(self.to_lat), str(self.from_lon), str(self.to_lon)
],
stdout = subprocess.PIPE
)
coords = p.communicate()[0].split()
return ZGridSelection(int(coords[0]), int(coords[1]), int(coords[2]), int(coords[3]))
When I run the script on another server everything works just fine.
Can I use something else instead of p.communicate()[0].split() ?
You might have previously run your server without daemonization i.e., you had functional stdin, stdout, stderr streams. To fix, you could redirect the streams to DEVNULL for the subprocess:
import os
from subprocess import Popen, PIPE
DEVNULL = os.open(os.devnull, os.O_RDWR)
p = Popen(tcl_cmd, stdin=DEVNULL, stdout=PIPE, stderr=DEVNULL, close_fds=True)
os.close(DEVNULL)
.communicate() may wait for EOF on stdout even if tcl_cmd already exited: the tcl script might have spawned a child process that inherited the standard streams and outlived its parent.
If you know that you don't need any stdout after the tcl_cmd exits then you could kill the whole process tree when you detect that tcl_cmd is done.
You might need start_new_session=True analog to be able to kill the whole process tree:
import os
import signal
from threading import Timer
def kill_tree_on_exit(p):
p.wait() # wait for tcl_cmd to exit
os.killpg(p.pid, signal.SIGTERM)
t = Timer(0, kill_tree_on_exit, [p])
t.start()
coords = p.communicate()[0].split()
t.cancel()
See How to terminate a python subprocess launched with shell=True