We Keep Coding

seeing garbled output with Popen communicate

Am seeing garbled output with below code. Can someone help on how to synchronize the output?
import subprocess
from subprocess import Popen, PIPE
def do_shell(cmd):
p = Popen('/bin/bash', shell=False, universal_newlines=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
o, e = p.communicate(cmd)
p.wait()
print('Output: ' + o.decode('utf8'))
Is it possible to wait for 5 sec after each command execution passed to communicate?
Note:Here cmd is string of commands (cmd=cmd1\ncmd2\ncmd3\n)

difference between subprocess check_output, Popen, getoutput python

I'm using python 3.6 in Windows, and my aim is to run a cmd command and save the output as a string in a variable.
I'm using subprocess and its objects like check_output, Popen and Communicate, and getoutput. But here is my problem with these:
subprocess.check_output the problem is if the code returns non-zero it raises an exception and I can't read the output, for example, executing the netstat -abcd.
stdout_value = (subprocess.check_output(command, shell=True, stdin=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=self.timeout)).decode()
subprocess.Popen and communicate() the problem is some commands like netstat -abcd returns empty from communicate().
self.process = subprocess.Popen(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
stdin=subprocess.PIPE)
try:
self.process.wait(timeout=5)
stdout_value = self.process.communicate()[0]
except:
self.process.kill()
self.process.wait()
subprocess.getoutput(Command) is ok but there is no timeout so my code would block forever on executing some commands like netstat. I also tried to run it as a thread but the code is blocking and I can't stop the thread itself.
stdout_value = subprocess.getoutput(command)
What I want is to run any cmd commands (blocking like netstat or nonblocking like dir) with timeout for example if the user executes netstat it only shows the lines generated in timeout and then kills it.
Thanks.
EDIT------
According to Jean's answer, I rewrote the code but the timeout doesn't work in running some commands like netstat.
# command = "netstat"
command = "test.exe" # running an executable program can't be killed after timeout
self.process = subprocess.run(command, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
stdin=subprocess.PIPE,
timeout=3,
universal_newlines=True
)
stdout_value = self.process.stdout

subprocess.run() with timeout doesn't seem to run properly on Windows.
You can try running the subprocess within a Timer-thread or in case of you dont need communicate(), you can do something like this:
import time
import subprocess
#cmd = 'cmd /c "dir c:\\ /s"'
#cmd = ['calc.exe']
cmd = ['ping', '-n', '25', 'www.google.com']
#_stdout = open('C:/temp/stdout.txt', 'w')
#_stderr = open('C:/temp/stderr.txt', 'w')
_stdout = subprocess.PIPE
_stderr = subprocess.PIPE
proc = subprocess.Popen(cmd, bufsize=0, stdout=_stdout, stderr=_stderr)
_startTime = time.time()
while proc.poll() is None and proc.returncode is None:
if (time.time() - _startTime) >= 5:
print ("command ran for %.6f seconds" % (time.time() - _startTime))
print ("timeout - killing process!")
proc.kill()
break
print (proc.stdout.read())
It works for all three commands on Win7/py3.6, but not for the 'killed-netstat' issue!

cannot kill a Sub process created by Popen when printing process.stdout

I have created a script which should run a command and kill it after 15 seconds
import logging
import subprocess
import time
import os
import sys
import signal
#cmd = "ping 192.168.1.1 -t"
cmd = "C:\\MyAPP\MyExe.exe -t 80 -I C:\MyApp\Temp -M Documents"
proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,shell=True)
**for line in proc.stdout:
print (line.decode("utf-8"), end='')**
time.sleep(15)
os.kill(proc.pid, signal.SIGTERM)
#proc.kill() #Tried this too but no luck
This doesnot terminate my subprocess. however if I comment out the logging to stdout part, ie
for line in proc.stdout:
print (line.decode("utf-8"), end='')
the subprocess has been killed.
I have tried proc.kill() and CTRL_C_EVENT too but no luck.
Any help would be highly appreciated. Please see me as novice to python

To terminate subprocess in 15 seconds while printing its output line-by-line:
#!/usr/bin/env python
from __future__ import print_function
from threading import Timer
from subprocess import Popen, PIPE, STDOUT
# start process
cmd = r"C:\MyAPP\MyExe.exe -t 80 -I C:\MyApp\Temp -M Documents"
process = Popen(cmd, stdout=PIPE, stderr=STDOUT,
bufsize=1, universal_newlines=True)
# terminate process in 15 seconds
timer = Timer(15, terminate, args=[process])
timer.start()
# print output
for line in iter(process.stdout.readline, ''):
print(line, end='')
process.stdout.close()
process.wait() # wait for the child process to finish
timer.cancel()
Notice, you don't need shell=True here. You could define terminate() as:
def terminate(process):
if process.poll() is None:
try:
process.terminate()
except EnvironmentError:
pass # ignore
If you want to kill the whole process tree then define terminate() as:
from subprocess import call
def terminate(process):
if process.poll() is None:
call('taskkill /F /T /PID ' + str(process.pid))
Use raw-string literals for Windows paths: r"" otherwise you should escape all backslashes in the string literal
Drop shell=True. It creates an additional process for no reason here
universal_newlines=True enables text mode (bytes are decode into Unicode text using the locale preferred encoding automatically on Python 3)
iter(process.stdout.readline, '') is necessary for compatibility with Python 2 (otherwise the data may be printed with a delay due to the read-ahead buffer bug)
Use process.terminate() instead of process.send_signal(signal.SIGTERM) or os.kill(proc.pid, signal.SIGTERM)
taskkill allows to kill a process tree on Windows

The problem is reading from stdout is blocking. You need to either read the subprocess's output or run the timer on a separate thread.
from subprocess import Popen, PIPE
from threading import Thread
from time import sleep
class ProcKiller(Thread):
def __init__(self, proc, time_limit):
super(ProcKiller, self).__init__()
self.proc = proc
self.time_limit = time_limit
def run(self):
sleep(self.time_limit)
self.proc.kill()
p = Popen('while true; do echo hi; sleep 1; done', shell=True)
t = ProcKiller(p, 5)
t.start()
p.communicate()
EDITED to reflect suggested changes in comment
from subprocess import Popen, PIPE
from threading import Thread
from time import sleep
from signal import SIGTERM
import os
class ProcKiller(Thread):
def __init__(self, proc, time_limit):
super(ProcKiller, self).__init__()
self.proc = proc
self.time_limit = time_limit
def run(self):
sleep(self.time_limit)
os.kill(self.proc.pid, SIGTERM)
p = Popen('while true; do echo hi; sleep 1; done', shell=True)
t = ProcKiller(p, 5)
t.start()
p.communicate()

subprocess.communicate() hangs on Windows 8 if parent process creates some child

I have a simple code which works fine on Win 2003:
proc = subprocess.Popen('<some python script which runs another process>', stdout = subprocess.PIPE, stderr = subprocess.PIPE, stdin = subprocess.PIPE)
out = proc.communicate()[0]
But on Windows 8 this part; out = proc.communicate()[0], hangs.
Have anybody seeen this issue?
I've checked that process is really ternimated (PID is absent when child process has been started)
It's also a problem to make proc.stdout.readlines(), it hangs too. How to check that stdout has EOF?
When I stop child process proc.communicate() works fine.
Here is the simplest example:
import subprocess
proc = subprocess.Popen([sys.executable, "D:\\test.py"], stdout = subprocess.PIPE)
print 'PID', proc.pid #When this PID is printed I see in the taskbr that process is already finished
print 'Output', proc.communicate() # but this part is hangs
And code od test.py:
import os, time
from subprocess import Popen, PIPE
CREATE_NEW_PROCESS_GROUP = 0x00000200 # note: could get it from subprocess
DETACHED_PROCESS = 0x00000008 # 0x8 | 0x200 == 0x208
p = Popen("start /B notepad", shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE,
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)
print 'Done'
exit()
Is it valid scenario?

To allow .communicate() to return without waiting for the grandchild (notepad) to exit, you could try in test.py:
import sys
from subprocess import Popen, PIPE
CREATE_NEW_PROCESS_GROUP = 0x00000200
DETACHED_PROCESS = 0x00000008
p = Popen('grandchild', stdin=PIPE, stdout=PIPE, stderr=PIPE,
creationflags=DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP)
See Popen waiting for child process even when the immediate child has terminated.

Popen stdout reading pipe, deadlock using sleep

Well, I have two scripts. The a.py which prints the output of the b.py script as follows:
#a.py
from subprocess import Popen, PIPE, STDOUT
p = Popen(['/Users/damian/Desktop/b.py'], shell=False, stdout=PIPE, stderr=STDOUT)
while p.poll() is None:
print p.stdout.readline()
#b.py
#!/usr/bin/env python
import time
while 1:
print 'some output'
#time.sleep(1)
This works.But,
Why do my scripts deadlock when I uncomment the time.sleep() line?

Your output is probably buffered. Add a .flush() for stdout to clear it:
import sys
import time
while 1:
print 'someoutput'
sys.stdout.flush()
time.sleep(1)

If you add -u to the call in a.py (make the output unbuffered) then you don't need to modify b.py script:
import sys
from subprocess import Popen, PIPE, STDOUT
p = Popen([sys.executable, '-u', '/Users/damian/Desktop/b.py'],
stdout=PIPE, stderr=STDOUT, close_fds=True)
for line in iter(p.stdout.readline, ''):
print line,
p.stdout.close()
if p.wait() != 0:
raise RuntimeError("%r failed, exit status: %d" % (cmd, p.returncode))
See more ways to get output from a subprocess.