We Keep Coding

Live output from Python subprocess that calls Python script

There's quite a bit of posts related to collecting live output from a process that was launched using Python's subprocess module. When I attempt these solutions between my two test scripts, one being a (ba)sh script and the other being a Python script, the Python script fails to have its output read live. Instead when the Python script is ran by subprocess it ends up waiting until the process has completed to flush it to PIPE. The constraints I'm bounded by is that I do want a way to retrieve live output from subprocess for the Python script.
Tested on Ubuntu 20.04 & Windows, Shell script ran on Ubuntu 20.04.
Calling code:
import shlex
import subprocess
# invoke process
process = subprocess.Popen('python test.py',shell=True,stdout=subprocess.PIPE) #Shell true/false results in "not live" output
# Poll process.stdout to show stdout live
while True:
output = process.stdout.readline() # <-- Hangs here on calling test.py, doesn't hang on test.sh
if process.poll() is not None:
break
if output:
print(output.strip())
rc = process.poll()
test.py <-- Waits until it has completed to print out entire output
import time
for x in range(10):
print(x)
time.sleep(1)
test.sh <-- Prints out live in Python script
#!/bin/bash
for i in $(seq 1 5); do
echo "iteration" $i
sleep 1
done

#stochastic13 Provided a very useful link where the -u switch and PYTHONUNBUFFERED variable being set would work. For my needs, I used PYTHONUNBUFFERED which solved my issue entirely. The Python test script actually executes another Python script to run, which I needed the output on. Despite -u helping for the first script, it wouldn't help for the second as I wouldn't have direct access to said script to add the argument. Instead I went with the environment variable, solution below:
def run_command(command):
os.environ['PYTHONUNBUFFERED'] = '1'
process = Popen(command, shell=False, stdout=PIPE, env=os.environ) # Shell doesn't quite matter for this issue
while True:
output = process.stdout.readline()
if process.poll() is not None:
break
if output:
print(output)
rc = process.poll()
return rc
Above the code passes PYTHONUNBUFFERED and sets it to the environment, any spawned process in subprocess with this environment set will inherit PYTHONUNBUFFERED.
Test Script
import subprocess
from io import TextIOWrapper, TextIOBase, StringIO
from subprocess import PIPE, Popen, call
from tempfile import TemporaryFile
from sarge import run, Capture
# process = Popen('python test2.py', shell=False)
# while True:
# if process.poll() is not None:
# break
# rc = process.poll()
subprocess.call('python test2.py')
Test Script 2
import time
import os
print(list(os.environ.keys()))
for x in range(10):
print('test2', x)
time.sleep(1)
The output is a live capture of stdout from any Python process, not just after completion.
...
b'test2 0\r\n'
b'test2 1\r\n'
b'test2 2\r\n'
b'test2 3\r\n'
...
0

Process hangs when communicating with an interactive program

I have created a simple echo.py like this:
import sys
while True:
s = sys.stdin.readline()
s = s.strip('\n')
if s == 'exit':
break
sys.stdout.write("You typed: %s\n" % s)
It works well on the terminal.
And another program to interact with echo.py named main.py
import subprocess
if __name__ == '__main__':
proc = subprocess.Popen(['python', 'echo.py'],stdin=subprocess.PIPE,
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
proc.stdin.write(b'Hello\n')
proc.stdin.flush()
print(proc.stdout.readline())
proc.stdin.write(b'Hello World\n')
proc.stdin.flush()
print(proc.stdout.readline())
proc.terminate()
The main.py just hangs forever.
The thing is if I create subprocess with ['python', '-i'], it works.

Add sys.stdout.flush() to echo.py. Buffering works differently if you run the process with Popen instead of the terminal.

I believe the problem is the while loop.. You're opening a subprocess, writing to it and flushing, and it does all that, but never finishes readline() because of the loop.

Can I close the CMD window that opened with subprocess.Popen in Python?

I have a program that need to run small tasks in new CMDs.
For example:
def main()
some code
...
proc = subprocess.Popen("start.bat")
some code...
proc.kill()
subprocess,Popen opens a new cmd window and runs "start.bat" in it.
proc.kill() kills the process but doesn't close the cmd window.
Is there a way to close this cmd window?
I thought about naming the opened cmd window so i can kill it with the command:
/taskkill /f /im cmdName.exe
Is it possible ?if no, What do you suggest ?
Edit, Added Minimal, Complete, and Verifiable example:
a.py:
import subprocess,time
proc = subprocess.Popen("c.bat",creationflags=subprocess.CREATE_NEW_CONSOLE)
time.sleep(5)
proc.kill()
b.py
while True:
print("IN")
c.bat
python b.py

that's expected when a subprocess is running. You're just killing the .bat process.
You can use psutil (third party, use pip install psutil to install) to compute the child processes & kill them, like this:
import subprocess,time,psutil
proc = subprocess.Popen("c.bat",creationflags=subprocess.CREATE_NEW_CONSOLE)
time.sleep(5)
pobj = psutil.Process(proc.pid)
# list children & kill them
for c in pobj.children(recursive=True):
c.kill()
pobj.kill()
tested with your example, the window closes after 5 seconds

here is another way you can do it
import subprocess
from subprocess import Popen,CREATE_NEW_CONSOLE
command ='cmd'
prog_start=Popen(command,creationflags=CREATE_NEW_CONSOLE)
pidvalue=prog_start.pid
#this will kill the invoked terminal
subprocess.Popen('taskkill /F /T /PID %i' % pidvalue)

Supressing output with os.system()

Why does the below MWE not redirect output to /dev/null.
#!/usr/bin/env python
import os
if __name__ == "__main__":
os.system ( 'echo hello &>/dev/null' )

Not sure, but another (better) way to do it is:
from os import devnull
from subprocess import call
if __name__ == "__main__":
with open(devnull, 'w') as dn:
call(['echo', 'hello'], stdout=dn, stderr=dn)
This opens /dev/null for writing, and redirects output of the spawned process to there.
UPDATE based on comments from #abarnert
In the specific case of echo, to get identically the same behavior you will want to use shell=True because otherwise it will call /bin/echo, not the shell builtin:
call('echo hello', shell=True, stdout=dn, stderr=dn)
Also, on Python 3.3+, you can do
from subprocess import call, DEVNULL
if __name__ == "__main__":
call('echo hello', shell=True, stdout=DEVNULL, stderr=DEVNULL)

python Popen: How do I stop(kill) the subprocess that created using Popen?

I need to stop the service(runs at the background in another thread) that I issued through Popen in python after I got the result, but the following approach failed(just use ping for the sake of explanation):
class sample(threading.Thread):
def __init__(self, command, queue):
threading.Thread.__init__(self)
self.command = command;
self.queue = queue
def run(self):
result = Popen(self.command, shell=True, stdout=PIPE, stderr=STDOUT)
while True:
output = result.stdout.readline()
if not self.queue.empty():
result.kill()
break
if output != "":
print output
else:
break
def main():
q = Queue()
command = sample("ping 127.0.0.1", q)
command.start()
time.sleep(10)
q.put("stop!")
command.join()
if __name__ == "__main__":
main()
After running above program, when I pgrep for ping, it's still there. How can I kill the subprocess opened by Popen? Thanks.
PS: I also tried result.terminate(), but doesn't really solve the problem either.

You don't really need to run a subprocess from a thread. Try running the subprocess without a thread. Also, you specified shell=True, so it's running the command in a shell. So there are two new processes, the shell and the command. You can also remove the shell by making shell=False.