There's quite a bit of posts related to collecting live output from a process that was launched using Python's subprocess module. When I attempt these solutions between my two test scripts, one being a (ba)sh script and the other being a Python script, the Python script fails to have its output read live. Instead when the Python script is ran by subprocess it ends up waiting until the process has completed to flush it to PIPE. The constraints I'm bounded by is that I do want a way to retrieve live output from subprocess for the Python script.
Tested on Ubuntu 20.04 & Windows, Shell script ran on Ubuntu 20.04.
Calling code:
import shlex
import subprocess
# invoke process
process = subprocess.Popen('python test.py',shell=True,stdout=subprocess.PIPE) #Shell true/false results in "not live" output
# Poll process.stdout to show stdout live
while True:
output = process.stdout.readline() # <-- Hangs here on calling test.py, doesn't hang on test.sh
if process.poll() is not None:
break
if output:
print(output.strip())
rc = process.poll()
test.py <-- Waits until it has completed to print out entire output
import time
for x in range(10):
print(x)
time.sleep(1)
test.sh <-- Prints out live in Python script
#!/bin/bash
for i in $(seq 1 5); do
echo "iteration" $i
sleep 1
done
#stochastic13 Provided a very useful link where the -u switch and PYTHONUNBUFFERED variable being set would work. For my needs, I used PYTHONUNBUFFERED which solved my issue entirely. The Python test script actually executes another Python script to run, which I needed the output on. Despite -u helping for the first script, it wouldn't help for the second as I wouldn't have direct access to said script to add the argument. Instead I went with the environment variable, solution below:
def run_command(command):
os.environ['PYTHONUNBUFFERED'] = '1'
process = Popen(command, shell=False, stdout=PIPE, env=os.environ) # Shell doesn't quite matter for this issue
while True:
output = process.stdout.readline()
if process.poll() is not None:
break
if output:
print(output)
rc = process.poll()
return rc
Above the code passes PYTHONUNBUFFERED and sets it to the environment, any spawned process in subprocess with this environment set will inherit PYTHONUNBUFFERED.
Test Script
import subprocess
from io import TextIOWrapper, TextIOBase, StringIO
from subprocess import PIPE, Popen, call
from tempfile import TemporaryFile
from sarge import run, Capture
# process = Popen('python test2.py', shell=False)
# while True:
# if process.poll() is not None:
# break
# rc = process.poll()
subprocess.call('python test2.py')
Test Script 2
import time
import os
print(list(os.environ.keys()))
for x in range(10):
print('test2', x)
time.sleep(1)
The output is a live capture of stdout from any Python process, not just after completion.
...
b'test2 0\r\n'
b'test2 1\r\n'
b'test2 2\r\n'
b'test2 3\r\n'
...
0
Related
I am working on UBUNTU and I have file main.py with a code inside:
#!/usr/bin/env python3
# coding=utf-8
import os
import time
from subprocess import Popen, PIPE, call, signal
base_path = os.path.abspath('')
path_to_file = base_path + '/test_subprocess.py'
p = Popen(['gnome-terminal', "--", path_to_file])
time.sleep(2)
os.kill(p.pid, signal.SIGKILL)
I have test_subprocess.py with code like that:
#!/usr/bin/env python3
# coding=utf-8
import time
def print_message():
while True:
print('I am working!')
time.sleep(0.5)
print_message()
I tried to kill the subprocess but after
os.kill(p.pid, signal.SIGKILL)
subprocess is still working and prints 'I am working!'
How can I finish subprocess and how to close gnome terminal?
If I selected completely wrong way. Can you show me working example?
New version of test_subprocess.py
#!/usr/bin/env python3
# coding=utf-8
import sys
from subprocess import signal
import time
def print_message():
while True:
print('I am working!')
time.sleep(0.5)
if signal.SIGKILL: # it is braking a loop when parent process terminate!
print('I am killing self!')
break
print_message()
Should I do it like above?
You could try the following:
p = Popen(['gnome-terminal', "--", path_to_file])
PIDs = p.pid
os.system("kill {0}".format(PIDs))
Popen.pid The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
http://docs.python.org/library/subprocess.html
This will at least kill the correct process. Not sure if it will close the terminal.
Edit: to kill the process and close the terminal:
p = Popen(['gnome-terminal', '--disable-factory', '-e', path_to_file], preexec_fn=os.setpgrp)
os.killpg(p.pid, signal.SIGINT)
Credit to https://stackoverflow.com/a/34690644/15793575, whih I modified for your command:
--disable-factory is used to avoid re-using an active terminal so that we can kill newly created terminal via the subprocess handle
os.setpgrp puts gnome-terminal in its own process group so that
os.killpg() could be used to send signal to this group
Popen.pid
The process ID of the child process.
Note that if you set the shell argument to True, this is the process
ID of the spawned shell.
Try setting the shell argument of the Popen constructor to False. (p = Popen(['gnome-terminal', "--", path_to_file]) -> p = Popen(['gnome-terminal', "--", path_to_file], shell=False)). I had a similar issue not long ago - this fixed it for me.
I want code like this:
if True:
run('ABC.PY')
else:
if ScriptRunning('ABC.PY):
stop('ABC.PY')
run('ABC.PY'):
Basically, I want to run a file, let's say abc.py, and based on some conditions. I want to stop it, and run it again from another python script. Is it possible?
I am using Windows.
You can use python Popen objects for running processes in a child process
So run('ABC.PY') would be p = Popen("python 'ABC.PY'")
if ScriptRunning('ABC.PY) would be if p.poll() == None
stop('ABC.PY') would be p.kill()
This is a very basic example for what you are trying to achieve
Please checkout subprocess.Popen docs to fine tune your logic for running the script
import subprocess
import shlex
import time
def run(script):
scriptArgs = shlex.split(script)
commandArgs = ["python"]
commandArgs.extend(scriptArgs)
procHandle = subprocess.Popen(commandArgs, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
return procHandle
def isScriptRunning(procHandle):
return procHandle.poll() is None
def stopScript(procHandle):
procHandle.terminate()
time.sleep(5)
# Forcefully terminate the script
if isScriptRunning(procHandle):
procHandle.kill()
def getOutput(procHandle):
# stderr will be redirected to stdout due "stderr=subprocess.STDOUT" argument in Popen call
stdout, _ = procHandle.communicate()
returncode = procHandle.returncode
return returncode, stdout
def main():
procHandle = run("main.py --arg 123")
time.sleep(5)
isScriptRunning(procHandle)
stopScript(procHandle)
print getOutput(procHandle)
if __name__ == "__main__":
main()
One thing that you should be aware about is stdout=subprocess.PIPE.
If your python script has a very large output, the pipes may overflow causing your script to block until .communicate is called over the handle.
To avoid this, pass a file handle to stdout, like this
fileHandle = open("main_output.txt", "w")
subprocess.Popen(..., stdout=fileHandle)
In this way, the output of the python process will be dumped into the file.(You will have to modily the getOutput() function too for this)
import subprocess
process = None
def run_or_rerun(flag):
global process
if flag:
assert(process is None)
process = subprocess.Popen(['python', 'ABC.PY'])
process.wait() # must wait or caller will hang
else:
if process.poll() is None: # it is still running
process.terminate() # terminate process
process = subprocess.Popen(['python', 'ABC.PY']) # rerun
process.wait() # must wait or caller will hang
I'm trying to terminate a subprocess pid if a string is in the output, but it is not working. What is wrong?
import subprocess
import shlex
if "PING" in subprocess.check_call(shlex.split("ping -c 10 gogole.com")):
subprocess.check_call(shlex.split("ping -c 10 gogole.com")).terminate()
Please refere to the documentation for the methods you call. First of all, check_call executes until the process is finished, then returns the return code from the process. I'm not sure how you intend to find "PING" from a return code, which is typically an integer.
If it is there, look at the body of your if statement: you fork a totally new instance of ping, wait for it to complete, and then try to terminate the return code.
I recommend that you work through a tutorial on subprocesses. Learn how to grab a process handle and invoke operations on that. You'll need to get a handle on the output stream, look for "PING" in that, and then call terminate on the process handle you got at invocation.
import subprocess, os
run = "ping -c 10 google.com"
log = ""
process = subprocess.Popen(run, stdout=subprocess.PIPE, shell=True)
while True:
out = process.stdout.read(1)
log +=out
print log
if out == '' and process.poll() != None:
break
if "PING" in log:
print "terminated!"
process.kill()
process.terminate()
break
This program should echo the pid of sleep immediately:
import subprocess
subprocess.check_output("sleep 1 & echo $!", shell=True)
Running this on the shell directly, it immediately prints the pid, but running it in python, the & is ignored and it takes 1 second before echo is executed.
How can I get this to work with only one execution of check_output (or another function of subprocess)?
(This is a simplified example, in reality instead of sleep 1 I'd put my own executable)
check_output waits for the output pipes to close and sleep has them too. You can redirect to /dev/null for an immediate return.
subprocess.check_output("sleep 1 >/dev/null 2>&1 & echo $!", shell=True)
UPDATE
Its hard to tell if sleep 1 really did run in the background so I wrote a slightly larger test.
test.py - writes time to stdout for 5 seconds
import time
for i in range(5):
print(time.strftime('%H:%M:%S'), flush=True)
time.sleep(1)
print('done', flush=True)
runner.py - runs the test redirecting stdout to a file and monitors the file.
import subprocess as subp
import time
import os
# run program in background
pid = int(subp.check_output("python3 test.py >test.out 2>&1 & echo $!",
shell=True))
print("pid", pid)
# monitor output file
pos = 0
done = False
while not done:
time.sleep(.1)
if os.stat('test.out').st_size > pos:
with open('test.out', 'rb') as fp:
fp.seek(pos)
for line in fp.readlines():
print(line.strip().decode())
done = b'done' in line
pos = fp.tell()
print("test complete")
Running it, I get
td#mintyfresh ~/tmp $ python3 runner.py
pid 24353
09:32:18
09:32:19
09:32:20
09:32:21
09:32:22
done
test complete
Please help me to write simple console application in python. It should redirect all input to system shell (bash or windows cmd or powershell) and give all their output to the screen.
Simply I can say run terminal from python application.
The next code works with some strange behavior: first 3 times after any key pressed it outputs (executes?) some previous commands (may be from cache)
#!/bin/python3
import subprocess
import sys
proc = subprocess.Popen(['bash'])
while True:
buff = sys.stdin.readline()
stdoutdata, stderrdata = proc.communicate(buff)
if( stdoutdata ):
print( stdoutdata )
else:
print('n')
break
I think you need
proc = subprocess.Popen(['bash'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
From the docs:
PIPE indicates that a new pipe to the child should be created.
DEVNULL indicates that the special file os.devnull will be used. With
the default settings of None, no redirection will occur; the child’s
file handles will be inherited from the parent.
I don't think you want your bash to be connected to your parent process's stdin directly. That would explain wierdness.