We Keep Coding

multiprocessing.Manager() hangs Popen.communicate() on Python

The use of multiprocessing.Manager prevents clean termination of Python child process using subprocess.Process.Popen.terminate() and subprocess.Process.Popen.kill().
This seems to be because Manager creates a child process behind the scenes for communicating, but this process does not know how to clean itself up when the parent is terminated.
What is the easiest way to use multiprocessing.Manager so that it does not prevent a process shutdown by a signal?
A demostration:
"""Multiprocess manager hang test."""
import multiprocessing
import subprocess
import sys
import time
def launch_and_read_process():
proc = subprocess.Popen(
[
"python",
sys.argv[0],
"run_unkillable"
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
)
# Give time for the process to run and print()
time.sleep(3)
status = proc.poll()
print("poll() is", status)
print("Terminating")
assert proc.returncode is None
proc.terminate()
exit_code = proc.wait()
print("Got exit code", exit_code)
stdout, stderr = proc.communicate()
print("Got output", stdout.decode("utf-8"))
def run_unkillable():
# Disable manager creation to make the code run correctly
manager = multiprocessing.Manager()
d = manager.dict()
d["foo"] = "bar"
print("This is an example output", flush=True)
time.sleep(999)
def main():
mode = sys.argv[1]
print("Doing subrouting", mode)
func = globals().get(mode)
func()
if __name__ == "__main__":
main()
Run as python test-script.py launch_and_read_process.
Good output (no multiprocessing.Manager):
Doing subrouting launch_and_read_process
poll() is None
Terminating
Got exit code -15
Got output Doing subrouting run_unkillable
This is an example output
Output when subprocess.Popen.communicate hangs because use of Manager:
Doing subrouting launch_and_read_process
poll() is None
Terminating
Got exit code -15

Like you pointed out, this happens because the manager spawns its own child processes. So when you do proc.communicate() the code hangs because that child process's stderr and stdout are still open. You can easily solve this on Unix by setting your own handlers for SIGTERM and SIGINT, but it becomes a little hairy on Windows since those two signals are pretty much useless. Also, keep in mind that signals are only delivered to the main thread. Depending on your OS and the signal, if the thread is busy (time.sleep(999)) then the whole timer may need to run out before the signal can be intercepted. Anyway, I have provided a solution for both Windows and Unix with a note at the end:
UNIX
This is pretty straightforward, you simply define your own handlers for the signals where you explicitly call manager.shutdown() to properly cleanup its child process:
def handler(manager, *args):
"""
Our handler, use functools.partial to fix arg manager (or you
can create a factory function too)
"""
manager.shutdown()
sys.exit()
def run_unkillable():
# Disable manager creation to make the code run correctly
manager = multiprocessing.Manager()
# Register our handler,
h = functools.partial(handler, manager)
signal.signal(signal.SIGINT, h)
signal.signal(signal.SIGTERM, h)
d = manager.dict()
d["foo"] = "bar"
print("This is an example output", flush=True)
time.sleep(999)
Windows
On Windows you will need to explicitly send the signal signal.CTRL_BREAK_EVENT rather than the plain proc.terminate() to ensure that the child process intercepts it (reference). Additionally, you'll also want to sleep in shorter durations in a loop instead of doing sleep(999) to make sure the signal interrupts the main thread rather than waiting for the whole duration of sleep (see this question for alternatives).
"""Multiprocess manager hang test."""
import functools
import multiprocessing
import subprocess
import sys
import time
import signal
def launch_and_read_process():
proc = subprocess.Popen(
[
"python",
sys.argv[0],
"run_unkillable"
],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
creationflags=subprocess.CREATE_NEW_PROCESS_GROUP # So that our current process does not get SIGBREAK signal
)
# Give time for the process to run and print()
time.sleep(5)
status = proc.poll()
print("poll() is", status)
print("Terminating")
assert proc.returncode is None
# Send this specific signal instead of doing terminate()
proc.send_signal(signal.CTRL_BREAK_EVENT)
exit_code = proc.wait()
print("Got exit code", exit_code)
stdout, stderr = proc.communicate()
print("Got output", stdout.decode("utf-8"))
def handler(manager, *args):
"""
Our handler, use functools.partial to fix arg manager (or you
can create a factory function too)
"""
manager.shutdown()
sys.exit()
def run_unkillable():
# Disable manager creation to make the code run correctly
manager = multiprocessing.Manager()
# Register our handler,
signal.signal(signal.SIGBREAK, functools.partial(handler, manager))
d = manager.dict()
d["foo"] = "bar"
print("This is an example output", flush=True)
# Sleep in a loop otherwise the signal won't interrupt the main thread
for _ in range(999):
time.sleep(1)
def main():
mode = sys.argv[1]
print("Doing subrouting", mode)
func = globals().get(mode)
func()
if __name__ == "__main__":
main()
Note: Keep in mind that there is a race condition in the above solution because we are registering the signal handler after the creation of a manager. Theoretically, one could kill the process right before the handler is registered and the proc.communicate() will then hang because the manager was not cleaned up. So you may want to supply a timeout parameter to .communicate with error handling to log these edge cases.

Tkinter read subprocess output and put it into the gui

I'm pretty new to python and tkinter so I'm struggling with creating a script that reads the terminal output into a label or Gui in tkinter. I've searched around and can't find any tutorials on how to do it, a lot of forums have specific or old code which makes it really difficult to adapt especially for a beginner. The code that I found that looks like it will work best for what I'm trying to accomplish was made by jfs, the only problem is that I keep getting errors which for the life of me I cant figure out.
Here is the code:
import logging
import os
import sys
from subprocess import Popen, PIPE, STDOUT
try:
import tkinter as tk
except ImportError: # Python 3
import tkinter as tk
info = logging.getLogger(__name__).info
# define dummy subprocess to generate some output
cmd = [sys.executable or "python", "-u", "-c", """
import itertools, time
for i in itertools.count():
print(i)
time.sleep(0.5)
"""]
class ShowProcessOutputDemo:
def __init__(self, root):
"""Start subprocess, make GUI widgets."""
self.root = root
# start subprocess
self.proc = Popen(cmd, stdout=PIPE, stderr=STDOUT)
# show subprocess' stdout in GUI
self.root.createfilehandler(
self.proc.stdout, tk.READABLE, self.read_output)
self._var = tk.StringVar() # put subprocess output here
tk.Label(root, textvariable=self._var).pack()
# stop subprocess using a button
tk.Button(root, text="Stop subprocess", command=self.stop).pack()
def read_output(self, pipe, mask):
"""Read subprocess' output, pass it to the GUI."""
data = os.read(pipe.fileno(), 1 << 20)
if not data: # clean up
info("eof")
self.root.deletefilehandler(self.proc.stdout)
self.root.after(5000, self.stop) # stop in 5 seconds
return
info("got: %r", data)
self._var.set(data.strip(b'\n').decode())
def stop(self, stopping=[]):
"""Stop subprocess and quit GUI."""
if stopping:
return # avoid killing subprocess more than once
stopping.append(True)
info('stopping')
self.proc.terminate() # tell the subprocess to exit
# kill subprocess if it hasn't exited after a countdown
def kill_after(countdown):
if self.proc.poll() is None: # subprocess hasn't exited yet
countdown -= 1
if countdown < 0: # do kill
info('killing')
self.proc.kill() # more likely to kill on *nix
else:
self.root.after(1000, kill_after, countdown)
return # continue countdown in a second
self.proc.stdout.close() # close fd
self.proc.wait() # wait for the subprocess' exit
self.root.destroy() # exit GUI
kill_after(countdown=5)
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')
root = tk.Tk()
app = ShowProcessOutputDemo(root)
root.protocol("WM_DELETE_WINDOW", app.stop) # exit subprocess if GUI is closed
root.mainloop()
info('exited')
Due to my lack of experience, i don't know how to fix the errors that keep being thrown. Here is a terminal output of what keeps occurring.
Traceback (most recent call last):
File "d:\coding\OtherProjects\testserver\tkinter-read-async-subprocess-output.py", line 83, in <module>
app = ShowProcessOutputDemo(root)
File "d:\coding\OtherProjects\testserver\tkinter-read-async-subprocess-output.py", line 37, in __init__
self.root.createfilehandler(
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1520.0_x64__qbz5n2kfra8p0\lib\tkinter\__init__.py", line 2354, in __getattr__
return getattr(self.tk, attr)
AttributeError: '_tkinter.tkapp' object has no attribute 'createfilehandler'
Thanks to everyone that took the time to read this I really appreciate it.
Also sorry if i didn't put this in the right forum I'm still trying to understand this website and am dedicated to improve.
Thank You - Connor

Try this:
import logging
import os
import sys
from subprocess import Popen, PIPE, STDOUT
from threading import Thread
try:
import tkinter as tk
except ImportError: # Python 3
import tkinter as tk
info = logging.getLogger(__name__).info
# define dummy subprocess to generate some output
cmd = [sys.executable or "python", "-u", "-c", """
import itertools, time
for i in itertools.count():
print(i)
time.sleep(0.5)
"""]
class ShowProcessOutputDemo:
def __init__(self, root):
"""Start subprocess, make GUI widgets."""
self.root = root
# start subprocess
self.proc = Popen(cmd, stdout=PIPE, stderr=STDOUT)
# stop subprocess using a button
tk.Button(root, text="Stop subprocess", command=self.stop).pack()
self.label = tk.Label(root) # put subprocess output here
self.label.pack()
# Create a buffer for the stdout
self.stdout_data = ""
# Create a new thread that will read stdout and write the data to
# `self.stdout_buffer`
thread = Thread(target=self.read_output, args=(self.proc.stdout, ))
thread.start()
# A tkinter loop that will show `self.stdout_data` on the screen
self.show_stdout()
def read_output(self, pipe):
"""Read subprocess' output and store it in `self.stdout_data`."""
while True:
data = os.read(pipe.fileno(), 1 << 20)
# Windows uses: "\r\n" instead of "\n" for new lines.
data = data.replace(b"\r\n", b"\n")
if data:
info("got: %r", data)
self.stdout_data += data.decode()
else: # clean up
info("eof")
self.root.after(5000, self.stop) # stop in 5 seconds
return None
def show_stdout(self):
"""Read `self.stdout_data` and put the data in the GUI."""
self.label.config(text=self.stdout_data.strip("\n"))
self.root.after(100, self.show_stdout)
def stop(self, stopping=[]):
"""Stop subprocess and quit GUI."""
if stopping:
return # avoid killing subprocess more than once
stopping.append(True)
info("stopping")
self.proc.terminate() # tell the subprocess to exit
# kill subprocess if it hasn't exited after a countdown
def kill_after(countdown):
if self.proc.poll() is None: # subprocess hasn't exited yet
countdown -= 1
if countdown < 0: # do kill
info("killing")
self.proc.kill() # more likely to kill on *nix
else:
self.root.after(1000, kill_after, countdown)
return # continue countdown in a second
self.proc.stdout.close() # close fd
self.proc.wait() # wait for the subprocess' exit
self.root.destroy() # exit GUI
kill_after(countdown=5)
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
root = tk.Tk()
app = ShowProcessOutputDemo(root)
root.protocol("WM_DELETE_WINDOW", app.stop) # exit subprocess if GUI is closed
root.mainloop()
info("exited")
This code starts a new thread that reads from self.proc.stdout and writes the data into self.stdout_data in a while True loop. There is also a tkinter loop that takes the data out of the self.stdout_data and puts it in the Label widget.
I didn't directly set the Label's text from the thread because sometimes tkinter can crash if you call it from a different thread.
Another thing: I removed the StringVar because I can just use: <tkinter.Label>.config(text=<new text>) instead.

I have the problem too, this was my solution.
I post it on [https://github.com/ianfun/notes/blob/main/editor/main.py].
data.py was generate by build.py.
I cannot found grate terminal like vscode in python.
note:use key right to select
you can read idlelib run.py

How & where to best retrieve sudo password via a native GUI on a macOS Python-based app - (while maintaining an interactive output stream (stdout))

Ok, so the situation is this: I am building a macOS GUI App using Python and wx (wxphoenix). The user can use the GUI (say: script1) to launch a file-deletion process (contained in script2). In order to run successfully script2 needs to run with sudo rights.
script2 will itterate over a long list of files and delete them. But I need it to communicate with the GUI contained in script1 after each round so that script1 can update the progressbar.
In it's absolute most basic form my current working setup looks like this:
Script1:
import io
from threading import Thread
import subprocess
import wx
# a whole lot of wx GUI stuff
def get_password():
"""Retrieve user password via a GUI"""
# A wx solution using wx.PasswordEntryDialog()
# Store password in a variable
return variable
class run_script_with_sudo(Thread):
"""Launch a script with administrator privileges"""
def __init__(self, path_to_script, wx_pubsub_sendmessage):
"""Set variables to self"""
self.path = path_to_script
self.sender = wx_pubsub_sendmessage
self.password = get_password()
Thread.__init__(self)
self.start()
def run(self):
"""Run thread"""
prepare_script = subprocess.Popen(["echo", password], stdout=subprocess.PIPE)
prepare_script.wait()
launch_script = subprocess.Popen(['sudo', '-S', '/usr/local/bin/python3.6', '-u', self.path], stdin=prepare_script.stdout, stdout=subprocess.PIPE)
for line in io.TextIOWrapper(launch_script.stdout, encoding="utf-8"):
print("Received line: ", line.rstrip())
# Tell progressbar to add another step:
wx.CallAfter(self.sender, "update", msg="")
Script2:
import time
# This is a test setup, just a very simple loop that produces an output.
for i in range(25):
time.sleep(1)
print(i)
The above setup works in that script1 receives the output of script2 in real-time and acts on it. (So in the given example: after each second script1 adds another step to the progress bar until it reaches 25 steps).
What I want to achieve = not storing the password in a variable and using macOS it's native GUI to retrieve the password.
However when I change:
prepare_script = subprocess.Popen(["echo", password], stdout=subprocess.PIPE)
prepare_script.wait()
launch_script = subprocess.Popen(['sudo', '-S', '/usr/local/bin/python3.6', '-u', self.path], stdin=prepare_script.stdout, stdout=subprocess.PIPE)
for line in io.TextIOWrapper(launch_script.stdout, encoding="utf-8"):
print("Received line: ", line.rstrip())
# Tell progressbar to add another step:
wx.CallAfter(self.sender, "update", msg="")
Into:
command = r"""/usr/bin/osascript -e 'do shell script "/usr/local/bin/python3.6 -u """ + self.path + """ with prompt "Sart Deletion Process " with administrator privileges'"""
command_list = shlex.split(command)
launch_script = subprocess.Popen(command_list, stdout=subprocess.PIPE)
for line in io.TextIOWrapper(launch_script.stdout, encoding="utf-8"):
print("Received line: ", line.rstrip())
# Tell progressbar to add another step:
wx.CallAfter(self.sender, "update", msg="")
It stops working because osascript apparently runs in a non-interactive shell. This means script2 doesn't sent any output until it is fully finished, causing the progress bar in script1 to stall.
My question thus becomes: How can I make sure to use macOS native GUI to ask for the sudo password, thus preventing having to store it in a variable, while still maintaining the possibility to catch the stdout from the privileged script in an interactive / real-time stream.
Hope that makes sense.
Would appreciate any insights!

My question thus becomes: How can I make sure to use macOS native GUI
to ask for the sudo password, thus preventing having to store it in a
variable, while still maintaining the possibility to catch the stdout
from the privileged script in an interactive / real-time stream.
I have found a solution myself, using a named pipe (os.mkfifo()).
That way, you can have 2 python scripts communicate with each other while 1 of them is launched with privileged rights via osascript (meaning: you get a native GUI window that asks for the users sudo password).
Working solution:
mainscript.py
import os
from pathlib import Path
import shlex
import subprocess
import sys
from threading import Thread
import time
class LaunchDeletionProcess(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
launch_command = r"""/usr/bin/osascript -e 'do shell script "/usr/local/bin/python3.6 -u /path/to/priviliged_script.py" with prompt "Sart Deletion Process " with administrator privileges'"""
split_command = shlex.split(launch_command)
print("Thread 1 started")
testprogram = subprocess.Popen(split_command)
testprogram.wait()
print("Thread1 Finished")
class ReadStatus(Thread):
def __init__(self):
Thread.__init__(self)
def run(self):
while not os.path.exists(os.path.expanduser("~/p1")):
time.sleep(0.1)
print("Thread 2 started")
self.wfPath = os.path.expanduser("~/p1")
rp = open(self.wfPath, 'r')
response = rp.read()
self.try_pipe(response)
def try_pipe(self, response):
rp = open(self.wfPath, 'r')
response = rp.read()
print("Receiving response: ", response)
rp.close()
if response == str(self.nr_of_steps-1):
print("Got to end")
os.remove(os.path.expanduser("~/p1"))
else:
time.sleep(1)
self.try_pipe(response)
if __name__ == "__main__":
thread1 = LaunchDeletionProcess()
thread2 = ReadStatus()
thread1.start()
thread2.start()
priviliged_script.py
import os
import time
import random
wfPath = os.path.expanduser("~/p1")
try:
os.mkfifo(wfPath)
except OSError:
print("error")
pass
result = 10
nr = 0
while nr < result:
random_nr = random.random()
wp = open(wfPath, 'w')
print("writing new number: ", random_nr)
wp.write("Number: " + str(random_nr))
wp.close()
time.sleep(1)
nr += 1
wp = open(wfPath, 'w')
wp.write("end")
wp.close()

Run subprocess in python and get stdout and kill process on timeout

Here is my code, it launches a subprocess, waits till it ends and returns stdout, or a timeout happens and it raises exception. Common use is print(Run('python --version').execute())
class Run(object):
def __init__(self, cmd, timeout=2*60*60):
self.cmd = cmd.split()
self.timeout = timeout
self._stdout = b''
self.dt = 10
self.p = None
def execute(self):
print("Execute command: {}".format(' '.join(self.cmd)))
def target():
self.p = Popen(self.cmd, stdout=PIPE, stderr=STDOUT)
self._stdout = self.p.communicate()[0]
thread = Thread(target=target)
thread.start()
t = 0
while t < self.timeout:
thread.join(self.dt)
if thread.is_alive():
t += self.dt
print("Running for: {} seconds".format(t))
else:
ret_code = self.p.poll()
if ret_code:
raise AssertionError("{} failed.\nretcode={}\nstdout:\n{}".format(
self.cmd, ret_code, self._stdout))
return self._stdout
else:
print('Timeout {} reached, kill task, pid={}'.format(self.timeout, self.p.pid))
self.p.terminate()
thread.join()
raise AssertionError("Timeout")
The problem is following case. The process that I launch spawns more child processes. So when the timeout is reached, I kill main process (the one I srarted using my class) with self.p.terminate(), the children are remaining and my code hangs on line self._stdout = self.p.communicate()[0]. And execution continues if I manually kill all child processes.
I tried soulution when instead of self.p.terminate() I kill whole process tree.
This also does not work if the main process finished by itself and its children are existing on their own, and I have no ability to find and kill them. But they are blocking self.p.communicate().
Is there way to effectively solve this?

You could use the ProcessWrapper from the PySys framework - it offers alot of this functionality as an abstraction in a cross platform way i.e.
import sys, os
from pysys.constants import *
from pysys.process.helper import ProcessWrapper
from pysys.exceptions import ProcessTimeout
command=sys.executable
arguments=['--version']
try:
process = ProcessWrapper(command, arguments=arguments, environs=os.environ, workingDir=os.getcwd(), stdout='stdout.log', stderr='stderr.log', state=FOREGROUND, timeout=5.0)
process.start()
except ProcessTimeout:
print "Process timeout"
process.stop()
It's at SourceForge (http://sourceforge.net/projects/pysys/files/ and http://pysys.sourceforge.net/) if of interest.

How to achieve desired results when using the subprocees Popen.send_signal(CTRL_C_EVENT) in Windows?

In python 2.7 in windows according to the documentation you can send a CTRL_C_EVENT
(Python 2.7 Subprocess Popen.send_signal documentation).
However when I tried it I did not receive the expected keyboard interrupt in the subprocess.
This is the sample code for for the parent process:
# FILE : parentProcess.py
import subprocess
import time
import signal
CREATE_NEW_PROCESS_GROUP = 512
process = subprocess.Popen(['python', '-u', 'childProcess.py'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
universal_newlines=True,
creationflags=CREATE_NEW_PROCESS_GROUP)
print "pid = ", process.pid
index = 0
maxLoops = 15
while index < maxLoops:
index += 1
# Send one message every 0.5 seconds
time.sleep(0.5)
# Send data to the subprocess
process.stdin.write('Bar\n')
# Read data from the subprocess
temp = process.stdout.readline()
print temp,
if (index == 10):
# Send Keyboard Interrupt
process.send_signal(signal.CTRL_C_EVENT)
This is the sample code for the child proceess:
# FILE : childProcess.py
import sys
while True:
try:
# Get data from main process
temp = sys.stdin.readline()
# Write data out
print 'Foo ' + temp,
except KeyboardInterrupt:
print "KeyboardInterrupt"
If I run the file parentProcess.py I expect to get "Foo Bar" ten times then a "KeyboardInterrupt" followed by "Foo Bar" 4 times but I get "Foo Bar" 15 times instead.
Is there a way to get the CTRL_C_EVENT to behave as a keyboard interrupt just as SIGINT behaves in Linux?
After doing some reading I found some information that seems to contradic the python documentation regarding CTRL_C_EVENT, in particular it says that
CTRL_C_EVENT
0 Generates a CTRL+C signal. This signal cannot be generated for process groups
The following site provide more inforamtion about creation flags:
Process Creation Flags.

This method of signal handling by subprocesses worked for me on both Linux and Windows 2008, both using Python 2.7.2, but it uses Ctrl-Break instead of Ctrl-C. See the note about process groups and Ctrl-C in http://msdn.microsoft.com/en-us/library/ms683155%28v=vs.85%29.aspx.
catcher.py:
import os
import signal
import sys
import time
def signal_handler(signal, frame):
print 'catcher: signal %d received!' % signal
raise Exception('catcher: i am done')
if hasattr(os.sys, 'winver'):
signal.signal(signal.SIGBREAK, signal_handler)
else:
signal.signal(signal.SIGTERM, signal_handler)
print 'catcher: started'
try:
while(True):
print 'catcher: sleeping...'
time.sleep(1)
except Exception as ex:
print ex
sys.exit(0)
thrower.py:
import signal
import subprocess
import time
import os
args = [
'python',
'catcher.py',
]
print 'thrower: starting catcher'
if hasattr(os.sys, 'winver'):
process = subprocess.Popen(args, creationflags=subprocess.CREATE_NEW_PROCESS_GROUP)
else:
process = subprocess.Popen(args)
print 'thrower: waiting a couple of seconds for catcher to start...'
time.sleep(2)
print 'thrower: sending signal to catch'
if hasattr(os.sys, 'winver'):
os.kill(process.pid, signal.CTRL_BREAK_EVENT)
else:
process.send_signal(signal.SIGTERM)
print 'thrower: i am done'

try with
win32api.GenerateConsoleCtrlEvent(CTRL_C_EVENT, pgroupid)
or
win32api.GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, pgroupid)
references:
http://docs.activestate.com/activepython/2.5/pywin3/win32process_CREATE_NEW_PROCESS_GROUP.html
http://msdn.microsoft.com/en-us/library/ms683155%28v=vs.85%29.aspx
read info about dwProcessGroupId, the groupid should be the same of the process id