We Keep Coding

Rationalizing and simplifying Python 3 Keyboard Interrupt behavior in a threaded program

Running the following minimized and reproducible code example, python (e.g. 3.7.3, and 3.8.3) will emit a message as follows when a first Ctrl+C is pressed, rather than terminate the program.
Traceback (most recent call last):
File "main.py", line 44, in <module>
Main()
File "main.py", line 41, in __init__
self.interaction_manager.join()
File "/home/user/anaconda3/lib/python3.7/threading.py", line 1032, in join
self._wait_for_tstate_lock()
File "/home/user/anaconda3/lib/python3.7/threading.py", line 1048, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
Only on a second Ctrl+C being pressed after that, the program will terminate.
What is the rationale behind this design? What would be an elegant way for avoiding the need for more than a single Ctrl+C or underlying signal?
Here's the code:
from threading import Thread
from queue import Queue, Empty
def get_event(queue: Queue, block=True, timeout=None):
""" just a convenience wrapper for avoiding try-except clutter in code """
try:
element = queue.get(block, timeout)
except Empty:
element = Empty
return element
class InteractionManager(Thread):
def __init__(self):
super().__init__()
self.queue = Queue()
def run(self):
while True:
event = get_event(self.queue, block=True, timeout=0.1)
class Main(object):
def __init__(self):
# kick off the user interaction
self.interaction_manager = InteractionManager()
self.interaction_manager.start()
# wait for the interaction manager object shutdown as a signal to shutdown
self.interaction_manager.join()
if __name__ == "__main__":
Main()
Prehistoric related question: Interruptible thread join in Python

Python waits for all non-daemon threads before exiting. The first Ctrl+C merely kills the explicit self.interaction_manager.join(), the second Ctrl+C kills the internal join() of threading. Either declare the thread as an expendable daemon thread, or signal it to shut down.
A thread can be declared as expendable by setting daemon=True, either as a keyword or attribute:
class InteractionManager(Thread):
def __init__(self):
super().__init__(daemon=True)
self.queue = Queue()
def run(self):
while True:
event = get_event(self.queue, block=True, timeout=0.1)
A daemon thread is killed abruptly, and may fail to cleanly release resources if it holds any.
Graceful shutdown can be coordinated using a shared flag, such as threading.Event or a boolean value:
shutdown = False
class InteractionManager(Thread):
def __init__(self):
super().__init__()
self.queue = Queue()
def run(self):
while not shutdown:
event = get_event(self.queue, block=True, timeout=0.1)
def main()
self.interaction_manager = InteractionManager()
self.interaction_manager.start()
try:
self.interaction_manager.join()
finally:
global shutdown
shutdown = True

Multiprocessing.Queue with hugh data causes _wait_for_tstate_lock

An Exception is raised in threading._wait_for_tstate_lock when I transfere hugh data between a Process and a Thread via multiprocessing.Queue.
My minimal working example looks a bit complex first - sorry. I will explain. The original application loads a lot of (not so important) files into RAM. This is done in a separate process to save ressources. The main gui thread shouldn't freez.
The GUI start a separate Thread to prevent the gui event loop from freezing.
This separate Thread then starts one Process which should does the work.
a) This Thread instanciates a multiprocess.Queue (be aware that this is a multiprocessing and not threading!)
b) This is givin to the Process for sharing data from Process back to the Thread.
The Process does some work (3 steps) and .put() the result into the multiprocessing.Queue.
When the Process ends the Thread takes over again and collect the data from the Queue, store it to its own attribute MyThread.result.
The Thread tells the GUI main loop/thread to call a callback function if it has time for.
The callback function (MyWindow::callback_thread_finished()) get the results from MyWindow.thread.result.
The problem is if the data put to the Queue is to big something happen I don't understand - the MyThread never ends. I have to cancle the application via Strg+C.
I got some hints from the docs. But my problem is I did not fully understand the documentation. But I have the feeling that the key of my problems can be found there.
Please see the two red boxex in "Pipes and Queues" (Python 3.5 docs).
That is the full output
MyWindow::do_start()
Running MyThread...
Running MyProcess...
MyProcess stoppd.
^CProcess MyProcess-1:
Exception ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'>
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 1288, in _shutdown
t.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/process.py", line 252, in _bootstrap
util._exit_function()
File "/usr/lib/python3.5/multiprocessing/util.py", line 314, in _exit_function
_run_finalizers()
File "/usr/lib/python3.5/multiprocessing/util.py", line 254, in _run_finalizers
finalizer()
File "/usr/lib/python3.5/multiprocessing/util.py", line 186, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/usr/lib/python3.5/multiprocessing/queues.py", line 198, in _finalize_join
thread.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
This is the minimal working example
#!/usr/bin/env python3
import multiprocessing
import threading
import time
import gi
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk
from gi.repository import GLib
class MyThread (threading.Thread):
"""This thread just starts the process."""
def __init__(self, callback):
threading.Thread.__init__(self)
self._callback = callback
def run(self):
print('Running MyThread...')
self.result = []
queue = multiprocessing.Queue()
process = MyProcess(queue)
process.start()
process.join()
while not queue.empty():
process_result = queue.get()
self.result.append(process_result)
print('MyThread stoppd.')
GLib.idle_add(self._callback)
class MyProcess (multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def run(self):
print('Running MyProcess...')
for i in range(3):
self.queue.put((i, 'x'*102048))
print('MyProcess stoppd.')
class MyWindow (Gtk.Window):
def __init__(self):
Gtk.Window.__init__(self)
self.connect('destroy', Gtk.main_quit)
GLib.timeout_add(2000, self.do_start)
def do_start(self):
print('MyWindow::do_start()')
# The process need to be started from a separate thread
# to prevent the main thread (which is the gui main loop)
# from freezing while waiting for the process result.
self.thread = MyThread(self.callback_thread_finished)
self.thread.start()
def callback_thread_finished(self):
result = self.thread.result
for r in result:
print('{} {}...'.format(r[0], r[1][:10]))
if __name__ == '__main__':
win = MyWindow()
win.show_all()
Gtk.main()
Possible duplicate but quite different and IMO without an answer for my situation: Thread._wait_for_tstate_lock() never returns.
Workaround
Using a Manager by modifing line 22 to queue = multiprocessing.Manager().Queue() solve the problem. But I don't know why. My intention of this question is to understand the things behind and not only to make my code work. Even I don't really know what a Manager() is and if it has other (problem causing) implications.

According to the second warning box in the documentation you are linking to you can get a deadlock when you join a process before processing all items in the queue. So starting the process and immediately joining it and then processing the items in the queue is the wrong order of steps. You have to start the process, then receive the items, and then only when all items are received you can call the join method. Define some sentinel value to signal that the process is finished sending data through the queue. None for example if that can't be a regular value you expect from the process.
class MyThread(threading.Thread):
"""This thread just starts the process."""
def __init__(self, callback):
threading.Thread.__init__(self)
self._callback = callback
self.result = []
def run(self):
print('Running MyThread...')
queue = multiprocessing.Queue()
process = MyProcess(queue)
process.start()
while True:
process_result = queue.get()
if process_result is None:
break
self.result.append(process_result)
process.join()
print('MyThread stoppd.')
GLib.idle_add(self._callback)
class MyProcess(multiprocessing.Process):
def __init__(self, queue):
multiprocessing.Process.__init__(self)
self.queue = queue
def run(self):
print('Running MyProcess...')
for i in range(3):
self.queue.put((i, 'x' * 102048))
self.queue.put(None)
print('MyProcess stoppd.')

The documentation in question
reads:
Warning
As mentioned above, if a child process has put items on a queue (and it has not used JoinableQueue.cancel_join_thread), then that process will not terminate until all buffered items have been flushed to the pipe.
This means that if you try joining that process you may get a deadlock unless you are sure that all items which have been put on the queue have been consumed. Similarly, if the child process is non-daemonic then the parent process may hang on exit when it tries to join all its non-daemonic children.
Note that a queue created using a manager does not have this issue. See Programming guidelines.
This is supplementary to the accepted answer, but the edit queue is full.

Main thread not catching KeyboardInterrupt

I've seen several threads on here about killing python threads in a clean way, but I think I'm having a more fundamental issue. Suppose I have some code that looks like this:
t1 = threading.Thread(target=method1, args=(args1,))
t1.daemon = True
t2 = threading.Thread(target=method2, args=(args2, ))
t2.daemon = True
t1.start()
t2.start()
while True:
time.sleep(1)
I would like for the main thread to notice a Ctrl-C keyboard interrupt, and from there I can handle the termination of the other threads (t1 and t2) as appropriate given the context. But no matter what I do, I can't get the main thread to catch the KeyboardInterrupt. I've tried something like this:
try:
while True: time.sleep(100)
except KeyboardInterrupt:
print "Quitting!"
Or something like this:
threads = []
threads.append(t1)
threads.append(t2)
while len(threads) > 0:
try:
threads = [t for t in threads if t is not None and t.isAlive()]
time.sleep(1)
except:
print "Ctrl - C received, kiling"
for t in threads:
t.kill_received = True
But none of these even print the messages in the exception handler, displaying only this:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/dist-packages/audioread/gstdec.py", line 149, in run
self.loop.run()
File "/usr/lib/python2.7/dist-packages/gi/overrides/GLib.py", line 576, in run
raise KeyboardInterrupt
KeyboardInterrupt
The main question here is not how to kill t1 and t2 safely (I've got to deal with that too, eventually), but why is the main thread not catching KeyboardInterrupt here?
Edit: As I've written the examples, I've already tried the approach involving sleeping in a while loop within the try-except block. Any other ideas?

The following works. Even though the try...except in the Thread subclass is never triggered, the main thread is able to catch KeyboardInterrupts.
The code in the run() method of the MyThread subclass is a modified version of what is the CPython 2.7 source for the Thread.run() method, which already contained a try/finally...to which I added an except BaseException as exc: in order to print a message when any type of one happens.
import threading
import time
def method1(args):
print('method1() running')
while True:
time.sleep(.001)
def method2(args):
print('method2() running')
while True:
time.sleep(.01)
class MyThread(threading.Thread):
def __init__(self, *args, **kwargs):
super(MyThread, self).__init__(*args, **kwargs)
def run(self):
""" Method representing the thread's activity.
You may override this method in a subclass. The standard run() method
invokes the callable object passed to the object's constructor as the
target argument, if any, with sequential and keyword arguments taken
from the args and kwargs arguments, respectively.
"""
try:
print('in MyThread.run()\n')
if self._Thread__target:
self._Thread__target(*self._Thread__args, **self._Thread__kwargs)
except BaseException as exc:
print('reraising Exception {}'.format(exc))
raise typeof(exc)(str(exc))
finally:
# Avoid a refcycle if the thread is running a function with
# an argument that has a member that points to the thread.
del self._Thread__target, self._Thread__args, self._Thread__kwargs
print('exiting')
args1 = 1
t1 = MyThread(target=method1, args=(args1,))
t1.daemon = True
args2 = 2
t2 = MyThread(target=method2, args=(args2,))
t2.daemon = True
t1.start()
t2.start()
try:
while True:
time.sleep(1)
except BaseException as exc:
print('exception "{}" occurred'.format(type(exc)))
quit()
Console output (I typed Ctrl-C right after the method1() running appeared):
> python not-catching-keyboardinterrupt.py
in MyThread.run()
in MyThread.run()
method2() running
method1() running
exception "<type 'exceptions.KeyboardInterrupt'>" occurred

My KeyboardInterrupt is only getting caught 90% of the time. What am I failing at?

Here's some slimmed down code that demonstrates my use of threading:
import threading
import Queue
import time
def example():
""" used in MainThread as the example generator """
while True:
yield 'asd'
class ThreadSpace:
""" A namespace to be shared among threads/functions """
# set this to True to kill the threads
exit_flag = False
class MainThread(threading.Thread):
def __init__(self, output):
super(MainThread, self).__init__()
self.output = output
def run(self):
# this is a generator that contains a While True
for blah in example():
self.output.put(blah)
if ThreadSpace.exit_flag:
break
time.sleep(0.1)
class LoggerThread(threading.Thread):
def __init__(self, output):
super(LoggerThread, self).__init__()
self.output = output
def run(self):
while True:
data = self.output.get()
print data
def main():
# start the logging thread
logging_queue = Queue.Queue()
logging_thread = LoggerThread(logging_queue)
logging_thread.daemon = True
logging_thread.start()
# launch the main thread
main_thread = MainThread(logging_queue)
main_thread.start()
try:
while main_thread.isAlive():
time.sleep(0.5)
except KeyboardInterrupt:
ThreadSpace.exit_flag = True
if __name__ == '__main__':
main()
I have one main thread which gets data yielded to it from a blocking generator. In the real code, this generator yields network related data it sniffs out over the socket.
I then have a logging, daemon, thread which prints the data to screen.
To exit the program cleanly, I'm catching a KeyboardInterrupt which will set an exit_flag to try - This tells the main thread to return.
9 times out of 10, this will work fine. The program will exit cleanly. However, there's a few occasions when I'll receive the following two errors:
Error 1:
^CTraceback (most recent call last):
File "demo.py", line 92, in <module>
main('')
File "demo.py", line 87, in main
time.sleep(0.5)
KeyboardInterrupt
Error 2:
Exception KeyboardInterrupt in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
I've run this exact sample code a few times and haven't been able to replicate the errors. The only difference between this and the real code is the example() generator. This, like I said, yields network data from the socket.
Can you see anything wrong with how I'm handling the threads?

KeyboardInterrupts are received by arbitrary threads. If the receiver isn't the main thread, it dies, the main thread is unaffected, ThreadSpace.exit_flag remains false, and the script keeps running.
If you want sigint to work, you can have each thread catch KeyboardInterrupt and call thread.interrupt_main() to get Python to exit, or use the signal module as the official documentation explains.

Python Multiprocessing atexit Error "Error in atexit._run_exitfuncs"

I am trying to run a simple multiple processes application in Python. The main thread spawns 1 to N processes and waits until they all done processing. The processes each run an infinite loop, so they can potentially run forever without some user interruption, so I put in some code to handle a KeyboardInterrupt:
#!/usr/bin/env python
import sys
import time
from multiprocessing import Process
def main():
# Set up inputs..
# Spawn processes
Proc( 1).start()
Proc( 2).start()
class Proc ( Process ):
def __init__ ( self, procNum):
self.id = procNum
Process.__init__(self)
def run ( self ):
doneWork = False
while True:
try:
# Do work...
time.sleep(1)
sys.stdout.write('.')
if doneWork:
print "PROC#" + str(self.id) + " Done."
break
except KeyboardInterrupt:
print "User aborted."
sys.exit()
# Main Entry
if __name__=="__main__":
main()
The problem is that when using CTRL-C to exit, I get an additional error even though the processes seem to exit immediately:
......User aborted.
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Python26\lib\atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "C:\Python26\lib\multiprocessing\util.py", line 281, in _exit_function
p.join()
File "C:\Python26\lib\multiprocessing\process.py", line 119, in join
res = self._popen.wait(timeout)
File "C:\Python26\lib\multiprocessing\forking.py", line 259, in wait
res = _subprocess.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
File "C:\Python26\lib\atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "C:\Python26\lib\multiprocessing\util.py", line 281, in _exit_function
p.join()
File "C:\Python26\lib\multiprocessing\process.py", line 119, in join
res = self._popen.wait(timeout)
File "C:\Python26\lib\multiprocessing\forking.py", line 259, in wait
res = _subprocess.WaitForSingleObject(int(self._handle), msecs)
KeyboardInterrupt
I am running Python 2.6 on Windows. If there is a better way to handle multiprocessing in Python, please let me know.

This is a very old question, but it seems like the accepted answer does not actually fix the problem.
The main issue is that you need to handle the keyboard interrupt in the parent process as well. Additionally to that, in the while loop, you just need to exit the loop, there's no need to call sys.exit()
I've tried to as closely match the example in the original question. The doneWork code does nothing in the example so have removed that for clarity.
import sys
import time
from multiprocessing import Process
def main():
# Set up inputs..
# Spawn processes
try:
processes = [Proc(1), Proc(2)]
[p.start() for p in processes]
[p.join() for p in processes]
except KeyboardInterrupt:
pass
class Proc(Process):
def __init__(self, procNum):
self.id = procNum
Process.__init__(self)
def run(self):
while True:
try:
# Do work...
time.sleep(1)
sys.stdout.write('.')
except KeyboardInterrupt:
print("User aborted.")
break
# Main Entry
if __name__ == "__main__":
main()

Rather then just forcing sys.exit(), you want to send a signal to your threads to tell them to stop. Look into using signal handlers and threads in Python.
You could potentially do this by changing your while True: loop to be while keep_processing: where keep_processing is some sort of global variable that gets set on the KeyboardInterrupt exception. I don't think this is a good practice though.