I have some code that is multi-threaded and any time I introduce a bug in any non-main thread, that thread fails silently, and that makes it difficult to debug the program. What is the best way to handle exceptions in threads that aren't the main thread? Ideally I just want uncaught exceptions to kill the process like they would in the main thread.
For example in this code the result will never be changed because I try to reference a function on an object that is set to None:
from threading import Thread
def thread_func(args):
global result
bad_obj = None
bad_obj.call_function() # Should cause error
result = 2
result = 1
thread = Thread(target=thread_func, args=None)
thread.start()
while True:
print(result)
I really would prefer not to surround the entire thread function in a try catch, is there a way to make uncaught exceptions non silent at least?
Thread class catches exceptions in the child thread and only reports them when Thread.join is called.
One work-around is to override Thread.run method in a derived class and use that instead of Thread:
from threading import Thread
import sys
import os
class MyThread(Thread):
def run(self):
try:
super(MyThread, self).run()
except Exception as e:
print("error:", e, file=sys.stderr)
os._exit(1)
Note that it uses os._exit because sys.exit works by throwing SystemExit exception and that doesn't terminate the process until Thread.join is called.
Related
I've made a program which has a main thread that spawns many other threads by subclassing the threading.Thread class.
Each such child thread runs an infinite while loop, and inside the loop I check a condition. If the condition is true, I make the thread sleep for 1 second using time.sleep(1) and if it's false, then the thread performs some computation.
The program itself works fine and I've achieved what I wanted to do, my only remaining problem is that I seem unable to stop the threads after my work is done. I want the user to be able to kill all the threads by pressing a button or giving a keyboard interrupt like Ctrl+C.
For this I had tried using the signal module and inserted a conditon in the threads' loops that breaks the loop when the main thread catches a signal but it didn't work for some reason. Can anyone please help with this?
EDIT: This is some of the relevant code snippets:
def sighandler(signal,frame):
BaseThreadClass.stop_flag = True
class BaseThreadClass(threading.Thread):
stop_flag = False
def __init__(self):
threading.Thread.__init__(self)
def run(self,*args):
while True:
if condition:
time.sleep(1)
else:
#do computation and stuff
if BaseThreadClass.stop_flag:
#do cleanup
break
Your basic method does work, but you've still not posted enough code to show the flaw. I added a few lines of code to make it runnable and produced a result like:
$ python3 test.py
thread alive
main alive
thread alive
main alive
^CSignal caught
main alive
thread alive
main alive
main alive
main alive
^CSignal caught
^CSignal caught
main alive
^Z
[2]+ Stopped python3 test.py
$ kill %2
The problem demonstrated above involves the signal handler telling all the threads to exit, except the main thread, which still runs and still catches interrupts. The full source of this variant of the sample snippet is:
import threading, signal, time
def sighandler(signal,frame):
BaseThreadClass.stop_flag = True
print("Signal caught")
class BaseThreadClass(threading.Thread):
stop_flag = False
def __init__(self):
threading.Thread.__init__(self)
def run(self,*args):
while True:
if True:
time.sleep(1)
print("thread alive")
else:
#do computation and stuff
pass
if BaseThreadClass.stop_flag:
#do cleanup
break
signal.signal(signal.SIGINT, sighandler)
t = BaseThreadClass()
t.start()
while True:
time.sleep(1)
print("main alive")
The problem here is that the main thread never checks for the quit condition. But as you never posted what the main thread does, nor how the signal handler is activated, or information regarding whether threads may go a long time without checking the quit condition... I still don't know what went wrong in your program. The signal example shown in the library documentation raises an exception in order to divert the main thread.
Signals are a rather low level concept for this task, however. I took the liberty of writing a somewhat more naïve version of the main thread:
try:
t = BaseThreadClass()
t.start()
while True:
time.sleep(1)
print("main alive")
except KeyboardInterrupt:
BaseThreadClass.stop_flag = True
t.join()
This version catches the exception thrown by the default interrupt handler, signals the thread to stop, and waits for it to do so. It might even be appropriate to change the except clause to a finally, since we could want to clean the threads up on other errors too.
If you want to do this kind of "cooperative" polled-shutdown, you can use a threading.Event to signal:
import threading
import time
def proc1():
while True:
print("1") # payload
time.sleep(1)
# have we been signalled to stop?
if not ev1.wait(0): break
# do any shutdown etc. here
print ("T1 exiting")
ev1 = threading.Event()
ev1.set()
thread1 = threading.Thread(target=proc1)
thread1.start()
time.sleep(3)
# signal thread1 to stop
ev1.clear()
But be aware that if the "payload" does something blocking like network or file IO, that op will not be interrupted. You can do those blocking ops with a timeout, but that obviously will complicate your code.
I have a GUI (PySide) application that uses QThread. I have a signal in my QThread that is emitted when an exception occurs so that I can handle the exception in the main thread. However, the rest of the function starting the thread is still executed. I tried the wait function to block the execution but it does not work. Here is my implementation:
QThread daughter
class LongTaskThread(QtCore.QThread):
task_finished = QtCore.Signal()
task_failed = QtCore.Signal(Exception)
def __init__(self, allow_log=True, test_mode=False, parent=None):
QtCore.QThread.__init__(self, parent)
def run(self):
self.task_failed.emit(Exception())
def wait_with_gui_refresh(self):
while self.isRunning():
time.sleep(0.1)
if not self.test_mode:
QtGui.QApplication.processEvents()
Main thread
def test():
my_thread = LongTaskThread()
my_thread.task_finished.connect(on_finished)
my_thread.task_failed.connect(on_failed)
my_thread.start()
# my_thread.wait() <---- tentative 1
# my_thread.wait_with_gui_refresh() <---- tentative 2
print('bla bla bla bla')
def on_finished)():
pass
def on_failed(err):
raise err
I expected that the print would never been hit, but whether I use the wait function or the wait_with_gui_refresh function, or nothing, the print is always printed.
How to stop the test function when an exception is raised inside the QThread ?
In your test function, the sequence of events is this:
The thread starts
The thread's run method is called
The task_failed signal is emitted asynchronously (i.e. it's posted to the receiver's event queue)
The thread's run method returns
If the thread's wait method is called here, it will return True immediately because there is nothing to wait for (i.e. run has already returned)
A message is printed, and test returns
Control returns to the event-loop, and the task_failed signal is processed
An exception is raised in on_failed
It's hard to see anything to object to here. Presumably, you don't want to block the gui whilst the worker thread is running, so it makes perfect sense to process any exceptions aynchronously. But for that to happen, control must return to the event-loop of the main thread - which means the test function must return immediately. If you want to run some code after the thread starts, connect a slot to its started signal.
I am trying to create a watchdog class, that will throw an exception after specified time:
from threading import Timer
from time import sleep
class watchdog():
def _timeout(self):
#raise self
raise TypeError
def __init__(self):
self.t = Timer(1, self._timeout)
def start(self):
self.t.start()
try:
w = watchdog()
w.start()
sleep(2)
except TypeError, e:
print "Exception caught"
else:
print "Of course I didn't catch the exception"
This exception is not caught, as the exception is thrown from completely different context, hence we will see the last message.
My question is, how can I modify the code, so the exception will be caught?
This is not possible, as you suggested, and there is no api for abruptly stopping a thread, either, which rules out other potential solutions.
I believe your best solution is to let the watchdog set a flag, and let the test read it at certain points. Similarly, your test can simply check the duration from time to time.
Note that if the "flag" would set in a way that will cause the main thread to raise an exception (for example, deleting attributes from objects), it'll be just as effective.
The other possibility is to use multiprocessing instead of multythreading, if it is possible for your application.
I have a Python GUI that I use to test various aspects of my work. Currently I have a "stop" button which kills the process at the end of each test (there can be multiple tests set up to run at once). However, some tests take a long time to run and if I need to stop the test I would like it to stop instantly. My thoughts are to use
import pdb; pdb.set_trace()
exit
But I'm not sure how I would inject this into the next run line of code. Is this possible?
If it's a thread, you can use the lower-level thread (or _thread in Python 3) module to kill the thread with an exception by calling thread.exit().
From the documentation:
thread.exit(): Raise the SystemExit exception. When not caught,
this will cause the thread to exit silently.
A cleaner method (depending on how your processing is set up) would be to signal the thread to stop processing and exit using an instance variable, then calling the join() method from your main thread to wait until the thread exits.
Example:
class MyThread(threading.Thread):
def __init__(self):
super(MyThread, self).__init__()
self._stop_req = False
def run(self):
while not self._stop_req:
pass
# processing
# clean up before exiting
def stop(self):
# triggers the threading event
self._stop_req = True;
def main():
# set up the processing thread
processing_thread = MyThread()
processing_thread.start()
# do other things
# stop the thread and wait for it to exit
processing_thread.stop()
processing_thread.join()
if __name__ == "__main__":
main()
I'm trying to create a method which runs on a timer in the background of my main script:
def hello_world(self):
print 'Hello!'
threading.Timer(2,hello_world).start()
if __name__ == "__main__":
try:
hello_world()
except KeyboardInterrupt:
print '\nGoodbye!'
I am getting this message when I attempt to keyboard interrupt my script:
Exception KeyboardInterrupt in <module 'threading' from '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py'> ignored
How do I close the thread so that I can exit my application cleanly?
To elaborate a bit on Aphex's answer, the main thread can't possibly catch the KeyboardInterrupt signal, unless you have very fast fingers. The main thread exits almost immediately! Try this:
import threading
def hello_world():
print 'Hello!'
threading.Timer(2,hello_world).start()
if __name__ == "__main__":
try:
hello_world()
except KeyboardInterrupt:
print '\nGoodbye!'
print "main thread exited"
More generally, I would not suggest using a self-calling timer like this, just because it creates a lot of threads. Just create one thread and call time.sleep inside it.
However, as long as you keep the main thread running, you seem to be able to catch KeyboardInterrupt inside. The trick then is to make the thread a daemon thread that exits when the main thread exits.
import threading
import time
def hello_world():
while(True):
print 'Hello!'
time.sleep(2)
if __name__ == "__main__":
hw_thread = threading.Thread(target = hello_world)
hw_thread.daemon = True
hw_thread.start()
try:
time.sleep(1000)
except KeyboardInterrupt:
print '\nGoodbye!'
This exits automatically after 1000 seconds -- you could make that number even bigger if you like. You could also use a busy-loop to repeat the sleep call, but I don't really see the point.
You just need to set the Timer thread as a daemon
def hello_world(self):
print 'Hello!'
t = threading.Timer(2,hello_world)
t.daemon = True
t.start()
That will cause it to exit when the main thread exits, e.g. due to KeyboardInterrupt.
The daemon setting causes the entire program to exit when the only threads left are daemon threads.
Try re-raising the KeyboardInterrupt exception: http://effbot.org/zone/stupid-exceptions-keyboardinterrupt.htm
This still may not work though; chances are you're running into this caveat:
Threads interact strangely with interrupts: the KeyboardInterrupt exception will be received by an arbitrary thread. (When the signal module is available, interrupts always go to the main thread.)
In short, you can't be sure that the KeyboardInterrupt is going to your main thread. To workaround this, you may want to look into the signal module.
Edit: A more elegant way to cancel the thread is to have a shared variable that the thread looks at, and exits if it becomes false. Then if you want to kill the thread from your main thread, you set the variable to false.