I need to pass exceptions across a thread boundary.
I'm using python embedded in a non thread safe app which has one thread safe call, post_event(callable), which calls callable from its main thread.
I am running a pygtk gui in a seperate thread, so when a button is clicked I post an event with post_event, and wait for it to finish before continuing. But I need the caller to know if the callee threw an exception, and raise it if so. I'm not too worried about the traceback, just the exception itself.
My code is roughly:
class Callback():
def __init__(self,func,*args):
self.func=func
self.args=args
self.event=threading.Event()
self.result=None
self.exception=None
def __call__(self):
gtk.gdk.threads_enter()
try:
self.result=self.func(*self.args)
except:
#what do I do here? How do I store the exception?
pass
finally:
gtk.gdk.threads_leave()
self.event.set()
def post(self):
post_event(self)
gtk.gdk.threads_leave()
self.event.wait()
gtk.gdk.threads_enter()
if self.exception:
raise self.exception
return self.result
Any help appreciated, thanks.
#what do I do here? How do I store the exception?
Use sys.exc_info()[:2], see this wiki
Best way to communicate among threads is Queue. Have the main thread instantiate a Queue.Queue instance and pass it to subthreads; when a subthread has something to communicate back to the master it uses .put on that queue (e.g. a tuple with thread id, exception type, exception value -- or, other useful info, not necessarily exception-related, just make sure the first item of a tuple identifies the kind of info;-). The master can .get those info-units back when it wants, with various choices for synchronization and so on.
Related
I have some code that is multi-threaded and any time I introduce a bug in any non-main thread, that thread fails silently, and that makes it difficult to debug the program. What is the best way to handle exceptions in threads that aren't the main thread? Ideally I just want uncaught exceptions to kill the process like they would in the main thread.
For example in this code the result will never be changed because I try to reference a function on an object that is set to None:
from threading import Thread
def thread_func(args):
global result
bad_obj = None
bad_obj.call_function() # Should cause error
result = 2
result = 1
thread = Thread(target=thread_func, args=None)
thread.start()
while True:
print(result)
I really would prefer not to surround the entire thread function in a try catch, is there a way to make uncaught exceptions non silent at least?
Thread class catches exceptions in the child thread and only reports them when Thread.join is called.
One work-around is to override Thread.run method in a derived class and use that instead of Thread:
from threading import Thread
import sys
import os
class MyThread(Thread):
def run(self):
try:
super(MyThread, self).run()
except Exception as e:
print("error:", e, file=sys.stderr)
os._exit(1)
Note that it uses os._exit because sys.exit works by throwing SystemExit exception and that doesn't terminate the process until Thread.join is called.
I've wrote a class that inherits from object and has instances of sub-objects that uses some threads for tasks. There are two socket listeners that creates other threads for each accepted connection. They do what they have to do. To finish them, they are looking on a Threading.Event object to know that they have to finish.
I've noticed that, when exit the python console they are not notified (or don't catch the notification) and the exit don't return control to the bash console, unless a Close() is called before.
First idea to fix it has been to implement the '__del__' method to use the garbage collector to clean it when exit.
class ServiceProvider(object):
def __init__(self):
super(ServiceProvider,self).__init__()
#...
self.Open()
def Open(self):
#... Some threads are created.
def Close(self):
#.... Threading.Event to report the threads to finish
def __del__(self):
self.Close()
But the behaviour is the same. If I place a print in those methods, non in '__del__', neither in 'Close' they are written. Unless it is closed before, then the print in the del is wrote.
Then I've implemented the '__enter__' and '__exit__' methods to manage the with statement. And the exit behaves as expected and when the with ends, things are release. But what I really want is to have something like the file descriptors that event if file.close() is not called, it is executed when exits the program.
class ServiceProvider(object):
#...
def __enter__(self):
return self
def __exit__(self):
self.Close()
Searching for more solutions I've tried with atexit but not. I have similar results that doesn't fix the issue. Even I collect all the objects created of this class, the doOnExit only writes its print if the objects in the list are already Close.
import atexit
global objects2Close
objects2Close = []
#atexit.register
def doOnExit():
for obj in objects2Close:
obj.Close()
class ServiceProvider(object):
def __init__(self):
super(ServiceProvider,self).__init__()
objects2Close.append(self)
It's usually a good idea to use with when you have resources that you don't want to leak (files, connections, whatever else you care about).
Somewhere, just outside your main loop you should have something like:
with ServiceProvider(some_params) as service_provider:
rest_of_the_code()
What this does is that regardless of how you exit rest_of_the_code() (except for kill -9) it will call service_provider.Close() at the end. This works for exceptions and interrupts as well. Kill -9 doesn't work because the process is kill at os level and doesn't have a chance to attempt to recover.
I've got a solution for this issue. The posted information in this question was not related with the real issue.
This is as simple as daemon threading.
A the implementation uses some threads for listening remote connections they have to finish their execution when the program goes to exit. But the program ends when all the no daemon thread has finished.
Mistakenly those listeners and talkers where not set to be daemons and that's why the execution waits for them.
I am trying to create a watchdog class, that will throw an exception after specified time:
from threading import Timer
from time import sleep
class watchdog():
def _timeout(self):
#raise self
raise TypeError
def __init__(self):
self.t = Timer(1, self._timeout)
def start(self):
self.t.start()
try:
w = watchdog()
w.start()
sleep(2)
except TypeError, e:
print "Exception caught"
else:
print "Of course I didn't catch the exception"
This exception is not caught, as the exception is thrown from completely different context, hence we will see the last message.
My question is, how can I modify the code, so the exception will be caught?
This is not possible, as you suggested, and there is no api for abruptly stopping a thread, either, which rules out other potential solutions.
I believe your best solution is to let the watchdog set a flag, and let the test read it at certain points. Similarly, your test can simply check the duration from time to time.
Note that if the "flag" would set in a way that will cause the main thread to raise an exception (for example, deleting attributes from objects), it'll be just as effective.
The other possibility is to use multiprocessing instead of multythreading, if it is possible for your application.
I want to add a timeout on one function which is getting called inside a child thread.
I can't use a signal, as a signal should be on the main thread.
I can't use thread.join(time_out), as that function can sometimes be executed in a few seconds, and in those cases the thread will always wait out the time_out.
Are there any other approaches?
Sources:
thread.join: Timeout function using threading in python does not work
signal: Timeout function if it takes too long to finish
You can use a little known ctyps hack to raise a TimeoutError targeting a specific thread. I made a non_blocking timeout script using this method and just released it on GitHub: https://github.com/levimluke/PyTimeoutAfter
It's SUPER simple to implement, but technically complex.
def raise_caller(self):
ret = ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(self._caller_thread._ident), ctypes.py_object(self._exception))
if ret == 0:
raise ValueError("Invalid thread ID")
elif ret > 1:
ctypes.pythonapi.PyThreadState_SetAsyncExc(self._caller_thread._ident, NULL)
raise SystemError("PyThreadState_SetAsyncExc failed")
https://github.com/levimluke/PyTimeoutAfter/blob/main/timeout_after.py#L47
I use a class object and save the calling thread to, that way I can raise an exception in the parent class from within the timed child thread.
Im trying to send a signal from a non-main thread in PyQt but i dont know what am doing wrong! And when i execute the program it fails with this error:
QObject::connect: Cannot queue arguments of type 'QTextCursor'
(Make sure 'QTextCursor' is registered using qRegisterMetaType().)
here is my code:
class Sender(QtCore.QThread):
def __init__(self,q):
super(Sender,self).__init__()
self.q=q
def run(self):
while True:
pass
try: line = q.get_nowait()
# or q.get(timeout=.1)
except Empty:
pass
else:
self.emit(QtCore.SIGNAL('tri()'))
class Workspace(QMainWindow, Ui_MainWindow):
""" This class is for managing the whole GUI `Workspace'.
Currently a Workspace is similar to a MainWindow
"""
def __init__(self):
try:
from Queue import Queue, Empty
except ImportError:
while True:
#from queue import Queue, Empty # python 3.x
print "error"
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
p= Popen(["java -Xmx256m -jar bin/HelloWorld.jar"],cwd=r'/home/karen/sphinx4-1.0beta5-src/sphinx4-1.0beta5/',stdout=PIPE, shell=True, bufsize= 4024)
q = Queue()
t = threading.Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()
self.sender= Sender(q)
self.connect(self.sender, QtCore.SIGNAL('tri()'), self.__action_About)
self.sender.start()
I think that my way of send parameter to the thread is wrong...
I need to know how to send parameters to a thread, in my case i need to send q to the worker thread.
Quite new to PyQt5, but this appears to happen when you try to do a GUI operation from a thread which is not the "application thread". I put this in quotes because it appears to be a mistake to think that, even in a fairly simple PyQt5 app, QApplication.instance().thread() will always return the same object.
The thing to do is to use the signal/slot mechanism to send any kind of data from a worker thread (a thread created in my case by extending QtCore.QRunnable, one other pattern apparently being QtCore.QThread and QtCore.QObject.moveToThread, see here).
Then also include a check in all your slot methods which are likely to receive data from a non-"application thread". Example which logs messages visually during execution:
def append_message(self, message):
# this "instance" method is very useful!
app_thread = QtWidgets.QApplication.instance().thread()
curr_thread = QtCore.QThread.currentThread()
if app_thread != curr_thread:
raise Exception('attempt to call MainWindow.append_message from non-app thread')
ms_now = datetime.datetime.now().isoformat(sep=' ', timespec='milliseconds')
self.messages_text_box.insertPlainText(f'{ms_now}: {message}\n')
# scroll to bottom
self.messages_text_box.moveCursor(QtGui.QTextCursor.End)
It's all too easy to just call this inadvertently and directly from a non-"application thread".
Making such a mistake then raise an exception is good, because it gives you a stack trace showing the culprit call. Then change the call so that it instead sends a signal to the GUI class, the slot for which could be the method in the GUI class (here append_message), or alternatively one which then in turn calls append_message.
In my example I've included the "scroll to bottom" line above because it was only when I added that line that these "cannot queue" errors started happening. In other words, it is perfectly possible to get away with a certain amount of non-compliant handling (in this case adding some more text with each call) without any error being raised... and only later do you then run into difficulties. To prevent this, I suggest that EVERY method in a GUI class with GUI functionality should include such a check!
Make sure 'QTextCursor' is registered using qRegisterMetaType().
Did you try to use qRegisterMetaType function?
The official manual says:
The class is used as a helper to marshall types in QVariant and in
queued signals and slots connections. It associates a type name to a
type so that it can be created and destructed dynamically at run-time.
Declare new types with Q_DECLARE_METATYPE() to make them available to
QVariant and other template-based functions. Call qRegisterMetaType()
to make type available to non-template based functions, such as the
queued signal and slot connections.
I would like to add the following notes to the #mike rodent's post which solved my problem (I'm using PyQt5):
Custom signals and slots can be used to avoid directly modifying GUI from thread other than "application thread" (I'm using Python threading module and the equivalent there to that is probably "main thread"). I find this website very useful for basic custom signal and slot setup. Pay attention to using a class (and not an instance) attribute.
To avoid the QObject::connect: Cannot queue arguments of type 'QTextCursor' message I needed to find the following locations and add some code:
Before the function __init__ of the class MainWindow: definition of class attribute; I needed to use something like class_attribute = pyqtSignal(str).
In the function __init__: self.class_attribute.connect(self.slot_name)
Inside of a thread (I mean the thread which is not the main thread): self.class_attribute.emit(str)
In the slot inside the main thread: "safety mechanism" proposed by #mike rodent.