We Keep Coding

PyQt A problem of wait() in QThread using moveToThread() method

I'm trying to write a multi-thread program with QThread in PyQt6.
The example code is below.
I create two threads by moveToThread() method and expect to join both of them after finish, but the result is crushing.
I know the other way is create subclass of QThread, it's easier to write, but I still want to understand why moveToThread() cannot do that.
Thank you!
import sys
import logging
from functools import partial
from PyQt6.QtWidgets import *
from PyQt6.QtCore import *
logging.basicConfig(format="%(message)s", level=logging.INFO)
class MyWork(QObject):
finished = pyqtSignal()
def run_work(self, obj_name):
# sleep 3 secs and finish
for i in range(3):
QThread.sleep(1)
logging.info(f'{obj_name}: {i} sec.')
self.finished.emit()
class MyWindow(QWidget):
def __init__(self):
super().__init__()
self.setWindowTitle('MyWork Example')
app = QApplication(sys.argv)
form = MyWindow()
form.show()
# create two threads
th1 = QThread()
wo1 = MyWork()
wo1.moveToThread(th1)
th1.started.connect(partial(wo1.run_work, 'obj1'))
wo1.finished.connect(th1.quit)
th1.start()
th2 = QThread()
wo2 = MyWork()
wo2.moveToThread(th2)
th2.started.connect(partial(wo2.run_work, 'obj2'))
wo2.finished.connect(th2.quit)
th2.start()
# join two threads and finish
th1.wait()
th2.wait()
logging.info('All threads finished.')
sys.exit(app.exec())
the output:
obj1: 0 sec.
obj2: 0 sec.
obj2: 1 sec.
obj1: 1 sec.
obj2: 2 sec.
obj1: 2 sec.

What you're seeing is caused by the fact that cross-thread signals call their connected functions in the thread of the receiver.
Remember that a QThread (just like a Thread object in python) is not "the thread", but the interface to access and run it.
When you do this:
wo1.finished.connect(th1.quit)
the result is that quit() will be called in the thread in which th1 was created, which is the main thread.
Since wait() blocks the event loop of the thread in which it was called, the call to quit() is queued and never processed.
For this specific case, the solution is to use a direct connection for the signal:
wo1.finished.connect(th1.quit, Qt.DirectConnection)
# ...
wo2.finished.connect(th2.quit, Qt.DirectConnection)
By doing this, quit() will be called from the actual thread, allowing its immediate processing.
Note that yours is a very peculiar case that normally won't be used. You normally connect the worker "finished" signal to both quit() and wait() (in this specific order), or call those functions in the same order when you actually need to quit, or connect the QThread finished signal to the function that eventually will print the completion of the thread execution.

Implementing threading in a Python GTK application (PyGObject) to prevent UI freezing

Simply put, I want to properly implement threading in a Python GTK application. This is in order to prevent UI freezing due to functions/code taking a long time to finish running. Hence, my approach was to move all code which took a long time to run into separate functions, and run them in their separate threads as needed. This however posed a problem when trying to run the functions in sequence.
For example, take a look at the following code:
class Main(Gtk.Window):
def __init__(self):
super().__init__()
self.button = Gtk.Button(label='button')
self.add(self.button)
self.button.connect('clicked', self.main_function)
def threaded_function(self):
time.sleep(20)
print('this is a threaded function')
def first_normal_function(self):
print('this is a normal function')
def second_normal_function(self):
print('this is a normal function')
def main_function(self, widget):
self.first_normal_function()
self.threaded_function()
self.second_normal_function()
Pressing the button starts main_function which then starts 3 functions in sequence. threaded_function represents a function which would take a long time to complete. Running this as is will freeze the UI. Hence it should be threaded as such:
...
...
def main_function(self, widget):
self.first_normal_function()
thread = threading.Thread(target=self.threaded_function)
thread.daemon = True
thread.start()
self.second_normal_function()
What should happen is that the following first_normal_function should run, then threaded_function in a background thread - the UI should remain responsive as the background thread is working. Finally, second_normal_function should run, but only when threaded_function is finished.
The issue with this is that the functions will not run in sequence. The behaviour I am looking for could be achieved by using thread.join() however this freezes the UI.
So I ask, what's the proper way of doing this? This is a general case, however it concerns the general issue of having code which takes a long time to complete in a graphical application, while needing code to run sequentially. Qt deals with this by using signals, and having a QThread emit a finished signal. Does GTK have an equivalent?
I'm aware that this could be partially solved using Queue , with a put() and get() in relevant functions, however I don't understand how to get this to work if the main thread is calling anything other than functions.
EDIT: Given that it's possible to have threaded_function call second_normal_function using GLib.idle_add, let's take an example where in main_function, the second_normal_function call is replaced with a print statement, such that:
def main_function(self, widget):
self.first_normal_function()
thread = threading.Thread(target=self.threaded_function)
thread.daemon = True
thread.start()
print('this comes after the thread is finished')
...
...
...
#some more code here
With GLib.idle_add, the print statement and all the code afterwards would need to be moved into a separate function. Is it possible to avoid moving the print statement into its own function while maintaining sequentiality, such that the print statement remains where it is and still gets called after threaded_function is finished?

Your suggestion on how to do this was very close to the actual solution, but it's indeed not going to work.
In essence, what you'll indeed want to do, is to run the long-running function in a different thread. That'll mean you get 2 threads: one which is running the main event loop that (amongs other things) updates your UI, and another thread which does the long-running logic.
Of course, that bears the question: how do I notify the main thread that some work is done and I want it to react to that? For example, you might want to update the UI while (or after) some complex calculation is going on. For this, you can use GLib.idle_add() from within the other thread. That function takes a single callback as an argument, which it will run as soon as it can ("on idle").
So a possibility to use here, would be something like this:
class Main(Gtk.Window):
def __init__(self):
super().__init__()
self.button = Gtk.Button(label='button')
self.add(self.button)
self.button.connect('clicked', self.main_function)
thread = threading.Thread(target=self.threaded_function)
thread.daemon = True
thread.start()
def threaded_function(self):
# Really intensive stuff going on here
sleep(20)
# We're done, schedule "on_idle" to be called in the main thread
GLib.idle_add(self.on_idle)
# Note, this function will be run in the main loop thread, *not* in this one
def on_idle(self):
second_normal_function()
return GLib.SOURCE_REMOVE # we only want to run once
# ...
For more context, you might want to read the pygobject documentation on threading and concurrency

PySide QThread.terminate() causing fatal python error

I am using PySide version 1.2.2, which wraps the Qt v4.8 framework. I am in a situation where I have to choose between having my application wait for a QThread that I no longer need to exit normally (it is quite possible that the thread will block indefinitely), and giving the unresponsive thread a grace period (of several seconds), then calling QThread.terminate() on it. Though I wish I could, I cannot let the QThread object go out of scope while the underlying thread is still running, since this will throw the error "QThread: Destroyed while thread is still running" and almost surely cause a segfault.
Please note that I am aware that terminating QThreads is dangerous and highly discouraged. I am just trying to explore my options here.
When I try to terminate a thread however, my application crashes with the following error:
Fatal Python error: This thread state must be current when releasing
You can try this out yourself by copy/pasting and running the following code:
from PySide import QtCore, QtGui
class Looper(QtCore.QThread):
"""QThread that prints natural numbers, one by one to stdout."""
def __init__(self, *args, **kwargs):
super(Looper, self).__init__(*args, **kwargs)
self.setTerminationEnabled(True)
def run(self):
i = 0
while True:
self.msleep(100)
print(i)
i += 1
# Initialize and start a looper.
looper = Looper()
looper.start()
# Sleep main thread for 5 seconds.
QtCore.QThread.sleep(5)
# Terminate looper.
looper.terminate()
# After calling terminate(), we should call looper.wait() or listen
# for the QThread.terminated signal, but that is irrelevant for
# the purpose of this example.
app = QtGui.QApplication([])
app.exec_()
How do you properly terminate QThreads in Python?
I reckon that the error I am getting has got something to do with releasing of the Global Interpreter Lock, but I am not sure exactly what is going wrong, and how to fix it.

It seems that the error may be specific to PySide: running your example with PyQt4 does not produce any errors at all.
As for the general issue of how to terminate a QThread safely: it entirely depends on how much control you have over the work that is being done in the thread. If it is effectively a loop where you can periodically check a flag, then the solution is simple:
class Looper(QtCore.QThread):
...
def interrupt(self):
self._active = False
def run(self):
i = 0
self._active = True
while self._active:
self.msleep(100)
print(i)
i += 1
app = QtGui.QApplication([])
looper = Looper()
looper.finished.connect(app.quit)
looper.start()
QtCore.QTimer.singleShot(3000, looper.interrupt)
app.exec_()
The thread will finish cleanly once the run method returns, so you must find some mechanism to allow that happen. If you can't do that (perhaps because the work being done in the thread is largely outside of your control), you should probably consider switching to a multiprocessing approach instead.

Why is the work being done by my QThread starving the main thread?

Here's my lovely thread I've written based on QThread. You'll notice it has an event queue. After 4 seconds an event fires and does some work in doWork. doWork should sleep in between all its printing and give other threads a chance to run. Suffice it to say with all the printing and sleeping doWork runs long enough that another thread really should get some time to execute.
from PySide.QtCore import *
from PySide.QtGui import *
class DoStuffPeriodically(QThread):
def __init__(self):
super(DoStuffPeriodically, self).__init__()
def doWork(self):
#... do work, post signal to consumer
print "Start work"
for i in range(0,100):
print "work %i" % i
QThread.msleep(10)
print "Done work"
return
def run(self):
""" Setup "pullFiles" to be called once a second"""
self.timer= QTimer()
self.timer.setSingleShot(True)
self.timer.timeout.connect(self.doWork)
self.timer.start(4000)
self.exec_()
Here's the top-level QT widget I'm using to control my thread. Its basically just a push button that starts/stops the thread.
class Widg(QWidget):
def __init__(self):
super(Widg, self).__init__()
self.thread = DoStuffPeriodically()
self.startStopButton = QPushButton()
hBoxLayout = QHBoxLayout()
hBoxLayout.addWidget(self.startStopButton)
self.startStopButton.pressed.connect(self.startStopThread)
self.setLayout(hBoxLayout)
self.threadRunning = False
def startStopThread(self):
if self.threadRunning:
print "Stopping..."
self.thread.exit(0)
self.threadRunning = False
print "Stopped"
else:
print "Starting..."
self.thread.start()
self.threadRunning = True
print "Started"
if __name__ == "__main__":
from sys import argv
qApp = QApplication(argv)
widg = Widg()
widg.show()
qApp.exec_()
If I click the startStopButton, I expect to see the thread begin printing
Starting...
Started...
Start Work
work 0
work 1
...
work 99
Done Work
But what I want to do is to be able to stop the thread while its doing work. I expect something along the lines of
Starting...
Started...
Start Work
work 0
work 1
...
work N
Stopping...
work 99
Done Work
Stopped...
Instead, the worker thread appears to be preventing the main thread from executing? And I have to wait for the work to be done before I can click the startStopButton, giving me
Starting...
Started...
Start Work
work 0
work 1
...
work 99
Done Work
Stopping...
Stopped...
It doesn't matter how long doWork runs. I`ve upped it to loop 10000 times. It doesn't appear to ever give time back to the main thread and the widget is unresponsive. Am I doing something thats preventing real threading from actually working?
(I'm using python 2.7 and pyside 1.10.)
Update
If I modify run to do the work directly, not based on the QTimer the threading appears to work correctly. Ie change run to:
def run(self):
self.doWork()
return
This doesn't solve my problem, because I want to run using the event queue. I suspect therefore, that this is some kind of signals/slots problem where the QTimer signal is associated with the wrong thread.
Note I'm not exeriencing that exit or quit blocks until the work is done. I'm simply experiencing the threading not work at all. Namely the main window is blocked and I can't even click the button to even initiating quiting the thread

The problem is that the QThread method is doing the work. The thread affinity of QThread is always the thread that created the QThread. Therefore, the signal tells QThread's owning thread to execute doWork--in this case the main thread. So even though doWork is defined in this QThread, the work is done by the main thread. I know kind of mind twisting. To explain, let me begin by quoting the docs
QThread object is living in another thread, namely, the one in which it was created.
so when this signal/slot connection is setup
self.timer.timeout.connect(self.doWork)
it, by default is an AutoConnection:
(default) If the signal is emitted from a different thread than the receiving object, the signal is queued, behaving as Qt::QueuedConnection. Otherwise, the slot is invoked directly, behaving as Qt::DirectConnection. The type of connection is determined when the signal is emitted.
The source of the signal is my QThread, because QTimer was created in the run method, but the destination is the main thread. Its being queued in the main thread's event queue! The solution is to create a second worker QObject which will have the affinity of the current thread:
class Worker(QObject):
def __init__(self, parent):
super(Worker, self).__init__(parent=parent)
def doWork(self):
#... do work, post signal to consumer
print "Start work"
for i in range(0,1000):
print "work %i" % i
QThread.msleep(100)
print "Done work"
return
Then run becomes:
def run(self):
""" Setup "pullFiles" to be called once a second"""
print "Running..."
self.worker = Worker(parent=None) #affinity = this thread
self.timer= QTimer() #affinity = this thread
print self.timer.thread()
self.timer.setSingleShot(True)
self.timer.timeout.connect(self.worker.doWork)
self.timer.start(4000)
self.exec_()
print "Exec_ done"
And this works. The source and destination of the signal is all in one thread and doesn't traverse back to the main thread. Voila!

From QThread::exit() documentation:
Tells the thread's event loop to exit with a return code.
Your doWork is a single event in the event loop. The event loop called your event, therefore it waits for it to finish. exit is yet another event, enqueued in the event loop and waiting for doWork to finish. The msleep helps the responsiveness of your GUI (gives it time to repaint and execute the button handler), but it really does not enable the exit event to sneak in somehow.
If you want your doWork to be interruptable at any time, you must change your logic. Make the timer fire more often and increment only by one. exit then can act anytime in between.

Is it still not enough to simply use threads to update GUI?

For example:
class DemoFrame(wx.Frame):
def __init__(self):
Initializing
...
self.TextA = wx.StaticText(MainPanel, id = -1, label = "TextAOrWhatever")
self.TextB = wx.StaticText(MainPanel, id = -1, label = "TextBOrWhatever")
...
def StaticTextUpdating(self, ObjectName, Message):
ObjectName.SetLabel(Message)
def WorkerA(self):
while True:
Work on something
UpdatingThread = threading.Thread(target = self.StaticTextUpdating, args = (self.TextA, "Something for TextA", ))
UpdatingThread.start()
time.sleep(randomSecs)
def WorkerB(self):
while True:
Work on something
UpdatingThread = threading.Thread(target = self.StaticTextUpdating, args = (self.TextB, "Something for TextB", ))
UpdatingThread.start()
time.sleep(randomSecs)
...
def StartWorking(self):
Spawn WorkerA thread
Spawn WorkerB thread
...
As you can see, I always update StaticText in new threads, and I'm 100% sure at a whatever certain time point there's only one thread updating a specific object, but the problem is, every now and then after running for a while, some objects just disappear. Why is this happening? Does it mean GUI updating is not thread safe? Maybe only one object can be updated at a certain time point?
Added:
OK, wx.CallAfter should be a good solution for above codes. But I got another question, what if a button event and SetLabel happens at the same time? Wouldn't things like this cause troubles although I don't see any?

Most wx methods are not thread-safe. Use wx.CallAfter if you want to invoke a wx method from another thread; replace
ObjectName.SetLabel(Message)
with:
wx.CallAfter(ObjectName.SetLabel, Message)
Edit: Some Background Information
In wx (And in most other UI platforms) all the UI updates get executed in a single thread called main thread (Or UI Thread). This is to make the UI work faster by avoiding the performance hit of thread synchronization.
But the down side of this is that If we write code to update the UI from a different thread the results are undefined. Sometimes it may work, sometimes it may crash, sometimes some other thing may happen. So we should always go to UI thread to do the UI updates. So we use CallAfter function to make UI update function execute in the UI thread.
UI thread in java
UI thread in C#

The main thing to remember is that you shouldn't update anything in wxPython without using a threadsafe method, such as wx.CallAfter, wx.CallLater or wx.PostEvent. See http://wiki.wxpython.org/LongRunningTasks or http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/ for more information.