Real time plotting with Matplotlib, PyQt and Threading ends to python crash - python

I've been struggling with my Python application and I can't find any answers.
I'm having PyQT GUI application which uses Matplotlib widget. GUI starts a new thread which handles plotting to mpl widget. I'm afraid I run now to a race condition by accessing matplotlib drawing components from another thread which leads to crash.
This is basically, what my code looks like:
class Analyzer(QMainWindow, Ui_MainWindow):
def __init__(self, parent=None):
self.timer = QTimer()
super(Analyzer, self).__init__(parent)
self.setupUi(self)
self.background = self.mpl.canvas.copy_from_bbox(self.mpl.canvas.ax.bbox)
self.plotQueue = Queue.Queue()
self.plotterStarted = False
self.plotter = Plotter(self.mpl, self.plotQueue)
self.cam = Cam(self.plotQueue, self.textEdit)
...
class Ui_MainWindow(object):
def setupUi(self, MainWindow):
...
self.mpl = MplWidget(self.centralWidget)
...
class MplWidget(QtGui.QWidget):
"""Widget defined in Qt Designer"""
def __init__(self, parent = None):
QtGui.QWidget.__init__(self, parent)
self.canvas = MplCanvas()
...
class MplCanvas(FigureCanvas):
"""Class to represent the FigureCanvas widget"""
def __init__(self):
# setup Matplotlib Figure and Axis
self.fig = Figure()
self.ax = self.fig.add_subplot(111)
# initialization of the canvas
FigureCanvas.__init__(self, self.fig)
FigureCanvas.updateGeometry(self)
And plotter class:
class Plotter():
def __init__(self, mpl="", plotQueue=""):
self.mpl = mpl
self.background = self.mpl.canvas.copy_from_bbox(self.mpl.canvas.ax.bbox)
self.plotQueue = plotQueue
...
def start(self):
threading.Thread(target=self.run).start()
''' Real time plotting '''
def run(self):
while True:
try:
inputData = self.plotQueue.get(timeout=1)
# Go through samples
for samples in inputData:
self.line, = self.mpl.canvas.ax.plot(x, y, animated=True, label='Jee')
for sample in samples:
x.append(sample['tick'])
y.append(sample['linear'])
self.line.set_data(x,y)
self.mpl.canvas.ax.draw_artist(self.line)
self.mpl.canvas.blit(self.mpl.canvas.ax.bbox)
...
So I pass mpl and plotQueue to Plotter class object. PlotQueue is populated in Cam class which processes incoming data from external hw. Plotter reads the plotQueue, processes it and calls drawing for mpl.
But is this a thread safe method to access mpl? If not, how should I do it? Any tips on this are appreciated.
Edit 1.
I added QTimer in main thread to handle drawing, as suggested in comments. After a small tweaking, I got it working fairly well.
class Analyzer(...):
def __init__(self, parent=None):
QObject.connect(self.timer, SIGNAL("timeout()"), self.periodicCall)
def periodicCall(self):
self.plotter.draw()
def startButton(self):
self.timer.start(10)
Thanks a lot for useful comments.

If matplotlib in your program is using the QT backend (which I assume it is since you are embedding it in a Qt application), then the drawing is going to be done in thread you call the matplotlib commands from. This is going to be a problem because Qt requires that all drawing is done from the main thread. So I'm fairly certain you can't fix it simply. (if you were using GTK you could use the gtk lock to prevent the main process from interacting with the GUI while you did GUI related things from your thread, but Qt got rid of their similar lock in v4 and above).
You have a few options:
Try and separate out the drawing parts of matplotlib (may not even be possible?) and have them run in the main thread by sending events with QApplication.postEvent()
Instead of using a thread, just use callbacks in the main thread (maybe called periodically using a QTimer or when the program is idle). This probbaly won't impact the performance of your application since the Python GIL prevents true multi-threading behaviour anyway.
Use a different plotting library. I had a look over PyQtGraph the other day, and it seems to be coming along nicely. From my brief glance I think it has the ability to handle all of this behind the scenes for you, using a RemoteGraphicsView. This would launch a second process for doing the CPU intensive plotting stuff, which gets around the aforementioned Python GIL issue. Check out the examples they provide if you are interested

Related

Keep PyQt UI Responsive With Threads

I have created a relatively complex PyQt program and am trying to implement threads so that when the program encounters a part of the program which is particularly CPU intensive, the GUI will remain refreshed and responsive throughout. Sadly though, I am having some difficulties with the threading.
I am using Python 2.7 for reasons that I don't believe to be relevant.
Anyway, the entire program runs within one class and calls upon a PyQt designer .ui file in order to display the actual GUI. When a particular button is pressed, in order to shred a file, it calls a function within that class that then starts a thread using the 'thread' module, yes, outdated, I know. The shredding function that is then called from this commences the shredding of the file. Throughout the shredding of the file, the actual shredding function interacts and adds bits to the GUI in order to keep the user up to date on what is happening.
During the execution of the function the GUI continues to be refreshed, however it does become a little laggy, I can cope with that. However, when that function is complete, instead of smoothly continuing and allowing the user to keep using the program, the program throws a complete hissy fit and simply just stops working and has to be closed.
Hopefully someone can assist me here. I would greatly appreciate as much detail as possible as I have been searching around for a way to cope with this for a good number of weeks now.
I am using PyQt4.
Here's a simple demo of threading in pyqt5. Qt has it's own threading class that works pretty well.
from PyQt5 import QtCore, QtGui, QtWidgets
from PyQt5.QtCore import pyqtSignal
import sys
import time
class TheBoss(QtWidgets.QWidget):
def __init__(self, parent=None):
super(TheBoss, self).__init__(parent)
self.resize(300,200)
self.VL = QtWidgets.QVBoxLayout(self)
self.label = QtWidgets.QLabel()
self.VL.addWidget(self.label)
self.logger = Logger()
self.logger.sec_signal.connect(self.label.setText)
self.logger.start()
def closeEvent(self,event):
self.logger.terminate()
class Logger(QtCore.QThread):
sec_signal = pyqtSignal(str)
def __init__(self, parent=None):
super(Logger, self).__init__(parent)
self.current_time = 0
self.go = True
def run(self):
#this is a special fxn that's called with the start() fxn
while self.go:
time.sleep(1)
self.sec_signal.emit(str(self.current_time))
self.current_time += 1
if __name__ == '__main__':
app = QtWidgets.QApplication(sys.argv)
app.setApplicationName("Thread Example")
window = TheBoss()
window.show()
sys.exit(app.exec_())

Qthread is still working when i close gui on python pyqt

my code has thread, but when i close the gui, it still works on background. how can i stop threads? is there something stop(), close()?
i dont use signal, slots? Must i use this?
from PyQt4 import QtGui, QtCore
import sys
import time
import threading
class Main(QtGui.QMainWindow):
def __init__(self, parent=None):
super(Main, self).__init__(parent)
self.kac_ders=QtGui.QComboBox()
self.bilgi_cek=QtGui.QPushButton("Save")
self.text=QtGui.QLineEdit()
self.widgetlayout=QtGui.QFormLayout()
self.widgetlar=QtGui.QWidget()
self.widgetlar.setLayout(self.widgetlayout)
self.bilgiler=QtGui.QTextBrowser()
self.bilgi_cek.clicked.connect(self.on_testLoop)
self.scrollArea = QtGui.QScrollArea()
self.scrollArea.setWidgetResizable(True)
self.scrollArea.setWidget(self.widgetlar)
self.analayout=QtGui.QVBoxLayout()
self.analayout.addWidget(self.text)
self.analayout.addWidget(self.bilgi_cek)
self.analayout.addWidget(self.bilgiler)
self.centralWidget=QtGui.QWidget()
self.centralWidget.setLayout(self.analayout)
self.setCentralWidget(self.centralWidget)
def on_testLoop(self):
self.c_thread=threading.Thread(target=self.kontenjan_ara)
self.c_thread.start()
def kontenjan_ara(self):
while(1):
self.bilgiler.append(self.text.text())
time.sleep(10)
app = QtGui.QApplication(sys.argv)
myWidget = Main()
myWidget.show()
app.exec_()
A few things:
You shouldn't be calling GUI code from outside the main thread. GUI elements are not thread-safe. self.kontenjan_ara updates and reads from GUI elements, it shouldn't be the target of your thread.
In almost all cases, you should use QThreads instead of python threads. They integrate nicely with the event and signal system in Qt.
If you just want to run something every few seconds, you can use a QTimer
def __init__(self, parent=None):
...
self.timer = QTimer(self)
self.timer.timeout.connect(self.kontenjan_ara)
self.timer.start(10000)
def kontenjan_ara(self):
self.bilgiler.append(self.text.text())
If your thread operations are more computationally complex you can create a worker thread and pass data between the worker thread and the main GUI thread using signals.
class Worker(QObject):
work_finished = QtCore.pyqtSignal(object)
#QtCore.pyqtSlot()
def do_work(self):
data = 'Text'
while True:
# Do something with data and pass back to main thread
data = data + 'text'
self.work_finished.emit(data)
time.sleep(10)
class MyWidget(QtGui.QWidget):
def __init__(self, ...)
...
self.worker = Worker()
self.thread = QtCore.QThread(self)
self.worker.work_finished.connect(self.on_finished)
self.worker.moveToThread(self.thread)
self.thread.started.connect(self.worker.do_work)
self.thread.start()
#QtCore.pyqtSlot(object)
def on_finished(self, data):
self.bilgiler.append(data)
...
Qt will automatically kill all the subthreads when the main thread exits the event loop.
I chose to rewrite a bit this answer, because I had failed to properly look at the problem's context. As the other answers and comments tell, you code lacks thread-safety.
The best way to fix this is to try to really think "in threads", to restrict yourself to only use objects living in the same thread, or functions that are known as "threadsafe".
Throwing in some signals and slots will help, but maybe you want to think back a bit to your original problem. In your current code, each time a button is pressed, a new thread in launched, that will, every 10 seconds, do 2 things :
- Read some text from self.text
- Append it to self.bilgiler
Both of these operations are non-threadsafe, and must be called from the thread that owns these objects (the main thread). You want to make the worker threads "schedule & wait" the read & append oeprations, instead of simply "executing" them.
I recommend using the other answer (the thread halting problem is automatically fixed by using proper QThreads that integrate well with Qt's event loop), which would make you use a cleaner approach, more integrated with Qt.
You may also want to rethink your problem, because maybe there is a simpler approach to your problem, for example : not spawning threads each time bilgi_cek is clicked, or using Queue objects so that your worker is completely agnostic of your GUI, and only interact with it using threadsafe objects.
Good luck, sorry if I caused any confusion. My original answer is still available here. I think it would be wise to mark the other answer as the valid answer for this question.

PyQt crashes and thread safety

Hello StackExchange community,
first off, you have all been a great help for me, thanks so much. First time question:
I am currently writing a PyQt GUI application and I see that it crashes on windows systems, also it gives me a segfault on my machine at home while it works on the one at work (both linux mint 17). After some research, I realize that I have probably created a thread-unsafe GUI because I have several objects that call each others methods.
From another stackoverflow question: GUI widgets may be accessed only from main thread, meaning the thread that calls QApplication.exec(). Access to GUI widgets from any other thread – what you're doing with your calls to self.parent() – is undefined behaviour, in your case this means crashes.
From Qt docs: Although QObject is reentrant, the GUI classes, notably QWidget and all its subclasses, are not reentrant. They can only be used from the main thread. As noted earlier, QCoreApplication::exec() must also be called from that thread.
So in the end, I figure that I should only use the signal-slot system for doing so.
Is this correct?
Is this only needed for function calls, or can I manipulate the fields of some objects from other objects at runtime in a thread safe manner? for example, I have a options object that is accessed from multiple other objects, and I frequently change parameters there from different sources. Thread safe or unsafe?
The next thing, is that I have a problem recreating this thread-unsafe behaviour in example code. Qt documentation says that QObjects live in different threads. This means, the following Qt application should be thread unsafe (if I got it correctly).
from PyQt4 import QtGui
import sys
class TestWidget(QtGui.QWidget):
def __init__(self,string):
super(TestWidget,self).__init__()
self.button = QtGui.QPushButton(string,parent=self)
self.button.clicked.connect(self.buttonClicked)
# just to check, and yes, lives in it's own thread
print self.thread()
def buttonClicked(self):
# the seemingly problematic line
self.parent().parent().statusBar().showMessage(self.button.text())
pass
pass
class MainWindow(QtGui.QMainWindow):
def __init__(self):
super(MainWindow,self).__init__()
Layout = QtGui.QHBoxLayout()
for string in ['foo','bar']:
Layout.addWidget(TestWidget(string))
CentralWidget = QtGui.QWidget(self)
CentralWidget.setLayout(Layout)
self.setCentralWidget(CentralWidget)
self.statusBar()
self.show()
pass
pass
if __name__ == '__main__':
app = QtGui.QApplication(sys.argv)
M = MainWindow()
sys.exit(app.exec_())
but it runs on my mine as well on windows machines fine.
Why? Is this actually thread-unsafe and could crash, but it just doesn't?
Thanks for helping me to sort this out ...
Is this correct?
Yes, you should only use the signal-slot system for interactions ammong q-objects.
This is how it's meant to be.
Is this only needed for function calls, or can I manipulate the fields of some objects
from other objects at runtime in a thread safe manner?
I have a options object that is accessed from multiple other objects...
If by objects here you meant Q-objects:
Your options object should have support for the signal-slot mechanism, you can achieve this
deriving options from QObject.
class Options(QtCore.QObject):
optionUpdated = QtCore.pyqtSignal(object)
def __init__(self):
self.__options = {
'option_1': None
}
def get_option(self, option):
return self.__options.get(option)
def set_option(self, option, value):
self.__options[option] = value
self.optionUpdated.emit(self)
and then all widgets/objects using this options should have a slot that connect to this singal.
A simple example:
options = Options()
some_widget = SomeWidget()
options.optionUpdated.connect(some_widget.options_updated) // Is like you implement the observer pattern, right?
Why? Is this actually thread-unsafe and could crash, but it just doesn't?
thread-unsafe doesn't mean "the crash is guaranteed" but "this might crash" or "there is a high probability for this to crash".
From pyqt API doc QObject.thread :
Returns the thread in which the object lives.
Erratum
As pointed by ekumoro, I've re-checked my previous position about each object leaving in a different thread, and ... I was wrong!
QObject.thread will return a different QThread instance for each object but QThread is not actually a thread is just a wrapper for those threads provided by the OS.
So the code don't really have the problem af several object sliving in different threads.
I have modified the code you have used for demostration a little bit for simplicity:
from PyQt4 import QtGui
import sys
class TestWidget(QtGui.QWidget):
def __init__(self,string):
super(TestWidget,self).__init__()
# just to check, and yes, lives in it's own thread
print("TestWidget thread: {}".format(self.thread()))
class MainWindow(QtGui.QMainWindow):
def __init__(self):
super(MainWindow,self).__init__()
print("Window thread: {}".format(self.thread()))
Layout = QtGui.QHBoxLayout()
for string in ['foo','bar']:
Layout.addWidget(TestWidget(string))
self.show()
if __name__ == '__main__':
app = QtGui.QApplication(sys.argv)
M = MainWindow()
sys.exit(app.exec_())
and yes, this prints:
Window thread: <PyQt4.QtCore.QThread object at 0x00000000025C1048>
TestWidget thread: <PyQt4.QtCore.QThread object at 0x00000000025C4168>
TestWidget thread: <PyQt4.QtCore.QThread object at 0x00000000025C41F8>
demostrating each control lives in its own thread.
Now, you have the signal-slot mechanism to deal with this "thread-safely", any other approach will not be thread-safe.
Answers to your questions:
GUI widgets can only be accessed from the main thread (the one that runs
QApplication.exec_()). Signals and slots are thread safe by default since
Qt 4
Any call leading to direct Qt graphics objects manipulation from another thread than the main one is not thread-safe => will crash
There is no threads involved in the code of your question (where are the threads???),
it's not true that different QObjects live in different threads. Maybe the crash you have has nothing to do
with threads ?
As a follow-up to some of the comments, below is an test script that shows how to check which thread the code is executing in:
from PyQt4 import QtCore, QtGui
class Worker(QtCore.QObject):
threadInfo = QtCore.pyqtSignal(object, object)
#QtCore.pyqtSlot()
def emitInfo(self):
self.threadInfo.emit(self.objectName(), QtCore.QThread.currentThreadId())
class Window(QtGui.QWidget):
def __init__(self):
super(Window, self).__init__()
self.button = QtGui.QPushButton('Test', self)
layout = QtGui.QVBoxLayout(self)
layout.addWidget(self.button)
self.thread = QtCore.QThread(self)
self.worker1 = Worker()
self.worker1.setObjectName('Worker1')
self.worker1.moveToThread(self.thread)
self.worker1.threadInfo.connect(self.handleShowThreads)
self.button.clicked.connect(self.worker1.emitInfo)
self.worker2 = Worker()
self.worker2.setObjectName('Worker2')
self.worker2.threadInfo.connect(self.handleShowThreads)
self.button.clicked.connect(self.worker2.emitInfo)
self.thread.start()
def handleShowThreads(self, name, identifier):
print('Main: %s' % QtCore.QThread.currentThreadId())
print('%s: %s\n' % (name, identifier))
def closeEvent(self, event):
self.thread.quit()
self.thread.wait()
if __name__ == '__main__':
import sys
app = QtGui.QApplication(sys.argv)
window = Window()
window.show()
sys.exit(app.exec_())

Using QThread in embedded PyQt application

I have written a Qt application in c++ which embeds the Python interpreter and makes use of PyQt to allow for scriptable user interfaces for data analysis. The Python code does some computations on the data and returns a QWidget (containing various plots etc.) which gets inserted into the main application.
I would like to spawn a new QThread from Python to allow control to return to the c++ application so that heavy computations do not block the main thread in which the GUI runs. The problem is that as soon as control is returned back to the c++ application, the thread appears to go to sleep until the Python interpreter is somehow invoked again, eg. a mouseover event on a matplotlib plot. I suspect this is somehow due to Python's GIL.
How can I have a QThread created from Python continue to run after control passes back to the embedding application?
I am using the following code to test. The embedding application calls test_thread() and adds the returned container to a QTabwidget. The thread only executes as I generate mouse events over the matplotlib plot.
import time
from PyQt4 import QtCore, QtGui
from PyQt4.QtCore import pyqtSignal, pyqtSlot
from matplotlib.figure import Figure
from matplotlib.backends.backend_qt4agg import FigureCanvasQTAgg as FigureCanvas
app = QtGui.QApplication.instance()
class Worker(QtCore.QObject):
finished = pyqtSignal()
def __init__(self):
QtCore.QObject.__init__(self)
def do_work(self):
for i in range(0, 10):
print 'worker running'
time.sleep(1)
self.finished.emit()
class Controller(QtCore.QObject):
def __init__(self):
QtCore.QObject.__init__(self)
self.thread = QtCore.QThread()
self.worker = Worker()
self.worker.moveToThread(self.thread)
self.worker.finished.connect(self.thread.quit)
self.thread.started.connect(self.worker.do_work)
self.thread.finished.connect(app.exit)
def start_thread(self):
self.thread.start()
class Container(QtGui.QWidget):
def __init__(self):
QtGui.QWidget.__init__(self)
self.controller = Controller()
self.controller.start_thread()
self.fig = Figure(dpi=72, facecolor=(1, 1, 1), edgecolor=(0, 0, 0))
self.fig.add_subplot(111)
self.canvas = FigureCanvas(self.fig)
self.layout = QtGui.QVBoxLayout()
self.layout.addWidget(self.canvas)
self.setLayout(self.layout)
def test_thread():
container = Container()
return container
CPython is single threaded because of GIL or Global Interpreter Lock. You cannot run more than 1 concurrent thread from python.
https://wiki.python.org/moin/GlobalInterpreterLock
Qt is not blocking you here, python is. The only workaround here is to spawn separate python process for your worker "thread" then use IPC (sockets, shared memory, etc) to communicate between processes. Qt provides interfaces for these IPC mechanisms. You will just have to serialize your interprocess communication.
http://qt-project.org/doc/qt-5/QSharedMemory.html
http://qt-project.org/doc/qt-5/QLocalSocket.html
This assumed you were asking about concurrency. Regarding something else, and editing answer to clarify jurisdiction,
How can I have a QThread created from Python continue to run after control passes back to the embedding application?
This is 100% up to the embedding application. The python interpreter must be allowed to continue to run by the embedding application. If the embedding application does not allow python to continue to run, then any attempts to circumvent this will result in problems.

wxPython, Threads, and PostEvent between modules

I'm relatively new to wxPython (but not Python itself), so forgive me if I've missed something here.
I'm writing a GUI application, which at a very basic level consists of "Start" and "Stop" buttons that start and stop a thread. This thread is an infinite loop, which only ends when the thread is stopped. The loop generates messages, which at the moment are just output using print.
The GUI class and the infinite loop (using threading.Thread as a subclass) are held in separate files. What is the best way to get the thread to push an update to something like a TextCtrl in the GUI? I've been playing around with PostEvent and Queue, but without much luck.
Here's some bare bones code, with portions removed to keep it concise:
main_frame.py
import wx
from loop import Loop
class MainFrame(wx.Frame):
def __init__(self, parent, title):
# Initialise and show GUI
# Add two buttons, btnStart and btnStop
# Bind the two buttons to the following two methods
self.threads = []
def onStart(self):
x = Loop()
x.start()
self.threads.append(x)
def onStop(self):
for t in self.threads:
t.stop()
loop.py
class Loop(threading.Thread):
def __init__(self):
self._stop = threading.Event()
def run(self):
while not self._stop.isSet():
print datetime.date.today()
def stop(self):
self._stop.set()
I did, at one point, have it working by having the classes in the same file by using wx.lib.newevent.NewEvent() along these lines. If anyone could point me in the right direction, that'd be much appreciated.
The easiest solution would be to use wx.CallAfter
wx.CallAfter(text_control.SetValue, "some_text")
You can call CallAfter from any thread and the function that you pass it to be called will be called from the main thread.

Categories