We Keep Coding

CPython extension using omp freezes Qt UI

I am working on a scientific algorithm (image processing), which is written in C++, and uses lots of parallelization, handled by OpenMP. I need it to be callable from Python, so I created a CPython package, which handles the wrapping of the algorithm.
Now I need some UI, as user interaction is essential for initializing some stuff. My problem is that the UI freezes when I run the algorithm. I start the algorithm in a separate thread, so this shouldn't be a problem (I even proved it by replacing the function call with time.sleep, and it works fine, not causing any freeze). For testing I reduced the UI to two buttons: one for starting the algorithm, and another just to print some random string to console (to check UI interactions).
I also experienced something really weird. If I started moving the mouse, then pressed the button to start the computation, and after that kept moving the mouse continuously, the UI did not freeze, so hovering over the buttons gave them the usual blueish Windows-style tint. But if I stopped moving my mouse for a several seconds over the application window, clicked a button, or swapped to another window, the UI froze again. It's even more strange that the UI stayed active if I rested my mouse outside of the application window.Here's my code (unfortunately I cannot share the algorithm for several reasons, but I hope I manage to get some help even like this):
if __name__ == "__main__":
from PyQt5.QtWidgets import QApplication, QPushButton, QVBoxLayout, QWidget
from PyQt5.QtCore import QThread, QObject, pyqtSignal
import time
from CustomAlgorithm import Estimator # my custom Python package implemented in C++
class Worker(QObject):
finished = pyqtSignal()
def run(self):
estimator = Estimator()
estimator.calculate()
# if the above two lines are commented, and the next line is uncommented,
# everything's fine
# time.sleep(5)
print("done")
app = QApplication([])
thread = QThread()
window = QWidget()
layout = QVBoxLayout()
# button to start the calculation
btn = QPushButton("start")
layout.addWidget(btn)
btn.clicked.connect(thread.start)
# button to print some text to console
btn2 = QPushButton("other button")
layout.addWidget(btn2)
btn2.clicked.connect(lambda: print("other button clicked"))
window.setLayout(layout)
# handling events
worker = Worker(app)
worker.moveToThread(thread)
thread.started.connect(worker.run)
worker.finished.connect(thread.quit)
worker.finished.connect(worker.deleteLater)
thread.finished.connect(thread.deleteLater)
window.show()
app.exec_()
I tried multiple variants of using threads, like threading.Thread, multiprocessing.Process, PyQt5.QtCore.QThread (as seen above), even napari's worker implementation, but the result was the same. I even tried removing omp from the code, just in case it interferes somehow with python threads, but it didn't help.
As for the reason I use python, is that the final goal is to make my implementation available in napari.
Any help would be highly appreciated!

Because of Python's "Global Interpreter Lock", only one thread can run Python code at a time. However, other threads can do I/O at the same time.
If you want to allow other threads to run (just like I/O does) you can surround your code with these macros:
Py_BEGIN_ALLOW_THREADS
// computation goes here
Py_END_ALLOW_THREADS
Other Python threads will be allowed to run while the computation is happening. You can't access anything from Python between these two lines - so get that data in order before Py_BEGIN_ALLOW_THREADS.
Reference

PyQt4, prevent app.exec_() from hanging the main thread

I have some simple code that loads google.com with the PyQt4 library.
This is the code:
import sys
from PyQt4.QtGui import QApplication
from PyQt4.QtCore import QUrl
from PyQt4.QtWebKit import QWebView
class Browser(QWebView):
def __init__(self):
QWebView.__init__(self)
self.loadFinished.connect(self._result_available)
def _result_available(self, ok):
frame = self.page().mainFrame()
#print(frame.toHtml())
app.exit()
if __name__ == '__main__':
app = QApplication(sys.argv)
view = Browser()
view.load(QUrl('http://www.google.com'))
print('start')
app.exec_()#hangs the main thread
print('end')
My problem with this code is that app.exec_() hangs the main thread for a while, between the print start and print end.
Is there a way to scrape a website in PyQt4 without making the main thread hang for a while.
I would like to resume the main thread's normal execution of code after app.exec_().

It's probably just taking time to download the content from the web. app.exec_() is what runs your application basically. So if anything is indeed hanging it's actually some other factor that influences the proper execution of your application. Any statements that are after the app.exec_() will be executed once you close the application. Usually app.exec_() is called at the end of the main (in C++ you do return app.exec_() which you can also do in PyQt of course).
When working with content that requires time to be downloaded and displayed in the UI, you have to add multithreading in order to allow the main thread to continue work properly (thus not creating the so called UI freeze) but at the same time do some work in the background. The sole purpose of the main thread is to keep track of the UI and run it. Anything else that you add to the main thread will prevent the UI from working in a fluent manner. You have several options here - inheriting QThread (I wouldn't suggest doing that for this scenario in particular), using QThread + QObject in order to incorporate slots and signal in your application (for example: page downloaded? -> if yes, signal the UI to display the content), QRunnable etc.
So my advice is to download the content in a separate thread and once it's done you can add it to your UI as well as provide some other form of visual feedback (this also includes print statements :P).

Role of Mainloops, Event Loops in DBus service

This is the standard example of DBus service.
import dbus
import gobject
from dbus import service
# from gi._gobject import MainLoop
from dbus.mainloop.glib import DBusGMainLoop
class DBusServer(service.Object):
def __init__(self, name, object_path):
# super(service.Object, self).__init__(name, object_path)
dbus.service.Object.__init__(self, name, object_path)
#dbus.service.method("com.test", in_signature='s', out_signature="s")
def test(self, args):
return args + " Sent by dbus client"
#dbus.service.method("com.test", in_signature='s', out_signature="s")
def foo(self, args):
return "foo"
bus_loop = DBusGMainLoop(set_as_default=True)
session_bus = dbus.SessionBus()
session_name = service.BusName("com.test", session_bus)
dbus_server = DBusServer(session_name, "/test")
loop = gobject.MainLoop()
try:
loop.run()
except KeyboardInterrupt:
loop.quit()
I have questions regarding two mainloops used here
1. What is the role of each mainloop or event loop in this code (if I am using the correct terminology. They both are event loops I guess)
2. If my app is not a GUI app why should I need gobject mainloop or qt mainloop since that is required to catch user generated events from X11 library (in case of Linux)
3. Why can't I use an infinite loop which does nothing instead of gobject main loop (as follows)
while True:
pass

Below are summarized short answers, see the details at the end for more explanations.
Question 1:
There is only one mainloop being used in the code you posted, the loop = gobject.MainLoop(). Its role is to process events, e.g. D-Bus messages sent to your service.
Question 2:
You need a mainloop in order for your code to not block D-Bus messages coming from outside your service. The dbus-python binding does not have a mainloop implementation so you need to use a mainloop from some other library (which also needs to be one supported by dbus-python, in this case GLib).
Question 3:
An infinite loop like that will never allow for any code other than that particular loop to execute, and there would be no way of receiving D-Bus messages (or any other events).
Details:
Simply put, a mainloop typically processes other events when there is no code currently running in your program. In this case, the nature of D-Bus requires there to be a way for your code to handle events from an external source, i.e. D-Bus messages. This is the reason that you need to have a mainloop. You could base the event processing on a loop you write yourself, as long as you have code in the loop that can process external events. The reason people often use libraries for mainloops is because things tend to grow complex in 'real' applications or systems and it's often more robust to use something which has been tested, used, and improved by others for years (e.g. GLib).
The D-Bus binding dbus-python is implemented in a way that requires an external mainloop to be used, as it does not have its own mainloop implementation. A typical reason for designing a library to use an external mainloop is to let the user of a library decide what mainloop to use, since the mainloop could potentially be used for other things as well (e.g. process other events than D-Bus events). This increases the potential use of the library and reduces the dependencies introduced as a side-effect of using the library.
The library must however support the mainloop of choice which means that there must be some integration provided specifically for each mainloop intended to be used with the library. Typically, the internals of a library like this would be designed to use a class/type that is an abstraction of a mainloop so the code internals doesn't need to know about what mainloop has been integrated. This reduces the amount of code which needs to be mainloop implementation specific within the library.
If you examine the bus_loop object received when doing:
bus_loop = DBusGMainLoop(set_as_default=True)
... you will see that it is an object of type dbus.mainloop.NativeMainLoop which is the mainloop wrapper object in dbus-python.
Now, dbus-python currently only provides an integration for the GLib mainloop so the discussion above becomes a bit theoretical in this case. The way dbus-python integrates a GLib mainloop is to require the user to import and instantiate the GLib mainloop wrapper, like this:
from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)
This creates a GLib main context which is later used by a GLib mainloop to process events that dbus-python associates with that context. The last part of the mainloop integration is to require the user to start a GLib mainloop. When you import and run the gobject mainloop, like this:
loop = gobject.MainLoop()
loop.run()
The mainloop will process events from the context created previously. This is the 'link' between the D-Bus binding dbus-python and the GObject binding gobject (which gives access to the GLib mainloop).
Simplified to the extreme, one could say that the context created and set when doing:
from dbus.mainloop.glib import DBusGMainLoop
DBusGMainLoop(set_as_default=True)
is a list of events to process if they appear, and the mainloop created and started when doing:
loop = gobject.MainLoop()
loop.run()
is what makes something check that list. The fact that GLib is used for both things makes it work, again simply put.
If you want to dig further into this, read up on GLib and the GMainLoop and GMainContext and the concepts related to these. Hope this helps.

How can I print to QTextEdit to mimic printing to console? (Python 3.3)

I have a piece of code that displays a Gui which has a QTextEdit Field. I would like to print to this field in real time similar to how the print function outputs to the console.
I have tried using multiple instances of the append function. Ex:
self.textEdit.append(_translate("MainWindow", ">>> Text", None))
The problem is that no matter where they are in the code, they seem to only show after the program is executed. My goal is to have them show in line like the print function does on the console.
I feel like this is an easy answer, but I have had no luck searching.. I am fairly new to Python and any help or guidance will be appreciated.
Thanks in advance!

Indeed, as mata mentioned the freezing comes from doing all your work in the same (main) thread, which also handles UI updates. One way to solve your responsiveness issue is indeed to frequently use QApplication.processEvents() in your blocking code. That will give the impression of a responsive GUI to the user if frequent enough.
However, using threads in Python (whether native or QThread) will not always work. That is because of the existence of the Global Interpreter Lock (GIL, the wiki has a good short intro). In short, Python does not allow more than one thread to execute code at the same time.
If your background task is light, or is based on IO, you can get around this as most IO-heavy modules for Python release the GIL while doing their job. However, if you are performing heavy computations in Python, the GIL will be locked by your processes, and as such your UI will still be unresponsive.
Consider the following example, built using PySide:
import sys, random
from threading import Thread
from time import sleep
from urllib import urlopen
from PySide import QtCore, QtGui
class Window(QtGui.QMainWindow):
update_signal = QtCore.Signal(int)
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.progress_bar = QtGui.QProgressBar(self)
self.progress_bar.setRange(0, 10)
self.setCentralWidget(self.progress_bar)
self.update_signal[int].connect(self.progress_bar.setValue)
self.show()
self.t = Thread(target=self.worker)
self.t.start()
def worker(self):
while self.progress_bar.value() < 10:
self.update_signal.emit(self.progress_bar.value()+1)
print "Starting Sleep"
sleep(5)
print "End of Sleep"
if __name__ == '__main__':
qapp = QtGui.QApplication(sys.argv)
win = Window()
sys.exit(qapp.exec_())
Then, try replacing the worker function to:
def worker(self):
while self.progress_bar.value() < 10:
self.update_signal.emit(self.progress_bar.value()+1)
v = 0
print "Starting Add"
for i in xrange(5000000):
v = v+random.uniform(0, 100)
print "End of Add"
The first case maintains a responsive UI, as the call to sleep() releases the GIL. But the second example does not, as the computationally-intense algorithm keeps the lock.
One solution could be using the multiprocessing package.
From the docs:
multiprocessing is a package that supports spawning processes using an
API similar to the threading module. The multiprocessing package
offers both local and remote concurrency, effectively side-stepping
the Global Interpreter Lock by using subprocesses instead of threads.
Due to this, the multiprocessing module allows the programmer to fully
leverage multiple processors on a given machine.
A simple, illustrative example of using python multipocessing.
Also, if you have further interest, this blog post about multi-processing techniques might be of interest.

This means that you're probably doing all your work in the GUI thread. This is a common mistake, and it means that the GUI will freeze and not respond while there is something else going on.
You can add a call to QApplication.processEvents() to allow for the GUI to update after you change the text in your QTextEdit, but that will only partially solve the problem, the GUI will nevertheless freeze up between those calls.
The solution is simple: do the work in a separate thread. You should read Threading Basics from the Qt documentation, that should get you started.

Explanation of need for Multi Threading GUI programming

I'm looking for a good explanation of the need to use multi-threading in graphical applications. In the examples below Python is used but the question is not python specific it applies to maybe the general design of graphical programming in any language.
Lets take a simple example. Let's assume there is an application that does some sort of time consuming operation on a collection of files, and that it outputs it's progress to the console. Lets assume that this operation takes 2 seconds per file and that there are 10 files to process called 1.txt, 2.txt,3.txt, ... 10.txt. Then an example implementation could look like the following:
console
import time
def process(file):
print 'processing {0}...'.format(file)
time.sleep(2.0) #simulate slow operation
files = ['{0}.txt'.format(i) for i in range(1, 11)]
map(process, files)
The console example is of course single threaded and does the job just fine. Now if we wanted to add a graphical progress bar a single threaded implementation might look like the following:
single-threaded gui
import time, gtk, gobject
def process(file):
print 'processing {0}...'.format(file)
time.sleep(2.0)
class MainWindow(gtk.Window):
def __init__(self):
super(MainWindow, self).__init__()
self.progress = gtk.ProgressBar()
self.progress.set_fraction(0)
self.add(self.progress)
self.connect("destroy", gtk.main_quit)
self.show_all()
files = ['{0}.txt'.format(i) for i in range(1, 11)]
gobject.timeout_add(100, self.submit, files, 0)
def submit(self, files, i):
process(files[i])
self.progress.set_fraction((i + 1.0)/len(files))
if i + 1 < len(files):
gobject.idle_add(self.submit, files, i + 1)
win = MainWindow()
gtk.main()
This seems to work fine, but as you run the application if you try and interact with the application like try and resize the window for example it will get stuck and only respond every two seconds when it gets freed up to process pending gui events. The final example is a multi-threaded implementation and stays responsive throughout execution.
multi-threaded gui
import time, gtk, gobject, threading
def process(file):
print 'processing {0}...'.format(file)
time.sleep(2.0)
class MainWindow(gtk.Window):
def __init__(self):
super(MainWindow, self).__init__()
self.progress = gtk.ProgressBar()
self.progress.set_fraction(0)
self.add(self.progress)
self.connect("destroy", gtk.main_quit)
self.show_all()
files = ['{0}.txt'.format(i) for i in range(1, 11)]
threading.Thread(target=self.submit, args=(files,)).start()
def submit(self, files):
for i, file in enumerate(files):
process(file)
gobject.idle_add(self.progress.set_fraction, (i + 1.0)/len(files))
if not self.get_visible():
return
gtk.gdk.threads_init()
win = MainWindow()
gtk.main()
It seems perfectly clear and logical to me that if you have a long-running blocking operation in your code and you want a responsive gui that you have to use a multi-threaded solution. There is no other way around it. Is this the case? I have tried to explain this many times to fellow developers but many don't understand or don't agree. Can someone provide an explanation of this concept, a link to an article on it, or correct me if my understanding is incorrect.

Your understanding is correct. If an application isn't muli-threaded the application waits for every operation to finish. When your application is multi-threaded you use one thread to handle GUI actions and another to process the files.
I don't have an article or a reference to something similar. Maybe it helps if you think of threads as people, each has it's own job and each can only do one thing at a time.

The main reason is, that GUI toolkit process all events (mouse movement, button click, keyboard input, system events, etc) in the mainloop, which is not part of the code you write for you GUI application.
This main loop call all the event handlers and other functions you provide. Now, if one of those functions take too long (e.g. > 100ms), then there will be a very noticeable effect on UI responsiveness, as the main loop will not be able to process more events.
This topic should be discussed in details in the "advanced concepts" section of any book on GUI programming, regardless of programming language of toolkit used.

Multithreading is one way of approaching this, but not necessarily the only (or best) way. As a python example, greenlets provide a way of running concurrent processes within the same thread, avoiding the problems with locking associated with multithreading. I would certainly consider greenlets to be a preferred solution in most cases simply because of the relative ease of coding them.