I have a wxPython application (http://www.OpenSTV.org) that counts ballots using methods that have multiple rounds. I'd like to do two things:
(1) For a large number of ballots, this can be a bit slow, so I'd like to show the user a progress dialog so he doesn't think the application is frozen.
(2) I'd like to allow the user to break ties manually, and this requires the counting code to show a dialog window.
To achieve (1), I create a thread to run the counting code, and this allows me to present a nice progress dialog to the user.
The problem with this, however, is that the counting code is not the main thread, and only the main thread in wxPython can process window events.
I suppose I could create a thread to run the progress dialog instead, but this seems awkward. Is there a better way of accomplishing both (1) and (2)?
Use Queue to communicate and synchronize among threads, with each thread "owning" and exclusively interacting with a resource that's not handy to share.
In GUI toolkits where only the main thread can really handle the GUI, the main thread should play along -- set up and start the threads doing the actual work, then do nothing but GUI work, using Queues to communicate to and from the other threads.
For (1), when your counting thread has an update, it should put it to the Queue where the main thread is waiting; when your main thread gets a suitable message on that Queue, it updates the progress dialog.
For (2), the counting thread sends the "have the user break a tie" request, main thread gets it and responds appropriately, and sends back the resolution on a separate Queue.
So in general, there are two kinds of communications: one that don't require a response, and others that do. For the former kind, just put the notification on the appropriate queue and simply proceed -- it will be acted on in due course. For the latter kind, my favorite idiom is to put on the appropriate queue a pair (request, response_queue). If otherwise identical requests differ in that some need a response and others don't, queueing (request, None) when no response is needed (and (request, q) where q's a Queue when a response IS needed) is a nice, easy, and general idiom, too.
There are several ways to call the main thread wxPython thread from a process thread. The simplest is wx.CallAfter() which will always execute the functional passed to it in the main thread. You can also use wx.PostEvent() and there's an example of this in the demo (labeled: Threads), and there are several more complicated but more customizable ways which are discussed in the last chapter of wxPython in Action.
Related
I have been trying to find the most elegant way to decouple my programs from the GUI, such that I can change my front-end without needing to re-write a whole lot of code.
I work with threads a lot, so I often have the need to notify the main GUI thread of asynchronous happenings either through events (for wxPython) or signals (for PyQt). I have experimented a bit with PyPubSub, which may be what I am looking for, but while there are tons of wxPython examples (since it was originally included with it in early development).
I am not aware if there is a 'proper' way to use it with PyQt without running into race conditions. If anyone has some insight on this, I would really appreciate it!
PyPubSub's sendMessage() will call listeners in the same thread as the sender (default Python behavior). In a multithreaded GUI app, you must ensure that listeners that interact with GUI are called in the main thread. Also threads execute independently, so you need each thread to call its own listeners, based on a timed or idle callback mechanism.
The way to call listeners in the correct thread in PyQt is via signals. PyPubSub can still be used in a multithreaded PyQt GUI, but the mechanism used to transfer the "message" from sender to listener would have to be via a signal. There isn't one best way to do it I don't think, depends on details of your app design. You could for example have a QtPubsubHandler that derives from QObject and gets created in main thread, and a QtPubsubSender class that also derives from QObject and gets created in each worker thread. The QtPubSubSender defines a custom signal, say pubsub, which QtPubsubHandler connects to. Then to send a message, the sender does qtPubsubHandler.sendMessage(topic, **data), which causes a pubsub signal to get emitted, which Qt properly queues and eventually signals the QtPubsubHandler, which actually calls pub.sendMessage().
There are many other ways, but you get the general idea: two classes that derive from QObject, and one of them does the actual sending in the same thread as the intended listener, the other uses a signal so everything is thread safe. Actually you don't have to use PyQt signals: but then you would have to have a queue in main thread and have an idle callback that allows it to process any items on the queue.
I have an application (actually a plugin for another application) that manages threads to communicate with external sensor devices. The external sensors send events to the application, but the application may also send actions to the sensors. There are several types of devices and each has unique qualities (temperature, Pressure, etc.) that require special coding. All communications with the sensor devices is over IP.
In the applications, I create a thread for each instance of a sensor. This is an example of the code
self.phThreadDict[phDevId] = tempsensor(self, phDevId, phIpAddr, phIpPort, phSerial, self.triggerDict)
self.phThreadDict[phDevId].start()
In each thread I setup callback handlers for events sent by the sensor and then go into a loop at the end.
while not self.shutdown:
self.plugin.sleep(0.5)
The threads then handle incoming events and make calls into the main thread, or the actual program that spawned the main thread. All of this works quite well.
But, at times I also need to send requests to a specific sensor. Methods are defined in each thread for that purpose and I call those methods from the main thread. For example:
self.phThreadDict[dev.id].writeDigitalOutput(textLine, lcdMessage)
This also works, but I believe the code is actually executed in the main thread rather than in the thread specific to the sensor.
My question is: What options do I have for passing work to the specific target thread and having the thread execute the work and then return success or fail?
Expanding a bit on Thomas Orozco's spot-on comments,
self.phThreadDict[dev.id].writeDigitalOutput(textLine, lcdMessage)
is executed in whichever thread runs it. If you run it from the main thread, then the main thread will do all of it. If from some other thread, then that thread will run it.
In addition to a Queue per thread, for the threads to receive descriptions of work items to process, you also want a single Queue for threads to put results on (you can also use another Queue per thread for this, but that's overkill).
The main thread will pull results off the latter Queue. Note that you can - and it's very common to do so - put tuples on Queues. So, for example, on the talk-back-to-the-main-thread Queue threads will likely put tuples of the form:
(result, my_thread_id, original_work_description)
That's enough to figure out which thread returned what result in response to which work item. Maybe you don't need all of that. Maybe you need more than that. Can't guess ;-)
Indeed, this is executing code in the main thread.
Use queues, that's what they're meant for (task synchronization and message passing between threads).
Use one queue per sensor manager thread.
Your sensor manager threads should be getting items from the queue instead of sleeping (this is a blocking call).
Your "main" thread should be putting items in the queue instead of running functions (this is generally a non-blocking call).
All you need to do is define a message format that lets the main thread tell the manager threads what functions to execute and what arguments to use.
I'm learning PyQt (I haven't used C++ Qt before).
I don't understand the reason for signals and slots. That is, it seems I can do everything by calling the methods of classes directly. Doing so seems more Pythonic, uses less code and is easier to read. What am I missing?
For clarification, I'm asking why I would do this:
def mouseReleaseEvent(self):
self.clicksignal.connect(ui.iconClicked)
self.clicksignal.emit()
when I can do this:
def mouseReleaseEvent(self):
ui.iconClicked()
The reason signals and slots exist is because you cannot change the GUI from any other thread of the application except the UI thread.
If you have some heavy CPU intensive calculation to do, or any task that waits for IO or something like that... if you do it the UI thread (for example if you fetch a url or something that lasts some time), your UI thread will be busy, and the GUI event loop wont be able to update itself, so the GUI will seem frozen.
To do these kind of operations. you execute them in a separate (background worker) thread, so the UI thread can continue updating the GUI.
Now the problem is that you cannot access the elements of the GUI and change their state from any other thread except the UI thread. So signals and slots are introduced. When you emit a signal it is guaranteed to be caught in the UI thread, and the slot will be executed in the UI thread.
I'm not exactly sure what you try to accomplish in your example, but this is the main reason why signals and slots exist. Basically UI thread should only deal with the UI, and everything else should be done in the background worker thread that sends a signal that gets caught in the UI thread and the slot that update the GUI is executed in the UI thread.
Well... yes, you can. But you need to think bigger. In your example code, caller of mouseReleaseEvent must have the reference to the object that receives the notification, and explicitly invoke appropriate method. Using slot & signals mechanism decouples event producer (e.g. widget) from event consumer - pretty much arbitrary other object. This makes setting up communication and control flow easier and external to the low-level UI components, which is the good thing. It also makes such components reusable - by moving wiring code outside, we make it independent of the application logic.
In addition to the answers by #ViktorKerkez and #Wilbur, signals and slots provide a notification system of fire and forget, as well as decoupling classes.
A great advantage of this is that a class can emit a signal, not knowing what or who is going to receive the message. It may be just one object that has a slot connected, or it could be dozens. Alternatively, you may want a single class with one slot that is connected to multiple signals. So it can be used as a notification system.
For example, imagine a program where many different types of objects send information to a log. The objects simply emit a Log(text) signal and do not care about what actually does the logging. These signals can be connected to a log class which can be either logging to a file, over a network, to the screen, or all at once. The objects logging don't care.
I'm writing a program, with a PyQt frontend. To ensure that the UI doesn't freeze up, I use QThreads to emit signals back to the parent. Now, I have reached a point where I need my thread to stop running, emit a signal back to the parent, then wait on the parent to return an approval for the thread to continue (after the user interacts with the UI a little bit).
I've been looking into the QMutex class, along with QThread's wait function.
How should I go about doing this properly?
One approach is using a condition variable.
In my code, however, I prefer using Python's built-in Queue objects to synchronize data between threads. While I'm at it, I use Python's threads as opposed to PyQt threads, mainly because it allows me to reuse the non-GUI part of the code without an actual GUI.
I've seen a few threaded downloaders online, and even a few multi-part downloaders (HTTP).
I haven't seen them together as a class/function.
If any of you have a class/function lying around, that I can just drop into any of my applications where I need to grab multiple files, I'd be much obliged.
If there is there a library/framework (or a program's back-end) that does this, please direct me towards it?
Threadpool by Christopher Arndt may be what you're looking for. I've used this "easy to use object-oriented thread pool framework" for the exact purpose you describe and it works great. See the usage examples at the bottom on the linked page. And it really is easy to use: just define three functions (one of which is an optional exception handler in place of the default handler) and you are on your way.
from http://www.chrisarndt.de/projects/threadpool/:
Object-oriented, reusable design
Provides callback mechanism to process results as they are returned from the worker threads.
WorkRequest objects wrap the tasks assigned to the worker threads and allow for easy passing of arbitrary data to the callbacks.
The use of the Queue class solves most locking issues.
All worker threads are daemonic, so they exit when the main program exits, no need for joining.
Threads start running as soon as you create them. No need to start or stop them. You can increase or decrease the pool size at any time, superfluous threads will just exit when they finish their current task.
You don't need to keep a reference to a thread after you have assigned the last task to it. You just tell it: "don't come back looking for work, when you're done!"
Threads don't eat up cycles while waiting to be assigned a task, they just block when the task queue is empty (though they wake up every few seconds to check whether they are dismissed).
Also available at http://pypi.python.org/pypi/threadpool, easy_install, or as a subversion checkout (see project homepage).