We Keep Coding

Checking threadsafe-ness in python?

This is my first foray into threading, so apologies for any obvious mistakes.
I have a PyQt widget, from which a new process, prog, is run in a different thread. In my main thread, I'm also redirecting stdout to a read-only QTextEdit. However, I get errors referring to recursion, and I'm worried that my threads are interfering in each other in a way which causes a print statement to go into an infinite loop. I only get these errors if I run prog from the GUI, and not from the command line. My stdout redirect is using the code in this SO answer
In pseudo-code, this is basically what I've got:
gui.py
class widget(QWidget):
def __init__(self):
self.button = QPushButton("GO!", self)
self.button.clicked.connect(self.start)
def start(self):
self.thread = TaskThread()
sys.stdout = EmittingStream(textWritten = self.outputText)
self.thread.start()
def outputText(self):
#as in answer provided in link (EmittingStream in separate module)
prog.py
class TaskThread(QThread):
def run(self):
'''
Long complicated program; putting in simpler code here (e.g. loop printing to 10000) doesn't reproduce errors
'''
Is there any way of finding out if my recursion is caused by an infinite loop, or by anything else?
Is my code obviously thread-unsafe?
How do you make functions guaranteed to be threadsafe? (Links to tutorials / books will be good!)

This is tricky, but I think that your code is thread-unsafe. Specifically, looking at other stackoverflow answers (here and here) it appears that you should not be accessing a Qt GUI object from another thread than the one it was created in (even a QThread).
Since any call to print in your code now accesses a Qt GUI object, it seems this is very thread unsafe.
My suggestion to make it safe would be to:
Have a QThread, inside of which you have instantiated the output box (it is thread safe to access a Qt GUI object from the thread it was created in, and it does not have to be created in the main thread) Qt GUI objects are not reentrant and must be created and used in the main thread only, see here As such you will need a QThread to post events back to the main thread (using the Qt signals/slots mechanism which can be thread safe when done correctly)
Have this QThread blocking on reading from a Python Queue. When it gets something from the queue, it places it in the Qt text box posts it back to the main thread, and the main thread will update the output box.
Modify your EmmittingStream to place things in the Queue, rather than directly into the Qt output box.

I can see you have located where the error is. But without the code there is not much that I can tell.
Filling your needs of direction, I'll point to you to Python profiles. Since it looks like you need some python profiling tools.
http://docs.python.org/2/library/profile.html
and a answer about the subject
How can you profile a Python script?

In a Qt application you must use one single thread to handle the all the gui part. You can use other threads for computations, but not for user interface.
Just post messages about updates in a queue in the worker threads and use the main thread to pick up those messages and update the GUI.

Qt processes signals after threads exit?

I am building an application with a GUI and several worker threads. Now I want it to be a multi-threaded application so I am executing the same thread several times, in a loop, each thread grabbing different input parameters defined in a class outside of the thread.
So my mainGui.py file looks something like this (relevant code shown only):
self.workers = [worker.Worker(), worker.Worker(), worker.Worker()]
for i in xrange(threadCount):
self.currentWorker = self.workers[i]
self.currentWorker.alterTable.connect(self.alterMainTable)
self.currentWorker.start()
time.sleep(0.1)
As you may imagine, I am connecting the Worker's alterTable signal to the alterMainTable() method I have defined in my main GUI thread. This method updates the table in the GUI.
The worker thread looks something like this:
class Worker(QThread):
alterTable = Signal(dict)
def __init__(self, parent=None):
super(Worker, self).__init__(parent)
def sendToTable(self, param1, param2, param3):
"""This method emits the signal with params as defined above"""
params = {}
params["param1"] = param1
params["param2"] = param2
params["param3"] = param3
self.alterTable.emit(params)
def run(self):
#Perform a lengthy task, do this every now and then:
self.sendToTable(param, param2, param3)
When I am running this application in a single worker thread (so when I'm not calling that loop in the main thread), it works fine - the signals are emitted, and the main table in the GUI is updated.
However, the problems arise when I run several threads at once. The Worker threads do their job, but the Signal is only emitted sometimes. Or, better yet, it is emitted as if Qt (or whatever) was waiting for all the threads to finish, and then update the table. This is literally what happens - I can see in Python console that the threads are performing their tasks, and once all of them are doing whatever they were doing, the table is suddenly populated with a bunch of data at once.
As you may imagine, another problem that comes out of this is that since there are no events being processed, after some time, my application appears to be frozen.
I have tried adding the Qt.DirectConnection to the connect() method, but that didn't really help.
Bonus question: I've been reading about this topic on SO and other websites, it seems that people recommend QRunnable() instead of QThread() especially when it comes to subclassing it. Consequently, I'd be using QThreadPool(). But when I tried this, it seems that I cannot emit a signal from a QRunnable - it gives me the AttributeError: 'PySide.QtCore.Signal' object has no attribute 'connect', even though the Signal is defined within the QRunnable class - which is quite odd, I must say.
EDIT: In another answer on SO, someone mentioned that one may be "spamming" the main GUI thread with events to be processed. However, I don't believe that this is the case here as the sendToTable() method from the QThread is only called 5-6 times at most from the thread, and the threadCount is never larger than 20 at most, but I usually keep it at around 5.

And, as usual, I answer my question after 2 days of debugging and minutes after posting on SO.
I had a leftover workerThread.wait() method call after all the threads have been started. So naturally my application did what it was told to do - waited for the thread to finish.
I removed that method call and also put the QCoreApplication.processEvents() inside the loop which started the threads, now it all works like a charm.
Once again, thank you, the invisible, almighty person of SO!

Detecting the module in python (which sends data to the GUI thread)

So I wrote this software which used to implement 4 tabs (now 8). Now, every tab is its own QThread, plus one general GUI thread, obviously. The worker threads (tab threads) send data back to the GUI thread when this needs updating. For example, I have a statusBar text in the GUI thread which needs to be updated every time worker thread does something. I'm using the SIGNAL/SLOT for that, and the way I detect an incoming SIGNAL is like this:
def setStatus(self, status):
#Defining SIGNAL sender
self.sender = QObject().sender()
if "module1" in str(self.sender):
self.module1CurrentStatus.setText(status)
Now this works and all, but like I said, the software has grown from implementing 4 tabs to implementing 8 tabs. Is there a better way to implement this, instead of using sender()? I'm simply looking for something that can be written only once for all the modules, a way for the GUI to know automatically which thread the SIGNAL came from. I know that I could write a separate SIGNAL for every module, but I accomplish nothing with that.

If I understand your requirements correctly, you could use a QSignalMapper to re-emit the signals with an identifier.

The module name can be acquired via __name__, unless it is the file being run as a script .

How to make a PyGTK widget respond quickly when the callback takes too long?

I do a time-consuming processing in the _init callback function. Therefore, at the GUI, when I activate the check-button, the program freezes, while the processing runs to completion, and then the check mark appears and execution resumes normally.
self.init = gtk.CheckButton("Init", False)
self.init.connect("toggled", self._init)
def _init(self, w):
....
How do I make the GUI more responsive while the processing takes place ?
And is there a way I can make the GUI change the state of the check-box right in the beginning and then enter the heavy loop ?

You have several solutions:
Make the computation in a worker thread, which will send the result to the main thread. But don't change GUI items in the worker thread. Like most toolkits, GTK is not thread safe. I don't really like that one, as I don't really like threads, as they tend to make debugging harder...
Split the processing in an idle handle. When I say split, I really mean to split it in several steps that take much less time to compute. You should use an automaton with several states taking each much less time. Putting a big fat calculus in an idle handler is like calling sleep(5) in that handler: the GUI message pump would be blocked for 5 seconds, with no way to handle redrawing events... Here's an example of lazy loading data using this technique.
Force a call to the message pump on a regular basis using gtk_main_iteration.
The examples I gave you are in C, but the spirit is the same in python.

In the handler, register the rest of the init as idle-handler and return. gtk.idle_add IIRC. That way it will run after the gui events are handled. Check documentation to see which value you need to return so the function is not started again.

WX.Python and multiprocessing

I have a wx.python application that takes some files and processes them when a button is clicked. I need to process them in parallel.
I use this code inside the bound button function:
my_pool = multiprocessing.Pool(POOLSIZE)
results=[digest_pool.apply_async(self.fun, [args]) for file in list_files() ]
my_pool.close()
my_pool.join()
for result in results :
print result.get()
But it seems this code is not run at all, even if I print something on fun. I didn't get any result and my GUI application got stuck. Could someone help? What is the problem here and how can I solve it using the pool multiprocessing module inside my wx frame class?

It looks like you're running up against a pretty common problem encountered by people attempting to use threading with GUI toolkits. The core of the issue is that you must never block the main GUI thread in your code. The graphical toolkit needs to be able to constantly respond to events. When you do the my_pool.join() call, you're putting the main thread to sleep and the result is that your entire process will appear to lock up.
I'm not particularly familiar with wxWidgets but I'm sure there are a few patterns out there for how to use threads with it. It's easy to spin off background threads from the GUI thread but getting the results back is usually the trick. You'll need some sort of asynchronous "work done" event that you can send to the main GUI thread when the background operation completes. Exactly how that's done differs from toolkit to toolkit. I'm not sure what the mechanism is for wxWidgets but I'd expect a quick google search would turn up an answer (or perhaps a kind commenter will provide a link ;-)