We Keep Coding

Python multithreading - interaction with child threads from parent/main thread

We are using python multithreading(I/O bound) for our application deployments. From main/parent thread we are creating multiple child threads (multiple hosts [application/web] for deployment within each thread) and waiting for them to finish using .join() in main thread. We are also using some Fabric package commands to ask user input whether to proceed with deployment or not in-case of an error for whatsoever reason.
Problem statement:
Since we are using multithreading, all the child threads are writing the messages to terminal (stdout)output and thus the errors from one thread on the screen are overlapping by other thread messages and it is difficult to comb through the console/terminal to see what thread has failures.
What could be the best way to gather all the failure steps in the deployment from all the threads collectively and display all of them at once on the console/terminal output ?
The difficulty I'm facing is how to interact with child threads while in main thread we are just waiting to finish them (if there are any fabric user interactive questions then will type y/n based on our decision about the failure).

Restart QThread with GUI

I am using QThread to do some calculations in a separate Thread.
The Thread gets started by a button click, witch launches the function StartMeasurement().
The Thread can finish the process by itself (after finished the calculations)
and emits the PyQT Signal finished. Or the thread can be stopped by the User by the stopBtn click.
The terminate() function is working, but I get a lot of troubles when I try to start the thread again.
Is it recommendable to use the movetoThread() approach here?
Or how could I ensure that the thread is stopped correctly to enable a proper restart. (means, starting new!)
# starts the measurment in a Thread: StartMeasurement()
def StartMeasurement(self):
self.thread = measure.CMeasurementThread(self.osziObj, self.genObj, self.measSetup)
self.thread.newSample.connect(self.plotNewSample)
self.thread.finished.connect(self.Done)
self.stopBtn.clicked.connect(self.thread.terminate)
self.stopBtn.clicked.connect(self.Stop)
self.thread.start()

It's not a problem. The general practice when working with QThread is to connect its finished() signal to the deleteLater() slot of the objects that have been moved to the separate thread via moveToThread(). It's done in order to properly manage the memory when you then destroy your thread because it's assumed that you will first quit the thread and then destroy its instance. ;) This should tell you that stopping a thread has nothing to do with the destruction of those objects UNLESS you have established the connection I've described above.
It is perfectly fine to restart a thread IF you have stopped it properly using quit() and wait() to actually wait untill the stopping is completed.
However my advice is to not do that unless that extra thread has a huge impact on your application for some reason (highly unlikely with modern machines).
Instead of restarting the thread consider the following options:
implement a pause flag that just makes the thread run without doing anything if it's set to true (I've used this example of mine many times to demonstrate such behaviour (check the worker.cpp and the doWork() function in particular) - it's in C++ but it can be ported to PyQt in no time)
use QRunnable - its designed to run something and then (unless autoDelete is set to true) return to the thread pool. It's really nice if you have tasks that occur every once in a while and you don't need a constatly running separate thread. If you want to use signals and slots (to get the result of the calculation done inside the QRunnable::run() you will have to first inherit from QObject and then from QRunnable
Use futures (check the Qt Concurrent module)
I suggest that you first read the Example use cases for the various threading technologies Qt provides.

Python message passing to unique threads

I have an application (actually a plugin for another application) that manages threads to communicate with external sensor devices. The external sensors send events to the application, but the application may also send actions to the sensors. There are several types of devices and each has unique qualities (temperature, Pressure, etc.) that require special coding. All communications with the sensor devices is over IP.
In the applications, I create a thread for each instance of a sensor. This is an example of the code
self.phThreadDict[phDevId] = tempsensor(self, phDevId, phIpAddr, phIpPort, phSerial, self.triggerDict)
self.phThreadDict[phDevId].start()
In each thread I setup callback handlers for events sent by the sensor and then go into a loop at the end.
while not self.shutdown:
self.plugin.sleep(0.5)
The threads then handle incoming events and make calls into the main thread, or the actual program that spawned the main thread. All of this works quite well.
But, at times I also need to send requests to a specific sensor. Methods are defined in each thread for that purpose and I call those methods from the main thread. For example:
self.phThreadDict[dev.id].writeDigitalOutput(textLine, lcdMessage)
This also works, but I believe the code is actually executed in the main thread rather than in the thread specific to the sensor.
My question is: What options do I have for passing work to the specific target thread and having the thread execute the work and then return success or fail?

Expanding a bit on Thomas Orozco's spot-on comments,
self.phThreadDict[dev.id].writeDigitalOutput(textLine, lcdMessage)
is executed in whichever thread runs it. If you run it from the main thread, then the main thread will do all of it. If from some other thread, then that thread will run it.
In addition to a Queue per thread, for the threads to receive descriptions of work items to process, you also want a single Queue for threads to put results on (you can also use another Queue per thread for this, but that's overkill).
The main thread will pull results off the latter Queue. Note that you can - and it's very common to do so - put tuples on Queues. So, for example, on the talk-back-to-the-main-thread Queue threads will likely put tuples of the form:
(result, my_thread_id, original_work_description)
That's enough to figure out which thread returned what result in response to which work item. Maybe you don't need all of that. Maybe you need more than that. Can't guess ;-)

Indeed, this is executing code in the main thread.
Use queues, that's what they're meant for (task synchronization and message passing between threads).
Use one queue per sensor manager thread.
Your sensor manager threads should be getting items from the queue instead of sleeping (this is a blocking call).
Your "main" thread should be putting items in the queue instead of running functions (this is generally a non-blocking call).
All you need to do is define a message format that lets the main thread tell the manager threads what functions to execute and what arguments to use.

How to subscribe the notification from worker threads?

I follow MVC design. I have three Controller parts which runs in main GUI thread. Also have another worker thread which processed some I/O stuffs. Once the worker thread finishes its task, all three controllers should get notification. Can anyone suggest me a solution for this ?
I am using PyQT. So is there any way like; subscribe the events from a worker thread to multiple controller parts ?

The easiest way is to just emit a signal in the worker thread. Create three connections from the worker thread's signal to the controllers' slots, and you should be good to go.
Signal/slot connections in Qt are thread-safe and will work as expected. Only the thread in which the object with the slot lives in needs to have an event loop, which is fine, since in your case the slots belong to the controller, which is in the GUI thread which has an event loop.
See the documentation on QObjects and threads for more details.

python: How to detect when my thread become orphan?

I have a program using a thread. When my program is closed, my thread is still running and that's normal. I would like to know how my thread can detect that the main program is terminated; by itself ONLY. How would I do that?
My thread is in an infinite loop and process many object in a Queue. I can't define my thread as a daemon, else I can lose some data at the end of the main program. I don't want that my main program set a boolean value when it closed.

If you can get a handle to the main thread, you can call is_alive() on it.
Alternatively, you can call threading.enumerate() to get a list of all currently living threads, and check to see if the main thread is in there.
Or if even that is impossible, then you might be able to check to see if the child thread is the only remaining non-daemon thread.

Would it work if your manager tracked how many open threads there were, then the children killed themselves when starved of input? So the parent would start pushing data on to the queue, and the workers would consume data from the queue. If a worker found nothing on the queue for a certain timeout period, it would kill itself. The main thread would then track how many workers were operating and periodically start new workers if the number of active workers were under a given threshold.