When running my code I start a thread that runs for around 50 seconds and does a lot of background stuff. If I run this program and then close it soon after, the stuff still goes on in the background for a while because the thread never dies. How can I kill the thread gracefully in my closeEvent method in my MianWindow class? I've tried setting up a method called exit(), creating a signal 'quitOperation' in the thread in question, and then tried to use
myThread.quitOperation.emit()
I expected that this would call my exit() function in my thread because I have this line in my constructor:
self.quitOperation.connect(self.exit)
However, when I use the first line it breaks, saying that 'myThread' has no attribute 'quitOperation'. Why is this? Is there a better way?
I'm not sure for python, but I assume this myThread.quitOperation.emit() emits a signal for the thread to exit. The point is that while your worker is using the thread and does not return, nor runs QCoreApplication::processEvents(), myThread will never have the chance to actually process your request (this is called thread starvation).
Correct answer may depend on the situation, and the nature of the "stuff" your thread is doing. The most common practice is that the main thread sends a signal to the worker thread where a slot sets a flag. In the blocking process you regularly check this flag. It it is set you stop whatever "stuff" you are doing, tell your worker thread that it can quit (with a signal preferably with queued connection), call a deleteLater() on the worker object itself, and return from any functions you are currently in, so that the thread's event handler can run, and clear your worker object and itself up, the finally quit.
In case your "stuff" is a huge cycle of very fast operation like simple mathematics or directory navigation one-by-one that takes only a few milliseconds each, this will be enough.
In case your "stuff" contain huge blocking parts that you have no control of (an thus you can't place this flag checking call in it), you may need to wait in the main thread until the worker thread quits.
In case you use direct connect to set the flag, or you set it directly, be sure to protect the read/write access of the flag with a QMutex to prevent inconsistent reads, or user a queued connection to ensure single thread access of the flag.
While highly discouraged, optionally you can use QThread's terminate() method to instantaneously kill the thread. You should never do this as it may cause memory leak, heap corruption, resource leaking and any nasty stuff as destructors and clean-up codes will not run, and the execution can be halted at an undesired state.
Related
I am trying to call a thread I define in a function from another function. Here is the first function, its purpose is to create and start a thread:
def startThread(func):
listen = threading.Thread(target = func)
listen.start()
I am trying to implement a function that will close the thread created in that first function, how should I go about it? I don't know how to successfully pass the thread.
def endThread(thread):
thread.exit()
Thank you!
This problem is almost FAQ material.
To summarise, there is no way to kill a thread from the outside. You can of course pass the thread object to any function you want, but threading library is missing kill and exit calls.
There are more or less two distinct ways around this, depending on what your thread does.
The first method is to make it so that your thread co-operates. This approach is discussed here: Is there any way to kill a Thread in Python? This method adds a check to your thread loop and a way to raise a "stop signal", which will then cause the thread to exit from the inside when detected.
This method works fine if your thread is a relatively busy loop. If it is something that is blocking in IO wait, not so much, as your thread could be blocking in a read call for days or weeks before receiving something and executing the signal check part. Many IO calls accept a timeout value, and if it is acceptable to wait a couple of seconds before your thread exits, you can use this to force the exit check every N seconds without making your thread a busy loop.
The other approach is to replace threads with processes. You can force kill a subprocess. If you can communicate with your main program with queues instead of shared variables, this is not too complicated, either. If your program relies heavily on sharing global variables, this would require a major redesign.
If your program is waiting in IO loops, you need instantaneous termination and you are using shared global variables, then you are somewhat out of luck, as you either need to accept your threads not behaving nicely or you need to redesign some parts of your code to untangle either the IO wait or shared variables.
I am using QThread to do some calculations in a separate Thread.
The Thread gets started by a button click, witch launches the function StartMeasurement().
The Thread can finish the process by itself (after finished the calculations)
and emits the PyQT Signal finished. Or the thread can be stopped by the User by the stopBtn click.
The terminate() function is working, but I get a lot of troubles when I try to start the thread again.
Is it recommendable to use the movetoThread() approach here?
Or how could I ensure that the thread is stopped correctly to enable a proper restart. (means, starting new!)
# starts the measurment in a Thread: StartMeasurement()
def StartMeasurement(self):
self.thread = measure.CMeasurementThread(self.osziObj, self.genObj, self.measSetup)
self.thread.newSample.connect(self.plotNewSample)
self.thread.finished.connect(self.Done)
self.stopBtn.clicked.connect(self.thread.terminate)
self.stopBtn.clicked.connect(self.Stop)
self.thread.start()
It's not a problem. The general practice when working with QThread is to connect its finished() signal to the deleteLater() slot of the objects that have been moved to the separate thread via moveToThread(). It's done in order to properly manage the memory when you then destroy your thread because it's assumed that you will first quit the thread and then destroy its instance. ;) This should tell you that stopping a thread has nothing to do with the destruction of those objects UNLESS you have established the connection I've described above.
It is perfectly fine to restart a thread IF you have stopped it properly using quit() and wait() to actually wait untill the stopping is completed.
However my advice is to not do that unless that extra thread has a huge impact on your application for some reason (highly unlikely with modern machines).
Instead of restarting the thread consider the following options:
implement a pause flag that just makes the thread run without doing anything if it's set to true (I've used this example of mine many times to demonstrate such behaviour (check the worker.cpp and the doWork() function in particular) - it's in C++ but it can be ported to PyQt in no time)
use QRunnable - its designed to run something and then (unless autoDelete is set to true) return to the thread pool. It's really nice if you have tasks that occur every once in a while and you don't need a constatly running separate thread. If you want to use signals and slots (to get the result of the calculation done inside the QRunnable::run() you will have to first inherit from QObject and then from QRunnable
Use futures (check the Qt Concurrent module)
I suggest that you first read the Example use cases for the various threading technologies Qt provides.
In a multi-threaded Python process I have a number of non-daemon threads, by which I mean threads which keep the main process alive even after the main thread has exited / stopped.
My non-daemon threads hold weak references to certain objects in the main thread, but when the main thread ends (control falls off the bottom of the file) these objects do not appear to be garbage collected, and my weak reference finaliser callbacks don't fire.
Am I wrong to expect the main thread to be garbage collected? I would have expected that the thread-locals would be deallocated (i.e. garbage collected)...
What have I missed?
Supporting materials
Output from pprint.pprint( threading.enumerate() ) showing the main thread has stopped while others soldier on.
[<_MainThread(MainThread, stopped 139664516818688)>,
<LDQServer(testLogIOWorkerThread, started 139664479889152)>,
<_Timer(Thread-18, started 139663928870656)>,
<LDQServer(debugLogIOWorkerThread, started 139664437925632)>,
<_Timer(Thread-17, started 139664463103744)>,
<_Timer(Thread-19, started 139663937263360)>,
<LDQServer(testLogIOWorkerThread, started 139664471496448)>,
<LDQServer(debugLogIOWorkerThread, started 139664446318336)>]
And since someone always asks about the use-case...
My network service occasionally misses its real-time deadlines (which causes a total system failure in the worst case). This turned out to be because logging of (important) DEBUG data would block whenever the file-system has a tantrum. So I am attempting to retrofit a number of established specialised logging libraries to defer blocking I/O to a worker thread.
Sadly the established usage pattern is a mix of short-lived logging channels which log overlapping parallel transactions, and long-lived module-scope channels which are never explicitly closed.
So I created a decorator which defers method calls to a worker thread. The worker thread is non-daemon to ensure that all (slow) blocking I/O completes before the interpreter exits, and holds a weak reference to the client-side (where method calls get enqueued). When the client-side is garbage collected the weak reference's callback fires and the worker thread knows no more work will be enqueued, and so will exit at its next convenience.
This seems to work fine in all but one important use-case: when the logging channel is in the main thread. When the main thread stops / exits the logging channel is not finalised, and so my (non-daemon) worker thread lives on keeping the entire process alive.
It's a bad idea for your main thread to end without calling join on all non-daemon threads, or to make any assumptions about what happens if you don't.
If you don't do anything very unusual, CPython (at least 2.0-3.3) will cover for you by automatically calling join on all non-daemon threads as pair of _MainThread._exitfunc. This isn't actually documented, so you shouldn't rely on it, but it's what's happening to you.
Your main thread hasn't actually exited at all; it's blocking inside its _MainThread._exitfunc trying to join some arbitrary non-daemon thread. Its objects won't be finalized until the atexit handler is called, which doesn't happen until after it finishes joining all non-daemon threads.
Meanwhile, if you avoid this (e.g., by using thread/_thread directly, or by detaching the main thread from its object or forcing it into a normal Thread instance), what happens? It isn't defined. The threading module makes no reference to it at all, but in CPython 2.0-3.3, and likely in any other reasonable implementation, it falls to the thread/_thread module to decide. And, as the docs say:
When the main thread exits, it is system defined whether the other threads survive. On SGI IRIX using the native thread implementation, they survive. On most other systems, they are killed without executing try ... finally clauses or executing object destructors.
So, if you manage to avoid joining all of your non-daemon threads, you have to write code that can handle both having them hard-killed like daemon threads, and having them continue running until exit.
If they do continue running, at least in CPython 2.7 and 3.3 on POSIX systems, that the main thread's OS-level thread handle, and various higher-level Python objects representing it, may be still retained, and not get cleaned up by the GC.
On top of that, even if everything were released, you can't rely on the GC ever deleting anything. If your code depends on deterministic GC, there are many cases you can get away with it in CPython (although your code will then break in PyPy, Jython, IronPython, etc.), but at exit time is not one of them. CPython can, and will, leak objects at exit time and let the OS sort 'em out. (This is why writable files that you never close may lose the last few writes—the __del__ method never gets called, and therefore there's nobody to tell them to flush, and at least on POSIX the underlying FILE* doesn't automatically flush either.)
If you want something to be cleaned up when the main thread finishes, you have to use some kind of close function rather than relying on __del__, and you have to make sure it gets triggered via a with block around the main block of code, an atexit function, or some other mechanism.
One last thing:
I would have expected that the thread-locals would be deallocated (i.e. garbage collected)...
Do you actually have thread locals somewhere? Or do you just mean locals and/or globals that are only accessed in one thread?
I have a program using a thread. When my program is closed, my thread is still running and that's normal. I would like to know how my thread can detect that the main program is terminated; by itself ONLY. How would I do that?
My thread is in an infinite loop and process many object in a Queue. I can't define my thread as a daemon, else I can lose some data at the end of the main program. I don't want that my main program set a boolean value when it closed.
If you can get a handle to the main thread, you can call is_alive() on it.
Alternatively, you can call threading.enumerate() to get a list of all currently living threads, and check to see if the main thread is in there.
Or if even that is impossible, then you might be able to check to see if the child thread is the only remaining non-daemon thread.
Would it work if your manager tracked how many open threads there were, then the children killed themselves when starved of input? So the parent would start pushing data on to the queue, and the workers would consume data from the queue. If a worker found nothing on the queue for a certain timeout period, it would kill itself. The main thread would then track how many workers were operating and periodically start new workers if the number of active workers were under a given threshold.
In my program I have a bunch of threads running and I'm trying
to interrupt the main thread to get it to do something asynchronously.
So I set up a handler and send the main process a SIGUSR1 - see the code
below:
def SigUSR1Handler(signum, frame):
self._logger.debug('Received SIGUSR1')
return
signal.signal(signal.SIGUSR1, SigUSR1Handler)
[signal.signal(signal.SIGUSR1, signal.SIG_IGN)]
In the above case, all the threads and the main process stops - from a 'c'
point of view this was unexpected - I want the threads to continue as they
were before the signal. If I put the SIG_IGN in instead, everything continues
fine.
Can somebody tell me how to do this? Maybe I have to do something with the 'frame'
manually to get back to where it was..just a guess though
thanks in advance,
Thanks for your help on this.
To explain a bit more, I have thread instances writing string information to
a socket which is also output to a file. These threads run their own timers so they
independently write their outputs to the socket. When the program runs I also see
their output on stdout but it all stops as soon as I see the debug line from the signal.
I need the threads to constantly send this info but I need the main program to
take a command so it also starts doing something else (in parallel) for a while.
I thought I'd just be able to send a signal from the command line to trigger this.
Mixing signals and threads is always a little precarious. What you describe should not happen, however. Python only handles signals in the main thread. If the OS delivered the signal to another thread, that thread may be briefly interrupted (when it's performing, say, a systemcall) but it won't execute the signal handler. The main thread will be asked to execute the signalhandler at the next opportunity.
What are your threads (including the main thread) actually doing when you send the signal? How do you notice that they all 'stop'? Is it a brief pause (easily explained by the fact that the main thread will need to acquire the GIL before handling the signal) or does the process break down entirely?
I'll sort-of answer my own question:
In my first attempt at this I was using time.sleep(run_time) in the main
thread to control how long the threads ran until they were stopped. By adding
debug I could see that the sleep loop seemed to be exiting as soon as the
signal handler returned so everything was shutting down normally but early!
I've replaced the sleep with a while loop and that doesn't jump out after
the signal handler returns so my threads keep running. So it solves the
problem but I'm still a bit puzzled about sleep()'s behaviour.
You should probably use a threading.Condition variable instead of sending signals. Have your main thread check it every loop and perform its special operation if it's been set.
If you insist on using signals, you'll want to move to using subprocess instead of threads, as your problem is likely due to the GIL.
Watch this presentation by David Beazley.
http://blip.tv/file/2232410
It also explains some quirky behavior related to threads and signals (Python specific, not the general quirkiness of the subject :-) ).
http://pyprocessing.berlios.de/ Pyprocessing is a neat library that makes it easier to work with separate processes in Python.