I want to know: what is the best practice for killing threads started by a main Python application in the case the main application receives a SIGINT?
I am doing the following thing, but I HIGHLY suspect that because needing to kill other started threads is such a common problem, that probably there is a better way to do it:
class Handler(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.keep_go = True
def run(self):
while self.keep_go:
#do something
def stop(self): #seems like i shouldn't have to do this myself
self.keep_go = False
try:
h = Handler()
h.start()
while True: #ENTER SOME OTHER LOOP HERE
#do something else
except KeyboardInterrupt: #seems like i shouldn't have to do this myself
pass
finally:
h.stop()
The following post is related, but it is not clear to me what the actual recommended practice is, because the answers are more of a "here's some possibly hackish way you can do this". Also, I do not need to kill somethng "abruptly"; I am ok with doing it "the right way": Is there any way to kill a Thread in Python?
Edit: I guess one minor flaw with my approach is that it does not kill the current processing in the while loop. It does not receive a "kill event" that "rolls" back this loop as a transaction, nor does it halt the remainder of the loop.
I usually just set each thread's daemon attribute to True. That way, when the main thread terminates, so does everything else.
The documentation has a little more to say on the matter:
Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
Related
This question is very much related to this one, which doesn't have a solution, but it is not exactly the same.
I would like to ask if there is a way of launching a background task in PyQt, and be able to kill it by pressing a button.
My problem is that I have an user interface and some external (3rd party) functions that take a while to compute. In order to not frozen the user interface while the task are computing, I run them on the background using QThread and synchronize the UI when they finish using signals.
However, I would like to add the option for the external user to press a button and cancel the current task (because the task is not needed/desired anymore).
Something that to me looks as simple as a kill -9 *task* in linux, is quite hard/ofuscated to obtain in Qt.
Right now I'm using custom Qthreads of the form of:
mythread = Mythread()
mythread.finished.connect(mycallback)
mythread.start()
Where Mythread inherits QThread overriding the run method.
In the user interface, there is one button that tries to kill that thread by either using:
mythread.exit(0)
mythread.quit()
mythread.terminate()
None of them works... I'm aware that the documentation states that the terminate method does have weird behaviours...
So the question is.. I'm facing this problem wrong? How to kill a QThread? If is not possible, is there any alternative to this?
Thanks!
It's a very common mistake to try to kill a QThread in the way you're suggesting. This seems to be due to a failure to realise that it's the long-running task that needs to be stopped, rather than the thread itself.
The task was moved to the worker thread because it was blocking the main/GUI thread. But that situation doesn't really change once the task is moved. It will block the worker thread in exactly the same way that it was blocking the main thread. For the thread to finish, the task itself either has to complete normally, or be programmatically halted in some way. That is, something must be done to allow the thread's run() method to exit normally (which often entails breaking out of a blocking loop).
A common way to cancel a long-running task is via a simple stop-flag:
class Thread(QThread):
def stop(self):
self._flag = False
def run(self):
self._flag = True
for item in get_items():
process_item(item)
if not self._flag:
break
self._flag = False
I am trying to call a thread I define in a function from another function. Here is the first function, its purpose is to create and start a thread:
def startThread(func):
listen = threading.Thread(target = func)
listen.start()
I am trying to implement a function that will close the thread created in that first function, how should I go about it? I don't know how to successfully pass the thread.
def endThread(thread):
thread.exit()
Thank you!
This problem is almost FAQ material.
To summarise, there is no way to kill a thread from the outside. You can of course pass the thread object to any function you want, but threading library is missing kill and exit calls.
There are more or less two distinct ways around this, depending on what your thread does.
The first method is to make it so that your thread co-operates. This approach is discussed here: Is there any way to kill a Thread in Python? This method adds a check to your thread loop and a way to raise a "stop signal", which will then cause the thread to exit from the inside when detected.
This method works fine if your thread is a relatively busy loop. If it is something that is blocking in IO wait, not so much, as your thread could be blocking in a read call for days or weeks before receiving something and executing the signal check part. Many IO calls accept a timeout value, and if it is acceptable to wait a couple of seconds before your thread exits, you can use this to force the exit check every N seconds without making your thread a busy loop.
The other approach is to replace threads with processes. You can force kill a subprocess. If you can communicate with your main program with queues instead of shared variables, this is not too complicated, either. If your program relies heavily on sharing global variables, this would require a major redesign.
If your program is waiting in IO loops, you need instantaneous termination and you are using shared global variables, then you are somewhat out of luck, as you either need to accept your threads not behaving nicely or you need to redesign some parts of your code to untangle either the IO wait or shared variables.
I'm working on a python project were I want the same behavior as in C for my threads. In C when the main thread exit, it kills all other threads.
The project contains a TCP error server that it used to get logs from other threads and other software .The TCP link is simplex.
Some errors must involve the end of the whole program.
For external software I can kill them using their PID.
For other threads I've tried sys._exit(), sometimes it works, and sometimes some threads remain.
If my other threads were looping I could use a semaphore or something like that, but it is only one iteration of a linear process.
I've thought about the design pattern Producer/Consumer or add a lot of lock.acquire()/lock.release() but I think it will add more complexity and it imply to break the linear thread.
I've had a look to other Stackoverflow question I've found those solutions:
Use sys._exit() but its success rate is not 100%.
Convert my threads into subprocess to kill them easily, but in my case I can't.
I'm looking for a solution, a design pattern or something else to solve it.
PS: I'm a C lover and each time I deal with Python I think to solutions as simple as to call exit() to kill all my threads.
If you make your worker threads daemon threads, they will die when all your non-daemon threads (e.g. the main thread) have exited.
http://docs.python.org/library/threading.html#threading.Thread.daemon
Thread daemon status isDaemon() is False, set it True by setDaemon(True)
Another solution :
To make the thread stop on Keyboard Interrupt signal (ctrl+c) you can catch the exception "KeyboardInterrup" and cleanup before exiting. Like this:
try:
start_thread() #And the rest of your main
except (KeyboardInterrupt, SystemExit):
cleanup_stop_thread();
sys.exit()
I am creating a custom job scheduler with a web frontend in python 3.4 on linux. This program creates a daemon (consumer) thread that waits for jobs to come available in a PriorityQueue. These jobs can manually be added through the web interface which adds them to the queue. When the consumer thread finds a job, it executes a program using subprocess.run, and waits for it to finish.
The basic idea of the worker thread:
class Worker(threading.Thread):
def __init__(self, queue):
self.queue = queue
# more code here
def run(self):
while True:
try:
job = self.queue.get()
#do some work
proc = subprocess.run("myprogram", timeout=my_timeout)
#do some more things
except TimeoutExpired:
#do some administration
self.queue.add(job)
However:
This consumer should be able to receive some kind of signal from the frontend (main thread) that it should stop the current job and instead work on the next job in the queue (saving the state of the current job and adding it to the end of the queue again). This can (and will most likely) happen while blocked on subprocess.run().
The subprocesses can simply be killed (the program that is executed saves sme state in a file) but the worker thread needs to do some administration on the killed job to make sure it can be resumed later on.
There can be multiple such worker threads.
Signal handlers are not an option (since they are always handled by the main thread which is a webserver and should not be bothered with this).
Having an event loop in which the process actively polls for events (such as the child exiting, the timeout occurring or the interrupt event) is in this context not really a solution but an ugly hack. The jobs are performance-heavy and constant context switches are unwanted.
What synchronization primitives should I use to interrupt this thread or to make sure it waits for several events at the same time in a blocking fashion?
I think you've accidentally glossed over a simple solution: your second bullet point says that you have the ability to kill the programs that are running in subprocesses. Notice that subprocess.call returns the return code of the subprocess. This means that you can let the main thread kill the subprocess, and just check the return code to see if you need to do any cleanup. Even better, you could use subprocess.check_call instead, which will raise an exception for you if the returncode isn't 0. I don't know what platform you're working on, but on Linux, killed processes generally don't return a 0 if they're killed.
It could look something like this:
class Worker(threading.Thread):
def __init__(self, queue):
self.queue = queue
# more code here
def run(self):
while True:
try:
job = self.queue.get()
#do some work
subprocess.check_call("myprogram", timeout=my_timeout)
#do some more things
except (TimeoutExpired, subprocess.CalledProcessError):
#do some administration
self.queue.add(job)
Note that if you're using Python 3.5, you can use subprocess.run instead, and set the check argument to True.
If you have a strong need to handle the cases where the worker needs to be interrupted when it isn't running the subprocess, then I think you're going to have to use a polling loop, because I don't think the behavior you're looking for is supported for threads in Python. You can use a threading.Event object to pass the "stop working now" pseudo-signal from your main thread to the worker, and have the worker periodically check the state of that event object.
If you're willing to consider using multiple processing stead of threads, consider switching over to the multiprocessing module, which would allow you to handle signals. There is more overhead to spawning full-blown subprocesses instead of threads, but you're essentially looking for signal-like asynchronous behavior, and I don't think Python's threading library supports anything like that. One benefit though, would be that you would be freed from the Global Interpreter Lock(PDF link), so you may actually see some speed benefits if your worker processes (formerly threads) are doing anything CPU intensive.
This question is very much related to this one, which doesn't have a solution, but it is not exactly the same.
I would like to ask if there is a way of launching a background task in PyQt, and be able to kill it by pressing a button.
My problem is that I have an user interface and some external (3rd party) functions that take a while to compute. In order to not frozen the user interface while the task are computing, I run them on the background using QThread and synchronize the UI when they finish using signals.
However, I would like to add the option for the external user to press a button and cancel the current task (because the task is not needed/desired anymore).
Something that to me looks as simple as a kill -9 *task* in linux, is quite hard/ofuscated to obtain in Qt.
Right now I'm using custom Qthreads of the form of:
mythread = Mythread()
mythread.finished.connect(mycallback)
mythread.start()
Where Mythread inherits QThread overriding the run method.
In the user interface, there is one button that tries to kill that thread by either using:
mythread.exit(0)
mythread.quit()
mythread.terminate()
None of them works... I'm aware that the documentation states that the terminate method does have weird behaviours...
So the question is.. I'm facing this problem wrong? How to kill a QThread? If is not possible, is there any alternative to this?
Thanks!
It's a very common mistake to try to kill a QThread in the way you're suggesting. This seems to be due to a failure to realise that it's the long-running task that needs to be stopped, rather than the thread itself.
The task was moved to the worker thread because it was blocking the main/GUI thread. But that situation doesn't really change once the task is moved. It will block the worker thread in exactly the same way that it was blocking the main thread. For the thread to finish, the task itself either has to complete normally, or be programmatically halted in some way. That is, something must be done to allow the thread's run() method to exit normally (which often entails breaking out of a blocking loop).
A common way to cancel a long-running task is via a simple stop-flag:
class Thread(QThread):
def stop(self):
self._flag = False
def run(self):
self._flag = True
for item in get_items():
process_item(item)
if not self._flag:
break
self._flag = False