I am writing a class that creates threads that timeout if not used within a certain time. The class allows you to pump data to a specific thread (by keyword), and if it doesn't exist it creates the thread.
Anywho, the problem I have is main supervisor class doesn't know when threads have ended. I can't put blocking code like join or poll to see if it's alive. What I want is an event handler, that is called when a thread ends (or is just about to end) so that I can inform the supervisor that the thread is no longer active.
Is this something that can be done with signal or something similar?
As psuedocode, I'm looking for something like:
def myHandlerFunc():
# inform supervisor the thread is dead
t1 = ThreadFunc()
t1.eventHandler(condition=thread_dies, handler=myHandlerFunc)
EDIT: Perhaps a better way would be to pass a ref to the parent down to the thread, and have the thread tell parent class directly. I'm sure someone will tell me off for data flow inversion.
EDIT: Here is some psuedocode:
class supervisor():
def __init__:
Setup thread dict with all threads as inactive
def dispatch(target, message):
if(target thread inactive):
create new thread
send message to thread
def thread_timeout_handler():
# Func is called asynchronously when a thread dies
# Does some stuff over here
def ThreadFunc():
while( !timeout ):
wait for message:
do stuff with message
(Tell supervisor thread is closing?)
return
The main point is that you send messages to the threads (referenced by keyword) through the supervisor. The supervisor makes sure the thread is alive (since they timeout after a while), creates a new one if it dies, and sends the data over.
Looking at this again, it's easy to avoid needing an event handler as I can just check if the thread is alive using threadObj.isAlive() instead of dynamically keeping a dict of thread statuses.
But out of curiosity, is it possible to get a handler to be called in the supervisor class by signals sent from the thread? The main App code would call the supervisor.dispatch() function once, then do other stuff. It would later be interrupted by the thread_timeout_handler function, as the thread had closed.
You still don't mention if you are using a message/event loop framework, which would provide a way for you to dispatch a call to the "main" thread and call an event handler.
Assuming you're not, than you can't just interrupt or call into the main thread.
You don't need to, though, as you only need to know if a thread is alive when you decide if you need to create a new one. You can do your checking at this time. This way, you only need a way to communicate the "finished" state between threads. There are a lot of ways to do this (I've never used .isAlive(), but you can pass information back in a Queue, Event, or even a shared variable).
Using Event it would look something like this:
class supervisor():
def __init__:
Setup thread dict with all threads as inactive
def dispatch(target, message):
if(thread.event.is_set()):
create new thread
thread.event = Event()
send message to thread
def ThreadFunc(event):
while( !timeout ):
wait for message:
do stuff with message
event.set()
return
Note that this way there is still a possible race condition. The supervisor thread might check is_set() right before the worker thread calls .set() which will lie about the thread's ability to do work. The same problem would exist with isAlive().
Is there a reason you don't just use a threadpool?
Related
This question is very much related to this one, which doesn't have a solution, but it is not exactly the same.
I would like to ask if there is a way of launching a background task in PyQt, and be able to kill it by pressing a button.
My problem is that I have an user interface and some external (3rd party) functions that take a while to compute. In order to not frozen the user interface while the task are computing, I run them on the background using QThread and synchronize the UI when they finish using signals.
However, I would like to add the option for the external user to press a button and cancel the current task (because the task is not needed/desired anymore).
Something that to me looks as simple as a kill -9 *task* in linux, is quite hard/ofuscated to obtain in Qt.
Right now I'm using custom Qthreads of the form of:
mythread = Mythread()
mythread.finished.connect(mycallback)
mythread.start()
Where Mythread inherits QThread overriding the run method.
In the user interface, there is one button that tries to kill that thread by either using:
mythread.exit(0)
mythread.quit()
mythread.terminate()
None of them works... I'm aware that the documentation states that the terminate method does have weird behaviours...
So the question is.. I'm facing this problem wrong? How to kill a QThread? If is not possible, is there any alternative to this?
Thanks!
It's a very common mistake to try to kill a QThread in the way you're suggesting. This seems to be due to a failure to realise that it's the long-running task that needs to be stopped, rather than the thread itself.
The task was moved to the worker thread because it was blocking the main/GUI thread. But that situation doesn't really change once the task is moved. It will block the worker thread in exactly the same way that it was blocking the main thread. For the thread to finish, the task itself either has to complete normally, or be programmatically halted in some way. That is, something must be done to allow the thread's run() method to exit normally (which often entails breaking out of a blocking loop).
A common way to cancel a long-running task is via a simple stop-flag:
class Thread(QThread):
def stop(self):
self._flag = False
def run(self):
self._flag = True
for item in get_items():
process_item(item)
if not self._flag:
break
self._flag = False
This question is very much related to this one, which doesn't have a solution, but it is not exactly the same.
I would like to ask if there is a way of launching a background task in PyQt, and be able to kill it by pressing a button.
My problem is that I have an user interface and some external (3rd party) functions that take a while to compute. In order to not frozen the user interface while the task are computing, I run them on the background using QThread and synchronize the UI when they finish using signals.
However, I would like to add the option for the external user to press a button and cancel the current task (because the task is not needed/desired anymore).
Something that to me looks as simple as a kill -9 *task* in linux, is quite hard/ofuscated to obtain in Qt.
Right now I'm using custom Qthreads of the form of:
mythread = Mythread()
mythread.finished.connect(mycallback)
mythread.start()
Where Mythread inherits QThread overriding the run method.
In the user interface, there is one button that tries to kill that thread by either using:
mythread.exit(0)
mythread.quit()
mythread.terminate()
None of them works... I'm aware that the documentation states that the terminate method does have weird behaviours...
So the question is.. I'm facing this problem wrong? How to kill a QThread? If is not possible, is there any alternative to this?
Thanks!
It's a very common mistake to try to kill a QThread in the way you're suggesting. This seems to be due to a failure to realise that it's the long-running task that needs to be stopped, rather than the thread itself.
The task was moved to the worker thread because it was blocking the main/GUI thread. But that situation doesn't really change once the task is moved. It will block the worker thread in exactly the same way that it was blocking the main thread. For the thread to finish, the task itself either has to complete normally, or be programmatically halted in some way. That is, something must be done to allow the thread's run() method to exit normally (which often entails breaking out of a blocking loop).
A common way to cancel a long-running task is via a simple stop-flag:
class Thread(QThread):
def stop(self):
self._flag = False
def run(self):
self._flag = True
for item in get_items():
process_item(item)
if not self._flag:
break
self._flag = False
Related: Catch a thread's exception in the caller thread in Python
When I catch exception in a child thread:
I need to let the thread caller to know
In the related posting, accepted answer checks for queue which is fine but it prevents the main caller from doing anything else because its keep checking the queue
So it would be ideal if it's event based.
The solution should be: main caller does things (not continuously checking the queue for error in child thread) and if something goes bad inside child thread, child thread lets main caller know through some event and main caller process the error.
I ve been looking at different articles and solutions but all event stuff is directed towards main caller communicating to a child thread and not vice versa
any solution that a child thread communicating to the caller via events?
There is no such thing as a "thread caller". All threads are equal.
In your head the "main thread" is something special because Python gives that to you for free but as far as Python is concerned, the main thread is just a tread like any other.
So your question boils down to: How can I exchange information between two threads?
With one or more queues.
Usually, the pseudo code looks like this:
Create N workers, connected to an input and an output queue.
Put work into the input queue
Wait for results on the output queue
Join the threads
Note that you don't have to block in step #3. You can also ask the queue whether it has any elements. If so, then get() won't block. If it's empty, then you can do other things.
Step #4 is often more simple to implement, when you define a "kill" command to which the threads respond with a "I'm done" reply.
You can then send N kill commands and wait for the N replies. Afterwards, you can be sure that all threads are done.
I have a class that inherits from threading.Thread.
After processing an item from a queue I want it to call a function that I handed over to the thread when starting it. Doing that callback from the run-Method will have it run in my worker thread. Is there a way to run that callback inside the MainThread?
Thanks & Greetings,
Sean
You didnt mention intially that you are using pyqt. Qt has signals and slots built in just for this purpose. If you are launching your thread using QThread then it has the 'finished' signal that your main thread can be connected to on a slot. If you need even more control of the resulting signal you can emit a custom one from the thread with any value you want. When you construct the thread just connect its signal to the slot that you want to act as the callback.
Otherwise you would be blocking your main event loop if you are specifically waiting on the thread.
Here is a link specifically about the new style signal/slots: http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/new_style_signals_slots.html
It makes it extremely easy now to create a python signal with whatever signature you want and connect it in an object oriented fashion to a slot.
Also as a side note about PyQt... Some of the other answers here have suggested creating a loop on your main thread that waits to collect responses in a queue from its dispatched threads. That is essentially what PyQt does. When you start your app the main thread goes into an event loop. This event loop is processing events that stack up in its event queue. There are different types of signal/slot connections. If your two endpoints are in the same thread, then a direct connection will be made and the emitting signal will call the slot. If they are in different threads, then I believe it goes through the event loop so that it can be processed outside of the emitters thread.
Push the method onto a queue and have the main thread sit and wait on that queue, running each method as it pulls them off.
You can use condition variables and have your main thread wait on a message from the sub-thread. The sub-thread can signal the main thread to call that method.
I'm using Python in a webapp (CGI for testing, FastCGI for production) that needs to send an occasional email (when a user registers or something else important happens). Since communicating with an SMTP server takes a long time, I'd like to spawn a thread for the mail function so that the rest of the app can finish up the request without waiting for the email to finish sending.
I tried using thread.start_new(func, (args)), but the Parent return's and exits before the sending is complete, thereby killing the sending process before it does anything useful. Is there anyway to keep the process alive long enough for the child process to finish?
Take a look at the thread.join() method. Basically it will block your calling thread until the child thread has returned (thus preventing it from exiting before it should).
Update:
To avoid making your main thread unresponsive to new requests you can use a while loop.
while threading.active_count() > 0:
# ... look for new requests to handle ...
time.sleep(0.1)
# or try joining your threads with a timeout
#for thread in my_threads:
# thread.join(0.1)
Update 2:
It also looks like thread.start_new(func, args) is obsolete. It was updated to thread.start_new_thread(function, args[, kwargs]) You can also create threads with the higher level threading package (this is the package that allows you to get the active_count() in the previous code block):
import threading
my_thread = threading.Thread(target=func, args=(), kwargs={})
my_thread.daemon = True
my_thread.start()
You might want to use threading.enumerate, if you have multiple workers and want to see which one(s) are still running.
Other alternatives include using threading.Event---the main thread sets the event to True and starts the worker thread off. The worker thread unsets the event when if finishes work, and the main check whether the event is set/unset to figure out if it can exit.