We Keep Coding

How can I terminate a thread in Python3 after a specified timeout? [duplicate]

I would like to run a process in a thread (which is iterating over a large database table). While the thread is running, I just want the program to wait. If that thread takes longer then 30 seconds, I want to kill the thread and do something else. By killing the thread, I mean that I want it to cease activity and release resources gracefully.
I figured the best way to do this was through a Thread()'s join(delay) and is_alive() functions, and an Event. Using the join(delay) I can have my program wait 30 seconds for the thread to finish, and by using the is_alive() function I can determine if the thread has finished its work. If it hasn't finished its work, the event is set, and the thread knows to stop working at that point.
Is this approach valid, and is this the most pythonic way to go about my problem statement?
Here is some sample code:
import threading
import time
# The worker loops for about 1 minute adding numbers to a set
# unless the event is set, at which point it breaks the loop and terminates
def worker(e):
data = set()
for i in range(60):
data.add(i)
if not e.isSet():
print "foo"
time.sleep(1)
else:
print "bar"
break
e = threading.Event()
t = threading.Thread(target=worker, args=(e,))
t.start()
# wait 30 seconds for the thread to finish its work
t.join(30)
if t.is_alive():
print "thread is not done, setting event to kill thread."
e.set()
else:
print "thread has already finished."

Using an Event in this case is works just fine as the signalling mechanism, and
is actually recommended in the threading module docs.
If you want your threads to stop gracefully, make them non-daemonic and use a
suitable signalling mechanism such as an Event.
When verifying thread termination, timeouts almost always introduce room for
error. Therefore, while using the .join() with a timeout for the initial
decision to trigger the event is fine, final verification should be made using a
.join() without a timeout.
# wait 30 seconds for the thread to finish its work
t.join(30)
if t.is_alive():
print "thread is not done, setting event to kill thread."
e.set()
# The thread can still be running at this point. For example, if the
# thread's call to isSet() returns right before this call to set(), then
# the thread will still perform the full 1 second sleep and the rest of
# the loop before finally stopping.
else:
print "thread has already finished."
# Thread can still be alive at this point. Do another join without a timeout
# to verify thread shutdown.
t.join()
This can be simplified to something like this:
# Wait for at most 30 seconds for the thread to complete.
t.join(30)
# Always signal the event. Whether the thread has already finished or not,
# the result will be the same.
e.set()
# Now join without a timeout knowing that the thread is either already
# finished or will finish "soon."
t.join()

I'm way late to this game, but I've been wrestling with a similar question and the following appears to both resolve the issue perfectly for me AND lets me do some basic thread state checking and cleanup when the daemonized sub-thread exits:
import threading
import time
import atexit
def do_work():
i = 0
#atexit.register
def goodbye():
print ("'CLEANLY' kill sub-thread with value: %s [THREAD: %s]" %
(i, threading.currentThread().ident))
while True:
print i
i += 1
time.sleep(1)
t = threading.Thread(target=do_work)
t.daemon = True
t.start()
def after_timeout():
print "KILL MAIN THREAD: %s" % threading.currentThread().ident
raise SystemExit
threading.Timer(2, after_timeout).start()
Yields:
0
1
KILL MAIN THREAD: 140013208254208
'CLEANLY' kill sub-thread with value: 2 [THREAD: 140013674317568]

I was also struggling to close a thread that was waiting to receive a notification.
Tried solution given here by user5737269 but it didn't really work for me. It was getting stuck in second join statement(without timeout one). Struggled a lot but didn't find any solution to this problem. Got this solution after thinking sometime:
My thread is waiting to receive a message in que. I want to close this thread, if no notification is received for 20 seconds. So, after 20 seconds, I am writing a message to this que so that thread terminates on its own.
Here's code:
q = Queue.Queue()
t.join(20)
if t.is_alive():
print("STOPPING THIS THREAD ....")
q.put("NO NOTIFICATION RECEIVED")
t.join(20)
else:
print("Thread completed successfully!!")
This worked for me.. Hope this idea helps someone!

How to stop threads running infinite loops in python?

I've made a program which has a main thread that spawns many other threads by subclassing the threading.Thread class.
Each such child thread runs an infinite while loop, and inside the loop I check a condition. If the condition is true, I make the thread sleep for 1 second using time.sleep(1) and if it's false, then the thread performs some computation.
The program itself works fine and I've achieved what I wanted to do, my only remaining problem is that I seem unable to stop the threads after my work is done. I want the user to be able to kill all the threads by pressing a button or giving a keyboard interrupt like Ctrl+C.
For this I had tried using the signal module and inserted a conditon in the threads' loops that breaks the loop when the main thread catches a signal but it didn't work for some reason. Can anyone please help with this?
EDIT: This is some of the relevant code snippets:
def sighandler(signal,frame):
BaseThreadClass.stop_flag = True
class BaseThreadClass(threading.Thread):
stop_flag = False
def __init__(self):
threading.Thread.__init__(self)
def run(self,*args):
while True:
if condition:
time.sleep(1)
else:
#do computation and stuff
if BaseThreadClass.stop_flag:
#do cleanup
break

Your basic method does work, but you've still not posted enough code to show the flaw. I added a few lines of code to make it runnable and produced a result like:
$ python3 test.py
thread alive
main alive
thread alive
main alive
^CSignal caught
main alive
thread alive
main alive
main alive
main alive
^CSignal caught
^CSignal caught
main alive
^Z
[2]+ Stopped python3 test.py
$ kill %2
The problem demonstrated above involves the signal handler telling all the threads to exit, except the main thread, which still runs and still catches interrupts. The full source of this variant of the sample snippet is:
import threading, signal, time
def sighandler(signal,frame):
BaseThreadClass.stop_flag = True
print("Signal caught")
class BaseThreadClass(threading.Thread):
stop_flag = False
def __init__(self):
threading.Thread.__init__(self)
def run(self,*args):
while True:
if True:
time.sleep(1)
print("thread alive")
else:
#do computation and stuff
pass
if BaseThreadClass.stop_flag:
#do cleanup
break
signal.signal(signal.SIGINT, sighandler)
t = BaseThreadClass()
t.start()
while True:
time.sleep(1)
print("main alive")
The problem here is that the main thread never checks for the quit condition. But as you never posted what the main thread does, nor how the signal handler is activated, or information regarding whether threads may go a long time without checking the quit condition... I still don't know what went wrong in your program. The signal example shown in the library documentation raises an exception in order to divert the main thread.
Signals are a rather low level concept for this task, however. I took the liberty of writing a somewhat more naïve version of the main thread:
try:
t = BaseThreadClass()
t.start()
while True:
time.sleep(1)
print("main alive")
except KeyboardInterrupt:
BaseThreadClass.stop_flag = True
t.join()
This version catches the exception thrown by the default interrupt handler, signals the thread to stop, and waits for it to do so. It might even be appropriate to change the except clause to a finally, since we could want to clean the threads up on other errors too.

If you want to do this kind of "cooperative" polled-shutdown, you can use a threading.Event to signal:
import threading
import time
def proc1():
while True:
print("1") # payload
time.sleep(1)
# have we been signalled to stop?
if not ev1.wait(0): break
# do any shutdown etc. here
print ("T1 exiting")
ev1 = threading.Event()
ev1.set()
thread1 = threading.Thread(target=proc1)
thread1.start()
time.sleep(3)
# signal thread1 to stop
ev1.clear()
But be aware that if the "payload" does something blocking like network or file IO, that op will not be interrupted. You can do those blocking ops with a timeout, but that obviously will complicate your code.

Python : how to stop a thread that's waiting for a .recv()

I have this thread running :
def run(self):
while 1:
msg = self.connection.recv(1024).decode()
I wish I could end this thread when I close the Tkinter Window like this :
self.window.protocol('WM_DELETE_WINDOW', self.closeThreads)
def closeThreads(self):
self.game.destroy()
#End the thread
Can't use thread._close() because it is deprecated and python 3.4 does not allow it.

The only really satisfactory solution I've seen for this problem is not to allow your thread to block inside recv(). Instead, set the socket to non-blocking and have the thread block inside select() instead. The advantage of blocking inside select() is that you can tell select() to return when any one of several sockets becomes ready-for-read, which brings us to the next part: as part of setting up your thread, create a second socket (either a locally-connected TCP socket e.g. as provided by socketpair, or a UDP socket listening on a port for packets from localhost). When your main thread wants your networking thread to go away, your main thread should send a byte to that socket (or in the TCP case, the main thread could just close its end of the socket-pair). That will cause select() to return ready-for-read on that socket, and when your network thread realizes that the socket is marked ready-for-read, it should respond by exiting immediately.
The advantages of doing it that way are that it works well on all OS's, always reacts immediately (unlike a polling/timeout solution), takes up zero extra CPU cycles when the network is idle, and doesn't have any nasty side effects in multithreaded environments. The downside is that it uses up a couple of extra sockets, but that's usually not a big deal.

Two solutions:
1) Don't stop the thread, just allow it to die when the process exits with sys.exit()
2) Start the thread with a "die now" flag. The Event class is specifically designed to signal one thread from another.
The following example starts a thread, which connects to a server. Any data is handled, and if the parent signals the thread to exit, it will. As an additional safety feature we have an alarm signal to kill everything, just it case something gets out of hand.
source
import signal, socket, threading
class MyThread(threading.Thread):
def __init__(self, conn, event):
super(MyThread,self).__init__()
self.conn = conn
self.event = event
def handle_data(self):
"process data if any"
try:
data = self.conn.recv(4096)
if data:
print 'data:',data,len(data)
except socket.timeout:
print '(timeout)'
def run(self):
self.conn.settimeout(1.0)
# exit on signal from caller
while not self.event.is_set():
# handle any data; continue loop after 1 second
self.handle_data()
print 'got event; returning to caller'
sock = socket.create_connection( ('example.com', 80) )
event = threading.Event()
# connect to server and start connection handler
th = MyThread(conn=sock, event=event)
# watchdog: kill everything in 3 seconds
signal.alarm(3)
# after 2 seconds, tell data thread to exit
threading.Timer(2.0, event.set).start()
# start data thread and wait for it
th.start()
th.join()
output
(timeout)
(timeout)
got event; returning to caller

Python how to kill threads blocked on queue with signals?

I start a bunch of threads working on a queue and I want to kill them when sending the SIGINT (Ctrl+C). What is the best way to handle this?
targets = Queue.Queue()
threads_num = 10
threads = []
for i in threads_num:
t = MyThread()
t.setDaemon(True)
threads.append(t)
t.start()
targets.join()

If you are not interested in letting the other threads shut down gracefully, simply start them in daemon mode and wrap the join of the queue in a terminator thread.
That way, you can make use of the join method of the thread -- which supports a timeout and does not block off exceptions -- instead of having to wait on the queue's join method.
In other words, do something like this:
term = Thread(target=someQueueVar.join)
term.daemon = True
term.start()
while (term.isAlive()):
term.join(3600)
Now, Ctrl+C will terminate the MainThread whereupon the Python Interpreter hard-kills all threads marked as "daemons". Do note that this means that you have to set "Thread.daemon" for all the other threads or shut them down gracefully by catching the correct exception (KeyboardInterrupt or SystemExit) and doing whatever needs to be done for them to quit.
Do also note that you absolutely need to pass a number to term.join(), as otherwise it will, too, ignore all exceptions. You can select an arbitrarily high number, though.

Isn't Ctrl+C SIGINT?
Anyway, you can install a handler for the appropriate signal, and in the handler:
set a global flag that instructs the workers to exit, and make sure they check it periodically
or put 10 shutdown tokens on the queue, and have the workers exit when they pop this magic token
or set a flag which instructs the main thread to push those tokens, make sure the main thread checks that flag
etc. Mostly it depends on the structure of the application you're interrupting.

One way to do it is to install a signal handler for SIGTERM that directly calls os._exit(signal.SIGTERM). However unless you specify the optional timeout argument to Queue.get the signal handler function will not run until after the get method returns. (That's completely undocumented; I discovered that on my own.) So you can specify sys.maxint as the timeout and put your Queue.get call in a retry loop for purity to get around that.

Why don't you set timeouts for any operation on the queue? Then your threads can regular check if they have to finish by checking if an Event is raised.

This is how I tackled this.
class Worker(threading.Thread):
def __init__(self):
self.shutdown_flag = threading.Event()
def run(self):
logging.info('Worker started')
while not self.shutdown_flag.is_set():
try:
task = self.get_task_from_queue()
except queue.Empty:
continue
self.process_task(task)
def get_task_from_queue(self) -> Task:
return self.task_queue.get(block=True, timeout=10)
def shutdown(self):
logging.info('Shutdown received')
self.shutdown_flag.set()
Upon receiving a signal the main thread sets the shutdown event on workers. The workers wait on a blocking queue, but keep checking every 10 seconds if they have received a shutdown signal.

I managed to solve the problem by emptying the queue on KeyboardInterrupt and letting threads to gracefully stop themselves.
I don't know if it's the best way to handle this but is simple and quite clean.
targets = Queue.Queue()
threads_num = 10
threads = []
for i in threads_num:
t = MyThread()
t.setDaemon(True)
threads.append(t)
t.start()
while True:
try:
# If the queue is empty exit loop
if self.targets.empty() is True:
break
# KeyboardInterrupt handler
except KeyboardInterrupt:
print "[X] Interrupt! Killing threads..."
# Substitute the old queue with a new empty one and exit loop
targets = Queue.Queue()
break
# Join every thread on the queue normally
targets.join()

Python - Cancel timer thread

I'm trying to create a method which runs on a timer in the background of my main script:
def hello_world(self):
print 'Hello!'
threading.Timer(2,hello_world).start()
if __name__ == "__main__":
try:
hello_world()
except KeyboardInterrupt:
print '\nGoodbye!'
I am getting this message when I attempt to keyboard interrupt my script:
Exception KeyboardInterrupt in <module 'threading' from '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py'> ignored
How do I close the thread so that I can exit my application cleanly?

To elaborate a bit on Aphex's answer, the main thread can't possibly catch the KeyboardInterrupt signal, unless you have very fast fingers. The main thread exits almost immediately! Try this:
import threading
def hello_world():
print 'Hello!'
threading.Timer(2,hello_world).start()
if __name__ == "__main__":
try:
hello_world()
except KeyboardInterrupt:
print '\nGoodbye!'
print "main thread exited"
More generally, I would not suggest using a self-calling timer like this, just because it creates a lot of threads. Just create one thread and call time.sleep inside it.
However, as long as you keep the main thread running, you seem to be able to catch KeyboardInterrupt inside. The trick then is to make the thread a daemon thread that exits when the main thread exits.
import threading
import time
def hello_world():
while(True):
print 'Hello!'
time.sleep(2)
if __name__ == "__main__":
hw_thread = threading.Thread(target = hello_world)
hw_thread.daemon = True
hw_thread.start()
try:
time.sleep(1000)
except KeyboardInterrupt:
print '\nGoodbye!'
This exits automatically after 1000 seconds -- you could make that number even bigger if you like. You could also use a busy-loop to repeat the sleep call, but I don't really see the point.

You just need to set the Timer thread as a daemon
def hello_world(self):
print 'Hello!'
t = threading.Timer(2,hello_world)
t.daemon = True
t.start()
That will cause it to exit when the main thread exits, e.g. due to KeyboardInterrupt.
The daemon setting causes the entire program to exit when the only threads left are daemon threads.

Try re-raising the KeyboardInterrupt exception: http://effbot.org/zone/stupid-exceptions-keyboardinterrupt.htm
This still may not work though; chances are you're running into this caveat:
Threads interact strangely with interrupts: the KeyboardInterrupt exception will be received by an arbitrary thread. (When the signal module is available, interrupts always go to the main thread.)
In short, you can't be sure that the KeyboardInterrupt is going to your main thread. To workaround this, you may want to look into the signal module.
Edit: A more elegant way to cancel the thread is to have a shared variable that the thread looks at, and exits if it becomes false. Then if you want to kill the thread from your main thread, you set the variable to false.