I've made a program which has a main thread that spawns many other threads by subclassing the threading.Thread class.
Each such child thread runs an infinite while loop, and inside the loop I check a condition. If the condition is true, I make the thread sleep for 1 second using time.sleep(1) and if it's false, then the thread performs some computation.
The program itself works fine and I've achieved what I wanted to do, my only remaining problem is that I seem unable to stop the threads after my work is done. I want the user to be able to kill all the threads by pressing a button or giving a keyboard interrupt like Ctrl+C.
For this I had tried using the signal module and inserted a conditon in the threads' loops that breaks the loop when the main thread catches a signal but it didn't work for some reason. Can anyone please help with this?
EDIT: This is some of the relevant code snippets:
def sighandler(signal,frame):
BaseThreadClass.stop_flag = True
class BaseThreadClass(threading.Thread):
stop_flag = False
def __init__(self):
threading.Thread.__init__(self)
def run(self,*args):
while True:
if condition:
time.sleep(1)
else:
#do computation and stuff
if BaseThreadClass.stop_flag:
#do cleanup
break
Your basic method does work, but you've still not posted enough code to show the flaw. I added a few lines of code to make it runnable and produced a result like:
$ python3 test.py
thread alive
main alive
thread alive
main alive
^CSignal caught
main alive
thread alive
main alive
main alive
main alive
^CSignal caught
^CSignal caught
main alive
^Z
[2]+ Stopped python3 test.py
$ kill %2
The problem demonstrated above involves the signal handler telling all the threads to exit, except the main thread, which still runs and still catches interrupts. The full source of this variant of the sample snippet is:
import threading, signal, time
def sighandler(signal,frame):
BaseThreadClass.stop_flag = True
print("Signal caught")
class BaseThreadClass(threading.Thread):
stop_flag = False
def __init__(self):
threading.Thread.__init__(self)
def run(self,*args):
while True:
if True:
time.sleep(1)
print("thread alive")
else:
#do computation and stuff
pass
if BaseThreadClass.stop_flag:
#do cleanup
break
signal.signal(signal.SIGINT, sighandler)
t = BaseThreadClass()
t.start()
while True:
time.sleep(1)
print("main alive")
The problem here is that the main thread never checks for the quit condition. But as you never posted what the main thread does, nor how the signal handler is activated, or information regarding whether threads may go a long time without checking the quit condition... I still don't know what went wrong in your program. The signal example shown in the library documentation raises an exception in order to divert the main thread.
Signals are a rather low level concept for this task, however. I took the liberty of writing a somewhat more naïve version of the main thread:
try:
t = BaseThreadClass()
t.start()
while True:
time.sleep(1)
print("main alive")
except KeyboardInterrupt:
BaseThreadClass.stop_flag = True
t.join()
This version catches the exception thrown by the default interrupt handler, signals the thread to stop, and waits for it to do so. It might even be appropriate to change the except clause to a finally, since we could want to clean the threads up on other errors too.
If you want to do this kind of "cooperative" polled-shutdown, you can use a threading.Event to signal:
import threading
import time
def proc1():
while True:
print("1") # payload
time.sleep(1)
# have we been signalled to stop?
if not ev1.wait(0): break
# do any shutdown etc. here
print ("T1 exiting")
ev1 = threading.Event()
ev1.set()
thread1 = threading.Thread(target=proc1)
thread1.start()
time.sleep(3)
# signal thread1 to stop
ev1.clear()
But be aware that if the "payload" does something blocking like network or file IO, that op will not be interrupted. You can do those blocking ops with a timeout, but that obviously will complicate your code.
Related
In this script:
import threading, socket
class send(threading.Thread):
def run(self):
try:
while True:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((url,port))
s.send(b"Hello world!")
print ("Request Sent!")
except:
s.close()
except KeyboardInterrupt:
# here i'd like to kill all threads if possible
for x in range(800):
send().start()
Is it possible to kill all threads in the except of KeyboardInterrupt? I've searched on the net and yeah, I know that it has been already asked, but I'm really new in python and I didn't get so well the method of these other question asked on stack.
No. Individual threads can't be terminated forcibly (it's unsafe, since it could leave locks held, leading to deadlocks, among other things).
Two ways to do something like this would be to either:
Have all threads launched as daemon threads, with the main thread waiting on an Event/Condition and exiting as soon as one of the threads sets the Event or notifies the Condition. The process terminates as soon as the (sole) non-daemon thread exits, ending all the daemon threads
Use a shared Event that all the threads poll intermittently, so they cooperatively exit shortly after it is set.
As a comment said, use exit(n) to terminate a threads, but it might not terminate all threads necessarily and threads will mostly continue to run.
So your approach is to use os._exit(n), This terminates all threads and does what you want.
But be warned os._exit is not a good way to exit a script, and should be used only in special cases.
#Allexj: You got an accurate answer from #ShadowRanger. You can do it as shown below.
import threading, socket, sys
event = Threading.Event() # define an Event
class send(threading.Thread):
def run(self):
try:
global event
while True:
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((url,port))
s.send(b"Hello world!")
print ("Request Sent!")
except:
s.close()
except KeyboardInterrupt:
# here i'd like to kill all threads if possible
event.set() # set the event as soon as any thread receives an keyboardInterrupt.
try:
for x in range(800):
send(daemon=True).start()
event.wait()
except KeyboardInterrupt: # If user pressed ctrl+c while loop was still running, then this will be useful
pass
sys.exit(0)
EDIT 9/15/16: In my original code (still posted below) I tried to use .join() with a function, which is a silly mistake because it can only be used with a thread object. I am trying to
(1) continuously run a thread that gets data and saves it to a file
(2) have a second thread, or incorporate queue, that will stop the program once a user enters a flag (i.e. "stop"). It doesn't interrupt the data gathering/saving thread.
I need help with multithreading. I am trying to run two threads, one that handles data and the second checks for a flag to stop the program.
I learned by trial and error that I can't interrupt a while loop without my computer exploding. Additionally, I have abandoned my GUI code because it made my code too complicated with the mulithreading.
What I want to do is run a thread that gathers data from an Arduino, saves it to a file, and repeats this. The second thread will scan for a flag -- which can be a raw_input? I can't think of anything else that a user can do to stop the data acquisition program.
I greatly appreciate any help on this. Here is my code (much of it is pseudocode, as you can see):
#threading
import thread
import time
global flag
def monitorData():
print "running!"
time.sleep(5)
def stopdata(flag ):
flag = raw_input("enter stop: ")
if flag == "stop":
monitorData.join()
flag = "start"
thread.start_new_thread( monitorData,())
thread.start_new_thread( stopdata,(flag,))
The error I am getting is this when I try entering "stop" in the IDLE.
Unhandled exception in thread started by
Traceback (most recent call last):
File "c:\users\otangu~1\appdata\local\temp\IDLE_rtmp_h_frd5", line 16, in stopdata
AttributeError: 'function' object has no attribute 'join'
Once again I really appreciate any help, I have taught myself Python so far and this is the first huge wall that I've hit.
The error you see is a result of calling join on the function. You need to call join on the thread object. You don't capture a reference to the thread so you have no way to call join anyway. You should join like so.
th1 = thread.start_new_thread( monitorData,())
# later
th1.join()
As for a solution, you can use a Queue to communicate between threads. The queue is used to send a quit message to the worker thread and if the worker does not pick anything up off the queue for a second it runs the code that gathers data from the arduino.
from threading import Thread
from Queue import Queue, Empty
def worker(q):
while True:
try:
item = q.get(block=True, timeout=1)
q.task_done()
if item == "quit":
print("got quit msg in thread")
break
except Empty:
print("empty, do some arduino stuff")
def input_process(q):
while True:
x = raw_input("")
if x == 'q':
print("will quit")
q.put("quit")
break
q = Queue()
t = Thread(target=worker, args=(q,))
t.start()
t2 = Thread(target=input_process, args=(q,))
t2.start()
# waits for the `task_done` function to be called
q.join()
t2.join()
t.join()
It's possibly a bit more code than you hoped for and having to detect the queue is empty with an exception is a little ugly, but this doesn't rely on any global variables and will always exit promptly. That wont be the case with sleep based solutions, which need to wait for any current calls to sleep to finish before resuming execution.
As noted by someone else, you should really be using threading rather than the older thread module and also I would recommend you learn with python 3 and not python 2.
You're looking for something like this:
from threading import Thread
from time import sleep
# "volatile" global shared by threads
active = True
def get_data():
while active:
print "working!"
sleep(3)
def wait_on_user():
global active
raw_input("press enter to stop")
active = False
th1 = Thread(target=get_data)
th1.start()
th2 = Thread(target=wait_on_user)
th2.start()
th1.join()
th2.join()
You made a few obvious and a few less obvious mistakes in your code. First, join is called on a thread object, not a function. Similarly, join doesn't kill a thread, it waits for the thread to finish. A thread finishes when it has no more code to execute. If you want a thread to run until some flag is set, you normally include a loop in your thread that checks the flag every second or so (depending on how precise you need the timing to be).
Also, the threading module is preferred over the lower lever thread module. The latter has been removed in python3.
This is not possible. The thread function has to finish. You can't join it from the outside.
I have a Python GUI that I use to test various aspects of my work. Currently I have a "stop" button which kills the process at the end of each test (there can be multiple tests set up to run at once). However, some tests take a long time to run and if I need to stop the test I would like it to stop instantly. My thoughts are to use
import pdb; pdb.set_trace()
exit
But I'm not sure how I would inject this into the next run line of code. Is this possible?
If it's a thread, you can use the lower-level thread (or _thread in Python 3) module to kill the thread with an exception by calling thread.exit().
From the documentation:
thread.exit(): Raise the SystemExit exception. When not caught,
this will cause the thread to exit silently.
A cleaner method (depending on how your processing is set up) would be to signal the thread to stop processing and exit using an instance variable, then calling the join() method from your main thread to wait until the thread exits.
Example:
class MyThread(threading.Thread):
def __init__(self):
super(MyThread, self).__init__()
self._stop_req = False
def run(self):
while not self._stop_req:
pass
# processing
# clean up before exiting
def stop(self):
# triggers the threading event
self._stop_req = True;
def main():
# set up the processing thread
processing_thread = MyThread()
processing_thread.start()
# do other things
# stop the thread and wait for it to exit
processing_thread.stop()
processing_thread.join()
if __name__ == "__main__":
main()
I'm trying to create a method which runs on a timer in the background of my main script:
def hello_world(self):
print 'Hello!'
threading.Timer(2,hello_world).start()
if __name__ == "__main__":
try:
hello_world()
except KeyboardInterrupt:
print '\nGoodbye!'
I am getting this message when I attempt to keyboard interrupt my script:
Exception KeyboardInterrupt in <module 'threading' from '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py'> ignored
How do I close the thread so that I can exit my application cleanly?
To elaborate a bit on Aphex's answer, the main thread can't possibly catch the KeyboardInterrupt signal, unless you have very fast fingers. The main thread exits almost immediately! Try this:
import threading
def hello_world():
print 'Hello!'
threading.Timer(2,hello_world).start()
if __name__ == "__main__":
try:
hello_world()
except KeyboardInterrupt:
print '\nGoodbye!'
print "main thread exited"
More generally, I would not suggest using a self-calling timer like this, just because it creates a lot of threads. Just create one thread and call time.sleep inside it.
However, as long as you keep the main thread running, you seem to be able to catch KeyboardInterrupt inside. The trick then is to make the thread a daemon thread that exits when the main thread exits.
import threading
import time
def hello_world():
while(True):
print 'Hello!'
time.sleep(2)
if __name__ == "__main__":
hw_thread = threading.Thread(target = hello_world)
hw_thread.daemon = True
hw_thread.start()
try:
time.sleep(1000)
except KeyboardInterrupt:
print '\nGoodbye!'
This exits automatically after 1000 seconds -- you could make that number even bigger if you like. You could also use a busy-loop to repeat the sleep call, but I don't really see the point.
You just need to set the Timer thread as a daemon
def hello_world(self):
print 'Hello!'
t = threading.Timer(2,hello_world)
t.daemon = True
t.start()
That will cause it to exit when the main thread exits, e.g. due to KeyboardInterrupt.
The daemon setting causes the entire program to exit when the only threads left are daemon threads.
Try re-raising the KeyboardInterrupt exception: http://effbot.org/zone/stupid-exceptions-keyboardinterrupt.htm
This still may not work though; chances are you're running into this caveat:
Threads interact strangely with interrupts: the KeyboardInterrupt exception will be received by an arbitrary thread. (When the signal module is available, interrupts always go to the main thread.)
In short, you can't be sure that the KeyboardInterrupt is going to your main thread. To workaround this, you may want to look into the signal module.
Edit: A more elegant way to cancel the thread is to have a shared variable that the thread looks at, and exits if it becomes false. Then if you want to kill the thread from your main thread, you set the variable to false.
I've seen a lot of questions related to this... but my code works on python 2.6.2 and fails to work on python 2.6.5. Am I wrong in thinking that the whole atexit "functions registered via this module are not called when the program is killed by a signal" thing shouldn't count here because I'm catching the signal and then exiting cleanly? What's going on here? Whats the proper way to do this?
import atexit, sys, signal, time, threading
terminate = False
threads = []
def test_loop():
while True:
if terminate:
print('stopping thread')
break
else:
print('looping')
time.sleep(1)
#atexit.register
def shutdown():
global terminate
print('shutdown detected')
terminate = True
for thread in threads:
thread.join()
def close_handler(signum, frame):
print('caught signal')
sys.exit(0)
def run():
global threads
thread = threading.Thread(target=test_loop)
thread.start()
threads.append(thread)
while True:
time.sleep(2)
print('main')
signal.signal(signal.SIGINT, close_handler)
if __name__ == "__main__":
run()
python 2.6.2:
$ python halp.py
looping
looping
looping
main
looping
main
looping
looping
looping
main
looping
^Ccaught signal
shutdown detected
stopping thread
python 2.6.5:
$ python halp.py
looping
looping
looping
main
looping
looping
main
looping
looping
main
^Ccaught signal
looping
looping
looping
looping
...
looping
looping
Killed <- kill -9 process at this point
The main thread on 2.6.5 appears to never execute the atexit functions.
The root difference here is actually unrelated to both signals and atexit, but rather a change in the behavior of sys.exit.
Before around 2.6.5, sys.exit (more accurately, SystemExit being caught at the top level) would cause the interpreter to exit; if threads were still running, they'd be terminated, just as with POSIX threads.
Around 2.6.5, the behavior changed: the effect of sys.exit is now essentially the same as returning from the main function of the program. When you do that--in both versions--the interpreter waits for all threads to be joined before exiting.
The relevant change is that Py_Finalize now calls wait_for_thread_shutdown() near the top, where it didn't before.
This behavioral change seems incorrect, primarily because it no longer functions as documented, which is simply: "Exit from Python." The practical effect is no longer to exit from Python, but simply to exit the thread. (As a side note, sys.exit has never exited Python when called from another thread, but that obscure divergance from documented behavior doesn't justify a much bigger one.)
I can see the appeal of the new behavior: rather than two ways to exit the main thread ("exit and wait for threads" and "exit immediately"), there's only one, as sys.exit is essentially identical to simply returning from the top function. However, it's a breaking change and diverges from documented behavior, which far outweighs that.
Because of this change, after sys.exit from the signal handler above, the interpreter sits around waiting for threads to exit and then runs atexit handlers after they do. Since it's the handler itself that tells the threads to exit, the result is a deadlock.
Exiting due to a signal is not the same as exiting from within a signal handler. Catching a signal and exiting with sys.exit is a clean exit, not an exit due to a signal handler. So, yes, I agree that it should run atexit handlers here--at least in principle.
However, there's something tricky about signal handlers: they're completely asynchronous. They can interrupt the program flow at any time, between any VM opcode. Take this code, for example. (Treat this as the same form as your code above; I've omitted code for brevity.)
import threading
lock = threading.Lock()
def test_loop():
while not terminate:
print('looping')
with lock:
print "Executing synchronized operation"
time.sleep(1)
print('stopping thread')
def run():
while True:
time.sleep(2)
with lock:
print "Executing another synchronized operation"
print('main')
There's a serious problem here: a signal (eg. ^C) may be received while run() is holding lock. If that happens, your signal handler will be run with the lock still held. It'll then wait for test_loop to exit, and if that thread is waiting for the lock, you'll deadlock.
This is a whole category of problems, and it's why a lot of APIs say not to call them from within signal handlers. Instead, you should set a flag to tell the main thread to shut down at an appropriate time.
do_shutdown = False
def close_handler(signum, frame):
global do_shutdown
do_shutdown = True
print('caught signal')
def run():
while not do_shutdown:
...
My preference is to avoid exiting the program with sys.exit entirely and to explicitly do cleanup at the main exit point (eg. the end of run()), but you can use atexit here if you want.
I'm not sure if this was entirely changed, but this is how I have my atexit done in 2.6.5
atexit.register(goodbye)
def goodbye():
print "\nStopping..."