We Keep Coding

Stopping eval code dinamically on event fired [duplicate]

What's the proper way to tell a looping thread to stop looping?
I have a fairly simple program that pings a specified host in a separate threading.Thread class. In this class it sleeps 60 seconds, the runs again until the application quits.
I'd like to implement a 'Stop' button in my wx.Frame to ask the looping thread to stop. It doesn't need to end the thread right away, it can just stop looping once it wakes up.
Here is my threading class (note: I haven't implemented looping yet, but it would likely fall under the run method in PingAssets)
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
def run(self):
config = controller.getConfig()
fmt = config['timefmt']
start_time = datetime.now().strftime(fmt)
try:
if onlinecheck.check_status(self.asset):
status = "online"
else:
status = "offline"
except socket.gaierror:
status = "an invalid asset tag."
msg =("{}: {} is {}. \n".format(start_time, self.asset, status))
wx.CallAfter(self.window.Logger, msg)
And in my wxPyhton Frame I have this function called from a Start button:
def CheckAsset(self, asset):
self.count += 1
thread = PingAssets(self.count, asset, self)
self.threads.append(thread)
thread.start()

Threaded stoppable function
Instead of subclassing threading.Thread, one can modify the function to allow
stopping by a flag.
We need an object, accessible to running function, to which we set the flag to stop running.
We can use threading.currentThread() object.
import threading
import time
def doit(arg):
t = threading.currentThread()
while getattr(t, "do_run", True):
print ("working on %s" % arg)
time.sleep(1)
print("Stopping as you wish.")
def main():
t = threading.Thread(target=doit, args=("task",))
t.start()
time.sleep(5)
t.do_run = False
if __name__ == "__main__":
main()
The trick is, that the running thread can have attached additional properties. The solution builds
on assumptions:
the thread has a property "do_run" with default value True
driving parent process can assign to started thread the property "do_run" to False.
Running the code, we get following output:
$ python stopthread.py
working on task
working on task
working on task
working on task
working on task
Stopping as you wish.
Pill to kill - using Event
Other alternative is to use threading.Event as function argument. It is by
default False, but external process can "set it" (to True) and function can
learn about it using wait(timeout) function.
We can wait with zero timeout, but we can also use it as the sleeping timer (used below).
def doit(stop_event, arg):
while not stop_event.wait(1):
print ("working on %s" % arg)
print("Stopping as you wish.")
def main():
pill2kill = threading.Event()
t = threading.Thread(target=doit, args=(pill2kill, "task"))
t.start()
time.sleep(5)
pill2kill.set()
t.join()
Edit: I tried this in Python 3.6. stop_event.wait() blocks the event (and so the while loop) until release. It does not return a boolean value. Using stop_event.is_set() works instead.
Stopping multiple threads with one pill
Advantage of pill to kill is better seen, if we have to stop multiple threads
at once, as one pill will work for all.
The doit will not change at all, only the main handles the threads a bit differently.
def main():
pill2kill = threading.Event()
tasks = ["task ONE", "task TWO", "task THREE"]
def thread_gen(pill2kill, tasks):
for task in tasks:
t = threading.Thread(target=doit, args=(pill2kill, task))
yield t
threads = list(thread_gen(pill2kill, tasks))
for thread in threads:
thread.start()
time.sleep(5)
pill2kill.set()
for thread in threads:
thread.join()

This has been asked before on Stack. See the following links:
Is there any way to kill a Thread in Python?
Stopping a thread after a certain amount of time
Basically you just need to set up the thread with a stop function that sets a sentinel value that the thread will check. In your case, you'll have the something in your loop check the sentinel value to see if it's changed and if it has, the loop can break and the thread can die.

I read the other questions on Stack but I was still a little confused on communicating across classes. Here is how I approached it:
I use a list to hold all my threads in the __init__ method of my wxFrame class: self.threads = []
As recommended in How to stop a looping thread in Python? I use a signal in my thread class which is set to True when initializing the threading class.
class PingAssets(threading.Thread):
def __init__(self, threadNum, asset, window):
threading.Thread.__init__(self)
self.threadNum = threadNum
self.window = window
self.asset = asset
self.signal = True
def run(self):
while self.signal:
do_stuff()
sleep()
and I can stop these threads by iterating over my threads:
def OnStop(self, e):
for t in self.threads:
t.signal = False

I had a different approach. I've sub-classed a Thread class and in the constructor I've created an Event object. Then I've written custom join() method, which first sets this event and then calls a parent's version of itself.
Here is my class, I'm using for serial port communication in wxPython app:
import wx, threading, serial, Events, Queue
class PumpThread(threading.Thread):
def __init__ (self, port, queue, parent):
super(PumpThread, self).__init__()
self.port = port
self.queue = queue
self.parent = parent
self.serial = serial.Serial()
self.serial.port = self.port
self.serial.timeout = 0.5
self.serial.baudrate = 9600
self.serial.parity = 'N'
self.stopRequest = threading.Event()
def run (self):
try:
self.serial.open()
except Exception, ex:
print ("[ERROR]\tUnable to open port {}".format(self.port))
print ("[ERROR]\t{}\n\n{}".format(ex.message, ex.traceback))
self.stopRequest.set()
else:
print ("[INFO]\tListening port {}".format(self.port))
self.serial.write("FLOW?\r")
while not self.stopRequest.isSet():
msg = ''
if not self.queue.empty():
try:
command = self.queue.get()
self.serial.write(command)
except Queue.Empty:
continue
while self.serial.inWaiting():
char = self.serial.read(1)
if '\r' in char and len(msg) > 1:
char = ''
#~ print('[DATA]\t{}'.format(msg))
event = Events.PumpDataEvent(Events.SERIALRX, wx.ID_ANY, msg)
wx.PostEvent(self.parent, event)
msg = ''
break
msg += char
self.serial.close()
def join (self, timeout=None):
self.stopRequest.set()
super(PumpThread, self).join(timeout)
def SetPort (self, serial):
self.serial = serial
def Write (self, msg):
if self.serial.is_open:
self.queue.put(msg)
else:
print("[ERROR]\tPort {} is not open!".format(self.port))
def Stop(self):
if self.isAlive():
self.join()
The Queue is used for sending messages to the port and main loop takes responses back. I've used no serial.readline() method, because of different end-line char, and I have found the usage of io classes to be too much fuss.

Depends on what you run in that thread.
If that's your code, then you can implement a stop condition (see other answers).
However, if what you want is to run someone else's code, then you should fork and start a process. Like this:
import multiprocessing
proc = multiprocessing.Process(target=your_proc_function, args=())
proc.start()
now, whenever you want to stop that process, send it a SIGTERM like this:
proc.terminate()
proc.join()
And it's not slow: fractions of a second.
Enjoy :)

My solution is:
import threading, time
def a():
t = threading.currentThread()
while getattr(t, "do_run", True):
print('Do something')
time.sleep(1)
def getThreadByName(name):
threads = threading.enumerate() #Threads list
for thread in threads:
if thread.name == name:
return thread
threading.Thread(target=a, name='228').start() #Init thread
t = getThreadByName('228') #Get thread by name
time.sleep(5)
t.do_run = False #Signal to stop thread
t.join()

I find it useful to have a class, derived from threading.Thread, to encapsulate my thread functionality. You simply provide your own main loop in an overridden version of run() in this class. Calling start() arranges for the object’s run() method to be invoked in a separate thread.
Inside the main loop, periodically check whether a threading.Event has been set. Such an event is thread-safe.
Inside this class, you have your own join() method that sets the stop event object before calling the join() method of the base class. It can optionally take a time value to pass to the base class's join() method to ensure your thread is terminated in a short amount of time.
import threading
import time
class MyThread(threading.Thread):
def __init__(self, sleep_time=0.1):
self._stop_event = threading.Event()
self._sleep_time = sleep_time
"""call base class constructor"""
super().__init__()
def run(self):
"""main control loop"""
while not self._stop_event.isSet():
#do work
print("hi")
self._stop_event.wait(self._sleep_time)
def join(self, timeout=None):
"""set stop event and join within a given time period"""
self._stop_event.set()
super().join(timeout)
if __name__ == "__main__":
t = MyThread()
t.start()
time.sleep(5)
t.join(1) #wait 1s max
Having a small sleep inside the main loop before checking the threading.Event is less CPU intensive than looping continuously. You can have a default sleep time (e.g. 0.1s), but you can also pass the value in the constructor.

Sometimes you don't have control over the running target. In those cases you can use signal.pthread_kill to send a stop signal.
from signal import pthread_kill, SIGTSTP
from threading import Thread
from itertools import count
from time import sleep
def target():
for num in count():
print(num)
sleep(1)
thread = Thread(target=target)
thread.start()
sleep(5)
pthread_kill(thread.ident, SIGTSTP)
result
0
1
2
3
4
[14]+ Stopped

End a Process early in Python 3.6+

I've read that it's considered bad practice to kill a thread. (Is there any way to kill a Thread?) There are a LOT of answers there, and I'm wondering if even using a thread in the first place is the right answer for me.
I have a bunch multiprocessing.Processes. Essentially, each Process is doing this:
while some_condition:
result = self.function_to_execute(i, **kwargs_i)
# outQ is a multiprocessing.queue shared between all Processes
self.outQ.put(Result(i, result))
Problem is... I need a way to interrupt function_to_execute, but can't modify the function itself. Initially, I was thinking simply process.terminate(), but that appears to be unsafe with multiprocessing.queue.
Most likely (but not guaranteed), if I need to kill a thread, the 'main' program is going to be done soon. Is my safest option to do something like this? Or perhaps there is a more elegant solution than using a thread in the first place?
def thread_task():
while some_condition:
result = self.function_to_execute(i, **kwargs_i)
if (this_thread_is_not_daemonized):
self.outQ.put(Result(i, result))
t = Thread(target=thread_task)
t.start()
if end_early:
t.daemon = True
I believe the end result of this is that the Process that spawned the thread will continue to waste CPU cycles on a task I no longer care about the output for, but if the main program finishes, it'll clean up all my memory nicely.
The main problem with daemonizing a thread is that the main program could potentially continue for 30+ minutes even when I don't care about the output of that thread anymore.

From the threading docs:
If you want your threads to stop gracefully, make them non-daemonic
and use a suitable signalling mechanism such as an Event
Here is a contrived example of what I was thinking - no idea if it mimics what you are doing or can be adapted for your situation. Another caveat: I've never written any real concurrent code.
Create an Event object in the main process and pass it all the way to the thread.
Design the thread so that it loops until the Event object is set. Once you don't need the processing anymore SET the Event object in the main process. No need to modify the function being run in the thread.
from multiprocessing import Process, Queue, Event
from threading import Thread
import time, random, os
def f_to_run():
time.sleep(.2)
return random.randint(1,10)
class T(Thread):
def __init__(self, evt,q, func, parent):
self.evt = evt
self.q = q
self.func = func
self.parent = parent
super().__init__()
def run(self):
while not self.evt.is_set():
n = self.func()
self.q.put(f'PID {self.parent}-{self.name}: {n}')
def f(T,evt,q,func):
pid = os.getpid()
t = T(evt,q,func,pid)
t.start()
t.join()
q.put(f'PID {pid}-{t.name} is alive - {t.is_alive()}')
q.put(f'PID {pid}:DONE')
return 'foo done'
if __name__ == '__main__':
results = []
q = Queue()
evt = Event()
# two processes each with one thread
p= Process(target=f, args=(T, evt, q, f_to_run))
p1 = Process(target=f, args=(T, evt, q, f_to_run))
p.start()
p1.start()
while len(results) < 40:
results.append(q.get())
print('.',end='')
print('')
evt.set()
p.join()
p1.join()
while not q.empty():
results.append(q.get_nowait())
for thing in results:
print(thing)
I initially tried to use threading.Event but the multiprocessing module complained that it couldn't be pickled. I was actually surprised that the multiprocessing.Queue and multiprocessing.Event worked AND could be accessed by the thread.
Not sure why I started with a Thread subclass - I think I thought it would be easier to control/specify what happens in it's run method. But it can be done with a function also.
from multiprocessing import Process, Queue, Event
from threading import Thread
import time, random
def f_to_run():
time.sleep(.2)
return random.randint(1,10)
def t1(evt,q, func):
while not evt.is_set():
n = func()
q.put(n)
def g(t1,evt,q,func):
t = Thread(target=t1,args=(evt,q,func))
t.start()
t.join()
q.put(f'{t.name} is alive - {t.is_alive()}')
return 'foo'
if __name__ == '__main__':
q = Queue()
evt = Event()
p= Process(target=g, args=(t1, evt, q, f_to_run))
p.start()
time.sleep(5)
evt.set()
p.join()

python picamera, keyboard ctrl+c/sigint not caught

From the pycamera docs I took the example for fast capture and processing and added a sigint event handler to catch the keyboard interrupt:
import io
import time
import threading
import picamera
# Create a pool of image processors
done = False
lock = threading.Lock()
pool = []
def signal_handler(signal, frame):
global done
print 'You pressed Ctrl+C!'
done=True
sys.exit()
signal.signal(signal.SIGINT, signal_handler)
class ImageProcessor(threading.Thread):
def __init__(self):
super(ImageProcessor, self).__init__()
self.stream = io.BytesIO()
self.event = threading.Event()
self.terminated = False
self.daemon=True;
self.start()
def run(self):
# This method runs in a separate thread
global done
while not self.terminated:
# Wait for an image to be written to the stream
if self.event.wait(1):
try:
self.stream.seek(0)
# Read the image and do some processing on it
#Image.open(self.stream)
#...
#...
# Set done to True if you want the script to terminate
# at some point
#done=True
finally:
# Reset the stream and event
self.stream.seek(0)
self.stream.truncate()
self.event.clear()
# Return ourselves to the pool
with lock:
pool.append(self)
def streams():
while not done:
with lock:
if pool:
processor = pool.pop()
else:
processor = None
if processor:
yield processor.stream
processor.event.set()
else:
# When the pool is starved, wait a while for it to refill
time.sleep(0.1)
with picamera.PiCamera() as camera:
pool = [ImageProcessor() for i in range(4)]
camera.resolution = (640, 480)
camera.framerate = 30
camera.start_preview()
time.sleep(2)
camera.capture_sequence(streams(), use_video_port=True)
# Shut down the processors in an orderly fashion
while pool:
with lock:
processor = pool.pop()
processor.terminated = True
processor.join()
but the interrupt signal is never caught.
Until the camera.capture_sequence(streams(), use_video_port=True) runs the signal is caught, after capture_sequence is started the signal handler is not called.
I'm new to python so maybe the answer is simple. What am i doing wrong in here?
EDIT:
If i remove the following code the signal is caught:
yield processor.stream

The problem there is that you are using thread.join(), it block the main thread,which means your program have to wait until that thread you joined finishes to continue.
The signals will always be caught by the main process, because it's the one that receives the signals, it's the process that has threads.
There are plenty of answer about how to deal with main thread and CTRL+C,and i give you three options,
First,add timeout to join() call:
thread1.join(60) detail here
Second, start a new process to deal with signal to kill the program.
class Watcher():
def __init__(self):
self.child = os.fork()
if self.child == 0:
return
else:
self.watch()
def watch(self):
try:
os.wait()
except KeyboardInterrupt:
self.kill()
sys.exit()
def kill(self):
try:
os.kill(self.child, signal.SIGKILL)
except OSError:
pass
start a Watcher before you start work thread,like
def main():
init()
Watcher()
start_your_thread1()
start_your_thread2()
start_your_thread3()
The final,your original way,the complicate Producer and Consumer way.
just delete the final join(),and add some task for the main thread.
i prefer the second option,it's easy use,and solves two problems with multithreaded programs in Python, (1) a signal might be delivered to any thread (which is just a malfeature) and (2) if the thread that gets the signal is waiting, the signal is ignored (which is a bug).
More detail about the Watcher is in Appendix A of the book The Little Book of Semaphores

In your code, the done variable is a global variable.
So, whenever you want to modify it inside a function, you need to use the keyword global, or else it become a local variable.
You should fix your code like this:
import signal
import sys
done = False
def signal_handler(signal, frame):
global done
print('You pressed Ctrl+C!')
done = True
sys.exit()
signal.signal(signal.SIGINT, signal_handler)

Python threading: will Event.set() really notify every waiting thread

If I have a threading.Event and the following two lines of code:
event.set()
event.clear()
and I have some threads who are waiting for that event.
My question is related to what happens when calling the set() method:
Can I be ABSOLUTELY sure that all the waiting thread(s) will be notified? (i.e. Event.set() "notifies" the threads)
Or could it happen that those two lines are executed so quickly after each other, that some threads might still be waiting? (i.e. Event.wait() polls the event's state, which might be already "cleared" again)
Thanks for your answers!

In the internals of Python, an event is implemented with a Condition() object.
When calling the event.set() method, the notify_all() of the condition is called (after getting the lock to be sure to be not interrupted), then all the threads receive the notification (the lock is released only when all the threads are notified), so you can be sure that all the threads will effectively be notified.
Now, clearing the event just after the notification is not a problem.... until you do not want to check the event value in the waiting threads with an event.is_set(), but you only need this kind of check if you were waiting with a timeout.
Examples :
pseudocode that works :
#in main thread
event = Event()
thread1(event)
thread2(event)
...
event.set()
event.clear()
#in thread code
...
event.wait()
#do the stuff
pseudocode that may not work :
#in main thread
event = Event()
thread1(event)
thread2(event)
...
event.set()
event.clear()
#in thread code
...
while not event.is_set():
event.wait(timeout_value)
#do the stuff
Edited : in python >= 2.7 you can still wait for an event with a timeout and be sure of the state of the event :
event_state = event.wait(timeout)
while not event_state:
event_state = event.wait(timeout)

It's easy enough to verify that things work as expected (Note: this is Python 2 code, which will need adapting for Python 3):
import threading
e = threading.Event()
threads = []
def runner():
tname = threading.current_thread().name
print 'Thread waiting for event: %s' % tname
e.wait()
print 'Thread got event: %s' % tname
for t in range(100):
t = threading.Thread(target=runner)
threads.append(t)
t.start()
raw_input('Press enter to set and clear the event:')
e.set()
e.clear()
for t in threads:
t.join()
print 'All done.'
If you run the above script and it terminates, all should be well :-) Notice that a hundred threads are waiting for the event to be set; it's set and cleared straight away; all threads should see this and should terminate (though not in any definite order, and the "All done" can be printed anywhere after the "Press enter" prompt, not just at the very end.

Python 3+
It's easier to check that it works
import threading
import time
lock = threading.Lock() # just to sync printing
e = threading.Event()
threads = []
def runner():
tname = threading.current_thread().name
with lock:
print('Thread waiting for event ', tname)
e.wait()
with lock:
print('Thread got event: ', tname)
for t in range(8): # Create 8 threads could be 100's
t = threading.Thread(target=runner)
threads.append(t)
t.start()
time.sleep(1) # force wait until set/clear
e.set()
e.clear()
for t in threads:
t.join()
print('Done')

ideal thread structure question (involves multiple thread communication)

I'm writing an application that listens for sound events (using messages passed in with Open Sound Control), and then based on those events pauses or resumes program execution. My structure works most of the time but always bombs out in the main loop, so I'm guessing it's a thread issue. Here's a generic, simplified version of what I'm talking about:
import time, threading
class Loop():
aborted = False
def __init__(self):
message = threading.Thread(target=self.message, args=((0),))
message.start()
loop = threading.Thread(target=self.loop)
loop.start()
def message(self,val):
if val > 1:
if not self.aborted:
self.aborted = True
# do some socket communication
else:
self.aborted = False
# do some socket communication
def loop(self):
cnt = 0
while True:
print cnt
if self.aborted:
while self.aborted:
print "waiting"
time.sleep(.1);
cnt += 1
class FakeListener():
def __init__(self,loop):
self.loop = loop
listener = threading.Thread(target=self.listener)
listener.start()
def listener(self):
while True:
loop.message(2)
time.sleep(1)
if __name__ == '__main__':
loop = Loop()
#fake listener standing in for the real OSC event listener
listener = FakeListener(loop)
Of course, this simple code seems to work great, so it's clearly not fully illustrating my real code, but you get the idea. What isn't included here is also the fact that on each loop pause and resume (by setting aborted=True/False) results in some socket communication which also involves threads.
What always happens in my code is that the main loop doesn't always pickup where it left off after a sound event. It will work for a number of events but then eventually it just doesn't answer.
Any suggestions for how to structure this kind of communication amongst threads?
UPDATE:
ok, i think i've got it. here's a modification that seems to work. there's a listener thread that periodically puts a value into a Queue object. there's a checker thread that keeps checking the queue looking for the value, and once it sees it sets a boolean to its opposite state. that boolean value controls whether the loop thread continues or waits.
i'm not entirely sure what the q.task_done() function is doing here, though.
import time, threading
import Queue
q = Queue.Queue(maxsize = 0)
class Loop():
aborted = False
def __init__(self):
checker = threading.Thread(target=self.checker)
checker.setDaemon(True)
checker.start()
loop = threading.Thread(target=self.loop)
loop.start()
def checker(self):
while True:
if q.get() == 2:
q.task_done()
if not self.aborted:
self.aborted = True
else:
self.aborted = False
def loop(self):
cnt = 0
while cnt < 40:
if self.aborted:
while self.aborted:
print "waiting"
time.sleep(.1)
print cnt
cnt += 1
time.sleep(.1)
class fakeListener():
def __init__(self):
listener = threading.Thread(target=self.listener)
listener.setDaemon(True)
listener.start()
def listener(self):
while True:
q.put(2)
time.sleep(1)
if __name__ == '__main__':
#fake listener standing in for the real OSC event listener
listener = fakeListener()
loop = Loop()

Umm.. I don't completely understand your question but i'll do my best to explain what I think you need to fix your problems.
1) The thread of your Loop.loop function should be set as a daemon thread so that it exits with your main thread (so you don't have to kill the python process every time you want to shut down your program). To do this just put loop.setDaemon(True) before you call the thread's "start" function.
2)The most simple and fail-proof way to communicate between threads is with a Queue. On thread will put an item in that Queue and another thread will take an item out, do something with the item and then terminate (or get another job)
In python a Queue can be anything from a global list to python's built-in Queue object. I recommend the python Queue because it is thread safe and easy to use.