import _thread
import time
def test1():
while True:
time.sleep(1)
print('TEST1')
def test2():
while True:
time.sleep(3)
print('TEST2')
try:
_thread.start_new_thread(test1,())
_thread.start_new_thread(test2,())
except:
print("ERROR")
How can I stop the two threads for example in case of KeyboardInterrupts?
Because for "except KeyboardInterrupt" the threads are still running :/
Important:
The question is about closing threads only with the module _thread!
Is it possible?
There's no way to directly interact with another thread, except for the main thread. While some platforms do offer thread cancel or kill semantics, Python doesn't expose them, and for good reason.1
So, the usual solution is to use some kind of signal to tell everyone to exit. One possibility is a done flag with a Lock around it:
done = False
donelock = _thread.allocate_lock()
def test1():
while True:
try:
donelock.acquire()
if done:
return
finally:
donelock.release()
time.sleep(1)
print('TEST1')
_thread.start_new_thread(test1,())
time.sleep(3)
try:
donelock.acquire()
done = True
finally:
donelock.release()
Of course the same thing is a lot cleaner if you use threading (or a different higher-level API like Qt's threads). Plus, you can use a Condition or Event to make the background threads exit as soon as possible, instead of only after their next sleep finishes.
done = threading.Event()
def test1():
while True:
if done.wait(1):
return
print('TEST1')
t1 = threading.Thread(target=test1)
t1.start()
time.sleep(3)
done.set()
The _thread module doesn't have an Event or Condition, of course, but you can always build one yourself—or just borrowing from the threading source.
Or, if you wanted the threads to be killed asynchronously (which obviously isn't safe if they're, e.g., writing files, but if they're just doing computation or downloads or the like that you don't care about if you're canceling, that's fine), threading makes it even easier:
t1 = threading.Thread(target=test1, daemon=True)
As a side note, the behavior you're seeing isn't actually reliable across platforms:
Background threads created with _thread may keep running, or shut down semi-cleanly, or terminate hard. So, when you use _thread in a portable application, you have to write code that can handle any of the three.
KeyboardInterrupt may be delivered to an arbitrary thread rather than the main thread. If it is, it will usually kill that thread, unless you've set up a handler. So, if you're using _thread, you usually want to handle KeyboardInterrupt and call _thread.interrupt_main().
Also, I don't think your except: is doing what you think it is. That try only covers the start_new_thread calls. If the threads start successfully, the main thread exits the try block and reaches the end of the program. If a KeyboardInterrupt or other exception is raised, the except: isn't going to be triggered. (Also, using a bare except: and not even logging which exception got handled is a really bad idea if you want to be able to understand what your code is doing.) Presumably, on your platform, background threads continue running, and the main thread blocks on them (and probably at the OS level, not the Python level, so there's no code you can write that gets involved there).
If you want your main thread to keep running to make sure it can handle a KeyboardInterrupt and so something with it (but see the caveats above!), you have to give it code to keep running:
try:
while True:
time.sleep(1<<31)
except KeyboardInterrupt:
# background-thread-killing code goes here.
1. TerminateThread on Windows makes it impossible to do all the cleanup Python needs to do. pthread_cancel on POSIX systems like Linux and macOS makes it possible, but very difficult. And the semantics are different enough between the two that trying to write a cross-platform wrapper would be a nightmare. Not to mention that Python supports systems (mostly older Unixes) that don't have the full pthread API, or even have a completely different threading API.
Related
I looked online and found some SO discussing and ActiveState recipes for running some code with a timeout. It looks there are some common approaches:
Use thread that run the code, and join it with timeout. If timeout elapsed - kill the thread. This is not directly supported in Python (used private _Thread__stop function) so it is bad practice
Use signal.SIGALRM - but this approach not working on Windows!
Use subprocess with timeout - but this is too heavy - what if I want to start interruptible task often, I don't want fire process for each!
So, what is the right way? I'm not asking about workarounds (eg use Twisted and async IO), but actual way to solve actual problem - I have some function and I want to run it only with some timeout. If timeout elapsed, I want control back. And I want it to work on Linux and Windows.
A completely general solution to this really, honestly does not exist. You have to use the right solution for a given domain.
If you want timeouts for code you fully control, you have to write it to cooperate. Such code has to be able to break up into little chunks in some way, as in an event-driven system. You can also do this by threading if you can ensure nothing will hold a lock too long, but handling locks right is actually pretty hard.
If you want timeouts because you're afraid code is out of control (for example, if you're afraid the user will ask your calculator to compute 9**(9**9)), you need to run it in another process. This is the only easy way to sufficiently isolate it. Running it in your event system or even a different thread will not be enough. It is also possible to break things up into little chunks similar to the other solution, but requires very careful handling and usually isn't worth it; in any event, that doesn't allow you to do the same exact thing as just running the Python code.
What you might be looking for is the multiprocessing module. If subprocess is too heavy, then this may not suit your needs either.
import time
import multiprocessing
def do_this_other_thing_that_may_take_too_long(duration):
time.sleep(duration)
return 'done after sleeping {0} seconds.'.format(duration)
pool = multiprocessing.Pool(1)
print 'starting....'
res = pool.apply_async(do_this_other_thing_that_may_take_too_long, [8])
for timeout in range(1, 10):
try:
print '{0}: {1}'.format(duration, res.get(timeout))
except multiprocessing.TimeoutError:
print '{0}: timed out'.format(duration)
print 'end'
If it's network related you could try:
import socket
socket.setdefaulttimeout(number)
I found this with eventlet library:
http://eventlet.net/doc/modules/timeout.html
from eventlet.timeout import Timeout
timeout = Timeout(seconds, exception)
try:
... # execution here is limited by timeout
finally:
timeout.cancel()
For "normal" Python code, that doesn't linger prolongued times in C extensions or I/O waits, you can achieve your goal by setting a trace function with sys.settrace() that aborts the running code when the timeout is reached.
Whether that is sufficient or not depends on how co-operating or malicious the code you run is. If it's well-behaved, a tracing function is sufficient.
An other way is to use faulthandler:
import time
import faulthandler
faulthandler.enable()
try:
faulthandler.dump_tracebacks_later(3)
time.sleep(10)
finally:
faulthandler.cancel_dump_tracebacks_later()
N.B: The faulthandler module is part of stdlib in python3.3.
If you're running code that you expect to die after a set time, then you should write it properly so that there aren't any negative effects on shutdown, no matter if its a thread or a subprocess. A command pattern with undo would be useful here.
So, it really depends on what the thread is doing when you kill it. If its just crunching numbers who cares if you kill it. If its interacting with the filesystem and you kill it , then maybe you should really rethink your strategy.
What is supported in Python when it comes to threads? Daemon threads and joins. Why does python let the main thread exit if you've joined a daemon while its still active? Because its understood that someone using daemon threads will (hopefully) write the code in a way that it wont matter when that thread dies. Giving a timeout to a join and then letting main die, and thus taking any daemon threads with it, is perfectly acceptable in this context.
I've solved that in that way:
For me is worked great (in windows and not heavy at all) I'am hope it was useful for someone)
import threading
import time
class LongFunctionInside(object):
lock_state = threading.Lock()
working = False
def long_function(self, timeout):
self.working = True
timeout_work = threading.Thread(name="thread_name", target=self.work_time, args=(timeout,))
timeout_work.setDaemon(True)
timeout_work.start()
while True: # endless/long work
time.sleep(0.1) # in this rate the CPU is almost not used
if not self.working: # if state is working == true still working
break
self.set_state(True)
def work_time(self, sleep_time): # thread function that just sleeping specified time,
# in wake up it asking if function still working if it does set the secured variable work to false
time.sleep(sleep_time)
if self.working:
self.set_state(False)
def set_state(self, state): # secured state change
while True:
self.lock_state.acquire()
try:
self.working = state
break
finally:
self.lock_state.release()
lw = LongFunctionInside()
lw.long_function(10)
The main idea is to create a thread that will just sleep in parallel to "long work" and in wake up (after timeout) change the secured variable state, the long function checking the secured variable during its work.
I'm pretty new in Python programming, so if that solution has a fundamental errors, like resources, timing, deadlocks problems , please response)).
solving with the 'with' construct and merging solution from -
Timeout function if it takes too long to finish
this thread which work better.
import threading, time
class Exception_TIMEOUT(Exception):
pass
class linwintimeout:
def __init__(self, f, seconds=1.0, error_message='Timeout'):
self.seconds = seconds
self.thread = threading.Thread(target=f)
self.thread.daemon = True
self.error_message = error_message
def handle_timeout(self):
raise Exception_TIMEOUT(self.error_message)
def __enter__(self):
try:
self.thread.start()
self.thread.join(self.seconds)
except Exception, te:
raise te
def __exit__(self, type, value, traceback):
if self.thread.is_alive():
return self.handle_timeout()
def function():
while True:
print "keep printing ...", time.sleep(1)
try:
with linwintimeout(function, seconds=5.0, error_message='exceeded timeout of %s seconds' % 5.0):
pass
except Exception_TIMEOUT, e:
print " attention !! execeeded timeout, giving up ... %s " % e
First of all, I've only started working with Python a month ago, so I don't have deep knowledge about anything.
In a project I'm trying to collect the results of multiple (simultaneous) functions in a database, infinitely until I tell it to stop.
In an earlier attempt, I successfully used multiprocessing to do what I need, but since I now need to collect all the results of those functions for a database within the main, I switched to threading instead.
Basically, what I'm trying to do is:
collect1 = Thread(target=collect_data1)
collect2 = Thread(target=collect_data2)
send1 = Thread(target=send_data1)
send2 = Thread(target=send_data2)
collect = (collect1, collect2)
send = (send1, send2)
while True:
try:
for thread in collect:
thread.start()
for thread in collect:
thread.join()
for thread in send:
thread.start()
for thread in send:
thread.join()
except KeyboardInterrupt:
break
Now, obviously I can't just restart the threads. Nor explicitly kill them. The functions within the threads can theoretically be stopped at any point, so terminate() from multiprocessing was fine.
I was thinking whether something like the following could work (at least PyCharm is fine with it, so it seems to work) or if it creates a memory leak (which I assume), because the threads are never properly closed or deleted, at least as far as I can tell from researching.
Again, I'm new to Python, so I don't know anything about this aspect.
Code:
while True:
try:
collect1 = Thread(target=collect_data1)
collect2 = Thread(target=collect_data2)
send1 = Thread(target=send_data1)
send2 = Thread(target=send_data2)
collect = (collect1, collect2)
send = (send1, send2)
for thread in collect:
thread.start()
for thread in collect:
thread.join()
for thread in send:
thread.start()
for thread in send:
thread.join()
except KeyboardInterrupt:
break
I feel like this approach seems too good to be true, especially since I've never came across a similar solution during my research.
Anyways, any input is appreciated.
Have a good day.
You explicitly await the threads' termination in loops containing thread.join(), so no memory leak takes place, and your code is fine. If you're worried that you don't dispose of the thread objects in any way after they terminate, it will be done automatically as soon as they aren't used anymore, so this souldn't be a concern neither.
I have a Python program that has some threads that do blocking calls.
For example:
#!/usr/bin/python
import threading, tty, sys, os, signal
# super-awesome thread launcher (re-inventing the wheel because I'm
# too lazy to research what they called this)
class Launch(threading.Thread):
def __init__(self, f):
threading.Thread.__init__(self)
self.f = f
self.start()
def run(self):
self.f()
# structure to hold unprocessed chars
Term_Lock = threading.Lock()
Term_Cond = threading.Condition(Term_Lock)
Term_In = []
# launch a thread to retrieve characters from the terminal
tty.setraw(sys.stdin.fileno())
#Launch
def Stdin_Reader():
while True:
c = sys.stdin.read(1)
with Term_Lock:
Term_In.append(c)
Term_Cond.notify()
# main thread
c = None
with Term_Lock:
Term_Cond.wait(1)
if Term_In:
c = Term_In.pop(0)
if c:
print "You pressed '%s'\r" % c
else:
print "You were too slow!\r"
# Lord have mercy on my soul
os.kill(os.getpid(), signal.SIGKILL)
Although this program works just fine, that os.kill() at the end is a little bit unsettling. I have programmed in many other languages, and never seen this sort of issue before. I don't have a problem with the language inventor removing the _Exit call that should happen at the end of the main thread. But then to completely hide _Exit from the system APIs, now that's nerve.
Indeed, what we're seeing is basic questions about how to stop a program in a reasonable way. For example:
Exit a process while threads are sleeping
They say use Python 3.0 daemon threads. I'll keep that in mind for when Python 3.0 finally introduces general 2.7 compatibility. So the next best idea is to stop all the threads:
Is there any way to kill a Thread in Python?
but the best-voted response is basically "don't do that". OK, fine. So take my example above. Blocking call to sys.stdin.read(). How do we fix that? They say use select():
Read file with timeout in Python
Hold on though. Select only works with file descriptors and timeouts. What if I want to receive other inputs from programs and/or libraries that aren't using file descriptors to produce data? So I have to create in-memory pipes or something?? This is getting ridiculous fast.
So, do I just have to keep using os.kill() until Python 3.0 gains acceptance?
Or is there a better way?
I think that os._exit(0) is what I want:
What is difference between sys.exit(0) and os._exit(0)
It seems to work perfectly. I can even put it in my own Exit() function that can do whatever cleanups I want.
I have some code which runs routinely, and every now and then (like once a month) the program seems to hang somewhere and I'm not sure where.
I thought I would implement [what has turned out to be not quite] a "quick fix" of checking how long the program has been running for. I decided to use multithreading to call the function, and then while it is running, check the time.
For example:
import datetime
import threading
def myfunc():
#Code goes here
t=threading.Thread(target=myfunc)
t.start()
d1=datetime.datetime.utcnow()
while threading.active_count()>1:
if (datetime.datetime.utcnow()-d1).total_seconds()>60:
print 'Exiting!'
raise SystemExit(0)
However, this does not close the other thread (myfunc).
What is the best way to go about killing the other thread?
The docs could be clearer about this. Raising SystemExit tells the interpreter to quit, but "normal" exit processing is still done. Part of normal exit processing is .join()-ing all active non-daemon threads. But your rogue thread never ends, so exit processing waits forever to join it.
As #roippi said, you can do
t.daemon = True
before starting it. Normal exit processing does not wait for daemon threads. Your OS should kill them then when the main process exits.
Another alternative:
import os
os._exit(13) # whatever exit code you want goes there
That stops the interpreter "immediately", and skips all normal exit processing.
Pick your poison ;-)
There is no way to kill a thread. You must kill the target from within the target. The best way is with a hook and a queue. It goes something like this.
import Threading
from Queue import Queue
# add a kill_hook arg to your function, kill_hook
# is a queue used to pass messages to the main thread
def myfunc(*args, **kwargs, kill_hook=None):
#Code goes here
# put this somewhere which is periodically checked.
# an ideal place to check the hook is when logging
try:
if q.get_nowait(): # or use q.get(True, 5) to wait a longer
print 'Exiting!'
raise SystemExit(0)
except Queue.empty:
pass
q = Queue() # the queue used to pass the kill call
t=threading.Thread(target=myfunc, args = q)
t.start()
d1=datetime.datetime.utcnow()
while threading.active_count()>1:
if (datetime.datetime.utcnow()-d1).total_seconds()>60:
# if your kill criteria are met, put something in the queue
q.put(1)
I originally found this answer somewhere online, possibly this. Hope this helps!
Another solution would be to use a separate instance of Python, and monitor the other Python thread, killing it from the system level, with psutils.
Wow, I like the daemon and stealth os._exit solutions too!
Is there some method where one could use a try statement to catch an error caused by a raise statement, execute code to handle the flag e.g. update some variables and then return to the line where the code had been operating when the flag was raised?
I am thinking specifically of an interrupt handler for a micro-controller (which does what ive just described).
I am writing some code that has a thread checking a file to see if it updates and I want it to interrupt the main program so it is aware of the update, deals with it appropriately, and returns to the line it was running when interrupted.
Ideally, the main program would recognize the flag from the thread regardless of where it is in execution. A try statement would do this but how could I return to the line where the flag was raised?
Thanks!
Paul
EDIT:
My attempt at ISR after comments albeit it looks like a pretty straight forward example of using locks. Small test routine at the bottom to demonstrate code
import os
import threading
import time
def isr(path, interrupt):
prev_mod = os.stat(path).st_mtime
while(1):
new_mod = os.stat(path).st_mtime
if new_mod != prev_mod:
print "Updates! Waiting to begin"
# Prevent enter into critical code and updating
# While the critical code is running.
with interrupt:
print "Starting updates"
prev_mod = new_mod
print "Fished updating"
else:
print "No updates"
time.sleep(1)
def func2(interrupt):
while(1):
with interrupt: # Prevent updates while running critical code
# Execute critical code
print "Running Crit Code"
time.sleep(5)
print "Finished Crit Code"
# Do other things
interrupt = threading.Lock()
path = "testfil.txt"
t1 = threading.Thread(target = isr, args = (path, interrupt))
t2 = threading.Thread(target = func2, args = (interrupt,))
t1.start()
t2.start()
# Create and "Update" to the file
time.sleep(12)
chngfile = open("testfil.txt","w")
chngfile.write("changing the file")
chngfile.close()
time.sleep(10)
One standard OS way to handle interrupts is to enqueue the interrupt so another kernel thread can process it.
This partially applies in Python.
I am writing some code that has a thread checking a file to see if it updates and I want it to interrupt the main program so it is aware of the update, deals with it appropriately, and returns to the line it was running when interrupted.
You have multiple threads. You don't need to "interrupt" the main program. Simply "deal with it appropriately" in a separate thread. The main thread will find the updates when the other thread has "dealt with it appropriately".
This is why we have locks. To be sure that shared state is updated correctly.
You interrupt a thread by locking a resource the thread needs.
You make a thread interruptable by acquiring locks on resources.
In python we call that pattern "function calls". You cannot do this with exceptions; exceptions only unroll the stack, and always to the first enclosing except clause.
Microcontrollers have interrupts to support asynchronous events; but the same mechanism is also used in software interrupts for system calls, because an interrupt can be configured to have a different set of protection bits; the system call can be allowed to do more than the user program calling it. Python doesn't have any kind of protection levels like this, and so software interrupts are not of much use here.
As for handling asynchronous events, you can do that in python, using the signal module, but you may want to step lightly if you are also using threads.