When you import and use package, this package can run non daemon threads. Until these threads are finished, python cannot exit properly (like with sys.exit(0)). For example imagine that thread t is from some package. When unhandled exception occurs in the main thread, you want to terminate. But this won't exit immediately, it will wait 60s till the thread terminates.
import time, threading
def main():
t = threading.Thread(target=time.sleep, args=(60,))
t.start()
a = 5 / 0
if __name__ == '__main__':
try:
main()
except:
sys.exit(1)
So I came up with 2 things. Replace sys.exit(1) with os._exit(1) or enumerate all threads and make them daemon. Both of them seems to work, but what do you thing is better? os._exit won't flush stdio buffers but setting daemon attribute to threads seems like a hack and maybe it's not guaranteed to work all the time.
import time, threading
def main():
t = thread.Thread(target=time.sleep, args=(60,))
t.start()
a = 5 / 0
if __name__ == '__main__':
try:
main()
except:
for t in threading.enumerate():
if not t.daemon and t.name != "MainThread":
t._daemonic = True
sys.exit(1)
Related
I am testing Python threading with the following script:
import threading
class FirstThread (threading.Thread):
def run (self):
while True:
print 'first'
class SecondThread (threading.Thread):
def run (self):
while True:
print 'second'
FirstThread().start()
SecondThread().start()
This is running in Python 2.7 on Kubuntu 11.10. Ctrl+C will not kill it. I also tried adding a handler for system signals, but that did not help:
import signal
import sys
def signal_handler(signal, frame):
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
To kill the process I am killing it by PID after sending the program to the background with Ctrl+Z, which isn't being ignored. Why is Ctrl+C being ignored so persistently? How can I resolve this?
Ctrl+C terminates the main thread, but because your threads aren't in daemon mode, they keep running, and that keeps the process alive. We can make them daemons:
f = FirstThread()
f.daemon = True
f.start()
s = SecondThread()
s.daemon = True
s.start()
But then there's another problem - once the main thread has started your threads, there's nothing else for it to do. So it exits, and the threads are destroyed instantly. So let's keep the main thread alive:
import time
while True:
time.sleep(1)
Now it will keep print 'first' and 'second' until you hit Ctrl+C.
Edit: as commenters have pointed out, the daemon threads may not get a chance to clean up things like temporary files. If you need that, then catch the KeyboardInterrupt on the main thread and have it co-ordinate cleanup and shutdown. But in many cases, letting daemon threads die suddenly is probably good enough.
KeyboardInterrupt and signals are only seen by the process (ie the main thread)... Have a look at Ctrl-c i.e. KeyboardInterrupt to kill threads in python
I think it's best to call join() on your threads when you expect them to die. I've taken the liberty to make the change your loops to end (you can add whatever cleanup needs are required to there as well). The variable die is checked on each pass and when it's True, the program exits.
import threading
import time
class MyThread (threading.Thread):
die = False
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run (self):
while not self.die:
time.sleep(1)
print (self.name)
def join(self):
self.die = True
super().join()
if __name__ == '__main__':
f = MyThread('first')
f.start()
s = MyThread('second')
s.start()
try:
while True:
time.sleep(2)
except KeyboardInterrupt:
f.join()
s.join()
An improved version of #Thomas K's answer:
Defining an assistant function is_any_thread_alive() according to this gist, which can terminates the main() automatically.
Example codes:
import threading
def job1():
...
def job2():
...
def is_any_thread_alive(threads):
return True in [t.is_alive() for t in threads]
if __name__ == "__main__":
...
t1 = threading.Thread(target=job1,daemon=True)
t2 = threading.Thread(target=job2,daemon=True)
t1.start()
t2.start()
while is_any_thread_alive([t1,t2]):
time.sleep(0)
One simple 'gotcha' to beware of, are you sure CAPS LOCK isn't on?
I was running a Python script in the Thonny IDE on a Pi4. With CAPS LOCK on, Ctrl+Shift+C is passed to the keyboard buffer, not Ctrl+C.
I am trying to run, pause and terminate the child processes in Python from the parent process. I have tried to use multiprocessing.Value but for some reason the parent process never finishes completely although I terminate and join all the processes. My use case is something like:
def child(flow_flag):
while True:
with flow_flag.get_lock():
flag_value = flow_flag.value
if flag_value == 0:
print("This is doing some work")
elif flag_value == 1:
print("This is waiting for some time to check back later")
time.sleep(5)
else:
print("Time to exit")
break
def main():
flow_flag = Value('i', 0)
processes = [Process(target=child, args=(flow_flag,)) for i in range(10)]
[p.start() for p in processes]
print("Waiting for some work")
with flow_flag.get_lock():
flow_flag.value = 1
print("Do something else")
with flow_flag.get_lock():
flow_flag.value = 0
print("Waiting for more work")
with flow_flag.get_lock():
flow_flag.value = 2
print("Exiting")
for p in processes:
p.terminate()
p.join()
This never finishes properly and I have to Ctrl+C eventually. Then I see this message:
Traceback (most recent call last):
File "/home/abcde/anaconda3/lib/python3.7/threading.py", line 1308, in _shutdown
lock.acquire()
KeyboardInterrupt
What is a better way? FYI, while waiting for something else, I am spawning some other processes. I also had them not terminating properly, and I was using Value with them too. It got fixed when I switched to using Queue for them. However, Queue does not seem to be appropriate for the case above.
P.S. : I am ssh'ing into Ubuntu 18.04.
EDIT: After a lot of debugging, not exiting turned out to be because of a library I am using that I did not suspect to cause this. My apologies for false alarm. Thanks for the suggestions on the better way of controlling the child processes.
Your program works for me, but let me chime in on "is there another way". Instead of polling at 5 second intervals you could create a shared event object that lets the child processes know when they can do their work. Instead of polling for Value 1, wait for the event.
from multiprocessing import *
import time
import os
def child(event, times_up):
while True:
event.wait()
if times_up.value:
print(os.getpid(), "time to exit")
return
print(os.getpid(), "doing work")
time.sleep(.5)
def main():
manager = Manager()
event = manager.Event()
times_up = manager.Value(bool, False)
processes = [Process(target=child, args=(event, times_up)) for i in range(10)]
[p.start() for p in processes]
print("Let processes work")
event.set()
time.sleep(2)
print("Make them stop")
event.clear()
time.sleep(4)
print("Make them go away")
times_up.value = True
event.set()
print("Exiting")
for p in processes:
p.join()
if __name__ == "__main__":
main()
With Python 3.7.7 running on FreeBSD 12.1 (64-bit) I cannot reproduce your problem.
After fixing the indentation and adding the necessary imports The changed program runs fine AFAICT.
BTW, you might want to import sys and add
sys.stdout.reconfigure(line_buffering=True)
to the beginning of your main().
One library that I need to use with python3 (iperf3) needs that the library 'run' function is executed in the main thread.
I'm performing some tests to verify if a new process with the multiprocessing library will let me use the main thread of the process but it seems that with the snippet above I cannot have a new 'main thread' for the process.
What would be the recommended way to run a forked process as the main thread of the new process? Is that possible? Will a system like Celery help with this? I'm planning to run this from a Flask app.
Thanks!
#! /usr/bin/python3
import threading
import multiprocessing as mp
def mp_call():
try:
print("mp_call is mainthread? {}".format(isinstance(threading.current_thread(), threading._MainThread)))
except Exception as e:
print('create iperf e:{}'.format(e))
def thread_call():
try:
print("thread_call is mainthread? {}".format(isinstance(threading.current_thread(), threading._MainThread)))
p = mp.Process(target=mp_call, args=[])
p.daemon = False
p.start()
p.join()
print('Process ended')
except Exception as e:
print('thread e:{}'.format(e))
t = threading.Thread(target=thread_call)
t.daemon = False
t.start()
t.join()
print('Thread ended')
In fact, all threads are dead after fork, you will get a new "main" thread which is your current thread, your checking method is wrong. threading._MainThread is not a public api, use threading.main_thread() instead:
assert threading.current_thread() == threading.main_thread()
because main thread got replaced after subprocess fork, no longer a _MainThread subclass.
I am writing a python script that needs to run a thread which listens to a network socket.
I'm having trouble with killing it using Ctrl+c using the code below:
#!/usr/bin/python
import signal, sys, threading
THREADS = []
def handler(signal, frame):
global THREADS
print "Ctrl-C.... Exiting"
for t in THREADS:
t.alive = False
sys.exit(0)
class thread(threading.Thread):
def __init__(self):
self.alive = True
threading.Thread.__init__(self)
def run(self):
while self.alive:
# do something
pass
def main():
global THREADS
t = thread()
t.start()
THREADS.append(t)
if __name__ == '__main__':
signal.signal(signal.SIGINT, handler)
main()
Appreciate any advise on how to catch Ctrl+c and terminate the script.
The issue is that after the execution falls off the main thread (after main() returned), the threading module will pause, waiting for the other threads to finish, using locks; and locks cannot be interrupted with signals. This is the case in Python 2.x at least.
One easy fix is to avoid falling off the main thread, by adding an infinite loop that calls some function that sleeps until some action is available, like select.select(). If you don't need the main thread to do anything at all, use signal.pause(). Example:
if __name__ == '__main__':
signal.signal(signal.SIGINT, handler)
main()
while True: # added
signal.pause() # added
It's because signals can only be caught by main thread. And here main thread ended his life long time ago (application is waiting for your thread to finish). Try adding
while True:
sleep(1)
to the end of your main() (and of course from time import sleep at the very top).
or as Kevin said:
for t in THREADS:
t.join(1) # join with timeout. Without timeout signal cannot be caught.
I am testing Python threading with the following script:
import threading
class FirstThread (threading.Thread):
def run (self):
while True:
print 'first'
class SecondThread (threading.Thread):
def run (self):
while True:
print 'second'
FirstThread().start()
SecondThread().start()
This is running in Python 2.7 on Kubuntu 11.10. Ctrl+C will not kill it. I also tried adding a handler for system signals, but that did not help:
import signal
import sys
def signal_handler(signal, frame):
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
To kill the process I am killing it by PID after sending the program to the background with Ctrl+Z, which isn't being ignored. Why is Ctrl+C being ignored so persistently? How can I resolve this?
Ctrl+C terminates the main thread, but because your threads aren't in daemon mode, they keep running, and that keeps the process alive. We can make them daemons:
f = FirstThread()
f.daemon = True
f.start()
s = SecondThread()
s.daemon = True
s.start()
But then there's another problem - once the main thread has started your threads, there's nothing else for it to do. So it exits, and the threads are destroyed instantly. So let's keep the main thread alive:
import time
while True:
time.sleep(1)
Now it will keep print 'first' and 'second' until you hit Ctrl+C.
Edit: as commenters have pointed out, the daemon threads may not get a chance to clean up things like temporary files. If you need that, then catch the KeyboardInterrupt on the main thread and have it co-ordinate cleanup and shutdown. But in many cases, letting daemon threads die suddenly is probably good enough.
KeyboardInterrupt and signals are only seen by the process (ie the main thread)... Have a look at Ctrl-c i.e. KeyboardInterrupt to kill threads in python
I think it's best to call join() on your threads when you expect them to die. I've taken the liberty to make the change your loops to end (you can add whatever cleanup needs are required to there as well). The variable die is checked on each pass and when it's True, the program exits.
import threading
import time
class MyThread (threading.Thread):
die = False
def __init__(self, name):
threading.Thread.__init__(self)
self.name = name
def run (self):
while not self.die:
time.sleep(1)
print (self.name)
def join(self):
self.die = True
super().join()
if __name__ == '__main__':
f = MyThread('first')
f.start()
s = MyThread('second')
s.start()
try:
while True:
time.sleep(2)
except KeyboardInterrupt:
f.join()
s.join()
An improved version of #Thomas K's answer:
Defining an assistant function is_any_thread_alive() according to this gist, which can terminates the main() automatically.
Example codes:
import threading
def job1():
...
def job2():
...
def is_any_thread_alive(threads):
return True in [t.is_alive() for t in threads]
if __name__ == "__main__":
...
t1 = threading.Thread(target=job1,daemon=True)
t2 = threading.Thread(target=job2,daemon=True)
t1.start()
t2.start()
while is_any_thread_alive([t1,t2]):
time.sleep(0)
One simple 'gotcha' to beware of, are you sure CAPS LOCK isn't on?
I was running a Python script in the Thonny IDE on a Pi4. With CAPS LOCK on, Ctrl+Shift+C is passed to the keyboard buffer, not Ctrl+C.