I don't manage to understand why my SIGINT is never caught by the piece of code below.
#!/usr/bin/env python
from threading import Thread
from time import sleep
import signal
class MyThread(Thread):
def __init__(self):
Thread.__init__(self)
self.running = True
def stop(self):
self.running = False
def run(self):
while self.running:
for i in range(500):
col = i**i
print col
sleep(0.01)
global threads
threads = []
for w in range(150):
threads.append(MyThread())
def stop(s, f):
for t in threads:
t.stop()
signal.signal(signal.SIGINT, stop)
for t in threads:
t.start()
for t in threads:
t.join()
To clean this code I would prefer to try/except the join() and closing all threads in case of exception, would that work?
One of the problems with multithreading in python is that join() more or less disables signals.
This is because the signal can only be delivered to the main thread, but the main thread is already busy with performing the join() and the join is not interruptible.
You can deduce this from the documentation of the signal module
Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), pause(), setitimer() or getitimer(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.
You can work your way around it, by busy-looping over the join operation:
for t in threads:
while t.isAlive():
t.join(timeout=1)
This is, however, none to efficient:
The workaround of calling join() with a timeout has a drawback:
Python's threading wait routine polls 20 times a second when
given any timeout. All this polling can mean lots of CPU
interrupts/wakeups on an otherwise idle laptop and drain the
battery faster.
Some more details are provided here:
Python program with thread can't catch CTRL+C
Bug reports for this problem with a discussion of the underlying issue can be found here:
https://bugs.python.org/issue1167930
https://bugs.python.org/issue1171023
Related
I am writing an queue processing application which uses threads for waiting on and responding to queue messages to be delivered to the app. For the main part of the application, it just needs to stay active. For a code example like:
while True:
pass
or
while True:
time.sleep(1)
Which one will have the least impact on a system? What is the preferred way to do nothing, but keep a python app running?
I would imagine time.sleep() will have less overhead on the system. Using pass will cause the loop to immediately re-evaluate and peg the CPU, whereas using time.sleep will allow the execution to be temporarily suspended.
EDIT: just to prove the point, if you launch the python interpreter and run this:
>>> while True:
... pass
...
You can watch Python start eating up 90-100% CPU instantly, versus:
>>> import time
>>> while True:
... time.sleep(1)
...
Which barely even registers on the Activity Monitor (using OS X here but it should be the same for every platform).
Why sleep? You don't want to sleep, you want to wait for the threads to finish.
So
# store the threads you start in a your_threads list, then
for a_thread in your_threads:
a_thread.join()
See: thread.join
If you are looking for a short, zero-cpu way to loop forever until a KeyboardInterrupt, you can use:
from threading import Event
Event().wait()
Note: Due to a bug, this only works on Python 3.2+. In addition, it appears to not work on Windows. For this reason, while True: sleep(1) might be the better option.
For some background, Event objects are normally used for waiting for long running background tasks to complete:
def do_task():
sleep(10)
print('Task complete.')
event.set()
event = Event()
Thread(do_task).start()
event.wait()
print('Continuing...')
Which prints:
Task complete.
Continuing...
signal.pause() is another solution, see https://docs.python.org/3/library/signal.html#signal.pause
Cause the process to sleep until a signal is received; the appropriate handler will then be called. Returns nothing. Not on Windows. (See the Unix man page signal(2).)
I've always seen/heard that using sleep is the better way to do it. Using sleep will keep your Python interpreter's CPU usage from going wild.
You don't give much context to what you are really doing, but maybe Queue could be used instead of an explicit busy-wait loop? If not, I would assume sleep would be preferable, as I believe it will consume less CPU (as others have already noted).
[Edited according to additional information in comment below.]
Maybe this is obvious, but anyway, what you could do in a case where you are reading information from blocking sockets is to have one thread read from the socket and post suitably formatted messages into a Queue, and then have the rest of your "worker" threads reading from that queue; the workers will then block on reading from the queue without the need for neither pass, nor sleep.
Running a method as a background thread with sleep in Python:
import threading
import time
class ThreadingExample(object):
""" Threading example class
The run() method will be started and it will run in the background
until the application exits.
"""
def __init__(self, interval=1):
""" Constructor
:type interval: int
:param interval: Check interval, in seconds
"""
self.interval = interval
thread = threading.Thread(target=self.run, args=())
thread.daemon = True # Daemonize thread
thread.start() # Start the execution
def run(self):
""" Method that runs forever """
while True:
# Do something
print('Doing something imporant in the background')
time.sleep(self.interval)
example = ThreadingExample()
time.sleep(3)
print('Checkpoint')
time.sleep(2)
print('Bye')
I have a specific problem.
Main content of program starts with creating Process with dbus loop, where I listen for signals.
Content of signals I store in queues. In next part of main I have a threadpool.
When some thread takes item from queue, it use specific function(detection) to handle request - based on content of item from queue. (There is operation on database, from where I take data and make some operations depends on request)
Every thread in thread pool starts one more thread, which should handle signals (current status and interrupt).
For example: I receive signal, which means I have to handle something on numbers. Any thread from threadpool takes this item from queue and starts function which handle something on numbers - it can take long time. So after any time, I receive signal for current status and I need to send current status of detection - that's why I use threads (for shared memory). Also I can receive interrupt signal from D-Bus ("it takes too long time, so stop this detection and be free for another request"). And the interrupt is the main problem...
So my main questions are:
Is there any way, I can raise exception on interrupt signal and stop function (detection)? (I just found solution, but only for catch in main... but I need to catch it in thread which is in threadpool and raise in thread which is in thread in threadpool)
Second question is about GIL... does my thread with signal receiving receive all signals? I think it doesn't... (Yes, I use threads_init())
program:
SERVICE = multiprocessing.Process(target=dbus_signal_receiver, args=(...))
SERVICE.daemon = True
SERVICE.start()
class worker(threading.Thread):
def __init__(self,...):
threading.Thread.__init__(self)
def run(self):
while True:
#get item from queue
s = threading.Thread(target=curr_and_interr_signal_handle, args=(ID of item from queue,...))
s.daemon = True
s.start()
#start specific detection based on request
for i in range(number of threads):
t = worker(...)
t.daemon = True
t.start()
and I hoped, something like this will work... (but it doesn't)
...
class worker(threading.Thread):
def __init__(self,...):
threading.Thread.__init__(self)
def run(self):
while True:
try:
#get item from queue
s = threading.Thread(target=curr_and_interr_signal_handle, args=(ID of item from queue,...))
s.daemon = True
s.start()
#start specific detection based on request
except raised_interrupt_exception:
#continue - wait for another request from queue
...
Read about 18.8.1.2. Signals and threads
Python signal handlers are always executed in the main Python thread,
even if the signal was received in another thread.
This means that signals can’t be used as a means of inter-thread communication.
You can use the synchronization primitives from the threading module instead.
Besides, only the main thread is allowed to set a new signal handler.
Read about 17.1.7. Event Objects
This is one of the simplest mechanisms for communication between threads: one thread signals an event and other threads wait for it
Isn't clear why you have to use thread in thread.
Why could your worker thread not handle detection?
For instance, the following should be do it:
def run(self):
while self.running.is_set():
#get item from queue
#start specific detection based on request
from threading import Thread
class MyClass:
#...
def method2(self):
while True:
try:
hashes = self.target.bssid.replace(':','') + '.pixie'
text = open(hashes).read().splitlines()
except IOError:
time.sleep(5)
continue
# function goes on ...
def method1(self):
new_thread = Thread(target=self.method2())
new_thread.setDaemon(True)
new_thread.start() # Main thread will stop there, wait until method 2
print "Its continues!" # wont show =(
# function goes on ...
Is it possible to do like that?
After new_thread.start() Main thread waits until its done, why is that happening? i didn't provide new_thread.join() anywhere.
Daemon doesn't solve my problem because my problem is that Main thread stops right after new thread start, not because main thread execution is end.
As written, the call to the Thread constructor is invoking self.method2 instead of referring to it. Replace target=self.method2() with target=self.method2 and the threads will run in parallel.
Note that, depending on what your threads do, CPU computations might still be serialized due to the GIL.
IIRC, this is because the program doesn't exit until all non-daemon threads have finished execution. If you use a daemon thread instead, it should fix the issue. This answer gives more details on daemon threads:
Daemon Threads Explanation
I found this simple code at https://code.google.com/p/pyloadtools/wiki/CodeTutorialMultiThreading
import _thread
def hello(num):
print('hello from thread %s\n' % num)
_thread.start_new_thread(hello, (0,))
_thread.start_new_thread(hello, (1,))
_thread.start_new_thread(hello, (2,))
But when I run this, it works on IDLE, but not on eclipse which uses PyDev. Any idea how to fix it?
Note: I think the main program terminates before the threads run. The threads dont get enough time to run I guess. How do I fix it? May be should I use join?
Quoting the Caveats section of _thread documentation,
When the main thread exits, it is system defined whether the other threads survive. On most systems, they are killed without executing try ... finally clauses or executing object destructors.
When the main thread exits, it does not do any of its usual cleanup (except that try ... finally clauses are honored), and the standard I/O files are not flushed.
There are two possibilities here.
The main thread starts three threads but it exits before the threads finish the execution. So, the standard I/O files are not flushed, as they are buffered, by default.
Or, the main thread dies, and as per the first bullet point quoted, all the child threads are killed in action.
Either way, you need to make sure the main thread doesn't die before the children complete.
But when you run from IDLE, the main thread still exists, so, the I/O buffers are flushed when the threads actually complete. That is why it works in IDLE but not in eclipse.
To make sure that the main thread exits only after all the threads complete, you can make it wait for the child threads with
1. Semaphore
You can use Semaphore, like this
import _thread
import threading
def hello(num):
print('hello from thread %s' % num)
# Release semaphore when the thread is actually done
sem.release()
def create_thread(value):
# Acquire semaphore when the thread is actually created
sem.acquire()
_thread.start_new_thread(hello, (value,))
# Counting semaphore. Maximum three threads can acquire.
# Next acquire call has to wait till somebody releases
sem = threading.Semaphore(3)
for i in range(3):
create_thread(i)
# We are capturing the semaphore three times again, because
# whenever a thread completes it releases it. So, only when we
# acquire it thrice to make sure that all threads have completed.
for i in range(3):
sem.acquire()
2. Lock Objects
Alternatively, you can use the _thread.lock objects, like this
import _thread
locks = []
def hello(num, lockobject):
print('hello from thread %s' % num)
# Release the lock as we are done here
lockobject.release()
def create_thread(value):
# Create a lock and acquire it
a_lock = _thread.allocate_lock()
a_lock.acquire()
# Store it in the global locks list
locks.append(a_lock)
# Pass it to the newly created thread which can release the lock once done
_thread.start_new_thread(hello, (value, a_lock))
for i in range(3):
create_thread(i)
# Acquire all the locks, which means all the threads have released the locks
all(lock.acquire() for lock in locks)
Now you will see that the program always prints the hello from message.
Note: As the documentation says, _thread is a Low-level threading API. So, better use higher level module like threading, where you can simply wait for the all the threads to exit with join method.
From https://docs.python.org/3/library/_thread.html#module-_thread
The threading module provides an easier to use and higher-level threading API built on top of this module.
The module is optional.
So please use threading, not the optional _thread module.
I have a simple app that listens to a socket connection. Whenever certain chunks of data come in a callback handler is called with that data. In that callback I want to send my data to another process or thread as it could take a long time to deal with. I was originally running the code in the callback function, but it blocks!!
What's the proper way to spin off a new task?
threading is the threading library usually used for resource-based multithreading. The multiprocessing library is another library, but designed more for running intensive parallel computing tasks; threading is generally the recommended library in your case.
Example
import threading, time
def my_threaded_func(arg, arg2):
print "Running thread! Args:", (arg, arg2)
time.sleep(10)
print "Done!"
thread = threading.Thread(target=my_threaded_func, args=("I'ma", "thread"))
thread.start()
print "Spun off thread"
The multiprocessing module has worker pools. If you don't need a pool of workers, you can use Process to run something in parallel with your main program.
import threading
from time import sleep
import sys
# assume function defs ...
class myThread (threading.Thread):
def __init__(self, threadID):
threading.Thread.__init__(self)
self.threadID = threadID
def run(self):
if self.threadID == "run_exe":
run_exe()
def main():
itemList = getItems()
for item in itemList:
thread = myThread("run_exe")
thread.start()
sleep(.1)
listenToSocket(item)
while (thread.isAlive()):
pass # a way to wait for thread to finish before looping
main()
sys.exit(0)
The sleep between thread.start() and listenToSocket(item) ensures that the thread is established before you begin to listen. I implemented this code in a unit test framework were I had to launch multiple non-blacking processes (len(itemList) number of times) because my other testing framework (listenToSocket(item)) was dependent on the processes.
un_exe() can trigger a subprocess call that can be blocking (i.e. invoking pipe.communicate()) so that output data from the execution will still be printed in time with the python script output. But the nature of threading makes this ok.
So this code solves two problems - print data of a subprocess without blocking script execution AND dynamically create and start multiple threads sequentially (makes maintenance of the script better if I ever add more items to my itemList later).