I have a main thread which starts child threads and then does signal.pause()
def run(self):
for thread in (some_thread, some_other_thread, another_thread):
thread.start()
signal.pause()
On completion of work of child threads, the main thread doesn't terminate until it receives SIG_TERM signal.
I can't use join method to wait for child threads to complete since on doing so my program is not listening to external kill signals.
What is the best way to make the main thread come out of signal.pause()?
One way could be to send kill signal from any one of the child threads, but that won't be a good approach.
Related
I have a query. I have seen examples where developers write something like the code as follows:
import threading
def do_something():
return true
t = threading.Thread(target=do_something)
t.start()
t.join()
I know that join() signals the interpreter to wait till the thread is completely executed. But what if I do not write t.join()? Will the thread get closed automatically and will it be reused later?
Please let me know the answer. It's my first attempt at creating a multi-threaded application in Python 3.5.0.
A Python thread is just a regular OS thread. If you don't join it, it still keeps running concurrently with the current thread. It will eventually die, when the target function completes or raises an exception. No such thing as "thread reuse" exists, once it's dead it rests in peace.
Unless the thread is a "daemon thread" (via a constructor argument daemon or assigning the daemon property) it will be implicitly joined for before the program exits, otherwise, it is killed abruptly.
One thing to remember when writing multithreading programs in Python, is that they only have limited use due to infamous Global interpreter lock. In short, using threads won't make your CPU-intensive program any faster. They can be useful only when you perform something involving waiting (e.g. you wait for certain file system event to happen in a thread).
The join part means the main program will wait for the thread to end before continuing. Without join, the main program will end and the thread will continue.
Now if you set the daemon parameter to "True", it means the thread will depends on the main program, and it will ends if the main program ends before.
Here is an example to understand better :
import threading
import time
def do_something():
time.sleep(2)
print("do_something")
return True
t = threading.Thread(target=do_something)
t.daemon = True # without the daemon parameter, the function in parallel will continue even your main program ends
t.start()
t.join() # with this, the main program will wait until the thread ends
print("end of main program")
no daemon, no join:
end of main program
do_something
daemon only:
end of main program
join only:
do_something
end of main program
daemon and join:
do_something
end of main program
# Note : in this case the daemon parameter is useless
Without join(), non-daemon threads are running and are completed with the main thread concurrently.
Without join(), daemon threads are running with the main thread concurrently and when the main thread is completed, the daemon threads are exited without completed if the daemon threads are still running.
You can see my answer in this post explaining about it in detail.
According to the documentation: https://docs.python.org/3/library/threading.html
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property or the daemon constructor argument.
Sample code:
thread = threading.Thread(target=MultiHandler().network, args=(conn, data), daemon=True)
thread.start()
Refering to many other StackOverflow answers, it is not clear to me if daemon threads are forced to close when the main thread calls sys.exit()
Referring to the comment posted by zwer,
When a program exits, all of its children threads are killed with it. Threads that are not daemonic will prevent the program from exiting, hence preventing their own destruction. - zwer
In short, yes daemon threads will not stop the program from exiting thus they will be killed on exit.
Please consider this code:
import threading
def printer():
for i in range(2):
with lock:
print ['foo', 'bar', 'baz']
def main():
global lock
lock = threading.Lock()
threads = [threading.Thread(target=printer) for x in xrange(2)]
for t in threads:
t.start()
t.join()
main()
I can understand this code and it is clear: We create two threads and we run them sequentially - we run second thread only when first thread is finished. Ok, now consider another variant:
import threading
def printer():
for i in range(2):
with lock:
print ['foo', 'bar', 'baz']
def main():
global lock
lock = threading.Lock()
threads = [threading.Thread(target=printer) for x in xrange(2)]
for t in threads:
t.start()
for t in threads:
t.join()
main()
What happens here? Ok, we run them in parallel, but what is the purpose of make main thread waiting for child threads in second variant? How it can influence on the output?
In the second variant, the ordering of execution is much less defined.
The lock is released each time through the loop in printer. In both variants, you have two threads and two loops within a thread.
In the first variant, since only one thread runs at a time, you know the total ordering.
In the second variant, each time the lock is released, the thread running may change.
So you might get
thread 1 loop 1
thread 1 loop 2
thread 2 loop 1
thread 2 loop 2
or perhaps
* thread 2 loop 1
* thread 1 loop 1
* thread 1 loop 2
* thread 2 loop 2
The only constraint is that loop1 within a given thread runs before loop 2 within that thread and that the two print statements come together since the lock is held for both of them.
In this particular case I'm not sure the call to t.join() in the second variant has an observable effect. It guarantees that the main thread will be the last thread to end, but I'm not sure that in this code you can observe that in any way. In more complex code, joining the threads can be important so that cleanup actions are only performed after all threads terminate. This can also be very important if you have daemon threads, because the entire program will terminate when all non-daemon threads terminate.
To better understand the multithreading in python, you need to first understand the relationship between the main thread and the children threads.
The main thread is the entry of the program, it is created by your system when you run your script. For example, in your script, the main function is run in the main thread.
While the children thread is created by your main thread when you instanate the Thread class.
The most important thing is how the main thread controls the children thread. Basically, the instance of the Thread is everything that the main thread know about and control over this child thread. At the time when a child thread is created, this child thread does not run immediately, until the main thread call start function on this thread instance. After the start the child thread, you can assume that the main thread and the child thread is running parallelly now.
But one more important thing is how the main thread knows that the task of child thread is done. Though the main thread knows nothing about how the task is done by the child thread, it does be aware of the running status of the child thread. Thread.is_alive can check the status of a thread by the main thread. In pratice, the Thread.join function is always used to tell the main thread wait until the child thread is done. This function will block the main thread.
Okay, let's examine the two script you are confused with. For the first script:
for t in threads:
t.start()
t.join()
The children threads in the loop are started and then joined one by one. Note that start does not block main thread, while join will block the main thread wait until this child thread is done. Thus they are running sequentially.
While for the second script:
for t in threads:
t.start()
for t in threads:
t.join()
All children threads are started in the first loop. As the Thread.start function will not block the main thread, all children threadings are running parallelly after the first loop. In the second loop, the main thread will wait for the task done of each child thread one by one.
Now I think you should notice the difference between these two script: in the first one, children threads running one by one, while in the second script, they are running simultaneously.
There are other useful topics for the python threading:
(1) How to handle the Keyboard Interrupt Exception, e.g., when I want to terminate the program by Ctrl-C? Only the main thread will receive the exception, you have to handle the termination of children threads.
(2) Multithreading vs Multiprocessing. Although we are saying that threading is parallel, it is not the real parallel in CPU level. So if your application is CPU intensive, try multiprocessing, and if your application is I/O intensive, multithreading maybe sufficient.
By the way, read through the documentation of python threading section and try some code may help you understand it.
Hope this would be helpful. Thanks.
There is one process, which creates thread in the middle of process's work. Now we have Main thread and Thread2. Main thread does some job and it coudln't wait for an event. Thread2 - is an infinite loop, which is waiting for a signal. When signal comes, thread2 should send command (e.g. print str) to Main thread.
Is there any way to pass the command from thread2 to Main thread without making Main thread waiting? I mean something like interrupts or callbacks.
And if there comes a signal to thread2, then thread2 interrupt Main thread work and force it to do my command (print str), or thread2 send command with callback, which goes to queue and runs when its turn in queue.
I have a class that inherits from threading.Thread.
After processing an item from a queue I want it to call a function that I handed over to the thread when starting it. Doing that callback from the run-Method will have it run in my worker thread. Is there a way to run that callback inside the MainThread?
Thanks & Greetings,
Sean
You didnt mention intially that you are using pyqt. Qt has signals and slots built in just for this purpose. If you are launching your thread using QThread then it has the 'finished' signal that your main thread can be connected to on a slot. If you need even more control of the resulting signal you can emit a custom one from the thread with any value you want. When you construct the thread just connect its signal to the slot that you want to act as the callback.
Otherwise you would be blocking your main event loop if you are specifically waiting on the thread.
Here is a link specifically about the new style signal/slots: http://www.riverbankcomputing.co.uk/static/Docs/PyQt4/html/new_style_signals_slots.html
It makes it extremely easy now to create a python signal with whatever signature you want and connect it in an object oriented fashion to a slot.
Also as a side note about PyQt... Some of the other answers here have suggested creating a loop on your main thread that waits to collect responses in a queue from its dispatched threads. That is essentially what PyQt does. When you start your app the main thread goes into an event loop. This event loop is processing events that stack up in its event queue. There are different types of signal/slot connections. If your two endpoints are in the same thread, then a direct connection will be made and the emitting signal will call the slot. If they are in different threads, then I believe it goes through the event loop so that it can be processed outside of the emitters thread.
Push the method onto a queue and have the main thread sit and wait on that queue, running each method as it pulls them off.
You can use condition variables and have your main thread wait on a message from the sub-thread. The sub-thread can signal the main thread to call that method.