I found this simple code at https://code.google.com/p/pyloadtools/wiki/CodeTutorialMultiThreading
import _thread
def hello(num):
print('hello from thread %s\n' % num)
_thread.start_new_thread(hello, (0,))
_thread.start_new_thread(hello, (1,))
_thread.start_new_thread(hello, (2,))
But when I run this, it works on IDLE, but not on eclipse which uses PyDev. Any idea how to fix it?
Note: I think the main program terminates before the threads run. The threads dont get enough time to run I guess. How do I fix it? May be should I use join?
Quoting the Caveats section of _thread documentation,
When the main thread exits, it is system defined whether the other threads survive. On most systems, they are killed without executing try ... finally clauses or executing object destructors.
When the main thread exits, it does not do any of its usual cleanup (except that try ... finally clauses are honored), and the standard I/O files are not flushed.
There are two possibilities here.
The main thread starts three threads but it exits before the threads finish the execution. So, the standard I/O files are not flushed, as they are buffered, by default.
Or, the main thread dies, and as per the first bullet point quoted, all the child threads are killed in action.
Either way, you need to make sure the main thread doesn't die before the children complete.
But when you run from IDLE, the main thread still exists, so, the I/O buffers are flushed when the threads actually complete. That is why it works in IDLE but not in eclipse.
To make sure that the main thread exits only after all the threads complete, you can make it wait for the child threads with
1. Semaphore
You can use Semaphore, like this
import _thread
import threading
def hello(num):
print('hello from thread %s' % num)
# Release semaphore when the thread is actually done
sem.release()
def create_thread(value):
# Acquire semaphore when the thread is actually created
sem.acquire()
_thread.start_new_thread(hello, (value,))
# Counting semaphore. Maximum three threads can acquire.
# Next acquire call has to wait till somebody releases
sem = threading.Semaphore(3)
for i in range(3):
create_thread(i)
# We are capturing the semaphore three times again, because
# whenever a thread completes it releases it. So, only when we
# acquire it thrice to make sure that all threads have completed.
for i in range(3):
sem.acquire()
2. Lock Objects
Alternatively, you can use the _thread.lock objects, like this
import _thread
locks = []
def hello(num, lockobject):
print('hello from thread %s' % num)
# Release the lock as we are done here
lockobject.release()
def create_thread(value):
# Create a lock and acquire it
a_lock = _thread.allocate_lock()
a_lock.acquire()
# Store it in the global locks list
locks.append(a_lock)
# Pass it to the newly created thread which can release the lock once done
_thread.start_new_thread(hello, (value, a_lock))
for i in range(3):
create_thread(i)
# Acquire all the locks, which means all the threads have released the locks
all(lock.acquire() for lock in locks)
Now you will see that the program always prints the hello from message.
Note: As the documentation says, _thread is a Low-level threading API. So, better use higher level module like threading, where you can simply wait for the all the threads to exit with join method.
From https://docs.python.org/3/library/_thread.html#module-_thread
The threading module provides an easier to use and higher-level threading API built on top of this module.
The module is optional.
So please use threading, not the optional _thread module.
Related
Suppose I have something like this :
import threading
import time
_FINISH = False
def hang():
while True:
if _FINISH:
break
print 'hanging..'
time.sleep(10)
def main():
global _FINISH
t = threading.Thread(target=hang)
t.setDaemon( True )
t.start()
time.sleep(10)
if __name__ == '__main__':
main()
If my thread is daemon, do I need to have a global _FINISH to control exit clause of break loop? I tried and I don't seem to need it - when program exits ( in that case after the sleep ) then program terminates, which closes the thread too.
But I've seen that code too - is it just bad practise? Can I get away with no global flag for controlling the loop?
According to [Python 3.Docs]: threading - Thread Objects (emphasis is mine):
A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property or the daemon constructor argument.
Note: Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
Per above, technically, you don't need the _FINISH logic, as the thread will end when the main one does. But, according to your code, no one (main thread) signals that the thread should end (something like _FINISH = True), so the logic in the thread is useless (therefore it can be removed).
Also, according to the above recommendation, you should implement the synchronization mechanism between your threads, and avoid making them daemons (in most of the cases).
I am very new to the concept of threading and the concepts are still somewhat fuzzy.
But as of now i have a requirement in which i spin up an arbitrary number of threads from my Python program and then my Python program should indicate to the user running the process which threads have finished executing. Below is my first try:
import threading
from threading import Thread
from time import sleep
def exec_thread(n):
name = threading.current_thread().getName()
filename = name + ".txt"
with open(filename, "w+") as file:
file.write(f"My name is {name} and my main thread is {threading.main_thread()}\n")
sleep(n)
file.write(f"{name} exiting\n")
t1 = Thread(name="First", target=exec_thread, args=(10,))
t2 = Thread(name="Second", target=exec_thread, args=(2,))
t1.start()
t2.start()
while len(threading.enumerate()) > 1:
print(f"Waiting ... !")
sleep(5)
print(f"The threads are done"
So this basically tells me when all the threads are done executing.
But i want to know as soon as any one of my threads have completed execution so that i can tell the user that please check the output file for the thread.
I cannot use thread.join() since that would block my main program and the user would not know anything unless everything is complete which might take hours. The user wants to know as soon as some results are available.
Now i know that we can check individual threads whether they are active or not by doing : thread.isAlive() but i was hoping for a more elegant solution in which if the child threads can somehow communicate with the main thread and say I am done !
Many thanks for any answers in advance.
The simplest and most straightforward way to indicate a single thread is "done" is to put the required notification in the thread's implementation method, as the very last step. For example, you could print out a notification to the user.
Or, you could use events, see: https://docs.python.org/3/library/threading.html#event-objects
This is one of the simplest mechanisms for communication between
threads: one thread signals an event and other threads wait for it.
An event object manages an internal flag that can be set to true with
the set() method and reset to false with the clear() method. The
wait() method blocks until the flag is true.
So, the "final act" in your thread implementation would be to set an event object, and your main thread can wait until it's set.
Or, for an even fancier and more mechanism, use queues: https://docs.python.org/3/library/queue.html
Each thread writes an "I'm done" object to the queue when done, and the main thread can read those notifications from the queue in sequence as each thread completes.
I was reading about Queue in the Python documentation and this book, and I don't fully understand why my thread hangs. I have the following mcve:
from threading import Thread
import queue
def print_number(number_queue_display):
while True:
number = number_queue_display.get()
print(number)
number_queue_display.task_done()
number_queue = queue.Queue()
printing_numbers = Thread(target=print_number, args=(number_queue,),)
printing_numbers.start()
number_queue.put(5)
number_queue.put(10)
number_queue.put(15)
number_queue.put(20)
number_queue.join()
printing_numbers.join()
The only time it works is if I set the thread to daemon like so:
printing_numbers.setDaemon(True)
but that's because as stated in the Python documentation, the program will exit when only the daemon threads are left. The Python docs example for Queue doesn't use a daemon thread.
A thread can be flagged as a “daemon thread”. The significance of this
flag is that the entire Python program exits when only daemon threads
are left.
Even if I were to remove the two joins(number_queue.join() printing_numbers.join()), it still hangs, but I'm unsure of why.
Questions:
Why is it hanging?
How do I keep it as a non-daemon thread, but prevent it from hanging?
print_number() is running an infinite loop - it never exits, so the thread never ends. It sits in number_queue_display.get() forever, waiting for another queue item that never appears. Then, since the thread never ends, printing_numbers.join() also waits forever.
So you need some way to tell the thread to quit. One common way is to put a special "sentinel" value on the queue, and have the thread exit when it sees that. For concreteness, here's a complete program, which is very much the same as what you started with. None is used as the sentinel (and is commonly used for this purpose), but any unique object would work. Note that the .task_done() parts were removed, because they no longer serve a purpose.
from threading import Thread
import queue
def print_number(number_queue_display):
while True:
number = number_queue_display.get()
if number is None:
break
print(number)
number_queue = queue.Queue()
printing_numbers = Thread(target=print_number, args=(number_queue,),)
printing_numbers.start()
number_queue.put(5)
number_queue.put(10)
number_queue.put(15)
number_queue.put(20)
number_queue.put(None) # tell the thread it's done
printing_numbers.join() # wait for the thread to exit
I don't manage to understand why my SIGINT is never caught by the piece of code below.
#!/usr/bin/env python
from threading import Thread
from time import sleep
import signal
class MyThread(Thread):
def __init__(self):
Thread.__init__(self)
self.running = True
def stop(self):
self.running = False
def run(self):
while self.running:
for i in range(500):
col = i**i
print col
sleep(0.01)
global threads
threads = []
for w in range(150):
threads.append(MyThread())
def stop(s, f):
for t in threads:
t.stop()
signal.signal(signal.SIGINT, stop)
for t in threads:
t.start()
for t in threads:
t.join()
To clean this code I would prefer to try/except the join() and closing all threads in case of exception, would that work?
One of the problems with multithreading in python is that join() more or less disables signals.
This is because the signal can only be delivered to the main thread, but the main thread is already busy with performing the join() and the join is not interruptible.
You can deduce this from the documentation of the signal module
Some care must be taken if both signals and threads are used in the same program. The fundamental thing to remember in using signals and threads simultaneously is: always perform signal() operations in the main thread of execution. Any thread can perform an alarm(), getsignal(), pause(), setitimer() or getitimer(); only the main thread can set a new signal handler, and the main thread will be the only one to receive signals (this is enforced by the Python signal module, even if the underlying thread implementation supports sending signals to individual threads). This means that signals can’t be used as a means of inter-thread communication. Use locks instead.
You can work your way around it, by busy-looping over the join operation:
for t in threads:
while t.isAlive():
t.join(timeout=1)
This is, however, none to efficient:
The workaround of calling join() with a timeout has a drawback:
Python's threading wait routine polls 20 times a second when
given any timeout. All this polling can mean lots of CPU
interrupts/wakeups on an otherwise idle laptop and drain the
battery faster.
Some more details are provided here:
Python program with thread can't catch CTRL+C
Bug reports for this problem with a discussion of the underlying issue can be found here:
https://bugs.python.org/issue1167930
https://bugs.python.org/issue1171023
I have a long process that I've scheduled to run in a thread, because otherwise it will freeze the UI in my wxpython application.
I'm using:
threading.Thread(target=myLongProcess).start()
to start the thread and it works, but I don't know how to pause and resume the thread. I looked in the Python docs for the above methods, but wasn't able to find them.
Could anyone suggest how I could do this?
I did some speed tests as well, the time to set the flag and for action to be taken is pleasantly fast 0.00002 secs on a slow 2 processor Linux box.
Example of thread pause test using set() and clear() events:
import threading
import time
# This function gets called by our thread.. so it basically becomes the thread init...
def wait_for_event(e):
while True:
print('\tTHREAD: This is the thread speaking, we are Waiting for event to start..')
event_is_set = e.wait()
print('\tTHREAD: WHOOOOOO HOOOO WE GOT A SIGNAL : %s' % event_is_set)
# or for Python >= 3.6
# print(f'\tTHREAD: WHOOOOOO HOOOO WE GOT A SIGNAL : {event_is_set}')
e.clear()
# Main code
e = threading.Event()
t = threading.Thread(name='pausable_thread',
target=wait_for_event,
args=(e,))
t.start()
while True:
print('MAIN LOOP: still in the main loop..')
time.sleep(4)
print('MAIN LOOP: I just set the flag..')
e.set()
print('MAIN LOOP: now Im gonna do some processing')
time.sleep(4)
print('MAIN LOOP: .. some more processing im doing yeahhhh')
time.sleep(4)
print('MAIN LOOP: ok ready, soon we will repeat the loop..')
time.sleep(2)
There is no method for other threads to forcibly pause a thread (any more than there is for other threads to kill that thread) -- the target thread must cooperate by occasionally checking appropriate "flags" (a threading.Condition might be appropriate for the pause/unpause case).
If you're on a unix-y platform (anything but windows, basically), you could use multiprocessing instead of threading -- that is much more powerful, and lets you send signals to the "other process"; SIGSTOP should unconditionally pause a process and SIGCONT continues it (if your process needs to do something right before it pauses, consider also the SIGTSTP signal, which the other process can catch to perform such pre-suspension duties. (There may be ways to obtain the same effect on Windows, but I'm not knowledgeable about them, if any).
You can use signals: http://docs.python.org/library/signal.html#signal.pause
To avoid using signals you could use a token passing system. If you want to pause it from the main UI thread you could probably just use a Queue.Queue object to communicate with it.
Just pop a message telling the thread the sleep for a certain amount of time onto the queue.
Alternatively you could simply continuously push tokens onto the queue from the main UI thread. The worker should just check the queue every N seconds (0.2 or something like that). When there are no tokens to dequeue the worker thread will block. When you want it to start again just start pushing tokens on to the queue from the main thread again.
The multiprocessing module works fine on Windows. See the documentation here (end of first paragraph):
http://docs.python.org/library/multiprocessing.html
On the wxPython IRC channel, we had a couple fellows trying multiprocessing out and they said it worked. Unfortunately, I have yet to see anyone who has written up a good example of multiprocessing and wxPython.
If you (or anyone else on here) come up with something, please add it to the wxPython wiki page on threading here: http://wiki.wxpython.org/LongRunningTasks
You might want to check that page out regardless as it has several interesting examples using threads and queues.
You might take a look at the Windows API for thread suspension.
As far as I'm aware there is no POSIX/pthread equivalent. Furthermore, I cannot ascertain if thread handles/IDs are made available from Python. There are also potential issues with Python, as its scheduling is done using the native scheduler, it's unlikely that it is expecting threads to suspend, particularly if threads suspended while holding the GIL, amongst other possibilities.
I had the same issue. It is more effective to use time.sleep(1800) in the thread loop to pause the thread execution.
e.g
MON, TUE, WED, THU, FRI, SAT, SUN = range(7) #Enumerate days of the week
Thread 1 :
def run(self):
while not self.exit:
try:
localtime = time.localtime(time.time())
#Evaluate stock
if localtime.tm_hour > 16 or localtime.tm_wday > FRI:
# do something
pass
else:
print('Waiting to evaluate stocks...')
time.sleep(1800)
except:
print(traceback.format_exc())
Thread 2
def run(self):
while not self.exit:
try:
localtime = time.localtime(time.time())
if localtime.tm_hour >= 9 and localtime.tm_hour <= 16:
# do something
pass
else:
print('Waiting to update stocks indicators...')
time.sleep(1800)
except:
print(traceback.format_exc())