I have set up a thread pool executor with 4 threads. I have added 2 items to my queue to be processed. When I submit the tasks and retrieve futures, it appears the other 2 threads not processing items in the queue keep running and hang, even if they are not processing anything!
import time
import queue
import concurrent
def _read_queue(queue):
msg = queue.get()
time.sleep(2)
queue.task_done()
n_threads = 4
q = queue.Queue()
q.put('test')
q.put("test2")
with concurrent.futures.ThreadPoolExecutor(max_workers=n_threads) as pool:
futures = []
for _ in range(n_threads):
future = pool.submit(_read_queue, q)
print(future.running())
print("Why am running forever?")
How can I adjust my code so that threads that are not processing anything from the queue are shutdown so my program can terminate?
Because queue.get() operation block your ThreadPoolExecutor threads.
for _ in range(n_threads):
future = pool.submit(_read_queue, q)
print(future.running())
Let's examine future = pool.submit(_read_queue, q) in every iteration of for loop
In first iteration of for loop, pool.submit(_read_queue, q) will put a job inside the ThreadPoolExecutor internal queue. When any job are put inside the ThreadPoolExecutor internal queue (it's name is self._work_queue), submit method will create a thread1(I say thread1,thread2.. for easily understand) thread. This thread will execute _read_queue func(This can be happen immediately or this can be happen after the fourth iteration of for loop. This ordering is depends on the Operating System Scheduler, please look at this) and queue.get() will return "test". Then, this thread will sleep for 2 seconds.
In second iteration of for loop, pool.submit(_read_queue, q) will put a job inside the ThreadPoolExecutor internal queue and then submit method will check that there is any thread which is waiting for a job ? No, there is no any waiting thread, first thread is sleeping(for 2 seconds). So submit method will do below steps :
if "there is a thread which will accept a job immediately": #Step 1
return
# Step 2
if numbe_of_created_threads(now this is 1) < self._max_workers:
threading.Thread().... #create a new thread
And then submit method will create a new thread2 thread and this thread will execute _read_queue func and queue.get() will return "test2". Then, this thread will sleep for 2 seconds. Also, q, queue object will be empty and subsequent get() call will block the calling thread
In third iteration of for loop, submit method will put a job inside the ThreadPoolExecutor internal queue and then submit method will check that there is any thread which is waiting for a job ? There is no any waiting thread, first thread is sleeping(for 2 seconds) and second thread also is sleeping, so submit method will create a new thread3 thread (It will check the both step1 and Step2) and this thread will execute _read_queue func same as other threads did. When thread3 run, it will execute queue.get() but this will block the thread3, because q,queue object is empty and if you call get(blocking=True) method of a empty queue object, your calling thread will be blocked .
In fourth iteration of for loop, this will be same as with third case, and then thread4 will be blocked on queue.get() operation.
I assume 2 seconds not passed now, and there will be 5 thread which is alive (can be sleep mode or not) currently. After 2 seconds passed, thread1 and thread2(because time.sleep(2) will return) will terminate*1 but thread3 and thread4 will not, because queue.get() blocking them. That's why your main thread (whole program) will wait them and not terminate.
What can we do in this situation ?
We can put two elements inside the q object because q.get() blocking your thread by using acquire a lock object. We can only release this lock by calling release() method, to do that we need to call queue.put(something)
Here is one of the solutions ;
import time,threading
import queue
from concurrent import futures
def _read_queue(queue):
msg = queue.get()
time.sleep(2)
queue.put(None)
n_threads = 4
q = queue.Queue()
q.put('test')
q.put("test2")
with futures.ThreadPoolExecutor(max_workers=n_threads) as pool:
futures = []
for _ in range(n_threads):
futures.append(pool.submit(_read_queue, q))
*1, I said ThreadPoolExecutor threads will terminate after function finished, but it is depend on calling the shutdown() method, If we don't call shutdown() method of pool object, thread will not terminate even if function finished. Because creating and destruction a thread is costly, that's why threadpool concept is there.(shutdown() method will be called end of the with statement)
If I'm wrong somewhere please correct me.
Related
operating system: windows 10
python 3.7.6
I have a task function
def mytest(task_queue):
while True:
print(task_queue.get())
I want run sub thread and waiting for others put something into task_queue.
If I use concurrent.futures.ThreadPoolExecutor().submit() start thread, then put something into queue, it will block, task_queue.put(1) never run.
if __name__ == '__main__':
import queue
task_queue = queue.Queue()
task_queue.put(0)
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.submit(mytest, task_queue)
task_queue.put(1)
task_queue.put(2)
# only print 0, then block
If I start thread by Thread().start(), it works as I expect.
if __name__ == '__main__':
import queue
task_queue = queue.Queue()
task_queue.put(0)
t1 = threading.Thread(target=mytest, args=(task_queue,))
t1.start()
task_queue.put(1)
task_queue.put(2)
# print 0, 1, 2. but the main thread does not exit
But I don't think either of these methods will block the code because they just start the thread.
So I have 2 question:
Why does submit() block the code?
Why main thread does not exit when use start() to start sub thread without join()?
THX
Q-1) Why does submit() block the code?
A-1) No submit() method not blocking, it is schedules the callable mytest to be executed as mytest(task_queue) and returns a Future object. Look at below code, you will see that submit() method will not block the main thread
if __name__ == '__main__':
import queue
task_queue = queue.Queue()
task_queue.put(0)
executor = ThreadPoolExecutor()
executor.submit(mytest, task_queue)
task_queue.put(1)
task_queue.put(2)
print("hello")
>> 0
hello
1
2
Or you can do like :
if __name__ == '__main__':
import queue
task_queue = queue.Queue()
task_queue.put(0)
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.submit(mytest, task_queue)
task_queue.put(1)
task_queue.put(2)
You will see that task_queue.put(1) and other will be called immediately
As you see above examples, submit() method not blocking, but when you use with statement with concurrent.futures.ThreadPoolExecutor(), __exit__() method will be called end of the with statement. This __exit__() method will call shutdown(wait=True) method of ThreadPoolExecutor() class. When we look at the doc about shutdown(wait=True) method :
If wait is True then this method will not return until all the pending
futures are done executing and the resources associated with the
executor have been freed. If wait is False then this method will
return immediately and the resources associated with the executor will
be freed when all pending futures are done executing. Regardless of
the value of wait, the entire Python program will not exit until all
pending futures are done executing.
That's why your main thread blocked end of the with statement.
I want to give an answer to your second question, but i am confused with something about main thread exit or not. I will edit this answer later (for second question)
Thread(...).start() creates a new thread. End of story. You can always create a new thread if there's still some memory left in which to create it.
executor.submit(mytest, task_queue) creates a new task and adds it to the task_queue. But adding the task to the queue will force the caller to wait until there is room for it in the queue.
Some time later, when the task eventually reaches the head of the queue, a worker thread will take the task and execute it.
I am learning about Thread in Python and am trying to make a simple program, one that uses threads to grab a number off the Queue and print it.
I have the following code
import threading
from Queue import Queue
test_lock = threading.Lock()
tests = Queue()
def start_thread():
while not tests.empty():
with test_lock:
if tests.empty():
return
test = tests.get()
print("{}".format(test))
for i in range(10):
tests.put(i)
threads = []
for i in range(5):
threads.append(threading.Thread(target=start_thread))
threads[i].daemon = True
for thread in threads:
thread.start()
tests.join()
When run it just prints the values and never exits.
How do I make the program exit when the Queue is empty?
From the docstring of Queue.join():
Blocks until all items in the Queue have been gotten and processed.
The count of unfinished tasks goes up whenever an item is added to the
queue. The count goes down whenever a consumer thread calls task_done()
to indicate the item was retrieved and all work on it is complete.
When the count of unfinished tasks drops to zero, join() unblocks.
So you must call tests.task_done() after processing the item.
Since your threads are daemon threads, and the queue will handle concurrent access correctly, you don't need to check if the queue is empty or use a lock. You can just do:
def start_thread():
while True:
test = tests.get()
print("{}".format(test))
tests.task_done()
My problem is the following: I have a multiprocessing.pool.ThreadPool object with worker_count workers and a main pqueue from which I feed tasks to the pool.
The flow is as follows: There is a main loop that gets an item of level level from pqueue and submits it tot the pool using apply_async. When the item is processed, it generates items of level + 1. The problem is that the pool accepts all tasks and processes them in the order they were submitted.
More precisely, what is happening is that the level 0 items are processed and each generates 100 level 1 items that are retrieved immediately from pqueue and added to the pool, each level 1 item produces 100 level 2 items that are submitted to the pool, and so on, and the items are processed in an BFS manner.
I need to tell the pool to not accept more than worker_count items in order to give a chance of higher level to be retrieved from pqueue in order to process items in a DFS manner.
The current solution I came with is: for each submitted task, save the AsyncResult object in a asyncres_list list, and before retrieving items from pqueue I remove the items that were processed (if any), check if the length of the asyncres_list is lower than the number of threads in the pool every 0.5 seconds, and like that only thread_number items will be processed at the same time.
I am wondering if there is a cleaner way to achieve this behaviour and I can't seem to find in the documentation some parameters to limit the maximum number of tasks that can be submitted to a pool.
ThreadPool is a simple tool for a common task. If you want to manage the queue yourself, to get DFS behavior; you could implement the necessary functionality on top threading and queue modules directly.
To prevent scheduling the next root task until all tasks spawned by the current task are done ("DFS"-like order), you could use Queue.join():
#!/usr/bin/env python3
import queue
import random
import threading
import time
def worker(q, multiplicity=5, maxlevel=3, lock=threading.Lock()):
for task in iter(q.get, None): # blocking get until None is received
try:
if len(task) < maxlevel:
for i in range(multiplicity):
q.put(task + str(i)) # schedule the next level
time.sleep(random.random()) # emulate some work
with lock:
print(task)
finally:
q.task_done()
worker_count = 2
q = queue.LifoQueue()
threads = [threading.Thread(target=worker, args=[q], daemon=True)
for _ in range(worker_count)]
for t in threads:
t.start()
for task in "01234": # populate the first level
q.put(task)
q.join() # block until all spawned tasks are done
for _ in threads: # signal workers to quit
q.put(None)
for t in threads: # wait until workers exit
t.join()
The code example is derived from the example in the queue module documentation.
The task at each level spawns multiplicity direct child tasks that spawn their own subtasks until maxlevel is reached.
None is used to signal the workers that they should quit. t.join() is used to wait until threads exit gracefully. If the main thread is interrupted for any reason then the daemon threads are killed unless there are other non-daemon threads (you might want to provide SIGINT hanlder, to signal the workers to exit gracefully on Ctrl+C instead of just dying).
queue.LifoQueue() is used, to get "Last In First Out" order (it is approximate due to multiple threads).
The maxsize is not set because otherwise the workers may deadlock--you have to put the task somewhere anyway. worker_count background threads are running regardless of the task queue.
In Python while using multiprocessing module there are 2 kinds of queues:
Queue
JoinableQueue.
What is the difference between them?
Queue
from multiprocessing import Queue
q = Queue()
q.put(item) # Put an item on the queue
item = q.get() # Get an item from the queue
JoinableQueue
from multiprocessing import JoinableQueue
q = JoinableQueue()
q.task_done() # Signal task completion
q.join() # Wait for completion
JoinableQueue has methods join() and task_done(), which Queue hasn't.
class multiprocessing.Queue( [maxsize] )
Returns a process shared queue implemented using a pipe and a few locks/semaphores. When a process first puts an item on the queue a feeder thread is started which transfers objects from a buffer into the pipe.
The usual Queue.Empty and Queue.Full exceptions from the standard library’s Queue module are raised to signal timeouts.
Queue implements all the methods of Queue.Queue except for task_done() and join().
class multiprocessing.JoinableQueue( [maxsize] )
JoinableQueue, a Queue subclass, is a queue which additionally has task_done() and join() methods.
task_done()
Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.
If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).
Raises a ValueError if called more times than there were items placed in the queue.
join()
Block until all items in the queue have been gotten and processed.
The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate that the item was retrieved and all work on it is complete. When the count of unfinished tasks drops to zero, join() unblocks.
If you use JoinableQueue then you must call JoinableQueue.task_done() for each task removed from the queue or else the semaphore used to count the number of unfinished tasks may eventually overflow, raising an exception.
Based on the documentation, it's hard to be sure that Queue is actually empty. With JoinableQueue you can wait for the queue to empty by calling q.join(). In cases where you want to complete work in distinct batches where you do something discrete at the end of each batch, this could be helpful.
For example, perhaps you process 1000 items at a time through the queue, then send a push notification to a user that you've completed another batch. This would be challenging to implement with a normal Queue.
It might look something like:
import multiprocessing as mp
BATCH_SIZE = 1000
STOP_VALUE = 'STOP'
def consume(q):
for item in iter(q.get, STOP_VALUE):
try:
process(item)
# Be very defensive about errors since they can corrupt pipes.
except Exception as e:
logger.error(e)
finally:
q.task_done()
q = mp.JoinableQueue()
with mp.Pool() as pool:
# Pull items off queue as fast as we can whenever they're ready.
for _ in range(mp.cpu_count()):
pool.apply_async(consume, q)
for i in range(0, len(URLS), BATCH_SIZE):
# Put `BATCH_SIZE` items in queue asynchronously.
pool.map_async(expensive_func, URLS[i:i+BATCH_SIZE], callback=q.put)
# Wait for the queue to empty.
q.join()
notify_users()
# Stop the consumers so we can exit cleanly.
for _ in range(mp.cpu_count()):
q.put(STOP_VALUE)
NB: I haven't actually run this code. If you pull items off the queue faster than you put them on, you might finish early. In that case this code sends an update AT LEAST every 1000 items, and maybe more often. For progress updates, that's probably ok. If it's important to be exactly 1000, you could use an mp.Value('i', 0) and check that it's 1000 whenever your join releases.
I have a process which spawns 2 types of thread classes. One thread is responsible for consuming a job_queue (100Threads of this class are usually running). And second thread is a killing thread. I am using a result_done flag which is being set from thread2 and the issue is my threads1 wait for X seconds and then check if result_done flag is set.
def run(self):
while True:
try:
val = self.job_queue.get(True,self.maxtimeout)
except:
pass
if self.result_done.isset():
return
Now, if maxtimeout is set to 500seconds and I set the result_done flag from another thread, this thread will wait for 500 seconds before exiting( if there's no data in the queue).
What I want to achieve is that all the threads die gracefully along with the current process, properly terminating the db,websocket,http connections etc as soon as result_done event is set from any of the threads of the process.
I am using python multiprocess library to create the process which spawns these threads.
Update: All threads are daemon=True threads.
To avoid waiting maxtimeout time, you could use Event.wait() method:
def run(self):
while not self.result_done.is_set():
try:
val = self.job_queue.get_nowait()
except Empty:
if self.result_done.wait(1): # wait a second to avoid busy loop
return
Event.wait(timeout) returns without waiting the full timeout if the event is set during the call.