I am a php developer trying to wing a python multiprocessing script. It is creating a queue that needs to be 40,000+ items long. The queue fills fine, but my processes die silently and randomly. Tried in 2.6 and 2.7.
I have put logging throughout to try to diagnose what is happening, to no avail. On random entries with no discernible data problems, all child processes will stop and the main thread will exit? Note: it is the SAME entries that it fails on, regardless of start point, unless it is under 20 entries being read.
Sometimes it will do 150 entries, sometimes 50 before it quits. Since I need it to do 40k, this is a deal-breaker. The script has no problem doing 20 or so, and logs the proper process starting and exiting messages. When it fails, there is not process exit logged.
My code is too sensitive to post, but here is the basic model I used, taken from a tutorial on multithreading and converted to the mutliprocessing module. Basically, just replaced "threading" with "multiprocessing" and used multiprocessing's Queue instead of the module Queue.
import Queue
import threading
import time
exitFlag = 0
class myThread (threading.Thread):
def __init__(self, threadID, name, q):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.q = q
def run(self):
print "Starting " + self.name
process_data(self.name, self.q)
print "Exiting " + self.name
def process_data(threadName, q):
while not exitFlag:
queueLock.acquire()
if not workQueue.empty():
data = q.get()
queueLock.release()
print "%s processing %s" % (threadName, data)
else:
queueLock.release()
time.sleep(1)
threadList = ["Thread-1", "Thread-2", "Thread-3"]
nameList = ["One", "Two", "Three", "Four", "Five"]
queueLock = threading.Lock()
workQueue = Queue.Queue(10)
threads = []
threadID = 1
# Create new threads
for tName in threadList:
thread = myThread(threadID, tName, workQueue)
thread.start()
threads.append(thread)
threadID += 1
# Fill the queue
queueLock.acquire()
for word in nameList:
workQueue.put(word)
queueLock.release()
# Wait for queue to empty
while not workQueue.empty():
pass
# Notify threads it's time to exit
exitFlag = 1
# Wait for all threads to complete
for t in threads:
t.join()
print "Exiting Main Thread"
Turns out, I installed 32-bit python on a 64-bit windows machine accidentally. Strange behavior ensued, mostly from the multiprocessing module. Uninstalling 32-bit and installing 64-bit python 2.7.10 made it work correctly.
So if nothing makes sense with how your code is executing, maybe you've got the wrong interpreter!
Related
I am building a watchdog timer that runs another Python program, and if it fails to find a check-in from any of the threads, shuts down the whole program. This is so it will, eventually, be able to take control of needed communication ports. The code for the timer is as follows:
from multiprocessing import Process, Queue
from time import sleep
from copy import deepcopy
PATH_TO_FILE = r'.\test_program.py'
WATCHDOG_TIMEOUT = 2
class Watchdog:
def __init__(self, filepath, timeout):
self.filepath = filepath
self.timeout = timeout
self.threadIdQ = Queue()
self.knownThreads = {}
def start(self):
threadIdQ = self.threadIdQ
process = Process(target = self._executeFile)
process.start()
try:
while True:
unaccountedThreads = deepcopy(self.knownThreads)
# Empty queue since last wake. Add new thread IDs to knownThreads, and account for all known thread IDs
# in queue
while not threadIdQ.empty():
threadId = threadIdQ.get()
if threadId in self.knownThreads:
unaccountedThreads.pop(threadId, None)
else:
print('New threadId < {} > discovered'.format(threadId))
self.knownThreads[threadId] = False
# If there is a known thread that is unaccounted for, then it has either hung or crashed.
# Shut everything down.
if len(unaccountedThreads) > 0:
print('The following threads are unaccounted for:\n')
for threadId in unaccountedThreads:
print(threadId)
print('\nShutting down!!!')
break
else:
print('No unaccounted threads...')
sleep(self.timeout)
# Account for any exceptions thrown in the watchdog timer itself
except:
process.terminate()
raise
process.terminate()
def _executeFile(self):
with open(self.filepath, 'r') as f:
exec(f.read(), {'wdQueue' : self.threadIdQ})
if __name__ == '__main__':
wd = Watchdog(PATH_TO_FILE, WATCHDOG_TIMEOUT)
wd.start()
I also have a small program to test the watchdog functionality
from time import sleep
from threading import Thread
from queue import SimpleQueue
Q_TO_Q_DELAY = 0.013
class QToQ:
def __init__(self, processQueue, threadQueue):
self.processQueue = processQueue
self.threadQueue = threadQueue
Thread(name='queueToQueue', target=self._run).start()
def _run(self):
pQ = self.processQueue
tQ = self.threadQueue
while True:
while not tQ.empty():
sleep(Q_TO_Q_DELAY)
pQ.put(tQ.get())
def fastThread(q):
while True:
print('Fast thread, checking in!')
q.put('fastID')
sleep(0.5)
def slowThread(q):
while True:
print('Slow thread, checking in...')
q.put('slowID')
sleep(1.5)
def hangThread(q):
print('Hanging thread, checked in')
q.put('hangID')
while True:
pass
print('Hello! I am a program that spawns threads!\n\n')
threadQ = SimpleQueue()
Thread(name='fastThread', target=fastThread, args=(threadQ,)).start()
Thread(name='slowThread', target=slowThread, args=(threadQ,)).start()
Thread(name='hangThread', target=hangThread, args=(threadQ,)).start()
QToQ(wdQueue, threadQ)
As you can see, I need to have the threads put into a queue.Queue, while a separate object slowly feeds the output of the queue.Queue into the multiprocessing queue. If instead I have the threads put directly into the multiprocessing queue, or do not have the QToQ object sleep in between puts, the multiprocessing queue will lock up, and will appear to always be empty on the watchdog side.
Now, as the multiprocessing queue is supposed to be thread and process safe, I can only assume I have messed something up in the implementation. My solution seems to work, but also feels hacky enough that I feel I should fix it.
I am using Python 3.7.2, if it matters.
I suspect that test_program.py exits.
I changed the last few lines to this:
tq = threadQ
# tq = wdQueue # option to send messages direct to WD
t1 = Thread(name='fastThread', target=fastThread, args=(tq,))
t2 = Thread(name='slowThread', target=slowThread, args=(tq,))
t3 = Thread(name='hangThread', target=hangThread, args=(tq,))
t1.start()
t2.start()
t3.start()
QToQ(wdQueue, threadQ)
print('Joining with threads...')
t1.join()
t2.join()
t3.join()
print('test_program exit')
The calls to join() means that the test program never exits all by itself since none of the threads ever exit.
So, as is, t3 hangs and the watchdog program detects this and detects the unaccounted for thread and stops the test program.
If t3 is removed from the above program, then the other two threads are well behaved and the watchdog program allows the test program to continue indefinitely.
Multithreading app on python 2.7.
I use "threading" and "thread" libraries.
Main thread had started other 10 threads, which do some work. They have one shared class with data (singletone). I don't use any thread blocking, and it's seems good.
Main thread starting this threads with "start" and "join" methods of threading class.
One of ten threads, was starting every 10 seconds and do some math calculation.
When the work is complete, the thread invoke "thread.exit()".
And sometimes main thread did not have the result of the one thread.
The thread is end! And all strings of code are complete, but main thread stops on "join" instruction and did not response.
p.s. I'm not native english speacker, and discribe that problem was very difficult. Please be tolerant.
Code example:
while True:
all_result = check_is_all_results()
time.sleep(1)
if (all_result):
print app_data.task_table
app_data.flag_of_close = True
time.sleep(2) # Задержка на всякий случай
if (app_data.flag_of_close):
terminate()
print u"TEST"
if len(app_data.ip_table[app_data.cfg.MY_IP]['tasks']):
if (app_data.cfg.MULTITHREADING or app_data.complete_task.is_set()):
job = Worker(app_data, SRV.taskResultSendToSlaves, app_data.ip_table[app_data.cfg.MY_IP]['tasks'].pop())
job.setDaemon(True)
job.start()
###########################################################
class Worker(threading.Thread):
def __init__(self, data, sender, taskname):
self.data = data
self.sender = sender
self.taskname = taskname
threading.Thread.__init__(self)
def run(self):
import thread
self.data.complete_task.clear()
tick_before = time.time()
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
p = subprocess.Popen(self.data.cfg.PATH_INTERPRETER + " " + self.data.cfg.PATH_TASKS + self.taskname, startupinfo=startupinfo, shell=False, stdout=subprocess.PIPE)
job_result, err = p.communicate()
tick_after = time.time()
work_time = tick_after - tick_before
self.data.task_table[self.taskname]['status'] = 'complete'
self.data.task_table[self.taskname]['result'] = job_result
self.data.task_table[self.taskname]['time'] = work_time
tr = threading.Thread(target=self.sender, name="SENDER", args=(self.taskname, ))
tr.setDaemon(True)
tr.start()
tr.join()
self.data.complete_task.set()
thread.exit()
Sometimes main infinite loop, which calls Worker, does not print "TEST", and does not response.
Your worker threads are spawning subprocesses. Unfortunately, this never works right, because this is first done with a fork that only copies the executing thread from the parent process. Sorry, but your program will not be reliable until you restructure it. Here is some background information with links to more information:
https://stackoverflow.com/a/32107436/3577601
Status of mixing multiprocessing and threading in Python
https://stackoverflow.com/a/6079669/3577601
I have a class which pulls items from a queue and then runs code on it. I also have code in the main function that adds items to the queue for processing.
For some reason, the program doesn't want to end properly.
Here is the code:
class Downloader(Thread):
def __init__(self, queue):
self.queue = queue
Thread.__init__(self)
def run(self):
while True:
download_file(self.queue.get())
self.queue.task_done()
def spawn_threads(Class, amount):
for t in xrange(amount):
thread = Class(queue)
thread.setDaemon = True
thread.start()
if __name__ == "__main__":
spawn_threads(Downloader, 20)
for item in items: queue.put(item)
#not the real code, but simplied because it isn't relevant
print 'Done scanning. Waiting for downloads to finish.'
queue.join()
print 'Done!'
The program waits for it to finish properly at the queue.join() and prints Done!, but something keeps the program from closing which i can't seem to put my finger on. I'd assume it was the while True loop, but i thought setting the threads as daemons was meant to solve that.
You are not using setDaemon() correctly. As a result, none of the Downloader threads are daemon threads.
Instead of
thread.setDaemon = True
write
thread.setDaemon(True)
or
thread.daemon = True
(The docs seem to imply that the latter is the preferred spelling in Python 2.6+.)
I am learning Python Threading Module right now. And since the output of each thread will point to the same target, either database or write to flatfile, I would assume using Lock.acquire and Lock.release before and after the output process of each Thread.
from threading import Thread, Lock
import time
def task(argument1, lock):
lock.acquire()
print 'processing ' + str(argument1)
lock.release()
time.sleep(5)
# Your baby code goes here
lock.acquire()
print 'finish processing' + str(argument1)
lock.release()
class Worker(Thread):
def __init__(self, argument1, lock):
Thread.__init__(self)
self.value = argument1
self.lock = lock
def run(self):
task(self.value, self.lock) # calling the routine
def main():
l = Lock()
worker1 = Worker('BAV99', l)
worker2 = Worker('ATMEGA328', l)
worker3 = Worker('LM340', l)
worker1.start()
worker2.start()
worker3.start()
if __name__ == '__main__':
You might got confused by the usage of Lock. In a short sentence, without the lock, you might get the print spread out.
WITHOUT LOCK:
processing BAV99
processing ATMEGA328
processing LM340
finish processingBAV99finish processingLM340finish processingATMEGA328
WITH LOCK:
processing BAV99
processing ATMEGA328
processing LM340
finish processingBAV99
finish processingLM340
finish processingATMEGA328
The problem is:
I have spotted that sometimes there exist empty line between the outputs like below and wondering what is really going on there, or it is just some illusion caused by eclipse.
processing BAV99
processing ATMEGA328
processing LM340
finish processingBAV99
finish processingLM340
finish processingATMEGA328
SORRY THAT THE ERROR COULD NOT BE REPEATED and that is what haunted me the most.
I have a python program that implements threads like this:
class Mythread(threading.Thread):
def __init__(self, name, q):
threading.Thread.__init__(self)
self.name = name
self.q = q
def run(self):
print "Starting %s..." % (self.name)
while True:
## Get data from queue
data = self.q.get()
## do_some_processing with data ###
process_data(data)
## Mark Queue item as done
self.q.task_done()
print "Exiting %s..." % (self.name)
def call_threaded_program():
##Setup the threads. Define threads,queue,locks
threads = []
q = Queue.Queue()
thread_count = n #some number
data_list = [] #some data list containing data
##Create Threads
for thread_id in range(1, thread_count+1):
thread_name = "Thread-" + str(thread_id)
thread = Mythread(thread_name,q)
thread.daemon = True
thread.start()
##Fill data in Queue
for data_item in data_list:
q.put(data_item)
try:
##Wait for queue to be exhausted and then exit main program
q.join()
except (KeyboardInterrupt, SystemExit) as e:
print "Interrupt Issued. Exiting Program with error state: %s"%(str(e))
exit(1)
The call_threaded_program() is called from a different program.
I have the code working under normal circumstances. However if an error/exception occurs in one of the threads, then the program is stuck (as the queue join is infinitely blocking). The only way I am able to quit this program is to close the terminal itself.
What is the best way to terminate this program when a thread bails out? Is there a clean (actually I would take any way) way of doing this? I know this question has been asked numerous times, but I am still unable to find a convincing answer. I would really appreciate any help.
EDIT:
I tried removing the join on the queue and used a global exit flag as suggested in Is there any way to kill a Thread in Python?
However, Now the behavior is so strange, I can't comprehend what is going on.
import threading
import Queue
import time
exit_flag = False
class Mythread (threading.Thread):
def __init__(self,name,q):
threading.Thread.__init__(self)
self.name = name
self.q = q
def run(self):
try:
# Start Thread
print "Starting %s...."%(self.name)
# Do Some Processing
while not exit_flag:
data = self.q.get()
print "%s processing %s"%(self.name,str(data))
self.q.task_done()
# Exit thread
print "Exiting %s..."%(self.name)
except Exception as e:
print "Exiting %s due to Error: %s"%(self.name,str(e))
def main():
global exit_flag
##Setup the threads. Define threads,queue,locks
threads = []
q = Queue.Queue()
thread_count = 20
data_list = range(1,50)
##Create Threads
for thread_id in range(1,thread_count+1):
thread_name = "Thread-" + str(thread_id)
thread = Mythread(thread_name,q)
thread.daemon = True
threads.append(thread)
thread.start()
##Fill data in Queue
for data_item in data_list:
q.put(data_item)
try:
##Wait for queue to be exhausted and then exit main program
while not q.empty():
pass
# Stop the threads
exit_flag = True
# Wait for threads to finish
print "Waiting for threads to finish..."
while threading.activeCount() > 1:
print "Active Threads:",threading.activeCount()
time.sleep(1)
pass
print "Finished Successfully"
except (KeyboardInterrupt, SystemExit) as e:
print "Interrupt Issued. Exiting Program with error state: %s"%(str(e))
if __name__ == '__main__':
main()
The program's output is as below:
#Threads get started correctly
#The output also is getting processed but then towards the end, All i see are
Active Threads: 16
Active Threads: 16
Active Threads: 16...
The program then just hangs or keeps on printing the active threads. However since the exit flag is set to True, the thread's run method is not being exercised. So I have no clue as to how these threads are kept up or what is happening.
EDIT:
I found the problem. In the above code, thread's get method were blocking and hence unable to quit. Using a get method with a timeout instead did the trick. I have the code for just the run method that I modified below
def run(self):
try:
#Start Thread
printing "Starting %s..."%(self.name)
#Do Some processing
while not exit_flag:
try:
data = self.q.get(True,self.timeout)
print "%s processing %s"%(self.name,str(data))
self.q.task_done()
except:
print "Queue Empty or Timeout Occurred. Try Again for %s"%(self.name)
# Exit thread
print "Exiting %s..."%(self.name)
except Exception as e:
print "Exiting %s due to Error: %s"%(self.name,str(e))
If you want to force all the threads to exit when the process exits, you can set the "daemon" flag of the thread to True before the thread is created.
http://docs.python.org/2/library/threading.html#threading.Thread.daemon
I did it once in C. Basically i had a main process that were starting the other ones and kept tracks of them, ie. stored the PID and waited for the return code. If you have an error in a process the code will indicate so and then you can stop every other process. Hope this helps
Edit:
Sorry i can have forgotten in my answer that you were using threads. But I think it still applies. You can either wrap or modify the thread to get a return value or you can use the multithread pool library.
how to get the return value from a thread in python?
Python thread exit code