In my development, I have shared file which need to write mutliple process but I found it write only header part(only first line i.e test_data). I am able to create same behavior using following script. Any Input what am I missing?
#! /usr/bin/env python
import multiprocessing
def check_proc(procs):
"""checking cpu utilization befor starting process"""
for proc in procs:
proc.start()
while(len(multiprocessing.active_children())>10):
continue
for proc in procs:
proc.join()
while (len(multiprocessing.active_children())>5):
time.sleep(1)
def create_file():
f_data = open("temp", "w")
f_data.write("test data\n")
return f_data
def write_data(f_data, number, lock):
lock.acquire()
f_data.write(number)
lock.release()
def main():
f_data = create_file()
procs = []
lock = multiprocessing.Lock()
for i in range(0, 20, 1):
proc = multiprocessing.Process(target=write_data,
args=(f_data, str(i), lock))
procs.append(proc)
check_proc(procs)
if __name__ == "__main__":
main()
Expected:
it should have number
Python2.7 and linux enviroment
I have gone through stack overflow question and block and found following link. But I am not able to relate or use in above code. Any input.
http://ptspts.blogspot.com/2013/08/how-to-send-unix-file-descriptors.html
Related
I'm wondering if there can be a sort of deadlock in the following code. I have to read each element of a database (about 1 million items), process it, then collect the results in a unique file.
I've parallelized the execution with multiprocessing using two Queue's and three types of processes:
Reader: Main process which reads the database and adds the read items in a task_queue
Worker: Pool of processes. Each worker gets an item from task_queue, processes the item, saves the results in an intermediate file stored in item_name/item_name.txt and puts the item_name in a completed_queue
Writer: Process which gets an item_name from completed_queue, gets the intermediate result from item_name/item_name.txt and writes it in results.txt
from multiprocessing import Pool, Process, Queue
class Computation():
def __init__(self,K):
self.task_queue = Queue()
self.completed_queue = Queue()
self.n_cpus = K
def reader(self,):
with open(db, "r") as db:
... # Read an item
self.task_queue.put(item)
def worker(self,):
while True:
item = self.task_queue.get(True)
if item == "STOP":
break
self.process_item(item)
def writer_process(self,):
while True:
f = self.completed_queue.get(True)
if f == "DONE":
break
self.write_f(f)
def run(self,):
pool = Pool(n_cpus, self.worker, args=())
writer = Process(target=self.writer_process, args=())
writer.start()
self.reader()
pool.close()
pool.join()
self.completed_queue.put("DONE")
writer.join()
The code works, but it seems that sometimes the writer or the pool stops working (or they are very slow). Is a deadlock possible in this scenario?
There are a couple of issues with your code. First, by using the queues as you are, you are in effect creating your own process pool and have no need for using the multiprocessing.Pool class at all. You are using a pool initializer as an actual pool worker and it's a bit of a misuse of this class; you would be better off to just use regular Process instances (my opinion, anyway).
Second, although it is well and good that you are putting message DONE to the writer_process to signal it to terminate, you have not done similarly for the self.n_cpus worker processes, which are looking for 'STOP' messages, and therefore the reader function needs to put self.n_cpus STOP messages in the task queue:
from multiprocessing import Process, Queue
class Computation():
def __init__(self, K):
self.task_queue = Queue()
self.completed_queue = Queue()
self.n_cpus = K
def reader(self,):
with open(db, "r") as db:
... # Read an item
self.task_queue.put(item)
# signal to the worker processes to terminate:
for _ in range(self.n_cpus):
self.task_queue.put('STOP')
def worker(self,):
while True:
item = self.task_queue.get(True)
if item == "STOP":
break
self.process_item(item)
def writer_process(self,):
while True:
f = self.completed_queue.get(True)
if f == "DONE":
break
self.write_f(f)
def run(self):
processes = [Process(target=self.worker) for _ in range(self.n_cpus)]
for p in processes:
p.start()
writer = Process(target=self.writer_process, args=())
writer.start()
self.reader()
for p in processes:
p.join()
self.completed_queue.put("DONE")
writer.join()
Personally, instead of using 'STOP' and 'DONE' as the sentinel messages, I would use None instead, assuming that is not a valid actual message. I have tested the above code where reader just processed strings in a list and self.process_item(item) simply appended ' done' to the each of those strings and put the modified string on the completed_queue and replaced self.write_f in the writer_process with a print call. I did not see any problems with the code as is.
Update to use a Managed Queue
Disclaimer: I have had no experience using mpi4py and have no idea how the queue proxies would get distributed across different computers. The above code may not be sufficient as suggested by the following article, How to share mutliprocessing queue object between multiple computers. However, that code is creating instances of Queue.Queue (that code is Python 2 code) and not the proxies that are returned by the multiprocessing.SyncManager. The documentation on this is very poor. Try the above change to see if it works better (it will be slower).
Because the proxy returned by manager.Queue(), I have had to rearrange the code a bit; the queues are now being passed explicitly as arguments to the process functions:
from multiprocessing import Process, Manager
class Computation():
def __init__(self, K):
self.n_cpus = K
def reader(self, task_queue):
with open(db, "r") as db:
... # Read an item
# signal to the worker processes to terminate:
for _ in range(self.n_cpus):
task_queue.put('STOP')
def worker(self, task_queue, completed_queue):
while True:
item = task_queue.get(True)
if item == "STOP":
break
self.process_item(item)
def writer_process(self, completed_queue):
while True:
f = completed_queue.get(True)
if f == "DONE":
break
self.write_f(f)
def run(self):
with Manager() as manager:
task_queue = manager.Queue()
completed_queue = manager.Queue()
processes = [Process(target=self.worker, args=(task_queue, completed_queue)) for _ in range(self.n_cpus)]
for p in processes:
p.start()
writer = Process(target=self.writer_process, args=(completed_queue,))
writer.start()
self.reader(task_queue)
for p in processes:
p.join()
completed_queue.put("DONE")
writer.join()
I learned that AWS Lambda does not support multiprocessing.Pool and multiprocessing.Queue from this other question.
I'm also working on Python multiprocessing in AWS Lambda. But my question: how do we terminate the main process when the first child process returns? (all child processes will return with different execution time)
What I have here:
import time
from multiprocessing import Process, Pipe
class run_func():
number = 0
def __init__(self, number):
self.number = number
def subrun(self, input, conn):
# subprocess function with different execution time based on input.
response = subprocess(input)
conn.send([input, response])
conn.close()
def run(self):
number = self.number
processes = []
parent_connections = []
for i in range(0, number):
parent_conn, child_conn = Pipe()
parent_connections.append(parent_conn)
process = Process(target=self.subrun, args=(i, child_conn,))
processes.append(process)
for process in processes:
process.start()
for process in processes:
process.join()
results = []
for parent_connection in parent_connections:
resp = parent_connection.recv()
print(resp)
results.append((resp[0],resp[1]))
return results
def lambda_handler(event, context):
starttime = time.time()
results = []
work = run_func(int(event['number']))
results = work.run()
print("Results : {}".format(results))
print('Time: {} seconds'.format(time.time() - starttime))
return output
The current program will return until all child processes finish (with for parent_connection in parent_connections). But I wonder how to terminate with the first child process finish? (terminate the main at least, other child processes - it's ok to leave it running)
Added:
To be clear, I mean the first returned child process (may not be the first created child).
So the join() loop is the one which waits for all child process to complete.
If we break that after completing the first child and terminate all other process forcefully it'll work for you
class run_func():
number = 0
def __init__(self, number):
self.number = number
def subrun(self, input, conn):
# subprocess function with different execution time based on input.
response = subprocess(input)
conn.send([input, response])
conn.close()
def run(self):
number = self.number
processes = []
parent_connections = []
for i in range(0, number):
parent_conn, child_conn = Pipe()
parent_connections.append(parent_conn)
process = Process(target=self.subrun, args=(i, child_conn,))
processes.append(process)
for process in processes:
process.start()
for process in processes:
process.join()
break
results = []
for parent_connection in parent_connections:
resp = parent_connection.recv()
print(resp)
I have my demo code shown as below. I realize that all subprocesses have finished but they do not exit. Is there anything wrong with my code? Python version: 3.7.4, Operation system: win10
import multiprocessing as mp
res_queue = mp.Queue()
def runCalculation(i):
count_list = []
total_count = i
for k in range(100000):
total_count += k
count_list.append(total_count)
print('task {} finished calculation, putting results to queue'.format(i))
for item in count_list: res_queue.put(item)
print('task {} has put all results to queue'.format(i))
def initPool(res_queue_):
global res_queue
res_queue = res_queue_
def mainFunc():
p = mp.Pool(initializer=initPool, initargs=(res_queue,))
for i in range(20): p.apply_async(runCalculation, args=(i,))
print('Waiting for all subprocesses done...')
p.close()
p.join()
print('All subprocesses done.')
if __name__ == '__main__':
mainFunc()
I wrote a python script to test the multi-process pool and use apply_async to call the class method. But why does the same process (same pid) output multiple times in the output?
OS: centos-7.4
PYTHON: python-2.7
#!/usr/bin/env python
import time
import os
from multiprocessing import Pool
class New(object):
def __init__(self):
self.pid = os.getpid()
def gen(self, num):
pid = os.getpid()
print 'NEW PROCESS PID IS {}'.format(pid)
return (pid, num)
def log(self, pid):
print 'START WRITE {} INTO FILE'.format(pid[0])
with open('log', 'a') as f:
f.write('CURRENT PROCESS IS {} <--> NUM IS {}\n'.format(pid[0], pid[1]))
def start(self):
print 'CREATE MAIN PROCESS {}'.format(self.pid)
self.pool = Pool()
num = 0
while True:
narg = num
self.pool.apply_async(self, args=(narg,), callback=self.log)
num += 1
time.sleep(2)
self.pool.close()
self.pool.join()
def __call__(self, num):
return self.gen(num)
def __getstate__(self):
self_dict = self.__dict__.copy()
del self_dict['pool']
return self_dict
def __setstate__(self, state):
self.__dict__.update(state)
if __name__ == '__main__':
new = New()
new.start()
The following is the result of the script print, the same process id output twice，The specific code is below。
eg:
NEW PROCESS PID IS 14459
START WRITE 14459 INTO FILE
NEW PROCESS PID IS 14459
START WRITE 14459 INTO FILE
callback of apply_async will write some lines into file.
The output at the same time is as follows
eg:
CURRENT PROCESS IS 14459 <--> NUM IS 29
CURRENT PROCESS IS 14459 <--> NUM IS 30
I just want to get one print and write for one process.
The behavior you're observing is expected. The point of using a multiprocessing.Pool() is to distribute the work over a pool of workers (i.e., processes). See multiprocessing.Pool with maxtasksperchild produces equal PIDs for one way to achieve what you want. But honestly, it seems to me you should just be using multiprocessing.Process() if you want to spawn a new process for each iteration of the inner loop.
Lets assume that i am starting a process in python with the following code:
from multiprocessing import Process
import time
def f(name):
print ('hello ', name)
if __name__ == '__main__':
p = Process(target=f,name = "Process-1", args=('bob',))
p.start()
Now,i want to terminate the process.I can simply do:
p.terminate()
However, i would like to terminate the process by its name.Is that possible?
To do that, you need to store a map between your process objects and their names. Using an helper function it makes your code even easier to read (IMO):
def terminate(procname):
return pmap[procname].terminate()
if __name__ == '__main__':
pmap = {}
pname = "process-1"
p = Process(target=f,name = pname, args=('bob',))
pmap[pname] = p
p.start()
Then to terminate:
terminate(pname)