We Keep Coding

Timing a multiprocessing script

I've stumbled across a weird timing issue while using the multiprocessing module.
Consider the following scenario. I have functions like this:
import multiprocessing as mp
def workerfunc(x):
# timehook 3
# something with x
# timehook 4
def outer():
# do something
mygen = ... (some generator expression)
pool = mp.Pool(processes=8)
# time hook 1
result = [pool.apply(workerfunc, args=(x,)) for x in mygen]
# time hook 2
if __name__ == '__main__':
outer()
I am utilizing the time module to get an arbitrary feeling for how long my functions run. I successfully create 8 separate processes, which terminate without error. The longest time for a worker to finish is about 130 ms (measured between timehook 3 and 4).
I expected (as they are running in parallel) that the time between hook 1 and 2 will be approximately the same. Surprisingly, I get 600 ms as a result.
My machine has 32 cores and should be able to handle this easily. Can anybody give me a hint where this difference in time comes from?
Thanks!

You are using pool.apply which is blocking. Use pool.apply_async instead and then the function calls will all run in parallel, and each will return an AsyncResult object immediately. You can use this object to check when the processes are done and then retrieve the results using this object also.

Since you are using multiprocessing and not multithreading your performance issue is not related to GIL (Python's Global Interpreter Lock).
I've found an interesting link explaining this with an example, you can find it in the bottom of this answer.
The GIL does not prevent a process from running on a different
processor of a machine. It simply only allows one thread to run at
once within the interpreter.
So multiprocessing not multithreading will allow you to achieve true
concurrency.
Lets understand this all through some benchmarking because only that
will lead you to believe what is said above. And yes, that should be
the way to learn — experience it rather than just read it or
understand it. Because if you experienced something, no amount of
argument can convince you for the opposing thoughts.
import random
from threading import Thread
from multiprocessing import Process
size = 10000000 # Number of random numbers to add to list
threads = 2 # Number of threads to create
my_list = []
for i in xrange(0,threads):
my_list.append([])
def func(count, mylist):
for i in range(count):
mylist.append(random.random())
def multithreaded():
jobs = []
for i in xrange(0, threads):
thread = Thread(target=func,args=(size,my_list[i]))
jobs.append(thread)
# Start the threads
for j in jobs:
j.start()
# Ensure all of the threads have finished
for j in jobs:
j.join()
def simple():
for i in xrange(0, threads):
func(size,my_list[i])
def multiprocessed():
processes = []
for i in xrange(0, threads):
p = Process(target=func,args=(size,my_list[i]))
processes.append(p)
# Start the processes
for p in processes:
p.start()
# Ensure all processes have finished execution
for p in processes:
p.join()
if __name__ == "__main__":
multithreaded()
#simple()
#multiprocessed()
Additional information
Here you can find the source of this information and a more detailed technical explanation (bonus: there's also Guido Van Rossum quotes in it :) )

How do I convert current python program to multi prosessing/multi threading

Sorry to ask this question as a new python starter, I have a working python program to be converted into multiprocessing or multithreading, here is the working py's structure:
class XMLToJson():
def __init__(self, region=None, flow=None, path=None, output=None):
def run(self):
def run_from_cmd():
XMLToJson().run()
if __name__ == '__main__':
XMLToJson().run()
It would be greatly appreciated if anyone can tell me how to do the conversion.
Thank you very much.
P.S.
The following is the framework I am thinking how to fit into it:
from threading import Thread, current_thread, Lock
import time
def worker(l):
while True:
l.acquire()
print ('in worker:' + str(current_thread()))
l.release()
time.sleep(0.5)
if __name__ == '__main__':
l = Lock()
print ('in main: ' + str(current_thread()))
threads = [Thread(target=worker, args=[l]) for i in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
I modified the original working program from run() to main_process(), and set the target from worker to main_process,
if __name__ == '__main__':
l = Lock()
print ('in main: ' + str(current_thread()))
threads = [Thread(target=main_process, args=[l]) for i in range(5)]
for t in threads:
t.start()
for t in threads:
t.join()
but the program doesn't even pass the compile, error out in target=main_process.
Thank you very much.

Your question lacks quite a bit of specifics. What is your program doing? Why do you want to multiprocess/thread? What is your input/output? What is there to multiprocess/multithread?
If what you have is a script that does input => transform => output and terminates, multiprocessing/threading would be just a way to process several sets of input at the same time to gain time. In that case you could either call your script several times with each set of inputs, or pass the multi-inputs to a single instance of your multi-threaded script where you use e.g. binge library (pip install binge) to deal with multiprocessing:
from binge import B
result = B(worker, n=5)(....)
where worker is your transform function, n the number of times it should happen, and .... your inputs to be sent to the 5 parallel worker instances - mind that if you have n=5, then your inputs should be either length 5 (distributed over workers), or 1 (given identically to each worker).
cf: binge documentation

Simple and safe way to wait for a Python Process to complete when using a Queue

I'm using the Process class to create and manage subprocesses, which may return non-trival quantities of data. The documentation states that join() is the correct way to wait for a Process to complete (https://docs.python.org/2/library/multiprocessing.html#the-process-class).
However, when using multiprocessing.Queue this can cause a hang after joining the process, as described here: https://bugs.python.org/issue8426 and here https://docs.python.org/2/library/multiprocessing.html#multiprocessing-programming (not a bug).
These docs suggest removing p.join() - but surely this will remove the guarantee that all processes have completed, as Queue.get() only waits for a single item to become available?
How can I wait for completion of all Processes in this case, and ensure I'm collecting output from them all?
A simple example of the hang I'd like to deal with:
from multiprocessing import Process, Queue
class MyClass:
def __init__(self):
pass
def example_run(output):
output.put([MyClass() for i in range(1000)])
print("Bottom of example_run() - note hangs after this is printed")
if __name__ == '__main__':
output = Queue()
processes = [Process(target=example_run, args=(output,)) for x in range(5)]
for p in processes:
p.start()
for p in processes:
p.join()
print("Processes completed")

https://bugs.python.org/issue8426
This means that whenever you use a queue you need to make sure that
all items which have been put on the queue will eventually be removed
before the process is joined. Otherwise you cannot be sure that
processes which have put items on the queue will terminate.
In your example I just added output.get() before calling to join() and every thing worked fine. We put data in queue to be used some where, so just make sure that.
for p in processes:
p.start()
print output.get()
for p in processes:
p.join()
print("Processes completed")

An inelegant solution is to add
output_final = []
for i in range(5): # we have 5 processes
output_final.append(output.get())
before attempting to join any of the processes. This simply tries to get the appropriate number of outputs for the number of processes we've started.
Turns out a much better, wider solution is not to use Process at all; use Pool instead. This way the hassles of starting worker processes and collecting the results is handled for you:
import multiprocessing
class MyClass:
def __init__(self):
pass
def example_run(someArbitraryInput):
foo = [MyClass() for i in range(10000)]
return foo
if __name__ == '__main__':
pool = multiprocessing.Pool(processes=5)
output = pool.map(example_run, range(5))
pool.close(); pool.join() # make sure the processes are complete and tidy
print("Processes completed")

The purpose of the pool in python multiprocessing

I'm having difficulty understanding the purpose of the pool in Python's multiprocessing module.
I know what this code is doing:
import multiprocessing
def worker():
"""worker function"""
print 'Worker'
return
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=worker)
jobs.append(p)
p.start()
So my question is, in what type of situation would a pool be used?

Pool objects are useful when you want to be able to submit more tasks to sub-processes, but you don't want to handle all the organization of these tasks(i.e. how many processes should be spawned to handle them; which task go to which process etc.) and you care only for the result value, and not any other kind of synchronisation etc. You don't want to have the control over the sub-process computation but simply the result.
On the other hand Process is used when you want to execute a specific action, and you need control over the sub-process, not only on the result of its computation.

Python multiprocess profiling

I want to profile a simple multi-process Python script. I tried this code:
import multiprocessing
import cProfile
import time
def worker(num):
time.sleep(3)
print 'Worker:', num
if __name__ == '__main__':
for i in range(5):
p = multiprocessing.Process(target=worker, args=(i,))
cProfile.run('p.start()', 'prof%d.prof' %i)
I'm starting 5 processes and therefore cProfile generates 5 different files. Each log only shows what happens inside the start method. How can I get logs that profile the worker function (and show that it took approximately 3 seconds in each case)?

You're profiling the process startup, which is why you're only seeing what happens in p.start() as you say—and p.start() returns once the subprocess is kicked off. You need to profile inside the worker method, which will get called in the subprocesses.

It's not cool enough having to change your source code for profiling. Let's see what your code is supposed to be like:
import multiprocessing
import time
def worker(num):
time.sleep(3)
print('Worker:', num)
if __name__ == '__main__':
processes = []
for i in range(5):
p = multiprocessing.Process(target=worker, args=(i,))
p.start()
processes.append(p)
for p in processes:
p.join()
I added join here so your main process will wait for your workers before quitting.
Instead of cProfile, try viztracer.
Install it by pip install viztracer. Then use the multiprocess feature
viztracer --log_multiprocess your_script.py
It will generate an html file showing every process on a timeline. (use AWSD to zoom/navigate)
Of course this includes some info that you are not interested in(like the structure of the actual multiprocessing library). If you are already satisfied with this, you are good to go. However, if you want a clearer graph for only your function worker(). Try log_sparse feature.
First, decorate the function you want to log with #log_sparse
from viztracer import log_sparse
#log_sparse
def worker(num):
time.sleep(3)
print('Worker:', num)
Then run viztracer --log_multiprocess --log_sparse your_script.py
Only your worker function, taking 3s, will be displayed on the timeline.

If you have a complex processes structure and you want to profile a particular portion of the code, or maybe the particular working core of the process you can point to the profiler to collect stats there (see enable and disable methods https://docs.python.org/3.6/library/profile.html#module-cProfile). This is what you can do:
import cProfile
def my_particular_worker_code()
pr = cProfile.Profile()
pr.enable()
# Process stuff to be profiled
pr.disable()
pr.print_stats(sort='tottime') # sort as you wish
You can drop the reports to a file as well.