We Keep Coding

object is not iterable with multiprocessing?

I keep getting Type errors on list not being callable, although the I receive the print of output on my terminal... what is calling the list if we are on a loop?
def work(page):
#-------------------------
#make obj of page and do something
grabthis = Some_class1(page)
f = Someclass_2(grabthis,page)
output = f.extract()
print(output)
pages='PDFPAGES'
#set page
save = []
for page in pages:
go = work(page)
start = multiprocessing.Process(target=go)
start.start()
save.append(start)
if go == 'norun':
continue
for items in save:
start.join()
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
TypeError: 'list' object is not callable
what is the correct way to iterate a bunch of files through multiprocessing or threading?

See the comment posted by Michael Butscher as it is most likely that your intention is to have function work process pages either concurrently (multithreading) or in parallel (multiprocessing). The difference is that with multithreading each thread must acquire the Global Interpreter Lock (GIL) before it can run Python code so no two threads will ever be executing Python code at the same time (in parallel). This is not too much of an issue if work is (1) mostly I/O bound and releases the GIL when it is waiting for I/O (or a network request) to complete so that most of the time it is just waiting or (2) you are executing C/C++ code (which some library modules use for their implementation) that releases the GIL. Otherwise you have CPU-intensive processing to do in which case multiprocessing is the way to go. However, multiprocessing has has additional overhead that serial processing does not, i.e. the creation of processes and moving data across processes (different address spaces). So unless work is significantly CPU-intensive, a multiprocessing solution will run more slowly than a serial one.
Let's assume work is such that multiprocessing is the correct approach. Assuming that in addition to its high CPU requirements there is also a fair amount of waiting involved. Then it might be advantageous to create more processes than you have CPU cores since the processes will from time to time go in a wait state and allow other processes to run. But if there is little or no I/O processing involved, you gain nothing by creating more processes than you have CPU cores. Let's assume the latter and let N be the number of CPU cores you have and M is the number of pages that you have to process. If M is <= N, then you could create a process for each page as you are doing since you do not seem to be returning a value back from work (but using multiprocessing pool is probably simpler):
from multiprocessing import Process
def work(page):
#-------------------------
#make obj of page and do something
grabthis = Some_class1(page)
f = Someclass_2(grabthis,page)
output = f.extract()
print(output)
pages='PDFPAGES'
# Required for Windows or any platform that uses the *spawn* method to
# create new processes:
if __name__ == '__main__':
processes = []
for page in pages:
p = multiprocessing.Process(target=go, args=(page,))
p.start()
processes.append(p)
for p in processes:
p.join()
But if M > P, i.e. you have more pages to process than CPU cores you have, or if work needs to return a result back to the main process, I would then use a multiprocessing pool, which is also suitable even if M < P:
from multiprocessing import Pool, cpu_count
def work(page):
#-------------------------
#make obj of page and do something
grabthis = Some_class1(page)
f = Someclass_2(grabthis,page)
output = f.extract()
print(output)
pages='PDFPAGES'
# Required for Windows or any platform that uses the *spawn* method to
# create new processes:
if __name__ == '__main__':
# This will create a pool whose size is never more than the number of
# CPU cores you have or the number of pages you have to process:
pool_size = min(cpu_count(), len(pages))
pool = Pool()
pool.map(go, pages) # or results = pool.map(go, pages) if `work` returns something
# Cleanup pool:
pool.close()
pool.join()
But if work is mostly I/O bound, then use a multithreading pool. The pool size can be quite large, but you should still keep it to a reasonable size (200?):
from multiprocessing.pool import ThreadPool
def work(page):
#-------------------------
#make obj of page and do something
grabthis = Some_class1(page)
f = Someclass_2(grabthis,page)
output = f.extract()
print(output)
pages='PDFPAGES'
if __name__ == '__main__':
# This will create a pool whose size is never more than 200 or
# the number of pages you have to process:
pool_size = min(200, len(pages))
pool = ThreadPool(pool_size)
pool.map(go, pages) # or results = pool.map(go, pages) if `work` returns something
# Cleanup pool:
pool.close()
pool.join()
Note
The above are just generalizations. But if your work function is iterating files as you say, then multithreading might be the best approach. But there is a maximum data rate your disk has so creating more threads will not help performance. Moreover, if you don't have a solid state drive, then the extra head movement caused by reading multiple files concurrently can hurt performance and two threads may run more slowly than the serial approach. You could start with a pool size of 2 and see if it improves performance and then slowly increasing the pool size. The only problem is that your operating system probably caches disk data so that when you rerun the code with a different pool size, it will run faster just due to the caching. You either need to find a way of purging the disk cache between runs or re-booting between runs.

Python Subprocess Popen Parallelization

Objective
a process (.exe) with multiple input arguments
Multiple files. For each the above mentioned process shall be executed
I want to use python to parallelize the process
I am using subprocess.Popen to create the processes and afterwards keep a maximum of N parallel processes.
For testing purposes, I want to parallelize a simple script like "cmd timeout 5".
State of work
import subprocess
count = 10
parallel = 2
processes = []
for i in range(0,count):
while (len(processes) >= parallel):
for process in processes:
if (process.poll() is None):
processes.remove(process)
break
process = subprocess.Popen(["cmd", "/c timeout 5"])
processes.append(process)
[...]
I read somewhere that a good approach for checking if a process is running would be is not None like shown in the code.
Question
I am somehow struggling to set it up correctly, especially the Popen([...]) part. In some cases, all processes are executed without considering the maximum parallel count and in other cases, it doesnt work at all.
I guess that there has to be a part where the process is closed if finished.
Thanks!

You will probably have a better time using the built-in multiprocessing module to manage the subprocesses running your tasks.
The reason I've wrapped the command in a dict is that imap_unordered (which is faster than imap but doesn't guarantee ordered execution since any worker process can grab any job – whether that's okay for you is your business problem) doesn't have a starmap alternative, so it's easier to unpack a single "job" within the callable.
import multiprocessing
import subprocess
def run_command(job):
# TODO: add other things here?
subprocess.check_call(job["command"])
def main():
with multiprocessing.Pool(2) as p:
jobs = [{"command": ["cmd", "/c timeout 5"]} for x in range(10)]
for result in p.imap_unordered(run_command, jobs):
pass
if __name__ == "__main__":
main()

What is the easiest way to make maximum cpu usage for nested for-loops?

I have code that makes unique combinations of elements. There are 6 types, and there are about 100 of each. So there are 100^6 combinations. Each combination has to be calculated, checked for relevance and then either be discarded or saved.
The relevant bit of the code looks like this:
def modconffactory():
for transmitter in totaltransmitterdict.values():
for reciever in totalrecieverdict.values():
for processor in totalprocessordict.values():
for holoarray in totalholoarraydict.values():
for databus in totaldatabusdict.values():
for multiplexer in totalmultiplexerdict.values():
newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
data_I_need = dosomethingwith(newconfiguration)
saveforlateruse_if_useful(data_I_need)
Now this takes a long time and that is fine, but now I realize this process (making the configurations and then calculations for later use) is only using 1 of my 8 processor cores at a time.
I've been reading up about multithreading and multiprocessing, but I only see examples of different processes, not how to multithread one process. In my code I call two functions: 'dosomethingwith()' and 'saveforlateruse_if_useful()'. I could make those into separate processes and have those run concurrently to the for-loops, right?
But what about the for-loops themselves? Can I speed up that one process? Because that is where the time consumption is. (<-- This is my main question)
Is there a cheat? for instance compiling to C and then the os multithreads automatically?

I only see examples of different processes, not how to multithread one process
There is multithreading in Python, but it is very ineffective because of GIL (Global Interpreter Lock). So if you want to use all of your processor cores, if you want concurrency, you have no other choice than use multiple processes, which can be done with multiprocessing module (well, you also could use another language without such problems)
Approximate example of multiprocessing usage for your case:
import multiprocessing
WORKERS_NUMBER = 8
def modconffactoryProcess(generator, step, offset, conn):
"""
Function to be invoked by every worker process.
generator: iterable object, the very top one of all you are iterating over,
in your case, totalrecieverdict.values()
We are passing a whole iterable object to every worker, they all will iterate
over it. To ensure they will not waste time by doing the same things
concurrently, we will assume this: each worker will process only each stepTH
item, starting with offsetTH one. step must be equal to the WORKERS_NUMBER,
and offset must be a unique number for each worker, varying from 0 to
WORKERS_NUMBER - 1
conn: a multiprocessing.Connection object, allowing the worker to communicate
with the main process
"""
for i, transmitter in enumerate(generator):
if i % step == offset:
for reciever in totalrecieverdict.values():
for processor in totalprocessordict.values():
for holoarray in totalholoarraydict.values():
for databus in totaldatabusdict.values():
for multiplexer in totalmultiplexerdict.values():
newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
data_I_need = dosomethingwith(newconfiguration)
saveforlateruse_if_useful(data_I_need)
conn.send('done')
def modconffactory():
"""
Function to launch all the worker processes and wait until they all complete
their tasks
"""
processes = []
generator = totaltransmitterdict.values()
for i in range(WORKERS_NUMBER):
conn, childConn = multiprocessing.Pipe()
process = multiprocessing.Process(target=modconffactoryProcess, args=(generator, WORKERS_NUMBER, i, childConn))
process.start()
processes.append((process, conn))
# Here we have created, started and saved to a list all the worker processes
working = True
finishedProcessesNumber = 0
try:
while working:
for process, conn in processes:
if conn.poll(): # Check if any messages have arrived from a worker
message = conn.recv()
if message == 'done':
finishedProcessesNumber += 1
if finishedProcessesNumber == WORKERS_NUMBER:
working = False
except KeyboardInterrupt:
print('Aborted')
You can adjust WORKERS_NUMBER to your needs.
Same with multiprocessing.Pool:
import multiprocessing
WORKERS_NUMBER = 8
def modconffactoryProcess(transmitter):
for reciever in totalrecieverdict.values():
for processor in totalprocessordict.values():
for holoarray in totalholoarraydict.values():
for databus in totaldatabusdict.values():
for multiplexer in totalmultiplexerdict.values():
newconfiguration = [transmitter, reciever, processor, holoarray, databus, multiplexer]
data_I_need = dosomethingwith(newconfiguration)
saveforlateruse_if_useful(data_I_need)
def modconffactory():
pool = multiprocessing.Pool(WORKERS_NUMBER)
pool.map(modconffactoryProcess, totaltransmitterdict.values())
You probably would like to use .map_async instead of .map
Both snippets do the same, but I would say in the first one you have more control over the program.
I suppose the second one is the easiest, though :)
But the first one should give you the idea of what is happening in the second one
multiprocessing docs: https://docs.python.org/3/library/multiprocessing.html

you can run your function in this way:
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
p = Pool(5)
print(p.map(f, [1, 2, 3]))
https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

multiprocessing.pool context and load balancing

I've encountered some unexpected behaviour of the python multiprocessing Pool class.
Here are my questions:
1) When does Pool creates its context, which is later used for serialization? The example below runs fine as long as the Pool object is created after the Container definition. If you swap the Pool initializations, serialization error occurs. In my production code I would like to initialize Pool way before defining the container class. Is it possible to refresh Pool "context" or to achieve this in another way.
2) Does Pool have its own load balancing mechanism and if so how does it work?
If I run a similar example on my i7 machine with the pool of 8 processes I get the following results:
- For a light evaluation function Pool favours using only one process for computation. It creates 8 processes as requested but for most of the time only one is used (I printed the pid from inside and also see this in htop).
- For a heavy evaluation function the behaviour is as expected. It uses all 8 processes equally.
3) When using Pool I always see 4 more processes that I requested (i.e. for Pool(processes=2) I see 6 new processes). What is their role?
I use Linux with Python 2.7.2
from multiprocessing import Pool
from datetime import datetime
POWER = 10
def eval_power(container):
for power in xrange(2, POWER):
container.val **= power
return container
#processes = Pool(processes=2)
class Container(object):
def __init__(self, value):
self.val = value
processes = Pool(processes=2)
if __name__ == "__main__":
cont = [Container(foo) for foo in xrange(20)]
then = datetime.now()
processes.map(eval_power, cont)
now = datetime.now()
print "Eval time:", now - then
EDIT - TO BAKURIU
1) I was afraid that that's the case.
2) I don't understand what the linux scheduler has to do with python assigning computations to processes. My situation can be ilustrated by the example below:
from multiprocessing import Pool
from os import getpid
from collections import Counter
def light_func(ind):
return getpid()
def heavy_func(ind):
for foo in xrange(1000000):
ind += foo
return getpid()
if __name__ == "__main__":
list_ = range(100)
pool = Pool(4)
l_func = pool.map(light_func, list_)
h_func = pool.map(heavy_func, list_)
print "light func:", Counter(l_func)
print "heavy func:", Counter(h_func)
On my i5 machine (4 threads) I get the following results:
light func: Counter({2967: 100})
heavy func: Counter({2969: 28, 2967: 28, 2968: 23, 2970: 21})
It seems that the situation is as I've described it. However I still don't understand why python does it this way. My guess would be that it tries to minimise communication expenses, but still the mechanism which it uses for load balancing is unknown. The documentation isn't very helpful either, the multiprocessing module is very poorly documented.
3) If I run the above code I get 4 more processes as described before. The screen comes from htop: http://i.stack.imgur.com/PldmM.png

The Pool object creates the subprocesses during the call to __init__ hence you must define Container before. By the way, I wouldn't include all the code in a single file but use a module to implement the Container and other utilities and write a small file that launches the main program.
The Pool does exactly what is described in the documentation. In particular it has no control over the scheduling of the processes hence what you see is what Linux's scheduler thinks it is right. For small computations they take so little time that the scheduler doesn't bother parallelizing them(this probably have better performances due to core affinity etc.)
Could you show this with an example and what you see in the task manager? I think they may be the processes that handle the queue inside the Pool, but I'm not sure. On my machine I can see only the main process plus the two subprocesses.
Update on point 2:
The Pool object simply puts the tasks into a queue, and the child processes get the arguments from this queue. If a process takes almost no time to execute an object, than Linux scheduler let the process execute more time(hence consuming more items from the queue). If the execution takes much time then this scheduler will change processes and thus the other child processes are also executed.
In your case a single process is consuming all items because the computation take so little time that before the other child processes are ready it has already finished all items.
As I said, Pool doesn't do anything about balancing the work of the subprocesses. It's simply a queue and a bunch of workers, the pool puts items in the queue and the processes get the items and compute the results. AFAIK the only thing that it does to control the queue is putting a certain number of tasks in a single item in the queue(see the documentation) but there is no guarantee about which process will grab which task. Everything else is left to the OS.
On my machine the results are less extreme. Two processes get about twice the number of calls than the other two for the light computation, while for the heavy one all have more or less the same number of items processed. Probably on different OSes and/or hardware we would obtain even different results.

Subprocess management with python

I have a data analysis script that takes an argument specifying the segments of the analysis to perform. I want to run up to 'n' instances of the script at a time where 'n' is the number of cores on the machine. The complication is that there are more segments of the analysis than there are cores so I want to run at most, 'n' processes at once, and one of them finishes, kick off another one. Has anyone done something like this before using the subprocess module?

I do think that multiprocessing module will help you achieve what you need.
Take look at the example technique.
import multiprocessing
def do_calculation(data):
"""
#note: you can define your calculation code
"""
return data * 2
def start_process():
print 'Starting', multiprocessing.current_process().name
if __name__ == '__main__':
analsys_jobs = list(range(10)) # could be your analysis work
print 'analsys_jobs :', analsys_jobs
pool_size = multiprocessing.cpu_count() * 2
pool = multiprocessing.Pool(processes=pool_size,
initializer=start_process,
maxtasksperchild=2, )
#maxtasksperchild = tells the pool to restart a worker process \
# after it has finished a few tasks. This can be used to avoid \
# having long-running workers consume ever more system resources
pool_outputs = pool.map(do_calculation, analsys_jobs)
#The result of the map() method is functionally equivalent to the \
# built-in map(), except that individual tasks run in parallel. \
# Since the pool is processing its inputs in parallel, close() and join()\
# can be used to synchronize the main process with the \
# task processes to ensure proper cleanup.
pool.close() # no more tasks
pool.join() # wrap up current tasks
print 'Pool :', pool_outputs
You could find good multiprocessing techniques here to start with

Use the multiprocessing module, specifically the Pool class. Pool creates a pool of processes (by default, as many processes as you have CPUs), and allows you to submit jobs to the pool which are executed on the next free process. It takes care of all the subprocess management and the details of passing data between tasks, so you can write code in a very straightforward manner. See the documentation for some examples on usage.