I am trying to acheive multi processing in python. I might have a minimum of 500 elements in the list at least. I have a function to which each element of a list needs to be passed as an argument. Then each of this function should be executed as a seperate process using mutli processing either starting a new interpretter or however. Following is some pseudo code.
def fiction(arrayElement)
perform some operations here
arrayList[]
for eachElement in arrayList:
fiction(eachElement)
I want multiprocess the function under
for eachElement in arrayList:
So that I can use the multiple cores of my box. All the help is appreciated.
The multiprocessing module contains all sorts of basic classes which can be helpful for this:
from multiprocessing import Pool
def f(x):
return x*x
p = Pool(5)
p.map(f, [1,2,3])
And the work will be distributed among 3 processes.
This is fairly simple, but you can achieve much more using an external packages, mostly a Message-oriented middleware.
Prime examples are ActiveMQ, RabbitMQ and ZeroMQ.
RabbitMQ has a combination of good python API and simplicity. You can see here how simple it is to create a dispatcher-workers pattern, in which one process is sending the workload, and other processes preform it.
ZeroMQ is a bit more low-level, but is very lightweight and does not require an extenal broker.
Related
Am a beginner in python threading.... I want to create a program that has multiple threads waiting in the background, and at some point, execute function f(x) asynchronously. f(x) really takes a lot of time to compute (it computes gradients)..
I plan to run the program for several steps (i.e. for 100 steps), and each step has several values for x (i.e. 10 values), but I want to compute f(x) for all 10 values in a parallel manner to save time..
I looked at the multiprocessing python module but I need help on how to implement the threads and processes..
It's as easy as a python import:
from multiprocessing import Pool
pool = Pool(5)
pool.map(f, [<list of inputs>])
Now if your asynchronous functions will need to save it's computational result into the same place, it will be a little bit trickier:
from multiprocessing import Pool, Manager
l = Manager.list()
def func(l, *args, **kwargs): # you need to use the manager list as it's multiprocess safe
blah blah
pool = Pool(5)
pool.map(func, [<list of inputs>])
# result will now be stored in l.
And there you go.
If you want to run a script that fires off parallel tasks and manages a pool of processes, you will want to use multiprocessing.Pool.
However, it's not clear what your platform is; you could look into something like celery to handle queues for you (or AWS Lambda for potentially larger-scale work that would benefit from third party infrastructure management).
I am wondering how to extract outputs from a multiprocessed function in Python. I am new to multiprocessing and have limited understanding of how it all works (not for lack of trying though).
I need to run the optimization with 31 different inputs for InfForecast and InitialStorage (for now... could be up to 10,000 inputs and independent optimizations being performed). I was hoping I could speed things up using multiprocessing to process more than one of these independent optimizations at a time. What I want is for the outputs (5 values for each optimization) to be put into the array "Nextday" which should have dimensions of (5,31). It seems the output Nextday as I've got the code written is either empty or not accessible. How do I extract/access the values and place them into Nextday?
Note: The function main(...) is a highly complex optimization problem. I hope the problem is easy enough to understand without providing it. It works when I loop over it and call it for each i in range(31).
from multiprocessing.pool import ThreadPool as Pool
Nextday=np.zeros((5,31))
pool_size = 4 # Should I set this to the number of cores my machine has?
pool = Pool(pool_size)
def optimizer(InfForecast, InitialStorage):
O=main(InfForecast,InitialStorage)
return [O[0][0], O[0][1], O[0][2], O[0][3], O[0][4]]
for i in range(31):
pool.apply_async(optimizer, (InfForecast[i],InitialStorage[i]))
pool.close()
Nextday=pool.join()
In addition to this, I'm not sure whether this is the best way to do things. If it's working (which I'm not sure it is) it sure seems slow. I was reading that it may be better to do multiprocessing vs threading and this seems to be threading? Forgive me if I'm wrong.
I am also curious about how to select pool_size as you can see in my comment in the code. I may be running this on a cloud server eventually, so I expect the pool_size I would want to use there would be slightly different than the number I will be using on my own machine. Is it just the number of cores?
Any advice would be appreciated.
You should use
from multiprocessing.pool import Pool
if you want to do multiprocessing.
Pool size should start out as multiprocessing.cpu_count() if you have the machine to yourself, and adjusted manually for best effect. If your processes are cpu-bound, then leaving a core available will make your machine more responsive -- if your code is not cpu-bound you can have more processes than cores (tuning this is finicky though, you'll just have to try).
You shouldn't have any code at the top-most level in your file when doing multiprocessing (or any other time really). Put everything in functions and call the start function from:
if __name__ == "__main__":
my_start_function()
(digression: using capital oh as a variable name is really bad, and you get statements that are almost unreadable in certain fonts like O[0][0]).
In regular python, the map function is "defined" by this equality:
map(fn, lst) == [fn(item) for item in lst]
so the Pool methods (imap/imap_unordered/map/map_async) has similar semantics, and in your case you would call them like:
def my_start_function():
...
results = pool.map(optimizer, zip(InfForecast, InitialStorage))
Since the map-functions take a function and a list, I've used the zip function to creates a list where each item has one element from each of its arguments (it functions as like a zipper).
I have a program, which opens a lot of urls and downloads pictures .
I have a function of the program, which manages link's opening and pictures downloading, which contains a for loop and performs some operations on the priority queue. I want to run this function, but no longer than the set time period. For example if this function is running longer than 1 hour I want to terminate it and run the rest of the program (other functions).
I was trying to find some solutions, and I found two question here on stack.
The first solution use only time module First solution
The second use also the multiprocessing module
Second solution. Can some one suggest which one will be more appropriate to use in my program? I will write a pseudocode of my function:
def fun():
for link in linkList:
if link not in queue:
queue.push(link)
else:
queue.updatePriority(link)
if queue:
top = queue.pop()
fun(top)
This function is called in other function:
def run(startLink):
fun(startLink)
And the run() function is called in other module.
Which method is better to use with a program which contains a lot of modules and performs a lot of
The asyncio module is ideal for this task.
You can create a future, then use asyncio.wait which supports a timeout parameter.
Using multiprocessing here would be a little bit tricky, because fun is consuming a priority queue (I'm assuming a Queue.PriorityQueue) that is coming from some other part of the program. That queue cannot easily be passed between processes - you would need to create a custom multiprocessing.BaseManager subclass, register the Queue.PriorityQueue class with it, and start up the Manager server, instantiate a PriorityQueue on the server, and use a Proxy to that instance everywhere you interact with the queue. That's a lot of overhead, and also hurts performance a bit.
Since it appears you don't actually want any concurrency here - you want the rest of the program to stop while fun is running - I don't think there's a compelling reason to use multiprocessing. Instead, I think using the time-based solution makes more sense.
The next code works to execute "someprogram" in parallel for the 50 inputfilenames, using a pool of 5 workers. How is it possible to use 5 workers on
this computer, and 7 on another computer, using only the standard libraries like multiprocessing. Ideally i would have a list tuples (hostname, amountofworkers), which can be used to speed up something, and maybe turn it into a decorator so that it can be reused more easily on methods like the "commandlinestuff". (Using linux and python 2.7)
import multiprocessing
import subprocess
def commandlinestuff(inputfilename):
p = subprocess.Popen("someprogram "+inputfilename, shell=True)
p.wait()
inputfilenames = ["something"+str(i).zfill(2)+".in" for i in range(50)]
p = multiprocessing.pool(5)
p.map(commandlinestuff, inputfilenames)
It sounds like you are trying to re-invent pyro which is itself written in pure python but is not currently a part of the standard libraries.
Basically you need a server running on the remote machine(s) that accepts a connection, receives a pickled object to execute, (and possibly some data), executes it and posts back a result. You will also need a client on the local machine that does the posts, gathers the results and possibly does some load balancing.
The Parallel Processing entry in the python wiki gives a long list of tools to do this sort of thing with various advantages and disadvantages.
https://code.google.com/p/distributed-python-for-scripting/ did the trick for me - nothing to install, and seems to be the shortest/easiest way possible to do multiprocessing in python
Lets say I have a big list of music of varying length that needs to be converted or images of varying sizes that need to be resized or something like that. The order doesn't matter so it is perfect for splitting across multiple processors.
If I use multiprocessing.Pool's map function it seems like all the work is divided up ahead of time and doesn't take into account the fact that some files may take longer to do that others.
What happens is that if I have 12 processors... near the end of processing, 1 or 2 processors will have 2 or 3 files left to process while other processors that could be utilized sit idle.
Is there some sort of queue implementation that can keep all processors loaded until there is no more work left to do?
There is a Queue class within the multiprocessing module specifically for this purpose.
Edit: If you are looking for a complete framework for parallel computing which features a map() function using a task queue, have a look at the parallel computing facilities of IPython. In particlar, you can use the TaskClient.map() function to get a load-balanced mapping to the available processors.
This is trivial to do with jug:
def process_image(img):
....
images = glob('*.jpg')
for im in images:
Task(process_image, im)
Now, just run jug execute a few times to spawn worker processes.
About queue implementations. There are some.
Look at the Celery project. http://celeryproject.org/
So, in your case, you can run 12 conversions (one on each CPU) as Celery tasks, add a callback function (to the conversion or to the task) and in that callback function add a new conversion task running when one of the previous conversions is finished.
The Python threading library that has brought me most joy is Parallel Python (PP). It is trivial with PP to use a thread pool approach with a single queue to achieve what you need.
This is not the case if you use Pool.imap_unordered.