I've got a function foo which takes a small object and a large one big_object. The large one is a constant. I'm using multiprocessing to process a list of the small objects. I want to avoid having to pickle/unpickle big_object each time foo is called.
It seems like the initialiser argument of multiprocessing.Pool would be useful for me. But I can't figure it out (memory explodes). My approach at the moment looks like:
big_object = None
def foo(small_object):
global big_object
# ... do stuff with big_object
return result
def init(big_object_arg):
global big_object
big_object = big_object_arg
def main():
[...]
with mp.Pool(4, initializer=init, initargs=(big_object,)) as pool:
lst_results = pool.map(foo, lst_small_objects)
This runs, but memory usage explodes for some reason. Why could this be happening?
big_object is a custom C++ object defined via pybind11 for which I have defined pickling functions. These are very slow though.
So, the memory blowing up was fine and actually expected. At the same time it was not actually running because I was using a custom C++ object and there was a bug in the pickling code.
In other words, the code I posted is fine. That works!
Related
I have a class with a method which modifies its internal state, for instance:
class Example():
def __init__(self, value):
self.param = value
def example_method(self, m):
self.param = self.param * m
# By convention, these methods in my implementation return the object itself
return self
I wanna run example_method in parallel (I am using the mpire lib, but other options are welcome as well), for many instances of Example, and have their internal states altered in my instances. Something like:
import mpire
list_of_instances = [Example(i) for i in range(1, 6)]
def run_method(ex):
ex.example_method(10)
print("Before parallel calls, this should print <1>}")
print(f"<{list_of_instances[0]}>")
with mpire.WorkerPool(n_jobs=3) as pool:
pool.map_unordered(run_method, [(example,) for example in list_of_instances])
print("After parallel calls, this should print <10>}")
print(f"<{list_of_instances[0]}>")
However, the way that mpire works, what is being modified are copies of example, and not the objects within list_of_instances, making any changes to internal state not being kept after the parallel processing. So the second print will print <1> instead, because that object`s internal state was not changed, a copy of it was.
I am wondering if there are any solutions to have the internal state changes be applied to the original objects in list_of_instances.
The only solutions I can think about is:
replace list_of_instances by the result of pool.map_unordered (changing to pool.map_ordered if order is important).
Since in any other case (even when using shared_objects) I have a copy of the original objects being made, resulting in the state changes being lost.
Is there any way to solve this with parallel processing? I also accept answers using other libs.
I am using the Pool class from python's multiprocessing library write a program that will run on an HPC cluster.
Here is an abstraction of what I am trying to do:
def myFunction(x):
# myObject is a global variable in this case
return myFunction2(x, myObject)
def myFunction2(x,myObject):
myObject.modify() # here I am calling some method that changes myObject
return myObject.f(x)
poolVar = Pool()
argsArray = [ARGS ARRAY GOES HERE]
output = poolVar.map(myFunction, argsArray)
The function f(x) is contained in a *.so file, i.e., it is calling a C function.
The problem I am having is that the value of the output variable is different each time I run my program (even though the function myObject.f() is a deterministic function). (If I only have one process then the output variable is the same each time I run the program.)
I have tried creating the object rather than storing it as a global variable:
def myFunction(x):
myObject = createObject()
return myFunction2(x, myObject)
However, in my program the object creation is expensive, and thus, it is a lot easier to create myObject once and then modify it each time I call myFunction2(). Thus, I would like to not have to create the object each time.
Do you have any tips? I am very new to parallel programming so I could be going about this all wrong. I decided to use the Pool class since I wanted to start with something simple. But I am willing to try a better way of doing it.
I am using the Pool class from python's multiprocessing library to do
some shared memory processing on an HPC cluster.
Processes are not threads! You cannot simply replace Thread with Process and expect all to work the same. Processes do not share memory, which means that the global variables are copied, hence their value in the original process doesn't change.
If you want to use shared memory between processes then you must use the multiprocessing's data types, such as Value, Array, or use the Manager to create shared lists etc.
In particular you might be interested in the Manager.register method, which allows the Manager to create shared custom objects(although they must be picklable).
However I'm not sure whether this will improve the performance. Since any communication between processes requires pickling, and pickling takes usually more time then simply instantiating the object.
Note that you can do some initialization of the worker processes passing the initializer and initargs argument when creating the Pool.
For example, in its simplest form, to create a global variable in the worker process:
def initializer():
global data
data = createObject()
Used as:
pool = Pool(4, initializer, ())
Then the worker functions can use the data global variable without worries.
Style note: Never use the name of a built-in for your variables/modules. In your case object is a built-in. Otherwise you'll end up with unexpected errors which may be obscure and hard to track down.
Global keyword works on the same file only. Another way is to set value dynamically in pool process initialiser, somefile.py can just be an empty file:
import importlib
def pool_process_init():
m = importlib.import_module("somefile.py")
m.my_global_var = "some value"
pool = Pool(4, initializer=pool_process_init)
How to use the var in task:
def my_coroutine():
m = importlib.import_module("somefile.py")
print(m.my_global_var)
well,i want to make my two different methods proceeding at the sametime.
i wish to accomplish by multiprocessing,but i found all of examples are proceeding the same methods with multiprocessors,not the different method with the multiprocessors.
my code is as follows,it proceeds in sequence,not out of sequence
#-*-coding:utf-8-*-
import multiprocessing
import time
#def thread_test(num1,num2):
#x=5
def haha(num1):
for i in range(num1):
time.sleep(1)
print('a')
def hehe(num2):
for i in range(num2):
time.sleep(1)
print('b')
if __name__=='__main__':
pool = multiprocessing.Pool(processes=4)
pool.apply_async(haha(5))
pool.apply_async(hehe(5))
pool.close()
pool.join()
print("done")
the print is as follows
a
a
a
a
a
b
b
b
b
b
done
I think this has been answered before but I cannot find where to duplicate this and I need a bit more than a comment to describe what is happening, so here it goes:
The problem with your code is that you are in fact not executing your functions in different processes. Instead you are trying to execute the return value of your functions haha and hehe in new processes. And since you have not defined any return value, they return None.
.apply_async (and similar functions) need to be called with the bare function name as first parameter and then the arguments as a second parameter (wraped as a tuple). This is needed due to the execution order defined by Python (and virtually all other programming languages) where the function argumnets are evaluated before the function itself. Thus, when you call a function with an argument being another function call then the inner function call is evaluated first.
The solution to this, is therefore to call the outer function, not with an inner function call, but with the bare function name (which then works as a reference to the inner function) and then the arguments to the inner function as a separate argument to the outer function. This way there is nothint to evaluate before the outer function starts to execute. For the situation at hand, the solution is thus simple. Just change your code as:
#-*-coding:utf-8-*-
import multiprocessing
import time
def haha(num1):
for i in range(num1):
time.sleep(1)
print('a')
def hehe(num2):
for i in range(num2):
time.sleep(1)
print('b')
if __name__=='__main__':
pool = multiprocessing.Pool(processes=4)
pool.apply_async(haha, (5,)) # instead of pool.apply_async(haha(5))
pool.apply_async(hehe, (5,)) # instead of pool.apply_async(hehe(5))
pool.close()
pool.join()
print("done")
I hope this explanation makes sense to you, and helps you to watch out for these situations in the future.
I can't find any reference to this question and it seems so trivial.
Is it even possible to share objects across Pythons workers from multiprocessing Pool ?
Here's a little example:
from multiprocessing import Pool
def work(a):
return do_work(obj_b)
def main(obj_a, obj_b):
my_iterable = get_iter(obj_a)
p = Pool(processes=6)
res = p.map(work, my_iterable)
Lets say get_iter(obj_a) returns an iterable object.
How does "work" know of obj_b?
After reading a lot of material I've come to realize a few things:
Python isn't commonly used for multiprocessing.
The so called "Shared Object" is probably (and I'm not sure of it, I'll be happy to stand corrected) is being fully copied by every process.
It works :>
Heres the code:
from multiprocessing import Pool, cpu_count
def work(a):
print("I'm aware of obj_b: {}".format(obj_b))
def initPoolResources(_obj_b):
# Define all your shared read obj here
global obj_b
# Initialize them
obj_b = _obj_b
def main(obj_a):
# Assume obj_a is an iterable object
# We want to create a "shared read only" object between the pool of processes.
p = Pool(processes=cpu_count()-1, initializer=initPoolResources, initargs(obj_b))
result = p.map(work, obj_a)
p.close()
p.join()
work(a) has never seen obj_b, but he's fully aware of it.
Yes it is possible, from the doc. You can create a shared object and if set globally you can do it. See this SO answer.
I am using the Pool class from python's multiprocessing library write a program that will run on an HPC cluster.
Here is an abstraction of what I am trying to do:
def myFunction(x):
# myObject is a global variable in this case
return myFunction2(x, myObject)
def myFunction2(x,myObject):
myObject.modify() # here I am calling some method that changes myObject
return myObject.f(x)
poolVar = Pool()
argsArray = [ARGS ARRAY GOES HERE]
output = poolVar.map(myFunction, argsArray)
The function f(x) is contained in a *.so file, i.e., it is calling a C function.
The problem I am having is that the value of the output variable is different each time I run my program (even though the function myObject.f() is a deterministic function). (If I only have one process then the output variable is the same each time I run the program.)
I have tried creating the object rather than storing it as a global variable:
def myFunction(x):
myObject = createObject()
return myFunction2(x, myObject)
However, in my program the object creation is expensive, and thus, it is a lot easier to create myObject once and then modify it each time I call myFunction2(). Thus, I would like to not have to create the object each time.
Do you have any tips? I am very new to parallel programming so I could be going about this all wrong. I decided to use the Pool class since I wanted to start with something simple. But I am willing to try a better way of doing it.
I am using the Pool class from python's multiprocessing library to do
some shared memory processing on an HPC cluster.
Processes are not threads! You cannot simply replace Thread with Process and expect all to work the same. Processes do not share memory, which means that the global variables are copied, hence their value in the original process doesn't change.
If you want to use shared memory between processes then you must use the multiprocessing's data types, such as Value, Array, or use the Manager to create shared lists etc.
In particular you might be interested in the Manager.register method, which allows the Manager to create shared custom objects(although they must be picklable).
However I'm not sure whether this will improve the performance. Since any communication between processes requires pickling, and pickling takes usually more time then simply instantiating the object.
Note that you can do some initialization of the worker processes passing the initializer and initargs argument when creating the Pool.
For example, in its simplest form, to create a global variable in the worker process:
def initializer():
global data
data = createObject()
Used as:
pool = Pool(4, initializer, ())
Then the worker functions can use the data global variable without worries.
Style note: Never use the name of a built-in for your variables/modules. In your case object is a built-in. Otherwise you'll end up with unexpected errors which may be obscure and hard to track down.
Global keyword works on the same file only. Another way is to set value dynamically in pool process initialiser, somefile.py can just be an empty file:
import importlib
def pool_process_init():
m = importlib.import_module("somefile.py")
m.my_global_var = "some value"
pool = Pool(4, initializer=pool_process_init)
How to use the var in task:
def my_coroutine():
m = importlib.import_module("somefile.py")
print(m.my_global_var)