I have created a shared class with a shared variable. My function is supposed to run two parallel processes and find the total number of perfectly square integers. I'm able to get the total number of perfectly square numbers in each array, but when the process is done, I'm not able to get the sum of both of these numbers. Could you check where I went wrong? Creating Shared class was unnecessary but I just did it to check if it would work.
Here is my execution:
from multiprocessing import *
import multiprocessing
import math
class Shared:
def __init__(self) -> None:
self.total = multiprocessing.Value('f', 0)
def setMP(self, value):
self.total.value = value
def getMP(self):
return self.total
# global total
# total = multiprocessing.Value('f', 0) # using a synchronized value for all processes
shared = Shared()
shared.setMP(0)
# function to determine if the number is perfect square
def is_perfect(number):
if float(math.sqrt(number)) *2 == int(math.sqrt(number))*2:
return True # the number is perfect square
return False
# function to find the total number of perfectly square numbers
def find_perfect(array):
# loop through each element in the array
for element in array:
if is_perfect(element):
# get value
shared.getMP().acquire()
i = shared.getMP().value + 1
shared.setMP(i)
shared.getMP().release()
print(shared.getMP())
def perfectSquares(listA, listB):
# multiprocess
p1 = Process(target=find_perfect, args=(listA,))
p2 = Process(target=find_perfect, args=(listB,))
p1.start()
p2.start()
p1.join()
p2.join()
return shared.getMP()
if __name__ == '__main__':
list1 = [7, 8, 23, 64, 2, 3]
list2 = [64, 54, 32, 35, 36]
total = perfectSquares(list1, list2)
print (total)
You are running under Windows, a platform that uses spawn rather than fork to create new processes. What this means is that when a new process is created, execution starts at the very top of the program. This is the reason why the code that creates the new process must be within a if __name__ == '__main__': block (if it weren't, you would get into a recursive loop creating new processes). But this means that each new process you are creating is re-executing any code that is at global scope and is therefore creating its own shared variable instance.
The easiest fix is to move the creation of shared to function perfectSquared and to then pass shared as an argument to findPerfect. Be aware that you have two processes running in parallel but that one must finish before the other. The first process to finish will most likly print a count of 1.0 or 2.0 depending upon which process completes first (although it is possible that it could even be 3.0 when the two processes finish very close together) and the second process to finish must print a count of 3.0.
from multiprocessing import *
import multiprocessing
import math
class Shared:
def __init__(self) -> None:
self.total = multiprocessing.Value('f', 0)
def setMP(self, value):
self.total.value = value
def getMP(self):
return self.total
# function to determine if the number is perfect square
def is_perfect(number):
if float(math.sqrt(number)) *2 == int(math.sqrt(number))*2:
return True # the number is perfect square
return False
# function to find the total number of perfectly square numbers
def find_perfect(array, shared):
# loop through each element in the array
for element in array:
if is_perfect(element):
# get value
shared.getMP().acquire()
i = shared.getMP().value + 1
shared.setMP(i)
shared.getMP().release()
print(shared.getMP())
def perfectSquares(listA, listB):
# global total
# total = multiprocessing.Value('f', 0) # using a synchronized value for all processes
shared = Shared()
shared.setMP(0)
# multiprocess
p1 = Process(target=find_perfect, args=(listA, shared))
p2 = Process(target=find_perfect, args=(listB, shared))
p1.start()
p2.start()
p1.join()
p2.join()
return shared.getMP()
if __name__ == '__main__':
list1 = [7, 8, 23, 64, 2, 3]
list2 = [64, 54, 32, 35, 36]
total = perfectSquares(list1, list2)
print (total)
Prints:
<Synchronized wrapper for c_float(1.0)>
<Synchronized wrapper for c_float(3.0)>
<Synchronized wrapper for c_float(3.0)>
Related
I have a i5-8600k with 6 cores and am running a windows 10 computer. I am trying to perform multi processing with 2 numpy functions. I have made an issue before hand but I have not been successful as to making running the issue: issue, the code down below is from the answer to that issue. I am trying to run func1() and func2() at the same time however, when I run the code below it keeps on running forever.
import multiprocessing as mp
import numpy as np
num_cores = mp.cpu_count()
Numbers = np.array([1,2,3,4,5,6,7,8,9,10,11,12])
def func1():
Solution_1 = Numbers + 10
return Solution_1
def func2():
Solution_2 = Numbers * 10
return Solution_2
# Getting ready my cores, I left one aside
pool = mp.Pool(num_cores-1)
# This is to use all functions easily
functions = [func1, func2]
# This is to store the results
solutions = []
for function in functions:
solutions.append(pool.apply(function, ()))
There are several issues with the code. First, if you want to run this under Jupyter Notebook in Windows then you need to put your worker functions func1 and func2 in an external module, for example, workers.py and import them and that means you now need to either pass the Numbers array as an argument to the workers or initialize static storage of each process with the array when you initialize the pool. We will you the second method with a function called init_pool, which also has to be imported if we are running under Notebook:
workers.py
def func1():
Solution_1 = Numbers + 10
return Solution_1
def func2():
Solution_2 = Numbers * 10
return Solution_2
def init_pool(n_array):
global Numbers
Numbers = n_array
The second issue is that when running under Windows, the code that creates sub-processes or a multiprocessing pool must be within a block that is governed by a conditional if __name__ == '__main__':. Third, it is wasteful to create a pool size greater than 2 if you are only trying to run two parallel "jobs." And fourth, and I think finally, you are using the wrong pool method. apply will block until the "job" submitted (i.e. the one processed by func1) completes and so you are not achieving any degree of parallelism at all. You should be using apply_async.
import multiprocessing as mp
import numpy as np
from workers import func1, func2, init_pool
if __name__ == '__main__':
#num_cores = mp.cpu_count()
Numbers = np.array([1,2,3,4,5,6,7,8,9,10,11,12])
pool = mp.Pool(2, initializer=init_pool, initargs=(Numbers,)) # more than 2 is wasteful
# This is to use all functions easily
functions = [func1, func2]
# This is to store the results
solutions = []
results = [pool.apply_async(function) for function in functions]
for result in results:
solutions.append(result.get()) # wait for completion and get the result
print(solutions)
Prints:
[array([11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]), array([ 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120])]
I am new to multiprocessing with python, I was following a course and i find thatthe code is not working as they say in the tutorials. for example:
this code :
import multiprocessing
# empty list with global scope
result = []
def square_list(mylist):
"""
function to square a given list
"""
global result
# append squares of mylist to global list result
for num in mylist:
result.append(num * num)
# print global list result
print("Result(in process p1): {}".format(result))
if __name__ == "__main__":
# input list
mylist = [1,2,3,4]
# creating new process
p1 = multiprocessing.Process(target=square_list, args=(mylist,))
# starting process
p1.start()
# wait until process is finished
p1.join()
# print global result list
print("Result(in main program): {}".format(result))
should print this result as they say in the tutorial:
Result(in process p1): [1, 4, 9, 16]
Result(in main program): []
but when I run it it prints
Result(in main program): []
I think the prosses did not even start.
I am using python 3.7.9 from anaconda.
how to fix this ?
Do not use global Variables which you access at the same time. Global Variables are most of the time a very bad idea and should be used very carefully.
The easiest way is to use p.map. (you don't have to start/join the processes)
with Pool(5) as p:
result=p.map(square_list,mylist)
If you do not want to use p.map you can use also q.put() to return the value and q.get() to get the value from the function
You can find also examples for getting the result in multiprocessed function here:
https://docs.python.org/3/library/multiprocessing.html
I want to parallelize the processing of a dictionary using the multiprocessing library.
My problem can be reduced to this code:
from multiprocessing import Manager,Pool
def modify_dictionary(dictionary):
if((3,3) not in dictionary):
dictionary[(3,3)]=0.
for i in range(100):
dictionary[(3,3)] = dictionary[(3,3)]+1
return 0
if __name__ == "__main__":
manager = Manager()
dictionary = manager.dict(lock=True)
jobargs = [(dictionary) for i in range(5)]
p = Pool(5)
t = p.map(modify_dictionary,jobargs)
p.close()
p.join()
print dictionary[(3,3)]
I create a pool of 5 workers, and each worker should increment dictionary[(3,3)] 100 times. So, if the locking process works correctly, I expect dictionary[(3,3)] to be 500 at the end of the script.
However; something in my code must be wrong, because this is not what I get: the locking process does not seem to be "activated" and dictionary[(3,3)] always have a valuer <500 at the end of the script.
Could you help me?
The problem is with this line:
dictionary[(3,3)] = dictionary[(3,3)]+1
Three things happen on that line:
Read the value of the dictionary key (3,3)
Increment the value by 1
Write the value back again
But the increment part is happening outside of any locking.
The whole sequence must be atomic, and must be synchronized across all processes. Otherwise the processes will interleave giving you a lower than expected total.
Holding a lock whist incrementing the value ensures that you get the total of 500 you expect:
from multiprocessing import Manager,Pool,Lock
lock = Lock()
def modify_array(dictionary):
if((3,3) not in dictionary):
dictionary[(3,3)]=0.
for i in range(100):
with lock:
dictionary[(3,3)] = dictionary[(3,3)]+1
return 0
if __name__ == "__main__":
manager = Manager()
dictionary = manager.dict(lock=True)
jobargs = [(dictionary) for i in range(5)]
p = Pool(5)
t = p.map(modify_array,jobargs)
p.close()
p.join()
print dictionary[(3,3)]
I ve managed many times to find here the correct solution to a programming difficulty I had. So I would like to contribute a little bit. Above code still has the problem of not updating right the dictionary. To have the right result you have to pass lock and correct jobargs to f. In above code you make a new dictionary in every proccess. The code I found to work fine:
from multiprocessing import Process, Manager, Pool, Lock
from functools import partial
def f(dictionary, l, k):
with l:
for i in range(100):
dictionary[3] += 1
if __name__ == "__main__":
manager = Manager()
dictionary = manager.dict()
lock = manager.Lock()
dictionary[3] = 0
jobargs = list(range(5))
pool = Pool()
func = partial(f, dictionary, lock)
t = pool.map(func, jobargs)
pool.close()
pool.join()
print(dictionary)
In the OP's code, it is locking the entire iteration. In general, you should only apply locks for the shortest time, as long as it is effective. The following code is much more efficient. You acquire the lock only to make the code atomic
def f(dictionary, l, k):
for i in range(100):
with l:
dictionary[3] += 1
Note that dictionary[3] += 1 is not atomic, so it must be locked.
I have a simulation that is currently running, but the ETA is about 40 hours -- I'm trying to speed it up with multi-processing.
It essentially iterates over 3 values of one variable (L), and over 99 values of of a second variable (a). Using these values, it essentially runs a complex simulation and returns 9 different standard deviations. Thus (even though I haven't coded it that way yet) it is essentially a function that takes two values as inputs (L,a) and returns 9 values.
Here is the essence of the code I have:
STD_1 = []
STD_2 = []
# etc.
for L in range(0,6,2):
for a in range(1,100):
### simulation code ###
STD_1.append(value_1)
STD_2.append(value_2)
# etc.
Here is what I can modify it to:
master_list = []
def simulate(a,L):
### simulation code ###
return (a,L,STD_1, STD_2 etc.)
for L in range(0,6,2):
for a in range(1,100):
master_list.append(simulate(a,L))
Since each of the simulations are independent, it seems like an ideal place to implement some sort of multi-threading/processing.
How exactly would I go about coding this?
EDIT: Also, will everything be returned to the master list in order, or could it possibly be out of order if multiple processes are working?
EDIT 2: This is my code -- but it doesn't run correctly. It asks if I want to kill the program right after I run it.
import multiprocessing
data = []
for L in range(0,6,2):
for a in range(1,100):
data.append((L,a))
print (data)
def simulation(arg):
# unpack the tuple
a = arg[1]
L = arg[0]
STD_1 = a**2
STD_2 = a**3
STD_3 = a**4
# simulation code #
return((STD_1,STD_2,STD_3))
print("1")
p = multiprocessing.Pool()
print ("2")
results = p.map(simulation, data)
EDIT 3: Also what are the limitations of multiprocessing. I've heard that it doesn't work on OS X. Is this correct?
Wrap the data for each iteration up into a tuple.
Make a list data of those tuples
Write a function f to process one tuple and return one result
Create p = multiprocessing.Pool() object.
Call results = p.map(f, data)
This will run as many instances of f as your machine has cores in separate processes.
Edit1: Example:
from multiprocessing import Pool
data = [('bla', 1, 3, 7), ('spam', 12, 4, 8), ('eggs', 17, 1, 3)]
def f(t):
name, a, b, c = t
return (name, a + b + c)
p = Pool()
results = p.map(f, data)
print results
Edit2:
Multiprocessing should work fine on UNIX-like platforms such as OSX. Only platforms that lack os.fork (mainly MS Windows) need special attention. But even there it still works. See the multiprocessing documentation.
Here is one way to run it in parallel threads:
import threading
L_a = []
for L in range(0,6,2):
for a in range(1,100):
L_a.append((L,a))
# Add the rest of your objects here
def RunParallelThreads():
# Create an index list
indexes = range(0,len(L_a))
# Create the output list
output = [None for i in indexes]
# Create all the parallel threads
threads = [threading.Thread(target=simulate,args=(output,i)) for i in indexes]
# Start all the parallel threads
for thread in threads: thread.start()
# Wait for all the parallel threads to complete
for thread in threads: thread.join()
# Return the output list
return output
def simulate(list,index):
(L,a) = L_a[index]
list[index] = (a,L) # Add the rest of your objects here
master_list = RunParallelThreads()
Use Pool().imap_unordered if ordering is not important. It will return results in a non-blocking fashion.
I have three functions, each returning a list. The problem is that running each function takes around 20-30 seconds. So running the entire script ends up taking about 2 min.
I want to use multiprocessing or multithreading (whichever is easier to implement) to have all three functions running at the same time.
The other hurdle I ran into was I that I'm not sure how to return the list from each of the functions.
def main():
masterlist = get_crs_in_snow()
noop_crs = get_noops_in_snow()
made_crs = get_crs_in_git()
# take the prod master list in SNOW, subtract what's been made or is in the noop list
create_me = [obj for obj in masterlist if obj not in made_crs and obj not in noop_crs]
print "There are {0} crs in Service Now not in Ansible".format(len(create_me))
for cr in create_me:
print str(cr[0]),
if __name__ == '__main__':
main()
I figure I can get some significant improvements in run time just by multithreading or multiprocessing the following line:
masterlist = get_crs_in_snow()
noop_crs = get_noops_in_snow()
made_crs = get_crs_in_git()
How do I have these three functions run at the same time?
This is completely untested since I don't have the rest of your code, but it may give you an idea of what can be done. I have adapted your code into the multiprocessing pattern:
from multiprocessing import Pool
def dispatcher(n):
if n == 0:
return get_crs_in_snow()
if n == 1:
return get_noops_in_snow()
if n == 2:
return get_crs_in_git()
def main():
pool = Pool(processes=3)
v = pool.map(dispatcher, range(3))
masterlist = v[0]
noop_crs = v[1]
made_crs = v[2]
# take the prod master list in SNOW, subtract what's been made or is in the noop list
create_me = [obj for obj in masterlist if obj not in made_crs and obj not in noop_crs]
print "There are {0} crs in Service Now not in Ansible".format(len(create_me))
for cr in create_me:
print str(cr[0]),
if __name__ == '__main__':
main()
Try the threading library.
import threading
threading.Thread(target=get_crs_in_snow).start()
threading.Thread(target=get_noops_in_snow).start()
threading.Thread(target=get_crs_in_git).start()
As far as getting their return values, you could wrap the calls to recommon in some class functions and have them save the result to a member variable. Or, you could wrap the calls to recommon in some local functions and simply pass in a mutable object (list or dictionary) to the function, and have the function modify that mutable object.
Or, as others have stated, multiprocessing may be a good way to do what you want.