We Keep Coding

Use multiprocessings 'Pool' together with 'RawArray'

Whenever I try to use shared memory with pythons 'multiprocessing' module to fill a huge array in parallel I use something like:
import numpy as np
from multiprocessing import Process, RawArray
def tf(x, arr):
arr = np.reshape( np.frombuffer( arr, dtype=np.float32 ), -1 ).reshape((10, 10, 10))
arr[x] = np.random.random((10, 10))
mpa = RawArray('f', 1000)
ncpu = 4
procs = []
for i in range(10):
procs.append(Process(target=tf, args=(i, mpa)))
procs[-1].start()
if len(procs) == ncpu:
procs[0].join()
procs.pop(0)
for p in procs:
p.join()
arr = np.reshape( np.frombuffer( mpa, dtype=np.uint32 ), -1).reshape((10, 10, 10))
to ensure that only as many processes are active as I have cpus. If I try to use 'Pool' and 'apply_async' the array is not altered for some reason. So I wonder if it is possible to either use 'Pool' or any other intended way to manage the amount of active processes.
The above code is working but is not the most efficient since I only if the process I added first is finished to decide if I should add another process.

Saving images in a loop faster than multithreading / multiprocessing

Here's a timed example of multiple image arrays of different sizes being saved in a loop as well as concurrently using threads / processes:
import tempfile
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
from pathlib import Path
from time import perf_counter
import numpy as np
from cv2 import cv2
def save_img(idx, image, dst):
cv2.imwrite((Path(dst) / f'{idx}.jpg').as_posix(), image)
if __name__ == '__main__':
l1 = np.random.randint(0, 255, (100, 50, 50, 1))
l2 = np.random.randint(0, 255, (1000, 50, 50, 1))
l3 = np.random.randint(0, 255, (10000, 50, 50, 1))
temp_dir = tempfile.mkdtemp()
workers = 4
t1 = perf_counter()
for ll in l1, l2, l3:
t = perf_counter()
for i, img in enumerate(ll):
save_img(i, img, temp_dir)
print(f'Time for {len(ll)}: {perf_counter() - t} seconds')
for executor in ThreadPoolExecutor, ProcessPoolExecutor:
with executor(workers) as ex:
futures = [
ex.submit(save_img, i, img, temp_dir) for (i, img) in enumerate(ll)
]
for f in as_completed(futures):
f.result()
print(
f'Time for {len(ll)} ({executor.__name__}): {perf_counter() - t} seconds'
)
And I get these durations on my i5 mbp:
Time for 100: 0.09495482999999982 seconds
Time for 100 (ThreadPoolExecutor): 0.14151873999999998 seconds
Time for 100 (ProcessPoolExecutor): 1.5136184309999998 seconds
Time for 1000: 0.36972280300000016 seconds
Time for 1000 (ThreadPoolExecutor): 0.619205703 seconds
Time for 1000 (ProcessPoolExecutor): 2.016624468 seconds
Time for 10000: 4.232915643999999 seconds
Time for 10000 (ThreadPoolExecutor): 7.251599262 seconds
Time for 10000 (ProcessPoolExecutor): 13.963426469999998 seconds
Aren't threads / processes expected to require less time to achieve the same thing? and why not in this case?

The timings in the code are wrong because the timer t is not reset before testing the Pools. Nevertheless, the relative order of the timings are correct. A possible code with a timer reset is:
import tempfile
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
from pathlib import Path
from time import perf_counter
import numpy as np
from cv2 import cv2
def save_img(idx, image, dst):
cv2.imwrite((Path(dst) / f'{idx}.jpg').as_posix(), image)
if __name__ == '__main__':
l1 = np.random.randint(0, 255, (100, 50, 50, 1))
l2 = np.random.randint(0, 255, (1000, 50, 50, 1))
l3 = np.random.randint(0, 255, (10000, 50, 50, 1))
temp_dir = tempfile.mkdtemp()
workers = 4
for ll in l1, l2, l3:
t = perf_counter()
for i, img in enumerate(ll):
save_img(i, img, temp_dir)
print(f'Time for {len(ll)}: {perf_counter() - t} seconds')
for executor in ThreadPoolExecutor, ProcessPoolExecutor:
t = perf_counter()
with executor(workers) as ex:
futures = [
ex.submit(save_img, i, img, temp_dir) for (i, img) in enumerate(ll)
]
for f in as_completed(futures):
f.result()
print(
f'Time for {len(ll)} ({executor.__name__}): {perf_counter() - t} seconds'
)
Multithreading is faster specially for I/O bound processes. In this case, compressing the images is cpu-intensive, so depending on the implementation of OpenCV and of the python wrapper, multithreading can be much slower. In many cases the culprit is CPython's GIL, but I am not sure if this is the case (I do not know if the GIL is released during the imwrite call). In my setup (i7 8th gen), Threading is as fast as the loop for 100 images and barely faster for 1000 and 10000 images. If ThreadPoolExecutor reuses threads, there is an overhead involved in assigning a new task to an existing thread. If it does not reuses threads, there is an overhead involved in launching a new thread.
Multiprocessing circumvents the GIL issue, but has some other problems. First, pickling the data to pass between processes takes some time, and in the case of images it can be very expensive. Second, in the case of windows, spawning a new process takes a lot of time. A simple test to see the overhead (both for processes and threads) is to change the save_image function by one that does nothing, but still need pickling, etc:
def save_img(idx, image, dst):
if idx != idx:
print("impossible!")
and by a similar one without parameters to see the overhead of spawning the processes, etc.
The timings in my setup show that 2.3 seconds are needed just to spawn the 10000 processes and 0.6 extra seconds for pickling, which is much more than the time needed for processing.
A way to improve the throughput and keep the overhead to a minimum is to break the work on chunks, and submit each chunk to the worker:
import tempfile
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
from pathlib import Path
from time import perf_counter
import numpy as np
from cv2 import cv2
def save_img(idx, image, dst):
cv2.imwrite((Path(dst) / f'{idx}.jpg').as_posix(), image)
def multi_save_img(idx_start, images, dst):
for idx, image in zip(range(idx_start, idx_start + len(images)), images):
cv2.imwrite((Path(dst) / f'{idx}.jpg').as_posix(), image)
if __name__ == '__main__':
l1 = np.random.randint(0, 255, (100, 50, 50, 1))
l2 = np.random.randint(0, 255, (1000, 50, 50, 1))
l3 = np.random.randint(0, 255, (10000, 50, 50, 1))
temp_dir = tempfile.mkdtemp()
workers = 4
for ll in l1, l2, l3:
t = perf_counter()
for i, img in enumerate(ll):
save_img(i, img, temp_dir)
print(f'Time for {len(ll)}: {perf_counter() - t} seconds')
chunk_size = len(ll)//workers
ends = [chunk_size * (_+1) for _ in range(workers)]
ends[-1] += len(ll) % workers
starts = [chunk_size * _ for _ in range(workers)]
for executor in ThreadPoolExecutor, ProcessPoolExecutor:
t = perf_counter()
with executor(workers) as ex:
futures = [
ex.submit(multi_save_img, start, ll[start:end], temp_dir) for (start, end) in zip(starts, ends)
]
for f in as_completed(futures):
f.result()
print(
f'Time for {len(ll)} ({executor.__name__}): {perf_counter() - t} seconds'
)
This should give you a significant boost over a simple for, both for a multiprocessing and multithreading approach.

Multiprocessing for a range of loops in Python?

I have a very big array that I need to create with over 10^7 columns that needs to get filtered/modified depending on some criteria. There is a set of 24 different criterias (2x4x3 due to combinations) which means the filtering/modification needs to be done 24 times and each result is saved in a different specified directory.
Since this takes a very long time, I am looking into using multiprocessing to speed up the process. Can anyone help me out? Here is an exemplary code:
import itertools
import numpy as np
sample_size = 1000000
variables = 25
x_array = np.random.rand(variables, sample_size)
x_dir = ['x1', 'x2']
y_dir = ['y1', 'y2', 'y3', 'y4']
z_dir = ['z1', 'z2', 'z3']
x_directories = [0, 1]
y_directories = [0, 1, 2, 3]
z_directories = [0, 1, 2]
directory_combinations = itertools.product(x_directories, y_directories, z_directories)
for k, t, h in directory_combinations:
target_dir=main_dir+'/'+x_dir[k]+'/'+y_dir[t]+'/'+z_dir[h]
for i in range(sample_size):
#x_array gets filtered/modified
#x_array gets saved in target_dir directory as a dataframe after modification'''
Basically with multiprocessing I am hoping for either each loop handled by a single core out of 16 I have available or for each loop iteration to be sped up by using all 16 cores.
Many thanks in advance!

Take one of loop and rewrite it to function
for k, t, h in directory_combinations:
Becomes for example
def func(k,t,h):
....
pool = multiprocessing.Pool(12)
pool.starmap_async(func, directory_combinations, 32)
It spawns 12 processes, that apply func on each iteration of 2 argument. Data tranfered to processes by 32-length chunks.

The following code first creates the x_array in shared memory and initialized each process in the pool with global variable x_array, which is this shared array.
I would move the code that creates a the copy of this global x_array, processes it and then writes out the dataframe to a function, worker, which is passed the target directory as an argument.
import itertools
import numpy as np
import ctypes
import multiprocessing as mp
SAMPLE_SIZE = 1000000
VARIABLES = 25
def to_numpy_array(shared_array, shape):
'''Create a numpy array backed by a shared memory Array.'''
arr = np.ctypeslib.as_array(shared_array)
return arr.reshape(shape)
def to_shared_array(arr, ctype):
shared_array = mp.Array(ctype, arr.size, lock=False)
temp = np.frombuffer(shared_array, dtype=arr.dtype)
temp[:] = arr.flatten(order='C')
return shared_array
def init_pool(shared_array, shape):
global x_array
# Recreate x_array using shared memory array:
x_array = to_numpy_array(shared_array, shape)
def worker(target_dir):
# make copy of x_array with np.copy
x_array_copy = np.copy(x_array)
for i in range(sample_size):
#x_array_copy gets filtered/modified
...
#x_array_copy gets saved in target_dir directory as a dataframe after modification
def main():
main_dir = '.' # for example
x_dir = ['x1', 'x2']
y_dir = ['y1', 'y2', 'y3', 'y4']
z_dir = ['z1', 'z2', 'z3']
x_directories = [0, 1]
y_directories = [0, 1, 2, 3]
z_directories = [0, 1, 2]
directory_combinations = itertools.product(x_directories, y_directories, z_directories)
target_dirs = [main_dir+'/'+x_dir[k]+'/'+y_dir[t]+'/'+z_dir[h] for k, t, h in directory_combinations]
x_array = np.random.rand(VARIABLES, SAMPLE_SIZE)
shape = x_array.shape
# Create array in shared memory
shared_array = to_shared_array(x_array, ctypes.c_int64)
# Recreate x_array using the shared memory array as the base:
x_array = to_numpy_array(shared_array, shape)
# Create pool of 12 processes copying the shared array to each process:
pool = mp.Pool(12, initializer=init_pool, initargs=(shared_array, shape))
pool.map(worker, target_dirs)
# This is required for Windows:
if __name__ == '__main__':
main()

How to Parallelize a GridSearch Scan with Talos

While talos supports GPU parallelization, how do you extend the Scan object to support CPU + GPU parallelization?

Following the approach of breaking up the scan experiments into processes:
import multiprocessing as mp
from itertools import product
import talos
import os
# Helper function to create configuration chunks
def chunkify(lst, n):
return [lst[i::n] for i in range(n)]
# a Talos Scan Configuration superset
playbook_configurations = {
"input_lstm_dim": [5, 15, 30, 50],
"dense_a_dim": [None, 5],
"dense_b_dim": [None, 5],
"dense_c_dim": [None, 5],
"dropout_a_rate": [None, 0.7, 0.5, 0.3],
"epochs": [100],
"verbose": [verbose_flag],
"batch_normalization": [None, 1]
}
# Threadsafe Queue for scan results
output = mp.Queue()
# Actual scan to run within each process
def process_scan(playbook_scan_settings, output):
scan = talos.Scan(
...
params=playbook_scan_settings,
)
...
output.put(results) # pump results onto queue
# Sample Process count based on core affinity
cpu_count = len(os.sched_getaffinity(0))
# Cartesian product of Talos Configuration
playbook_configurations_cartesian_product = [dict(zip(playbook_configurations, v)) for v in product(
*playbook_configurations.values())]
# Configuration chunks to assign to each process
playbook_configuration_groups = chunkify(
playbook_configurations_cartesian_product, cpu_count)
processes = []
for playbook_configuration_group in playbook_configuration_groups:
# merged (array) configuration for process group
playbook_scan_settings = {}
for g in playbook_configuration_group:
for k, v in g.items():
if not k in playbook_scan_settings:
playbook_scan_settings[k] = []
if not v in playbook_scan_settings[k]:
playbook_scan_settings[k].append(v)
if bool(playbook_scan_settings):
# process to scan on merged configuration for process group
processes.append(mp.Process(
target=process_scan, args=(playbook_scan_settings, output)))
for p in processes:
p.start()
for p in processes:
p.join()
# Will be the result from the message queue
results = [output.get() for p in processes]
You can easily pump the Report object, winning models, and metrics per scan tier into the message queue for final selection.

If statement over dask array

Hi everybody can you tell me why a If statement over dask array is so slow and how to solve it ?
import dask.array as da
import time
x = da.random.binomial(1, 0.5, 200, 200)
s = time.time()
if da.any(x):
e = time.time()
print('duration = ', e-s)
output: duration = 0.368

Dask array is lazy by default, so no work happens until you call .compute() on your array.
In your case you are implicitly calling .compute() when you place your dask array into an if statement, which converts things into booleans.
x = da.random.random(...) # this is free
y = x + x.T # this is free
z = y.any() # this is free
if z: # everything above happens now,
...

I took at look at the dask source code. Essentially, when you call functions on dask arrays it performs a "reduction" of the array. Intuitively this is necessary because, behind the scenes, dask arrays are stored as separate "blocks" that can live individually in memory, on disk, etc. but you need to somehow pull pieces of them together for function calls.
So the time you are noticing is in the initial overhead of performing the reduction. Note that if you increase the size of the array to 2M, it takes about the same time as for 200. At 20M it only takes about 1s.
import dask.array as da
import time
# 200 case
x = da.random.binomial(1, 0.5, 200, 200)
print x.shape
s = time.time()
print "start"
if da.any(x):
e = time.time()
print 'duration = ', e-s
# duration = 0.362557172775
# 2M case
x = da.random.binomial(1, 0.5, 2000000, 2000000)
print x.shape
s = time.time()
print "start"
if da.any(x):
e = time.time()
print 'duration = ', e-s
# duration = 0.132781982422
# 20M case
x = da.random.binomial(1, 0.5, 20000000, 20000000)
print x.shape
s = time.time()
print "start"
if da.any(x):
e = time.time()
print 'duration = ', e-s
# duration = 1.08430886269
# 200M case
x = da.random.binomial(1, 0.5, 200000000, 200000000)
print x.shape
s = time.time()
print "start"
if da.any(x):
e = time.time()
print 'duration = ', e-s
# duration = 8.83682179451