Lets say I have two 3 dimensional arrays (a & b) of shape (1.000.000, ???, 50), (??? = see below).
How to merge them,
so that the result will be (1.000.000, {shape of a's + b's second dimension} , 50)?
Here are the samples, as you can see below: (np.arrays are also possible)
EDIT: added usable code, please scroll^^
[ #a
[
],
[
[1 2 3]
],
[
[0 2 7]
[1 Nan 3]
],
[
[10 0 3]
[NaN 9 9]
[10 NaN 3]
],
[
[8 2 0]
[2 2 3]
[8 1 3]
[1 2 3]
],
[
[0 2 3]
[1 2 9]
[1 2 3]
[1 0 3]
[1 2 3]
]
]
[#b
[
[7 2 3]
[1 2 9]
[1 2 3]
[8 0 3]
[1 7 3]
]
[
[3 9 0]
[2 2 3]
[8 1 3]
[0 2 3]
],
[
[10 0 3]
[0 NaN 9]
[10 NaN 3]
],
[
[0 2 NaN]
[1 Nan 3]
],
[
[1 2 NaN]
],
[
]
]
a = [ [ ],
[ [1, 2, 3] ],
[ [0, 2, 7], [1,np.nan,3] ],
[
[10,0,3], [np.nan,9,9], [10,np.nan,3]
],
[
[8,2,0], [2,2,3], [8,1,3], [1,2,3]
],
[
[0,2,3], [1,2,9], [1,2,3], [1,0,3], [1,2,3]
]
]
b = [
[
[7,2,3], [1,2,9], [1,2,3], [8,0,3], [1,7,3]
],
[
[3,9,0], [2,2,3], [8,1,3], [0,2,3]
],
[
[10,0,3], [0,np.nan,9], [10,np.nan,3]
],
[
[0,2,np.nan], [1,np.nan,3]
],
[
[1,2,np.nan]
],
[
]
]
expected outcome:
[
[ [7 2 3]# from b
[1 2 9]# from b
[1 2 3]# from b
[8 0 3]# from b
[1 7 3]# from b
],
[
[1 2 3]
[3 9 0]# from b
[2 2 3]# from b
[8 1 3]# from b
[0 2 3]# from b
],
[
[0 2 7]
[1 Nan 3]
[10 0 3]# from b
[0 NaN 9]# from b
[10 NaN 3]# from b
],
[
[10 0 3]
[NaN 9 9]
[10 NaN 3]
[0 2 NaN]# from b
[1 Nan 3]# from b
],
[
[8 2 0]
[2 2 3]
[8 1 3]
[1 2 3]
[1 2 NaN]# from b
],
[
[0 2 3]
[1 2 9]
[1 2 3]
[1 0 3]
[1 2 3]
]
]
Do you know a way to do that efficiently?
EDIT: tried concatenate (didnt work):
DF_LEN, COL_LEN, cols = 20,5,['A', 'B']
a = np.asarray(pd.DataFrame(1, index=range(DF_LEN), columns=cols))
a = list((map(lambda i: a[:i], range(1,a.shape[0]+1))))
b = np.asarray(pd.DataFrame(np.nan, index=range(DF_LEN), columns=cols))
b = list((map(lambda i: b[:i], range(1,b.shape[0]+1))))
b = b[::-1]
a_first = a[0]; del a[0]
b_last = b[-1]; del b[-1]
result = np.concatenate([a, b], axis=1)
>>>AxisError: axis 1 is out of bounds for array of dimension 1
You cannot have an array with variable length in a dimension. a and b are most likely list of lists and not arrays. You can use list comprehension along with zip:
np.array([x+y for x,y in zip(a,b)])
EDIT: or based on comment provided if a and b are lists of arrays:
np.array([np.vstack((x,y)) for x,y in zip(a,b)])
The output for your example looks like:
[[[ 7. 2. 3.]
[ 1. 2. 9.]
[ 1. 2. 3.]
[ 8. 0. 3.]
[ 1. 7. 3.]]
[[ 1. 2. 3.]
[ 3. 9. 0.]
[ 2. 2. 3.]
[ 8. 1. 3.]
[ 0. 2. 3.]]
[[ 0. 2. 7.]
[ 1. nan 3.]
[10. 0. 3.]
[ 0. nan 9.]
[10. nan 3.]]
[[10. 0. 3.]
[nan 9. 9.]
[10. nan 3.]
[ 0. 2. nan]
[ 1. nan 3.]]
[[ 8. 2. 0.]
[ 2. 2. 3.]
[ 8. 1. 3.]
[ 1. 2. 3.]
[ 1. 2. nan]]
[[ 0. 2. 3.]
[ 1. 2. 9.]
[ 1. 2. 3.]
[ 1. 0. 3.]
[ 1. 2. 3.]]]
To perform your concatenation, run:
result = np.concatenate([a, b], axis=1)
To test this code, I created a and b as:
a = np.stack([ np.full((2, 3), i) for i in range(1, 6)], axis=1)
b = np.stack([ np.full((2, 3), i + 10) for i in range(1, 4)], axis=1)
So they contain:
array([[[1, 1, 1], array([[[11, 11, 11],
[2, 2, 2], [12, 12, 12],
[3, 3, 3], [13, 13, 13]],
[4, 4, 4],
[5, 5, 5]], [[11, 11, 11],
[12, 12, 12],
[[1, 1, 1], [13, 13, 13]]])
[2, 2, 2],
[3, 3, 3],
[4, 4, 4],
[5, 5, 5]]])
and their shapes are: (2, 5, 3) and (2, 3, 3)
The result of my concatenation is:
array([[[ 1, 1, 1],
[ 2, 2, 2],
[ 3, 3, 3],
[ 4, 4, 4],
[ 5, 5, 5],
[11, 11, 11],
[12, 12, 12],
[13, 13, 13]],
[[ 1, 1, 1],
[ 2, 2, 2],
[ 3, 3, 3],
[ 4, 4, 4],
[ 5, 5, 5],
[11, 11, 11],
[12, 12, 12],
[13, 13, 13]]])
and the shape is (2, 8, 3), just as it should be.
Edit following the comment as of 19:56Z
I tried the code from your comment.
After you executed a = list((map(lambda i: a[:i], range(1,a.shape[0]+1)))),
the result is:
[array([[1, 1]], dtype=int64),
array([[1, 1],
[1, 1]], dtype=int64),
array([[1, 1],
[1, 1],
[1, 1]], dtype=int64),
array([[1, 1],
[1, 1],
[1, 1],
[1, 1]], dtype=int64),
array([[1, 1],
[1, 1],
[1, 1],
[1, 1],
[1, 1]], dtype=int64),
...
so a is a list of arrays of varying sizes.
Theres is something wrong in the way you construct your data.
First check that your both arrays are 3-D and their shapes differ
only in axis 1. Only then you can run my code on them.
For now both a and b are plain pythonic lists, not Numpy arrays!
How would you separate a 2D numpy array into a nxn chunks?
For example, the following array of shape (4,4):
arr = [[1,2,3,4],
[5,6,7,8],
[9,10,11,12],
[13,14,15,16]]
Transformed to this array, of shape (4,2,2), by subsampling with a different (2x2) array:
new_arr = [[[1,2],
[5,6]],
[[3,4],
[7,8]],
[[9,10],
[13,14]],
[[11,12],
[15,16]]]
You can use np.vsplit to split the array into multiple subarrays vertically. Similarly you can use np.hsplit to split the array into multiple subarrays horizontally. To better understand this examine the generalized resample function which makes the use of np.vsplit and np.hsplit methods.
Use this:
def ressample(arr, N):
A = []
for v in np.vsplit(arr, arr.shape[0] // N):
A.extend([*np.hsplit(v, arr.shape[0] // N)])
return np.array(A)
Example 1:
The given 2D array is of shape 4x4 and we want to subsample it into the chunks of shape 2x2.
arr = np.array([[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16]])
print(ressample(arr, 2)) #--> chunk size 2
Output 1:
[[[ 1 2]
[ 5 6]]
[[ 3 4]
[ 7 8]]
[[ 9 10]
[13 14]]
[[11 12]
[15 16]]]
Example 2:
Consider the given 2D array contains 8 rows and 8 columns. Now we subsample this array into the chunks of shape 4x4.
arr = np.random.randint(0, 10, 64).reshape(8, 8)
print(ressample(arr, 4)) #--> chunck size 4
Sample Output 2:
[[[8 3 7 5]
[7 2 6 1]
[7 9 2 2]
[3 1 8 8]]
[[2 0 3 2]
[2 9 0 8]
[2 6 3 9]
[2 4 4 8]]
[[9 9 1 8]
[9 1 5 0]
[8 5 1 2]
[2 7 5 1]]
[[7 8 9 6]
[9 0 9 5]
[8 9 8 3]
[7 3 6 3]]]
You could do the following, and adjust it to your array:
import numpy as np
arr = [[1,2,3,4],
[5,6,7,8],
[9,10,11,12],
[13,14,15,16]]
arr_new = np.array([[arr[i][j:j+2], arr[i+1][j:j+2]] for j in range(len(arr[0])-2) for i in range(len(arr)-2)])
print(arr_new)
print(arr_new.shape)
This gives the following output:
[[[ 1 2]
[ 5 6]]
[[ 5 6]
[ 9 10]]
[[ 2 3]
[ 6 7]]
[[ 6 7]
[10 11]]]
(4, 2, 2)
You could use hsplit() and vsplit() methods to achieve the above.
import numpy as np
arr = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12],[13,14,15,16]])
ls1,ls2 = np.hsplit(arr, 2)
ls1 = np.vsplit(ls1,2)
ls2 = np.vsplit(ls2,2)
ls = ls1 + ls2
result = np.array(ls)
print(result)
>>>
[[[ 1 2]
[ 5 6]]
[[ 9 10]
[13 14]]
[[ 3 4]
[ 7 8]]
[[11 12]
[15 16]]]
print(result.tolist())
>>> [[[1, 2], [5, 6]], [[9, 10], [13, 14]], [[3, 4], [7, 8]], [[11, 12], [15, 16]]]
There is no need to split or anything; the same can be achieved by reshaping and reordering the axes.
result = np.swapaxes(arr.reshape(2, 2, 2, 2), 1, 2).reshape(-1, 2, 2)
Dividing an (N, N) array to (n, n) chunks is also basically a sliding window op with an (n, n) window and a stride of n.
from numpy.lib.stride_tricks import sliding_window_view
result = sliding_window_view(arr, (2, 2))[::2, ::2].reshape(-1, 2, 2)
I have an array like this:
array = np.array([[[[ 2, -3],[ 3, 2]],[[-4, -1],[-5, 1]],
[[-7, -5],[-1, 6]],[[-5, 0],[-4, 2]]],
[[[-1, 4],[ 6, 1]],[[-2, -3],[-5, 5]],
[[-2, -8],[-1, 7]],[[-1, 8],[-4, 2]]]])
If I sum(array) then I get the sum of (4x2x2) + (4x2x2).
How can I sum the elements inside of the first arrays, opposite of what sum() function did. Like (2-3) = -1 in the first group, (3+2) = 5 in the second, etc.
Thanks
summing along the 3rd axis should do what you want:
res = np.sum(array, axis=3)
# or:
# res = array.sum(axis=3)
which produces
[[[ -1 5]
[ -5 -4]
[-12 5]
[ -5 -2]]
[[ 3 7]
[ -5 0]
[-10 6]
[ 7 -2]]]
I'm doing image processing for object detection using python. I need to divide my image into all possible blocks. For example given this toy image:
x = np.arange(25)
x = x.reshape((5, 5))
[[ 0 1 2 3 4]
[ 5 6 7 8 9]
[10 11 12 13 14]
[15 16 17 18 19]
[20 21 22 23 24]]
I want to retrieve all possible blocks of a given size, for example the 2x2 blocks are:
[[0 1]
[5 6]]
[[1 2]
[6 7]]
.. and so on. How can I do this?
The scikit image extract_patches_2d does that
>>> from sklearn.feature_extraction import image
>>> one_image = np.arange(16).reshape((4, 4))
>>> one_image
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
>>> patches = image.extract_patches_2d(one_image, (2, 2))
>>> print(patches.shape)
(9, 2, 2)
>>> patches[0]
array([[0, 1],
[4, 5]])
>>> patches[1]
array([[1, 2],
[5, 6]])
>>> patches[8]
array([[10, 11],
[14, 15]])
You can use something like this:
def rolling_window(arr, window):
"""Very basic multi dimensional rolling window. window should be the shape of
of the desired subarrays. Window is either a scalar or a tuple of same size
as `arr.shape`.
"""
shape = np.array(arr.shape*2)
strides = np.array(arr.strides*2)
window = np.asarray(window)
shape[arr.ndim:] = window # new dimensions size
shape[:arr.ndim] -= window - 1
if np.any(shape < 1):
raise ValueError('window size is too large')
return np.lib.stride_tricks.as_strided(arr, shape=shape, strides=strides)
# Now:
slices = rolling_window(arr, 2)
# Slices will be 4-d not 3-d as you wanted. You can reshape
# but it may need to copy (not if you have done no slicing, etc. with the array):
slices = slices.reshape(-1,slices.shape[2:])
Simple code with a double loop and slice:
>>> a = np.arange(12).reshape(3,4)
>>> print(a)
[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]]
>>> r = 2
>>> n_rows, n_cols = a.shape
>>> for row in range(n_rows - r + 1):
... for col in range(n_cols - r + 1):
... print(a[row:row + r, col:col + r])
...
[[0 1]
[4 5]]
[[1 2]
[5 6]]
[[2 3]
[6 7]]
[[4 5]
[8 9]]
[[ 5 6]
[ 9 10]]
[[ 6 7]
[10 11]]