Related
I have a multidimensional array of shape (n,x,y). For this example can use this array
A = array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]]])
I then have another multidimensional array that has index values that I want to use on the original array, A. This has shape (z,2) and the values represent row values index’s
Row_values = array([[0,1],
[0,2],
[1,2],
[1,3]])
So I want to use all the index values in row_values to apply to each of the three arrays in A so I end up with a final array of shape (12,2,3)
Result = ([[[0,1,2],
[3,4,5]],
[[0,1,2],
[6,7,8]],
[[3,4,5],
[6,7,8]]
[[3,4,5],
[9,10,11],
[[12,13,14],
[15,16,17]],
[[12,13,14],
[18,19,20]],
[[15,16,17],
[18,19,20]],
[[15,16,17],
[21,22,23]],
[[24,25,26],
[27,28,29]],
[[24,25,26],
[30,31,32]],
[[27,28,29],
[30,31,32]],
[[27,28,29],
[33,34,35]]]
I have tried using np.take() but haven’t been able to make it work. Not sure if there’s another numpy function that is easier to use
We can advantage of NumPy's advanced indexing and using np.repeat and np.tile along with it.
cidx = np.tile(Row_values, (A.shape[0], 1))
ridx = np.repeat(np.arange(A.shape[0]), Row_values.shape[0])
out = A[ridx[:, None], cidx]
# out.shape -> (12, 2, 3)
Using np.take
np.take(A, Row_values, axis=1).reshape((-1, 2, 3))
# Or
A[:, Row_values].reshape((-1, 2, 3))
Output:
array([[[ 0, 1, 2],
[ 3, 4, 5]],
[[ 0, 1, 2],
[ 6, 7, 8]],
[[ 3, 4, 5],
[ 6, 7, 8]],
[[ 3, 4, 5],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17]],
[[12, 13, 14],
[18, 19, 20]],
[[15, 16, 17],
[18, 19, 20]],
[[15, 16, 17],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29]],
[[24, 25, 26],
[30, 31, 32]],
[[27, 28, 29],
[30, 31, 32]],
[[27, 28, 29],
[33, 34, 35]]])
I have a 3D matrix, in below example it's a (5, 4, 2) matrix: data_matrix
I have a another index array of shape (5, 4) where each row of array represent the element location: indx_array
I don't know how can I get the required_output. I'm trying to arrange (1,2) elements of each row based on the indx_array
I don't want to use for loops!
data_matrix = np.array([
[[0, 1], [2, 3], [4, 5], [6, 7]],
[[8, 9], [10, 11], [12, 13], [14, 15]],
[[16, 17], [18, 19], [20, 21], [22, 23]],
[[24, 25], [26, 27], [28, 29], [30, 31]],
[[32, 33], [34, 35], [36, 37], [38, 39]]
])
indx_array = np.array([[3,2,1,0], [0,1,2,3], [1,0,3,2], [0,3,1,2], [1,2,3,0]])
# I want following result:
required_output = [
[[6, 7], [4, 5], [2, 3], [0, 1]]
[[8, 9], [10, 11], [12, 13], [14, 15]]
[[18, 19], [16, 17], [22, 23], [20, 21]]
[[24, 25], [30, 31], [26, 27], [28, 29]]
[[34, 35], [36, 37], [38, 39], [32, 33]]
]
EDIT: Updated the indx_array to better illustrate the situation.
Can be done with a little bit of handing of the index array.
import numpy as np
_x = np.repeat(np.arange(indx_array.shape[0]),indx_array.shape[1])
_y = indx_array.ravel()
output = data_matrix[_x, _y].reshape(data_matrix.shape)
which results in the expected numpy array
array([[[ 6, 7],
[ 4, 5],
[ 2, 3],
[ 0, 1]],
[[ 8, 9],
[10, 11],
[12, 13],
[14, 15]],
[[18, 19],
[16, 17],
[22, 23],
[20, 21]],
[[24, 25],
[30, 31],
[26, 27],
[28, 29]],
[[34, 35],
[36, 37],
[38, 39],
[32, 33]]])
Numpy: Indexing
Numpy: Indexing Multi-dimensional arrays
In [637]: data_matrix.shape
Out[637]: (5, 4, 2)
In [638]: indx_array.shape
Out[638]: (5, 4)
You need advanced-indexing on the first 2 dimensions. The first dimension array needs to broadcast with the second (5,4). To do that I make a (5,1) arange:
In [639]: data_matrix[np.arange(5)[:,None], indx_array]
Out[639]:
array([[[ 6, 7],
[ 4, 5],
[ 2, 3],
[ 0, 1]],
[[ 8, 9],
[10, 11],
[12, 13],
[14, 15]],
[[18, 19],
[16, 17],
[22, 23],
[20, 21]],
[[24, 25],
[30, 31],
[26, 27],
[28, 29]],
[[34, 35],
[36, 37],
[38, 39],
[32, 33]]])
Contrast my (5,1) index with the accepted _x (which is (5,4) ravelled):
In [640]: np.arange(5)[:,None]
Out[640]:
array([[0],
[1],
[2],
[3],
[4]])
In [641]: _x = np.repeat(np.arange(indx_array.shape[0]),indx_array.shape[1])
In [643]: _x
Out[643]: array([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4])
With broadcasting the _x doesn't need the repeat, (5,4); (5,1) is enough.
Broadcasting does a virtual repetition. This can be illustrated with the broadcast_to function:
In [648]: np.broadcast_to(np.arange(5)[:,None],(5,4))
Out[648]:
array([[0, 0, 0, 0],
[1, 1, 1, 1],
[2, 2, 2, 2],
[3, 3, 3, 3],
[4, 4, 4, 4]])
In [649]: _.strides
Out[649]: (8, 0)
It's that 0 strides that repeats without making copies. as_strided is the most useful stride_tricks function, especially when doing things like moving windows. Usually we just let the automatic broadcasting do the work without worrying too much about the how.
I have a random length list that contains ranges info:
list = [
[[7, 12], [6, 12], [38, 44], [25, 30], [25, 29]],
[[0, 5], [1, 5], [2, 5], [12, 16], [13, 16], [20, 23], [29, 33], [30, 33]],
[[5, 7], [6, 8], [7, 9], [8, 10], [9, 11], [10, 12], [16, 18], [17, 19], [18, 20], [23, 25], [24, 26], [25, 27], [26, 28], [27, 29], [33, 35], [34, 36], [35, 37], [36, 38], [37, 39], [38, 40], [39, 41], [40, 42], [41, 43], [42, 44]]
]
For example, first element [[7, 12], [6, 12], [38, 44], [25, 30]] contains 4 ranges 7-12, 6-12, 38-44 and 25-30 etc.
I need to find the all possible chains (a chain is an array of consecutive ranges where ending of first range == beginning of next range) of length given list length, given that I could and should take only one range from each row in the exact order of rows.
So, for this example list:
The chains would be
[[6, 12], [12, 16], [16, 18]],
[[7, 12], [12, 16], [16, 18]],
[[25, 30], [30, 33], [33, 35]]
and [[25, 29], [29, 33], [33, 35]]
Right now I am stuck on working with more than three length list, could not come up with recursive solution.
You can use itertools.product to iterate over all possible chains (all combinations of 1 range from each "row"),
then filter them by a simple function that checks if a specific chain is legal.
try this:
from itertools import product
def check_chain(chain):
prev_end = chain[0][1]
for start, end in chain[1:]:
if start != prev_end:
return False
prev_end = end
return True
all_candidate_chains = product(*list)
result = [[*chain] for chain in all_candidate_chains if check_chain(chain)]
print(result)
Output:
[[[7, 12], [12, 16], [16, 18]], [[6, 12], [12, 16], [16, 18]], [[25, 30], [30, 33], [33, 35]]]
EDIT:
can also use zip and all to replace check_chain with a 1-liner:
from itertools import product
result = [[*chain] for chain in product(*list) if all(end1 == start2 for (_, end1), (start2, _) in zip(chain, chain[1:]))]
print(result)
You can do this without looking at all the permutation. Start at with the last item and make a dictionary where the keys are the first value in the dictionary. Then work backward through the list and lookup the previous key based on the second value of the tuple adding to the array as you go:
In the end you'll have a dictionary keyed to the first value in the tuples of the first list. You can just flatten the values at this point.
Here I added one more pair [12,9] to the middle list so I could see it work with branching paths:
from collections import defaultdict
from itertools import chain
l = [
[[7, 12], [6, 12], [38, 44], [25, 30]],
[[0, 5], [1, 5], [2, 5], [12, 16], [12, 9],[13, 16], [20, 23], [29, 33], [30, 33]],
[[5, 7], [6, 8], [7, 9], [8, 10], [9, 11], [10, 12], [16, 18], [17, 19], [18, 20], [23, 25], [24, 26], [25, 27], [26, 28], [27, 29], [33, 35], [34, 36], [35, 37], [36, 38], [37, 39], [38, 40], [39, 41], [40, 42], [41, 43], [42, 44]]
]
d = defaultdict(list)
for k, v in l[-1]:
d[k].append([[k,v]])
for sub in reversed(l[:-1]):
ds = defaultdict(list)
for k, v in sub:
if v in d:
ds[k].extend([[k,v], *v2] for v2 in d[v] )
d = ds
list(chain.from_iterable(d.values()))
Output:
[[[7, 12], [12, 16], [16, 18]],
[[7, 12], [12, 9], [9, 11]],
[[6, 12], [12, 16], [16, 18]],
[[6, 12], [12, 9], [9, 11]],
[[25, 30], [30, 33], [33, 35]]]
These is a function in tensorflow called tf.space_to_depth. The implementation of this function in Tensorflow source code is so difficult for me. Could you please help me to implement it using numpy?
Here is some code to visualize how this function works. By the way, before everything, it is good to mention that the input to tensorflow's function should have input shape : [batch, height, width, depth]
Assuming this code. First we need to define a tensor:
norm = tf.reshape(tf.range(0,72),(1,6,6,2))
Here is the value of depth 1 (norm[0,:,:,0]):
[[ 0, 2, 4, 6, 8, 10],
[12, 14, 16, 18, 20, 22],
[24, 26, 28, 30, 32, 34],
[36, 38, 40, 42, 44, 46],
[48, 50, 52, 54, 56, 58],
[60, 62, 64, 66, 68, 70]]
Here is the value of depth 2 (norm[0,:,:,1]):
[[ 1, 3, 5, 7, 9, 11],
[13, 15, 17, 19, 21, 23],
[25, 27, 29, 31, 33, 35],
[37, 39, 41, 43, 45, 47],
[49, 51, 53, 55, 57, 59],
[61, 63, 65, 67, 69, 71]]
In the next step, I would like to apply tf.space_to_depth function and here it is:
trans = tf.space_to_depth(norm,2)
Output shape is : (1,3,3,8) and here is the output of this function:
trans[0,:,:,0]
[[ 0, 4, 8],
[24, 28, 32],
[48, 52, 56]]
trans[0,:,:,1]
[[ 1, 5, 9],
[25, 29, 33],
[49, 53, 57]]
trans[0,:,:,2]
[[ 2, 6, 10],
[26, 30, 34],
[50, 54, 58]]
trans[0,:,:,3]
[[ 3, 7, 11],
[27, 31, 35],
[51, 55, 59]]
trans[0,:,:,4]
[[12, 16, 20],
[36, 40, 44],
[60, 64, 68]]
trans[0,:,:,5]
[[13, 17, 21],
[37, 41, 45],
[61, 65, 69]]
trans[0,:,:,6]
[[14, 18, 22],
[38, 42, 46],
[62, 66, 70]]
trans[0,:,:,7]
[[15, 19, 23],
[39, 43, 47],
[63, 67, 71]]
May someone help me how can I implement a vectorized version of this function in numpy?
Appreciating in advance for any response!
You can implement space_to_depth with appropriate calls to the reshape() and swapaxes() functions:
import numpy as np
def space_to_depth(x, block_size):
x = np.asarray(x)
batch, height, width, depth = x.shape
reduced_height = height // block_size
reduced_width = width // block_size
y = x.reshape(batch, reduced_height, block_size,
reduced_width, block_size, depth)
z = np.swapaxes(y, 2, 3).reshape(batch, reduced_height, reduced_width, -1)
return z
Here are the examples from the documentation of tf.space_to_depth:
In [328]: x = [[[[1], [2]],
...: [[3], [4]]]]
...:
In [329]: space_to_depth(x, 2)
Out[329]: array([[[[1, 2, 3, 4]]]])
In [330]: x = [[[[1, 2, 3], [4, 5, 6]],
...: [[7, 8, 9], [10, 11, 12]]]]
...:
In [331]: space_to_depth(x, 2)
Out[331]: array([[[[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]]]])
In [332]: x = [[[[1], [2], [5], [6]],
...: [[3], [4], [7], [8]],
...: [[9], [10], [13], [14]],
...: [[11], [12], [15], [16]]]]
...:
In [333]: space_to_depth(x, 2)
Out[333]:
array([[[[ 1, 2, 3, 4],
[ 5, 6, 7, 8]],
[[ 9, 10, 11, 12],
[13, 14, 15, 16]]]])
And here is your example:
In [334]: norm = np.arange(72).reshape(1, 6, 6, 2)
In [335]: trans = space_to_depth(norm, 2)
In [336]: trans[0, :, :, 0]
Out[336]:
array([[ 0, 4, 8],
[24, 28, 32],
[48, 52, 56]])
In [337]: trans[0, :, :, 1]
Out[337]:
array([[ 1, 5, 9],
[25, 29, 33],
[49, 53, 57]])
In [338]: trans[0, :, :, 7]
Out[338]:
array([[15, 19, 23],
[39, 43, 47],
[63, 67, 71]])
I am working with images through numpy. I want to set a chunk of the image to its average color. I am able to do this, but I have to re-index the array, when I would like to use the original view to do this. In other words, I would like to use that 4th line of code, but I'm stuck with the 3rd one.
I have read a few posts about the as_strided function, but it is confusing to me, and I was hoping there might be a simpler solution. So is there a way to slightly modify that last line of code to do what I want?
box = im[x-dx:x+dx, y-dy:y+dy, :]
avg = block(box) #returns a 1D numpy array with 3 values
im[x-dx:x+dx, y-dy:y+dy, :] = avg[None,None,:] #sets box to average color
#box = avg[None,None,:] #does not affect original array
box = blah
just reassigns the box variable. The array that the box variable previously referred to is unaffected. This is not what you want.
box[:] = blah
is a slice assignment. It modifies the contents of the array. This is what you want.
Note that slice assignment is dependent on the syntactic form of the statement. The fact that box was assigned by box = im[stuff] does not make further assignments to box slice assignments. This is similar to how if you do
l = [1, 2, 3]
b = l[2]
b = 0
the assignment to b doesn't affect l.
Gray-scale Images
This will set a chunk of an array to its average (mean) value:
im[2:4, 2:4] = im[2:4, 2:4].mean()
For example:
In [9]: im
Out[9]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
In [10]: im[2:4, 2:4] = im[2:4, 2:4].mean()
In [11]: im
Out[11]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 12, 12],
[12, 13, 12, 12]])
Color Images
Suppose that we want to average over each component of color separately:
In [22]: im = np.arange(48).reshape((4,4,3))
In [23]: im
Out[23]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]],
[[36, 37, 38],
[39, 40, 41],
[42, 43, 44],
[45, 46, 47]]])
In [24]: im[2:4, 2:4, :] = im[2:4, 2:4, :].mean(axis=0).mean(axis=0)[np.newaxis, np.newaxis, :]
In [25]: im
Out[25]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[37, 38, 39],
[37, 38, 39]],
[[36, 37, 38],
[39, 40, 41],
[37, 38, 39],
[37, 38, 39]]])