Related
I have the tensors:
ids: shape (7000,1) containing indices like [[1],[0],[2],...]
x: shape(7000,3,255)
ids tensor encodes the index of bold marked dimension of x which should be selected.
I want to gather the selected slices in a resulting vector:
result: shape (7000,255)
Background:
I have some scores (shape = (7000,3)) for each of the 3 elements and want only to select the one with the highest score. Therefore, I used the function
ids = torch.argmax(scores,1,True)
giving me the maximum ids. I already tried to do it with gather function:
result = x.gather(1,ids)
but that didn't work.
Here is a solution you may look for
ids = ids.repeat(1, 255).view(-1, 1, 255)
An example as below:
x = torch.arange(24).view(4, 3, 2)
"""
tensor([[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]],
[[12, 13],
[14, 15],
[16, 17]],
[[18, 19],
[20, 21],
[22, 23]]])
"""
ids = torch.randint(0, 3, size=(4, 1))
"""
tensor([[0],
[2],
[0],
[2]])
"""
idx = ids.repeat(1, 2).view(4, 1, 2)
"""
tensor([[[0, 0]],
[[2, 2]],
[[0, 0]],
[[2, 2]]])
"""
torch.gather(x, 1, idx)
"""
tensor([[[ 0, 1]],
[[10, 11]],
[[12, 13]],
[[22, 23]]])
"""
using the example of David Ng I found out another way to do it:
idx = ids.flatten() + torch.arange(0,4*3,3)
tensor([ 0, 5, 6, 11])
x.view(-1,2)[idx]
tensor([[ 0, 1],
[10, 11],
[12, 13],
[22, 23]])
Another solution may provide better memory read pattern in cases where the dimensions are higher.
# data
x = torch.arange(60).reshape(3, 4, 5)
# index
y = torch.randint(0, 4, (12,), dtype=torch.int64).reshape(3, 4)
# result
z = x[torch.arange(x.shape[0]).repeat_interleave(x.shape[1]), y.flatten()]
z = z.reshape(x.shape)
An example result of the x, y, z will be
Tensor([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
tensor([[1, 1, 2, 3],
[3, 1, 1, 0],
[1, 1, 1, 1]])
tensor([[[ 5, 6, 7, 8, 9],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[35, 36, 37, 38, 39],
[25, 26, 27, 28, 29],
[25, 26, 27, 28, 29],
[20, 21, 22, 23, 24]],
[[45, 46, 47, 48, 49],
[45, 46, 47, 48, 49],
[45, 46, 47, 48, 49],
[45, 46, 47, 48, 49]]])
I have a 2d array
test = np.arange(25).reshape((5, 5))
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
for which I need a "checkboard" like selector. Basically, I want to select odd elements in the first row, even elements in the second row, odd elements in the third row etc..
In this case (because the row numbers are odd) I can get that via
test.flatten()[::2]
Out[22]: array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24])
which is exactly "every other element" in both dimensions. But if we try that with
test2 = np.arange(16).reshape((4, 4))
test2
Out[23]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
test2.flatten()[::2]
Out[27]: array([ 0, 2, 4, 6, 8, 10, 12, 14])
unsurprisingly, it doesn't work out.
What's a general way of generating the type of selection that I want, no matter the shape of the (2d) matrix? Is it easily (or even possible) to extend to 3d matrices?
I do prefer efficient (vectorized or cython-based) approaches.
Bonus: 3D:
test3 = np.arange(25*3).reshape((3, 5, 5))
Out[30]:
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]],
[[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39],
[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49]],
[[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59],
[60, 61, 62, 63, 64],
[65, 66, 67, 68, 69],
[70, 71, 72, 73, 74]]])
Here it's not about odd and even rows, but rather about neighboring elements. No two neighboring elements should have the same color (be part of the same bool selector).
That is, if we select test3[0, ...].flatten()[::2] we get [0, ... 24]. The neighboring element of 0 in the first dimension is 25, so we don't want that. The expected output is
np.hstack((test3[0, ...].flatten()[::2], test3[1, ...].flatten()[1::2], test3[2, ...].flatten()[::2], ))
Out[42]:
array([ 0, 2, 4, 6, 8, 10, 12, 14, 17, 19, 21, 23, 25, 27, 29, 31, 32,
34, 36, 38, 40, 42, 44, 46])
Here's one vectorized solution -
def checker_select(a):
m,n = a.shape[-2:]
i,j = np.ogrid[:m,:n]
mask = (i+j)%2==0
return a.reshape(-1,m,n)[:,mask].ravel()
Probably a faster one with array-initialization -
def checker_select_v2(a):
m,n = a.shape[-2:]
mask = np.zeros((m,n), dtype=bool)
mask[::2,::2] = 1
mask[1::2,1::2] = 1
return a.reshape(-1,m,n)[:,mask].ravel()
Sample run for 2D -
In [117]: a
Out[117]:
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19],
[20, 21, 22, 23, 24]])
In [118]: checker_select(a)
Out[118]: array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24])
Sample run for 3D -
In [144]: a
Out[144]:
array([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[ 10, 11, 12, 13, 14],
[ 15, 16, 17, 18, 19],
[ 20, 21, 22, 23, 24]],
[[ 25, 26, 27, 28, 29],
[ 30, 31, 32, 33, 34],
[ 35, 36, 37, 38, 39],
[ 40, 41, 42, 43, 44],
[ 45, 46, 47, 48, 49]],
....
In [145]: checker_select(a)
Out[145]:
array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24,
25, 27, 29, 31, 33, 35, 37, 39, 41, 43, 45, 47, 49,
....
You can do that in plain old python
mat = np.arange(16).reshape((4, 4))
result = []
for row in range(len(mat)):
for col in range(len(mat[row])):
if row % 2 == 0 and col % 2 == 0:
result.append(mat[row][col])
elif row % 2 == 1 and col % 2 == 1:
result.append(mat[row][col])
print(result)
Output
[0, 2, 5, 7, 8, 10, 13, 15]
If you are looking for a more pythonic way.
n = 6
m = 4
mat = np.arange(n * m).reshape((n, m))
flat = mat.flatten()
result = [flat[i] for i in range(len(flat)) if i // m % 2 == i % m % 2]
print(result)
Output
[0, 2, 5, 7, 8, 10, 13, 15, 16, 18, 21, 23]
Example using basic lists and plain python.
matrix = [
[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]
]
result = (row[i % 2::2] for i, row in enumerate(matrix))
flattened_result = [cell for filtered_row in result for cell in filtered_row]
print(flattened_result)
I have a big two-dimensional array like this:
array([[ 1, 2, 3, 4, 5, 6, 7, 8],
[ 9,10,11,12,13,14,15,16],
[17,18,19,20,21,22,23,24],
[25,26,27,28,29,30,31,32],
[33,34,35,36,37,38,39,40],
[41,42,43,44,45,46,47,48],
....])
and I need to convert it into:
array([ 1, 9,17, 2,10,18, 3,11,19, 4,12,20, 5,13,21, 6,14,22, 7,15,23, 8,16,24],
[25,33,41,26,34,42,27,35,43,28,36,44,29,37,45,30,38,46,31,39,47,32,40,48],
...
Note that this should only be a demonstration what it should do.
The original array contains only boolean values and has the size of 512x8. In my example, I order only 3 rows with 8 elements into one row but what I really need are respectively 32 rows with 8 elements.
I am really sorry, but after 30 minutes of writing, this is the only description I got of my problem. I hope it is enough.
I think you can achieve your desired result using two reshape operations and a transpose:
x = np.array([[ 1, 2, 3, 4, 5, 6, 7, 8],
[ 9,10,11,12,13,14,15,16],
[17,18,19,20,21,22,23,24],
[25,26,27,28,29,30,31,32],
[33,34,35,36,37,38,39,40],
[41,42,43,44,45,46,47,48]])
y = x.reshape(2, 3, 8).transpose(0, 2, 1).reshape(2, -1)
print(repr(y))
# array([[ 1, 9, 17, 2, 10, 18, 3, 11, 19, 4, 12, 20, 5, 13, 21, 6, 14,
# 22, 7, 15, 23, 8, 16, 24],
# [25, 33, 41, 26, 34, 42, 27, 35, 43, 28, 36, 44, 29, 37, 45, 30, 38,
# 46, 31, 39, 47, 32, 40, 48]])
To break that down a bit:
#hpaulj's first reshape operation gives us this:
x1 = x.reshape(2, 3, 8)
print(repr(x1))
# array([[[ 1, 2, 3, 4, 5, 6, 7, 8],
# [ 9, 10, 11, 12, 13, 14, 15, 16],
# [17, 18, 19, 20, 21, 22, 23, 24]],
# [[25, 26, 27, 28, 29, 30, 31, 32],
# [33, 34, 35, 36, 37, 38, 39, 40],
# [41, 42, 43, 44, 45, 46, 47, 48]]])
print(x1.shape)
# (2, 3, 8)
In order to get the desired output we need to 'collapse' this array along the second dimension (with size 3), then along the third dimension (with size 8).
The easiest way to achieve this sort of thing is to first transpose the
array so that the dimensions you want to collapse along are ordered from first to last:
x2 = x1.transpose(0, 2, 1) # you could also use `x2 = np.rollaxis(x1, 1, 3)`
print(repr(x2))
# array([[[ 1, 9, 17],
# [ 2, 10, 18],
# [ 3, 11, 19],
# [ 4, 12, 20],
# [ 5, 13, 21],
# [ 6, 14, 22],
# [ 7, 15, 23],
# [ 8, 16, 24]],
# [[25, 33, 41],
# [26, 34, 42],
# [27, 35, 43],
# [28, 36, 44],
# [29, 37, 45],
# [30, 38, 46],
# [31, 39, 47],
# [32, 40, 48]]])
print(x2.shape)
# (2, 8, 3)
Finally I can use reshape(2, -1) to collapse the array over the last two dimensions. The -1 causes numpy to infer the appropriate size in the last dimension based on the number of elements in x.
y = x2.reshape(2, -2)
Looks like a starting point is to reshape it, for example
In [49]: x.reshape(2,3,8)
Out[49]:
array([[[ 1, 2, 3, 4, 5, 6, 7, 8],
[ 9, 10, 11, 12, 13, 14, 15, 16],
[17, 18, 19, 20, 21, 22, 23, 24]],
[[25, 26, 27, 28, 29, 30, 31, 32],
[33, 34, 35, 36, 37, 38, 39, 40],
[41, 42, 43, 44, 45, 46, 47, 48]]])
.ravel(order='F') doesn't get it right, so I think we need to swap some axes before flattening. It will need to be a copy.
Using #ali_m's transpose:
In [65]: x1=x.reshape(2,3,8)
In [66]: x1.transpose(0,2,1).flatten()
Out[66]:
array([ 1, 9, 17, 2, 10, 18, 3, 11, 19, 4, 12, 20, 5, 13, 21, 6, 14,
22, 7, 15, 23, 8, 16, 24, 25, 33, 41, 26, 34, 42, 27, 35, 43, 28,
36, 44, 29, 37, 45, 30, 38, 46, 31, 39, 47, 32, 40, 48])
oops - there's an inner layer of nesting that's easy to miss
array([1,9,17,2,10,18,3,11,19,4,12,20,5,13,21,6,14,22,7,15,23,8,16,24],
[25,33,41,26,34,42,27,35,43,28,36,44,29,37,45,30,38,46,31,39,47,32,40,4],
You are missing a [] set. So #ali_m got it right.
I'm tempted to delete this, but my trial and error might be instructive.
I have a 4x4 array and wonder if there's a way to randomly sample a 2x2 square from it at any location, allowing for the square to wrapped around when it reaches an edge.
For example:
>> A = np.arange(16).reshape(4,-1)
>> start_point = A[0,3]
start_point = 3
the square would be [[15, 12], [3,0]]
I've generalized (a little...) my answer, permitting a rectangular input array
and even a rectangular random sample
def rand_sample(arr, nr, nc):
"sample a nr x nc submatrix starting from a random element of arr, with wrap"
r, c = arr.shape
i, j = np.random.randint(r), np.random.randint(c)
r = np.arange(i, i+nr) % r
c = np.arange(j, j+nc) % c
return arr[r][:,c]
You may want to check that arr is a 2D array
You could use np.lib.pad with 'wrap' option to have a padded version of the input array that has wrapped around elements at the end along the rows and columns. Finally, index into the padded array to get the desired output. Here's the implementation -
import numpy as np
# Input array and start row-col indices and neighbourhood extent
A = np.arange(42).reshape(6,-1)
start_pt = [0,3] # start_point
N = 5 # neighbourhood extent
# Pad boundary with wrapped elements
A_pad = np.lib.pad(A, ((0,N-1),(0,N-1)), 'wrap')
# Final output after indexing into padded array
out = A_pad[start_pt[0]:start_pt[0]+N,start_pt[1]:start_pt[1]+N]
Sample run -
In [192]: A
Out[192]:
array([[ 0, 1, 2, 3, 4, 5, 6],
[ 7, 8, 9, 10, 11, 12, 13],
[14, 15, 16, 17, 18, 19, 20],
[21, 22, 23, 24, 25, 26, 27],
[28, 29, 30, 31, 32, 33, 34],
[35, 36, 37, 38, 39, 40, 41]])
In [193]: start_pt
Out[193]: [0, 3]
In [194]: A_pad
Out[194]:
array([[ 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3],
[ 7, 8, 9, 10, 11, 12, 13, 7, 8, 9, 10],
[14, 15, 16, 17, 18, 19, 20, 14, 15, 16, 17],
[21, 22, 23, 24, 25, 26, 27, 21, 22, 23, 24],
[28, 29, 30, 31, 32, 33, 34, 28, 29, 30, 31],
[35, 36, 37, 38, 39, 40, 41, 35, 36, 37, 38],
[ 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3],
[ 7, 8, 9, 10, 11, 12, 13, 7, 8, 9, 10],
[14, 15, 16, 17, 18, 19, 20, 14, 15, 16, 17],
[21, 22, 23, 24, 25, 26, 27, 21, 22, 23, 24]])
In [195]: out
Out[195]:
array([[ 3, 4, 5, 6, 0],
[10, 11, 12, 13, 7],
[17, 18, 19, 20, 14],
[24, 25, 26, 27, 21],
[31, 32, 33, 34, 28]])
I am working with images through numpy. I want to set a chunk of the image to its average color. I am able to do this, but I have to re-index the array, when I would like to use the original view to do this. In other words, I would like to use that 4th line of code, but I'm stuck with the 3rd one.
I have read a few posts about the as_strided function, but it is confusing to me, and I was hoping there might be a simpler solution. So is there a way to slightly modify that last line of code to do what I want?
box = im[x-dx:x+dx, y-dy:y+dy, :]
avg = block(box) #returns a 1D numpy array with 3 values
im[x-dx:x+dx, y-dy:y+dy, :] = avg[None,None,:] #sets box to average color
#box = avg[None,None,:] #does not affect original array
box = blah
just reassigns the box variable. The array that the box variable previously referred to is unaffected. This is not what you want.
box[:] = blah
is a slice assignment. It modifies the contents of the array. This is what you want.
Note that slice assignment is dependent on the syntactic form of the statement. The fact that box was assigned by box = im[stuff] does not make further assignments to box slice assignments. This is similar to how if you do
l = [1, 2, 3]
b = l[2]
b = 0
the assignment to b doesn't affect l.
Gray-scale Images
This will set a chunk of an array to its average (mean) value:
im[2:4, 2:4] = im[2:4, 2:4].mean()
For example:
In [9]: im
Out[9]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
In [10]: im[2:4, 2:4] = im[2:4, 2:4].mean()
In [11]: im
Out[11]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 12, 12],
[12, 13, 12, 12]])
Color Images
Suppose that we want to average over each component of color separately:
In [22]: im = np.arange(48).reshape((4,4,3))
In [23]: im
Out[23]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]],
[[36, 37, 38],
[39, 40, 41],
[42, 43, 44],
[45, 46, 47]]])
In [24]: im[2:4, 2:4, :] = im[2:4, 2:4, :].mean(axis=0).mean(axis=0)[np.newaxis, np.newaxis, :]
In [25]: im
Out[25]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[37, 38, 39],
[37, 38, 39]],
[[36, 37, 38],
[39, 40, 41],
[37, 38, 39],
[37, 38, 39]]])