Related
I have the following numpy 3d array, in which I need to duplicate the last column
array([[[ 7, 5, 93],
[19, 4, 69],
[62, 2, 52]],
[[ 6, 1, 65],
[41, 9, 94],
[39, 4, 49]]])
The desired output is:
array([[[ 7, 5, 93, 93],
[19, 4, 69, 69],
[62, 2, 52, 52]],
[[ 6, 1, 65, 65],
[41, 9, 94, 94],
[39, 4, 49, 49]]])
Is there a clever way of doing this?
You could concatenate along the last axis as follows-
numpy.concatenate([a, numpy.expand_dims(a[:, :, -1], axis=2)], axis=2)
There is a built-in numpy function for this purpose:
np.insert(x,-1,x[...,-1],-1)
output:
array([[[ 7, 5, 93, 93],
[19, 4, 69, 69],
[62, 2, 52, 52]],
[[ 6, 1, 65, 65],
[41, 9, 94, 94],
[39, 4, 49, 49]]])
I have the tensors:
ids: shape (7000,1) containing indices like [[1],[0],[2],...]
x: shape(7000,3,255)
ids tensor encodes the index of bold marked dimension of x which should be selected.
I want to gather the selected slices in a resulting vector:
result: shape (7000,255)
Background:
I have some scores (shape = (7000,3)) for each of the 3 elements and want only to select the one with the highest score. Therefore, I used the function
ids = torch.argmax(scores,1,True)
giving me the maximum ids. I already tried to do it with gather function:
result = x.gather(1,ids)
but that didn't work.
Here is a solution you may look for
ids = ids.repeat(1, 255).view(-1, 1, 255)
An example as below:
x = torch.arange(24).view(4, 3, 2)
"""
tensor([[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]],
[[12, 13],
[14, 15],
[16, 17]],
[[18, 19],
[20, 21],
[22, 23]]])
"""
ids = torch.randint(0, 3, size=(4, 1))
"""
tensor([[0],
[2],
[0],
[2]])
"""
idx = ids.repeat(1, 2).view(4, 1, 2)
"""
tensor([[[0, 0]],
[[2, 2]],
[[0, 0]],
[[2, 2]]])
"""
torch.gather(x, 1, idx)
"""
tensor([[[ 0, 1]],
[[10, 11]],
[[12, 13]],
[[22, 23]]])
"""
using the example of David Ng I found out another way to do it:
idx = ids.flatten() + torch.arange(0,4*3,3)
tensor([ 0, 5, 6, 11])
x.view(-1,2)[idx]
tensor([[ 0, 1],
[10, 11],
[12, 13],
[22, 23]])
Another solution may provide better memory read pattern in cases where the dimensions are higher.
# data
x = torch.arange(60).reshape(3, 4, 5)
# index
y = torch.randint(0, 4, (12,), dtype=torch.int64).reshape(3, 4)
# result
z = x[torch.arange(x.shape[0]).repeat_interleave(x.shape[1]), y.flatten()]
z = z.reshape(x.shape)
An example result of the x, y, z will be
Tensor([[[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[20, 21, 22, 23, 24],
[25, 26, 27, 28, 29],
[30, 31, 32, 33, 34],
[35, 36, 37, 38, 39]],
[[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])
tensor([[1, 1, 2, 3],
[3, 1, 1, 0],
[1, 1, 1, 1]])
tensor([[[ 5, 6, 7, 8, 9],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]],
[[35, 36, 37, 38, 39],
[25, 26, 27, 28, 29],
[25, 26, 27, 28, 29],
[20, 21, 22, 23, 24]],
[[45, 46, 47, 48, 49],
[45, 46, 47, 48, 49],
[45, 46, 47, 48, 49],
[45, 46, 47, 48, 49]]])
These is a function in tensorflow called tf.space_to_depth. The implementation of this function in Tensorflow source code is so difficult for me. Could you please help me to implement it using numpy?
Here is some code to visualize how this function works. By the way, before everything, it is good to mention that the input to tensorflow's function should have input shape : [batch, height, width, depth]
Assuming this code. First we need to define a tensor:
norm = tf.reshape(tf.range(0,72),(1,6,6,2))
Here is the value of depth 1 (norm[0,:,:,0]):
[[ 0, 2, 4, 6, 8, 10],
[12, 14, 16, 18, 20, 22],
[24, 26, 28, 30, 32, 34],
[36, 38, 40, 42, 44, 46],
[48, 50, 52, 54, 56, 58],
[60, 62, 64, 66, 68, 70]]
Here is the value of depth 2 (norm[0,:,:,1]):
[[ 1, 3, 5, 7, 9, 11],
[13, 15, 17, 19, 21, 23],
[25, 27, 29, 31, 33, 35],
[37, 39, 41, 43, 45, 47],
[49, 51, 53, 55, 57, 59],
[61, 63, 65, 67, 69, 71]]
In the next step, I would like to apply tf.space_to_depth function and here it is:
trans = tf.space_to_depth(norm,2)
Output shape is : (1,3,3,8) and here is the output of this function:
trans[0,:,:,0]
[[ 0, 4, 8],
[24, 28, 32],
[48, 52, 56]]
trans[0,:,:,1]
[[ 1, 5, 9],
[25, 29, 33],
[49, 53, 57]]
trans[0,:,:,2]
[[ 2, 6, 10],
[26, 30, 34],
[50, 54, 58]]
trans[0,:,:,3]
[[ 3, 7, 11],
[27, 31, 35],
[51, 55, 59]]
trans[0,:,:,4]
[[12, 16, 20],
[36, 40, 44],
[60, 64, 68]]
trans[0,:,:,5]
[[13, 17, 21],
[37, 41, 45],
[61, 65, 69]]
trans[0,:,:,6]
[[14, 18, 22],
[38, 42, 46],
[62, 66, 70]]
trans[0,:,:,7]
[[15, 19, 23],
[39, 43, 47],
[63, 67, 71]]
May someone help me how can I implement a vectorized version of this function in numpy?
Appreciating in advance for any response!
You can implement space_to_depth with appropriate calls to the reshape() and swapaxes() functions:
import numpy as np
def space_to_depth(x, block_size):
x = np.asarray(x)
batch, height, width, depth = x.shape
reduced_height = height // block_size
reduced_width = width // block_size
y = x.reshape(batch, reduced_height, block_size,
reduced_width, block_size, depth)
z = np.swapaxes(y, 2, 3).reshape(batch, reduced_height, reduced_width, -1)
return z
Here are the examples from the documentation of tf.space_to_depth:
In [328]: x = [[[[1], [2]],
...: [[3], [4]]]]
...:
In [329]: space_to_depth(x, 2)
Out[329]: array([[[[1, 2, 3, 4]]]])
In [330]: x = [[[[1, 2, 3], [4, 5, 6]],
...: [[7, 8, 9], [10, 11, 12]]]]
...:
In [331]: space_to_depth(x, 2)
Out[331]: array([[[[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]]]])
In [332]: x = [[[[1], [2], [5], [6]],
...: [[3], [4], [7], [8]],
...: [[9], [10], [13], [14]],
...: [[11], [12], [15], [16]]]]
...:
In [333]: space_to_depth(x, 2)
Out[333]:
array([[[[ 1, 2, 3, 4],
[ 5, 6, 7, 8]],
[[ 9, 10, 11, 12],
[13, 14, 15, 16]]]])
And here is your example:
In [334]: norm = np.arange(72).reshape(1, 6, 6, 2)
In [335]: trans = space_to_depth(norm, 2)
In [336]: trans[0, :, :, 0]
Out[336]:
array([[ 0, 4, 8],
[24, 28, 32],
[48, 52, 56]])
In [337]: trans[0, :, :, 1]
Out[337]:
array([[ 1, 5, 9],
[25, 29, 33],
[49, 53, 57]])
In [338]: trans[0, :, :, 7]
Out[338]:
array([[15, 19, 23],
[39, 43, 47],
[63, 67, 71]])
I have looked into documentations and also other questions here, but it seems I
have not got the hang of subsetting in numpy arrays yet.
I have a numpy array,
and for the sake of argument, let it be defined as follows:
import numpy as np
a = np.arange(100)
a.shape = (10,10)
# array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
# [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
# [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
# [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
# [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
# [50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
# [60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
# [70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
# [80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
# [90, 91, 92, 93, 94, 95, 96, 97, 98, 99]])
now I want to choose rows and columns of a specified by vectors n1 and n2. As an example:
n1 = range(5)
n2 = range(5)
But when I use:
b = a[n1,n2]
# array([ 0, 11, 22, 33, 44])
Then only the first fifth diagonal elements are chosen, not the whole 5x5 block. The solution I have found is to do it like this:
b = a[n1,:]
b = b[:,n2]
# array([[ 0, 1, 2, 3, 4],
# [10, 11, 12, 13, 14],
# [20, 21, 22, 23, 24],
# [30, 31, 32, 33, 34],
# [40, 41, 42, 43, 44]])
But I am sure there should be a way to do this simple task in just one command.
You've gotten a handful of nice examples of how to do what you want. However, it's also useful to understand the what's happening and why things work the way they do. There are a few simple rules that will help you in the future.
There's a big difference between "fancy" indexing (i.e. using a list/sequence) and "normal" indexing (using a slice). The underlying reason has to do with whether or not the array can be "regularly strided", and therefore whether or not a copy needs to be made. Arbitrary sequences therefore have to be treated differently, if we want to be able to create "views" without making copies.
In your case:
import numpy as np
a = np.arange(100).reshape(10,10)
n1, n2 = np.arange(5), np.arange(5)
# Not what you want
b = a[n1, n2] # array([ 0, 11, 22, 33, 44])
# What you want, but only for simple sequences
# Note that no copy of *a* is made!! This is a view.
b = a[:5, :5]
# What you want, but probably confusing at first. (Also, makes a copy.)
# np.meshgrid and np.ix_ are basically equivalent to this.
b = a[n1[:,None], n2[None,:]]
Fancy indexing with 1D sequences is basically equivalent to zipping them together and indexing with the result.
print "Fancy Indexing:"
print a[n1, n2]
print "Manual indexing:"
for i, j in zip(n1, n2):
print a[i, j]
However, if the sequences you're indexing with match the dimensionality of the array you're indexing (2D, in this case), The indexing is treated differently. Instead of "zipping the two together", numpy uses the indices like a mask.
In other words, a[[[1, 2, 3]], [[1],[2],[3]]] is treated completely differently than a[[1, 2, 3], [1, 2, 3]], because the sequences/arrays that you're passing in are two-dimensional.
In [4]: a[[[1, 2, 3]], [[1],[2],[3]]]
Out[4]:
array([[11, 21, 31],
[12, 22, 32],
[13, 23, 33]])
In [5]: a[[1, 2, 3], [1, 2, 3]]
Out[5]: array([11, 22, 33])
To be a bit more precise,
a[[[1, 2, 3]], [[1],[2],[3]]]
is treated exactly like:
i = [[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
j = [[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]
a[i, j]
In other words, whether the input is a row/column vector is a shorthand for how the indices should repeat in the indexing.
np.meshgrid and np.ix_ are just convienent ways to turn your 1D sequences into their 2D versions for indexing:
In [6]: np.ix_([1, 2, 3], [1, 2, 3])
Out[6]:
(array([[1],
[2],
[3]]), array([[1, 2, 3]]))
Similarly (the sparse argument would make it identical to ix_ above):
In [7]: np.meshgrid([1, 2, 3], [1, 2, 3], indexing='ij')
Out[7]:
[array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]]),
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])]
Another quick way to build the desired index is to use the np.ix_ function:
>>> a[np.ix_([n1, n2])]
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44]])
This provides a convenient way to construct an open mesh from sequences of indices.
You could use np.meshgrid to give the n1, n2 arrays the proper shape to perform the desired indexing:
In [104]: a[np.meshgrid(n1,n2, sparse=True, indexing='ij')]
Out[104]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44]])
Or, without meshgrid:
In [117]: a[np.array(n1)[:,np.newaxis], np.array(n2)[np.newaxis,:]]
Out[117]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44]])
There is a similar example with an explanation of how this integer array indexing works in the docs.
See also the Cookbook recipe Picking out rows and columns.
A nice Trick I've managed to pull (for lazy people only)
Is filter + Transpose + filter.
a = np.arange(100).reshape(10,10)
subsetA = [1,3,5,7]
a[subsetA].T[subsetA]
array([[11, 31, 51, 71],
[13, 33, 53, 73],
[15, 35, 55, 75],
[17, 37, 57, 77]])
It seems that a use case for your particular question would deal with image manipulation. To the extent that you are using your example to edit numpy arrays arising from images, you can use the Python Imaging Library (PIL).
# Import Pillow:
from PIL import Image
# Load the original image:
img = Image.open("flowers.jpg")
# Crop the image
img2 = img.crop((0, 0, 5, 5))
The img2 object is a numpy array of the resulting cropped image.
You can read more about image manipulation here with the Pillow package (a user friendly fork on the PIL package):
I have looked into documentations and also other questions here, but it seems I
have not got the hang of subsetting in numpy arrays yet.
I have a numpy array,
and for the sake of argument, let it be defined as follows:
import numpy as np
a = np.arange(100)
a.shape = (10,10)
# array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
# [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
# [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
# [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
# [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
# [50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
# [60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
# [70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
# [80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
# [90, 91, 92, 93, 94, 95, 96, 97, 98, 99]])
now I want to choose rows and columns of a specified by vectors n1 and n2. As an example:
n1 = range(5)
n2 = range(5)
But when I use:
b = a[n1,n2]
# array([ 0, 11, 22, 33, 44])
Then only the first fifth diagonal elements are chosen, not the whole 5x5 block. The solution I have found is to do it like this:
b = a[n1,:]
b = b[:,n2]
# array([[ 0, 1, 2, 3, 4],
# [10, 11, 12, 13, 14],
# [20, 21, 22, 23, 24],
# [30, 31, 32, 33, 34],
# [40, 41, 42, 43, 44]])
But I am sure there should be a way to do this simple task in just one command.
You've gotten a handful of nice examples of how to do what you want. However, it's also useful to understand the what's happening and why things work the way they do. There are a few simple rules that will help you in the future.
There's a big difference between "fancy" indexing (i.e. using a list/sequence) and "normal" indexing (using a slice). The underlying reason has to do with whether or not the array can be "regularly strided", and therefore whether or not a copy needs to be made. Arbitrary sequences therefore have to be treated differently, if we want to be able to create "views" without making copies.
In your case:
import numpy as np
a = np.arange(100).reshape(10,10)
n1, n2 = np.arange(5), np.arange(5)
# Not what you want
b = a[n1, n2] # array([ 0, 11, 22, 33, 44])
# What you want, but only for simple sequences
# Note that no copy of *a* is made!! This is a view.
b = a[:5, :5]
# What you want, but probably confusing at first. (Also, makes a copy.)
# np.meshgrid and np.ix_ are basically equivalent to this.
b = a[n1[:,None], n2[None,:]]
Fancy indexing with 1D sequences is basically equivalent to zipping them together and indexing with the result.
print "Fancy Indexing:"
print a[n1, n2]
print "Manual indexing:"
for i, j in zip(n1, n2):
print a[i, j]
However, if the sequences you're indexing with match the dimensionality of the array you're indexing (2D, in this case), The indexing is treated differently. Instead of "zipping the two together", numpy uses the indices like a mask.
In other words, a[[[1, 2, 3]], [[1],[2],[3]]] is treated completely differently than a[[1, 2, 3], [1, 2, 3]], because the sequences/arrays that you're passing in are two-dimensional.
In [4]: a[[[1, 2, 3]], [[1],[2],[3]]]
Out[4]:
array([[11, 21, 31],
[12, 22, 32],
[13, 23, 33]])
In [5]: a[[1, 2, 3], [1, 2, 3]]
Out[5]: array([11, 22, 33])
To be a bit more precise,
a[[[1, 2, 3]], [[1],[2],[3]]]
is treated exactly like:
i = [[1, 1, 1],
[2, 2, 2],
[3, 3, 3]])
j = [[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]
a[i, j]
In other words, whether the input is a row/column vector is a shorthand for how the indices should repeat in the indexing.
np.meshgrid and np.ix_ are just convienent ways to turn your 1D sequences into their 2D versions for indexing:
In [6]: np.ix_([1, 2, 3], [1, 2, 3])
Out[6]:
(array([[1],
[2],
[3]]), array([[1, 2, 3]]))
Similarly (the sparse argument would make it identical to ix_ above):
In [7]: np.meshgrid([1, 2, 3], [1, 2, 3], indexing='ij')
Out[7]:
[array([[1, 1, 1],
[2, 2, 2],
[3, 3, 3]]),
array([[1, 2, 3],
[1, 2, 3],
[1, 2, 3]])]
Another quick way to build the desired index is to use the np.ix_ function:
>>> a[np.ix_([n1, n2])]
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44]])
This provides a convenient way to construct an open mesh from sequences of indices.
You could use np.meshgrid to give the n1, n2 arrays the proper shape to perform the desired indexing:
In [104]: a[np.meshgrid(n1,n2, sparse=True, indexing='ij')]
Out[104]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44]])
Or, without meshgrid:
In [117]: a[np.array(n1)[:,np.newaxis], np.array(n2)[np.newaxis,:]]
Out[117]:
array([[ 0, 1, 2, 3, 4],
[10, 11, 12, 13, 14],
[20, 21, 22, 23, 24],
[30, 31, 32, 33, 34],
[40, 41, 42, 43, 44]])
There is a similar example with an explanation of how this integer array indexing works in the docs.
See also the Cookbook recipe Picking out rows and columns.
A nice Trick I've managed to pull (for lazy people only)
Is filter + Transpose + filter.
a = np.arange(100).reshape(10,10)
subsetA = [1,3,5,7]
a[subsetA].T[subsetA]
array([[11, 31, 51, 71],
[13, 33, 53, 73],
[15, 35, 55, 75],
[17, 37, 57, 77]])
It seems that a use case for your particular question would deal with image manipulation. To the extent that you are using your example to edit numpy arrays arising from images, you can use the Python Imaging Library (PIL).
# Import Pillow:
from PIL import Image
# Load the original image:
img = Image.open("flowers.jpg")
# Crop the image
img2 = img.crop((0, 0, 5, 5))
The img2 object is a numpy array of the resulting cropped image.
You can read more about image manipulation here with the Pillow package (a user friendly fork on the PIL package):