Suppose I have a numpy array img, with img.shape == (468,832,3). What does img[::2, ::2] do? It reduces the shape to (234,416,3) Can you please explain the logic?
Let's read documentation together (Source).
(Just read the bold part first)
The basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step (k \neq 0). This selects the m elements (in the corresponding dimension) with index values i, i + k, ..., i + (m - 1) k where m = q + (r\neq0) and q and r are the quotient and remainder obtained by dividing j - i by k: j - i = q k + r, so that i + (m - 1) k < j.
...
Assume n is the number of elements in the dimension being sliced.
Then, if i is not given it defaults to 0 for k > 0 and n - 1 for k < 0
. If j is not given it defaults to n for k > 0 and -n-1 for k < 0 . If
k is not given it defaults to 1. Note that :: is the same as : and
means select all indices along this axis.
Now looking at your part.
[::2, ::2] will be translated to [0:468:2, 0:832:2] because you do not specify the first two or i and j in the documentation. (You only specify k here. Recall the i:j:k notation above.) You select elements on these axes at the step size 2 which means you select every other elements along the axes specified.
Because you did not specify for the 3rd dimension, all will be selected.
It slices every alternate row, and then every alternate column, from an array, returning an array of size (n // 2, n // 2, ...).
Here's an example of slicing with a 2D array -
>>> a = np.arange(16).reshape(4, 4)
>>> a
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
>>> a[::2, ::2]
array([[ 0, 2],
[ 8, 10]])
And, here's another example with a 3D array -
>>> a = np.arange(27).reshape(3, 3, 3)
>>> a
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
>>> a[::2, ::2] # same as a[::2, ::2, :]
array([[[ 0, 1, 2],
[ 6, 7, 8]],
[[18, 19, 20],
[24, 25, 26]]])
Well, we have the RGB image as a 3D array of shape:
img.shape=(468,832,3)
Now, what does img[::2, ::2] do?
we're just downsampling the image (i.e. we're shrinking the image size by half by taking only every other pixel from the original image and we do this by using a step size of 2, which means to skip one pixel). This should be clear from the example below.
Let's take a simple grayscale image for easier understanding.
In [13]: arr
Out[13]:
array([[10, 11, 12, 13, 14, 15],
[20, 21, 22, 23, 24, 25],
[30, 31, 32, 33, 34, 35],
[40, 41, 42, 43, 44, 45],
[50, 51, 52, 53, 54, 55],
[60, 61, 62, 63, 64, 65]])
In [14]: arr.shape
Out[14]: (6, 6)
In [15]: arr[::2, ::2]
Out[15]:
array([[10, 12, 14],
[30, 32, 34],
[50, 52, 54]])
In [16]: arr[::2, ::2].shape
Out[16]: (3, 3)
Notice which pixels are in the sliced version. Also, observe how the array shape changes after slicing (i.e. it is reduced by half).
Now, this downsampling happens for all three channels in the image since there's no slicing happening in the third axis. Thus, you will get the shape reduced only for the first two axis in your example.
(468, 832, 3)
. . |
. . |
(234, 416, 3)
Related
If I have a large 2D numpy array and 2 arrays which correspond to the x and y indices I want to extract, It's easy enough:
h = np.arange(49).reshape(7,7)
# h = [[0, 1, 2, 3, 4, 5, 6],
# [7, 8, 9, 10, 11, 12, 13],
# [14, 15, 16, 17, 18, 19, 20],
# [21, 22, 23, 24, 25, 26, 27],
# [28, 29, 30, 31, 32, 33, 34],
# [35, 36, 37, 38, 39, 40, 41],
# [42, 43, 44, 45, 46, 47, 48]]
x_indices = np.array([1,3,4])
y_indices = np.array([2,3,5])
reduced_h = h[x_indices, y_indices]
#reduced_h = [ 9, 24, 33]
However, I would like to, for each x,y pair cut out a square (denoted by 'a' - the number of indices in each direction from the centre) surrounding this 'coordinate' and return an array of these little 2D arrays.
For example, for h, x,y_indices as above and a=1:
reduced_h = [[[1,2,3],[8,9,10],[15,16,17]], [[16,17,18],[23,24,25],[30,31,32]], [[25,26,27],[32,33,34],[39,40,41]]]
i.e one 3x3 array for each x-y index pair corresponding to the 3x3 square of elements centred on the x-y index. In general, this should return a numpy array which has shape (len(x_indices),2a+1, 2a+1)
By analogy to reduced_h[0] = h[x_indices[0]-1:x_indices[0]+1 , y_indices[0]-1:y_indices[0]+1] = h[1-1:1+1 , 2-1:2+1] = h[0:2, 1:3] my first try was the following:
h[x_indices-a : x_indices+a, y_indices-a : y_indices+a]
However, perhaps unsurprisingly, slicing between the arrays fails.
So the obvious next thing to try is to create this slice manually. np.arange seems to struggle with this but linspace works:
a=1
xrange = np.linspace(x_indices-a, x_indices+a, 2*a+1, dtype=int)
# xrange = [ [0, 2, 3], [1, 3, 4], [2, 4, 5] ]
yrange = np.linspace(y_indices-a, y_indices+a, 2*a+1, dtype=int)
Now can try h[xrange,yrange] but this unsurprisingly does this element-wise meaning I get only one (2a+1)x(2a+1) array (the same dimensions as xrange and yrange). It there a way to, for every index, take the right slices from these ranges (without loops)? Or is there a way to make the broadcast work initially without having to set up linspace explicitly? Thanks
You can index np.lib.stride_tricks.sliding_window_view using your x and y indices:
import numpy as np
h = np.arange(49).reshape(7,7)
x_indices = np.array([1,3,4])
y_indices = np.array([2,3,5])
a = 1
window = (2*a+1, 2*a+1)
out = np.lib.stride_tricks.sliding_window_view(h, window)[x_indices-a, y_indices-a]
out:
array([[[ 1, 2, 3],
[ 8, 9, 10],
[15, 16, 17]],
[[16, 17, 18],
[23, 24, 25],
[30, 31, 32]],
[[25, 26, 27],
[32, 33, 34],
[39, 40, 41]]])
Note that you may need to pad h first to handle windows around your coordinates that reach "outside" h.
unigram is an array shape (N, M, 100)
I would like to remove the for loop and perform all the calculations.
seq is a 1D array of size M, and the size of M maybe up to 10000.
I would like to remove the for loop and vectorize it for easier computation.
batch_size, seq_len, num_labels = unigram_scores.shape
broadcast = np.broadcast_to(seq, (batch_size, seq_len))
for i in range(0, broadcast.shape[1]):
n_seq[i] = unigram_scores[np.arange(batch_size), i , broadcast[:,i]]
edit:
answer by #hpaulj worked perfectly and also has the advantage of not having to install any extra dependency
the speed up was much lower than I expected
I ended up finally installing numba
import numpy as np
from numba import njit, prange
#njit(parallel=True)
def calculate_unigram_probability(unigram_scores,seq):
batch_size, seq_len, num_labels = unigram_scores.shape
broadcast = np.broadcast_to(seq, (batch_size, seq_len))
for i in prange( broadcast.shape[1]):
n_seq[i] = unigram_scores[np.arange(batch_size), i , broadcast[:,i]]
return n_seq
which is also taking a a bit too long, Currently I am trying to move it from the cpu to cuda which should bring about the speedup I am hoping for
In [129]: N,M = 5,3
In [130]: unigram=np.arange(N*M*4).reshape(N,M,4)
In [131]: seq = np.arange(M)
In [132]: b_seq = np.broadcast_to(seq, (N,M))
For a single i:
In [133]: i=0; unigram[np.arange(N),i,b_seq[:,i]]
Out[133]: array([ 0, 12, 24, 36, 48])
For all i in the range:
In [136]: i=np.arange(M)[:,None]
In [137]: unigram[np.arange(N),i,b_seq[:,i]]
Out[137]:
array([[[ 0, 12, 24, 36, 48],
[ 5, 17, 29, 41, 53],
[10, 22, 34, 46, 58]],
...
[[ 0, 12, 24, 36, 48],
[ 5, 17, 29, 41, 53],
[10, 22, 34, 46, 58]]])
A (5,3,5) array. This (5,3) might be better)
In [141]: i=np.arange(M); unigram[np.arange(N)[:,None],i,b_seq[:,i]]
Out[141]:
array([[ 0, 5, 10],
[12, 17, 22],
[24, 29, 34],
[36, 41, 46],
[48, 53, 58]])
We don't need to index b_seq: unigram[np.arange(N)[:,None],i,b_seq]
Or even use; let the indexing broadcast seq:
unigram[np.arange(N)[:,None],i,seq]
and with the help of ix_:
In [145]: I,J=np.ix_(np.arange(N), np.arange(M))
In [146]: unigram[I,J,seq]
To get a visual idea of what this indexing does, look at unigram. It's pull 'diagonals' from successive blocks/batches:
In [147]: unigram
Out[147]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]],
[[12, 13, 14, 15],
[16, 17, 18, 19],
[20, 21, 22, 23]],
...
you can use x.flatten() to reshape a 3d array to 1d array (x must be a numpy array )
in your case :
broadcast = broadcast.flatten()
this will transform an array of shape (NM1000) to an array of one dimension
I want to define a 3d numpy array with shape = [3, 3, 5] and also with values as a range starting with 11 and step = 3 in column-wise manner. I mean:
B[:,:,0] = [[ 11, 20, 29],
[ 14, 23, 32],
[17, 26, 35]]
B[:,:,1] = [[ 38, ...],
[ 41, ...],
[ 44, ...]]
...
I am new to numpy and I doubt doing it with np.arange or np.mgrid maybe. but I don't how to do.
How can this be done with minimal lines of code?
You can calculate the end of the range by multiplying the shape by the step and adding the start. Then it's just reshape and transpose to move the column around:
start = 11
step = 3
shape = [5, 3, 3]
end = np.prod(shape) * step + start
B = np.arange(start, end, step).reshape([5, 3, 3]).transpose(2, 1, 0)
B[:, :, 0]
# array([[11, 20, 29],
# [14, 23, 32],
# [17, 26, 35]])
Having an array A with the shape (2,6, 60), is it possible to index it based on a binary array B of shape (6,)?
The 6 and 60 is quite arbitrary, they are simply the 2D data I wish to access.
The underlying thing I am trying to do is to calculate two variants of the 2D data (in this case, (6,60)) and then efficiently select the ones with the lowest total sum - that is where the binary (6,) array comes from.
Example: For B = [1,0,1,0,1,0] what I wish to receive is equal to stacking
A[1,0,:]
A[0,1,:]
A[1,2,:]
A[0,3,:]
A[1,4,:]
A[0,5,:]
but I would like to do it by direct indexing and not a for-loop.
I have tried A[B], A[:,B,:], A[B,:,:] A[:,:,B] with none of them providing the desired (6,60) matrix.
import numpy as np
A = np.array([[4, 4, 4, 4, 4, 4], [1, 1, 1, 1, 1, 1]])
A = np.atleast_3d(A)
A = np.tile(A, (1,1,60)
B = np.array([1, 0, 1, 0, 1, 0])
A[B]
Expected results are a (6,60) array containing the elements from A as described above, the received is either (2,6,60) or (6,6,60).
Thank you in advance,
Linus
You can generate a range of the indices you want to iterate over, in your case from 0 to 5:
count = A.shape[1]
indices = np.arange(count) # np.arange(6) for your particular case
>>> print(indices)
array([0, 1, 2, 3, 4, 5])
And then you can use that to do your advanced indexing:
result_array = A[B[indices], indices, :]
If you always use the full range from 0 to length - 1 (i.e. 0 to 5 in your case) of the second axis of A in increasing order, you can simplify that to:
result_array = A[B, indices, :]
# or the ugly result_array = A[B, np.arange(A.shape[1]), :]
Or even this if it's always 6:
result_array = A[B, np.arange(6), :]
An alternative solution using np.take_along_axis (from version 1.15 - docs)
import numpy as np
x = np.arange(2*6*6).reshape((2,6,6))
m = np.zeros(6, int)
m[0] = 1
#example: [1, 0, 0, 0, 0, 0]
np.take_along_axis(x, m[None, :, None], 0) #add dimensions to mask to match array dimensions
>>array([[[36, 37, 38, 39, 40, 41],
[ 6, 7, 8, 9, 10, 11],
[12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35]]])
Lets say I have an array of the below nature:
x = arange(30).reshape((10,3))
x
Out[52]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23],
[24, 25, 26],
[27, 28, 29]])
How do I add a fourth column to each of the row such that this column is an exponential function of the index number and ends up with something like this:
array([[ 0, 1, 2, 2.718281828],
[ 3, 4, 5, 7.389056099], ,
[ 6, 7, 8, 20.08553692],
[ 9, 10, 11, 54.59815003 ],
[12, 13, 14, 148.4131591],
[15, 16, 17, 403.4287935],
[18, 19, 20, 1096.633158 ],
[21, 22, 23, 2980.957987],
[24, 25, 26, 8103.083928],
[27, 28, 29, 22026.46579]])
Computing the exponential is easy:
ex = np.exp(np.arange(x.shape[0]) + 1)
What you want to do with it is a whole different story. Numpy doesn't allow heterogeneous arrays, unlike say pandas. So with the simple answer, your result will be float64 (x is most likely int64 or int32):
x = np.concatenate((x, ex[:, None]), axis=1)
An alternative is using structured arrays, which will let you preserve the input types:
d = [('', x.dtype)] * x.shape[1] + [('', ex.dtype)]
out = np.empty(ex.shape, dtype=d)
Bulk assignment is a bit tricky, but can be done with a view obtained from the raw ndarray constructor:
view = np.ndarray(buffer=out, dtype=x.dtype, shape=x.shape, strides=(out.dtype.itemsize, x.dtype.itemsize))
view[...] = x
np.ndarray(buffer=out, dtype=ex.dtype, shape=ex.shape, strides=(out.dtype.itemsize,), offset=x.strides[0])[:] = ex
A simpler approach would be to use recarray, as #PaulPanzer suggests:
out = np.core.records.fromarrays([*x.T, ex])
Try this:
import numpy as np
a = np.arange(30).reshape((10,3))
b = np.zeros((a.shape[0], a.shape[1] + 1))
b[:, :-1] = a
b[:, 3] = np.exp(np.arange(len(b)))
To create a single array of powers of e starting at one, you can use
powers = np.power(np.e, np.arange(10) + 1)
Which basically takes the number e and rases it to the powers given by array np.arange(10) + 1, i.e. the numbers [1...10].
You can then add this as an additional column by first reshaping it and then adding it using np.hstack.
powers = powers.reshape(-1, 1)
x = np.hstack((x, powers))
You can construct such column with:
>>> np.exp(np.arange(1, 11))
array([2.71828183e+00, 7.38905610e+00, 2.00855369e+01, 5.45981500e+01,
1.48413159e+02, 4.03428793e+02, 1.09663316e+03, 2.98095799e+03,
8.10308393e+03, 2.20264658e+04])
So we can first obtain the number of rows, and then use np.hstack:
rows = x.shape[0]
result = np.hstack((x, np.exp(np.arange(1, rows+1)).reshape(-1, 1)))
We then otain:
>>> np.hstack((x, np.exp(np.arange(1, 11)).reshape(-1, 1)))
array([[0.00000000e+00, 1.00000000e+00, 2.00000000e+00, 2.71828183e+00],
[3.00000000e+00, 4.00000000e+00, 5.00000000e+00, 7.38905610e+00],
[6.00000000e+00, 7.00000000e+00, 8.00000000e+00, 2.00855369e+01],
[9.00000000e+00, 1.00000000e+01, 1.10000000e+01, 5.45981500e+01],
[1.20000000e+01, 1.30000000e+01, 1.40000000e+01, 1.48413159e+02],
[1.50000000e+01, 1.60000000e+01, 1.70000000e+01, 4.03428793e+02],
[1.80000000e+01, 1.90000000e+01, 2.00000000e+01, 1.09663316e+03],
[2.10000000e+01, 2.20000000e+01, 2.30000000e+01, 2.98095799e+03],
[2.40000000e+01, 2.50000000e+01, 2.60000000e+01, 8.10308393e+03],
[2.70000000e+01, 2.80000000e+01, 2.90000000e+01, 2.20264658e+04]])