This code is swapping first and the last channels of an RBG image which is loaded into a Numpy array:
img = imread('image1.jpg')
# Convert from RGB -> BGR
img = img[..., [2, 1, 0]]
While I understand the use of Ellipsis for slicing in Numpy arrays, I couldn't understand the use of Ellipsis here. Could anybody explain what is exactly happening here?
tl;dr
img[..., [2, 1, 0]] produces the same result as taking the slices img[:, :, i] for each i in the index array [2, 1, 0], and then stacking the results along the last dimension of img. In other words:
img[..., [2,1,0]]
will produce the same output as:
np.stack([img[:,:,2], img[:,:,1], img[:,:,0]], axis=2)
The ellipsis ... is a placeholder that tells numpy which axis to apply the index array to. Without the ... the index array will be applied to the first axis of img instead of the last. Thus, without ..., the index statement:
img[[2,1,0]]
will produce the same output as:
np.stack([img[2,:,:], img[1,:,:], img[0,:,:]], axis=0)
What the docs say
This is an example of what the docs call "Combining advanced and basic indexing":
When there is at least one slice (:), ellipsis (...) or np.newaxis in the index (or the array has more dimensions than there are advanced indexes), then the behaviour can be more complicated. It is like concatenating the indexing result for each advanced index element.
It goes on to describe that in this
case, the dimensions from the advanced indexing operations [in your example [2, 1, 0]] are inserted into the result array at the same spot as they were in the initial array (the latter logic is what makes simple advanced indexing behave just like slicing).
The 2D case
The docs aren't the easiest to understand, but in this case it's not too hard to pick apart. Start with a simpler 2D case:
arr = np.arange(12).reshape(4,3)
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]])
Using the same kind of advanced indexing with a single index value yields:
arr[:, [1]]
array([[ 1],
[ 4],
[ 7],
[10]])
which is the 1st column of arr. In other words, it's like you yielded all possible values from arr while holding the index of the last axis fixed. Like #hpaulj said in his comment, the ellipsis is there to act as a placeholder. It effectively tells numpy to iterate freely over all of the axes except for the last, to which the indexing array is applied.
You can use also this indexing syntax to shuffle the columns of arr around however you'd like:
arr[..., [1,0,2]]
array([[ 1, 0, 2],
[ 4, 3, 5],
[ 7, 6, 8],
[10, 9, 11]])
This is essentially the same operation as in your example, but on a 2D array instead of a 3D one.
You can explain what's going on with arr[..., [1,0,2]] by breaking it down to simpler indexing ops. It's kind of like you first take the return value of arr[..., [1]]:
array([[ 1],
[ 4],
[ 7],
[10]])
then the return value of arr[..., [0]]:
array([[0],
[3],
[6],
[9]])
then the return value of arr[..., [1]]:
array([[ 2],
[ 5],
[ 8],
[11]])
and then finally concatenated all of those results into a single array of shape (*arr.shape[:-1], len(ix)), where ix = [2, 0, 1] is the index array. The data along the last axis are ordered according to their order in ix.
One good way to understand exactly the ellipsis is doing is to perform the same op without it:
arr[[1,0,2]]
array([[6, 7, 8],
[0, 1, 2],
[3, 4, 5]])
In this case, the index array is applied to the first axis of arr, so the output is an array containing the [1,0,2] rows of arr. Adding an ... before the index array tells numpy to apply the index array to the last axis of arr instead.
Your 3D case
The case you asked about is the 3D equivalent of the 2D arr[..., [1,0,2]] example above. Say that img.shape is (480, 640, 3). You can think about img[..., [2, 1, 0]] as looping over each value i in ix=[2, 1, 0]. For every i, the indexing operation will gather the slab of shape (480, 640, 1) that lies along the ith index of the last axis of img. Once all three slabs are collected, the final result will be the equivalent of concatenating along their last axis (and in the order they were found).
notes
The only difference between arr[..., [1]] and arr[:,1] is that arr[..., [1]] preserves the shape of the data from the original array.
For a 2D array, arr[:, [1]] is equivalent to arr[..., [1]]. : acts as a placeholder just like ..., but only for a single dimension.
Related
Say I have a matrix of shape (2,3), I need to diagonalize the 3-elements vector into matrix of shape (3,3), for all the 2 vectors at once. That is, I need to return matrix with shape (2,3,3). How can I do that with Numpy elegantly ?
given data = np.array([[1,2,3],[4,5,6]])
i want the result [[[1,0,0],
[0,2,0],
[0,0,3]],
[[4,0,0],
[0,5,0],
[0,0,6]]]
Thanks
tl;dr, my one-liner: mydiag=np.vectorize(np.diag, signature='(n)->(n,n)')
I suppose here that by "diagonalize" you mean "applying np.diag".
Which, as a teacher of linear algebra, tickles me a bit. Since "diagonalizing" has a specific meaning, which is not that (it is computing eigen vectors and values, and from there, writing M=P⁻¹ΛP. Which you cannot do from the inputs you have).
So, I suppose that if input matrix is
[[1, 2, 3],
[9, 8, 7]]
The output matrix you want is
[[[1, 0, 0],
[0, 2, 0],
[0, 0, 3]],
[[9, 0, 0],
[0, 8, 0],
[0, 0, 7]]]
If not, you can ignore this answer [Edit: in the meantime, you explained exactly that. So yo may continue to read].
There are many way to do that.
My one liner would be
mydiag=np.vectorize(np.diag, signature='(n)->(n,n)')
Which build a new functions which does what you want (it interprets the input as a list of 1D-array, call np.diag of each of them, to get a 2D-array, and put each 2D-array in a numpy array, thus getting a 3D-array)
Then, you just call mydiag(M)
One advantage of vectorize, is that it uses numpy broadcasting. In other words, the loops are executed in C, not in python. In yet other words, it is faster. Well it is supposed to be (on small matrix, it is in fact slower than Michael's method - in comment; on large matrix, it is has the exact same speed. Which is frustrating, since einsum doc itself specify that it sacrifices broadcasting).
Plus, it is a one-liner, which has no other interest than bragging on forums. But well, here we are.
Here is one way with indexing:
out = np.zeros(data.shape+(data.shape[-1],), dtype=data.dtype)
x,y = np.indices(data.shape).reshape(2, -1)
out[x,y,y] = data.ravel()
output:
array([[[1, 0, 0],
[0, 2, 0],
[0, 0, 3]],
[[4, 0, 0],
[0, 5, 0],
[0, 0, 6]]])
We use array indexing to precisely grab those elements that are on the diagonal. Note that array indexing allows broadcasting between the indices, so we have index1 contain the index of the array, and index2 contain the index of the diagonal element.
index1 = np.arange(2)[:, None] # 2 is the number of arrays
index2 = np.arange(3)[None, :] # 3 is the square size of each matrix
result = np.zeros((2, 3, 3))
result[index1, index2, index2] = data
In practice, using both [..., :2] and [:2] on np.array([1,2,3]) results in np.array([1,2]). Are there also cases where the result differs when you use an ellipsis like this on an array?
np.arrays are designed to handle n-dimensional arrays, specified as [rows, columns]In the case of np.array([1, 2, 3]), [:2] and [:, :2] will yield the same result because our array input is 1-dimensional of shape [1, 3], e.g. with 1 row and 3 columns.
If we instead input np.array([[1,2,3], [4,5,6]]), e.g. a 2-dimensional array of shape [2, 3], this will change. On this array, if we, e.g., do [:1, :2] we will get array([[1, 2]]) because we are asking for everything up to the first (i.e. the 2nd since we count from zero) row and everything up to the second (i.e. the 3rd) column.
Hope this makes sense.
Given an array X of shape (100,8192), I want to copy the subarrays of length 8192 for each of the 100 outer dimensions 10 times, so that the resulting array has shape (100,8192,10).
I'm kind of confused about how the tile function works, I can sort of only copy a 1d array (although probably not really elegantly), e.g. if I'm given a 1d array of shape (8192,), I can create a 2d array by copying the 1d array like this: np.tile(x,(10,1)).transpose(), but once I try to do this on a 2d array, I have no idea what the tile function is actually doing when you provide a tuple of values, the documentation is kind of unclear about that.
Can anybody tell me how to do this please?
EDIT: Example, given the 2d array:
In [229]: x
Out[229]:
array([[1, 2, 3],
[4, 5, 6]])
I want to get by copying along the columns 3 times in this case, the following array:
In [233]: y
Out[233]:
array([[[1, 1, 1],
[2, 2, 2],
[3, 3, 3]],
[[4, 4, 4],
[5, 5, 5],
[6, 6, 6]]])
One way to do this is using np.repeat, e.g.:
Let X be the array of shape (100,8192), to replicate the subarray of dimension 8192 10-times across the column dimension, do the following:
X_new = np.repeat(X,10).reshape(100,8192,10)
Are you really asking for a shape (100,8192,10)? By reading you, I would have rather thought of something like (100,10,8192)? Could you provide an example? If you're actually asking for (100,10,8192), maybe you want:
np.tile(x,10).reshape((100,10,8192))
Is it what you're asking for?
I have need for hstacking multple arrays with with the same number of rows (although the number of rows is variable between uses) but different number of columns. However some of the arrays only have one column, eg.
array = np.array([1,2,3,4,5])
which gives
#array.shape = (5,)
but I'd like to have the shape recognized as a 2d array, eg.
#array.shape = (5,1)
So that hstack can actually combine them.
My current solution is:
array = np.atleast_2d([1,2,3,4,5]).T
#array.shape = (5,1)
So I was wondering, is there a better way to do this? Would
array = np.array([1,2,3,4,5]).reshape(len([1,2,3,4,5]), 1)
be better?
Note that my use of [1,2,3,4,5] is just a toy list to make the example concrete. In practice it will be a much larger list passed into a function as an argument. Thanks!
Check the code of hstack and vstack. One, or both of those, pass the arguments through atleast_nd. That is a perfectly acceptable way of reshaping an array.
Some other ways:
arr = np.array([1,2,3,4,5]).reshape(-1,1) # saves the use of len()
arr = np.array([1,2,3,4,5])[:,None] # adds a new dim at end
np.array([1,2,3],ndmin=2).T # used by column_stack
hstack and vstack transform their inputs with:
arrs = [atleast_1d(_m) for _m in tup]
[atleast_2d(_m) for _m in tup]
test data:
a1=np.arange(2)
a2=np.arange(10).reshape(2,5)
a3=np.arange(8).reshape(2,4)
np.hstack([a1.reshape(-1,1),a2,a3])
np.hstack([a1[:,None],a2,a3])
np.column_stack([a1,a2,a3])
result:
array([[0, 0, 1, 2, 3, 4, 0, 1, 2, 3],
[1, 5, 6, 7, 8, 9, 4, 5, 6, 7]])
If you don't know ahead of time which arrays are 1d, then column_stack is easiest to use. The others require a little function that tests for dimensionality before applying the reshaping.
Numpy: use reshape or newaxis to add dimensions
If I understand your intent correctly, you wish to convert an array of shape (N,) to an array of shape (N,1) so that you can apply np.hstack:
In [147]: np.hstack([np.atleast_2d([1,2,3,4,5]).T, np.atleast_2d([1,2,3,4,5]).T])
Out[147]:
array([[1, 1],
[2, 2],
[3, 3],
[4, 4],
[5, 5]])
In that case, you could use avoid reshaping the arrays and use np.column_stack instead:
In [151]: np.column_stack([[1,2,3,4,5], [1,2,3,4,5]])
Out[151]:
array([[1, 1],
[2, 2],
[3, 3],
[4, 4],
[5, 5]])
I followed Ludo's work and just changed the size of v from 5 to 10000. I ran the code on my PC and the result shows that atleast_2d seems to be a more efficient method in the larger scale case.
import numpy as np
import timeit
v = np.arange(10000)
print('atleast2d:',timeit.timeit(lambda:np.atleast_2d(v).T))
print('reshape:',timeit.timeit(lambda:np.array(v).reshape(-1,1))) # saves the use of len()
print('v[:,None]:', timeit.timeit(lambda:np.array(v)[:,None])) # adds a new dim at end
print('np.array(v,ndmin=2).T:', timeit.timeit(lambda:np.array(v,ndmin=2).T)) # used by column_stack
The result is:
atleast2d: 1.3809496470021259
reshape: 27.099974197000847
v[:,None]: 28.58291715100131
np.array(v,ndmin=2).T: 30.141663907001202
My suggestion is that use [:None] when dealing with a short vector and np.atleast_2d when your vector goes longer.
Just to add info on hpaulj's answer. I was curious about how fast were the four methods described. The winner is the method adding a column at the end of the 1d array.
Here is what I ran:
import numpy as np
import timeit
v = [1,2,3,4,5]
print('atleast2d:',timeit.timeit(lambda:np.atleast_2d(v).T))
print('reshape:',timeit.timeit(lambda:np.array(v).reshape(-1,1))) # saves the use of len()
print('v[:,None]:', timeit.timeit(lambda:np.array(v)[:,None])) # adds a new dim at end
print('np.array(v,ndmin=2).T:', timeit.timeit(lambda:np.array(v,ndmin=2).T)) # used by column_stack
And the results:
atleast2d: 4.455070924214851
reshape: 2.0535152913971615
v[:,None]: 1.8387219828073285
np.array(v,ndmin=2).T: 3.1735243063353664
General Problem
Suppose that I have a ndarray v of shape (nrow,ncols,3). I want to compute the ndarray outer_array of shape (nrow,ncols,3,3) containing all outer products of the vectors of shape (3) at each index (nrow,ncol). Of course, this is the the kind of problem for which numpy.einsum exists.
Now, what I've tried is:
outer_array = numpy.einsum("xyi,xyj->xyij",v,v.conjugate())
Now, I'm not sure that this will work: despite the fact that outer_array has the expected shape, the elements of the matrices of outer products do not correspond to what I'm expecting.
I think this is due to the choice of labels in the einsum expression: the product is supposed to be summed over x and y because the indices are repeated, but since I'm reusing them in the output expression, the result of the sum is somehow broadcast.
On the other hand, if I write:
outer_array = numpy.einsum("xyi,uvj->...ij",v,v.conjugate())
numpy will compute all possible combinations of outer products for each pair (x,y) and (u,v), resulting in an array of shape (ncols,nrow,ncols,nrow,3,3), where the diagonals (u,v) = (x,y) will contain the desired output.
The Precise Question
How do I choose the first two indices in the einsum notation in order to obtain an array where at each index x,y I get the outer product of vector v with itself without having to resort to the second expression?
Edit
apparently, this form seems to work too:
outer_array = numpy.einsum("...i,...j->...ij",v,v.conjugate())
I can only admire how useful numpy broadcasting is!
The key to 'xyi,xyj->xyij' working is that the xy is repeated in the output string.
Let's use a simpler array:
x = np.arange(6).reshape(3,2)
np.einsum.einsum('ij->j',x)
# array([6, 9]) # sums on the 1st dimension of `x`
Now for an outer product on this x:
In [20]: x[:,:,None]*x[:,None,:] # shape (3,2,2)
Out[20]:
array([[[ 0, 0],
[ 0, 1]],
[[ 4, 6],
[ 6, 9]],
[[16, 20],
[20, 25]]])
This is an example of numpy broadcasting (i.e. adding dimensions and expanding them)
In "...i,...j->...ij", ... is functioning more as a place holder for existing, but anonymous dimensions.
The equivalent with einsum is:
np.einsum('ij,ik->ijk',x,x)
(I should really do a calculation that isn't symmetric in the last 2 dimensions).
I've worked out a pure Python work-alike of einsum. The focus is on parsing the argument string, and how it creates the inputs for an iter object. It's available on github: https://github.com/hpaulj/numpy-einsum/blob/master/einsum_py.py You are welcome to play around with it. It has a debug flag to show intermediate steps.
With my einsum with debugging output:
In [23]: einsum_py.myeinsum('ij,ik->ijk',x,x,debug=True)
# ... some parsing information
[('ij', [105, 106], 'NONE'), ('ik', [105, 107], 'NONE')]
('ijk', [105, 106, 107], 'NONE')
iter labels: [105, 106, 107],'ijk'
op_axes [[0, 1, -1], [0, -1, 1], [0, 1, 2]]
op_axes is the key argument that is used in creating a iter, the object that iterates over the axes of the input and output arrays. It iterates over the 1st axis of all arrays. The 2nd axis is 1 for 1st op and output, and a newaxis (-1) for the 2nd op.
With the ellipsis:
In [24]: einsum_py.myeinsum('...j,...k->...jk',x,x,debug=True)
...
iter labels: [0, 106, 107],'0jk'
op_axes [[0, 1, -1], [0, -1, 1], [0, 1, 2]]
This generates the same op_axes, and hence the same calculation.