Related
I think I've misunderstood something with indexing in numpy.
I have a 3D-numpy array of shape (dim_x, dim_y, dim_z) and I want to find the maximum along the third axis (dim_z), and set its value to 1 and all the others to zero.
The problem is that I end up with several 1 in the same row, even if values are different.
Here is the code :
>>> test = np.random.rand(2,3,2)
>>> test
array([[[ 0.13110146, 0.07138861],
[ 0.84444158, 0.35296986],
[ 0.97414498, 0.63728852]],
[[ 0.61301975, 0.02313646],
[ 0.14251848, 0.91090492],
[ 0.14217992, 0.41549218]]])
>>> result = np.zeros_like(test)
>>> result[:test.shape[0], np.arange(test.shape[1]), np.argmax(test, axis=2)]=1
>>> result
array([[[ 1., 0.],
[ 1., 1.],
[ 1., 1.]],
[[ 1., 0.],
[ 1., 1.],
[ 1., 1.]]])
I was expecting to end with :
array([[[ 1., 0.],
[ 1., 0.],
[ 1., 0.]],
[[ 1., 0.],
[ 0., 1.],
[ 0., 1.]]])
Probably I'm missing something here. From what I've understood, 0:dim_x, np.arange(dim_y) returns dim_x of dim_y tuples and np.argmax(test, axis=dim_z) has the shape (dim_x, dim_y) so if the indexing is of the form [x, y, z] a couple [x, y] is not supposed to appear twice.
Could someone explain me where I'm wrong ? Thanks in advance.
What we are looking for
We get the argmax indices along the last axis -
idx = np.argmax(test, axis=2)
For the given sample data, we have idx :
array([[0, 0, 0],
[0, 1, 1]])
Now, idx covers the first and second axes, while getting those argmax indices.
To assign the corresponding ones in the output, we need to create range arrays for the first two axes covering the lengths along those and aligned according to the shape of idx. Now, idx is a 2D array of shape (m,n), where m = test.shape[0] and n = test.shape[1].
Thus, the range arrays for assignment into first two axes of output must be -
X = np.arange(test.shape[0])[:,None]
Y = np.arange(test.shape[1])
Notice, the extension of the first range array to 2D is needed to have it aligned against the rows of idx and Y would align against the cols of idx -
In [239]: X
Out[239]:
array([[0],
[1]])
In [240]: Y
Out[240]: array([0, 1, 2])
Schematically put -
idx :
Y array
--------->
x x x | X array
x x x |
v
The fault in original code
Your code was -
result[:test.shape[0], np.arange(test.shape[1]), ..
This is essentially :
result[:, np.arange(test.shape[1]), ...
So, you are selecting all elements along the first axis, instead of only selecting the corresponding ones that correspond to idx indices. In that process, you were selecting a lot more than required elements for assignment and hence you were seeing many more than required 1s in result array.
The correction
Thus, the only correction needed was indexing into the first axis with the range array and a working solution would be -
result[np.arange(test.shape[0])[:,None], np.arange(test.shape[1]), ...
The alternative(s)
Alternatively, using the range arrays created earlier with X and Y -
result[X,Y,idx] = 1
Another way to get X,Y would be with np.mgrid -
m,n = test.shape[:2]
X,Y = np.ogrid[:m,:n]
I think there's a problem with mixing basic (slice) and advanced indexing. It's easier to see when selecting value from an array than with this assignment; but it can result in transposed axes. For a problem like this it is better use advanced indexing all around, as provided by ix_
In [24]: test = np.random.rand(2,3,2)
In [25]: idx=np.argmax(test,axis=2)
In [26]: idx
Out[26]:
array([[1, 0, 1],
[0, 1, 1]], dtype=int32)
with basic and advanced:
In [31]: res1 = np.zeros_like(test)
In [32]: res1[:, np.arange(test.shape[1]), idx]=1
In [33]: res1
Out[33]:
array([[[ 1., 1.],
[ 1., 1.],
[ 0., 1.]],
[[ 1., 1.],
[ 1., 1.],
[ 0., 1.]]])
with advanced:
In [35]: I,J = np.ix_(range(test.shape[0]), range(test.shape[1]))
In [36]: I
Out[36]:
array([[0],
[1]])
In [37]: J
Out[37]: array([[0, 1, 2]])
In [38]: res2 = np.zeros_like(test)
In [40]: res2[I, J , idx]=1
In [41]: res2
Out[41]:
array([[[ 0., 1.],
[ 1., 0.],
[ 0., 1.]],
[[ 1., 0.],
[ 0., 1.],
[ 0., 1.]]])
On further thought, the use of the slice for the 1st dimension is just wrong , if the goal is to set or find the 6 argmax values
In [54]: test
Out[54]:
array([[[ 0.15288242, 0.36013289],
[ 0.90794601, 0.15265616],
[ 0.34014976, 0.53804266]],
[[ 0.97979479, 0.15898605],
[ 0.04933804, 0.89804999],
[ 0.10199319, 0.76170911]]])
In [55]: test[I, J, idx]
Out[55]:
array([[ 0.36013289, 0.90794601, 0.53804266],
[ 0.97979479, 0.89804999, 0.76170911]])
In [56]: test[:, J, idx]
Out[56]:
array([[[ 0.36013289, 0.90794601, 0.53804266],
[ 0.15288242, 0.15265616, 0.53804266]],
[[ 0.15898605, 0.04933804, 0.76170911],
[ 0.97979479, 0.89804999, 0.76170911]]])
With the slice it selects a (2,3,2) set of values from test (or res), not the intended (2,3). There 2 extra rows.
Here is an easier way to do it:
>>> test == test.max(axis=2, keepdims=1)
array([[[ True, False],
[ True, False],
[ True, False]],
[[ True, False],
[False, True],
[False, True]]], dtype=bool)
...and if you really want that as floating-point 1.0 and 0.0, then convert it:
>>> (test==test.max(axis=2, keepdims=1)).astype(float)
array([[[ 1., 0.],
[ 1., 0.],
[ 1., 0.]],
[[ 1., 0.],
[ 0., 1.],
[ 0., 1.]]])
Here is a way to do it with only one winner per row-column combo (i.e. no ties, as discussed in comments):
rowmesh, colmesh = np.meshgrid(range(test.shape[0]), range(test.shape[1]), indexing='ij')
maxloc = np.argmax(test, axis=2)
flatind = np.ravel_multi_index( [rowmesh, colmesh, maxloc ], test.shape )
result = np.zeros_like(test)
result.flat[flatind] = 1
UPDATE after reading hpaulj's answer:
rowmesh, colmesh = np.ix_(range(test.shape[0]), range(test.shape[1]))
is a more-efficient, more numpythonic, alternative to my meshgrid call (the rest of the code stays the same)
The issue of why your approach fails is hard to explain, but here's one place where intuition could start: your slicing approach says "all rows, times all columns, times a certain sequence of layers". How many elements is that slice in total? By contrast, how many elements do you actually want to set to 1? It can be instructive to look at the values you get when you view the corresponding test values of the slice you're trying to assign to:
>>> test[:, :, maxloc].shape
(2, 3, 2, 3) # oops! it's because maxloc itself is 2x3
>>> test[:, :, maxloc]
array([[[[ 0.13110146, 0.13110146, 0.13110146],
[ 0.13110146, 0.07138861, 0.07138861]],
[[ 0.84444158, 0.84444158, 0.84444158],
[ 0.84444158, 0.35296986, 0.35296986]],
[[ 0.97414498, 0.97414498, 0.97414498],
[ 0.97414498, 0.63728852, 0.63728852]]],
[[[ 0.61301975, 0.61301975, 0.61301975],
[ 0.61301975, 0.02313646, 0.02313646]],
[[ 0.14251848, 0.14251848, 0.14251848],
[ 0.14251848, 0.91090492, 0.91090492]],
[[ 0.14217992, 0.14217992, 0.14217992],
[ 0.14217992, 0.41549218, 0.41549218]]]]) # note the repetition, because in maxloc you're repeatedly asking for layer 0 sometimes, and sometimes repeatedly for layer 1
I have these vectors :
a = [1,2,3,4]
b = [1,2,3,5]
and I could like to have this at the end :
A = [ [1,0,0,0,0]
[0,1,0,0,0]
[0,0,1,0,0]
[0,0,0,1,0] ]
B = [ [1,0,0,0,0]
[0,1,0,0,0]
[0,0,1,0,0]
[0,0,0,0,1] ]
I have been using np.reshape from python this way:
A = np.reshape(a,(1,4,1))
B = np.reshape(b,(1,4,1))
And it does just partially the job as I have the following result:
A = [[1]
[2]
[3]
[4]]
B = [[1]
[2]
[3]
[5]]
Ideally I would like something like this:
A = np.reshape(a,(1,4,(1,5))
but when reading the docs, this is not possible.
Thanks in advance for your help
Alternatively, numpy can assign value to multiple indexes on rows/columns in one go, example:
In [1]: import numpy as np
In [2]: b = [1,2,3,5]
...:
...:
In [3]: zero = np.zeros([4,5])
In [4]: brow, bcol = range(len(b)), np.array(b) -1 # logical transform
In [5]: zero[brow, bcol] = 1
In [6]: zero
Out[6]:
array([[ 1., 0., 0., 0., 0.],
[ 0., 1., 0., 0., 0.],
[ 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1.]])
What you're trying to do is not actually a reshape, as you alter the structure of the data.
Make a new array with the shape you want:
A = np.zeros(myshape)
B = np.zeros(myshape)
and then index those arrays
n = 0
for i_a, i_b in zip(a, b):
A[n, i_a - 1] = 1
B[n, i_b - 1] = 1
n += 1
The i_a/i_b - 1 in the assignment is only there to make 1 index the 0th element. This also only works if a and b have the same length. Make this two loops if they are not the same length.
There might be a more elegant solution but this should get the job done :)
Input:
array length (Integer)
indexes (Set or List)
Output:
A boolean numpy array that has a value 1 for the indexes 0 for the others.
Example:
Input: array_length=10, indexes={2,5,6}
Output:
[0,0,1,0,0,1,1,0,0,0]
Here is a my simple implementation:
def indexes2booleanvec(size, indexes):
v = numpy.zeros(size)
for index in indexes:
v[index] = 1.0
return v
Is there more elegant way to implement this?
One way is to avoid the loop
In [7]: fill = np.zeros(array_length) # array_length = 10
In [8]: fill[indexes] = 1 # indexes = [2,5,6]
In [9]: fill
Out[9]: array([ 0., 0., 1., 0., 0., 1., 1., 0., 0., 0.])
Another way to do it (in one line):
np.isin(np.arange(array_length), indexes)
However this is slower than Zero's solution.
Say I had the following:
x = np.array([[1.,2.,3.,4.],[2.,3.,4.,5,],[1.,3.,5.,6.]])
What would the syntax be in order to select say, the first two columns of every row? (So [[1.,2.],[2.,3.],[1.,3.]]).
Ultimately I want to run a loop of the form:
for j in range(len(x)):
a = x[1,2:j] * #something
Where x[1,2:j] refers to what I am trying to achieve in my question. Thanks in advance!
You can use np.hsplit() (Split an array into multiple sub-arrays horizontally (column-wise).) then chose the fist part :
>>> np.hsplit(x,2)[0]
array([[ 1., 2.],
[ 2., 3.],
[ 1., 3.]])
Or you can just use slicing :
>>> x[:, :2]
array([[ 1., 2.],
[ 2., 3.],
[ 1., 3.]])
You can slice axis 1 of the array x:
>>> x[:, :2]
array([[ 1., 2.],
[ 2., 3.],
[ 1., 3.]])
The : for axis 0 effectively means "every row". The :2 in axis 1 means "get the first two columns (0 and 1)".
Slicing in multiple dimensions works similarly to Python lists and other iterables,
start:stop:step
You can specify a slice for each dimension of the array, or use : to get everything along the axis.
New at Python and Numpy, trying to create 3-dimensional arrays. My problem is that the order of the dimensions are off compared to Matlab. In fact the order doesn't make sense at all.
Creating a matrix:
x = np.zeros((2,3,4))
In my world this should result in 2 rows, 3 columns and 4 depth dimensions and it should be presented as:
[0 0 0 [0 0 0 [0 0 0 [0 0 0
0 0 0] 0 0 0] 0 0 0] 0 0 0]
Seperating on each depth dimensions.
Instead it is presented as
[0 0 0 0 [0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0] 0 0 0 0]
That is, 3 rows, 4 column and 2 depth dimensions. That is, the first dimension is the "depth". To further add to this problem, importing an image with OpenCV the color dimension is the last dimension, that is, I see the color information as the depth dimension. This complicates things greatly if all I want to do is try something on a known smaller 3-dimensional array.
Have I misunderstood something? If not, why the heck is numpy using such a unintuitive way of working with 3D-dimensional arrays?
You have a truncated array representation. Let's look at a full example:
>>> a = np.zeros((2, 3, 4))
>>> a
array([[[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]],
[[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]]])
Arrays in NumPy are printed as the word array followed by structure, similar to embedded Python lists. Let's create a similar list:
>>> l = [[[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]],
[[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]]]
>>> l
[[[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]],
[[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]]]
The first level of this compound list l has exactly 2 elements, just as the first dimension of the array a (# of rows). Each of these elements is itself a list with 3 elements, which is equal to the second dimension of a (# of columns). Finally, the most nested lists have 4 elements each, same as the third dimension of a (depth/# of colors).
So you've got exactly the same structure (in terms of dimensions) as in Matlab, just printed in another way.
Some caveats:
Matlab stores data column by column ("Fortran order"), while NumPy by default stores them row by row ("C order"). This doesn't affect indexing, but may affect performance. For example, in Matlab efficient loop will be over columns (e.g. for n = 1:10 a(:, n) end), while in NumPy it's preferable to iterate over rows (e.g. for n in range(10): a[n, :] -- note n in the first position, not the last).
If you work with colored images in OpenCV, remember that:
2.1. It stores images in BGR format and not RGB, like most Python libraries do.
2.2. Most functions work on image coordinates (x, y), which are opposite to matrix coordinates (i, j).
You are right, you are creating a matrix with 2 rows, 3 columns and 4 depth. Numpy prints matrixes different to Matlab:
Numpy:
>>> import numpy as np
>>> np.zeros((2,3,2))
array([[[ 0., 0.],
[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.]]])
Matlab
>> zeros(2, 3, 2)
ans(:,:,1) =
0 0 0
0 0 0
ans(:,:,2) =
0 0 0
0 0 0
However you are calculating the same matrix. Take a look to Numpy for Matlab users, it will guide you converting Matlab code to Numpy.
For example if you are using OpenCV, you can build an image using numpy taking into account that OpenCV uses BGR representation:
import cv2
import numpy as np
a = np.zeros((100, 100,3))
a[:,:,0] = 255
b = np.zeros((100, 100,3))
b[:,:,1] = 255
c = np.zeros((100, 200,3))
c[:,:,2] = 255
img = np.vstack((c, np.hstack((a, b))))
cv2.imshow('image', img)
cv2.waitKey(0)
If you take a look to matrix c you will see it is a 100x200x3 matrix which is exactly what it is shown in the image (in red as we have set the R coordinate to 255 and the other two remain at 0).
No need to go in such deep technicalities, and get yourself blasted. Let me explain it in the most easiest way. We all have studied "Sets" during our school-age in Mathematics. Just consider 3D numpy array as the formation of "sets".
x = np.zeros((2,3,4))
Simply Means:
2 Sets, 3 Rows per Set, 4 Columns
Example:
Input
x = np.zeros((2,3,4))
Output
Set # 1 ---- [[[ 0., 0., 0., 0.], ---- Row 1
[ 0., 0., 0., 0.], ---- Row 2
[ 0., 0., 0., 0.]], ---- Row 3
Set # 2 ---- [[ 0., 0., 0., 0.], ---- Row 1
[ 0., 0., 0., 0.], ---- Row 2
[ 0., 0., 0., 0.]]] ---- Row 3
Explanation:
See? we have 2 Sets, 3 Rows per Set, and 4 Columns.
Note: Whenever you see a "Set of numbers" closed in double brackets from both ends. Consider it as a "set". And 3D and 3D+ arrays are always built on these "sets".
As much as people like to say "order doesn't matter its just convention" this breaks down when entering cross domain interfaces, IE transfer from C ordering to Fortran ordering or some other ordering scheme. There, precisely how your data is layed out and how shape is represented in numpy is very important.
By default, numpy uses C ordering, which means contiguous elements in memory are the elements stored in rows. You can also do FORTRAN ordering ("F"), this instead orders elements based on columns, indexing contiguous elements.
Numpy's shape further has its own order in which it displays the shape. In numpy, shape is largest stride first, ie, in a 3d vector, it would be the least contiguous dimension, Z, or pages, 3rd dim etc... So when executing:
np.zeros((2,3,4)).shape
you will get
(2,3,4)
which is actually (frames, rows, columns). doing np.zeros((2,2,3,4)).shape instead would mean (metaframs, frames, rows, columns). This makes more sense when you think of creating multidimensional arrays in C like langauges. For C++, creating a non contiguously defined 4D array results in an array [ of arrays [ of arrays [ of elements ]]]. This forces you to de reference the first array that holds all the other arrays (4th dimension) then the same all the way down (3rd, 2nd, 1st) resulting in syntax like:
double element = array4d[w][z][y][x];
In fortran, this indexed ordering is reversed (x is instead first array4d[x][y][z][w]), most contiguous to least contiguous and in matlab, it gets all weird.
Matlab tried to preserve both mathematical default ordering (row, column) but also use column major internally for libraries, and not follow C convention of dimensional ordering. In matlab, you order this way:
double element = array4d[y][x][z][w];
which deifies all convention and creates weird situations where you are sometimes indexing as if row ordered and sometimes column ordered (such as with matrix creation).
In reality, Matlab is the unintuitive one, not Numpy.
Read this article for better insight: numpy: Array shapes and reshaping arrays
Note: NumPy reports the shape of 3D arrays in the order layers, rows, columns.
I also got confused initially in NumPy. When you say :
x = np.zeros((2,3,4))
It interprets as:
Generate a 3d matrix with 2 matrices of 3 rows each. Each row must contain 4 elements each;
Numpy always starts assigning dimensions from the outermost then moves in
Thumb rule: A 2d array is a matrix