This question already has an answer here:
Logical indexing in Numpy with two indices as in MATLAB
(1 answer)
Closed 7 years ago.
I am trying to write some code that uses logical numpy arrays to index a larger array, similar to how MATLAB allows array indexing with logical arrays.
import numpy as np
m = 4
n = 4
unCov = np.random.randint(10, size = (m,n) )
rowCov = np.zeros( m, dtype = bool )
colCov = np.ones( n, dtype = bool )
>>> unCov[rowCov, rowCov]
[] # as expected
>>> unCov[colCov, colCov]
[0 8 3 3] # diagonal values of unCov, as expected
>>> unCov[rowCov, colCov]
ValueError: shape mismatch: objects cannot be broadcast to a single shape
For this last evaluation, I expected an empty array, similar to what MATLAB returns. I'd rather not have to check rowCov/colCov for True elements prior to indexing. Why is this happening, and is there a better way to do this?
As I understand it, numpy will translate your 2d logical indices to actual index vectors: arr[[True,False],[False,True]] would become arr[0,1] for an ndarray of shape (2,2). However, in your last case the second index array is full False, hence it corresponds to an index array of length 0. This is paired with the other full True index vector, corresponding to an index array of length 4.
From the numpy manual:
If the index arrays do not have the same shape, there is an attempt to
broadcast them to the same shape. If they cannot be broadcast to the
same shape, an exception is raised:
In your case, the error is exactly due to this:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-1411-28e41e233472> in <module>()
----> 1 unCov[colCov,rowCov]
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (4,) (0,)
MATLAB, on the other hand, automatically returns an empty array if the index array is empty along any given dimension.
This actually highlights a fundamental difference between the logical indexing in MATLAB and numpy. In MATLAB, vectors in subscript indexing always slice out a subarray. That is, both
arr([1,2],[1,2])
and
arr([true,true],[true,true])
will return the 2 x 2 submatrix of the matrix arr. If the logical index vectors are shorter than the given dimension of the array, the missing indexing elements are assumed to be false. Fun fact: the index vector can also be longer than the given dimension, as long as the excess elements are all false. So the above is also equivalent to
arr([true,true,false,false],[true,true])
and
arr([true,true,false,false,false,false,false],[true,true])
for a 4 x 4 array (for the sake of argument).
In numpy, however, indexing with boolean-valued numpy arrays in this way will try to extract a vector. Furthermore, the boolean index vectors should be the same length as the dimension they are indexing into. In your 4 x 4 example,
unCov[np.array([True,True]),np.array([True,True])]
and
unCov[np.array([True,True,False,False,False]),np.array([True,True,False,False,False])]
both return the two first diagonal elements, so not a submatrix but rather a vector. Furthermore, they also give the less-then-encouraging warning along the lines of
/usr/bin/ipython:1: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 4 but corresponding boolean dimension is 5
So, in numpy, your logical indexing vectors should be the same length as the corresponding dimensions of the ndarray. And then what I wrote above holds true: the logical values are translated into indices, and the result is expected to be a vector. The length of this vector is the number of True elements in every index vector, so if your boolean index vectors have a different number of True elements, then the referencing doesn't make sense, and you get the error that you get.
Related
Suppose I have a flat NumPy array a and want to define an index array i to index a with and thus obtain a again by a[i].
I tried
import numpy as np
a = np.array([1,2]).reshape(-1)
i = True
But this does not preserve shape: a[i] has shape (1, 2) while a has shape (2,).
I know I could reshape a[i] or use i = np.full_like(a, True, dtype=bool). I want neither: The reshape is unnecessary if i is per some conditional definition sometimes not plain True but a boolean array matching the shape of a. The second approach means I need different is for doing this on arrays of different shapes.
So... is there something build-in in NumPy to just get the array back when used as index?
Numpy can not preserve the shape of a boolean masked result because it may be ragged. When you pass in a single boolean scalar, things get special-case weird.
You must therefore use a fancy index. With a fancy index, the shape of the result is exactly the shape of the index. For a 1-D array the following is fine:
i = np.arange(a.size)
For more dimensions, you'll want to create a full indexing tuple, using np.indices for example. The elements of the tuple can broadcast to the final desired shape, so you can use sparse=True:
i = np.indices(a.shape, sparse=True)
If you want i to be a numpy array, you can set sparse=False, in which case i will be of shape (a.ndim, *a.shape).
If you want to cheat, you can use slices. slice(None) is the object representing the literal index ::
i = (slice(None),) * a.ndim
Or just index the first dimension only, which returns the entire array:
i = slice(None)
Or if you're feeling really lazy, use Ellipsis directly. This is the object that stands for the literal ..., meaning :, :, etc, as many times as necessary:
i = Ellipsis
Going back to the boolean mask option, you can use it for the same effect if you create a separate array for each dimension:
i = tuple(np.ones(k, dtype=bool) for k in a.shape)
You could save some memory by only allocating the largest shape and creating views:
s = np.ones(max(a.shape), dtype=bool)
i = tuple(s[:k] for k in a.shape)
I'm trying to write a function to compute the dot product of two 2D lists passed in as arguments, lets call them x and y.
My idea is to first create a 2D list of zeros of the proper dimensions for the result of the dot product. In order to do so, I need to find the column size of y when computing x * y
dim1 = len(x)
dim2 = len(y[0])
result = [0]*dim1*dim2
The above code was my idea for getting these dimensions, however it fails on the second line due to an error:
dim2 = len(y[0])
TypeError: object of type 'int' has no len()
My python interpreter seems to not like that I am assuming my arguments will be 2D lists. It seems to think it'll be a 1D list. How can I get the column length of the 2D list. I am assuming the 2D lists passed in will be of dimensions NxM, so it should be a clean rectangle shape list/matrix.
I am not able to use numpy for this case.
I want to find the values in a numpty multidimensional array (2D example below) be passing in an array of indicies.
It appears that I can only pass in upto 2 indices without getting an error:
V2 = [[1,2],[2,1]]
V3 = [[1,2],[2,1],[0,0]]
lookup = np.random.rand(3,3)
lookup[V2] #OK
lookup[V3] #IndexError: too many indices for array
The number of indexes as you use it is the number of dimensions.
I think you are making that assumption that every subelement of the list is 1 point while actually the syntax:
V2 = [[a1,a2,a3],[b1,b2,b3]]
lookup[V2]
is equivalent to accessing:
[V2[a1,b1],
V2[a2,b2],
V2[a3,b3]]
using a 3rd dimension gives you an error since you have an array with only 2 dimensions
I have a numpy array y:
y = np.arange(35).reshape(5,7)
5 rows, 7 columns. Now I create a boolean 1-d 5-element mask which picks out the rows I want (following the doc at numpy indexing):
b = np.array([False, False, True, False, True])
Then y[b] returns the rows of interest. But the doc is confusing: it says
Boolean arrays must be of the same shape as the array being indexed, or broadcastable to the same shape.
And b is not broadcastable with y:
>>> np.broadcast_arrays(y, b)
ValueError: shape mismatch: two or more arrays have incompatible dimensions on axis 1.
because broadcasting works by matching the trailing dimensions and working backwards.
In this case of boolean indexing, there's clearly some different rule at work; is the doc wrong or am I just misunderstanding it? If I did what the doc suggests and make b be of shape (5,1) it doesn't pick out the rows; it just gets the first column of each selected row and returns that as a 1-d array.
I suspect the real rule is that the boolean object's dims must match the original array's initial dims, and it selects the values of each of those dims where the boolean is true, returning all elements of any trailing dims. But I can't find anything official that says that's how it works.
So my question is, (a) am I doing it right and the doc is just wrong? (b) am I reading the wrong doc? (c) is there a better/different way to do it or to understand it?
The reduction of y[b] appears to do what you want. I don't think it's out of step with the docs and there is not a special case for boolean vs numbers for broadcasting here.
y[b] # gives what you want
# broadcast booleans
np.broadcast_arrays(b, y) #gives the error you saw
np.broadcast_arrays(b[:,np.newaxis], y) #broadcasts.
# same as broadcast numbers
np.broadcast_arrays(np.arange(5), y) #ALSO gives the error you saw
np.broadcast_arrays(np.arange(5)[:,np.newaxis], y) #broadcasts.
I'm having some trouble understanding the rules for array broadcasting in Numpy.
Obviously, if you perform element-wise multiplication on two arrays of the same dimensions and shape, everything is fine. Also, if you multiply a multi-dimensional array by a scalar it works. This I understand.
But if you have two N-dimensional arrays of different shapes, it's unclear to me exactly what the broadcasting rules are. This documentation/tutorial explains that: In order to broadcast, the size of the trailing axes for both arrays in an operation must either be the same size or one of them must be one.
Okay, so I assume by trailing axis they are referring to the N in a M x N array. So, that means if I attempt to multiply two 2D arrays (matrices) with equal number of columns, it should work? Except it doesn't...
>>> from numpy import *
>>> A = array([[1,2],[3,4]])
>>> B = array([[2,3],[4,6],[6,9],[8,12]])
>>> print(A)
[[1 2]
[3 4]]
>>> print(B)
[[ 2 3]
[ 4 6]
[ 6 9]
[ 8 12]]
>>>
>>> A * B
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape
Since both A and B have two columns, I would have thought this would work. So, I'm probably misunderstanding something here about the term "trailing axis", and how it applies to N-dimensional arrays.
Can someone explain why my example doesn't work, and what is meant by "trailing axis"?
Well, the meaning of trailing axes is explained on the linked documentation page.
If you have two arrays with different dimensions number, say one 1x2x3 and other 2x3, then you compare only the trailing common dimensions, in this case 2x3. But if both your arrays are two-dimensional, then their corresponding sizes have to be either equal or one of them has to be 1. Dimensions along which the array has size 1 are called singular, and the array can be broadcasted along them.
In your case you have a 2x2 and 4x2 and 4 != 2 and neither 4 or 2 equals 1, so this doesn't work.
From http://cs231n.github.io/python-numpy-tutorial/#numpy-broadcasting:
Broadcasting two arrays together follows these rules:
If the arrays do not have the same rank, prepend the shape of the lower rank array with 1s until both shapes have the same length.
The two arrays are said to be compatible in a dimension if they have the same size in the dimension, or if one of the arrays has size 1 in that dimension.
The arrays can be broadcast together if they are compatible in all dimensions.
After broadcasting, each array behaves as if it had shape equal to the elementwise maximum of shapes of the two input arrays.
In any dimension where one array had size 1 and the other array had size greater than 1, the first array behaves as if it were copied along that dimension
If this explanation does not make sense, try reading the explanation from the documentation or this explanation.
we should consider two points about broadcasting. first: what is possible. second: how much of the possible things is done by numpy.
I know it might look a bit confusing, but I will make it clear by some example.
lets start from the zero level.
suppose we have two matrices. first matrix has three dimensions (named A) and the second has five (named B). numpy tries to match last/trailing dimensions. so numpy does not care about the first two dimensions of B. then numpy compares those trailing dimensions with each other. and if and only if they be equal or one of them be 1, numpy says "O.K. you two match". and if it these conditions don't satisfy, numpy would "sorry...its not my job!".
But I know that you may say comparison was better to be done in way that can handle when they are devisable(4 and 2 / 9 and 3). you might say it could be replicated/broadcasted by a whole number(2/3 in out example). and i am agree with you. and this is the reason I started my discussion with a distinction between what is possible and what is the capability of numpy.