Numpy arrays - Multidimensional logic comparison - python

I am trying find the entries in a two-dimensional array that are above a certain threshold. The thresholds for the individual columns is given by a one-dimensional array. To exemplify,
[[1, 2, 3],
[4, 5, 6],
[2, 0, 4]]
is the two-dimensional array and I want to see if where in the columns values are bigger than
[2, 1, 3]
so the output of running the operation should be
[[False, True, False]
[True, True, True],
[False, False, True]]
Thanks!


Well, assuming there's an error in the example, I would simply do:
import numpy as np
A = np.array([[1, 2, 3],[4, 5, 6],[2, 0, 4]])
T = np.array([2, 1, 3])
X = A > T
Which gives
array([[False, True, False],
[ True, True, True],
[False, False, True]], dtype=bool)


I think there may be inconsistencies in your example (e.g. 2 > 1 is True, yet 2 > 4 is True) - can you clarify this?
Assuming you you want to know, for each row, which columns of the values in the first list are greater than the [2,1,3] list you gave, I suggest the following:
import numpy as np
tmp = [[1, 2, 3],
[4, 5, 6],
[2, 0, 4]]
output = [ np.less([2, 1, 3], tmp[i]) for i in range(len(tmp))]
Similarly, try greater or greater_equal or less_equal for the result you're after:
http://docs.scipy.org/doc/numpy/reference/routines.logic.html

Related

is there a method for finding the indexes of a 2d-array based on a given array

suppose we have two arrays like these two:
A=np.array([[1, 4, 3, 0, 5],[6, 0, 7, 12, 11],[20, 15, 34, 45, 56]])
B=np.array([[4, 5, 6, 7]])
I intend to write a code in which I can find the indexes of an array such as A based on values in
the array B
for example, I want the final results to be something like this:
C=[[0 1]
[0 4]
[1 0]
[1 2]]
can anybody provide me with a solution or a hint?

Do you mean?
In [375]: np.isin(A,B[0])
Out[375]:
array([[False, True, False, False, True],
[ True, False, True, False, False],
[False, False, False, False, False]])
In [376]: np.argwhere(np.isin(A,B[0]))
Out[376]:
array([[0, 1],
[0, 4],
[1, 0],
[1, 2]])
B shape of (1,4) where the initial 1 isn't necessary. That's why I used B[0], though isin, via in1d ravels it anyways.
where is result is often more useful
In [381]: np.where(np.isin(A,B))
Out[381]: (array([0, 0, 1, 1]), array([1, 4, 0, 2]))
though it's a bit harder to understand.
Another way to get the isin array:
In [383]: (A==B[0,:,None,None]).any(axis=0)
Out[383]:
array([[False, True, False, False, True],
[ True, False, True, False, False],
[False, False, False, False, False]])

You can try in this way by using np.where().
index = []
for num in B:
for nums in num:
x,y = np.where(A == nums)
index.append([x,y])
print(index)
>>array([[0,1],
[0,4],
[1,0],
[1,2]])

With zip and np.where:
>>> list(zip(*np.where(np.in1d(A, B).reshape(A.shape))))
[(0, 1), (0, 4), (1, 0), (1, 2)]
Alternatively:
>>> np.vstack(np.where(np.isin(A,B))).transpose()
array([[0, 1],
[0, 4],
[1, 0],
[1, 2]], dtype=int64)

For each row in one array: is it in another array?

How can check for each row in a numpy array if it can be found in another array? So if I have
import numpy as np
a = np.array([[1, 2.6], [3, 4], [2.6, 1]])
b = np.array([[0, 0], [3, 4], [1, 2.6], [4.3, 4], [5, 5]])
I would like to get
array([True, True, False])
Obvioulsly
np.isin(a,b)
array([[ True, True],
[ True, True],
[ True, True]])
is not an answer and of course I can write something like
return_ = np.zeros(a.shape[0], dtype=bool)
for index, loc in enumerate(a):
for loc2 in b:
if np.allclose(loc, loc2):
return_[index] = True
break
but this is slow and looks horrible. I would prefer using proper numpy commands.

You could try creating a boolean index with appropriate broadcasting and then checking in the result for a True subarray
np.any(np.all(b == a[:, None, :], axis=-1), axis=1)

Finding a matching row in a numpy matrix

Using numpy, I have a matrix called points.
points
=> matrix([[0, 2],
[0, 0],
[1, 3],
[4, 6],
[0, 7],
[0, 3]])
If I have the tuple (1, 3), I want to find the row in points that matches these numbers (in this case, the row index is 2).
I tried using np.where:
np.where(points == (1, 3))
=> (array([2, 2, 5]), array([0, 1, 1]))
What is the meaning of this output? Can it be used to find the row where (1, 3) occurs?

You were just needed to look for ALL matches along each row, like so -
np.where((a==(1,3)).all(axis=1))[0]
Steps involved using given sample -
In [17]: a # Input matrix
Out[17]:
matrix([[0, 2],
[0, 0],
[1, 3],
[4, 6],
[0, 7],
[0, 3]])
In [18]: (a==(1,3)) # Matrix of broadcasted matches
Out[18]:
matrix([[False, False],
[False, False],
[ True, True],
[False, False],
[False, False],
[False, True]], dtype=bool)
In [19]: (a==(1,3)).all(axis=1) # Look for ALL matches along each row
Out[19]:
matrix([[False],
[False],
[ True],
[False],
[False],
[False]], dtype=bool)
In [20]: np.where((a==(1,3)).all(1))[0] # Use np.where to get row indices
Out[20]: array([2])

Using numpy.where() to iterate through a matrix

There's something about numpy.where() I do not understand:
Let's say I have a 2D numpy ndarray:
import numpy as np
twodim = np.array([[1, 2, 3, 4], [1, 6, 7, 8], [1, 1, 1, 12], [17, 3, 15, 16], [17, 3, 18, 18]])
Now, would like to create a function which "checks" this numpy array for a variety of conditions.
array([[ 1, 2, 3, 4],
[ 1, 6, 7, 8],
[ 1, 1, 1, 12],
[17, 3, 15, 16],
[17, 3, 18, 18]])
For example, which entries in this array have (A) even numbers (B) greater than 7 (C) divisible by 3?
I would like to use numpy.where() for this, and iterate through each entry of this array, finally finding the elements which match all conditions (if such an entry exists):
even_entries = np.where(twodim % 2 == 0)
greater_seven = np.where(twodim > 7 )
divisible_three = np.where(twodim % 3 == 0)
How does one do this? I am not sure how to iterate through Booleans...
I could access the indices of the matrix (i,j) via
np.argwhere(even_entries)
We could do something like
import numpy as np
twodim = np.array([[1, 2, 3, 4], [1, 6, 7, 8], [1, 1, 1, 12], [17, 3, 15, 16], [17, 3, 18, 18]])
even_entries = np.where(twodim % 2 == 0)
greater_seven = np.where(twodim > 7 )
divisible_three = np.where(twodim % 3 == 0)
for row in even_entries:
for item in row:
if item: #equivalent to `if item == True`
for row in greater_seven:
for item in row:
if item: #equivalent to `if item == True`
for row in divisible_three:
for item in row:
if item: #equivalent to `if item == True`
# something like print(np.argwhere())
Any advice?
EDIT1: Great ideas below. As #hpaulj mentions "Your tests produce a boolean matrix of the same shape as twodim"
This is a problem I'm running into as I toy around---not all conditionals produce matrices the same shape as my starting matrix. For instance, let's say I'm comparing whether the array element has a matching array to the left or right (i.e. horizontally)
twodim[:, :-1] == twodim[:, 1:]
That results in a (5,3) Boolean array, whereas our original matrix is a (5,4) array
array([[False, False, False],
[False, False, False],
[ True, True, False],
[False, False, False],
[False, False, True]], dtype=bool)
If we do the same vertically, that results in a (4,4) Boolean array, whereas the original matrix is (5,4)
twodim[:-1] == twodim[1:]
array([[ True, False, False, False],
[ True, False, False, False],
[False, False, False, False],
[ True, True, False, False]], dtype=bool)
If we wished to know which entries have both vertical and horizontal pairs, it is non-trivial to figure out which dimension we are in.

Your tests produce a boolean matrix of the same shape as twodim:
In [487]: mask3 = twodim%3==0
In [488]: mask3
Out[488]:
array([[False, False, True, False],
[False, True, False, False],
[False, False, False, True],
[False, True, True, False],
[False, True, True, True]], dtype=bool)
As other answers noted you can combine tests logically - with and and or.
np.where is the same as np.nonzero (in this use), and just returns the coordinates of the True values - as a tuple of 2 arrays.
In [489]: np.nonzero(mask3)
Out[489]:
(array([0, 1, 2, 3, 3, 4, 4, 4], dtype=int32),
array([2, 1, 3, 1, 2, 1, 2, 3], dtype=int32))
argwhere returns the same values, but as a transposed 2d array.
In [490]: np.argwhere(mask3)
Out[490]:
array([[0, 2],
[1, 1],
[2, 3],
[3, 1],
[3, 2],
[4, 1],
[4, 2],
[4, 3]], dtype=int32)
Both the mask and tuple can be used to index your array directly:
In [494]: twodim[mask3]
Out[494]: array([ 3, 6, 12, 3, 15, 3, 18, 18])
In [495]: twodim[np.nonzero(mask3)]
Out[495]: array([ 3, 6, 12, 3, 15, 3, 18, 18])
The argwhere can't be used directly for indexing, but may be more suitable for iteration, especially if you want the indexes as well as the values:
In [496]: for i,j in np.argwhere(mask3):
.....: print(i,j,twodim[i,j])
.....:
0 2 3
1 1 6
2 3 12
3 1 3
3 2 15
4 1 3
4 2 18
4 3 18
The same thing with where requires a zip:
for i,j in zip(*np.nonzero(mask3)): print(i,j,twodim[i,j])
BUT in general in numpy we try to avoid iteration. If you can use twodim[mask] directly your code will be much faster.
Logical combinations of the boolean masks are easier to produce than combinations of the where indices. To use the indices I'd probably resort to set operations (union, intersect, difference).
As for a reduced size test, you have to decide how that maps on to the original array (and other tests). e.g.
A (5,3) mask (difference between columns):
In [505]: dmask=np.diff(twodim, 1).astype(bool)
In [506]: dmask
Out[506]:
array([[ True, True, True],
[ True, True, True],
[False, False, True],
[ True, True, True],
[ True, True, False]], dtype=bool)
It can index 3 columns of the original array
In [507]: twodim[:,:-1][dmask]
Out[507]: array([ 1, 2, 3, 1, 6, 7, 1, 17, 3, 15, 17, 3])
In [508]: twodim[:,1:][dmask]
Out[508]: array([ 2, 3, 4, 6, 7, 8, 12, 3, 15, 16, 3, 18])
It can also be combined with 3 columns of another mask:
In [509]: dmask & mask3[:,:-1]
Out[509]:
array([[False, False, True],
[False, True, False],
[False, False, False],
[False, True, True],
[False, True, False]], dtype=bool)
It is still easier to combine tests in the boolean array form than with where indices.

import numpy as np
twodim = np.array([[1, 2, 3, 4], [1, 6, 7, 8], [1, 1, 1, 12], [17, 3, 15, 16], [17, 3, 18, 18]])
condition = (twodim % 2. == 0.) & (twodim > 7.) & (twodim % 3. ==0.)
location = np.argwhere(condition == True)
for i in location:
print i, twodim[i[0],i[1]],
>>> [2 3] 12 [4 2] 18 [4 3] 18

If you want to find where all three conditions are satisfied:
import numpy as np
twodim = np.array([[1, 2, 3, 4], [1, 6, 7, 8], [1, 1, 1, 12], [17, 3, 15, 16], [17, 3, 18, 18]])
mask = (twodim % 2 == 0) & (twodim > 7) & (twodim % 3 =0)
print(twodim[mask])
[12 18 18]
Not sure what you want in the end whether all elements in the row must satisfy the condition and to find those rows or if you want individual elements.

Sort on multiple NumPy arrays

I am creating a 2 dimensional numpy array that contains stock returns. I want to sum the return every 2 days, and if the sum is in the top two, I will set every element in a similar shaped array to True.
For example, returns below is the daily returns for four different stocks.
returns=np.array([
[0, 0, 4, 8],
[7, 5, 4, 1],
[10, 5, 7, 6],
[7, 5, 4, 2]])
For the first two days, columns 2 and 3 (using 0 based indexing) have the highest sums. For the second set of two days, columns 0 and 2 have the highest sums. The output array I want is
bools=np.array([
[False, False, True, True],
[False, False, True, True],
[True, False, True, False],
[True, False, True, False]])
What are good ways to accomplish this?
If there are ties with the sums of two days, I want to use another similarly shaped numpy array as tiebreakers.
For example, if
returns=np.array([
[0, 9, 4, 8],
[7, 5, 4, 0],
[10, 5, 7, 6],
[7, 5, 4, 2]])
For the first set of two days, columns 2 and 3 are tied for the second highest sum. I want to decide the tiebreaker by the greatest value in the last row for the tied columns so that the tie break between columns 2 and 3 look at tiebreaks[1][2] vs tiebreaks[1][3] (4 v 5), and that the ultimate output is bools2.
tiebreaks=np.array([
[0, 0, 1, 1],
[2, 3, 4, 5],
[0, 5, 7, 6],
[-7, 5, -4, 2]])
bools2=np.array([
[False, True, False, True],
[False, True, False, True],
[True, False, True, False],
[True, False, True, False]])
Thanks for your help.

You can use numpy.lexsort() to get the indices that sort your arrays using prices as primary key and names as secondary key. Applying advanced indexing using these indices yields the sorted arrays:
col_indices = numpy.lexsort((names, prices))
row_indices = numpy.arange(len(names))[:, None]
print(prices[row_indices, col_indices])
print(names[row_indices, col_indices])
(Note that in your example, names and prices don't have compatible shapes.)

Categories