Say I have an array like:
a1 = np.array([.1, .2, 23., 4.3, 3.2, .1, .05, .2, .3, 4.2, 7.6])
And I filter out, and create a mask, of all values less than 1, like:
a2 = a1[a1 >= 1]
a2_mask = np.ma.masked_where(a1 < 1, a1)
And then search for a specific value:
a2_idx = np.where(a2==3.2)[0][0]
How would I convert that index to the corresponding index in the original array?
e.g.
>>> a2_idx
2
>>> a1_idx = reframe_index(a2_idx, a2_mask)
>>> a1_idx
4
My naive implementation would be:
def reframe_index(old_idx, mask):
cnt = 0
ref = 0
for v in mask:
if not isinstance(v, (int, float)):
cnt += 1
else:
if ref == old_idx:
return ref + cnt
ref += 1
Does Numpy have a more efficient way to do this?
a2 is a copy, so there's no link between it an a1 - except for some values.
In [19]: a2
Out[19]: array([23. , 4.3, 3.2, 4.2, 7.6])
In [20]: np.nonzero(a2 == 3.2)
Out[20]: (array([2]),)
In [21]: a2[2]
Out[21]: 3.2
The mask of a2_mask, just a1<1, does give us a way of finding the corresponding element of a1:
In [22]: a2_mask = np.ma.masked_where(a1 < 1, a1)
In [23]: a2_mask
Out[23]:
masked_array(data=[--, --, 23.0, 4.3, 3.2, --, --, --, --, 4.2, 7.6],
mask=[ True, True, False, False, False, True, True, True,
True, False, False],
fill_value=1e+20)
In [24]: a2_mask.compressed()
Out[24]: array([23. , 4.3, 3.2, 4.2, 7.6])
In [25]: a2_mask.mask
Out[25]:
array([ True, True, False, False, False, True, True, True, True,
False, False])
In [26]: np.nonzero(~a2_mask.mask)
Out[26]: (array([ 2, 3, 4, 9, 10]),)
In [27]: np.nonzero(~a2_mask.mask)[0][2]
Out[27]: 4
In [28]: a1[4]
Out[28]: 3.2
So you need the mask or indices used to select a2 in the first place. a2 itself does not have the information.
In [30]: np.nonzero(a1>=1)
Out[30]: (array([ 2, 3, 4, 9, 10]),)
In [31]: np.nonzero(a1 >= 1)[0][2]
Out[31]: 4
I had a similar problem recently, so I made haggis.npy_util.unmasked_index1. This function has a lot of overkill for your relatively simple case, because it's intended to operate on an arbitrary number of dimensions. That being said, given
>>> arr = np.array([.1, .2, 23., 4.3, 3.2, .1, .05, .2, .3, 4.2, 7.6])
and
>>> mask = arr >= 1
>>> mask
array([False, False, True, True, True, False, False, False, False,
True, True])
You can do something like
>>> idx = unmasked_index(np.flatnonzero(arr[mask] == 3.2), mask)
>>> idx
array([4])
If you ever need it, there is also an inverse function haggis.npy_util.masked_index that converts a location in a multidimensional input array into its index in the masked array.
1Disclaimer: I am the author of haggis.
Related
Say there's a np.float32 matrix A of shape (N, M). Together with A, I possess another matrix B, of type np.bool, of the exact same shape (elements from A can be mapped 1:1 to B). Example:
A =
[
[0.1, 0.2, 0.3],
[4.02, 123.4, 534.65],
[2.32, 22.0, 754.01],
[5.41, 23.1, 1245.5],
[6.07, 0.65, 22.12],
]
B =
[
[True, False, True],
[False, False, True],
[True, True, False],
[True, True, True],
[True, False, True],
]
Now, I'd like to perform np.max, np.min, np.argmax and np.argmin on axis=1 of A, but only considering elements A[i,j] for which B[i,j] == True. Is it possible to do something like this in NumPy? The for-loop version is trivial, but I'm wondering whether I can get some of that juicy NumPy speed.
The result for A, B and np.max (for example) would be:
[ 0.3, 534.65, 22.0, 1245.5, 22.12 ]
I've avoided ma because I've heard that the computation gets very slow and I don't feel like specifying fill_value makes sense in this context. I just want the numbers to be ignored.
Also, if it matters at all in my case, N ranges in thousands and M ranges in units.
This is a textbook application for masked arrays. But as always there are other ways to do it.
import numpy as np
A = np.array([[ 0.1, 0.2, 0.3],
[ 4.02, 123.4, 534.65],
[ 2.32, 22.0, 754.01],
[ 5.41, 23.1, 1245.5],
[ 6.07, 0.65, 22.12]])
B = np.array([[ True, False, True],
[False, False, True],
[ True, True, False],
[ True, True, True],
[ True, False, True]])
With nanmax etc.
You could cast the 'invalid' values to NaN (say), then use NumPy's special NaN-ignoring functions:
>>> A[~B] = np.nan # <-- Note this mutates A
>>> np.nanmax(A, axis=1)
array([3.0000e-01, 5.3465e+02, 2.2000e+01, 1.2455e+03, 2.2120e+01])
The catch is that, while np.nanmax, np.nanmin, np.nanargmax, and np.nanargmin all exist, lots of functions don't have a non-NaN twin, so you might have to come up with something else eventually.
With ma
It seems weird not to mention masked arrays, which are straightforward. Notice that the mask is (to my mind anyway) 'backwards'. That is, True means the value is 'masked' or invalid and will be ignored. Hence having to negate B with the tilde. Then you can do what you want with the masked array:
>>> X = np.ma.masked_array(A, mask=~B) # <--- Note the tilde.
>>> np.max(X, axis=1)
masked_array(data=[0.3, 534.65, 22.0, 1245.5, 22.12],
mask=[False, False, False, False, False],
fill_value=1e+20)
I am trying to delete an element from an array. When trying to delete integer values(using numpy.delete) it's working but it doesn't work for decimal values.
For integer deletion
X = [1. 2. 2.5 5.7 3. 6. ]
to_delete_key = [3, 7.3]
Y = np.delete(X, to_delete_key, None)
Output is [1. 2. 2.5 5.7 6. ]
The value 3 got deleted
Whereas in the case of decimal deletion
For decimal deletion
X = [6. 7.3 9.1]
to_delete_key = [3, 7.3]
Y = np.delete(X, to_delete_key, None)
Output is [6. 7.3 9.1]
The value 7.3 didn't get deleted.
I know how to do it the normal way but is there any efficient pythonic way to do it
In [249]: X = np.array([1., 2., 2.5, 5.7, 3., 6. ])
...: to_delete_key = [3, 7.3]
In [252]: np.delete(X, to_delete_key)
Traceback (most recent call last):
File "<ipython-input-252-f9031065a548>", line 1, in <module>
np.delete(X, to_delete_key)
File "<__array_function__ internals>", line 5, in delete
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py", line 4406, in delete
keep[obj,] = False
IndexError: arrays used as indices must be of integer (or boolean) type
Using an integer:
In [253]: np.delete(X, 3)
Out[253]: array([1. , 2. , 2.5, 3. , 6. ])
It was the 5.7 that was deleted, X[3].
np.delete does not delete by value! From the docs:
obj : slice, int or array of ints
Indicate indices of sub-arrays to remove along the specified axis.
We can look for value matches
In [267]: vals = [3, 2.5]
In [268]: X[:,None]==vals
Out[268]:
array([[False, False],
[False, False],
[False, True],
[False, False],
[ True, False],
[False, False]])
But equality match on floats can be unreliable. isclose operates with a tolerance:
In [269]: np.isclose(X[:,None],vals)
Out[269]:
array([[False, False],
[False, False],
[False, True],
[False, False],
[ True, False],
[False, False]])
Then find the rows where there's a match:
In [270]: _.any(axis=1)
Out[270]: array([False, False, True, False, True, False])
In [271]: X[_]
Out[271]: array([2.5, 3. ])
In [272]: X[~__]
Out[272]: array([1. , 2. , 5.7, 6. ])
Lists have a remove by value:
In [284]: alist=X.tolist()
In [285]: alist.remove(3.0)
In [286]: alist.remove(2.5)
In [287]: alist
Out[287]: [1.0, 2.0, 5.7, 6.0]
You are dealing with floating-point numbers that cannot be compared exactly. Google out "What every programmer should know about floating-point numbers".
1/3 + 1/3 + 1/3 might not be equal to 1 due to rounding errors.
So the explanation is that your value of 7.3 is not found. Numpy probably converted 7.3 to a 32-bit float or whatever that is not exactly equal to what is in the array.
As mentioned by #elPastor, you are misusing Numpy.
I have a numpy array like
np.array([[1.0, np.nan, 5.0, 1, True, True, np.nan, True],
[np.nan, 4.0, 7.0, 2, True, np.nan, False, True],
[2.0, 5.0, np.nan, 3, False, False, True, np.nan]], dtype=object)
Now I want to sort the values with key as isnan? How can I do that? So that I would end up in the array
np.array([[1.0, 5.0, 1, True, True, True, np.nan, np.nan],
[4.0, 7.0, 2, True, False, True, np.nan, np.nan],
[2.0, 5.0, 3, False, False, True, np.nan, np.nan]], dtype=object)
np.sort() didn't work. The same can be achieved in pandas by applying sorted over columns with sorted function with key as pd.isnull(), but looking for a numpy answer for speed.
In pandas
data = pd.DataFrame({'Key': [1, 2, 3], 'Var': [True, True, False], 'ID_1':[1, np.NaN, 2],
'Var_1': [True, np.NaN, False], 'ID_2': [np.NaN, 4, 5], 'Var_2': [np.NaN, False, True],
'ID_3': [5, 7, np.NaN], 'Var_3': [True, True, np.NaN]})
data.apply(lambda x : sorted(x,key=pd.isnull),1).values
Output :
array([[1.0, 5.0, 1, True, True, True, nan, nan],
[4.0, 7.0, 2, True, False, True, nan, nan],
[2.0, 5.0, 3, False, False, True, nan, nan]], dtype=object)
Approach #1
Here's a vectorized approach borrowing the concept of masking from this post -
def mask_app(a):
out = np.empty_like(a)
mask = np.isnan(a.astype(float))
mask_sorted = np.sort(mask,1)
out[mask_sorted] = a[mask]
out[~mask_sorted] = a[~mask]
return out
Sample run -
# Input dataframe
In [114]: data
Out[114]:
ID_1 ID_2 ID_3 Key Var Var_1 Var_2 Var_3
0 1.0 NaN 5.0 1 True True NaN True
1 NaN 4.0 7.0 2 True NaN False True
2 2.0 5.0 NaN 3 False False True NaN
# Use pandas approach for verification
In [115]: data.apply(lambda x : sorted(x,key=pd.isnull),1).values
Out[115]:
array([[1.0, 5.0, 1, True, True, True, nan, nan],
[4.0, 7.0, 2, True, False, True, nan, nan],
[2.0, 5.0, 3, False, False, True, nan, nan]], dtype=object)
# Use proposed approach and verify
In [116]: mask_app(data.values)
Out[116]:
array([[1.0, 5.0, 1, True, True, True, nan, nan],
[4.0, 7.0, 2, True, False, True, nan, nan],
[2.0, 5.0, 3, False, False, True, nan, nan]], dtype=object)
Approach #2
With few more modifications, a simplified version with the idea from this post -
def mask_app2(a):
out = np.full(a.shape,np.nan,dtype=a.dtype)
mask = ~np.isnan(a.astype(float))
out[np.sort(mask,1)[:,::-1]] = a[mask]
return out
Since you have an object array anyway, do the sorting in Python, then make your array. You can write a key that does something like this:
from math import isnan
def key(x):
if isnan(x):
t = 3
x = 0
elif isinstance(x, bool):
t = 2
else:
t = 1
return t, x
This key returns a two-element tuple, where the first element gives the preliminary ordering by type. It considers all NaNs to be equal and greater than any other type.
Even if you start with data in a DataFrame, you can do something like:
values = [list(sorted(row, key=key)) for row in data.values]
values = np.array(values, dtype=np.object)
You can replace the list comprehension with np.apply_along_axis if that suits your needs better:
values = np.apply_along_axis(lambda row: np.array(list(sorted(row, key=key))),
axis=1, arr=data.values)
You can't do this with an object array and nan You would need to find a numeric type everything would fit into. When used as an object instead of as a float, nan returns false for <, >, and ==.
Additionally, True and False are equivalent to 0 and 1, so I don't think there is any way to get your expected result.
You would have to see if converting the dtype to float would give you proper results for your use case.
I can understand following numpy behavior.
>>> a
array([[ 0. , 0. , 0. ],
[ 0. , 0.7, 0. ],
[ 0. , 0.3, 0.5],
[ 0.6, 0. , 0.8],
[ 0.7, 0. , 0. ]])
>>> argmax_overlaps = a.argmax(axis=1)
>>> argmax_overlaps
array([0, 1, 2, 2, 0])
>>> max_overlaps = a[np.arange(5),argmax_overlaps]
>>> max_overlaps
array([ 0. , 0.7, 0.5, 0.8, 0.7])
>>> gt_argmax_overlaps = a.argmax(axis=0)
>>> gt_argmax_overlaps
array([4, 1, 3])
>>> gt_max_overlaps = a[gt_argmax_overlaps,np.arange(a.shape[1])]
>>> gt_max_overlaps
array([ 0.7, 0.7, 0.8])
>>> gt_argmax_overlaps = np.where(a == gt_max_overlaps)
>>> gt_argmax_overlaps
(array([1, 3, 4]), array([1, 2, 0]))
I understood 0.7, 0.7 and 0.8 is a[1,1],a[3,2] and a[4,0] so I got the tuple (array[1,3,4] and array[1,2,0]) each array of which composed of 0th and 1st indices of those three elements. I then tried other examples to see my understanding is correct.
>>> np.where(a == [0.3])
(array([2]), array([1]))
0.3 is in a[2,1] so the outcome looks as I expected. Then I tried
>>> np.where(a == [0.3, 0.5])
(array([], dtype=int64),)
?? I expected to see (array([2,2]),array([2,3])). Why do I see the output above?
>>> np.where(a == [0.7, 0.7, 0.8])
(array([1, 3, 4]), array([1, 2, 0]))
>>> np.where(a == [0.8,0.7,0.7])
(array([1]), array([1]))
I can't understand the second result either. Could someone please explain it to me? Thanks.
The first thing to realize is that np.where(a == [whatever]) is just showing you the indices where a == [whatever] is True. So you can get a hint by looking at the value of a == [whatever]. In your case that "works":
>>> a == [0.7, 0.7, 0.8]
array([[False, False, False],
[False, True, False],
[False, False, False],
[False, False, True],
[ True, False, False]], dtype=bool)
You aren't getting what you think you are. You think that is asking for the indices of each element separately, but instead it's getting the positions where the values match at the same position in the row. Basically what this comparison is doing is saying "for each row, tell me whether the first element is 0.7, whether the second is 0.7, and whether the third is 0.8". It then returns the indices of those matching positions. In other words, the comparison is done between entire rows, not just individual values. For your last example:
>>> a == [0.8,0.7,0.7]
array([[False, False, False],
[False, True, False],
[False, False, False],
[False, False, False],
[False, False, False]], dtype=bool)
You now get a different result. It's not asking for "the indices where a has value 0.8", it's asking for only the indices where there is a 0.8 at the beginning of the row -- and likewise a 0.7 in either of the later two positions.
This type of row-wise comparison can only be done if the value you compare against matches the shape of a single row of a. So when you try it with a two-element list, it returns an empty set, because there it is trying to compare the list as a scalar value against individual values in your array.
The upshot is that you can't use == on a list of values and expect it to just tell you where any of the values occurs. The equality will match by value and position (if the value you compare against is the same shape as a row of your array), or it will try to compare the whole list as a scalar (if the shape doesn't match). If you want to search for the values independently, you need to do something like what Khris suggested in a comment:
np.where((a==0.3)|(a==0.5))
That is, you need to make two (or more) separate comparisons against separate values, not a single comparison against a list of values.
Given a matrix of values that represent probabilities I am trying to write an efficient process that returns the bin that the value belongs to. For example:
sample = 0.5
x = np.array([0.1]*10)
np.digitize( sample, np.cumsum(x))-1
#returns 5
is the result I am looking for.
According to timeit for x arrays with few elements it is more efficient to do it as:
cdf = 0
for key,val in enumerate(x):
cdf += val
if sample<=cdf:
print key
break
while for bigger x arrays the numpy solution is faster.
The question:
Is there a way to further accelerate it, e.g., a function that combines the steps?
Can we vectorize the process it for the case where sample is a list, whose each item is associated with its own x array (x will then be 2-D)?
In the application x contains the marginal probabilities; this is way I need to decrement the results of np.digitize
You could use some broadcasting magic there -
(x.cumsum(1) > sample[:,None]).argmax(1)-1
Steps involved :
I. Perform cumsum along each row.
II. Use broadcasted comparison for each cumsum row against each sample value and look for the first occurrence of sample being lesser than cumsum values, signalling that the element before that in x is the index we are looking for.
Step-by-step run -
In [64]: x
Out[64]:
array([[ 0.1 , 0.1 , 0.1 , 0.1 , 0.1 , 0.1 , 0.1 ],
[ 0.8 , 0.96, 0.88, 0.36, 0.5 , 0.68, 0.71],
[ 0.37, 0.56, 0.5 , 0.01, 0.77, 0.88, 0.36],
[ 0.62, 0.08, 0.37, 0.93, 0.65, 0.4 , 0.79]])
In [65]: sample # one elem per row of x
Out[65]: array([ 0.5, 2.2, 1.9, 2.2])
In [78]: x.cumsum(1)
Out[78]:
array([[ 0.1 , 0.2 , 0.3 , 0.4 , 0.5 , 0.6 , 0.7 ],
[ 0.8 , 1.76, 2.64, 2.99, 3.49, 4.18, 4.89],
[ 0.37, 0.93, 1.43, 1.45, 2.22, 3.1 , 3.47],
[ 0.62, 0.69, 1.06, 1.99, 2.64, 3.04, 3.83]])
In [79]: x.cumsum(1) > sample[:,None]
Out[79]:
array([[False, False, False, False, False, True, True],
[False, False, True, True, True, True, True],
[False, False, False, False, True, True, True],
[False, False, False, False, True, True, True]], dtype=bool)
In [80]: (x.cumsum(1) > sample[:,None]).argmax(1)-1
Out[80]: array([4, 1, 3, 3])
# A loopy solution to verify results against
In [81]: [np.digitize( sample[i], np.cumsum(x[i]))-1 for i in range(x.shape[0])]
Out[81]: [4, 1, 3, 3]
Boundary cases :
The proposed solution automatically handles the cases where sample values are lesser than smallest of cumulative summed values -
In [113]: sample[0] = 0.08 # editing first sample to be lesser than 0.1
In [114]: [np.digitize( sample[i], np.cumsum(x[i]))-1 for i in range(x.shape[0])]
Out[114]: [-1, 1, 3, 3]
In [115]: (x.cumsum(1) > sample[:,None]).argmax(1)-1
Out[115]: array([-1, 1, 3, 3])
For cases where a sample value is greater than largest of cumulative summed values, we need one extra step -
In [116]: sample[0] = 0.8 # editing first sample to be greater than 0.7
In [121]: mask = (x.cumsum(1) > sample[:,None])
In [122]: idx = mask.argmax(1)-1
In [123]: np.where(mask.any(1),idx,x.shape[1]-1)
Out[123]: array([6, 1, 3, 3])
In [124]: [np.digitize( sample[i], np.cumsum(x[i]))-1 for i in range(x.shape[0])]
Out[124]: [6, 1, 3, 3]