In numpy.argmax function, tie breaking between multiple max elements is so that the first element is returned.
Is there a functionality for randomizing tie breaking so that all maximum numbers have equal chance of being selected?
Below is an example directly from numpy.argmax documentation.
>>> b = np.arange(6)
>>> b[1] = 5
>>> b
array([0, 5, 2, 3, 4, 5])
>>> np.argmax(b) # Only the first occurrence is returned.
1
I am looking for ways so that 1st and 5th elements in the list are returned with equal probability.
Thank you!
Use np.random.choice -
np.random.choice(np.flatnonzero(b == b.max()))
Let's verify for an array with three max candidates -
In [298]: b
Out[298]: array([0, 5, 2, 5, 4, 5])
In [299]: c=[np.random.choice(np.flatnonzero(b == b.max())) for i in range(100000)]
In [300]: np.bincount(c)
Out[300]: array([ 0, 33180, 0, 33611, 0, 33209])
In the case of a multi-dimensional array, choice won't work.
An alternative is
def randargmax(b,**kw):
""" a random tie-breaking argmax"""
return np.argmax(np.random.random(b.shape) * (b==b.max()), **kw)
If for some reason generating random floats is slower than some other method, random.random can be replaced with that other method.
Easiest way is
np.random.choice(np.where(b == b.max())[0])
Since the accepted answer may not be obvious, here is how it works:
b == b.max() will return an array of booleans, with values of true where items are max and values of false for other items
flatnonzero() will do two things: ignore the false values (nonzero part) then return indices of true values. In other words, you get an array with indices of items matching the max value
Finally, you pick a random index from the array
Additional to #Manux's answer,
Changing b.max() to np.amax(b,**kw, keepdims=True) will let you do it along axes.
def randargmax(b,**kw):
""" a random tie-breaking argmax"""
return np.argmax(np.random.random(b.shape) * (b==b.max()), **kw)
randargmax(b,axis=None)
Here is a comparison between the two main solutions by #divakar and #shyam-padia :
method (1) - using np.where
np.random.choice(np.where(b == b.max())[0])
method (2) - using np.flatnonzero
np.random.choice(np.flatnonzero(b == b.max())
Code
Here is the code I wrote for the comparison:
def method1(b, bmax,):
return np.random.choice(np.where(b == bmax)[0])
def method2(b, bmax):
return np.random.choice(np.flatnonzero(b == bmax))
def time_it(n):
b = np.array([1.0, 2.0, 5.0, 5.0, 0.4, 0.1, 5.0, 0.3, 0.1])
bmax = b.max()
start = time.perf_counter()
for i in range(n):
method1(b, bmax)
elapsed1 = time.perf_counter() - start
start = time.perf_counter()
for i in range(n):
method2(b, bmax)
elapsed2 = time.perf_counter() - start
print(f'method1 time: {elapsed1} - method2 time: {elapsed2}')
return elapsed1, elapsed2
Results
The following figure shows the computation time for running each method for [100, 1000, 10000, 100000, 1000000] iterations where x-axis represents number of iterations, y-axis shows time in seconds. It can be seen that np.where performs better than np.flatnonzero when number of iterations increases. Note that the x-axis has a logarithmic scale.
To show how the two methods compare in the lower iteration, we can re-plot the previous results by making the y-axis being a logarithmic scale. We can see that np.where stays always better than np.flatnonzero.
Related
This is an example of what I am trying to do. Suppose the following numpy array:
A = np.array([3, 0, 1, 5, 7]) # in practice, this array is a huge array of float numbers: A.shape[0] >= 1000000
I need the fastest possible way to get the following result:
result = []
for a in A:
result.append( 1 / np.exp(A - a).sum() )
result = np.array(result)
print(result)
>>> [1.58297157e-02 7.88115138e-04 2.14231906e-03 1.16966657e-01 8.64273193e-01]
Option 1 (faster than previous code):
result = 1 / np.exp(A - A[:,None]).sum(axis=1)
print(result)
>>> [1.58297157e-02 7.88115138e-04 2.14231906e-03 1.16966657e-01 8.64273193e-01]
Is there a faster way to get "result" ?
EDIT: yes, scipy.special.softmax did the trick
Rather than trying to compute each value by normalizing it in place (effectively adding up all the values, repeatedly for each value), instead just get the exponentials and then normalize once at the end. So:
raw = np.exp(A)
result = A / sum(A)
(In my testing, the builtin sum is over 2.5x as fast as np.sum for summing a small array. I did not test with larger ones.)
Yes: scipy.special.softmax did the trick
from scipy.special import softmax
result = softmax(A)
Thank you #j1-lee and #Karl Knechtel
numpy.nanpercentile is extremely slow.
So, I wanted to use cupy.nanpercentile; but there is not cupy.nanpercentile implemented yet.
Do someone have solution for it?
I also had a problem with np.nanpercentile being very slow for my datasets. I found a wokraround that lets you use the standard np.percentile. And it can also be applied to many other libs.
This one should solve your problem. And it also works alot faster than np.nanpercentile:
arr = np.array([[np.nan,2,3,1,2,3],
[np.nan,np.nan,1,3,2,1],
[4,5,6,7,np.nan,9]])
mask = (arr >= np.nanmin(arr)).astype(int)
count = mask.sum(axis=1)
groups = np.unique(count)
groups = groups[groups > 0]
p90 = np.zeros((arr.shape[0]))
for g in range(len(groups)):
pos = np.where (count == groups[g])
values = arr[pos]
values = np.nan_to_num (values, nan=(np.nanmin(arr)-1))
values = np.sort (values, axis=1)
values = values[:,-groups[g]:]
p90[pos] = np.percentile (values, 90, axis=1)
So instead of taking the percentile with the nans, it sorts the rows by the amount of valid data, and takes the percentile of those rows separated. Then adds everything back together. This also works for 3D-arrays, just add y_pos and x_pos instead of pos. And watch out for what axis you are calculating over.
def testset_gen(num):
init=[]
for i in range (num):
a=random.randint(65,122) # Dummy name
b=random.randint(1,100) # Dummy value: 11~100 and 10% of nan
if b<11:
b=np.nan # 10% = nan
init.append([a,b])
return np.array(init)
np_testset=testset_gen(30000000) # 468,751KB
def f1_np (arr, num):
return np.percentile (arr[:,1], num)
# 55.0, 0.523902416229248 sec
print (f1_np(np_testset[:,1], 50))
def cupy_nanpercentile (arr, num):
return len(cp.where(arr > num)[0]) / (len(arr) - cp.sum(cp.isnan(arr))) * 100
# 55.548758317136446, 0.3640251159667969 sec
# 43% faster
# If You need same result, use int(). But You lose saved time.
print (cupy_nanpercentile(cp_testset[:,1], 50))
I can't imagine How test result takes few days. With my computer, It seems 1 Trillion line of data or more. Because of this, I can't reproduce same problem due to lack of resource.
Here's an implementation with numba. After it's been compiled it is more than 7x faster than the numpy version.
Right now it is set up to take the percentile along the first axis, however it could be changed easily.
#numba.jit(nopython=True, cache=True)
def nan_percentile_axis0(arr, percentiles):
"""Faster implementation of np.nanpercentile
This implementation always takes the percentile along axis 0.
Uses numba to speed up the calculation by more than 7x.
Function is equivalent to np.nanpercentile(arr, <percentiles>, axis=0)
Params:
arr (np.array): Array to calculate percentiles for
percentiles (np.array): 1D array of percentiles to calculate
Returns:
(np.array) Array with first dimension corresponding to
values as passed in percentiles
"""
shape = arr.shape
arr = arr.reshape((arr.shape[0], -1))
out = np.empty((len(percentiles), arr.shape[1]))
for i in range(arr.shape[1]):
out[:,i] = np.nanpercentile(arr[:,i], percentiles)
shape = (out.shape[0], *shape[1:])
return out.reshape(shape)
I want to find the number of occurrences of vector v in matrix M.
What I have is a matrix the size (60K, 10)
and I initialised a test vector v (1,10):
tester = np.zeros((1, 10))
Now I want to check how much time that vector entirely repeats itself in the matrix rows.
I did it iterative and it works, but the fact that the matrix is very large, it affects the performance and im trying to find some more elegant and faster way.
would appreciate some help
Thanks.
you can do the following:
temp = np.where((prediction == tester)).all(axis=1))
len(temp[0])
what np.where() returns in the case it has no values [x,y] accept for the condition is the indices, in your case it will return the True and False option, starting from the True.
so using this will sure to lower your running time, and for me its much more elegant then looping through the matrix.
you can check np.where api:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.where.html
Just compare and use all, so each row will result in a True value only if all its elements compare equal to the reference array. Then, you can simply sum the result, since int(True) == 1.
Example:
np.random.seed(0)
data = np.random.randint(0, 2, size=(50, 3))
to_match = np.random.randint(0, 2, size=(1, 3))
print(to_match)
print((data == to_match).all(axis=1).sum())
Output:
[[0 0 0]]
4
...which means that there are 4 instances of [0, 0, 0] in data.
There is two case, the first one, my function Y1 return always the same number. In that case, it doesn't work, y is equal to a integer of 10 and not a array of one thousand 10. The second case, where it's return differents numbers, it works!
First case (Doesn't work as expected)
def Y1(x, N):
return 10
x= np.linspace(-2,2,1000)
y= Y1(x,0) #In that case, it should create a array with 1000 numbers, but it only return one int, 10.
y value: 10 #when it should be [10 10 10 10 10 10...]
The othercases (Does work as expected)
def Y1(x, N):
return x**2
x= np.linspace(-2,2,1000)
y= Y1(x,0) #it returns a array, all numbers are differents
y value:
[4.00000000e+00 3.98400002e+00 3.96803210e+00 3.95209624e+00
3.93619245e+00 3.92032072e+00 3.90448106e+00 3.88867346e+00
3.87289792e+00 3.85715445e+00 3.84144304e+00 3.82576370e+00
3.81011642e+00 3.79450121e+00 3.77891806e+00 3.76336697e+00
...]
Thank you!
In the first case, you are returning the constant 10.
In the second case, you are applying the operator **2 to a np array, which overloads ** and applies the power operation element-wise to the full array. This behaviour is known as broadcasting or vectorized aritmethic operations.
This broadcasting overload of the aritmethic methods allows behaviours as the following:
np.array([1, 2, 3, 4]) + 1
>>> array([2, 3, 4, 5])
Which is what your second case uses and your first one does not.
You can learn more about this topic, for example, here.
If you want an array of shape 1000 full of 10s use numpy.full instead:
import numpy as np
y = np.full(1000, 10)
I need to fill a numpy array of three elements with random integers such that the sum total of the array is three (e.g. [0,1,2]).
By my reckoning there are 10 possible arrays:
111,
012,
021,
102,
120,
201,
210,
300,
030,
003
My ideas is to randomly generate an integer between 1 and 10 using randint, and then use a look-up table to fill the array from the above list of combinations.
Does anyone know of a better approach?
Here is how I did it:
>>> import numpy as np
>>> a=np.array([[1,1,1],[0,1,2],[0,2,1],[1,0,2],[1,2,0],[2,0,1],[2,1,0],[3,0,0],[0,3,0],[0,0,3]])
>>> a[np.random.randint(0,10)]
array([1, 2, 0])
>>> a[np.random.randint(0,10)]
array([0, 1, 2])
>>> a[np.random.randint(0,10)]
array([1, 0, 2])
>>> a[np.random.randint(0,10)]
array([3, 0, 0])
Here’s a naive programmatic way to do this for arbitrary array sizes/sums:
def n_ints_summing_to_v(n, v):
elements = (np.arange(n) == np.random.randint(0, n)) for i in range(v))
return np.sum(elements, axis=0)
This will, of course, slow down proportionally to the desired sum, but would be ok for small values.
Alternatively, we can phrase this in terms of drawing samples from the Multinomial distribution, for which there is a function available in NumPy (see here), as follows:
def n_ints_summing_to_v(n, v):
return np.random.multinomial(v, ones((n)) / float(n))
This is a lot quicker!
This problem can be solved in the generic case, where the number of elements and their sum are both configurable. One advantage of the solution below is that it does not require generating a list of all the possibilities. The idea is to pick random numbers sequentially, each of which is less than the required sum. The required sum is reduced every time you pick a number:
import numpy
def gen(numel = 3, sum = 3):
arr = numpy.zeros((numel,), dtype = numpy.int)
for i in range(len(arr) - 1): # last element must be free to fill in the sum
arr[i] = numpy.random.randint(0, sum + 1)
sum -= arr[i]
if sum == 0: break # Nothing left to do
arr[-1] = sum # ensure that everything adds up
return arr
print(gen())
This solution does not guarantee that the possibilities will all occur with the same frequency. Among the ten possibilities you list, four start with 0, three with 1, two with 2 and one with 3. This is clearly not the uniform distribution that numpy.random.randint() provides for the first digit.