Is it possible to use np.bincount but get the max instead of sum of weights? Here, bbb at index 3 has two values, 11.1 and 55.5. I want to have 55.5, not 66.6. I doubt I choose use other function but not so sure which one is good for this purpose.
bbb = np.array([ 3, 7, 11, 13, 3])
weight = np.array([ 11.1, 22.2, 33.3, 44.4, 55.5])
print np.bincount(bbb, weight, minlength=15)
OUT >> [ 0. 0. 0. 66.6 0. 0. 0. 22.2 0. 0. 0. 33.3 0. 44.4 0. ]
Note that, in fact, bbb and weight are very large (about 5e6 elements).
The solution to your 2D question is also valid for the 1D case, so you can use np.maxmimum.at
out = np.zeros(15)
np.maximum.at(out, bbb, weight)
# array([ 0. , 0. , 0. , 55.5, 0. , 0. , 0. , 22.2, 0. ,
# 0. , 0. , 33.3, 0. , 44.4, 0. ])
Approach #1 : Here's one way with np.maximum.reduceat to get the binned maximum values -
def binned_max(bbb, weight, minlength):
sidx = bbb.argsort()
weight_s = weight[sidx]
bbb_s = bbb[sidx]
cut_idx = np.flatnonzero(np.concatenate(([True], bbb_s[1:] != bbb_s[:-1])))
bbb_unq = bbb_s[cut_idx]
#Or bbb_unq, cut_idx = np.unique(bbb_s, return_index=1)
max_val = np.maximum.reduceat(weight_s, cut_idx)
out = np.zeros(minlength, dtype=weight.dtype)
out[bbb_unq] = max_val
return out
Sample run -
In [36]: bbb = np.array([ 3, 7, 11, 13, 3])
...: weight = np.array([ 11.1, 22.2, 33.3, 44.4, 55.5])
In [37]: binned_max(bbb, weight, minlength=15)
Out[37]:
array([ 0. , 0. , 0. , 55.5, 0. , 0. , 0. , 22.2, 0. ,
0. , 0. , 33.3, 0. , 44.4, 0. ])
Approach #2 : Well I was trying to check out/having fun with numba to solve this and it seems quite efficient. Here's one numba way -
from numba import njit
#njit
def numba_func(out, bins, weight, minlength):
l = len(bins)
for i in range(l):
if out[bins[i]] < weight[i]:
out[bins[i]] = weight[i]
return out
def maxat_numba(bins, weight, minlength):
out = np.zeros(minlength, dtype=weight.dtype)
out[bins] = weight.min()
numba_func(out, bins, weight, minlength)
return out
Runtime test -
The built-in with np.maximum.at looks quite neat and would be the preferred one in most scenarios, so testing the proposed one against it -
# #Nils Werner's soln with np.maximum.at
def maxat_numpy(bins, weight, minlength):
out = np.zeros(minlength)
np.maximum.at(out, bins, weight)
return out
Timings -
Case #1 :
In [155]: bbb = np.random.randint(1,1000, (10000))
In [156]: weight = np.random.rand(*bbb.shape)
In [157]: %timeit maxat_numpy(bbb, weight, minlength=bbb.max()+1)
1000 loops, best of 3: 686 µs per loop
In [158]: %timeit maxat_numba(bbb, weight, minlength=bbb.max()+1)
10000 loops, best of 3: 60.6 µs per loop
Case #2 :
In [159]: bbb = np.random.randint(1,10000, (1000000))
In [160]: weight = np.random.rand(*bbb.shape)
In [161]: %timeit maxat_numpy(bbb, weight, minlength=bbb.max()+1)
10 loops, best of 3: 66 ms per loop
In [162]: %timeit maxat_numba(bbb, weight, minlength=bbb.max()+1)
100 loops, best of 3: 5.42 ms per loop
Probably not quite as fast as the answer by Nils, but the numpy_indexed package (disclaimer: I am its author) has a more flexible syntax for performing these type of operations:
import numpy_indexed as npi
unique_keys, maxima_per_key = npi.group_by(bbb).max(weight)
Related
I want to eliminate the unefficient for loop from this code
import numpy as np
x = np.zeros((5,5))
for i in range(5):
x[i] = np.random.choice(i+1, 5)
While maintaining the output given
[[0. 0. 0. 0. 0.]
[0. 0. 1. 0. 0.]
[0. 2. 2. 1. 0.]
[1. 2. 3. 1. 0.]
[1. 0. 3. 3. 1.]]
I have tried this
i = np.arange(5)
x[i] = np.random.choice(i+1, 5)
But it outputs
[[0. 1. 1. 3. 3.]
[0. 1. 1. 3. 3.]
[0. 1. 1. 3. 3.]
[0. 1. 1. 3. 3.]
[0. 1. 1. 3. 3.]]
Is it possible to remove the loop? If not, which is the most efficient way to proceed for a big array and a lot of repetitions?
Create a random int array with the highest number per row as the number of columns. Hence, we can use np.random.randint with its high arg set as the no. of cols. Then, perform modulus operation to set across each row a different limit defined by the row number. Thus, we would have a vectorized implementation like so -
def create_rand_limited_per_row(m,n):
s = np.arange(1,m+1)
return np.random.randint(low=0,high=n,size=(m,n))%s[:,None]
Sample run -
In [45]: create_rand_limited_per_row(m=5,n=5)
Out[45]:
array([[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[1, 2, 0, 2, 1],
[0, 0, 1, 3, 0],
[1, 2, 3, 3, 2]])
To leverage multi-core with numexpr module for large data -
import numexpr as ne
def create_rand_limited_per_row_numepxr(m,n):
s = np.arange(1,m+1)[:,None]
a = np.random.randint(0,n,(m,n))
return ne.evaluate('a%s')
Benchmarking
# Original approach
def create_rand_limited_per_row_loopy(m,n):
x = np.empty((m,n),dtype=int)
for i in range(m):
x[i] = np.random.choice(i+1, n)
return x
Timings on 1k x 1k data -
In [71]: %timeit create_rand_limited_per_row_loopy(m=1000,n=1000)
10 loops, best of 3: 20.6 ms per loop
In [72]: %timeit create_rand_limited_per_row(m=1000,n=1000)
100 loops, best of 3: 14.3 ms per loop
In [73]: %timeit create_rand_limited_per_row_numepxr(m=1000,n=1000)
100 loops, best of 3: 6.98 ms per loop
I have the following numpy array:
a = np.array([[1.1,0.8,0.5,0,0],[1,0.85,0.5,0,0],[1,0.8,0.5,1,0]])
with shape = (3,5).
I would like to reshape and resize it to a new array with shape = (3,8), filling the new values in each row with 0. So far I tried the following approach:
b = np.resize(a,(3,8))
But it returns:
[[ 1.1 0.8 0.5 0. 0. 1. 0.85 0.5 ]
[ 0. 0. 1. 0.8 0.5 1. 0. 1.1 ]
[ 0.8 0.5 0. 0. 1. 0.85 0.5 0. ]]
instead of the expected (for me):
[[ 1.1 0.8 0.5 0. 0. 0. 0. 0. ]
[ 1. 0.85 0.5 0. 0. 0. 0. 0. ]
[ 1. 0.8 0.5 1. 0. 0. 0. 0. ]]
Use np.lib.pad -
np.lib.pad(a, ((0,0),(0,3)), 'constant', constant_values=(0))
Sample run -
In [156]: a
Out[156]:
array([[ 1.1 , 0.8 , 0.5 , 0. , 0. ],
[ 1. , 0.85, 0.5 , 0. , 0. ],
[ 1. , 0.8 , 0.5 , 1. , 0. ]])
In [157]: np.lib.pad(a, ((0,0),(0,3)), 'constant', constant_values=(0))
Out[157]:
array([[ 1.1 , 0.8 , 0.5 , 0. , 0. , 0. , 0. , 0. ],
[ 1. , 0.85, 0.5 , 0. , 0. , 0. , 0. , 0. ],
[ 1. , 0.8 , 0.5 , 1. , 0. , 0. , 0. , 0. ]])
Runtime tests -
This section covers runtime tests for the approaches posted thus far for the size listed in the question and scaling it up by 100x. Here are the timing test results -
In [212]: def init_based(a,N):
...: b = np.zeros((a.shape[0], a.shape[1]+N))
...: b[:, :a.shape[1]] = a
...: return b
...:
In [213]: a = np.random.rand(3,5)
In [214]: N = 3
In [215]: %timeit np.lib.pad(a, ((0,0),(0,N)), 'constant', constant_values=(0))
...: %timeit np.hstack([a, np.zeros([a.shape[0], N])])
...: %timeit np.concatenate((a,np.zeros((a.shape[0],N))), axis=1)
...: %timeit init_based(a,N)
...:
10000 loops, best of 3: 32.7 µs per loop
100000 loops, best of 3: 11.2 µs per loop
100000 loops, best of 3: 4.49 µs per loop
100000 loops, best of 3: 5.67 µs per loop
In [216]: a = np.random.rand(300,500)
In [217]: N = 300
In [218]: %timeit np.lib.pad(a, ((0,0),(0,N)), 'constant', constant_values=(0))
...: %timeit np.hstack([a, np.zeros([a.shape[0], N])])
...: %timeit np.concatenate((a,np.zeros((a.shape[0],N))), axis=1)
...: %timeit init_based(a,N)
...:
100 loops, best of 3: 2.99 ms per loop
1000 loops, best of 3: 1.72 ms per loop
1000 loops, best of 3: 1.71 ms per loop
1000 loops, best of 3: 1.72 ms per loop
From the doc of np.resize():
If the new array is larger than the original array, then the new
array is filled with repeated copies of a.
Zeros are not used, but actual values of a.
Instead, you could use np.hstack() and np.zeros() :
np.hstack([a, np.zeros([3, 3])])
Edit: I have not tested the speed, so I suggest you have a look a other solutions too.
Definitely you can use resize().
If the new array is larger than the original array, then the new array is filled with repeated copies of a. Note that this behavior is different from a.resize(new_shape) which fills with zeros instead of repeated copies of a.
b = a.transpose().copy()
b.resize((8,3), refcheck=False)
b = a.transpose()
which outputs:
[[ 1.1 0.8 0.5 0. 0. 0. 0. 0. ]
[ 1. 0.85 0.5 0. 0. 0. 0. 0. ]
[ 1. 0.8 0.5 1. 0. 0. 0. 0. ]]
Limitation:
Filling with 0s can be only applied to the 1st dimension.
Another option (although np.hstack is probably best as in M. Massias' answer).
Initialise an array of zeros:
b = np.zeros((3, 8))
Fill using slice syntax:
b[:3, :5] = a
np.concatenate
np.concatenate((a,np.zeros((3,3))), axis=1)
array([[ 1.1 , 0.8 , 0.5 , 0. , 0. , 0. , 0. , 0. ],
[ 1. , 0.85, 0.5 , 0. , 0. , 0. , 0. , 0. ],
[ 1. , 0.8 , 0.5 , 1. , 0. , 0. , 0. , 0. ]])
This is a VERY simplified example of a larger problem I'm working with. How do I replace multiple rows in a two dimensional numpy array? For example, I have an array...
main_list = array( [[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]])
I have a list of indexes...
indexes = array([3, 6, 2])
I have a list of substitutions that will always be the same length as the list of indexes.
substitutions = array( [[ 2.4 5.2]
[ 10.1 1.3]
[ 5.6 9.5]])
I want...
main_list = array( [[ 0. 0.]
[ 0. 0.]
[ 5.6 9.5]
[ 2.4 5.2]
[ 0. 0.]
[ 0. 0.]
[ 10.1 1.3]
[ 0. 0.]
[ 0. 0.]
[ 0. 0.]])
Right now I'm doing...
for i, ind in enumerate(indexes):
main_list[ind] = substitutions[i]
Keeping in mind that this is a simple example, in the production version of what I'm doing the length of all these lists will be large. Is there a faster way to do these substitutions? Thanks.
main_list[indexes,:] = substitutions
my attempt at timing says this is 3x faster than what you posted
In [51]: %%timeit
for i, ind in enumerate(indexes):
main_list[ind] = substitutions[i]
....:
100000 loops, best of 3: 6.83 us per loop
In [52]: %timeit main_list[indexes,:] = substitutions
100000 loops, best of 3: 2.27 us per loop
I wanted to interleave the rows of two numpy arrays of the same size.
I came up with this solution.
# A and B are same-shaped arrays
A = numpy.ones((4,3))
B = numpy.zeros_like(A)
C = numpy.array(zip(A[::1], B[::1])).reshape(A.shape[0]*2, A.shape[1])
print(C)
Outputs
[[ 1. 1. 1.]
[ 0. 0. 0.]
[ 1. 1. 1.]
[ 0. 0. 0.]
[ 1. 1. 1.]
[ 0. 0. 0.]
[ 1. 1. 1.]
[ 0. 0. 0.]]
Is there a cleaner, faster, better, numpy-only way?
It is maybe a bit clearer to do:
A = np.ones((4,3))
B = np.zeros_like(A)
C = np.empty((A.shape[0]+B.shape[0],A.shape[1]))
C[::2,:] = A
C[1::2,:] = B
and it's probably a bit faster as well, I'm guessing.
I find the following approach using numpy.hstack() quite readable:
import numpy as np
a = np.ones((2,3))
b = np.zeros_like(a)
c = np.hstack([a, b]).reshape(4, 3)
print(c)
Output:
[[ 1. 1. 1.]
[ 0. 0. 0.]
[ 1. 1. 1.]
[ 0. 0. 0.]]
It is easy to generalize this to a list of arrays of the same shape:
arrays = [a, b, c,...]
shape = (len(arrays)*a.shape[0], a.shape[1])
interleaved_array = np.hstack(arrays).reshape(shape)
It seems to be a bit slower than the accepted answer of #JoshAdel on small arrays but equally fast or faster on large arrays:
a = np.random.random((3,100))
b = np.random.random((3,100))
%%timeit
...: C = np.empty((a.shape[0]+b.shape[0],a.shape[1]))
...: C[::2,:] = a
...: C[1::2,:] = b
...:
The slowest run took 9.29 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 3.3 µs per loop
%timeit c = np.hstack([a,b]).reshape(2*a.shape[0], a.shape[1])
The slowest run took 5.06 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 10.1 µs per loop
a = np.random.random((4,1000000))
b = np.random.random((4,1000000))
%%timeit
...: C = np.empty((a.shape[0]+b.shape[0],a.shape[1]))
...: C[::2,:] = a
...: C[1::2,:] = b
...:
10 loops, best of 3: 23.2 ms per loop
%timeit c = np.hstack([a,b]).reshape(2*a.shape[0], a.shape[1])
10 loops, best of 3: 21.3 ms per loop
You can stack, transpose, and reshape:
numpy.dstack((A, B)).transpose(0, 2, 1).reshape(A.shape[0]*2, A.shape[1])
Given an index and a size, is there a more efficient way to produce the standard basis vector:
import numpy as np
np.array([1.0 if i == index else 0.0 for i in range(size)])
In [2]: import numpy as np
In [9]: size = 5
In [10]: index = 2
In [11]: np.eye(1,size,index)
Out[11]: array([[ 0., 0., 1., 0., 0.]])
Hm, unfortunately, using np.eye for this is rather slow:
In [12]: %timeit np.eye(1,size,index)
100000 loops, best of 3: 7.68 us per loop
In [13]: %timeit a = np.zeros(size); a[index] = 1.0
1000000 loops, best of 3: 1.53 us per loop
Wrapping np.zeros(size); a[index] = 1.0 in a function makes only a modest difference, and is still much faster than np.eye:
In [24]: def f(size, index):
....: arr = np.zeros(size)
....: arr[index] = 1.0
....: return arr
....:
In [27]: %timeit f(size, index)
1000000 loops, best of 3: 1.79 us per loop
x = np.zeros(size)
x[index] = 1.0
at least i think thats it...
>>> t = timeit.Timer('np.array([1.0 if i == index else 0.0 for i in range(size)]
)','import numpy as np;size=10000;index=5123')
>>> t.timeit(10)
0.039461429317952934 #original method
>>> t = timeit.Timer('x=np.zeros(size);x[index]=1.0','import numpy as np;size=10000;index=5123')
>>> t.timeit(10)
9.4077963240124518e-05 #zeros method
>>> t = timeit.Timer('x=np.eye(1.0,size,index)','import numpy as np;size=10000;index=5123')
>>> t.timeit(10)
0.0001398340635319073 #eye method
looks like np.zeros is fastest...
I'm not sure if this is faster, but it's definitely more clear to me.
a = np.zeros(size)
a[index] = 1.0
Often, you need not one but all basis vectors. If this is the case, consider np.eye:
basis = np.eye(3)
for vector in basis:
...
Not exactly the same, but closely related: This even works to get a set of basis matrices with a bit of tricks:
>>> d, e = 2, 3 # want 2x3 matrices
>>> basis = np.eye(d*e,d*e).reshape((d*e,d,e))
>>> print(basis)
[[[ 1. 0. 0.]
[ 0. 0. 0.]]
[[ 0. 1. 0.]
[ 0. 0. 0.]]
[[ 0. 0. 1.]
[ 0. 0. 0.]]
[[ 0. 0. 0.]
[ 1. 0. 0.]]
[[ 0. 0. 0.]
[ 0. 1. 0.]]
[[ 0. 0. 0.]
[ 0. 0. 1.]]]
and so on.
Another way to implement this is :
>>> def f(size, index):
... return (np.arange(size) == index).astype(float)
...
Which gives a slightly slower execution time :
>>> timeit.timeit('f(size, index)', 'from __main__ import f, size, index', number=1000000)
2.2554846050043125
It may not be the fastest, but the method scipy.signal.unit_impulse generalizes the above concept to numpy arrays of any shape.