I just came across a showstopper for a part of my code and I am not sure what I am doing wrong...
I simply have a large data cube and want to change the maximum values along the z axis to some other number:
import numpy as np
from time import time
x, y, z = 100, 100, 10
a = np.arange(x*y*z).reshape((z, y, x))
t = time()
a[np.argmax(a, axis=0)] = 1
print(time() - t)
This takes about 0.02s which is a bit slow for such a small array, but ok. My problem is that I need to do this with arrays as large as (32, 4096, 4096) and I have not had the patience to let this finish with the above code...it's just too inefficient, but it should actually be very fast! Am I doing something wrong with setting the array elements?
You are basically indexing your numpy array with a numpy array containing numbers. I think that is the reason why it is so slow (and I'm not sure if it really does what you want it to do).
If you create a boolean numpy array and use this as slice it's orders of magnitudes faster.
For example:
pos_max = np.expand_dims(np.argmax(a, axis=0), axis=0)
pos_max_indices = np.arange(a.shape[0]).reshape(10,1,1) == pos_max
a[pos_max_indices] = 1
is 20 times faster than the original and does the same.
I don't think it is the indexing with numbers that's slowing it down. Usually indexing a single dimension with a boolean vector is slower than indexing with the corresponding np.where.
Something else is going on here. Look at these shapes:
In [14]: a.shape
Out[14]: (10, 100, 100)
In [15]: np.argmax(a,axis=0).shape
Out[15]: (100, 100)
In [16]: a[np.argmax(a,axis=0)].shape
Out[16]: (100, 100, 100, 100)
The indexed a is much larger than the original, 1000x.
#MSeifert's solution is faster, but I can't help feeling it is more complex than needed.
In [35]: %%timeit
....: a=np.arange(x*y*z).reshape((z,y,x))
....: pos_max = np.expand_dims(np.argmax(a, axis=0), axis=0)
....: pos_max_indices = np.arange(a.shape[0]).reshape(10,1,1) == pos_max
....: a[pos_max_indices]=1
....:
1000 loops, best of 3: 1.28 ms per loop
I'm still working on an improvement.
The sample array isn't a good one - it's too big to display, and all the max values on the last z plane:
In [46]: x,y,z=4,2,3
In [47]: a=np.arange(x*y*z).reshape((z,y,x))
In [48]: a
Out[48]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[16, 17, 18, 19],
[20, 21, 22, 23]]])
In [49]: a[np.argmax(a,axis=0)]=1
In [50]: a
Out[50]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[ 1, 1, 1, 1],
[ 1, 1, 1, 1]]])
I could access those same argmax values with:
In [51]: a[-1,...]
Out[51]:
array([[1, 1, 1, 1],
[1, 1, 1, 1]])
Let's try a random array, where the argmax can be in any plane:
In [57]: a=np.random.randint(2,10,(z,y,x))
In [58]: a
Out[58]:
array([[[9, 7, 6, 5],
[6, 3, 5, 2]],
[[5, 6, 2, 3],
[7, 9, 6, 9]],
[[7, 7, 8, 9],
[2, 4, 9, 7]]])
In [59]: a[np.argmax(a,axis=0)]=0
In [60]: a
Out[60]:
array([[[0, 0, 0, 0],
[0, 0, 0, 0]],
[[0, 0, 0, 0],
[0, 0, 0, 0]],
[[0, 0, 0, 0],
[0, 0, 0, 0]]])
Oops - I turned everything to 0. Is that what you want?
Let's try the pos_max method:
In [61]: a=np.random.randint(0,10,(z,y,x))
In [62]: a
Out[62]:
array([[[9, 3, 9, 0],
[6, 6, 2, 4]],
[[9, 9, 4, 9],
[5, 9, 7, 9]],
[[1, 8, 1, 7],
[1, 0, 2, 3]]])
In [63]: pos_max = np.expand_dims(np.argmax(a, axis=0), axis=0)
In [64]: pos_max
Out[64]:
array([[[0, 1, 0, 1],
[0, 1, 1, 1]]], dtype=int32)
In [66]: pos_max_indices = np.arange(a.shape[0]).reshape(z,1,1) == pos_max
In [67]: pos_max_indices
Out[67]:
array([[[ True, False, True, False],
[ True, False, False, False]],
[[False, True, False, True],
[False, True, True, True]],
[[False, False, False, False],
[False, False, False, False]]], dtype=bool)
In [68]: a[pos_max_indices]=0
In [69]: a
Out[69]:
array([[[0, 3, 0, 0],
[0, 6, 2, 4]],
[[9, 0, 4, 0],
[5, 0, 0, 0]],
[[1, 8, 1, 7],
[1, 0, 2, 3]]])
That looks more reasonable. There still is a 9 in the 2nd plane, but that's because there was also a 9 in the 1st.
This still needs to be cleaned up, but here's a non-boolean mask solution:
In [98]: a=np.random.randint(0,10,(z,y,x))
In [99]: a1=a.reshape(z,-1) # it's easier to work with a 2d view
In [100]: ind=np.argmax(a1,axis=0)
In [101]: ind
Out[101]: array([2, 2, 1, 0, 2, 0, 1, 2], dtype=int32)
In [102]: a1[ind,np.arange(a1.shape[1])] # the largest values
Out[102]: array([9, 8, 7, 4, 9, 7, 9, 6])
In [104]: a1
Out[104]:
array([[3, 1, 5, 4, 2, 7, 4, 5],
[4, 4, 7, 1, 3, 7, 9, 4],
[9, 8, 3, 3, 9, 1, 2, 6]])
In [105]: a1[ind,np.arange(a1.shape[1])]=0
In [106]: a
Out[106]:
array([[[3, 1, 5, 0],
[2, 0, 4, 5]],
[[4, 4, 0, 1],
[3, 7, 0, 4]],
[[0, 0, 3, 3],
[0, 1, 2, 0]]])
Working with a1 the 2d view is easier; the exact shape of the x,y dimensions is not important to this problem. We are changing individual values, not columns or planes. Still I'd like to do get it working without `a1.
Here are two functions that replace the maximum value (in the 1st plane). I use copy since it makes repeated time testing easier.
def setmax0(a, value=-1):
# #MSeifert's
a = a.copy()
z = a.shape[0]
# a=np.arange(x*y*z).reshape((z,y,x))
pos_max = np.expand_dims(np.argmax(a, axis=0), axis=0)
pos_max_indices = np.arange(z).reshape(z,1,1) == pos_max
a[pos_max_indices]=value
return a
def setmax1(a, value=-2):
a = a.copy()
z = a.shape[0]
a1 = a.reshape(z, -1)
ind = np.argmax(a1, axis=0)
a1[ind, np.arange(a1.shape[1])] = value
return a
They produce the same result in a test like:
ab = np.random.randint(0,100,(20,1000,1000))
test = np.allclose(setmax1(ab,-1),setmax0(ab,-1))
Timings (using ipython timeit) are basically the same.
They do assign values in different orders, so setmax0(ab,-np.arange(...)) will be different.
Related
I am trying to index an np.array with another array so that I can have zeros everywhere after a certain index but it gives me the error
TypeError: only integer scalar arrays can be converted to a scalar
index
Basically what I would like my code to do is that if I have:
a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
d = np.array([2, 1, 3])
that I could do something like
a[d:] = 0
to give the output
a = [[ 1 2 3]
[ 4 0 6]
[ 0 0 9]
[ 0 0 0]]
It can be done with array indexing but it doesn't feel natural.
import numpy as np
a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
d = np.array([2, 1, 3])
col_ix = [ 0, 0, 1, 1, 1, 2 ] # column ix for each item to change
row_ix = [ 2, 3, 1, 2, 3, 3 ] # row index for each item to change
a[ row_ix, col_ix ] = 0
a
# array([[1, 2, 3],
# [4, 0, 6],
# [0, 0, 9],
# [0, 0, 0]])
With a for loop
a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
for ix_col, ix_row in enumerate( d ): # iterate across the columns
a[ ix_row:, ix_col ] = 0
a
# array([[1, 2, 3],
# [4, 0, 6],
# [0, 0, 9],
# [0, 0, 0]])
A widely used approach for this kind of problem is to construct a boolean mask, comparing the index array with the appropriate arange:
In [619]: mask = np.arange(4)[:,None]>=d
In [620]: mask
Out[620]:
array([[False, False, False],
[False, True, False],
[ True, True, False],
[ True, True, True]])
In [621]: a[mask]
Out[621]: array([ 5, 7, 8, 10, 11, 12])
In [622]: a[mask] = 0
In [623]: a
Out[623]:
array([[1, 2, 3],
[4, 0, 6],
[0, 0, 9],
[0, 0, 0]])
That's not necessarily faster than a row (or in this case column) iteration. Since slicing is basic indexing, it may be faster, even if done several times.
In [624]: for i,v in enumerate(d):
...: print(a[v:,i])
...:
[0 0]
[0 0 0]
[0]
Generally if a result involves multiple arrays or lists with different lengths, there isn't a "neat" multidimensional solution. Either iterate over those lists, or step back and "think outside the box".
how can I execute the following piece of code without using loop structure, instead using the numpy einsum function? I want the product matrix to be a 2D matrix and not 3D. Simply doing
"D = np.einsum('ijk,ijk->jk',A,B)
D += np.einsum('ijk,ijk->jk',B,C) " gives different result. Should I introduce any intermediate temporary array or something to use the einsum function?
import numpy as np
A = np.array( [[[1, 2, 3, 0],
[ 4, 2, 1, 1]],
[[2, 0, 0, 3],
[1, 0, 2, 4]]] )
B = np.array( [[[0, 2, 3, 1],
[0, 2, 5, 0]],
[[0, 1, 2, 2],
[3, 3, 2, 1]]] )
C = np.array( [[[0, 2, 2, 1],
[0, 2, 1, 0]],
[[0, 0, 2, 0],
[3, 1, 2, 1]]] )
X = np.zeros([2,4])
for i in range(2):
for j in range(2):
for k in range(4):
X[j,k] = A[i,j,k]*B[i,j,k]
X[j,k] += B[i,j,k]*C[i,j,k]
D = np.einsum('ijk,ijk->jk',A,B)
D += np.einsum('ijk,ijk->jk',B,C)
Following up on my comment, replace the i loop with just 1 step
In [64]: X = np.zeros([2,4])
...: i=1
...: for j in range(2):
...: for k in range(4):
...:
...: X[j,k] = A[i,j,k]*B[i,j,k]
...: X[j,k] += B[i,j,k]*C[i,j,k]
In [65]: X
Out[65]:
array([[ 0., 0., 4., 6.],
[12., 3., 8., 5.]])
This is the value as produced by your loop. You've thrown away the i=0 values.
But you don't need the loops to do:
In [68]: A[1,:,:]*B[1,:,:]+B[1,:,:]*C[1,:,:]
Out[68]:
array([[ 0, 0, 4, 6],
[12, 3, 8, 5]])
In [69]: A*B+B*C
Out[69]:
array([[[ 0, 8, 15, 1],
[ 0, 8, 10, 0]],
[[ 0, 0, 4, 6],
[12, 3, 8, 5]]])
The same thing with einsum:
In [71]: np.einsum('ijk,ijk->ijk',A,B)+np.einsum('ijk,ijk->ijk',B,C)
Out[71]:
array([[[ 0, 8, 15, 1],
[ 0, 8, 10, 0]],
[[ 0, 0, 4, 6],
[12, 3, 8, 5]]])
and if you want to sum across i:
In [72]: np.einsum('ijk,ijk->jk',A,B)+np.einsum('ijk,ijk->jk',B,C)
Out[72]:
array([[ 0, 8, 19, 7],
[12, 11, 18, 5]])
In [73]: (A*B+B*C).sum(axis=0)
Out[73]:
array([[ 0, 8, 19, 7],
[12, 11, 18, 5]])
I was checking indexing in numpy array but got confused in below case, please tell me why I am getting different output when I am converting a list to an array. what am I doing wrong?
In [124]: a = np.arange(12).reshape(3, 4)
Out[125]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
In [126]: j = [[0, 1], [1, 2]]
In [127]: a[j]
Out[127]: array([1, 6])
In [128]: a[np.array(j)]
Out[128]:
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7]],
[[ 4, 5, 6, 7],
[ 8, 9, 10, 11]]])
There's a bit of undocumented backward compatibility handling where if the selection object is a single short non-ndarray sequence containing embedded sequences, the sequence is handled as if it were a tuple. That means this:
a[j]
is treated as
a[[0, 1], [1, 2]]
instead of as the docs would have you expect.
I have two sets of array data and result. result contains the same elements in data but with an extra column and in unsorted order. I want to rearrange the result array so that it is in the same order as the rows in data , while bringing the associated value into the last column with the rest of the row when doing the sorting.
data = np.array([[0,1,0,0],[1,0,0,0],[0,1,1,0],[0,1,0,1]])
result = np.array([[0,1,1,0,1],[1,0,0,0,0],[0,1,0,0,1],[0,1,0,1,0]])
# this is what the final sorted array should look like:
'''
array([[0, 1, 0, 0, 1],
[1, 0, 0, 0, 0],
[0, 1, 1, 0, 1],
[0, 1, 0, 1, 0]])
'''
I've tried doing argsort in order to reverse data into the sorted order then applying that to result but argsort seems to sort the order of the array based on each element, whereas I want the sort to treat each row of the data[:,4] as a whole.
ind = np.argsort(data)
indind =np.argsort(ind)
ind
array([[0, 2, 3, 1],
[1, 2, 3, 0],
[0, 3, 1, 2],
[0, 2, 1, 3]])
What is a good way to do this kind of sorting by rows?
The numpy_indexed package (disclaimer: I am its author) can be used to efficiently and elegantly solve these kind of problems:
import numpy_indexed as npi
result[npi.indices(result[:, :-1], data)]
npi.indices is essentially a vectorized equivalent of list.index; so for each element (row) in data, we get where that same row is located in result, minus the last column.
Note that this solution works for any number of columns, and is fully vectorized (ie, no python loops anywhere).
Approach #1
Here's an approach considering each row as an indexing tuple and then finding the matching indices between data and result corresponding to those linear index equivalents. These indices would represent the new order of rows, which when indexed into result would give us the desired output. The implementation would look like this -
# Slice out from result everything except the last column
r = result[:,:-1]
# Get linear indices equivalent of each row from r and data
ID1 = np.ravel_multi_index(r.T,r.max(0)+1)
ID2 = np.ravel_multi_index(data.T,r.max(0)+1)
# Search for ID2 in ID1 and use those indices index into result
out = result[np.where(ID1[:,None] == ID2)[1]]
Approach #2
If all the rows from data are guaranteed to be in result, you can use another approach based on just argsort, like so -
# Slice out from result everything except the last column
r = result[:,:-1]
# Get linear indices equivalent of each row from r and data
ID1 = np.ravel_multi_index(r.T,r.max(0)+1)
ID2 = np.ravel_multi_index(data.T,r.max(0)+1)
sortidx_ID1 = ID1.argsort()
sortidx_ID2 = ID2.argsort()
out = result[sortidx_ID1[sortidx_ID2]]
Sample run for a bit more generic case -
In [37]: data
Out[37]:
array([[ 3, 2, 1, 5],
[ 4, 9, 2, 4],
[ 7, 3, 9, 11],
[ 5, 9, 4, 4]])
In [38]: result
Out[38]:
array([[ 7, 3, 9, 11, 55],
[ 4, 9, 2, 4, 8],
[ 3, 2, 1, 5, 7],
[ 5, 9, 4, 4, 88]])
In [39]: r = result[:,:-1]
...: ID1 = np.ravel_multi_index(r.T,r.max(0)+1)
...: ID2 = np.ravel_multi_index(data.T,r.max(0)+1)
...:
In [40]: result[np.where(ID1[:,None] == ID2)[1]] # Approach 1
Out[40]:
array([[ 3, 2, 1, 5, 7],
[ 4, 9, 2, 4, 8],
[ 7, 3, 9, 11, 55],
[ 5, 9, 4, 4, 88]])
In [41]: sortidx_ID1 = ID1.argsort() # Approach 2
...: sortidx_ID2 = ID2.argsort()
...:
In [42]: result[sortidx_ID1[sortidx_ID2]]
Out[42]:
array([[ 3, 2, 1, 5, 7],
[ 4, 9, 2, 4, 8],
[ 7, 3, 9, 11, 55],
[ 5, 9, 4, 4, 88]])
Just to try to clarify what you are doing. With an index list [2,1,0,3] I can reorder the rows of result thus:
In [37]: result[[2,1,0,3],:]
Out[37]:
array([[0, 1, 0, 0, 1],
[1, 0, 0, 0, 0],
[0, 1, 1, 0, 1],
[0, 1, 0, 1, 0]])
In [38]: result[[2,1,0,3],:4]==data
Out[38]:
array([[ True, True, True, True],
[ True, True, True, True],
[ True, True, True, True],
[ True, True, True, True]], dtype=bool)
I don't see how argsort or sort is going to help come up with this indexing order.
With np.lexsort I can order the rows of both arrays the same:
In [54]: data[np.lexsort(data.T),:]
Out[54]:
array([[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 1, 1, 0],
[0, 1, 0, 1]])
In [55]: result[np.lexsort(result[:,:-1].T),:]
Out[55]:
array([[1, 0, 0, 0, 0],
[0, 1, 0, 0, 1],
[0, 1, 1, 0, 1],
[0, 1, 0, 1, 0]])
I found by trial and error that I needed to use the transpose. We need to check the docs of lexsort to understand why.
A little more trial and error produces:
In [66]: i=np.lexsort(data.T)
In [67]: j=np.lexsort(result[:,:-1].T)
In [68]: j[i]
Out[68]: array([2, 1, 0, 3], dtype=int64)
In [69]: result[j[i],:]
Out[69]:
array([[0, 1, 0, 0, 1],
[1, 0, 0, 0, 0],
[0, 1, 1, 0, 1],
[0, 1, 0, 1, 0]])
This is a tentative solution. It needs to be tested on other samples. And needs to be explained.
I have an array H of dimension MxN, and an array A of dimension M . I want to scale H rows with array A. I do it this way, taking advantage of element-wise behaviour of Numpy
H = numpy.swapaxes(H, 0, 1)
H /= A
H = numpy.swapaxes(H, 0, 1)
It works, but the two swapaxes operations are not very elegant, and I feel there is a more elegant and consise way to achieve the result, without creating temporaries. Would you tell me how ?
I think you can simply use H/A[:,None]:
In [71]: (H.swapaxes(0, 1) / A).swapaxes(0, 1)
Out[71]:
array([[ 8.91065496e-01, -1.30548362e-01, 1.70357901e+00],
[ 5.06027691e-02, 3.59913305e-01, -4.27484490e-03],
[ 4.72868136e-01, 2.04351398e+00, 2.67527572e+00],
[ 7.87239835e+00, -2.13484271e+02, -2.44764975e+02]])
In [72]: H/A[:,None]
Out[72]:
array([[ 8.91065496e-01, -1.30548362e-01, 1.70357901e+00],
[ 5.06027691e-02, 3.59913305e-01, -4.27484490e-03],
[ 4.72868136e-01, 2.04351398e+00, 2.67527572e+00],
[ 7.87239835e+00, -2.13484271e+02, -2.44764975e+02]])
because None (or newaxis) extends A in dimension (example link):
In [73]: A
Out[73]: array([ 1.1845468 , 1.30376536, -0.44912446, 0.04675434])
In [74]: A[:,None]
Out[74]:
array([[ 1.1845468 ],
[ 1.30376536],
[-0.44912446],
[ 0.04675434]])
You just need to reshape A so that it will broad cast properly:
A = A.reshape((-1, 1))
so:
In [21]: M
Out[21]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17],
[18, 19, 20]])
In [22]: A
Out[22]: array([1, 2, 3, 4, 5, 6, 7])
In [23]: M / A.reshape((-1, 1))
Out[23]:
array([[0, 1, 2],
[1, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2],
[2, 2, 2]])