Related
I have two numpy arrays which look like this:
x = [v1, v2, v3, ..., vm]
y = [w1, w2, w3, ..., wn]
where vi, wj are numpy arrays of length 3.
I want to perform a pairwise summation of v's and w's and get a final array
z = [v1+w1, v1+w2,...,v1+wn,v2+w1, ..., vi+wj, ..., vm+wn]
A simple way of obtaining z is as follows:
z = np.zeros ((m*n, 3))
for i in range(m):
for j in range(n):
z[n*i+j] = x[i] + y[j]
This computation is not feasible is m, n are very large.
I know scipy.spatial has methods to enumerate pairwise distances using distance_matrix in a vectorized fashion.
I want to ask if there is a vectorized version of performing such pairwise additions for numpy arrays?
You can take advantage of broadcasting, creating a 2D array, then you can easily get z[i,j] = x[i] + y[j]
x = np.reshape(x, (-1, 1)) # shape (N, 1)
y = np.reshape(y, (-1, 1)) # shape (N, 1)
z = x + y.T # shape (N, N)
If you want to have z as a 1D array you can do z.reshape(-1).
If x is mx3 matrix, y is a nx3
x.shape # (m,3)
y.shape # (n,3)
x1 = x.reshape(m,1,3)
y1 = y.reshape(1,n,3)
z = x1 + y1 # shape (m,n,3)
z1 = z.reshape(-1,3) # (m*n, 3)
equivalently
z = x[:,None]+y
test:
In [263]: x=np.arange(12).reshape(4,3); y=np.arange(6).reshape(2,3)
In [264]: z = x[:,None]+y
In [265]: z.shape
Out[265]: (4, 2, 3)
In [266]: z
Out[266]:
array([[[ 0, 2, 4],
[ 3, 5, 7]],
[[ 3, 5, 7],
[ 6, 8, 10]],
[[ 6, 8, 10],
[ 9, 11, 13]],
[[ 9, 11, 13],
[12, 14, 16]]])
import numpy
A = numpy.array([
[0,1,1],
[2,2,0],
[3,0,3]
])
B = numpy.array([
[1,1,1],
[2,2,2],
[3,2,9],
[4,4,4],
[5,9,5]
])
Dimension of A: N * N(3*3)
Dimension of B: K * N(5*3)
Expected result is:
C = [ A * B[0], A * B[1], A * B[2], A * B[3], A * B[4]] (Dimension of C is also 5*3)
I am new to numpy and not sure how to perform this operation without using for loops.
Thanks!
By the math you provide, I think you are evaluating A times B transpose. If you want the resultant matrix to have the size 5*3, you can transpose it (equivalent to numpy.matmul(B.transpose(),A)).
import numpy
A = numpy.array([
[0,1,1],
[2,2,0],
[3,0,3]
])
B = numpy.array([
[1,1,1],
[2,2,2],
[3,2,9],
[4,4,4],
[5,9,5]
])
print(numpy.matmul(A,B.transpose()))
output :array([[ 2, 4, 11, 8, 14],
[ 4, 8, 10, 16, 28],
[ 6, 12, 36, 24, 30]])
for i in range(5):
print (numpy.matmul(A,B[i]))
Output:
[2 4 6]
[ 4 8 12]
[11 10 36]
[ 8 16 24]
[14 28 30]
You can move forward like this:
import numpy as np
matrix_a = np.array([
[0, 1, 1],
[2, 2, 0],
[3, 0, 3]
])
matrix_b = np.array([
[1, 1, 1],
[2, 2, 2],
[3, 2, 9],
[4, 4, 4],
[5, 9, 5]
])
Remember:
For matrix multiplication , Order of first Column of matrix-A == Order of first row of matrix-B - Such as: B -> (3, 3) == (3, 5), to get order of column and row of matrices, you can use:
rows_of_second_matrix = matrix_b.shape[0]
columns_of_first_matrix = matrix_a.shape[1]
Here, you can check whether Order of first Column of matrix-A == Order of first row of matrix-B or not. If order is not same then go for transpose of matrix-B, else simply multiply.
if columns_of_first_matrix != rows_of_second_matrix:
transpose_matrix_b = np.transpose(matrix_b)
output_1 = np.dot(matrix_a, transpose_matrix_b)
print('Shape of dot product:', output_1.shape)
print('Dot product:\n {}\n'.format(output_1))
output_2 = np.matmul(matrix_a, transpose_matrix_b)
print('Shape of matmul product:', output_2.shape)
print('Matmul product:\n {}\n'.format(output_2))
# In order to obtain -> Output_Matrix of shape (5, 3), Again take transpose
output_matrix = np.transpose(output_1)
print("Shape of required matrix: ", output_matrix.shape)
else:
output_1 = np.dot(matrix_a, matrix_b)
print('Shape of dot product:', output_1.shape)
print('Dot product:\n {}\n'.format(output_1))
output_2 = np.matmul(matrix_a, matrix_b)
print('Shape of matmul product:', output_2.shape)
print('Matmul product:\n {}\n'.format(output_2))
output_matrix = output_2
print("Shape of required matrix: ", output_matrix.shape)
Output:
- Shape of dot product: (3, 5)
Dot product:
[[ 2 4 11 8 14]
[ 4 8 10 16 28]
[ 6 12 36 24 30]]
- Shape of matmul product: (3, 5)
Matmul product:
[[ 2 4 11 8 14]
[ 4 8 10 16 28]
[ 6 12 36 24 30]]
- Shape of required matrix: (5, 3)
I have an Nx3 numpy array:
A = [[01,02,03]
[11,12,13]
[21,22,23]]
I need an array where the second and third columns are swapped if the sum of the second and third numbers is greater then 20:
[[01,02,03]
[11,13,12]
[21,23,22]]
Is it possible to achieve this without a loop?
UPDATE:
So, the story behind this is that I want to swap colors in a RGB image, namely green and blue, but not yellow - this is my condition. Empirically I found out it is abs(green - blue) > 15 && (blue > green)
swapped = np.array(img).reshape(img.shape[0] * img.shape[1], img.shape[2])
idx = ((np.abs(swapped[:,1] - swapped[:,2]) < 15) & (swapped[:, 2] < swapped[:, 1]))
swapped[idx, 1], swapped[idx, 2] = swapped[idx, 2], swapped[idx, 1]
plt.imshow(swapped.reshape(img.shape[0], img.shape[1], img.shape[2]))
this actually works, but partially. The first column will be swapped, but the second one will be overwritten.
# tested in pyton3
a = np.array([[1,2,3],[11,12,13],[21,22,23]])
a[:,1], a[:,2] = a[:,2], a[:,1]
array([[ 1, 3, 3],
[11, 13, 13],
[21, 23, 23]])
Here's one way with masking -
# Get 1D mask of length same as the column length of array and with True
# values at places where the combined sum is > 20
m = A[:,1] + A[:,2] > 20
# Get the masked elements off the second column
tmp = A[m,2]
# Assign into the masked places in the third col from the
# corresponding masked places in second col.
# Note that this won't change `tmp` because `tmp` isn't a view into
# the third col, but holds a separate memory space
A[m,2] = A[m,1]
# Finally assign into the second col from tmp
A[m,1] = tmp
Sample run -
In [538]: A
Out[538]:
array([[ 1, 2, 3],
[11, 12, 13],
[21, 22, 23]])
In [539]: m = A[:,1] + A[:,2] > 20
...: tmp = A[m,2]
...: A[m,2] = A[m,1]
...: A[m,1] = tmp
In [540]: A
Out[540]:
array([[ 1, 2, 3],
[11, 13, 12],
[21, 23, 22]])
How about using np.where along with "fancy" indexing, and np.flip to swap the elements.
In [145]: A
Out[145]:
array([[ 1, 2, 3],
[11, 12, 13],
[21, 22, 23]])
# extract matching sub-array
In [146]: matches = A[np.where(np.sum(A[:, 1:], axis=1) > 20)]
In [147]: matches
Out[147]:
array([[11, 12, 13],
[21, 22, 23]])
# swap elements and update the original array using "boolean" indexing
In [148]: A[np.where(np.sum(A[:, 1:], axis=1) > 20)] = np.hstack((matches[:, :1], np.flip(matches[:, 1:], axis=1)))
In [149]: A
Out[149]:
array([[ 1, 2, 3],
[11, 13, 12],
[21, 23, 22]])
One more approach based on #Divakar's suggestion would be to:
First get the indices which are nonzero for the condition specified (here
sum of the elements in second and third column > 20)
In [70]: idx = np.flatnonzero(np.sum(A[:, 1:3], axis=1) > 20)
Then create an open mesh using np.ix_
In [71]: gidx = np.ix_(idx,[1,2])
# finally update the original array `A`
In [72]: A[gidx] = A[gidx][:,::-1]
x = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15])
I want to grab first 2 rows of array x from every block of 5, result should be:
x[fancy_indexing] = [1,2, 6,7, 11,12]
It's easy enough to build up an index like that using a for loop.
Is there a one-liner slicing trick that will pull it off? Points for simplicity here.
Approach #1 Here's a vectorized one-liner using boolean-indexing -
x[np.mod(np.arange(x.size),M)<N]
Approach #2 If you are going for performance, here's another vectorized approach using NumPy strides -
n = x.strides[0]
shp = (x.size//M,N)
out = np.lib.stride_tricks.as_strided(x, shape=shp, strides=(M*n,n)).ravel()
Sample run -
In [61]: # Inputs
...: x = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15])
...: N = 2
...: M = 5
...:
In [62]: # Approach 1
...: x[np.mod(np.arange(x.size),M)<N]
Out[62]: array([ 1, 2, 6, 7, 11, 12])
In [63]: # Approach 2
...: n = x.strides[0]
...: shp = (x.size//M,N)
...: out=np.lib.stride_tricks.as_strided(x,shape=shp,strides=(M*n,n)).ravel()
...:
In [64]: out
Out[64]: array([ 1, 2, 6, 7, 11, 12])
I first thought you need this to work for 2d arrays due to your phrasing of "first N rows of every block of M rows", so I'll leave my solution as this.
You could work some magic by reshaping your array into 3d:
M = 5 # size of blocks
N = 2 # number of columns to cut
x = np.arange(3*4*M).reshape(4,-1) # (4,3*N)-shaped dummy input
x = x.reshape(x.shape[0],-1,M)[:,:,:N+1].reshape(x.shape[0],-1) # (4,3*N)-shaped output
This will extract every column according to your preference. In order to use it for your 1d case you'd need to make your 1d array into a 2d one using x = x[None,:].
Reshape the array to multiple rows of five columns then take (slice) the first two columns of each row.
>>> x
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15])
>>> x.reshape(x.shape[0] / 5, 5)[:,:2]
array([[ 1, 2],
[ 6, 7],
[11, 12]])
Or
>>> x.reshape(x.shape[0] / 5, 5)[:,:2].flatten()
array([ 1, 2, 6, 7, 11, 12])
>>>
It only works with 1-d arrays that have a length that is a multiple of five.
import numpy as np
x = np.array(range(1, 16))
y = np.vstack([x[0::5], x[1::5]]).T.ravel()
y
// => array([ 1, 2, 6, 7, 11, 12])
Taking the first N rows of every block of M rows in the array [1, 2, ..., K]:
import numpy as np
K = 30
M = 5
N = 2
x = np.array(range(1, K+1))
y = np.vstack([x[i::M] for i in range(N)]).T.ravel()
y
// => array([ 1, 2, 6, 7, 11, 12, 16, 17, 21, 22, 26, 27])
Notice that .T and .ravel() are fast operations: they don't copy any data, but just manipulate the dimensions and strides of the array.
If you insist on getting your slice using fancy indexing:
import numpy as np
K = 30
M = 5
N = 2
x = np.array(range(1, K+1))
fancy_indexing = [i*M+n for i in range(len(x)//M) for n in range(N)]
x[fancy_indexing]
// => array([ 1, 2, 6, 7, 11, 12, 16, 17, 21, 22, 26, 27])
I have three numpy arrays:
X: a 3073 x 49000 matrix
W: a 10 x 3073 matrix
y: a 49000 x 1 vector
y contains values between 0 and 9, each value represents a row in W.
I would like to add the first column of X to the row in W given by the first element in y. I.e. if the first element in y is 3, add the first column of X to the fourth row of W. And then add the second column of X to the row in W given by the second element in y and so on, until all columns of X has been aded to the row in W specified by y, which means a total of 49000 added rows.
W[y] += X.T does not work for me, because this will not add more than one vector to a row in W.
Please note: I'm only looking for vectorized solutions. I.e. no for-loops.
EDIT: To clarify I'll add an example with small matrix sizes adapted from Salvador Dali's example below.
In [1]: import numpy as np
In [2]: a, b, c = 3, 4, 5
In [3]: np.random.seed(0)
In [4]: X = np.random.randint(10, size=(b,c))
In [5]: W = np.random.randint(10, size=(a,b))
In [6]: y = np.random.randint(a, size=(c,1))
In [7]: X
Out[7]:
array([[5, 0, 3, 3, 7],
[9, 3, 5, 2, 4],
[7, 6, 8, 8, 1],
[6, 7, 7, 8, 1]])
In [8]: W
Out[8]:
array([[5, 9, 8, 9],
[4, 3, 0, 3],
[5, 0, 2, 3]])
In [9]: y
Out[9]:
array([[0],
[1],
[1],
[2],
[0]])
In [10]: W[y.ravel()] + X.T
Out[10]:
array([[10, 18, 15, 15],
[ 4, 6, 6, 10],
[ 7, 8, 8, 10],
[ 8, 2, 10, 11],
[12, 13, 9, 10]])
In [11]: W[y.ravel()] = W[y.ravel()] + X.T
In [12]: W
Out[12]:
array([[12, 13, 9, 10],
[ 7, 8, 8, 10],
[ 8, 2, 10, 11]])
The problem is to get BOTH column 0 and column 4 in X added to row 0 in W, as well as both column 1 and 2 in X added to row 1 in W.
The desired outcome is thus:
W = [[17, 22, 16, 16],
[ 7, 11, 14, 17],
[ 8, 2, 10, 11]]
First the straight forward loop solution as reference:
In [65]: for i,j in enumerate(y):
W[j]+=X[:,i]
....:
In [66]: W
Out[66]:
array([[17, 22, 16, 16],
[ 7, 11, 14, 17],
[ 8, 2, 10, 11]])
An add.at solution:
In [67]: W=W1.copy()
In [68]: np.add.at(W,(y.ravel()),X.T)
In [69]: W
Out[69]:
array([[17, 22, 16, 16],
[ 7, 11, 14, 17],
[ 8, 2, 10, 11]])
add.at does an unbuffered calculation, getting around the buffering that prevents W[y.ravel()] += X.T from working. It is still iterative, but the loop has been moved to compiled code. It isn't true vectorization because the order of application matters. The addition for one row of X.T depends on the results from the previous rows.
https://stackoverflow.com/a/20811014/901925 is the answer I gave a couple of years ago to a similar question (for 1d arrays).
But when dealing with your large arrays:
X: a 3073 x 49000 matrix
W: a 10 x 3073 matrix
y: a 49000 x 1 vector
this can run into speed issues. Note that W[y.ravel()] is the same size as X.T (why did you pick these sizes that require transpose?). And it's a copy, not a view. So there's already a time penalty.
bincount has been suggested in previous questions, and I think it is faster. Making for loop with index arrays faster (both bincount and add.at solutions)
Iterating over the small 3073 dimension could also have speed advantages. Or better yet on the size 10 dimension as Divakar demonstrates.
For the small test case, a,b,c=3,4,5, the add.at solution is fastest, with Divakar's bincount and einseum next. For a larger a,b,c=10,1000,20000, add.at gets very slow, with bincount being the fastest.
Related SO answers
https://stackoverflow.com/a/28205888/901925 (notes that bincount requires complete coverage for y).
https://stackoverflow.com/a/30041823/901925 (where Divakar again shows that bincount rules!)
Vectorized approaches
Approach #1
Based on this answer, here's a vectorized solution using np.bincount -
N = y.max()+1
id = y.ravel() + np.arange(X.shape[0])[:,None]*N
W[:N] += np.bincount(id.ravel(), weights=X.ravel()).reshape(-1,N).T
Approach #2
You can make good usage of boolean indexing and np.einsum to get the job done in a concise vectorized manner -
N = y.max()+1
W[:N] += np.einsum('ijk,lk->il',(np.arange(N)[:,None,None] == y.ravel()),X)
Loopy approaches
Approach #3
Since you are selecting and adding up a huge number of columns from X per unique y, it might be better in terms of performance to run a loop with complexity equal to the number of such unique y's, which seems to be at max equal to the number of rows in W and that in your case is just 10. Thus, the loop has just 10 iterations, not bad! Here's the implementation to fulfill those aspirations -
for k in range(W.shape[0]):
W[k] += X[:,(y==k).ravel()].sum(1)
Approach #4
You can bring in np.einsum to do the columnwise summations and have the final output like so -
for k in range(W.shape[0]):
W[k] += np.einsum('ij->i',X[:,(y==k).ravel()])
This will achieve what you want: X + W[y.ravel()].T
To see that this really does the work, here is a reproducible example:
import numpy as np
np.random.seed(0)
a, b, c = 3, 5, 4 # you can use your 3073, 49000, 10 later
X = np.random.rand(a, b)
W = np.random.rand(c, a)
y = np.random.randint(c, size=(b, 1))
Now your matrices are:
[[ 0.0871293 0.0202184 0.83261985]
[ 0.77815675 0.87001215 0.97861834]
[ 0.79915856 0.46147936 0.78052918]
[ 0.11827443 0.63992102 0.14335329]]
[[3]
[0]
[3]
[2]
[0]]
[[ 0.5488135 0.71518937 0.60276338 0.54488318 0.4236548 ]
[ 0.64589411 0.43758721 0.891773 0.96366276 0.38344152]
[ 0.79172504 0.52889492 0.56804456 0.92559664 0.07103606]]
And W[y.ravel()] gives you " W given by the first element in y". By transposing it, you will get a matrix ready to be added to X:
[[ 0.11827443 0.0871293 0.11827443 0.79915856 0.0871293 ]
[ 0.63992102 0.0202184 0.63992102 0.46147936 0.0202184 ]
[ 0.14335329 0.83261985 0.14335329 0.78052918 0.83261985]]
While I can't say that this is very pythonic, it is a solution (I think):
for column in range(x.shape[1]):
w[y[column]] = x[:,column].T