Related
I have an array of shape [batch_size, N], for example:
[[1 2]
[3 4]
[5 6]]
and I need to create a 3 indices array with shape [batch_size, N, N] where for every batch I have a N x N diagonal matrix, where diagonals are taken by the corresponding batch element, for example in this case, In this simple case, the result I am looking for is:
[
[[1,0],[0,2]],
[[3,0],[0,4]],
[[5,0],[0,6]],
]
How can I make this operation without for loops and exploting vectorization? I guess it is an extension of dimension, but I cannot find the correct function to do this.
(I need it as I am working with tensorflow and prototyping with numpy).
Try it in tensorflow:
import tensorflow as tf
A = [[1,2],[3 ,4],[5,6]]
B = tf.matrix_diag(A)
print(B.eval(session=tf.Session()))
[[[1 0]
[0 2]]
[[3 0]
[0 4]]
[[5 0]
[0 6]]]
Approach #1
Here's a vectorized one with np.einsum for input array, a -
# Initialize o/p array
out = np.zeros(a.shape + (a.shape[1],),dtype=a.dtype)
# Get diagonal view and assign into it input array values
diag = np.einsum('ijj->ij',out)
diag[:] = a
Approach #2
Another based on slicing for assignment -
m,n = a.shape
out = np.zeros((m,n,n),dtype=a.dtype)
out.reshape(-1,n**2)[...,::n+1] = a
Using np.expand_dims with an element-wise product with np.eye
a = np.array([[1, 2],
[3, 4],
[5, 6]])
N = a.shape[1]
a = np.expand_dims(a, axis=1)
a*np.eye(N)
array([[[1., 0.],
[0., 2.]],
[[3., 0.],
[0., 4.]],
[[5., 0.],
[0., 6.]]])
Explanation
np.expand_dims(a, axis=1) adds a new axis to a, which will now be a (3, 1, 2) ndarray:
array([[[1, 2]],
[[3, 4]],
[[5, 6]]])
You can now multiply this array with a size N identity matrix, which you can generate with np.eye:
np.eye(N)
array([[1., 0.],
[0., 1.]])
Which will yield the desired output:
a*np.eye(N)
array([[[1., 0.],
[0., 2.]],
[[3., 0.],
[0., 4.]],
[[5., 0.],
[0., 6.]]])
Yu can use numpy.diag
m = [[1, 2],
[3, 4],
[5, 6]]
[np.diag(b) for b in m]
EDIT The following plot shows the average execution time for the solution above (solid line), and compared it against #Divakar's (dashed line) for different batch-sizes and different matrix sizes
I don't believe you get much of an improvement, but this is just based on this simple metric
You basically want a function that does the opposite of/reverses np.block(..)
I needed the same thing, so I wrote this little function:
def split_blocks(x, m=2, n=2):
"""
Reverse the action of np.block(..)
>>> x = np.random.uniform(-1, 1, (2, 18, 20))
>>> assert (np.block(split_blocks(x, 3, 4)) == x).all()
:param x: (.., M, N) input matrix to split into blocks
:param m: number of row splits
:param n: number of column, splits
:return:
"""
x = np.array(x, copy=False)
nd = x.ndim
*shape, nr, nc = x.shape
return list(map(list, x.reshape((*shape, m, nr//m, n, nc//n)).transpose(nd-2, nd, *range(nd-2), nd-1, nd+1)))
I'd like to accumulate the sum of a numpy array at specific positions of another numpy array without using any loop. This is similar to what numpy.put does, but I'd like to accumule the sum instead of replacing the elements.
Consider the following example:
import numpy as np
a = np.zeros([2, 2])
b = np.array([[1, 2], [3, 4], [5, 6]])
indices = np.array([[0, 2], [0, 1], [0, 1]])
Here, np.put(a, indices, b) gives the following result:
a = [[ 5. 6.][ 2. 0.]]
Instead, I'd like to obtain:
a = [[ 9. 10.][ 2. 0.]]
Is there an efficient way to do this?
You could use np.add.at:
>>> import numpy as np
>>> a = np.zeros([2, 2])
>>> b = np.array([[1, 2], [3, 4], [5, 6]])
>>> indices = np.array([[0, 2], [0, 1], [0, 1]])
>>>
>>> np.add.at(a.ravel(), indices.ravel(), b.ravel())
>>> a
array([[ 9., 10.],
[ 2., 0.]])
>>>
an alternative, and in my experience often quite a bit faster is np.bincount:
>>> a = np.zeros([2, 2])
>>> a += np.bincount(indices.ravel(), b.ravel(), minlength=a.size).reshape(a.shape)
>>> a
array([[ 9., 10.],
[ 2., 0.]])
>>>
Lets say , S is the large scipy-csr-matrix(sparse) and a dictionary D with key -> index(position) of the row vector A in S & values -> list of all the indices(positions) of other row vectors l in S. For each row vector in l you subtract A and get the new vector which will be nothing but the new row vector to be updated in the new sparse matrix.
dictionary of form -> { 1 : [4 , 5 ,... ,63] }
then have to create a new sparse matrix with....
new_row_vector_1 -> S_vec1 - S_vec4
new_row_vector_2 -> S_vec1 - S_vec5
.
new_row_vector_n -> S_vec1 - S_vec63
where S_vecX is the Xth row vector of matrix S
Check out the pictorial explanation of the above statements
Numpy Example:
>>> import numpy as np
>>> s = np.array([[1,5,3,4],[3,0,12,7],[5,6,2,4],[4,6,6,4],[7,12,5,67]])
>>> s
array([[ 1, 5, 3, 4],
[ 3, 0, 12, 7],
[ 5, 6, 2, 4],
[ 4, 6, 6, 4],
[ 7, 12, 5, 67]])
>>> index_dictionary = {0: [2, 4], 1: [3, 4], 2: [1], 3: [1, 2], 4: [1, 3, 2]}
>>> n = np.zeros((10,4)) #sum of all lengths of values in index_dictionary would be the number of rows for the new array(n) and columns remain the same as s.
>>> n
array([[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]])
>>> idx = 0
>>> for index in index_dictionary:
... for k in index_dictionary[index]:
... n[idx] = s[index]-s[k]
... idx += 1
...
>>> n
array([[ -4., -1., 1., 0.],
[ -6., -7., -2., -63.],
[ -1., -6., 6., 3.],
[ -4., -12., 7., -60.],
[ 2., 6., -10., -3.],
[ 1., 6., -6., -3.],
[ -1., 0., 4., 0.],
[ 4., 12., -7., 60.],
[ 3., 6., -1., 63.],
[ 2., 6., 3., 63.]])
n is what i want.
Here's a simple demonstration of what I think you are trying to do:
First the numpy array version:
In [619]: arr=np.arange(12).reshape(4,3)
In [620]: arr[[1,0,2,3]]-arr[0]
Out[620]:
array([[3, 3, 3],
[0, 0, 0],
[6, 6, 6],
[9, 9, 9]])
Now the sparse equivalent:
In [622]: M=sparse.csr_matrix(arr)
csr implements row indexing:
In [624]: M[[1,0,2,3]]
Out[624]:
<4x3 sparse matrix of type '<class 'numpy.int32'>'
with 11 stored elements in Compressed Sparse Row format>
In [625]: M[[1,0,2,3]].A
Out[625]:
array([[ 3, 4, 5],
[ 0, 1, 2],
[ 6, 7, 8],
[ 9, 10, 11]], dtype=int32)
But not broadcasting:
In [626]: M[[1,0,2,3]]-M[0]
....
ValueError: inconsistent shapes
So we can use an explicit form of broadcasting
In [627]: M[[1,0,2,3]]-M[[0,0,0,0]] # or M[[0]*4]
Out[627]:
<4x3 sparse matrix of type '<class 'numpy.int32'>'
with 9 stored elements in Compressed Sparse Row format>
In [628]: _.A
Out[628]:
array([[3, 3, 3],
[0, 0, 0],
[6, 6, 6],
[9, 9, 9]], dtype=int32)
This may not be the fastest or most efficient, but it's a start.
I found in a previous SO question that M[[1,0,2,3]] indexing is performed with a matrix multiplication, in this case the equivalent of:
idxM = sparse.csr_matrix(([1,1,1,1],([0,1,2,3],[1,0,2,3])),(4,4))
M1 = idxM * M
Sparse matrix slicing using list of int
So my difference expression requires 2 such multiplication along with the subtraction.
We could try a row by row iteration, and building a new matrix from the result, but there's no guarantee that it will be faster. Depending on the arrays, converting to dense and back might even be faster.
=================
I can imagine 2 ways of applying this to the dictionary.
One is to iterate through the dictionary (what order?), perform this difference for each key, collect the results in a list (list of sparse matrices), and use sparse.bmat to join them into one matrix.
Another is to collect two lists of indexes, and apply the above indexed difference just once.
In [8]: index_dictionary = {0: [2, 4], 1: [3, 4], 2: [1], 3: [1, 2], 4: [1, 3, 2]}
In [10]: alist=[]
...: for index in index_dictionary:
...: for k in index_dictionary[index]:
...: alist.append((index, k))
In [11]: idx = np.array(alist)
In [12]: idx
Out[12]:
array([[0, 2],
[0, 4],
[1, 3],
[1, 4],
[2, 1],
[3, 1],
[3, 2],
[4, 1],
[4, 3],
[4, 2]])
applied to your dense s:
In [15]: s = np.array([[1,5,3,4],[3,0,12,7],[5,6,2,4],[4,6,6,4],[7,12,5,67]])
In [16]: s[idx[:,0]]-s[idx[:,1]]
Out[16]:
array([[ -4, -1, 1, 0],
[ -6, -7, -2, -63],
[ -1, -6, 6, 3],
[ -4, -12, 7, -60],
[ 2, 6, -10, -3],
[ 1, 6, -6, -3],
[ -1, 0, 4, 0],
[ 4, 12, -7, 60],
[ 3, 6, -1, 63],
[ 2, 6, 3, 63]])
and to the sparse equivalent
In [19]: arr= sparse.csr_matrix(s)
In [20]: arr
Out[20]:
<5x4 sparse matrix of type '<class 'numpy.int32'>'
with 19 stored elements in Compressed Sparse Row format>
In [21]: res=arr[idx[:,0]]-arr[idx[:,1]]
In [22]: res
Out[22]:
<10x4 sparse matrix of type '<class 'numpy.int32'>'
with 37 stored elements in Compressed Sparse Row format>
I have a 2D numpy array defined A, for example. I want to transform it into another 2D arrays according to the following statements:
B = A - mean(A), the mean by the second axis
C = B / mean(A)
An example:
>>> import numpy as np
>>> A = np.array([[1, 2, 3], [4, 6, 8]])
>>> A
array([[1, 2, 3],
[4, 6, 8]])
>>> M = np.mean(A, axis=1)
>>> M
array([ 2., 6.])
>>> B = ... # ???
>>> B
array([[-1., 0., 1.],
[-2., 0., 2.]])
>>> C = ... # ???
>>> C
array([[-0.5, 0., 0.5],
[-0.33333333, 0., 0.33333333]])
Annoyingly, numpy.mean(axis=...) gives you an array where the relevant axis has been deleted rather than set to size 1. So when you apply this to a 2x3 array with axis=1, you get a (rank-1) array of size 2 rather than the 2x1 array you really want.
You can fix this up by supplying the keepdims argument to numpy.mean:
M = np.mean(A, axis=1, keepdims=True)
If that hadn't existed, an alternative would have been to use reshape.
Gareht McCaughan's solution is more elegant, but in the case keepdims did not exist, you could add a new axis to M:
B = A - M[:, None]
(M[:, None].shape is (2, 1), so broadcasting happens)
You can use the functions subtract and divide from numpy. Solving your example:
import numpy as np
A = np.array([[1, 2, 3], [4, 6, 8]])
M = np.mean(A, axis=1)
B = np.subtract(A.T,M).T
C = np.divide(B.T,M).T
print(B)
print(C)
, results in:
[[-1. 0. 1.]
[-2. 0. 2.]]
[[-0.5 0. 0.5 ]
[-0.33333333 0. 0.33333333]]
m is a ndarray with shape (12, 21, 21), now I want to take only a sparse slice of it to form a new 2D array, with
sliceid = 0
indx = np.array([0, 2, 4, 6, 8, 10])
so that sparse_slice is, intuitively,
sparse_slice = m[sliceid, indx, indx]
but apparently the above operation does not work, currently what I am using is
sparse_slice = m[sliceid,indx,:][:, indx]
why is the first "intuitive" way not working? and is there a more compact way than my current solution? all my previous ndarray slicing trials were based on nothing but intuition, maybe I shall switch to read some serious manual now...
The more compact way is to do new = m[0, :12:2, :12:2]. This is what the numpy docs call "basic indexing" meaning that you slice with an integer or a slice object (ie 0:12:2). When you use basic indexing numpy returns a view of the original array. For example:
In [3]: a = np.zeros((2, 3, 4))
In [4]: b = a[0, 1, ::2]
In [5]: b
Out[5]: array([ 0., 0.])
In [6]: b[:] = 7
In [7]: a
Out[7]:
array([[[ 0., 0., 0., 0.],
[ 7., 0., 7., 0.],
[ 0., 0., 0., 0.]],
[[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]]])
In you "intuitive" approach what you're doing is indexing an array with another array. When you index an numpy array with another array the arrays need to be the same size (or they need to broadcast against each other, more on this in a sec). In the docs this is called fancy indexing or advanced indexing. For example:
In [10]: a = np.arange(9).reshape(3,3)
In [11]: a
Out[11]:
array([[0, 1, 2],
[3, 4, 5],
[6, 7, 8]])
In [12]: index = np.array([0,1,2])
In [13]: b = a[index, index]
In [14]: b
Out[14]: array([0, 4, 8])
You see that I get a[0,0], a[1,1], and a[2,2] not a[0,0], a[0,1] ... If you want the "outer product" of index with index you can do the following.
In [22]: index1 = np.array([[0,0],[1,1]])
In [23]: index2 = np.array([[0,1],[0,1]])
In [24]: b = a[index1, index2]
In [25]: b
Out[25]:
array([[0, 1],
[3, 4]])
There is a shorthand for doing the above, like this:
In [28]: index = np.array([0,1])
In [29]: index1, index2 = np.ix_(index, index)
In [31]: index1
Out[31]:
array([[0],
[1]])
In [32]: index2
Out[32]: array([[0, 1]])
In [33]: a[index1, index2]
Out[33]:
array([[0, 1],
[3, 4]])
In [34]: a[np.ix_(index, index)]
Out[34]:
array([[0, 1],
[3, 4]])
You'll notice that index1 is (2, 1) and index2 is (1, 2), not (2, 2). That's because the two arrays get broadcast against one another, you can read more about broadcasting here. Keep in mind that when you're using fancy indexing you get a copy of the original data not a view. Sometimes this is better (if you want to leave the original data unchanged) and sometimes it just takes more memory. More about indexing here.
If I'm not mistaken, for input m = np.array(range(5292)).reshape(12,21,21) you are expecting output sparse_slice = m[sliceid,indx,:][:, indx] of
array([[ 0, 2, 4, 6, 8, 10],
[ 42, 44, 46, 48, 50, 52],
[ 84, 86, 88, 90, 92, 94],
[126, 128, 130, 132, 134, 136],
[168, 170, 172, 174, 176, 178],
[210, 212, 214, 216, 218, 220]])
In that case, you can get it using the step part of a slice:
m[0, :12:2, :12:2]