I have a matrix which has to be transformed to a symmetric matrix using
python numpy.
Apparently, using the following code in am able to transform to symmetric matrix .. this work fine for small matrix but for a large 150 * 151
I get the following error...operands could not be broadcast together
with shapes (151,150) (150,151)
import numpy as np
from numpy import genfromtxt
my_data = genfromtxt('C:\\Users\\vish_\\firstest\\demo\\002.csv',
delimiter=',')
print(my_data)
[[ 0. nan nan]
[ 1. 0. nan]
[ 2. 3. 0.]]
<!-- code for symmetric matrix in file!>
m = np.tril(my_data) + np.tril(my_data, -1).T
print(m)
[[0. 1. 2.]
[1. 0. 3.]
[2. 3. 0.]]
<!-- code for saving matrix in file!>
np.set_printoptions(suppress=True)
np.set_printoptions(precision=3)
m.tofile('foo2.csv',sep=',',format='%10.5f')
np.savetxt("foo2.csv", m ,delimiter=",",fmt='%f')
I get the following error...operands could not be broadcast together
with shapes (151,150) (150,151)
You don't have a square matrix. 150 x 151 is not a square matrix! The reason it worked for smaller example because it was a square matrix (3 x 3). The example on which you are trying to work is not a square matrix. Please there is a transpose operator in the code (see below towards right).
m = np.tril(my_data) + np.tril(my_data, -1).T
For 150 x 151 the transpose will be 151 x 150 shape. The dimensions will not line up for the addition to happen. Best wishes
Related
The correlation matrix is a symmetric matrix, meaning that its upper diagonal and lower diagonal elements are mirror images of each other, together called off-diagonal elements (as opposed to the diagonal elements, which are all equal to 1 in any correlation matrix since any variable's correlation with itself is just 1).
The off-diagonal elements of a correlation matrix are the same wherever the i'th row number and j'th column number in the lower diagonal are swapped in the upper diagonal, i.e. correlation of variables 1 and 2 (row 1, column 2) are the same for variables 2 and 1 (row 2, column 1). Therefore, we only need to re-calculate the lower-diagonal elements, and copy them to corresponding positions in the matrix's upper-diagonal after
import numpy as np
from numpy.random import randn
X = randn(20,3)
Rho = np.corrcoef(X.T) #correlation matrix
print(np.tril(Rho)) #lower off-diagonal of matrix Rho to re-calculate, then copy to other side
shows
array([[ 1. , 0. , 0. ],
[-0.03003281, 1. , 0. ],
[-0.02602238, 0.06137713, 1. ]])
What is the most efficient way to code a "i not-equal-to j" loop for the following sequence of steps:
re-calculate the lower off-diagonal elements of the symmetric matrix according to some apply function (to make it simple, we will just add +2 to each of these elements)
flip those same calculations onto its mirror image (the corresponding upper off-diagonals)
Also, replace the diagonal elements of the symmetric matrix with a vector filled with 10's (instead of 1's as found in the correlation matrix)
The aim is to generate a new matrix that is a re-calculation of the original.
Let us generate Rho first (note that I'm initializing the pseudo-random number generator in order to obtain the same Rho in different runs of the code):
In [526]: import numpy as np
In [527]: np.random.seed(0)
...: n = 3
...: X = np.random.randn(20, n)
...: Rho = np.corrcoef(X.T)
In [528]: Rho
Out[528]:
array([[1. , 0.03224462, 0.05021998],
[0.03224462, 1. , 0.15140358],
[0.05021998, 0.15140358, 1. ]])
Then you can use NumPy's tril_indices_from and advanced indexing to generate the new matrix:
In [548]: result = np.zeros_like(Rho)
In [549]: lrows, lcols = np.tril_indices_from(Rho, k=-1)
In [550]: result[lrows, lcols] = Rho[lrows, lcols] + 2
In [551]: result
Out[551]:
array([[0. , 0. , 0. ],
[2.03224462, 0. , 0. ],
[2.05021998, 2.15140358, 0. ]])
In [552]: result[lcols, lrows] = result[lrows, lcols]
In [553]: result
Out[553]:
array([[0. , 2.03224462, 2.05021998],
[2.03224462, 0. , 2.15140358],
[2.05021998, 2.15140358, 0. ]])
In [554]: result[np.arange(n), np.arange(n)] = 10
In [555]: result
Out[555]:
array([[10. , 2.03224462, 2.05021998],
[ 2.03224462, 10. , 2.15140358],
[ 2.05021998, 2.15140358, 10. ]])
I have a 3d numpy array of point (484,3,1) and a 2d transformation matrix (3,3). I want to compute the transformation for all 484 points.
I have tried to reshape the arrays and compute the dot product, but I am struggling to get it to output a (484,3,1) shaped array where all the points are transformed.
points = np.random.randint(0, 979, (484,3,1))
transformation = array([[0.94117647, 0. , 0. ],
[0. , 0.94117647, 0. ],
[0. , 0. , 1. ]])
points.shape = (484,3,1)
transformation = (3,3)
transformation.dot(points).shape = (3,484,1)
I would like this to be as optimized as possible. Any advice would be greatly appreciated.
Just do a reshape to (484,3) dimensions and use the np.matmul (np.dot is also possible but since you are looking for a matrix multiplication matmul is prefered according to the documentation) product
np.matmul(points.reshape(484,-1), transformation).reshape(484,3,-1)
resulting shape is the same of course given by the last reshaping: (484,3,1)
Suppose I have a rank two tensor, [[a,b],[c,d]](in general a m-by-n matrix).
I want to expand every element by a 2-2 identity matrix(outer product) and result in
[[a, 0, b, 0],[0,a,0,b],[c,0,d,0],[0,c,0,d]].
What is the most efficient way to implement it in tensorflow?
This operation appears a lot in the framework.
I would like to do this in 2-step process.
If I have a m-by-n matrix, and a 2-2 identity matrix.
First I will enlarge (repeat) the matrix to "2m-2n matrix"
and then multiply by the enlarged identity matrix (2m-2n matrix). As following shows.
import tensorflow as tf
#Process 1 repeat the tensor in 2D.
#e.g. [1,2,3,4] --> [[1,1,2,2],[1,1,2,2],[3,3,4,4],[3,3,4,4]]
# assuming m x n idenity matrix e.g. [1,2,3],[4,5,6]] , m=2,n=3
id_matrix_size=2 # size of identity matrix (e.g. 2x2 3x3 ...)
m=2
n=3
mytensor=tf.constant([[1,2,3],[4,5,6] ],dtype = tf.float32)
# similar to np.repeat in x-dimension.
flatten_mytensor=tf.reshape(mytensor,[-1,1])
a=tf.tile(flatten_mytensor, [1, id_matrix_size])
b=tf.reshape( a, [m,-1])
# tile in y-dimension
c=tf.tile(b,[1,id_matrix_size])
d=tf.reshape(c,[id_matrix_size*m,id_matrix_size*n])
#Process 2 tile identity matrix in 2D.
identity_matrix=tf.eye(id_matrix_size) # identity matrix
identity_matrix_2D= tf.tile(identity_matrix,[m,n])
#Process 3 elementwise multiply
output = tf.multiply(d,identity_matrix_2D )
with tf.Session() as sess:
print(sess.run(output) )
#output :
#[[1. 0. 2. 0. 3. 0.]
# [0. 1. 0. 2. 0. 3.]
# [4. 0. 5. 0. 6. 0.]
# [0. 4. 0. 5. 0. 6.]]
And also, a def is more convenient if lots of implements are needed.
def Expand_tensor(mytensor,id_matrix_size):
m=mytensor.shape[0]
n=mytensor.shape[1]
# similar to np.repeat in x-dimension.
flatten_mytensor=tf.reshape(mytensor,[-1,1])
a=tf.tile(flatten_mytensor, [1, id_matrix_size])
b=tf.reshape( a, [m,-1])
# tile in y-dimension
c=tf.tile(b,[1,id_matrix_size])
d=tf.reshape(c,[id_matrix_size*m,id_matrix_size*n])
# tile identity matrix in 2D.
identity_matrix=tf.eye(id_matrix_size) # identity matrix
identity_matrix_2D= tf.tile(identity_matrix,[m,n])
#elementwise multiply
output = tf.multiply(d,identity_matrix_2D )
return output
mytensor=tf.constant([[1,2,3],[4,5,6] ],dtype = tf.float32)
with tf.Session() as sess:
print(sess.run(Expand_tensor(mytensor,2)) )
I have an array of shape (3,2):
import numpy as np
arr = np.array([[0.,0.],[0.25,-0.125],[0.5,-0.125]])
I was trying to build a matrix (matrix) of dimensions (6,2), with the results of the outer product of the elements i,i of arr and arr.T. At the moment I am using a for loop such as:
size = np.shape(arr)
matrix = np.zeros((size[0]*size[1],size[1]))
for i in range(np.shape(arr)[0]):
prod = np.outer(arr[i],arr[i].T)
matrix[size[1]*i:size[1]+size[1]*i,:] = prod
Resulting:
matrix =array([[ 0. , 0. ],
[ 0. , 0. ],
[ 0.0625 , -0.03125 ],
[-0.03125 , 0.015625],
[ 0.25 , -0.0625 ],
[-0.0625 , 0.015625]])
Is there any way to build this matrix without using a for loop (e.g. broadcasting)?
Extend arrays to 3D with None/np.newaxis keeping the first axis aligned, while letting the second axis getting pair-wise multiplied, perform multiplication leveraging broadcasting and reshape to 2D -
matrix = (arr[:,None,:]*arr[:,:,None]).reshape(-1,arr.shape[1])
We can also use np.einsum -
matrix = np.einsum('ij,ik->ijk',arr,arr).reshape(-1,arr.shape[1])
einsum string representation might be more intuitive as it lets us visualize three things :
Axes that are aligned (axis=0 here).
Axes that are getting summed up (none here).
Axes that are kept i.e. element-wise multiplied (axis=1 here).
Let's say I do some calculation and I get a matrix of size 3 by 3 each time in a loop. Assume that each time, I want to save such matrix in a column of a bigger matrix, whose number of rows is equal to 9 (total number of elements in the smaller matrix). first I reshape the smaller matrix and then try to save it into one column of the big matrix. A simple code for only one column looks something like this:
import numpy as np
Big = np.zeros((9,3))
Small = np.random.rand(3,3)
Big[:,0]= np.reshape(Small,(9,1))
print Big
But python throws me the following error:
Big[:,0]= np.reshape(Small,(9,1))
ValueError: could not broadcast input array from shape (9,1) into shape (9)
I also tried to use flatten, but that didn't work either. Is there any way to create a shape(9) array from the small matrix or any other way to handle this error?
Your help is greatly appreciated!
try:
import numpy as np
Big = np.zeros((9,3))
Small = np.random.rand(3,3)
Big[:,0]= np.reshape(Small,(9,))
print Big
or:
import numpy as np
Big = np.zeros((9,3))
Small = np.random.rand(3,3)
Big[:,0]= Small.reshape((9,1))
print Big
or:
import numpy as np
Big = np.zeros((9,3))
Small = np.random.rand(3,3)
Big[:,[0]]= np.reshape(Small,(9,1))
print Big
Either case gets me:
[[ 0.81527817 0. 0. ]
[ 0.4018887 0. 0. ]
[ 0.55423212 0. 0. ]
[ 0.18543227 0. 0. ]
[ 0.3069444 0. 0. ]
[ 0.72315677 0. 0. ]
[ 0.81592963 0. 0. ]
[ 0.63026719 0. 0. ]
[ 0.22529578 0. 0. ]]
Explanation
the shape of Big you are trying to assign to is (9, ) one-dimensional. The shape you are trying to assign with is (9, 1) two-dimensional. You need to reconcile this by making the two-dim a one-dim np.reshape(Small, (9,1)) into np.reshape(Small, (9,)). Or, make the one-dim into a two-dim Big[:, 0] into Big[:, [0]]. The exception is when I assigned 'Big[:, 0] = Small.reshape((9,1))`. In this case, numpy must be checking.