How can I efficiently insert sub-matrices at specific positions into my sparse matrix? Also, which scipy sparse matrix class is recommended for such an incremental construction?
More specifically, how can I fill the matrix M in the code below?
def rrd(mesh, rel_rotations, neighbors, R_0):
M = scipy.sparse.lil_matrix((N_FACES*9*3,N_FACES*9))
for i in range(0,N_FACES*27,27):
for j in range(3):
for k in range(0,N_FACES*9,9):
M[i+j*9:i+(j+1)*9,k:k+9] = -np.eye(9)
for i in range(len(rel_rotations)):
diagonals = [
rel_rotations[i][0][2],
np.append(rel_rotations[i][0][1].repeat(3), rel_rotations[i][1][2].repeat(3)),
np.append(rel_rotations[i][0][0].repeat(3), np.append(rel_rotations[i][1][1].repeat(3),
rel_rotations[i][2][2].repeat(3))),
np.append(rel_rotations[i][1][0].repeat(3), rel_rotations[i][2][1].repeat(3)),
rel_rotations[i][2][0].repeat(3)
]
diag_rel_rotations = scipy.sparse.diags(diagonals, [-6,-3,0,3,6], shape=(9,9)).todense()
mod = i % 3
div = int((i-mod)/3)
n_idx = neighbors[div][mod]
M[i+mod*9:i+(mod+1)*9][n_idx*9:(n_idx+1)*9] = diag_rel_rotations
Slicing doesn't work here and I looked through some different types of sparse matrices but couldn't figure out which is the appropriate one for this problem.
lil is the right one for assignment.
In [553]: M = sparse.lil_matrix((9,9), dtype=int)
In [554]: M
Out[554]:
<9x9 sparse matrix of type '<class 'numpy.int64'>'
with 0 stored elements in LInked List format>
In [555]: M[2:5, 3:6] = np.eye(3)
In [556]: M
Out[556]:
<9x9 sparse matrix of type '<class 'numpy.int64'>'
with 3 stored elements in LInked List format>
In [557]: M.A
Out[557]:
array([[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 1, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0]])
In [558]: d = sparse.diags([[1,2],[1,2,3],[2,3]], [-1,0,1])
In [562]: M[0:3, 6:9] = d
Related
So I have an n*K integer matrix [Note: its a representation of the number of samples drawn from K-distributions (K-columns)]
a =[[0,1,0,0,2,0],
[0,0,1,0,0,0],
[3,0,0,0,0,0],
]
[Note: in the application context this matrix basically means that for the i row (sim instance) we drew 1 element from the "distribution 1" (1 \in [0,..K]) (a[0,1] = 1) and 2 from the distribution 4(a[0,4] = 2)].
What I need is to generate a 0-1 matrix that represents the same integer matrix but with ones(1). In this case, is a 3D matrix of n*a.max()*K that has a 1 for each sample that is drawn from the distributions. [Note: we need this matrix so we can multiply by our K-distribution sample matrix]
Output
b = [[[0,1,0,0,1,0], # we don't care if they samples are stack
[0,0,0,0,1,0],
[0,0,0,0,0,0]], # this is the first row representation
[[0,0,1,0,0,0],
[0,0,0,0,0,0],
[0,0,0,0,0,0]], # this is the second row representation
[[1,0,0,0,0,0],
[1,0,0,0,0,0],
[1,0,0,0,0,0]], # this is the third row representation
]
how to do that in NumPy ?
Thanks !
from #michael-szczesny comment
a = np.array([[0,1,0,0,2,0],
[0,0,1,0,0,0],
[3,0,0,0,0,0],
])
b = (np.arange(1, a.max()+1)[:,None] <= a[:,None]).astype('uint8')
print(b)
array([[[0, 1, 0, 0, 1, 0],
[0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0]],
[[0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]],
[[1, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0]]], dtype=uint8)
I have created a adjacency matrix using networkx as below:
from networkx.algorithms.bipartite.matrix import biadjacency_matrix as adj
user_node_list = data['user_id'].unique()
item_node_list = data['item_id'].unique()
adj_matrix = adj(B, user_node_list, column_order=item_node_list, dtype=None, weight='rating', format='csr')
I want to visualize this adj_matrix. How can I do this?
You can use Pandas to visualize your adj_matrix as following:
import pandas as pd
A = pd.DataFrame(adj_matrix)
Much of the time we're working with graphs with sparse adjacency matrices, so networkx returns a SciPy Compressed Sparse Row matrix rather than a numpy.ndarray or numpy.matrix. The former representation uses more efficient data structures and algorithms for representing and processing sparse matrices. In particular the __repr__ representation of the matrix differs from that of a vanilla (dense) NumPy matrix. It will look something like
<11x11 sparse matrix of type '<class 'numpy.int64'>'
with 28 stored elements in Compressed Sparse Row format>
This makes sense because if the representation of a CSR matrix were the same as what we see with a dense matrix, a simple print statement or logging message could have serious performance impacts if the matrix were very large.
Compare the above output with the __repr__ output of a vanilla (dense) NumPy matrix:
matrix([[0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1],
[0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0],
[0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0],
[0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1],
[0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0]])
which allows us to inspect the matrix elements visually (I am guessing that this is what was meant with "visualize the adj_matrix").
To convert a sparse CSR matrix to a dense NumPy matrix, simply do sparse_matrix.todense(). Note that this representation of a sparse matrix will require substantially more memory, so be mindful of that when working with larger graphs.
I have a matrix that has a quite sparse index (the largest values in both rows and columns are beyond 130000), but only a few of those rows/columns actually have non-zero values.
Thus, I want to have the row and column indices shifted to only represent the non-zero ones, by the first N natural numbers.
Visually, I want a example matrix like this
1 0 1
0 0 0
0 0 1
to look like this
1 1
0 1
but only if all values in the row/column are zero.
Since I do have the matrix in a sparse format, I could simply create a dictionary, store every value by an increasing counter (for row and matrix separately), and get a result.
row_dict = {}
col_dict = {}
row_ind = 0
col_ind = 0
# el looks like this: (row, column, value)
for el in sparse_matrix:
if el[0] not in row_dict.keys():
row_dict[el[0]] = row_ind
row_ind += 1
if el[1] not in col_dict.keys():
col_dict[el[1]] = col_ind
col_ind += 1
# now recreate matrix with new index
But I was looking for maybe an internal function in NumPy. Also note that I do not really know how to word the question, so there might well be a duplicate out there that I do not know of; Any pointers in the right direction are appreciated.
You can use np.unique:
>>> import numpy as np
>>> from scipy import sparse
>>>
>>> A = np.random.randint(-100, 10, (10, 10)).clip(0, None)
>>> A
array([[6, 0, 5, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 7, 0, 0, 0, 0, 4, 9],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 4, 0],
[9, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 4, 0, 0, 0, 0, 0, 0]])
>>> B = sparse.coo_matrix(A)
>>> B
<10x10 sparse matrix of type '<class 'numpy.int64'>'
with 8 stored elements in COOrdinate format>
>>> runq, ridx = np.unique(B.row, return_inverse=True)
>>> cunq, cidx = np.unique(B.col, return_inverse=True)
>>> C = sparse.coo_matrix((B.data, (ridx, cidx)))
>>> C.A
array([[6, 5, 0, 0, 0],
[0, 0, 7, 4, 9],
[0, 0, 0, 4, 0],
[9, 0, 0, 0, 0],
[0, 0, 4, 0, 0]])
Alright, so consider that I have a matrix m, as follows:
m = [[0, 1, 0, 0, 0, 1],
[4, 0, 0, 3, 2, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]]
My goal is to check each row of the matrix and see if the sum of that row is zero. If the sum is not zero, I want to shift the column that corresponds to that row to the end of the matrix. If the sum of the row is zero, nothing happens. So in the given matrix above the following should occur:
The program discovers that the 0th row has a sum that does not equal zero
The 0th column of the matrix is shifted to the end of the matrix, as follows:
m = [[1, 0, 0, 0, 1, 0],
[0, 0, 3, 2, 0, 4],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]]
The program checks the next row and does the same, shifting the column to the end of the matrix
m = [[0, 0, 0, 1, 0, 1],
[0, 3, 2, 0, 4, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]]
Each of the other rows are checked, but since all of their sums are zero no shift is made, and the final result is the matrix above.
The issue arises after shifting the columns of the matrix for the first time, once all of the values are shifted it becomes tricky to tell which column corresponds to the correct row.
I can't use numpy to solve this problem as I can only use the original Python 2 libraries.
Use a simple loop and when the sum is not equal to zero loop over rows again and append the popped first item to each row.
>>> from pprint import pprint
>>> m = [[0, 1, 0, 0, 0, 1],
[4, 0, 0, 3, 2, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]]
>>> for row in m:
# If all numbers are >= 0 then we can short-circuit this using `if any(row):`.
if sum(row) != 0:
for row in m:
row.append(row.pop(0))
...
>>> pprint(m)
[[0, 0, 0, 1, 0, 1],
[0, 3, 2, 0, 4, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]]
list.pop is O(N) operation, if you need something fast then use collections.deque.
deque can rotate elements.
from collections import deque
def rotate(matrix):
matrix_d = [deque(row) for row in matrix]
for row in matrix:
if sum(row) != 0:
for row_d in matrix_d:
row_d.rotate(-1)
return [list(row) for row in matrix_d]
How to construct sparse matrix from diagonal vectors like this:
Lets say my matrix is square with dimension N=6 and i have the following vector
vec = np.array([[1], [1,2]])
and I want to put those parts on diagonals
offset = np.array([2,3])
but vec[0] should start at Mat[0,2] and vec[1] should start at Mat[1,4]
I know about scipy.sparse.diags() but I don't think there is a way to specify just part of a diagonal where non-zero elements are present.
This is just an example to illustrate the problem. In reality I deal with very big arrays and I dont want to waste memory for useless zeros.
Is this the matrix that you want?
In [200]: sparse.dia_matrix(([[0,0,1,0,0,0],[0,0,0,0,1,2]],[2,3]),(6,6)).A
Out[200]:
array([[0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 2],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]])
Yes, the specification includes zeros, which could be annoying in larger cases.
spdiags just wraps the dia_matrix, with the option of converting the result to another format. In your example that converts a 7 element sparse to a 3.
sparse.diags accepts a ragged list of values, but they still need to match the diagonals in length. And internally it converts them to the rectangular array that dia_matrix takes.
S3=sparse.diags([[1,0,0,0],[0,1,2]],[2,3],(6,6))
So if you really need to be parsimonious about the zeros you need to go the coo route.
For example:
In [363]: starts = [[0,2],[1,4]]
In [364]: data = np.concatenate(vec)
In [365]: rows=np.concatenate([range(s[0],s[0]+len(v)) for s,v in zip(starts, vec)])
In [366]: cols=np.concatenate([range(s[1],s[1]+len(v)) for s,v in zip(starts, vec)])
In [367]: sparse.coo_matrix((data,(rows,cols)),(6,6)).A
Out[367]:
array([[0, 0, 1, 0, 0, 0],
[0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 2],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]])