I am trying to optimise my computation for the eigenvalue/vectors of a tridiagonal matrix. My code currently takes about 30s to run and I need it to run much faster, at around 1s. I am currently using the scipy.sparse.linalg.eigsh method for finding the eigenvalue/vectors but I believe that there should be other ways which are much faster. I will post my code below so it is a little easier to see what I am trying to do. Any suggestions are greatly welcome and if there is anything unclear please let me know.
H = (dx**-2)*diags([-1, 2, -1], [-1, 0, 1], shape=(N, N))
H = sp.lil_matrix(H)
H[0,0]=0.5*H[0,0]
H[N-1,N-1]=0.5*H[N-1,N-1]
lam, phi = eigsh(H, 400, which="SM")
You're unlikely to find a simple way to speed up this calculation by 30x. As you can see from the timings in this post, even Google's impressively optimized jax library only manages a bit less than a 3x speedup over the base SVD implementation in scipy. Since SVD is equivalent to eigsh for symmetric matrices, we can expect that the speedup for eigsh will be about the same at best.
So even Google can't speed this calculation up that much, at least in the traditional way.
However, for very large, sparse matrices, there are specialized stochastic algorithms that can speed things up by much larger factors under the right circumstances. One of them has an sklearn implementation: sklearn.utils.extmath.randomized_svd. (I believe it's based on the algorithm described here, and it may also be the same as sklearn.decomposition.TruncatedSVD when algorithm='randomized'.)
Here's a slightly modified version of your example, showing how to use it:
from scipy.sparse import diags, lil_matrix
from scipy.sparse.linalg import eigsh
from sklearn.utils.extmath import randomized_svd
N = 3200
k = 400
dx = 1.0
H = (dx**-2)*diags([-1, 2, -1], [-1, 0, 1], shape=(N, N))
H = lil_matrix(H)
H[0, 0] = 0.5*H[0, 0]
H[N-1, N-1] = 0.5*H[N-1, N-1]
# lam, phi = eigsh(H, k, which="SM")
U, s, V = randomized_svd(H, k)
Here, s contains the singular values (equivalent here to eigenvalues), and U and V contain the singular vectors (equivalent here to eigenvectors). In theory, if H is a symmetric matrix, U should be the same as V.T. I didn't find that to be the case for this example, which puzzles me a bit... but I'm going ahead and posting this anyway, because this does in fact run in about a second, and so might be a solution for you.
However, there is still one more big caveat. You're passing which="SM" to eigsh, which as I understand it means you are asking for the 400 smallest eigenvalues. Is that really what you want? In almost all applications I am aware of, you'd want the 400 largest eigenvalues. If in fact you really do want the 400 smallest eigenvalues, then this won't work, because it relies on the fact that the largest eigenvalues are easier to coax out using random matrix projections.
The upshot is, you will want to test this to see if it actually gives results you can use. But it's an interesting possible solution, given what you've said so far about your problem.
Related
Suppose I have a symmetric matrix A and a vector b and want to find A^(-1) b. Now, this is well-known to be doable in time O(N^2) (where N is the dimension of the vector\matrix), and I believe that in MATLAB this can be done as b\A. But all I can find in python is numpy.linalg.solve() which will do Gaussian elimination, which is O(N^3). I must not be looking in the right place...
scipy.linalg.solve has an argument to make it assume a symmetric matrix:
x = scipy.linalg.solve(A, b, assume_a="sym")
If you know your matrix is not just symmetric but positive definite you can give this stronger assumption instead, as "pos".
I have data which forms a sparse matrix in shape of 1000 x 1e9. I want to cluster the 1000 examples into 10 clusters using K-means.
The matrix is very sparse, less than 1/1e6 values.
My laptop got 16 RAM. I tried sparse matrix in scipy. Unfortunately, the matrix makes the clustering process need much more memory than I have. Could anyone suggest a way to do this?
My system crashed when running the following test snippet
import numpy as np
from scipy.sparse import csr_matrix
from sklearn.cluster import KMeans
row = np.array([0, 0, 1, 2, 2, 2, 3, 3, 4, 5, 5, 5, 6, 6, 7, 8, 8, 8])
col = np.array([0, 2, 2, 0, 1, 2] * 3)
data = np.array([1, 2, 3, 4, 5, 6] * 3)
X = csr_matrix((data, (row, col)), shape=(9, 1e9))
resC = KMeans(n_clusters=3).fit(X)
resC.labels_
Any helpful suggestion is appreciated.
KMeans centers will not be sparse anymore, so this would need careful optimization for the sparse case (that may be costly for the usual case, so it probably isn't optimized this way).
You can try ELKI (not python but Java) which often is much faster, and also has sparse data types. You can also try using single-precision float will also help.
But in the end, the results will be questionable: k-means is statistically rooted in least-squares. It assumes your data is coming from k signals plus some Gaussian error. Because your data is sparse, it obviously does not have this kind of Gaussian shape. When the majority of values is 0, it cannot be a Gaussian.
With just 1000 data points, I'd rather use HAC.
Whatever you do (for your data; given your memory-constraints): kmeans is not ready for that!
This includes:
Online KMeans / MiniBatch Kmeans; as proposed in another answer
it only helps to handle many samples (and is hurt by the same effect mentioned later)!
Various KMeans-implementation in different languages (it's an algorithmic problem; not bound by an implementation)
Ignoring potential theoretic reasons (high-dimensionality and non-convex heuristic optimization) i'm just mentioning the practical problem here:
your centroids might become non-sparse! (mentioned in sidenote by SOs clustering-expert; this link also mentions alternatives!)
this means: the sparse data-structures used will get very non-sparse and eventually blow up your memory!
(i changed sklearn's code to observe what the above link already mentioned)
relevant sklearn code: center_shift_total = squared_norm(centers_old - centers)
Even if you remove / turn-off all the memory-heavy components like:
init=some_sparse_ndarray (instead of k-means++)
n_init=1 instead of 10
precompute_distances=False instead of True (unclear if it helps)
n_jobs=1 instead of -1
the above will be your problem to care!
Although KMeans accepts sparse matrices as input, the centroids used within the algorithm have a dense representation, and your feature space is so big that even 10 centroids will not fit into 16GB of RAM.
I have 2 ideas:
Can you fit the clustering into RAM if you discard all empty columns? If you have 1000 samples and only about 1/1e6 values are occupied, then probably less than 1 in 1000 columns will contain any non-zero entries.
Several clustering algorithms in scikit-learn will accept a matrix of distances between samples in stead of the full data e.g. sklearn.cluster.SpectralClustering. You could precompute the pairwise distances in a 1000x1000 matrix and pass that to your clustering algorithm in stead. (I can't make a specific recommendation of a clustering method, or a suitable function to calculate the distances, as it will depend on your application)
Consider using dict, since it will only store the values wich were assigned. I guess a nice way to do this is by creating a SparseMatrix object like this:
class SparseMatrix(dict):
def __init__(self, mapping=[]):
dict.__init__(self, {i:mapping[i] for i in range(len(mapping))})
#overriding this method makes never-accessed indexes return 0.0
def __getitem__(self, i):
try:
return dict.__getitem__(self, i)
except KeyError:
return 0.0
>>> my_matrix = SparseMatrix([1,2,3])
>>> my_matrix[0]
1
>>> my_matrix[5]
0.0
Edit:
For the multi-dimensional case you may need to override the two item-management methods as follows:
def __getitem__(self, ij):
i,j = ij
dict.__setitem__(i*self.n + j)
def __getitem__(self, ij):
try:
i,j = ij
return dict.__getitem__(self, i*self.n + j)
except KeyError:
return 0.0
>>> my_matrix[0,0] = 10
>>> my_matrix[1,2]
0.0
>>> my_matrix[0,0]
10
Also assuming you defined self.n as the length of the matrix rows.
I found some examples online showing how to find the null space of a regular matrix in Python, but I couldn't find any examples for a sparse matrix (scipy.sparse.csr_matrix).
By null space I mean x such that M·x = 0, where '·' is matrix multiplication. Does anybody know how to do this?
Furthermore, in my case I know that the null space will consist of a single vector. Can this information be used to improve the efficiency of the method?
This isn't a complete answer yet, but hopefully it will be a starting point towards one. You should be able to compute the null space using a variant on the SVD-based approach shown for dense matrices in this question:
import numpy as np
from scipy import sparse
import scipy.sparse.linalg
def rand_rank_k(n, k, **kwargs):
"generate a random (n, n) sparse matrix of rank <= k"
a = sparse.rand(n, k, **kwargs)
b = sparse.rand(k, n, **kwargs)
return a.dot(b)
# I couldn't think of a simple way to generate a random sparse matrix with known
# rank, so I'm currently using a dense matrix for proof of concept
n = 100
M = rand_rank_k(n, n - 1, density=1)
# # this seems like it ought to work, but it doesn't
# u, s, vh = sparse.linalg.svds(M, k=1, which='SM')
# this works OK, but obviously converting your matrix to dense and computing all
# of the singular values/vectors is probably not feasible for large sparse matrices
u, s, vh = np.linalg.svd(M.todense(), full_matrices=False)
tol = np.finfo(M.dtype).eps * M.nnz
null_space = vh.compress(s <= tol, axis=0).conj().T
print(null_space.shape)
# (100, 1)
print(np.allclose(M.dot(null_space), 0))
# True
If you know that x is a single row vector then in principle you would only need to compute the smallest singular value/vector of M. It ought to be possible to do this using scipy.sparse.linalg.svds, i.e.:
u, s, vh = sparse.linalg.svds(M, k=1, which='SM')
null_space = vh.conj().ravel()
Unfortunately, scipy's svds seems to be badly behaved when finding small singular values of singular or near-singular matrices and usually either returns NaNs or throws an ArpackNoConvergence error.
I'm not currently aware of an alternative implementation of truncated SVD with Python bindings that will work on sparse matrices and can selectively find the smallest singular values - perhaps someone else knows of one?
Edit
As a side note, the second approach seems to work reasonably well using MATLAB or Octave's svds function:
>> M = rand(100, 99) * rand(99, 100);
% svds converges much more reliably if you set sigma to something small but nonzero
>> [U, S, V] = svds(M, 1, 1E-9);
>> max(abs(M * V))
ans = 1.5293e-10
I have been trying to find a solution to the same problem. Using Scipy's svds function provides unreliable results for small singular values. Therefore i have been using QR decomposition instead. The sparseqr https://github.com/yig/PySPQR provides a wrapper for Matlabs SuiteSparseQR method, and works reasonably well. Using this the null space can be calculated as:
from sparseqr import qr
Q, _, _,r = qr( M.transpose() )
N = Q.tocsr()[:,r:]
I am trying to factorize very large matrixes with the python library Nimfa.
Since the matrix is so large I am unable to instanciate it in a dence format in memory, so instead I use scipy.sparse.csr_matrix.
The library has a sparse matrix function that is called Snmf: Sparse Nonnegative Matrix Factorization (SNMF), which appears to be what I am looking for.
When trying it out I had serious performance issues with the factorization (not memory representation but in speed) I have not yet been able to factor a simple 10 x 95 matrix that is sparse.
This is how I build the test matrix:
m1 = lil_matrix((10, 95))
for i in xrange(10):
for j in xrange(95):
if random.random() > 0.8: m1[i, j] = 1
m1 = csc_matrix(m1)
and this is how I run it
t = time()
fctr = nimfa.mf(m1,
seed = "random_vcol",
rank = 2,
method = "snmf",
max_iter = 15,
initialize_only = True,
version = 'r',
eta = 1.,
beta = 1e-4,
i_conv = 10,
w_min_change = 0)
print numpy.shape(m1)
a = nimfa.mf_run(fctr)
print a.coef()
print a.basis()
print time() - t
This doesn't seem to finish at all. But if i do m1.todense() it finishes in seconds. Since I am unable to instanciate my real matrix this is not really a good solution for me.
I have tried different scipy.sparse matrix format but to no avail: csc_matrix, csr_matrix and dok_matrix.
Am I using wrong matrix format? What matrix operations does the snmf algorithm need to execute quickly? Is there some other mistake I am overlooking?
I did some digging, and there seems to be a bug in their sparse implementation. What it is, I don't know, but if you look at line 289 in _spfcnlls len(f_set) never decreases and the loop runs forever. When the matrix is not sparse, that method is never called. I opened an issue on the github repository here.
In the meantime, is there a factorization function in numpy or scipy that would fit your needs?
How do I get the inverse of a matrix in python? I've implemented it myself, but it's pure python, and I suspect there are faster modules out there to do it.
You should have a look at numpy if you do matrix manipulation. This is a module mainly written in C, which will be much faster than programming in pure python. Here is an example of how to invert a matrix, and do other matrix manipulation.
from numpy import matrix
from numpy import linalg
A = matrix( [[1,2,3],[11,12,13],[21,22,23]]) # Creates a matrix.
x = matrix( [[1],[2],[3]] ) # Creates a matrix (like a column vector).
y = matrix( [[1,2,3]] ) # Creates a matrix (like a row vector).
print A.T # Transpose of A.
print A*x # Matrix multiplication of A and x.
print A.I # Inverse of A.
print linalg.solve(A, x) # Solve the linear equation system.
You can also have a look at the array module, which is a much more efficient implementation of lists when you have to deal with only one data type.
Make sure you really need to invert the matrix. This is often unnecessary and can be numerically unstable. When most people ask how to invert a matrix, they really want to know how to solve Ax = b where A is a matrix and x and b are vectors. It's more efficient and more accurate to use code that solves the equation Ax = b for x directly than to calculate A inverse then multiply the inverse by B. Even if you need to solve Ax = b for many b values, it's not a good idea to invert A. If you have to solve the system for multiple b values, save the Cholesky factorization of A, but don't invert it.
See Don't invert that matrix.
It is a pity that the chosen matrix, repeated here again, is either singular or badly conditioned:
A = matrix( [[1,2,3],[11,12,13],[21,22,23]])
By definition, the inverse of A when multiplied by the matrix A itself must give a unit matrix. The A chosen in the much praised explanation does not do that. In fact just looking at the inverse gives a clue that the inversion did not work correctly. Look at the magnitude of the individual terms - they are very, very big compared with the terms of the original A matrix...
It is remarkable that the humans when picking an example of a matrix so often manage to pick a singular matrix!
I did have a problem with the solution, so looked into it further. On the ubuntu-kubuntu platform, the debian package numpy does not have the matrix and the linalg sub-packages, so in addition to import of numpy, scipy needs to be imported also.
If the diagonal terms of A are multiplied by a large enough factor, say 2, the matrix will most likely cease to be singular or near singular. So
A = matrix( [[2,2,3],[11,24,13],[21,22,46]])
becomes neither singular nor nearly singular and the example gives meaningful results... When dealing with floating numbers one must be watchful for the effects of inavoidable round off errors.
For those like me, who were looking for a pure Python solution without pandas or numpy involved, check out the following GitHub project: https://github.com/ThomIves/MatrixInverse.
It generously provides a very good explanation of how the process looks like "behind the scenes". The author has nicely described the step-by-step approach and presented some practical examples, all easy to follow.
This is just a little code snippet from there to illustrate the approach very briefly (AM is the source matrix, IM is the identity matrix of the same size):
def invert_matrix(AM, IM):
for fd in range(len(AM)):
fdScaler = 1.0 / AM[fd][fd]
for j in range(len(AM)):
AM[fd][j] *= fdScaler
IM[fd][j] *= fdScaler
for i in list(range(len(AM)))[0:fd] + list(range(len(AM)))[fd+1:]:
crScaler = AM[i][fd]
for j in range(len(AM)):
AM[i][j] = AM[i][j] - crScaler * AM[fd][j]
IM[i][j] = IM[i][j] - crScaler * IM[fd][j]
return IM
But please do follow the entire thing, you'll learn a lot more than just copy-pasting this code! There's a Jupyter notebook as well, btw.
Hope that helps someone, I personally found it extremely useful for my very particular task (Absorbing Markov Chain) where I wasn't able to use any non-standard packages.
You could calculate the determinant of the matrix which is recursive
and then form the adjoined matrix
Here is a short tutorial
I think this only works for square matrices
Another way of computing these involves gram-schmidt orthogonalization and then transposing the matrix, the transpose of an orthogonalized matrix is its inverse!
Numpy will be suitable for most people, but you can also do matrices in Sympy
Try running these commands at http://live.sympy.org/
M = Matrix([[1, 3], [-2, 3]])
M
M**-1
For fun, try M**(1/2)
If you hate numpy, get out RPy and your local copy of R, and use it instead.
(I would also echo to make you you really need to invert the matrix. In R, for example, linalg.solve and the solve() function don't actually do a full inversion, since it is unnecessary.)