We Keep Coding

What is the most efficient way to calculate the eigen values of a large covariance matrix?

I have been trying for some days to calculate the nearest positive semi-definite matrix from a very large covariance matrix to be able to sample from it.
I have tried MATLAB for the effect, but the memory usage is insane and it always crashes eventually, no error message or log file as far as I searched. The function used for the calculation can be found here https://www.mathworks.com/matlabcentral/fileexchange/42885-nearestspd. Optimizing the function to remove intermediate matrices seemed to reduce the memory usage, but it eventually crashes much in the same way.
Found this approach for doing the calculation https://stackoverflow.com/a/63131309/18660401 and switched to Python, in hopes of finding some GPU libraries to accelerate the calculations, but it seems I cannot find an up-to-date library that suports calculating eigenvectors using the numpy function. This is the function I am using:
import numpy as np
def get_near_psd(A):
C = (A + A.T)/2
eigval, eigvec = np.linalg.eig(C)
eigval[eigval < 0] = 0
return eigvec.dot(np.diag(eigval)).dot(eigvec.T)
I am currently trying to run the same function with numba in hopes that the translation to LLVM is enough to make the calculations in reasonable time, only modified the above version to include the #jit decorator from numba.
There does not seem to be a very optimized way to do this as far as I can find on my own, so any suggestion is very appreciated to crack this.
Edit: The matrix is a two-dimensional 60416x60416 covariance matrix and it is to be used to generate new samples from the distribution of the mean and covariance matrix calculated from a set of samples using a GAN. For training purposes, samples also need to be generated from randomly sampling the distribution, which I am intending to use the function multivariate_normal from numpy for.

A very up to date library that does have these capabilities including GPU support is pytorch, check out the examples on the torch.linalg.eig-function and the corresponding accelerated function torch.linalg.eigh for symmetric/hermitian matrices. You do have to convert these matrices from numpy to pytorch-tensors first to do the computation (and then convert it back), but you can definitely use it in a very similar way.
Of course also this library can't just magically give you more memory, but it is highly optimized.

are scipys solver method interior-point and revised-simplex really slow?

I'm working on some LP and I'm testing the scipy solvers
https://docs.scipy.org/doc/scipy/reference/optimize.linprog-interior-point.html
and
https://docs.scipy.org/doc/scipy/reference/optimize.linprog-revised_simplex.html
I know that there are the newer highs methods
https://docs.scipy.org/doc/scipy/reference/optimize.linprog-highs.html#optimize-linprog-highs
but I want to use the old ones as I want to test a "normal" simplex implementation...
I'm solving a 200x200 Problem with a constraint matrix of 400,40000
the revised simplex method took 16 minutes to solve...IP took about 10 minutes.
this is unbelievable slow?!
Am I doing something wrong or are those old methods really so bad?
If someone knows a good running python implementation of the simplex, IP or dual simplex feel free to share.. I want to see how the perform in comparison to scipy
cheers

Fastest way to compute large amount of fixed points in python?

I have a large amount of one-dimensional nonlinear fixed point problems to solve, what is the most efficient numerical solver? I'm currently using scipy.optimize.fixed_point, it takes around 17s to run 1000 of my tasks. Thanks for any suggestions.

If these are all 1D, you can take the fixed_point source,
https://github.com/scipy/scipy/blob/v1.5.2/scipy/optimize/minpack.py#L876
simplify it (can decide once on the acceleration strategy, no need for _lazywhere etc) and compile it with either cython or numba.

How to implement Python code to solve quadratic program?

I'm a total novice to this and I need some help implementing a solver in Python to optimize the following.
I want to minimize (1/4b)[(Π1-s)'K(Π1-s)+(Π'1-t)'K(Π'1-t)] - tr(KΠ) with respect to Π.
Π is an nxn matrix and 1 denotes the all ones' vector. Also s and t are vectors of dimension n and b is a fixed scalar. So the only quantity that varies is Π, and for that matrix, we have the constraint that all the entries sum up to 1.
How would I do this? Or if this isn't the correct place, where should I ask this?

First you need to express your equation as a python code. Raw python is not that great at pure number crunching, so you should consider a library like Numpy to do the heavy lifting for you.
Once you do that. You can try using one of the optimizers that come with scikit-learn package
If the domain of Π is weird (non continous for example) try using a HyperOpt package

convert a sparse matrix to dense and get the full eigenvalues

recently I'm working on a problem which requires
diagonalizing a huge hermitian matrix to get all the eigenvalues.
Currently I'm using Mathematica to do the job.
However it is not applicable due to the limitation of memory
when the matrix size approaches (2^15,2^15), where the diagonalization costs approximately 32 GBs memory.
I've tried using python by importing the matrix from mathematica,
import numpy as np
from scipy.io import mmread
from scipy.sparse import csc_matrix
#importing sparse matrix to save space
h = mmread("h.mtx")
h = csc_matrix(h)
#diagonlizing the dense one
ev = np.linalg.eigvalsh(h.todense())
It works but unfortunately an order of magnitude slower than Mathematica.
So, is there any other possible solutions, say, C++?
I know nothing about C++ so I guess the simplest way may be importing the
matrix to C++ and diagonalizing.
Thanks!

Running some preliminary test using this matrix:
http://math.nist.gov/MatrixMarket/data/NEP/h2plus/qc2534.html
I determined that the conversion to dense does not take up much of the time. The eigenvalue calculation does.
Numpy uses highly-optimized Lapack routines to calculate. These are the same you'd use in C++. Therefore C++ won't give you much of a speedup. If you want a speedup use the sparseness as a property, go to a better computer or switch to a distributed matrix storage(lot's of labor here).
P.S: if you do this for a university project you might want to look around if your university has a cluster of some sort. A cluster node typically has lots of memory. If not, check amazons AWS EC2 or googles compute engine for instances with lot's of ram.
Edit:
Here Wolfram says what Mathematica does behind the scenes: http://reference.wolfram.com/language/tutorial/LinearAlgebraAppendix.html#83486633
Arpack is a (arnoldi)subspace solver, giving you only the highest or lowest k-eigenvalues, ATLAS is just a Lapack implementation and the rest seems to be for solving linear systems.
All methods giving you the full eigenspectrum will require the matrix decomposition of a NxN matrix. If you only want k vectors there are methods which reduce it to a decomposition of a k x k-matrix.
There are modern alternatives to Arpack(http://slepc.upv.es/ or the one that comes with MKL), but they all give you a subspace.

c++ won't help much.
In python you can delegate easily to C++ and a lot of scipy routines will do just that (for performance). I also expect that if you only time the eigen value line you will get similar performance to Matematica and the difference in performance comes from reading the data.
The best solution is to look for a more appropriate algorithm, maybe something that operates on the sparse matrix directly, or decompose the original into smaller matrices and combine them.
To make the original solution more tractable you could try increasing the amount of swap space. In linux it's a dedicated partition, in windows it's a setting. This should allow Matematica/python to use more memory, but it's going to be much slower due to memory trashing. Get an SSD to speed this setup up, but note that it's going to be destroyed faster due to often writes. Or even better buy more RAM.