I am trying to use the scipy.linalg package but without the overhead. The funtions I currently use are scipy.linalg.cho_factor and scipy.linalg.cho_solve for cholesky decomposition. I have to use scipy.linalg.lapack or scipy.linalg.cython_lapack (or even blas), especially for the latter function which is being called thousands of times.
The problem is I don't know where to start, since there are tons of functions and their names are encoded e.g. cbbcsd, cbdsqr etc. The question is: how do you find the right function?
One option is to look inside the Python/Cython code for scipy.linalg.cho_factor and see what it uses. For cho_factor it looks like it might be ?potrf. The question mark is replaced with a letter depending on the data types.
More generally, if you know which operation you want to apply, the names are constructed pretty systematically. See http://www.netlib.org/lapack/lug_old/node26.html and http://www.netlib.org/lapack/lug_old/node26.html for example. Typically you have to know the data-type (single precision (i.e. float), double, complex, etc...), the symmetry (if any) of the matrices and then the operation.
BLAS is largely just multiplications of various combinations of matrices and vectors and some solutions of systems of linear equations, while Lapack is anything more advanced (decompositions, eigenvalues, etc).
Related
Is there any place with a brief description of each of the algorithms for the parameter method in the minimize function of the lmfit package? Both there and in the documentation of SciPy there is no explanation about the details of each algorithm. Right now I know I can choose between them but I don't know which one to choose...
My current problem
I am using lmfit in Python to minimize a function. I want to minimize the function within a finite and predefined range where the function has the following characteristics:
It is almost zero everywhere, which makes it to be numerically identical to zero almost everywhere.
It has a very, very sharp peak in some point.
The peak can be anywhere within the region.
This makes many minimization algorithms to not work. Right now I am using a combination of the brute force method (method="brute") to find a point close to the peak and then feed this value to the Nelder-Mead algorithm (method="nelder") to finally perform the minimization. It is working approximately 50 % of the times, and the other 50 % of the times it fails to find the minimum. I wonder if there are better algorithms for cases like this one...
I think it is a fair point that docs for lmfit (such as https://lmfit.github.io/lmfit-py/fitting.html#fit-methods-table) and scipy.optimize (such as https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html#optimization-scipy-optimize) do not give detailed mathematical descriptions of the algorithms.
Then again, most of the docs for scipy, numpy, and related libraries describe how to use the methods, but do not describe in much mathematical detail how the algorithms work.
In fairness, the different optimization algorithms share many features and the differences between them can get pretty technical. All of these methods try to minimize some metric (often called "cost" or "residual") by changing the values of parameters for the supplied function.
It sort of takes a text book (or at least a Wikipedia page) to establish the concepts and mathematical terms used for these methods, and then a paper (or at least a Wikipedia page) to describe how each method differs from the others. So, I think the basic answer would be to look up the different methods.
I am preconditioning a matrix using spilu, however, to pass this preconditioner into cg (the built in conjugate gradient method) it is necessary to use the LinearOperator function, can someone explain to me the parameter matvec, and why I need to use it. Below is my current code
Ainv=scla.spilu(A,drop_tol= 1e-7)
Ainv=scla.LinearOperator(Ainv.shape,matvec=Ainv)
scla.cg(A,b,maxiter=maxIterations, M = Ainv)
However this doesnt work and I am given the error TypeError: 'SuperLU' object is not callable. I have played around and tried
Ainv=scla.LinearOperator(Ainv.shape,matvec=Ainv.solve)
instead. This seems to work but I want to know why matvec needs Ainv.solve rather than just Ainv, and is it the right thing to feed LinearOperator?
Thanks for your time
Without having much experience with this part of scipy, some comments:
According to the docs you don't have to use LinearOperator, but you might do
M : {sparse matrix, dense matrix, LinearOperator}, so you can use explicit matrices too!
The idea/advantage of the LinearOperator:
Many iterative methods (e.g. cg, gmres) do not need to know the individual entries of a matrix to solve a linear system A*x=b. Such solvers only require the computation of matrix vector products docs
Depending on the task, sometimes even matrix-free approaches are available which can be much more efficient
The working approach you presented is indeed the correct one (some other source doing it similarily, and some course-materials doing it like that)
The idea of not using the inverse matrix, but using solve() here is not to form the inverse explicitly (which might be very costly)
A similar idea is very common in BFGS-based optimization algorithms although wiki might not give much insight here
scipy has an extra LinearOperator for this not forming the inverse explicitly! (although i think it's only used for statistics / completing/finishing some optimization; but i successfully build some LBFGS-based optimizers with this one)
Source # scicomp.stackexchange discussing this without touching scipy
And because of that i would assume spilu is completely going for this too (returning an object with a solve-method)
In lapack there are much more functions than in "lapack interface" in scipy. Is there any reason behind this and is there OS-independentway to call lapack functions directly?
I realize that I may call dinamic library directly, but this means writing my own wrapper and this is not what I want.
To make a real usecase, I need to call dsbgv to solve generalized eigenproblem for banded matrix. It is orders of magnitude faster than using eig which is for general matrix.
scipy.linalg.lapack is an organically grown (tm) set of wrappers, added by different people with different goals, needs, motivations and time budgets over quite a few years.
cython_lapack is a complete set of wrappers for a certain (old enough) version of LAPACK. It is lower level however: you need to supply all lapack arguments, ensure the correct array ordering, alignment etc.
I've searched quite a bit, but I've only found homegrown reimplementations of Strassen matrix multiplication.
Wikipedia says that numpy uses BLAS (which includes a high-performance implementations of sub-cubic matrix multiplication algorithms, e.g. Strassen's method), but I couldn't find if numpy matrix multiplication has some sort of size check and then chooses either naive $O(n^3)$ multiplication or some more sophisticated approach based on the size of the matrix (this is similar to choosing the leaf size or recursion limit in Strassen sub-calls).
I also tried just plotting the log of the runtime vs. the log of the problem size, but looking for a subtle change in slope is non-trivial (because of cache effects, etc. as the problems get larger).
Since the documentation for numpy matrix didn't have any mention of Strassen (or alternative sub-cubic algorithm) or the runtime, and since numpy source in question is in C++ for performance (the C++ code in turn uses the BLAS library), it isn't so easy to tell from the source code, so I thought I would ask:
Does anyone know about the algorithm or big-oh runtime of a numpy.matrix(...) * numpy.matrix(...) call?
Several SciPy functions are documented as taking a "condensed distance matrix as returned by scipy.spatial.distance.pdist". Now, inspection shows that what pdist returns is the row-major 1D-array form of the upper off-diagonal part of the distance matrix. This is all well and good, and natural and obvious, but is it documented or defined anywhere? I'd rather not assume anything about a data structure that'll suddenly change. (Granted, there isn't a lot of things it could change to, but I guess one possibility would be to wrap the array in an object that allows matrix-like indexing.)
Honestly, this is a better question for the scipy users or dev list, as it's about future plans for scipy.
However, the structure is fairly rigorously documented in the docstrings for both scipy.spatial.pdist and in scipy.spatial.squareform.
E.g. for pdist:
Returns a condensed distance matrix Y. For
each :math:`i` and :math:`j` (where :math:`i<j<n`), the
metric ``dist(u=X[i], v=X[j])`` is computed and stored in the
:math:`ij`th entry.
See ``squareform`` for information on how to calculate the index of
this entry or to convert the condensed distance matrix to a
redundant square matrix.
Becuase of this, and the fact that so many other functions in scipy.spatial expect a distance matrix in this form, I'd seriously doubt it's going to change without a number of depreciation warnings and announcements.
Modules in scipy itself (as opposed to scipy's scikits) are fairly stable, and there's a great deal of consideration put into backwards compatibility when changes are made (and because of this, there's quite a bit of legacy "cruft" in scipy: e.g. the fact that the core scipy module is just numpy with different defaults on a couple of functions.).