The documentation of scipy.sparse.linalg.eigs reads
The number of eigenvalues and eigenvectors desired. k must be smaller than N-1. It is not possible to compute all eigenvectors of a matrix.
Why can it not compute all eigenvalues? This is possible for standard (non-sparse) matrices.
The docs also say:
This function is a wrapper to the ARPACK SNEUPD, DNEUPD, CNEUPD, ZNEUPD, functions which use the Implicitly Restarted Arnoldi Method to find the eigenvalues and eigenvectors.
Digging into the ARPACK docs, the description of the algorithm says:
The algorithm is capable of computing a few (k) eigenvalues with user specified features such as largest real part or largest magnitude using storage.
So this is a limitation of the algorithm, as explained by the Wikipedia:
The Arnoldi method belongs to a class of linear algebra algorithms that give a partial result after a small number of iterations, in contrast to so-called direct methods which must complete to give any useful results.
I work with Lie group SO(3). I know there exists an analytical expression for tangent application
from which the derivative of matrix exponential can be simply computed as
What I would like to do is to compute T(w).
The problem with simply implementing function T myself is that it is numerically imprecise for small norms of vector omega. I know there are exponential scipy.linalg.expm and logarithmic maps scipy.linalg.logm already in SciPy. Since this is similar I wonder if there is also the tangent map or the derivative. I noticed the scipy.linalg.expm_frechet, however, I didn't know how to connect it to T.
This does not directly answer your question but for
it is numerically imprecise for small norms of vector omega
you can just use Taylor series expansion for small |\omega| ie use something like
(1-cos(x))/x^2 = 1/2 - x^2/24, if |x| < eps, normal expression otherwise
for suitably small eps (say 1e-4)
and the same idea for (x-sin(x))/x^3
For numeric evaluation of the derivatives you can translate your code to use pytorch, and then autograd will do the work for you a.
For symbolic differentiation you can use sympy b
How can one generate random sparse orthogonal matrix?
I know there is a sparse matrices in scipy library but they are generally non-orthogonal. One can exploit QR-factorization, but it is not necessarily preserves sparsity.
As a preliminary thought you could partition the matrix into diagonal blocks, fill those blocks with QR and then permute rows/columns. The resulting matrices will remain orthagonal. Alternatively, you could define some sparsity pattern for Q and try to minimize f(Q, xi) subject to QQ^T=I where f is some (preferably) convex function that adds entropy through the random variable xi. Can't say anything about the efficacy of either method since I haven't actually tried them.
EDIT: A bit more about the second method. f can really be any function. One choice might be similarity of the non-zero elements to a random gaussian vector (or any other random variate): f = ||vec(Q) - x||_2^2, x ~ N(0, sigma * I). You could handle this using any general constrained optimizer. The problem of course, is that not every pattern S is guaranteed to have a (full rank) orthogonal filling. If you have the memory, L1 regularization (or a smooth approximation) could encourage sparsity in a dense matrix variable: g(Q) = f(Q) + P(Q) where P is any sparsity-inducing penalty function. Check out Wen & Yen (2010) "A feasible Method for Optimization with Orthogonality Constraints" for an algorithm specifically designed for optimization of general (differentiable) functions over (dense) orthogonal matrices and Liu, Wu, So (2015) "Quadratic Optimization with Orthogonality Constraints" for more theorical evaluation of several line/arc search algorithms for quadratic functions. If memory is a problem, you could generate each row/column separately using sparse basis pursuit, for which there are many algorithms depending on the nature of your problem. See Qu, Sun and Wright (2015) "Finding a sparse vector in a subspace: linear sparsity using alternate directions" and Bian et al (2015) "Sparse null space basis pursuit and analysis dictionary learning for high-dimensional data analysis" for algorithm details, though in both cases you will have to incorporate/replace constraints to promote orthogonality to all previous vectors.
It's also worth noting there are sparse QR algorithms that return Q as the product of sparse/structured matrices. If you are concerned about storage space alone, this might be the simplest method to create large, efficient orthogonal operators.
I'm currently using the curve_fit function of the scipy.optimize package in Python, and know that if you take the square root of the diagonal entries of the covariance matrix that you get from curve_fit, you get the standard deviation on the parameters that curve_fit calculated. What I'm not sure about, is what exactly this standard deviation means. It's an approximation using a Hesse matrix as far as I understand, but what would the exact calculation be? Standard deviation on the Gaussian Bell Curve tells you what percentage of area is within a certain range of the curve, so I assumed for curve_fit it tells you how many datapoints are between certain parameter values, but apparently that isn't right...
I'm sorry if this should be basic knowledge for curve fitting, but I really can't figure out what the standard deviations do, they express an error on the parameters, but those parameters are calculated as the best possible fit for the function, it's not like there's a whole collection of optimal parameters, and we get the average value of that collection and consequently also have a standard deviation. There's only one optimal value, what is there to compare it with? I guess my question really comes down to this: how can I manually and accurately calculate these standard deviations, and not just get an approximation using a Hesse matrix?
The variance in the fitted parameters represents the uncertainty in the best-fit value based on the quality of the fit of the model to the data. That is, it describes by how much the value could change away from the best-fit value and still have a fit that is almost as good as the best-fit value.
With standard definition of chi-square,
chi_square = ( ( (data - model)/epsilon )**2 ).sum()
and reduced_chi_square = chi_square / (ndata - nvarys) (where data is the array of the data values, model the array of the calculated model, epsilon is uncertainty in the data, ndata is the number of data points, and nvarys the number of variables), a good fit should have reduced_chi_square around 1 or chi_square around ndata-nvary. (Note: not 0 -- the fit will not be perfect as there is noise in the data).
The variance in the best-fit value for a variable gives the amount by which you can change the value (and re-optimize all other values) and increase chi-square by 1. That gives the so-called '1-sigma' value of the uncertainty.
As you say, these values are expressed in the diagonal terms of the covariance matrix returned by scipy.optimize.curve_fit (the off-diagonal terms give the correlations between variables: if a value for one variable is changed away from its optimal value, how would the others respond to make the fit better). This covariance matrix is built using the trial values and derivatives near the solution as the fit is being done -- it calculates the "curvature" of the parameter space (ie, how much chi-square changes when a variables value changes).
You can calculate these uncertainties by hand. The lmfit library (https://lmfit.github.io/lmfit-py/) has routines to more explicitly explore the confidence intervals of variables from least-squares minimization or curve-fitting. These are described in more detail at
https://lmfit.github.io/lmfit-py/confidence.html. It's probably easiest to use lmfit for the curve-fitting rather than trying to re-implement the confidence interval code for curve_fit.
I am using the logistic regression function from sklearn, and was wondering what each of the solver is actually doing behind the scenes to solve the optimization problem.
Can someone briefly describe what "newton-cg", "sag", "lbfgs" and "liblinear" are doing?
Well, I hope I'm not too late for the party! Let me first try to establish some intuition before digging into loads of information (warning: this is not a brief comparison, TL;DR)
Introduction
A hypothesis h(x), takes an input and gives us the estimated output value.
This hypothesis can be as simple as a one-variable linear equation, .. up to a very complicated and long multivariate equation with respect to the type of algorithm we’re using (e.g. linear regression, logistic regression..etc).
Our task is to find the best Parameters (a.k.a Thetas or Weights) that give us the least error in predicting the output. We call the function that calculates this error a Cost or Loss Function, and apparently, our goal is to minimize the error in order to get the best-predicted output!
One more thing to recall is, the relation between the parameter value and its effect on the cost function (i.e. the error) looks like a bell curve (i.e. Quadratic; recall this because it’s important).
So if we start at any point in that curve and keep taking the derivative (i.e. tangent line) of each point we stop at (assuming it's a univariate problem, otherwise, if we have multiple features, we take the partial derivative), we will end up at what so-called the Global Optima as shown in this image:
If we take the partial derivative at the minimum cost point (i.e. global optima) we find the slope of the tangent line = 0 (then we know that we reached our target).
That’s valid only if we have a Convex Cost Function, but if we don’t, we may end up stuck at what is called Local Optima; consider this non-convex function:
Now you should have the intuition about the heck relationship between what we are doing and the terms: Derivative, Tangent Line, Cost Function, Hypothesis ..etc.
Side Note: The above-mentioned intuition is also related to the Gradient Descent Algorithm (see later).
Background
Linear Approximation:
Given a function, f(x), we can find its tangent at x=a. The equation of the tangent line L(x) is: L(x)=f(a)+f′(a)(x−a).
Take a look at the following graph of a function and its tangent line:
From this graph we can see that near x=a, the tangent line and the function have nearly the same graph. On occasion, we will use the tangent line, L(x), as an approximation to the function, f(x), near x=a. In these cases, we call the tangent line the "Linear Approximation" to the function at x=a.
Quadratic Approximation:
Same as a linear approximation, yet this time we are dealing with a curve where we cannot find the point near to 0 by using only the tangent line.
Instead, we use the parabola as it's shown in the following graph:
In order to fit a good parabola, both parabola and quadratic function should have the same value, the same first derivative, AND the same second derivative. The formula will be (just out of curiosity): Qa(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)2/2
Now we should be ready to do the comparison in detail.
Comparison between the methods
1. Newton’s Method
Recall the motivation for the gradient descent step at x: we minimize the quadratic function (i.e. Cost Function).
Newton’s method uses in a sense a better quadratic function minimisation.
It's better because it uses the quadratic approximation (i.e. first AND second partial derivatives).
You can imagine it as a twisted Gradient Descent with the Hessian (the Hessian is a square matrix of second-order partial derivatives of order n X n).
Moreover, the geometric interpretation of Newton's method is that at each iteration one approximates f(x) by a quadratic function around xn, and then takes a step towards the maximum/minimum of that quadratic function (in higher dimensions, this may also be a saddle point). Note that if f(x) happens to be a quadratic function, then the exact extremum is found in one step.
Drawbacks:
It’s computationally expensive because of the Hessian Matrix (i.e. second partial derivatives calculations).
It attracts to Saddle Points which are common in multivariable optimization (i.e. a point that its partial derivatives disagree over whether this input should be a maximum or a minimum point!).
2. Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm:
In a nutshell, it is an analogue of Newton’s Method, yet here the Hessian matrix is approximated using updates specified by gradient evaluations (or approximate gradient evaluations). In other words, using estimation to the inverse Hessian matrix.
The term Limited-memory simply means it stores only a few vectors that represent the approximation implicitly.
If I dare say that when the dataset is small, L-BFGS relatively performs the best compared to other methods especially because it saves a lot of memory, however, there are some “serious” drawbacks such that if it is unsafeguarded, it may not converge to anything.
Side note: This solver has become the default solver in sklearn LogisticRegression since version 0.22, replacing LIBLINEAR.
3. A Library for Large Linear Classification:
It’s a linear classification that supports logistic regression and linear support vector machines.
The solver uses a Coordinate Descent (CD) algorithm that solves optimization problems by successively performing approximate minimization along coordinate directions or coordinate hyperplanes.
LIBLINEAR is the winner of the ICML 2008 large-scale learning challenge. It applies automatic parameter selection (a.k.a L1 Regularization) and it’s recommended when you have high dimension dataset (recommended for solving large-scale classification problems)
Drawbacks:
It may get stuck at a non-stationary point (i.e. non-optima) if the level curves of a function are not smooth.
Also cannot run in parallel.
It cannot learn a true multinomial (multiclass) model; instead, the optimization problem is decomposed in a “one-vs-rest” fashion, so separate binary classifiers are trained for all classes.
Side note: According to Scikit Documentation: The “liblinear” solver was the one used by default for historical reasons before version 0.22. Since then, the default use is Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm.
4. Stochastic Average Gradient:
The SAG method optimizes the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values, the SAG method achieves a faster convergence rate than black-box SG methods.
It is faster than other solvers for large datasets when both the number of samples and the number of features are large.
Drawbacks:
It only supports L2 penalization.
This is not really a drawback, but more like a comparison: although SAG is suitable for large datasets, with a memory cost of O(N), it can be less practical for very large N (as the most recent gradient evaluation for each function needs to be maintained in the memory). This is usually not a problem, but a better option would be SVRG 1, 2 which is unfortunately not implemented in scikit-learn!
5. SAGA:
The SAGA solver is a variant of SAG that also supports the non-smooth penalty L1 option (i.e. L1 Regularization). This is therefore the solver of choice for sparse multinomial logistic regression. It also has a better theoretical convergence compared to SAG.
Drawbacks:
This is not really a drawback, but more like a comparison: SAGA is similar to SAG with regard to memory cost. That's it's suitable for large datasets, yet in edge cases where the dataset is very large, the SVRG 1, 2 would be a better option (unfortunately not implemented in scikit-learn)!
Side note: According to Scikit Documentation: The SAGA solver is often the best choice.
Please note the attributes "Large" and "Small" used in Scikit-Learn and in this comparison are relative. AFAIK, there is no universal unanimous and accurate definition of the dataset boundaries to be considered as "Large", "Too Large", "Small", "Too Small"...etc!
Summary
The following table is taken from Scikit Documentation
Updated Table from the same link above (accessed 02/11/2021):
I would like to add my two cents to the terrific answer given by Yahia
My goal is to establish intuition how to get from full gradient descent method to SG then to SAG and then to SAGA.
On the Stochastic Gradient (SG) methods.
SG takes advantage of the fact that commonly used loss functions can be written as a sum of per-sample loss functions
, where w is the weight vector being optimized.
The gradient vector then is written as a sum of per-sample gradient vectors:
.
E.g. least square error has this form
, where are features of i-th sample and the i-th ground truth value (target, dependent variable).
And the logistic regression loss has this form (in notation 2)
.
SG
The main idea of stochastic gradient that instead of computing the gradient of the whole loss function, we can compute the gradient of , the loss function for a single random sample and descent towards that sample gradient direction instead of full gradient of f(x). This is much faster. The reasoning is that uniformly randomly chosen sample gradient represents an unbiased estimate of the gradient of the whole loss function.
In practice, SG descent has worse convergence rate than full gradient descent where k is the number of iterations. But it has faster convergence in terms of number of flops (simple arithmetic operations) as each iteration requires computation of only one gradient instead of n. It also suffers from high variance (indeed we may not necesserily descent when picking random i, we may as well ascent)
SAG
SAG achieves convergence rate of full gradient descent without making each iteration more expensive in flops compared to SG (if only by a constant).
SAG algorithm minimizing f(w) is straightforward (for dense matrices of features).
At step 0 pick a point (leaving aside how you pick it). Initialize with 0 memory cells for saving gradients of at later steps.
At step k update weights with an average of lagged gradients taken from the memory cells (lagged as they are not updated at every step):
Pick uniformly randomly index from 1..n and update only one single memory cell
It seems that we're computing the whole sum of lagged gradients at each step but the nice part is that we can store the cumulative sum as a variable and make a cheap update to it at every step.
We may rewrite the update step a little
and see that the sum is updated by the amount
However, when we do this descent step we're not anymore going in a direction of an unbiased estimate of the full gradient at step k. We're going in a direction of a reduced variance estimate (in part because we're making a small step) but biased. I think this is an important and beautiful thing to understand so I will cite an excerpt from SAGA paper:
Suppose that we want to use Monte Carlo samples to estimate EX and
that we can compute efficiently EY for another random variable Y that
is highly correlated with X. One variance reduction approach is to use
the following estimator θ as an approximation to EX: θ := α(X − Y) +
EY , for a step size α ∈ [0, 1]. We have that Eθ is a convex
combination of EX and EY : Eθ = αEX + (1 − α)EY . The standard
variance reduction approach uses α = 1 and the estimate is unbiased Eθ
= EX. The variance of θ is: Var(θ) = α^2*[Var(X) + Var(Y ) − 2 Cov(X, Y )], and so if Cov(X, Y ) is big enough, the variance of θ is reduced
compared to X, giving the method its name. By varying α from 0 to 1,
we increase the variance of θ towards its maximum value (which
usually is still smaller than the one for X) while decreasing its bias
towards zero.
So we applied a more or less standard variance reduction approach to get from SG to SAG. The variance reduction constant α is equal to 1/n in SAG algorithm. If Y is the randomly picked , X is the , the update
uses the estimate of full gradient in the form 1/n*(X − Y) + EY
We mentioned that SG suffers from high variance. So we may say that SAG is SG with a clever method of variance reduction applied to it. I don't want to diminish the significance of the findings - picking suitable Y random variable is not simple. Now we can play with variance reduction constants. What if we take the variance reduction constant of 1 and therefore use an unbiased estimate of the full gradient?
SAGA
This is the main idea of SAGA. Take SAG algorithm and apply unbiased estimate of full gradient with variance reduction constant α=1.
The update step gets bigger and becomes
Due to lack of bias the proof of convergence becomes simple and has better constants than in SAG case. It also allows for additional trick allowing for l1 regularization. What I mean is proximal operator.
Proximal gradient descent step in SAGA
If you don't need l1 regularisation you can skip this part as there is whole mathematical theory on proximal operators.
Proximal operator is a generalization of gradient descent in some sense. (Operator is just a function from a vector into a vector. Gradient is an operator for example)
where h(u) is a continous convex function.
In other words it it same as finding minimum of h(u) but also getting penalized for going too far from the initial point v. Proximal operator is a function from to (vector to vector, just like gradient) parametrized by h(x). It is non-expansional (i.e distance between x and y does not get bigger after applying proximal operator to x and y). Its' fixed point () is the solution of the optimization problem. Proximal operator applied iteratively actually converges to its fixed point (although this is generally not true for non-expansive operators, i.e. not true for rotation). So most simple algorithm to find minimum using proximal operator is just applying the operator multiple times . And this is similar to gradient descent in some sense. Here is why:
Suppose a differentiable convex function h and instead of gradient descent update a similar backward Euler update: . This update can be viewed as a proximal operator update , since for proximal operator we need to find minimizing or find such that so
Ok why even consider changing one minimization problem by another (computing proximal operator is a minimization problem inside a minimization problem). The answer is for most common loss functions proximal operator either has a closed form or has efficient appoximation method. Take l1 regularizer. Its proximal operator is called soft-thresholding operator and it has a simple form (I tried to insert it here but failed).
Now back to SAGA.
Assume we minimize g(x) + h(x) where g(x) is a smooth convex function and h(x) is a non-smooth convex function (e.g. l1 regularization) but for which we are able to efficiently compute the proximal operator. So the algorithm could first make a gradient descent step for g to reduce g and then apply the proximal operator of h to the result to reduce h. This is the additional trick in SAGA and it is called proximal gradient descent.
Why SAG and SAGA are well suited for very large dataset
Here I am not sure what sklearn authors meant. Making a guess - very large dataset probably means that the feature matrix is sparse (has many 0).
Now let's consider a linearly-parameterized loss function
. Each sum term has a special form
. is a function of a single variable. Notice that both cross entropy loss and least square loss have this form. By chain rule
So it's evident that the gradient is also sparse.
SAG (and SAGA) apply a clever trick for sparse matrices. The idea is that weight vector does not need to be updated in every index at every step. Update may be skipped for the indices of the weight vector that are sparse in the current randomly chosen sample at the step k.
There are other clever tricks in SAG and SAGA. But if you made it so far I invite you to look at the original papers 1 and 2. They are well written.