I am using the logistic regression function from sklearn, and was wondering what each of the solver is actually doing behind the scenes to solve the optimization problem.
Can someone briefly describe what "newton-cg", "sag", "lbfgs" and "liblinear" are doing?
Well, I hope I'm not too late for the party! Let me first try to establish some intuition before digging into loads of information (warning: this is not a brief comparison, TL;DR)
Introduction
A hypothesis h(x), takes an input and gives us the estimated output value.
This hypothesis can be as simple as a one-variable linear equation, .. up to a very complicated and long multivariate equation with respect to the type of algorithm we’re using (e.g. linear regression, logistic regression..etc).
Our task is to find the best Parameters (a.k.a Thetas or Weights) that give us the least error in predicting the output. We call the function that calculates this error a Cost or Loss Function, and apparently, our goal is to minimize the error in order to get the best-predicted output!
One more thing to recall is, the relation between the parameter value and its effect on the cost function (i.e. the error) looks like a bell curve (i.e. Quadratic; recall this because it’s important).
So if we start at any point in that curve and keep taking the derivative (i.e. tangent line) of each point we stop at (assuming it's a univariate problem, otherwise, if we have multiple features, we take the partial derivative), we will end up at what so-called the Global Optima as shown in this image:
If we take the partial derivative at the minimum cost point (i.e. global optima) we find the slope of the tangent line = 0 (then we know that we reached our target).
That’s valid only if we have a Convex Cost Function, but if we don’t, we may end up stuck at what is called Local Optima; consider this non-convex function:
Now you should have the intuition about the heck relationship between what we are doing and the terms: Derivative, Tangent Line, Cost Function, Hypothesis ..etc.
Side Note: The above-mentioned intuition is also related to the Gradient Descent Algorithm (see later).
Background
Linear Approximation:
Given a function, f(x), we can find its tangent at x=a. The equation of the tangent line L(x) is: L(x)=f(a)+f′(a)(x−a).
Take a look at the following graph of a function and its tangent line:
From this graph we can see that near x=a, the tangent line and the function have nearly the same graph. On occasion, we will use the tangent line, L(x), as an approximation to the function, f(x), near x=a. In these cases, we call the tangent line the "Linear Approximation" to the function at x=a.
Quadratic Approximation:
Same as a linear approximation, yet this time we are dealing with a curve where we cannot find the point near to 0 by using only the tangent line.
Instead, we use the parabola as it's shown in the following graph:
In order to fit a good parabola, both parabola and quadratic function should have the same value, the same first derivative, AND the same second derivative. The formula will be (just out of curiosity): Qa(x) = f(a) + f'(a)(x-a) + f''(a)(x-a)2/2
Now we should be ready to do the comparison in detail.
Comparison between the methods
1. Newton’s Method
Recall the motivation for the gradient descent step at x: we minimize the quadratic function (i.e. Cost Function).
Newton’s method uses in a sense a better quadratic function minimisation.
It's better because it uses the quadratic approximation (i.e. first AND second partial derivatives).
You can imagine it as a twisted Gradient Descent with the Hessian (the Hessian is a square matrix of second-order partial derivatives of order n X n).
Moreover, the geometric interpretation of Newton's method is that at each iteration one approximates f(x) by a quadratic function around xn, and then takes a step towards the maximum/minimum of that quadratic function (in higher dimensions, this may also be a saddle point). Note that if f(x) happens to be a quadratic function, then the exact extremum is found in one step.
Drawbacks:
It’s computationally expensive because of the Hessian Matrix (i.e. second partial derivatives calculations).
It attracts to Saddle Points which are common in multivariable optimization (i.e. a point that its partial derivatives disagree over whether this input should be a maximum or a minimum point!).
2. Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm:
In a nutshell, it is an analogue of Newton’s Method, yet here the Hessian matrix is approximated using updates specified by gradient evaluations (or approximate gradient evaluations). In other words, using estimation to the inverse Hessian matrix.
The term Limited-memory simply means it stores only a few vectors that represent the approximation implicitly.
If I dare say that when the dataset is small, L-BFGS relatively performs the best compared to other methods especially because it saves a lot of memory, however, there are some “serious” drawbacks such that if it is unsafeguarded, it may not converge to anything.
Side note: This solver has become the default solver in sklearn LogisticRegression since version 0.22, replacing LIBLINEAR.
3. A Library for Large Linear Classification:
It’s a linear classification that supports logistic regression and linear support vector machines.
The solver uses a Coordinate Descent (CD) algorithm that solves optimization problems by successively performing approximate minimization along coordinate directions or coordinate hyperplanes.
LIBLINEAR is the winner of the ICML 2008 large-scale learning challenge. It applies automatic parameter selection (a.k.a L1 Regularization) and it’s recommended when you have high dimension dataset (recommended for solving large-scale classification problems)
Drawbacks:
It may get stuck at a non-stationary point (i.e. non-optima) if the level curves of a function are not smooth.
Also cannot run in parallel.
It cannot learn a true multinomial (multiclass) model; instead, the optimization problem is decomposed in a “one-vs-rest” fashion, so separate binary classifiers are trained for all classes.
Side note: According to Scikit Documentation: The “liblinear” solver was the one used by default for historical reasons before version 0.22. Since then, the default use is Limited-memory Broyden–Fletcher–Goldfarb–Shanno Algorithm.
4. Stochastic Average Gradient:
The SAG method optimizes the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method's iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values, the SAG method achieves a faster convergence rate than black-box SG methods.
It is faster than other solvers for large datasets when both the number of samples and the number of features are large.
Drawbacks:
It only supports L2 penalization.
This is not really a drawback, but more like a comparison: although SAG is suitable for large datasets, with a memory cost of O(N), it can be less practical for very large N (as the most recent gradient evaluation for each function needs to be maintained in the memory). This is usually not a problem, but a better option would be SVRG 1, 2 which is unfortunately not implemented in scikit-learn!
5. SAGA:
The SAGA solver is a variant of SAG that also supports the non-smooth penalty L1 option (i.e. L1 Regularization). This is therefore the solver of choice for sparse multinomial logistic regression. It also has a better theoretical convergence compared to SAG.
Drawbacks:
This is not really a drawback, but more like a comparison: SAGA is similar to SAG with regard to memory cost. That's it's suitable for large datasets, yet in edge cases where the dataset is very large, the SVRG 1, 2 would be a better option (unfortunately not implemented in scikit-learn)!
Side note: According to Scikit Documentation: The SAGA solver is often the best choice.
Please note the attributes "Large" and "Small" used in Scikit-Learn and in this comparison are relative. AFAIK, there is no universal unanimous and accurate definition of the dataset boundaries to be considered as "Large", "Too Large", "Small", "Too Small"...etc!
Summary
The following table is taken from Scikit Documentation
Updated Table from the same link above (accessed 02/11/2021):
I would like to add my two cents to the terrific answer given by Yahia
My goal is to establish intuition how to get from full gradient descent method to SG then to SAG and then to SAGA.
On the Stochastic Gradient (SG) methods.
SG takes advantage of the fact that commonly used loss functions can be written as a sum of per-sample loss functions
, where w is the weight vector being optimized.
The gradient vector then is written as a sum of per-sample gradient vectors:
.
E.g. least square error has this form
, where are features of i-th sample and the i-th ground truth value (target, dependent variable).
And the logistic regression loss has this form (in notation 2)
.
SG
The main idea of stochastic gradient that instead of computing the gradient of the whole loss function, we can compute the gradient of , the loss function for a single random sample and descent towards that sample gradient direction instead of full gradient of f(x). This is much faster. The reasoning is that uniformly randomly chosen sample gradient represents an unbiased estimate of the gradient of the whole loss function.
In practice, SG descent has worse convergence rate than full gradient descent where k is the number of iterations. But it has faster convergence in terms of number of flops (simple arithmetic operations) as each iteration requires computation of only one gradient instead of n. It also suffers from high variance (indeed we may not necesserily descent when picking random i, we may as well ascent)
SAG
SAG achieves convergence rate of full gradient descent without making each iteration more expensive in flops compared to SG (if only by a constant).
SAG algorithm minimizing f(w) is straightforward (for dense matrices of features).
At step 0 pick a point (leaving aside how you pick it). Initialize with 0 memory cells for saving gradients of at later steps.
At step k update weights with an average of lagged gradients taken from the memory cells (lagged as they are not updated at every step):
Pick uniformly randomly index from 1..n and update only one single memory cell
It seems that we're computing the whole sum of lagged gradients at each step but the nice part is that we can store the cumulative sum as a variable and make a cheap update to it at every step.
We may rewrite the update step a little
and see that the sum is updated by the amount
However, when we do this descent step we're not anymore going in a direction of an unbiased estimate of the full gradient at step k. We're going in a direction of a reduced variance estimate (in part because we're making a small step) but biased. I think this is an important and beautiful thing to understand so I will cite an excerpt from SAGA paper:
Suppose that we want to use Monte Carlo samples to estimate EX and
that we can compute efficiently EY for another random variable Y that
is highly correlated with X. One variance reduction approach is to use
the following estimator θ as an approximation to EX: θ := α(X − Y) +
EY , for a step size α ∈ [0, 1]. We have that Eθ is a convex
combination of EX and EY : Eθ = αEX + (1 − α)EY . The standard
variance reduction approach uses α = 1 and the estimate is unbiased Eθ
= EX. The variance of θ is: Var(θ) = α^2*[Var(X) + Var(Y ) − 2 Cov(X, Y )], and so if Cov(X, Y ) is big enough, the variance of θ is reduced
compared to X, giving the method its name. By varying α from 0 to 1,
we increase the variance of θ towards its maximum value (which
usually is still smaller than the one for X) while decreasing its bias
towards zero.
So we applied a more or less standard variance reduction approach to get from SG to SAG. The variance reduction constant α is equal to 1/n in SAG algorithm. If Y is the randomly picked , X is the , the update
uses the estimate of full gradient in the form 1/n*(X − Y) + EY
We mentioned that SG suffers from high variance. So we may say that SAG is SG with a clever method of variance reduction applied to it. I don't want to diminish the significance of the findings - picking suitable Y random variable is not simple. Now we can play with variance reduction constants. What if we take the variance reduction constant of 1 and therefore use an unbiased estimate of the full gradient?
SAGA
This is the main idea of SAGA. Take SAG algorithm and apply unbiased estimate of full gradient with variance reduction constant α=1.
The update step gets bigger and becomes
Due to lack of bias the proof of convergence becomes simple and has better constants than in SAG case. It also allows for additional trick allowing for l1 regularization. What I mean is proximal operator.
Proximal gradient descent step in SAGA
If you don't need l1 regularisation you can skip this part as there is whole mathematical theory on proximal operators.
Proximal operator is a generalization of gradient descent in some sense. (Operator is just a function from a vector into a vector. Gradient is an operator for example)
where h(u) is a continous convex function.
In other words it it same as finding minimum of h(u) but also getting penalized for going too far from the initial point v. Proximal operator is a function from to (vector to vector, just like gradient) parametrized by h(x). It is non-expansional (i.e distance between x and y does not get bigger after applying proximal operator to x and y). Its' fixed point () is the solution of the optimization problem. Proximal operator applied iteratively actually converges to its fixed point (although this is generally not true for non-expansive operators, i.e. not true for rotation). So most simple algorithm to find minimum using proximal operator is just applying the operator multiple times . And this is similar to gradient descent in some sense. Here is why:
Suppose a differentiable convex function h and instead of gradient descent update a similar backward Euler update: . This update can be viewed as a proximal operator update , since for proximal operator we need to find minimizing or find such that so
Ok why even consider changing one minimization problem by another (computing proximal operator is a minimization problem inside a minimization problem). The answer is for most common loss functions proximal operator either has a closed form or has efficient appoximation method. Take l1 regularizer. Its proximal operator is called soft-thresholding operator and it has a simple form (I tried to insert it here but failed).
Now back to SAGA.
Assume we minimize g(x) + h(x) where g(x) is a smooth convex function and h(x) is a non-smooth convex function (e.g. l1 regularization) but for which we are able to efficiently compute the proximal operator. So the algorithm could first make a gradient descent step for g to reduce g and then apply the proximal operator of h to the result to reduce h. This is the additional trick in SAGA and it is called proximal gradient descent.
Why SAG and SAGA are well suited for very large dataset
Here I am not sure what sklearn authors meant. Making a guess - very large dataset probably means that the feature matrix is sparse (has many 0).
Now let's consider a linearly-parameterized loss function
. Each sum term has a special form
. is a function of a single variable. Notice that both cross entropy loss and least square loss have this form. By chain rule
So it's evident that the gradient is also sparse.
SAG (and SAGA) apply a clever trick for sparse matrices. The idea is that weight vector does not need to be updated in every index at every step. Update may be skipped for the indices of the weight vector that are sparse in the current randomly chosen sample at the step k.
There are other clever tricks in SAG and SAGA. But if you made it so far I invite you to look at the original papers 1 and 2. They are well written.
Related
I am working on a problem which requires some nonlinear least squares curve fitting.
In this problem, I have regions of the curve which have very small gradients (they simply don't effect the output much); however, it's important that the input curves remain rather smooth, so I'd like to bias the least squares solution to favor curves with lower second derivatives.
I'd like to use scipy.optimize.least_squares for their implementation of Levenberg-Marquardt, but am struggling to regularize. In particular, the problem I'd like to solve is
[problem]
where L is a discretized second difference matrix and r_k are the residuals (f(x) - y)**2 for some function f. In LM typically the update step for such a regularized least squares is performed by solving the linear problem
[linear problem]
where J is the Jacobian of f. According to this manual for MINPACK (the backend of Levenberg-Marquardt in scipy.optimize.least_squares) this is exactly the update step that scipy is performing (once you concatenate the second difference operator onto J and 0s onto the residual vector).
Am I missing somthing here? Any help on getting regularization to work here is much appreciated!
I trained a neural network to do a regression on the sine function and would like to compute the first and second derivative with respect to the input.
I tried using the tf.gradients() function like this (neural_net is an instance of tf.keras.Sequential):
prediction = neural_net(x_value)
dx_f = tf.gradients(prediction, x_value)
dx_dx_f = tf.gradients(dx_f, x_value)
x_value is an array that has the length of the test size.
However, this results in predictions and derivatives. The prediction of the network (blue curve) basically exactly catches the sine function, but I had to divide the first derivative (orange) with a factor of 10 and the second derivative (green) with a factor of 100 in order for it to be in the same order of magnitude. So the, the first derivative looks (after that rescale) ok, but the seond derivative is completely erratic. Since the prediction of the sine function works really well there is clearly something funny going on here.
One possible explanation for what you observed, could be that your function is not derivable two times. It looks as if there are jumps in the 1st derivative around the extrema. If so, the 2nd derivative of the function doesn't really exist and the plot you get higly depends on how the library handles such places.
Consider the following picture of a non-smooth function, that jumps from 0.5 to -0.5 for all x in {1, 2, ....}. It's slope is 1 in all places except when x is an integer. If you'd try to plot it's derivative, you would probably see a straight line at y=1, which can be easily misinterpreted because if someone just looks at this plot, they could think the function is completely linear and starts from -infinity to +infinity.
If your results are produced by a neural net which uses RELU, you can try to do the same with the sigmoid activation function. I suppose you won't see that many spikes with this function.
I don't think you can calculate second order derivatives using tf.gradients. Take a look at tf.hessians (what you really want is the diagonal of the Hessian matrix), e.g. [1].
An alternative is to use tf.GradientTape: [2].
[1] https://github.com/gknilsen/pyhessian
[2] https://www.tensorflow.org/api_docs/python/tf/GradientTape
What you learned was the sinus function and not its derivative : during the training process, you are controlling the error with your cost function that takes into account only the values, but it does not control the slope at all : you could have learned a very noisy function but matching the data points exactly.
If you are just using the data point in your cost function, you have no guarantee about the derivative you've learned. However, with some advanced training technics, you could also learn such a derivative : https://arxiv.org/abs/1706.04859
So as a summary, it is not a code issue but only
a theoritical issue
I have a polynomial function for which I would like to find all local extrema. I can evaluate the polynomial via P(x) and to its derivative via d_P(x).
My first thought was to use minimize_scalar, however this does not seem to be able to take advantage of the fact that I can evaluate the derivative. Alternatively, I can use the more general minimize function and provide the gradient.
Is there a rule of thumb about which method will work better, or is this something where I should test out both methods and see what works better. Since the function I am optimizing is a polynomial (well behaved) I wonder if it really matters so much which I use, but if someone has a more background that would be great.
In particular, P(x) is the (unique) polynomial of degree n which alternatively attains a value of 1 or -1 on a set of n-1 points.
Here is a sample of the P(x) scaled so that P(0)=1. Note that the y axis is plotted on a symlog scale.
Since you have a continuous scalar function, the documentation of minimize_scalar suggests a more discrete optimization approach. Since it doesn't use gradient information you won't have trouble with noise/discontinuities/discreteness in your objective. However, if you use minimize in conjunction with a gradient based method then you will have trouble with convergence for noise/discontinuities/discreteness.
If the objective function is fist order continuous then both minimize and minimize_scalar should yield the same solution for a given bound.
I have the following problem. Let's say we have x_{jk} it is an expression value of gene j in a sample k. It is the average of expression levels across the cell types s_{ij}, weighted by respective proportions a_{ki} (i = 1 ... N,N is the disease type):
Generally this can be expressed as matrix form
What I want to do is to solve this equation
How can it be done using Theano?
You can do this in theano or not do it in theano. The only thing theano can help you with here is the gradient of the euclidean norm, which it can calculate for you, but which is also easy to write by hand. The algorithm to solve the problem needs to be implemented by yourself. Either you write the lagrangian and then solve the dual problem by gradient ascent and projection onto the constraint, or you solve the primal problem directly by gradient descent and projection onto the constraints. You need to program these optimization steps yourself, which is also the case for any other optimization you do in theano.
With the L-BFGS-B minimizer in scipy, is it possible to retrieve the approximate inverse Hessian that's calculated internally?
Having it in the implicit factored form, so that it's possible to compute arbitrary inverse Hessian matrix - vector products, would be fine.
It doesn't appear so. I'm not an expert on these algorithms but it seems that with L-BFGS specifically it is not possible. According to Wikipedia:
Instead of the inverse Hessian H_k, L-BFGS maintains a history of the past m updates of the position x and gradient ∇f(x), where generally the history size m can be small (often m<10). These updates are used to implicitly do operations requiring the H_k-vector product.
However, if you use scipy.fmin_bfgs it does return the approximate (inverse of the) Hessian matrix, at the cost of the greater memory needed to maintain it.