I am using scikit-learn's linear_model.LogisticRegression to perform multinomial logistic regress. I would like to initialize the solver's seed value, i.e. I want to give the solver its initial guess as the coefficients' values.
Does anyone know how to do that? I have looked online and sifted through the code too, but haven't found an answer.
Thanks!
You can use the warm_start option (with solver not liblinear), and manually set coef_ and intercept_ prior to fitting.
warm_start : bool, default=False
When set to True, reuse the solution of the previous call to fit as initialization, otherwise, just erase the previous solution. Useless for liblinear solver.
Related
I have a simple keras model (normal Lasso linear model) where the inputs are moved to a single 'neuron' Dense(1, kernel_regularizer=l1(fdr))(input_layer) but the weights from this model are never set exactly to zero. I find this interesting since scikit-learn's Lasso can set coefficients exactly to zero.
I have used Adam and tensorflow's FtrlOptimizer for optimisation and they have the same problem.
I've already checked this question already but this does not explain why sklearn can set values exactly to zero, no to mention how their models converge in ~500ms on my server when the same model in Keras takes 2.4secs with early terminations.
Is this all because of the optimizer being used or am I missing something?
Is this all because of the optimizer being used or am I missing
something?
Indeed. If you look into the actual function that gets called when you fit Lasso from scikit-learn (it's called from ElasticNet class) you see that it uses different optimization algorithm.
Coordinate Descent in scikit-learn's ElasticNet starts with coefficient vector equal to zero, and then considers adding nonzero entries one at a time (this is related to stepwise feature selection for linear regression).
Other methods that are used to optimize L1 regularized regression also are work in that way: for example LARS (Least-angle regression) can be also used from scikit-learn.
In contrast to that, a paper on FTRL algorithm says
Unfortunately, OGD is not particularly effective at producing
sparse models. In fact, simply adding a subgradient
of the L1 penalty to the gradient of the loss (Ow`t(w))
will essentially never produce coefficients that are exactly
zero.
fit method of LogisticRegression has a optional sample_weight parameter. I follow the python code and find it just does some trivial things and dispatches to underlying solvers (e.g. liblinear). How sample_weight works? does it work thought oversampling or some other method?
update
as #Alexander McFarlane said, it isn't immediately obvious that the sample weight in the decision tree is the same as the logistic regression unless you look at the source.
I am trying out logistic regression from scratch in python.(through finding probability estimates,cost function,applying gradient descent for increasing the maximum likelihood).But I have a confusion regarding which estimates should I take for the first iteration process.I took all the estimates as 0(including the intercept).But the results are different from that we get in Scikit-learn.I want to know which are the initial estimates taken in Scikit-learn for logistic regression?
First of all scikit learn's LogsiticRegerssion uses regularization. So unless you apply that too , it is unlikely you will get exactly the same estimates. if you really want to test your method versus scikit's , it is better to use their gradient decent implementation of Logistic regersion which is called SGDClassifier . Make certain you put loss='log' for logistic regression and set alpha=0 to remove regularization, but again you will need to adjust the iterations and eta as their implementation is likely to be slightly different than yours.
To answer specifically about the initial estimates, I don't think it matters, but most commonly you set everything to 0 (including the intercept) and should converge just fine.
Also bear in mind GD (gradient Decent) models are hard to tune sometimes and you may need to apply some scaling(like StandardScaler) to your data beforehand as very high values are very likely to drive your gradient out of its slope. Scikit's implementation adjusts for that.
I am trying to find an optimal parameter set for an XGB_Classifier using GridSearchCV.
Since my data is very unbalanced, both fitting and scoring (in cross_validation) must be performed using weights, therefore I have to use a custom scorer, which takes a 'weights' vector as a parameter.
However, I can't find a way to have GridSearchCV pass 'weights' vector to a scorer.
There were some attempts to add this functionality to gridsearch:
https://github.com/ndawe/scikit-learn/commit/3da7fb708e67dd27d7ef26b40d29447b7dc565d7
But they were not merged into master and now I am afraid that this code is not compatible with upstream changes.
Has anyone faced a similar problem and is there any 'easy' way to cope with it?
You could manually balance your training dataset as in the answer to Scikit-learn balanced subsampling
I'm trying to find a way to fit a linear regression model with positive coefficients.
The only way I found is sklearn's Lasso model, which has a positive=True argument, but doesn't recommend using with alpha=0 (means no other constraints on the weights).
Do you know of another model/method/way to do it?
IIUC, this is a problem which can be solved by the scipy.optimize.nnls, which can do non-negative least squares.
Solve argmin_x || Ax - b ||_2 for x>=0.
In your case, b is the y, A is the X, and x is the β (coefficients), but, otherwise, it's the same, no?
Many functions can keep linear regression model with positive coefficients.
scipy.optimize.nnls can solve above problem.
scikit-learn LinearRegression can set the parameter positive=True to solve this. And, the sklearn also uses the scipy.optimize.nnls. Interestingly, you can learn how to write multiple targets outputs in source code.
Additionally, if you want to solve linear least squares with bounds on the variables. You can see lsq_linear
.
As of version 0.24, scikit-learn LinearRegression includes a similar argument positive, which does exactly that; from the docs:
positive : bool, default=False
When set to True, forces the coefficients to be positive. This option is only supported for dense arrays.
New in version 0.24.