I want to get the marginal effects of a logistic regression from a sklearn model
I know you can get these for a statsmodel logistic regression using '.get_margeff()'. Is there nothing for sklearn? I want to avoid doing the calculation my self as I feel there would be a lot of room for error.
import statsmodels.formula.api as sm
from statsmodels.tools.tools import add_constant
from sklearn.datasets import load_breast_cancer
import pandas as pd
import numpy as np
data = load_breast_cancer()
x = data.data
y= data.target
x=add_constant(x,has_constant='add')
model = sm.Logit(y, x).fit_regularized()
margeff = model.get_margeff(dummy=True,count=True)
##print the margal effect
print(margeff.margeff)
>> [ 6.73582136e-02 2.15779589e-04 1.28857837e-02 -1.06718136e-03
-1.96032750e+00 1.36137385e+00 -1.16303369e+00 -1.37422595e+00
8.14539021e-01 -1.95330095e+00 -4.86235558e-01 4.84260993e-02
7.16675627e-02 -2.89644712e-03 -5.18982198e+00 -5.93269894e-01
3.22934080e+00 -1.28363008e+01 3.07823155e+00 5.84122170e+00
1.92785670e-02 -9.86284081e-03 -7.53298463e-03 -3.52349287e-04
9.13527446e-01 1.69938656e-01 -2.89245493e-01 -4.65659522e-01
-8.32713335e-01 -1.15567833e+00]
# manual calculation, doing this as you can get the coef_ from a sklearn model and use in the function
def PDF(XB):
var1 = np.exp(XB)
var2 = np.power((1+np.exp(XB)),2)
var3 = (var1 / var2)
return var3
arrPDF = PDF(np.dot(x,model.params))
ME=pd.DataFrame(np.dot(arrPDF[:,None],model.params[None,:]))
print(ME.iloc[:,1:].mean().to_list())
>>
[0.06735821358791198, 0.0002157795887363032, 0.012885783711597246, -0.0010671813611730326, -1.9603274961356965, 1.361373851981879, -1.1630336876543224, -1.3742259536619654, 0.8145390210646809, -1.9533009514684947, -0.48623555805230195, 0.04842609927469917, 0.07166756271689229, -0.0028964471200298475, -5.189821981601878, -0.5932698935239838, 3.229340802910038, -12.836300822253634, 3.0782315528664834, 5.8412217033605245, 0.019278567008384557, -0.009862840813512401, -0.007532984627259091, -0.0003523492868714151, 0.9135274456151128, 0.16993865598225097, -0.2892454926120402, -0.46565952159093893, -0.8327133347971125, -1.1556783345783221]
the custom function gives the same as ".get_margeff()" but there might be a lot of room for error when using the sklearn ceof_ in the custom function above.
Is there some method/function/Attribute in sklearn that can give me
the marginal effects
If there is not, is there another library get from the ceof_ and
data to the marginal effects
if the answer to both the above is no, are there any circumstances
in which the custom function will not work (e.g. with a particular solver or
penalty in sklearn)
I just hit this demand a few days ago.
My supervisor gave me this information that I want to share. Hope this can help you.
partial_dependence: This method can get the partial dependence or marginal effects you meant.
plot_partial_dependence: This method can plot the partial dependence.
Here is the sample code from the API Reference.
scikit-learn version: 0.21.2
from sklearn.inspection import plot_partial_dependence, partial_dependence
from sklearn.datasets import make_friedman1
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import GradientBoostingRegressor
%matplotlib inline
X, y = make_friedman1()
# case1: linear model
lm = LinearRegression().fit(X, y)
# plot the partial dependence
plot_partial_dependence(lm, X, [0, (0, 1)])
# get the partial dependence
partial_dependence(lm, X, [0])
# case2: classifier
clf = GradientBoostingRegressor(n_estimators=10).fit(X, y)
# plot the partial dependence
plot_partial_dependence(clf, X, [0, (0, 1)])
# get the partial dependence
partial_dependence(clf, X, [0])
Related
I'm attempting to perform a lasso regression for a larger than main memory dataset by using Dask, but there doesn't seem to be a cleanly documented way to do so.
I did previously find a somewhat related question but no actual answer.
Looking into how scikit sets up the Lasso regression, I thought I could set it up the same way. For example, here is one approach I tried
from dask_ml.datasets import make_regression
import dask_glm.families
import dask_glm.regularizers
import dask_glm.algorithms
import pandas as pd
# dask dataframes
X, y = make_regression(n_samples=1000, chunks=100)
# pandas dataframes
df_X = X.compute()
df_y = y.compute()
family = dask_glm.families.Normal()
regularizer = dask_glm.regularizers.ElasticNet(weight=1)
b = dask_glm.algorithms.gradient_descent(X=X, y=y, max_iter=100000, family=family, regularizer=regularizer, alpha=0.01, normalize=False, fit_intercept=False)
print(b)
reg = linear_model.Lasso(alpha=0.01, fit_intercept=False)
reg.fit(df_X, df_y)
print(reg.coef_)
However, the coefficients don't match up at all, and the dask version's coefficients seem more unstable than scikit's.
Here's another approach I tried, this time based on a comment from this GH issue
from dask_ml.datasets import make_regression
from dask_glm.regularizers import L1
from dask_glm.estimators import LinearRegression
X, y = make_regression(n_samples=1000, chunks=100)
lr = LinearRegression(regularizer=L1())
lr.fit(X, y)
print(lr.coef_)
Again, the coefficients seem very unstable.
Ideally there would already be an implementation of Lasso using Dask for this, but I can't seem to find much on the internet except for running LassoCV with dask as the backend to joblib, which is a little different than I want.
I am trying to run a regression without predictor, just constant and error term. The model is y = a + error.
I have tried as follows:
import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
y = np.random.normal(size=50)
sm.OLS(y, sm.add_constant(), missing='drop').fit()
However, this does not work.
As denoted here, using regression without predictors is not a major data analysis tool. Logistic regression is not a classifier, while this was already discussed as "Linear vs. Logistic Regression on Classification Problems" and "Regression for Binary Classification". However, it can still be a requirement for any reason like the one you pointed in this question. Thus, we will try to provide a proper solution as an answer to your case.
Similar to this question, you can use DummyRegressor from Sklearn as follows:
import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt
from sklearn.dummy import DummyRegressor
X = np.random.normal(size=50)
y = np.random.normal(size=50)
dummy_regr = DummyRegressor(strategy="mean")
dummy_regr.fit(X, y)
...
I would like to perform a simple linear regression using statsmodels and I've tried several different methods by now but I just don't get it to work. The code that I have constructed now doesn't give me any errors but it also doesn't show me the result
I am trying to create a model for the variable "Direction" which takes the value 0 if the return for the corresponding date was negative and 1 if it was positive. The explinatory variables are the (5) lags of the returns. The df13 contains the lags and also the direction for each observed date. I tried this code and as I mentioned it doesn't give an error but says " Optimization terminated successfully.
Current function value: 0.682314
Iterations 5
However, I would like to see the typical table with all the beta values, their significance etc.
Also, what would you say, since Direction is a binary variable may it be better to use a logit instead of a linear model? However, in the assignment it appeared as a linear model.
And lastly, I am sorry its not displayed here correctly but I don't know how to write as code or insert my dataframe
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
import os
import itertools
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import statsmodels.api as sm
import matplotlib.pyplot as plt
from statsmodels.sandbox.regression.predstd import wls_prediction_std
...
X = df13[['Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5']]
Y = df13['Direction']
X = sm.add_constant(X)
model = sm.Logit(Y.astype(float), X.astype(float)).fit()
predictions = model.predict(X)
print_model = model.summary
print(print_model)
Edit: I'm sure it has to be a logit regression so I updated that part
I don't know if this is unintentional, but it looks like you need to define X and Y separately:
X = df13[['Lag1', 'Lag2', 'Lag3', 'Lag4', 'Lag5']]
Y = df13['Direction']
Secondly, I'm not familiar with statsmodel, but I would try converting your dataframes to numpy arrays. You can do this with
Xnum = X.to_numpy()
ynum = y.to_numpy()
And try passing those to the regressors.
I am new to machine learning.
I created a data, random numbers in two sets. I am trying how to find a sample, however when doing following, I receive very low accuracy score:
from random import randint as R
from matplotlib import pyplot as plt
import numpy as np
from sklearn.neighbors import KNeighborsClassifier as KNC
from sklearn.cross_validation import train_test_split as tts
from sklearn.metrics import accuracy_score
a = [R(100,200) for x in range(100)]
b = [R(1000,2000) for x in range(100)]
c = a+b
X = np.array(c).reshape(len(c),1)
y = np.arange(len(c))
train_X, test_X, train_y,test_y = tts(X,y,test_size=0.4)
mimi = KNC()
mimi.fit(train_X, train_y)
y__pred = mimi.predict(train_X)
print(accuracy_score(train_y,y__pred))
print(mimi.score(train_X,train_y))
I receive a result of 0.18... What exactly does this mean? That a prediction score is just 18%? Please, can you explain to me in most simple way. I would really appreciate it.
By doing y = np.arange(len(c)) you have c different classes (here 200 classes) with only one example for each class. Learning the nearest neighbors on such a setup does not have any sense.
What you want (If I'm guessing right) is to have one class for data a and another class for data b.
Change y to:
y = np.concatenate([[0] *len(a), [1] *len(b)])
You'll see that you obtain an accuracy score of 1.0, which means that you successfully classify all your testing example.
Can anyone explain to me the difference between ols in statsmodel.formula.api versus ols in statsmodel.api?
Using the Advertising data from the ISLR text, I ran an ols using both, and got different results. I then compared with scikit-learn's LinearRegression.
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
import statsmodels.api as sm
from sklearn.linear_model import LinearRegression
df = pd.read_csv("C:\...\Advertising.csv")
x1 = df.loc[:,['TV']]
y1 = df.loc[:,['Sales']]
print "Statsmodel.Formula.Api Method"
model1 = smf.ols(formula='Sales ~ TV', data=df).fit()
print model1.params
print "\nStatsmodel.Api Method"
model2 = sm.OLS(y1, x1)
results = model2.fit()
print results.params
print "\nSci-Kit Learn Method"
model3 = LinearRegression()
model3.fit(x1, y1)
print model3.coef_
print model3.intercept_
The output is as follows:
Statsmodel.Formula.Api Method
Intercept 7.032594
TV 0.047537
dtype: float64
Statsmodel.Api Method
TV 0.08325
dtype: float64
Sci-Kit Learn Method
[[ 0.04753664]]
[ 7.03259355]
The statsmodel.api method returns a different parameter for TV from the statsmodel.formula.api and the scikit-learn methods.
What kind of ols algorithm is statsmodel.api running that would produce a different result? Does anyone have a link to documentation that could help answer this question?
Came across this issue today and wanted to elaborate on #stellasia's answer because the statsmodels documentation is perhaps a bit ambiguous.
Unless you are using actual R-style string-formulas when instantiating OLS, you need to add a constant (literally a column of 1s) under both statsmodels.formulas.api and plain statsmodels.api. #Chetan is using R-style formatting here (formula='Sales ~ TV'), so he will not run into this subtlety, but for people with some Python knowledge but no R background this could be very confusing.
Furthermore it doesn't matter whether you specify the hasconst parameter when building the model. (Which is kind of silly.) In other words, unless you are using R-style string formulas, hasconst is ignored even though it is supposed to
[Indicate] whether the RHS includes a user-supplied constant
because, in the footnotes
No constant is added by the model unless you are using formulas.
The example below shows that both .formulas.api and .api will require a user-added column vector of 1s if not using R-style string formulas.
# Generate some relational data
np.random.seed(123)
nobs = 25
x = np.random.random((nobs, 2))
x_with_ones = sm.add_constant(x, prepend=False)
beta = [.1, .5, 1]
e = np.random.random(nobs)
y = np.dot(x_with_ones, beta) + e
Now throw x and y into Excel and run Data>Data Analysis>Regression, making sure "Constant is zero" is unchecked. You'll get the following coefficients:
Intercept 1.497761024
X Variable 1 0.012073045
X Variable 2 0.623936056
Now, try running this regression on x, not x_with_ones, in either statsmodels.formula.api or statsmodels.api with hasconst set to None, True, or False. You'll see that in each of those 6 scenarios, there is no intercept returned. (There are only 2 parameters.)
import statsmodels.formula.api as smf
import statsmodels.api as sm
print('smf models')
print('-' * 10)
for hc in [None, True, False]:
model = smf.OLS(endog=y, exog=x, hasconst=hc).fit()
print(model.params)
# smf models
# ----------
# [ 1.46852293 1.8558273 ]
# [ 1.46852293 1.8558273 ]
# [ 1.46852293 1.8558273 ]
Now running things correctly with a column vector of 1.0s added to x. You can use smf here but it's really not necessary if you're not using formulas.
print('sm models')
print('-' * 10)
for hc in [None, True, False]:
model = sm.OLS(endog=y, exog=x_with_ones, hasconst=hc).fit()
print(model.params)
# sm models
# ----------
# [ 0.01207304 0.62393606 1.49776102]
# [ 0.01207304 0.62393606 1.49776102]
# [ 0.01207304 0.62393606 1.49776102]
The difference is due to the presence of intercept or not:
in statsmodels.formula.api, similarly to the R approach, a constant is automatically added to your data and an intercept in fitted
in statsmodels.api, you have to add a constant yourself (see the documentation here). Try using add_constant from statsmodels.api
x1 = sm.add_constant(x1)
I had a similar issue with the Logit function.
(I used patsy to create my matrices, so the intercept was there.)
My sm.logit was not converging.
My sm.formula.logit was converging however.
Data going in was exactly the same.
I changed the solver method to 'newton' and the sm.logit converged also.
Is it possible the two versions have different default solver methods.