We Keep Coding

Generate linear fit samples using the standard errors from scipy.stats.linregress

I'm using scipy.stats.linregress to fit a group of points as seen in the plot below. The points are the blue circles, the linear fit is the black line and the grey lines are samples taken using the stderr and intercept_stderr values to sample the slope and intercept values using numpy.random.normal (code below).
My question is: given that stderr and intercept_stderr are standard errors and numpy.random.normal expects standard deviations, should I multiply stderr and intercept_stderr by $\sqrt{N}$ when sampling?
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
x = np.array([-4.12078708, -3.89764352, -3.77248038, -3.66125475, -3.56117129,
-3.47019951, -3.3868179 , -3.30985686, -3.2383979 , -3.17170652,
-3.10918616, -3.05034566, -2.99477581, -2.94213208, -2.89212166,
-2.84449361, -2.79903124, -2.75554612, -2.71387343, -2.67386809,
-2.63540181, -2.59836054, -2.56264246, -2.52815628, -2.49481986,
-2.462559 , -2.43130646, -2.40100111, -2.37158722, -2.34301385,
-2.31523428, -2.28820561, -2.2618883 , -2.23624587, -2.21124457,
-2.18685312, -2.16304247, -2.13978561, -2.11705736, -2.09483422,
-2.07309423, -2.05181683, -2.03098275, -2.01057388])
y = np.array([10.54683181, 10.37020828, 10.93819231, 10.1338195 , 10.68036321,
10.48930797, 10.2340761 , 10.52002056, 10.20343913, 10.29089844,
10.36190947, 10.26050936, 10.36528216, 10.41799894, 10.40077834,
10.2513676 , 10.30768792, 10.49377725, 9.73298189, 10.1158334 ,
10.29359023, 10.38660209, 10.30087358, 10.49464606, 10.23305099,
10.34389097, 10.29016557, 10.0865885 , 10.338077 , 10.34950896,
10.15110388, 10.33316701, 10.22837808, 10.3848174 , 10.56872297,
10.24457621, 10.48255182, 10.39029786, 10.0208671 , 10.17400544,
9.82086658, 10.51361151, 10.4376062 , 10.18610696])
res = stats.linregress(x, y)
s_vals = np.random.normal(res.slope, res.stderr, 100)
i_vals = np.random.normal(res.intercept, res.intercept_stderr, 100)
for i in range(100):
plt.plot(x, i_vals[i] + s_vals[i]*x, c='grey', alpha=.1)
plt.scatter(x, y)
plt.plot(x, res.intercept + res.slope*x, c='k')
plt.show()

TL;DR
Indeed, it is an estimate of the standard deviation since standard error means standard deviation of the error of a particular parameter. No, there is no need to "de-normalize" by multiplying with np.sqrt(n). Finally, however, you might want to change the distribution from which you sample the simulated parameters with that of a t-distribution.
Qualitative explanation
No further multiplication (e.g. with np.sqrt(n)) is needed, i.e. the normalization stays in place. Why is that? Intuitively speaking, slope and intercept parameters are, in a sense, summary statistics of a dataset consisting of pairs x and y. They characterize the dataset as a whole rather than a single pair of points like (x_i, y_i). Similar to sampling a summary statistic (e.g. the mean over x), we use a normalized estimator for standard deviation. In case of a regression, the average variability of all datapoints in a dataset impacts the estimated variability of the resulting intercept. The square root of the sample size merely balances the sum of the variability across the datapoints relative to their absolute number.
A more rigorous explanation would concern the variance-covariance matrix of the estimator (β^). In it, the square root of the elements along the diagonal are the standard errors of the elements of the estimator. In particular, the square root of the first element on the diagonal which represents the standard error of the intercept parameter. With a bit of linear algebra, one can establish a connection between each parameter's standard error (in your case, those of intercept and slope) with that of the regression model. Since the standard error of the regression model s is an asymptotically unbiased estimate of the standard deviation of the noise in the data σ, a quantitative rationale can be established requiring no re-scaling of the intercept's standard error.
Regarding the distributions from which you sample/simulate the intercept and slope. Rather than a Normal distribution, the standard errors follow the (Student's) t-distribution. See slide 18. In turn,
s_vals = np.random.standard_t(df=len(x)-2, size=100) * res.stderr + res.slope
i_vals = np.random.standard_t(df=len(x)-2, size=100) * res.intercept_stderr + res.intercept
However, with sample sizes beyond n=30, the realizations will be almost statistically indistinguishable as compares to those sampled from a Gaussian distribution. This is because the t-distribution converges to that of a standard normal distribution rather quickly.
Visual explanation
We can skip the quantitative arguments though. What do we expect from estimators based on datasets? The more data we have the more certainty we have about the fixed but unknown location. In turn, if we increase the size n of the data, the simulated grey lines should move closer together. This happens when we use the standard error as the scale parameter. Increasing the sample size by a factor of 14 brings the grey lines closer. Instead, using the standard error multiplied by np.sqrt(n) leaves the grey lines equally far apart even when the dataset size is drastically increased. In fact, we exactly undid the advantage of a higher sample size by multiplying with the square roots of n.

Reducing difference between two graphs by optimizing more than one variable in MATLAB/Python?

Suppose 'h' is a function of x,y,z and t and it gives us a graph line (t,h) (simulated). At the same time we also have observed graph (observed values of h against t). How can I reduce the difference between observed (t,h) and simulated (t,h) graph by optimizing values of x,y and z? I want to change the simulated graph so that it imitates closer and closer to the observed graph in MATLAB/Python. In literature I have read that people have done same thing by Lavenberg-marquardt algorithm but don't know how to do it?

You are actually trying to fit the parameters x,y,z of the parametrized function h(x,y,z;t).
MATLAB
You're right that in MATLAB you should either use lsqcurvefit of the Optimization toolbox, or fit of the Curve Fitting Toolbox (I prefer the latter).
Looking at the documentation of lsqcurvefit:
x = lsqcurvefit(fun,x0,xdata,ydata);
It says in the documentation that you have a model F(x,xdata) with coefficients x and sample points xdata, and a set of measured values ydata. The function returns the least-squares parameter set x, with which your function is closest to the measured values.
Fitting algorithms usually need starting points, some implementations can choose randomly, in case of lsqcurvefit this is what x0 is for. If you have
h = #(x,y,z,t) ... %// actual function here
t_meas = ... %// actual measured times here
h_meas = ... %// actual measured data here
then in the conventions of lsqcurvefit,
fun <--> #(params,t) h(params(1),params(2),params(3),t)
x0 <--> starting guess for [x,y,z]: [x0,y0,z0]
xdata <--> t_meas
ydata <--> h_meas
Your function h(x,y,z,t) should be vectorized in t, such that for vector input in t the return value is the same size as t. Then the call to lsqcurvefit will give you the optimal set of parameters:
x = lsqcurvefit(#(params,t) h(params(1),params(2),params(3),t),[x0,y0,z0],t_meas,h_meas);
h_fit = h(x(1),x(2),x(3),t_meas); %// best guess from curve fitting
Python
In python, you'd have to use the scipy.optimize module, and something like scipy.optimize.curve_fit in particular. With the above conventions you need something along the lines of this:
import scipy.optimize as opt
popt,pcov = opt.curve_fit(lambda t,x,y,z: h(x,y,z,t), t_meas, y_meas, p0=[x0,y0,z0])
Note that the p0 starting array is optional, but all parameters will be set to 1 if it's missing. The result you need is the popt array, containing the optimal values for [x,y,z]:
x,y,z = popt
h_fit = h(x,y,z,t_meas)

Gaussian fit in Python - parameters estimation

I want to fit an array of data (in the program called "data", of size "n") with a Gaussian function and I want to get the estimations for the parameters of the curve, namely the mean and the sigma. Is the following code, which I found on the Web, a fast way to do that? If so, how can I actually get the estimated values of the parameters?
import pylab as plb
from scipy.optimize import curve_fit
from scipy import asarray as ar,exp
x = ar(range(n))
y = data
n = len(x) #the number of data
mean = sum(x*y)/n #note this correction
sigma = sum(y*(x-mean)**2)/n #note this correction
def gaus(x,a,x0,sigma,c):
return a*exp(-(x-x0)**2/(sigma**2))+c
popt,pcov = curve_fit(gaus,x,y,p0=[1,mean,sigma,0.0])
print popt
print pcov
plt.plot(x,y,'b+:',label='data')
plt.plot(x,gaus(x,*popt),'ro:',label='fit')
plt.legend()
plt.title('Fig. 3 - Fit')
plt.xlabel('q')
plt.ylabel('data')
plt.show()

To answer your first question, "Is the following code, which I found on the Web, a fast way to do that?"
The code that you have is in fact the right way to proceed with fitting your data, when you believe is Gaussian and know the fitting function (except change the return function to
a*exp(-(x-x0)**2/(sigma**2)).
I believe for a Gaussian function you don't need the constant c parameter.
A common use of least-squares minimization is curve fitting, where one has a parametrized model function meant to explain some phenomena and wants to adjust the numerical values for the model to most closely match some data. With scipy, such problems are commonly solved with scipy.optimize.curve_fit.
To answer your second question, "If so, how can I actually get the estimated values of the parameters?"
You can go to the link provided for scipy.optimize.curve_fit and find that the best fit parameters reside in your popt variable. In your example, popt will contain the mean and sigma of your data. In addition to the best fit parameters, pcov will contain the covariance matrix, which will have the errors of your mean and sigma. To obtain 1sigma standard deviations, you can simply use np.sqrt(pcov) and obtain the same.

Python / Scipy - implementing optimize.curve_fit 's sigma into optimize.leastsq

I am fitting data points using a logistic model. As I sometimes have data with a ydata error, I first used curve_fit and its sigma argument to include my individual standard deviations in the fit.
Now I switched to leastsq, because I needed also some Goodness of Fit estimation that curve_fit could not provide. Everything works well, but now I miss the possibility to weigh the least sqares as "sigma" does with curve_fit.
Has someone some code example as to how I could weight the least squares also in leastsq?
Thanks, Woodpicker

I just found that it is possible to combine the best of both worlds, and to have the full leastsq() output also from curve_fit(), using the option full_output:
popt, pcov, infodict, errmsg, ier = curve_fit(func, xdata, ydata, sigma = SD, full_output = True)
This gives me infodict that I can use to calculate all my Goodness of Fit stuff, and lets me use curve_fit's sigma option at the same time...

Assuming your data are in arrays x, y with yerr, and the model is f(p, x), just define the error function to be minimized as (y-f(p,x))/yerr.

The scipy.optimize.curve_fit docs say:
pcov : 2d array
The estimated covariance of popt. The diagonals provide the variance of the parameter estimate. To compute one standard deviation errors on the parameters use perr = np.sqrt(np.diag(pcov)). How the sigma parameter affects the estimated covariance depends on absolute_sigma argument, as described above.
And the section on
absolute_sigma : bool, optional
If True, sigma is used in an absolute sense and the estimated parameter covariance pcov reflects these absolute values.
If False, only the relative magnitudes of the sigma values matter. The returned parameter covariance matrix pcov is based on scaling sigma by a constant factor. This constant is set by demanding that the reduced chisq for the optimal parameters popt when using the scaled sigma equals unity. In other words, sigma is scaled to match the sample variance of the residuals after the fit. Mathematically, pcov(absolute_sigma=False) = pcov(absolute_sigma=True) * chisq(popt)/(M-N)
So, you could just leave absolute_sigma to the default value (False) and then use
perr = np.sqrt(np.diag(pcov))
fitStdErr0 = perr[0]
fitStdErr1 = perr[1]
...
to get the standard deviation error of each fit parameter (as a 1D numpy array). Now you can just pick the useful members (and combine them in a way that is most representative of your data).

numpy.polyfit has no keyword 'cov'

I'm trying to use polyfit to find the best fitting straight line to a set of data, but I also need to know the uncertainty on the parameters, so I want the covariance matrix too. The online documentation suggests I write:
polyfit(x, y, 2, cov=True)
but this gives the error:
TypeError: polyfit() got an unexpected keyword argument 'cov'
And sure enough help(polyfit) shows no keyword argument 'cov'.
So does the online documentation refer to a previous release of numpy? (I have 1.6.1, the newest one). I could write my own polyfit function, but has anyone got any suggestions for why I don't have a covariance option on my polyfit?
Thanks

For a solution that comes from a library, I find that using scikits.statsmodels is a convenient choice. In statsmodels, regression objects have callable attributes that return the parameters and standard errors. I put an example of how this would work for you below:
# Imports, I assume NumPy for forming your data.
import numpy as np
import scikits.statsmodels.api as sm
# Form the data here
(X, Y) = ....
reg_x_data = np.ones(X.shape); # 0th degree term.
for ii in range(1,deg+1):
reg_x_data = np.hstack(( reg_x_data, X**(ii) )); # Append the ii^th degree term.
# Store OLS regression results into `result`
result = sm.OLS(Y,reg_x_data).fit()
# Print the estimated coefficients
print result.params
# Print the basic OLS standard error in the coefficients
print result.bse
# Print the estimated basic OLS covariance matrix
print result.cov_params() # <-- Notice, this one is a function by convention.
# Print the heteroskedasticity-consistent standard error
print result.HC0_se
# Print the heteroskedasticity-consistent covariance matrix
print result.cov_HC0
There are additional robust covariance attributes in the result object as well. You can see them by printing out dir(result). Also, by convention, the covariance matrices for the heteroskedasticity-consistent estimators are only available after you already call the corresponding standard error, such as: you must call result.HC0_se prior to result.cov_HC0 because the first reference causes the second one to be computed and stored.
Pandas is another library that probably provides more advanced support for these operations.
Non-library function
This might be useful when you don't want to / can't rely on an extra library function.
Below is a function that I wrote to return the OLS regression coefficients, as well as a bunch of stuff. It returns the residuals, the regression variance and standard error (standard error of the residuals-squared), the asymptotic formula for large-sample variance, the OLS covariance matrix, the heteroskedasticity-consistent "robust" covariance estimate (which is the OLS covariance but weighted according to the residuals), and the "White" or "bias-corrected" heteroskedasticity-consistent covariance.
import numpy as np
###
# Regression and standard error estimation functions
###
def ols_linreg(X, Y):
""" ols_linreg(X,Y)
Ordinary least squares regression estimator given explanatory variables
matrix X and observations matrix Y.The length of the first dimension of
X and Y must be the same (equal to the number of samples in the data set).
Note: these methods should be adapted if you need to use this for large data.
This is mostly for illustrating what to do for calculating the different
classicial standard errors. You would never really want to compute the inverse
matrices for large problems.
This was developed with NumPy 1.5.1.
"""
(N, K) = X.shape
t1 = np.linalg.inv( (np.transpose(X)).dot(X) )
t2 = (np.transpose(X)).dot(Y)
beta = t1.dot(t2)
residuals = Y - X.dot(beta)
sig_hat = (1.0/(N-K))*np.sum(residuals**2)
sig_hat_asymptotic_variance = 2*sig_hat**2/N
conv_st_err = np.sqrt(sig_hat)
sum1 = 0.0
for ii in range(N):
sum1 = sum1 + np.outer(X[ii,:],X[ii,:])
sum1 = (1.0/N)*sum1
ols_cov = (sig_hat/N)*np.linalg.inv(sum1)
PX = X.dot( np.linalg.inv(np.transpose(X).dot(X)).dot(np.transpose(X)) )
robust_se_mat1 = np.linalg.inv(np.transpose(X).dot(X))
robust_se_mat2 = np.transpose(X).dot(np.diag(residuals[:,0]**(2.0)).dot(X))
robust_se_mat3 = np.transpose(X).dot(np.diag(residuals[:,0]**(2.0)/(1.0-np.diag(PX))).dot(X))
v_robust = robust_se_mat1.dot(robust_se_mat2.dot(robust_se_mat1))
v_modified_robust = robust_se_mat1.dot(robust_se_mat3.dot(robust_se_mat1))
""" Returns:
beta -- The vector of coefficient estimates, ordered on the columns on X.
residuals -- The vector of residuals, Y - X.beta
sig_hat -- The sample variance of the residuals.
conv_st_error -- The 'standard error of the regression', sqrt(sig_hat).
sig_hat_asymptotic_variance -- The analytic formula for the large sample variance
ols_cov -- The covariance matrix under the basic OLS assumptions.
v_robust -- The "robust" covariance matrix, weighted to account for the residuals and heteroskedasticity.
v_modified_robust -- The bias-corrected and heteroskedasticity-consistent covariance matrix.
"""
return beta, residuals, sig_hat, conv_st_err, sig_hat_asymptotic_variance, ols_cov, v_robust, v_modified_robust
For your problem, you would use it like this:
import numpy as np
# Define or load your data:
(Y, X) = ....
# Desired polynomial degree
deg = 2;
reg_x_data = np.ones(X.shape); # 0th degree term.
for ii in range(1,deg+1):
reg_x_data = np.hstack(( reg_x_data, X**(ii) )); # Append the ii^th degree term.
# Get all of the regression data.
beta, residuals, sig_hat, conv_st_error, sig_hat_asymptotic_variance, ols_cov, v_robust, v_modified_robust = ols_linreg(reg_x_data,Y)
# Print the covariance matrix:
print ols_cov
If you spot any bugs in my computations (especially the heteroskedasticity-consistent estimators) please let me know and I'll fix it asap.