Sorry for the noob question...here's my code:
from __future__ import division
import sklearn
import numpy as np
from scipy import stats
from sklearn.linear_model import LinearRegression
import matplotlib.pyplot as plt
X =np.array([6,8,10,14,18])
Y = np.array([7,9,13,17.5,18])
X = np.reshape(X,(1,5))
Y = np.reshape(Y,(1,5))
print X
print Y
plt.figure()
plt.title('Pizza Price as a function of Pizza Diameter')
plt.xlabel('Pizza Diameter (Inches)')
plt.ylabel('Pizza Price (Dollars)')
axis = plt.axis([0, 25, 0 ,25])
m, b = np.polyfit(X,Y,1)
plt.grid(True)
plt.plot(X,Y, 'k.')
plt.plot(X, m*X + b, '-')
#plt.show()
#training data
#x= [[6],[8],[10],[14],[18]]
#y= [[7],[9],[13],[17.5],[18]]
# create and fit linear regression model
model = LinearRegression()
model.fit(X,Y)
print 'A 12" pizza should cost $% .2f' % model.predict(19)
#work out cost function, which is residual sum of squares
print 'Residual sum of squares: %.2f' % np.mean((model.predict(x)- y) ** 2)
#work out variance (AKA Mean squared error)
xMean = np.mean(x)
print 'Variance is: %.2f' %np.var([x], ddof=1)
#work out covariance (this is whether the x axis data and y axis data correlate with eachother)
#When a and b are 1-dimensional sequences, numpy.cov(x,y)[0][1] calculates covariance
print 'Covariance is: %.2f' %np.cov(X, Y, ddof = 1)[0][1]
#test the model on new test data, printing the r squared coefficient
X_test = [[8], [9], [11], [16], [12]]
y_test = [[11], [8.5], [15], [18], [11]]
print 'R squared for model on test data is: %.2f' %model.score(X_test,y_test)
Basically, some of these functions work for the variables I have called X and Y and some don't.
For example, as the code is, it throws up this error:
TypeError: expected 1D vector for x
for the line
m, b = np.polyfit(X,Y,1)
However, when I comment out the two lines reshaping the variables like this:
#X = np.reshape(X,(1,5))
#Y = np.reshape(Y,(1,5))
I get the error:
ValueError: Found input variables with inconsistent numbers of samples: [1, 5]
on the line
model.fit(X,Y)
So, how do I get the array to work for all the functions in my script, without having different arrays of the same data with slightly different structures?
Thanks for your help!
Change these lines
X = np.reshape(X,(5))
Y = np.reshape(Y,(5))
or just removed them both
Related
I want to fit a function to the independant (X) and dependent (y) variables:
import numpy as np
y = np.array([1.45952016, 1.36947283, 1.31433227, 1.24076599, 1.20577963,
1.14454815, 1.13068077, 1.09638278, 1.08121406, 1.04417094,
1.02251471, 1.01268524, 0.98535659, 0.97400591])
X = np.array([4.571428571362048, 8.771428571548313, 12.404761904850602, 17.904761904850602,
22.904761904850602, 31.238095237873495, 37.95833333302289,
44.67857142863795, 51.39880952378735, 64.83928571408615,
71.5595238097012, 85., 98.55357142863795, 112.1071428572759])
I already tried scipy package in this way:
from scipy.optimize import curve_fit
def func (x, a, b, c):
return 1/(a*(x**2) + b*(x**1) + c)
g = [1, 1, 1]
c, cov = curve_fit (func, X.flatten(), y.flatten(), g)
test_ar = np.arange(min(X), max(X), 0.25)
pred = np.empty(len(test_ar))
for i in range (len(test_ar)):
pred[i] = func(test_ar[i], c[0], c[1], c[2])
I can add higher orders of polynomial to make my func more accurate but I want to keep it simple. I very much appreciate if anyone an give me some help on how to find another function or make my prediction better. The figure also shows the result of the prediction:
First thing you want to do is to specify how do you measure "accuracy" which in your case is not an appropriate term at all.
What are you essentially doing is called linear regression. Suitable metrics in this case are mean squared error (MSE), root mean squared error (RMSE), mean absolute error (MAE). It is up to you to decide which metric to use and what threshold to set for being "acceptable".
The image that you are showing above (where you've fitted the line) looks fine BUT please expand your X-axis from -100 to 300 and show us the image again this is a problem with high degree polynomials.
This is a 101 example how to use regression in scikit-learn. In your case if you want to use x^2 or x^3 for predicting y, you just need to add them in to the data ... Currently your X variable is an array (a vector) you need to expand that to become a matrix where each column is a feature (x, x^2, x^3 ...)
here is some code:
import pandas as pd
from sklearn import linear_model
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error, r2_score
y = [1.45952016, 1.36947283, 1.31433227, 1.24076599,
1.20577963, 1.14454815, 1.13068077, 1.09638278,
1.08121406, 1.04417094, 1.02251471, 1.01268524, 0.98535659,
0.97400591]
x = [4.571428571362048, 8.771428571548313, 12.404761904850602,
17.904761904850602, 22.904761904850602, 31.238095237873495,
37.95833333302289, 44.67857142863795, 51.39880952378735,
64.83928571408615, 71.5595238097012, 85., 98.55357142863795, 112.1071428572759]
df = pd.DataFrame({
'x' : x,
'x^2': [i**2 for i in x],
'x^3': [i**3 for i in x],
'y': y
})
X = df[['x','x^2','x^3']]
y = df['y']
model = linear_model.LinearRegression()
model.fit(X, y)
y1 = model.predict(X)
coef = model.coef_
intercept = model.intercept_
you can see the coefficients from the coef variable:
array([-1.67456732e-02, 2.03899728e-04, -8.70976426e-07])
you can see the intercept from the intercept variable:
1.5042389677980577
which in your case means -> y1 = -1.67e-2x +2.03e-4x^2 -8.70e-7x^3 + 1.5
I am trying to fit a line to the 9.0 to 10.0 um regime of my data set. Here is my plot:
Unfortunately, it's a scatter plot with the x values not being indexed from small numbers to large numbers so I can't just apply the optimize.curve_fit function to a specific range of indices to get the desired range in x values.
Below is my go-to procedure for curve fitting. How would I modify it to only get a fit for the 9.0 to 10.0 um x-value range (in my case, the x_dist variable) which has points scattered randomly throughout the indices?
def func(x,a,b): # Define your fitting function
return a*x+b
initialguess = [-14.0, 0.05] # initial guess for the parameters of the function func
fit, covariance = optimize.curve_fit( # call to the fitting routine curve_fit. Returns optimal values of the fit parameters, and their estimated variance
func, # function to fit
x_dist, # data for independant variable
xdiff_norm, # data for dependant variable
initialguess, # initial guess of fit parameters
) # uncertainty in dependant variable
print("linear coefficient:",fit[0],"+-",np.sqrt(covariance[0][0])) #print value and one std deviation of first fit parameter
print("offset coefficient:",fit[1],"+-",np.sqrt(covariance[1][1])) #print value and one std deviation of second fit parameter
print(covariance)
You correctly identified that the problem arises because your x-value data are not ordered. You can address this problem differently. One way is to use Boolean masks to filter out the unwanted values. I tried to be as close as possible to your example:
from matplotlib import pyplot as plt
import numpy as np
from scipy import optimize
#fake data generation
np.random.seed(1234)
arr = np.linspace(0, 15, 100).reshape(2, 50)
arr[1, :] = np.random.random(50)
arr[1, 20:45] += 2 * arr[0, 20:45] -5
rng = np.random.default_rng()
rng.shuffle(arr, axis = 1)
x_dist = arr[0, :]
xdiff_norm = arr[1, :]
def func(x, a, b):
return a * x + b
initialguess = [5, 3]
mask = (x_dist>2.5) & (x_dist<6.6)
fit, covariance = optimize.curve_fit(
func,
x_dist[mask],
xdiff_norm[mask],
initialguess)
plt.scatter(x_dist, xdiff_norm, label="data")
x_fit = np.linspace(x_dist[mask].min(), x_dist[mask].max(), 100)
y_fit = func(x_fit, *fit)
plt.plot(x_fit, y_fit, c="red", label="fit")
plt.legend()
plt.show()
Sample output:
This approach does not modify x_dist and xdiff_norm which might or might not be a good thing for further data evaluation. If you wanted to use a line plot instead of a scatter plot, it might be rather useful to sort your arrays in advance (try a line plot with the above method to see why):
from matplotlib import pyplot as plt
import numpy as np
from scipy import optimize
#fake data generation
np.random.seed(1234)
arr = np.linspace(0, 15, 100).reshape(2, 50)
arr[1, :] = np.random.random(50)
arr[1, 20:45] += 2 * arr[0, 20:45] -5
rng = np.random.default_rng()
rng.shuffle(arr, axis = 1)
x_dist = arr[0, :]
xdiff_norm = arr[1, :]
def func(x, a, b):
return a * x + b
#find indexes of a sorted x_dist array, then sort both arrays based on this index
ind = x_dist.argsort()
x_dist = x_dist[ind]
xdiff_norm = xdiff_norm[ind]
#identify index where linear range starts for normal array indexing
start = np.argmax(x_dist>2.5)
stop = np.argmax(x_dist>6.6)
initialguess = [5, 3]
fit, covariance = optimize.curve_fit(
func,
x_dist[start:stop],
xdiff_norm[start:stop],
initialguess)
plt.plot(x_dist, xdiff_norm, label="data")
x_fit = np.linspace(x_dist[start], x_dist[stop], 100)
y_fit = func(x_fit, *fit)
plt.plot(x_fit, y_fit, c="red", ls="--", label="fit")
plt.legend()
plt.show()
Sample output (unsurprisingly not much different):
I have a very simple case of 3 Datapoints and I would like to do a linear fit y=a0 + a1x through those points using np.polyfit or scipy.stats.linregress.
For the further error propagation I need the errors in the slope and the intercept. I am by far no expert in statistics but on the scipy side I am only aware of the stderr which does not split in slope and intercept.
Polyfit has the possibly to estimate the covariance matrix, but this does not work with only 3 datapoints.
When using qtiplot for example it yields errors for slope and intercept.
B (y-intercept) = 9,291335740072202e-12 +/- 2,391260092282606e-13
A (slope) = 2,527075812274368e-12 +/- 6,878180102259077e-13
What would be the appropiate way to calculate these in python?
EDIT:
np.polyfit(x, y, 1, cov=True)
results in
ValueError: the number of data points must exceed order + 2 for
Bayesian estimate the covariance matrix
scipy.stats.linregress gives you slope, intercept, correleation coefficient, p value & standard error. The fitted line does not have errors associated with its slope or intercept, the errors are to do with the distances of the points from the line. Have a read through this to clear up the point
An example...
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
points = np.array([[1, 3], [2, 4], [2, 7]])
slope, intercept, r_value, p_value, std_err = stats.linregress(points)
print("slope = ", slope)
print("intercept = ", intercept)
print("R = ", r_value)
print("p = ", p_value)
print("Standard error = ", std_err)
for xy in points:
plt.plot(xy[0], xy[1], 'ob')
x = np.linspace(0, 10, 100)
y = slope * x + intercept
plt.plot(x, y, '-r')
plt.grid()
plt.show()
Following the recommendations in this answer I have used several combination of values for beta0, and as shown here, the values from polyfit.
This example is UPDATED in order to show the effect of relative scales of values of X versus Y (X range is 0.1 to 100 times Y):
from random import random, seed
from scipy import polyfit
from scipy import odr
import numpy as np
from matplotlib import pyplot as plt
seed(1)
X = np.array([random() for i in range(1000)])
Y = np.array([i + random()**2 for i in range(1000)])
for num in range(1, 5):
plt.subplot(2, 2, num)
plt.title('X range is %.1f times Y' % (float(100 / max(X))))
X *= 10
z = np.polyfit(X, Y, 1)
plt.plot(X, Y, 'k.', alpha=0.1)
# Fit using odr
def f(B, X):
return B[0]*X + B[1]
linear = odr.Model(f)
mydata = odr.RealData(X, Y)
myodr = odr.ODR(mydata, linear, beta0=z)
myodr.set_job(fit_type=0)
myoutput = myodr.run()
a, b = myoutput.beta
sa, sb = myoutput.sd_beta
xp = np.linspace(plt.xlim()[0], plt.xlim()[1], 1000)
yp = a*xp+b
plt.plot(xp, yp, label='ODR')
yp2 = z[0]*xp+z[1]
plt.plot(xp, yp2, label='polyfit')
plt.legend()
plt.ylim(-1000, 2000)
plt.show()
It seems that no combination of beta0 helps... The only way to get polyfit and ODR fit similar is to swap X and Y, OR as shown here to increase the range of values of X with regard to Y, still not really a solution :)
=== EDIT ===
I do not want ODR to be the same as polyfit. I am showing polyfit just to emphasize that the ODR fit is wrong and it is not a problem of the data.
=== SOLUTION ===
thanks to #norok2 answer when Y range is 0.001 to 100000 times X:
from random import random, seed
from scipy import polyfit
from scipy import odr
import numpy as np
from matplotlib import pyplot as plt
seed(1)
X = np.array([random() / 1000 for i in range(1000)])
Y = np.array([i + random()**2 for i in range(1000)])
plt.figure(figsize=(12, 12))
for num in range(1, 10):
plt.subplot(3, 3, num)
plt.title('Y range is %.1f times X' % (float(100 / max(X))))
X *= 10
z = np.polyfit(X, Y, 1)
plt.plot(X, Y, 'k.', alpha=0.1)
# Fit using odr
def f(B, X):
return B[0]*X + B[1]
linear = odr.Model(f)
mydata = odr.RealData(X, Y,
sy=min(1/np.var(Y), 1/np.var(X))) # here the trick!! :)
myodr = odr.ODR(mydata, linear, beta0=z)
myodr.set_job(fit_type=0)
myoutput = myodr.run()
a, b = myoutput.beta
sa, sb = myoutput.sd_beta
xp = np.linspace(plt.xlim()[0], plt.xlim()[1], 1000)
yp = a*xp+b
plt.plot(xp, yp, label='ODR')
yp2 = z[0]*xp+z[1]
plt.plot(xp, yp2, label='polyfit')
plt.legend()
plt.ylim(-1000, 2000)
plt.show()
The key difference between polyfit() and the Orthogonal Distance Regression (ODR) fit is that polyfit works under the assumption that the error on x is negligible. If this assumption is violated, like it is in your data, you cannot expect the two methods to produce similar results.
In particular, ODR() is very sensitive to the errors you specify.
If you do not specify any error/weighting, it will assign a value of 1 for both x and y, meaning that any scale difference between x and y will affect the results (the so-called numerical conditioning).
On the contrary, polyfit(), before computing the fit, applies some sort of pre-whitening to the data (see around line 577 of its source code) for better numerical conditioning.
Therefore, if you want ODR() to match polyfit(), you could simply fine-tune the error on Y to change your numerical conditioning.
I tested that this works for any numerical conditioning between 1e-10 and 1e10 of your Y (it is / 10. or 1e-1 in your example).
mydata = odr.RealData(X, Y)
# equivalent to: odr.RealData(X, Y, sx=1, sy=1)
to:
mydata = odr.RealData(X, Y, sx=1, sy=1/np.var(Y))
(EDIT: note there was a typo on the line above)
I tested that this works for any numerical conditioning between 1e-10 and 1e10 of your Y (it is / 10. or 1e-1 in your example).
Note that this would only make sense for well-conditioned fits.
I cannot format source code in a comment, and so place it here. This code uses ODR to calculate fit statistics, note the line that has "parameter order for odr" such that I use a wrapper function for the ODR call to my "actual" function.
from scipy.optimize import curve_fit
import numpy as np
import scipy.odr
import scipy.stats
x = np.array([5.357, 5.797, 5.936, 6.161, 6.697, 6.731, 6.775, 8.442, 9.861])
y = np.array([0.376, 0.874, 1.049, 1.327, 2.054, 2.077, 2.138, 4.744, 7.104])
def f(x,b0,b1):
return b0 + (b1 * x)
def f_wrapper_for_odr(beta, x): # parameter order for odr
return f(x, *beta)
parameters, cov= curve_fit(f, x, y)
model = scipy.odr.odrpack.Model(f_wrapper_for_odr)
data = scipy.odr.odrpack.Data(x,y)
myodr = scipy.odr.odrpack.ODR(data, model, beta0=parameters, maxit=0)
myodr.set_job(fit_type=2)
parameterStatistics = myodr.run()
df_e = len(x) - len(parameters) # degrees of freedom, error
cov_beta = parameterStatistics.cov_beta # parameter covariance matrix from ODR
sd_beta = parameterStatistics.sd_beta * parameterStatistics.sd_beta
ci = []
t_df = scipy.stats.t.ppf(0.975, df_e)
ci = []
for i in range(len(parameters)):
ci.append([parameters[i] - t_df * parameterStatistics.sd_beta[i], parameters[i] + t_df * parameterStatistics.sd_beta[i]])
tstat_beta = parameters / parameterStatistics.sd_beta # coeff t-statistics
pstat_beta = (1.0 - scipy.stats.t.cdf(np.abs(tstat_beta), df_e)) * 2.0 # coef. p-values
for i in range(len(parameters)):
print('parameter:', parameters[i])
print(' conf interval:', ci[i][0], ci[i][1])
print(' tstat:', tstat_beta[i])
print(' pstat:', pstat_beta[i])
print()
I am trying to do multiple variables linear regression. But I find that the sklearn.linear_model working very weird. Here's my code:
import numpy as np
from sklearn import linear_model
b = np.array([3,5,7]).transpose() ## the right answer I am expecting
x = np.array([[1,6,9], ## 1*3 + 6*5 + 7*9 = 96
[2,7,7], ## 2*3 + 7*5 + 7*7 = 90
[3,4,5]]) ## 3*3 + 4*5 + 5*7 = 64
y = np.array([96,90,64]).transpose()
clf = linear_model.LinearRegression()
clf.fit([[1,6,9],
[2,7,7],
[3,4,5]], [96,90,64])
print clf.coef_ ## <== it gives me [-2.2 5 4.4] NOT [3, 5, 7]
print np.dot(x, clf.coef_) ## <== it gives me [ 67.4 61.4 35.4]
In order to find your initial coefficients back you need to use the keyword fit_intercept=False when construction the linear regression.
import numpy as np
from sklearn import linear_model
b = np.array([3,5,7])
x = np.array([[1,6,9],
[2,7,7],
[3,4,5]])
y = np.array([96,90,64])
clf = linear_model.LinearRegression(fit_intercept=False)
clf.fit(x, y)
print clf.coef_
print np.dot(x, clf.coef_)
Using fit_intercept=False prevents the LinearRegression object from working with x - x.mean(axis=0), which it would otherwise do (and capture the mean using a constant offset y = xb + c) - or equivalently by adding a column of 1 to x.
As a side remark, calling transpose on a 1D array doesn't have any effect (it reverses the order of your axes, and you only have one).