Using NumPy's polyfit (or something similar) is there an easy way to get a solution where one or more of the coefficients are constrained to a specific value?
For example, we could find the ordinary polynomial fitting using:
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
z = np.polyfit(x, y, 3)
yielding
array([ 0.08703704, -0.81349206, 1.69312169, -0.03968254])
But what if I wanted the best fit polynomial where the third coefficient (in the above case z[2]) was required to be 1? Or will I need to write the fitting from scratch?
In this case, I would use curve_fit or lmfit; I quickly show it for the first one.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c, d):
return a + b * x + c * x ** 2 + d * x ** 3
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
print(np.polyfit(x, y, 3))
popt, _ = curve_fit(func, x, y)
print(popt)
popt_cons, _ = curve_fit(func, x, y, bounds=([-np.inf, 2, -np.inf, -np.inf], [np.inf, 2.001, np.inf, np.inf]))
print(popt_cons)
xnew = np.linspace(x[0], x[-1], 1000)
plt.plot(x, y, 'bo')
plt.plot(xnew, func(xnew, *popt), 'k-')
plt.plot(xnew, func(xnew, *popt_cons), 'r-')
plt.show()
This will print:
[ 0.08703704 -0.81349206 1.69312169 -0.03968254]
[-0.03968254 1.69312169 -0.81349206 0.08703704]
[-0.14331349 2. -0.95913556 0.10494372]
So in the unconstrained case, polyfit and curve_fit give identical results (just the order is different), in the constrained case, the fixed parameter is 2, as desired.
The plot looks then as follows:
In lmfit you can also choose whether a parameter should be fitted or not, so you can then also just set it to a desired value (check this answer).
For completeness, with lmfit the solution would look like this:
import numpy as np
import matplotlib.pyplot as plt
from lmfit import Model
def func(x, a, b, c, d):
return a + b * x + c * x ** 2 + d * x ** 3
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
pmodel = Model(func)
params = pmodel.make_params(a=1, b=2, c=1, d=1)
params['b'].vary = False
result = pmodel.fit(y, params, x=x)
print(result.fit_report())
xnew = np.linspace(x[0], x[-1], 1000)
ynew = result.eval(x=xnew)
plt.plot(x, y, 'bo')
plt.plot(x, result.best_fit, 'k-')
plt.plot(xnew, ynew, 'r-')
plt.show()
which would print a comprehensive report, including uncertainties, correlations and fit statistics as:
[[Model]]
Model(func)
[[Fit Statistics]]
# fitting method = leastsq
# function evals = 10
# data points = 6
# variables = 3
chi-square = 0.066
reduced chi-square = 0.022
Akaike info crit = -21.089
Bayesian info crit = -21.714
[[Variables]]
a: -0.14331348 +/- 0.109441 (76.37%) (init= 1)
b: 2 (fixed)
c: -0.95913555 +/- 0.041516 (4.33%) (init= 1)
d: 0.10494371 +/- 0.008231 (7.84%) (init= 1)
[[Correlations]] (unreported correlations are < 0.100)
C(c, d) = -0.987
C(a, c) = -0.695
C(a, d) = 0.610
and produce a plot of
Note that lmfit.Model has many improvements over curve_fit, including automatically naming parameters based on function arguments, allowing any parameter to have bounds or simply be fixed without requiring nonsense like having upper and lower bounds that are almost equal. The key is that lmfit uses Parameter objects that have attributes instead of plain arrays of fitting variables. lmfit also supports mathematical constraints, composite models (eg, adding or multiplying models), and has superior reports.
Sorry for the resurrection
..but I felt that this answer was missing.
To fit a polynomial we solve the following system of equations:
a0*x0^n + a1*x0^(n-1) .. + an*x0^0 = y0
a0*x1^n + a1*x1^(n-1) .. + an*x1^0 = y1
...
a0*xm^n + a1*xm^(n-1) .. + an*xm^0 = ym
Which is a problem of the form V # a = y
where "V" is a Vandermonde matrix:
[[x0^n x0^(n-1) 1],
[x1^n x1^(n-1) 1],
...
[xm^n xm^(n-1) 1]]
"y" is a column vector holding the y-values:
[[y0],
[y1],
...
[ym]]
..and "a" is the column vector of coefficients that we are solving for:
[[a0],
[a1],
...
[an]]
This problem can be solved using linear least squares as follows:
import numpy as np
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
deg = 3
V = np.vander(x, deg + 1)
z, *_ = np.linalg.lstsq(V, y, rcond=None)
print(z)
# [ 0.08703704 -0.81349206 1.69312169 -0.03968254]
..which produces the same solution as the polyfit method:
z = np.polyfit(x, y, deg)
print(z)
# [ 0.08703704 -0.81349206 1.69312169 -0.03968254]
Instead we want a solution where a2 = 1
substituting a2 = 1 into the system of equations from the beginning of the answer, and then moving the corresponding term from the lhs to the rhs we get:
a0*x0^n + a1*x0^(n-1) + 1*x0^(n-2) .. + an*x0^0 = y0
a0*x1^n + a1*x1^(n-1) + 1*x0^(n-2) .. + an*x1^0 = y1
...
a0*xm^n + a1*xm^(n-1) + 1*x0^(n-2) .. + an*xm^0 = ym
=>
a0*x0^n + a1*x0^(n-1) .. + an*x0^0 = y0 - 1*x0^(n-2)
a0*x1^n + a1*x1^(n-1) .. + an*x1^0 = y1 - 1*x0^(n-2)
...
a0*xm^n + a1*xm^(n-1) .. + an*xm^0 = ym - 1*x0^(n-2)
This corresponds to removing column 2 from the Vandermonde matrix and subtracting it from the y-vector as follows:
y_ = y - V[:, 2]
V_ = np.delete(V, 2, axis=1)
z_, *_ = np.linalg.lstsq(V_, y_, rcond=None)
z_ = np.insert(z_, 2, 1)
print(z_)
# [ 0.04659264 -0.48453866 1. 0.19438046]
Notice that I inserted the 1 in the coefficient vector after solving the linear least-squares problem, we are no longer solving for a2 since we set it to 1 and removed it from the problem.
For completeness this is what the solution looks like when plotted:
and the complete code that I used:
import numpy as np
x = np.array([0.0, 1.0, 2.0, 3.0, 4.0, 5.0])
y = np.array([0.0, 0.8, 0.9, 0.1, -0.8, -1.0])
deg = 3
V = np.vander(x, deg + 1)
z, *_ = np.linalg.lstsq(V, y, rcond=None)
print(z)
# [ 0.08703704 -0.81349206 1.69312169 -0.03968254]
z = np.polyfit(x, y, deg)
print(z)
# [ 0.08703704 -0.81349206 1.69312169 -0.03968254]
y_ = y - V[:, 2]
V_ = np.delete(V, 2, axis=1)
z_, *_ = np.linalg.lstsq(V_, y_, rcond=None)
z_ = np.insert(z_, 2, 1)
print(z_)
# [ 0.04659264 -0.48453866 1. 0.19438046]
from matplotlib import pyplot as plt
plt.plot(x, y, 'o', label='data')
plt.plot(x, V # z, label='polyfit')
plt.plot(x, V # z_, label='constrained (a2 = 0)')
plt.legend()
plt.show()
Here is a way to do this using scipy.optimize.curve_fit:
First, let's recreate your example (as a sanity check):
import numpy as np
from scipy.optimize import curve_fit
def f(x, x3, x2, x1, x0):
"""this is the polynomial function"""
return x0 + x1*x + x2*(x*x) + x3*(x*x*x)
popt, pcov = curve_fit(f, x, y)
print(popt)
#array([ 0.08703704, -0.81349206, 1.69312169, -0.03968254])
Which matches the values you get from np.polyfit().
Now adding the constraints for x1:
popt, pcov = curve_fit(
f,
x,
y,
bounds = ([-np.inf, -np.inf, .999999999, -np.inf], [np.inf, np.inf, 1.0, np.inf])
)
print(popt)
#array([ 0.04659264, -0.48453866, 1. , 0.19438046])
I had to use .999999999 because the lower bound must be strictly less than the upper bound.
Alternatively, you could define your function with the constrained coefficient as a constant, and get the values for the other 3:
def f_new(x, x3, x2, x0):
x1 = 1
return x0 + x1*x + x2*(x*x) + x3*(x*x*x)
popt, pcov = curve_fit(f_new, x, y)
print(popt)
#array([ 0.04659264, -0.48453866, 0.19438046])
Here is also a way by using scipy.optimize.curve_fit but aiming to fix whatever the polynomial coefficients are desired. (The code is not so long after removing the comments.)
The guy that does the job:
import numpy as np
from scipy.optimize import curve_fit
def polyfit(x, y, deg, which=-1, to=0):
"""
An extension of ``np.polyfit`` to fix values of the vector
of polynomial coefficients. By default, the last coefficient
(i.e., the constant term) is kept at zero.
Parameters
----------
x : array_like
x-coordinates of the sample points.
y : array_like
y-coordinates of the sample points.
deg : int
Degree of the fitting polynomial.
which : int or array_like, optional
Indexes of the coefficients to remain fixed. By default, -1.
to : float or array_like, optional
Values of the fixed coefficients. By default, 0.
Returns
-------
np.ndarray
(deg + 1) polynomial coefficients.
"""
p0 = np.polyfit(x, y, deg)
# if which == None it is reduced to np.polyfit
if which is None:
return p0
# indexes of the coeffs being fitted
which_not = np.delete(np.arange(deg + 1), which)
# create the array of coeffs
def _fill_p(p):
p_ = np.empty(deg + 1) # empty array
p_[which] = to # fill with custom coeffs
p_[which_not] = p # fill with `p`
return p_
# callback function for fitting
def _polyfit(x, *p):
p_ = _fill_p(p)
return np.polyval(p_, x)
# get the array of coeffs
p0 = np.delete(p0, which) # use `p0` as initial condition
p, _ = curve_fit(_polyfit, x, y, p0=p0) # fitting
p = _fill_p(p) # merge fixed and no-fixed coeffs
return p
Two simple examples on how to use the function above:
import matplotlib.pyplot as plt
# just create some fake data (a parabola)
np.random.seed(0) # seed to reproduce the example
deg = 2 # second order polynomial
p = np.random.randint(1, 5, size=deg+1) # random vector of coefficients
x = np.linspace(0, 10, num=20) # fake data: x-array
y = np.polyval(p, x) + 1.5*np.random.randn(20) # fake data: y-array
print(p) # output:[1, 4, 2]
# fitting
p1 = polyfit(x, y, deg, which=2, to=p[2]) # case 1: last coeff is fixed
p2 = polyfit(x, y, deg, which=[1,2], to=p[1:3]) # case 2: last two coeffs are fixed
y1 = np.polyval(p1, x) # y-array for case 1
y2 = np.polyval(p2, x) # y-array for case 2
print(p1) # output: [1.05, 3.67, 2.]
print(p2) # output: [1.08, 4., 2.]
# plotting
plt.plot(x, y, '.', label='fake data: y = p[0]*x**2 + p[1]*x + p[2]')
plt.plot(x, y1, label='p[2] fixed at 2')
plt.plot(x, y2, label='p[2] and p[1] fixed at [4, 2]')
plt.legend()
plt.show()
Related
I am trying to fit x y data which look something like
x = np.linspace(-2, 2, 1000)
a = 0.5
yl = np.ones_like(x[x < a]) * -0.4 + np.random.normal(0, 0.05, x[x < a].shape[0])
yr = np.ones_like(x[x >= a]) * 0.4 + np.random.normal(0, 0.05, x[x >= a].shape[0])
y = np.concatenate((yl, yr))
plt.scatter(x, y, s=2, color='k')
I'm using a variation of the Heaviside step function
def f(x, a, b): return 0.5 * b * (np.sign(x - a))
and fitting with
popt, pcov = curve_fit(f, x, y, p0=p)
where p is some initial guess.
for any p curve_fit fit only b and not a
for example:
popt, pcov = curve_fit(f, x, y, p0=[-1.0, 0])
we get that popt is [-1., 0.20117665]
popt, pcov = curve_fit(f, x, y, p0=[.5, 2])
we get taht popt is [.5, 0.79902]
popt, pcov = curve_fit(f, x, y, p0=[1.5, -2])
we get taht popt is [1.5, 0.40128229]
why curve_fit not fitting a?
As mentioned by others, curve_fit (and all the other solvers in scipy.optimize) work well for optimizing continuous but not discrete variables. They all work by making small (like, at the 1.e-7 level) changes to the parameter values and seeing what (if any) change that makes in the result, and using that change to refine those values until the smallest residual is found. With your model function using np.sign:
def f(x, a, b): return 0.5 * b * (np.sign(x - a))
such a small change in the value of a will not change the model or fit result at all. That is, first the fit will try the starting value of, say, a=-1.0 or a=0.5, and then will try a=-0.999999995 or a=0.500000005. Those will both give the same result for np.sign(x-a). The fit does not know that it would need to change a by 1 to have any effect on the result. It cannot know this. np.sign() and np.sin() differ by one letter, but behave very differently in this respect.
It is pretty common for real data to take a step but to be sampled finely enough so that the step does not happen completely in one step. In that case, you would be able to model the step with a variety of functional forms (linear ramp, error function, arc-tangent, logistic, etc). The thorough answer from #JamesPhilipps gives one approach. I would probably use lmfit (being one of its main authors) and be willing to guess starting values for the parameters from looking at the data, perhaps:
import numpy as np
x = np.linspace(-2, 2, 1000)
a = 0.5
yl = np.ones_like(x[x < a]) * -0.4 + np.random.normal(0, 0.05, x[x < a].shape[0])
yr = np.ones_like(x[x >= a]) * 0.4 + np.random.normal(0, 0.05, x[x >= a].shape[0])
y = np.concatenate((yl, yr))
from lmfit.models import StepModel, ConstantModel
model = StepModel() + ConstantModel()
params = model.make_params(center=0, sigma=1, amplitude=1., c=-0.5)
result = model.fit(y, params, x=x)
print(result.fit_report())
import matplotlib.pyplot as plt
plt.scatter(x, y, label='data')
plt.plot(x, result.best_fit, marker='o', color='r', label='fit')
plt.show()
which would give a good fit and print out results of
[[Model]]
(Model(step, form='linear') + Model(constant))
[[Fit Statistics]]
# fitting method = leastsq
# function evals = 50
# data points = 1000
# variables = 4
chi-square = 2.32729556
reduced chi-square = 0.00233664
Akaike info crit = -6055.04839
Bayesian info crit = -6035.41737
## Warning: uncertainties could not be estimated:
[[Variables]]
amplitude: 0.80013762 (init = 1)
center: 0.50083312 (init = 0)
sigma: 4.6009e-04 (init = 1)
c: -0.40006255 (init = -0.5)
Note that it will find the center of the step because it assumed there was some finite width (sigma) to the step, but then found that width to be smaller than the step size in x. But also note that it cannot calculate the uncertainties in the parameters because, as above, a small change in center (your a) near the solution does not change the resulting fit. FWIW the StepModel can use a linear, error-function, arc-tangent, or logistic as the step function.
If you had constructed the test data to have a small width to the step, say with
something like
from scipy.special import erf
y = 0.638 * erf((x-0.574)/0.005) + np.random.normal(0, 0.05, len(x))
then the fit would have been able to find the best solution and evaluate the uncertainties.
I hope that explains why the fit with your model function could not refine the value of a, and what might be done about it.
Here is a graphical Python fitter using your data and function, with scipy's differential_evolution genetic algorithm module used to provide the initial parameter estimates for curve_fit. That module uses the Latin Hypercube algorithm to ensure a thorough search of parameter space, which requires bounds within which to search. In this example, those bounds are taken from the data max and min values.
import numpy, scipy, matplotlib
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from scipy.optimize import differential_evolution
import warnings
# generate data for testing
x = numpy.linspace(-2, 2, 1000)
a = 0.5
yl = numpy.ones_like(x[x < a]) * -0.4 + numpy.random.normal(0, 0.05, x[x < a].shape[0])
yr = numpy.ones_like(x[x >= a]) * 0.4 + numpy.random.normal(0, 0.05, x[x >= a].shape[0])
y = numpy.concatenate((yl, yr))
# alias data to match pervious example
xData = x
yData = y
def func(x, a, b): # variation of the Heaviside step function
return 0.5 * b * (numpy.sign(x - a))
# function for genetic algorithm to minimize (sum of squared error)
def sumOfSquaredError(parameterTuple):
warnings.filterwarnings("ignore") # do not print warnings by genetic algorithm
val = func(xData, *parameterTuple)
return numpy.sum((yData - val) ** 2.0)
def generate_Initial_Parameters():
# min and max used for bounds
maxX = max(xData)
minX = min(xData)
parameterBounds = []
parameterBounds.append([minX, maxX]) # search bounds for a
parameterBounds.append([minX, maxX]) # search bounds for b
# "seed" the numpy random number generator for repeatable results
result = differential_evolution(sumOfSquaredError, parameterBounds, seed=3)
return result.x
# by default, differential_evolution completes by calling curve_fit() using parameter bounds
geneticParameters = generate_Initial_Parameters()
# now call curve_fit without passing bounds from the genetic algorithm,
# just in case the best fit parameters are aoutside those bounds
fittedParameters, pcov = curve_fit(func, xData, yData, geneticParameters)
print('Fitted parameters:', fittedParameters)
print()
modelPredictions = func(xData, *fittedParameters)
absError = modelPredictions - yData
SE = numpy.square(absError) # squared errors
MSE = numpy.mean(SE) # mean squared errors
RMSE = numpy.sqrt(MSE) # Root Mean Squared Error, RMSE
Rsquared = 1.0 - (numpy.var(absError) / numpy.var(yData))
print()
print('RMSE:', RMSE)
print('R-squared:', Rsquared)
print()
##########################################################
# graphics output section
def ModelAndScatterPlot(graphWidth, graphHeight):
f = plt.figure(figsize=(graphWidth/100.0, graphHeight/100.0), dpi=100)
axes = f.add_subplot(111)
# first the raw data as a scatter plot
axes.plot(xData, yData, 'D')
# create data for the fitted equation plot
xModel = numpy.linspace(min(xData), max(xData))
yModel = func(xModel, *fittedParameters)
# now the model as a line plot
axes.plot(xModel, yModel)
axes.set_xlabel('X Data') # X axis data label
axes.set_ylabel('Y Data') # Y axis data label
plt.show()
plt.close('all') # clean up after using pyplot
graphWidth = 800
graphHeight = 600
ModelAndScatterPlot(graphWidth, graphHeight)
Or you could say that a heavyside can be approximated by a sigmoïd function:
or in your case
You add a parameter k, but hopefully it will be big enough in the end, and you get rid of it to find the two other parameters.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
x = np.linspace(-2, 2, 1000)
a = 0.5
yl = np.ones_like(x[x < a]) * -0.4 + np.random.normal(0, 0.05, x[x < a].shape[0])
yr = np.ones_like(x[x >= a]) * 0.4 + np.random.normal(0, 0.05, x[x >= a].shape[0])
y = np.concatenate((yl, yr))
plt.scatter(x, y, s=2, color='k')
# def f(x, a, b): return 0.5 * b * (np.sign(x - a))
def g(x, a, b, k): return b / (1 + np.exp(-2 * k * (x - a))) - b / 2
y_sigmoid = g(x, a, 0.8, 10)
plt.scatter(x, y_sigmoid, s=2, color='g')
popt, pcov = curve_fit(g, x, y, p0=[-1.0, 0, 1])
# popt, pcov = curve_fit(f, x, y, p0=[-1.0, 0])
print(popt)
plt.scatter(x, g(x, *popt), s=2, color='r')
which gives, as expected:
[5.02081214e-01 8.03257583e-01 3.33970547e+03]
(green: random soft sigmoid, red: curve_fit result)
I am trying to learn how to optimize data fits in higher dimensions using python / numpy / scipy. Having successfully used scipy to fit a line of the form y = f(x), I tried extending the logic to fit an ellipse of the form z = f(x, y); both are shown below. I'm hoping this approach can be generalized to fit shapes in higher dimensions (ie, sphere).
import numpy as np
from scipy.optimize import minimize
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
## LINE EXAMPLE
npoints = 30
x = np.linspace(-5, 5, npoints)
m, b = 3, -4 # slope, y-intercept
initial_parameter_guess = (2.5, -1)
y = m * x + b # exact line
noise = np.random.uniform(-1, 1, size=x.size)
yn = y + noise # line with noise
def get_residuals(prms, x, y):
""" """
return (y - (prms[0] * x + prms[1]))**2
def f_error(prms, x, y):
""" """
resid = get_residuals(prms, x, y)
return np.sum(resid)
result = minimize(f_error, x0=initial_parameter_guess, args=(x, yn))
# print(result)
yf = result.x[0] * x + result.x[1]
fig, ax = plt.subplots()
ax.scatter(x, yn, color='b', marker='.')
ax.plot(x, yf, color='r', alpha=0.5)
ax.grid(color='k', alpha=0.3, linestyle=':')
plt.show()
plt.close(fig)
Applying this logic to the case of an ellipse,
## ELLIPSE EXAMPLE
npoints = 75
theta = np.random.uniform(0, 2*np.pi, size=npoints)
a, b = (3, 5)
initial_parameter_guess = (2.5, 6)
xnoise = np.random.uniform(-1, 1, size=theta.size)
ynoise = np.random.uniform(-1, 1, size=theta.size)
x = a**2 * np.cos(theta)
xn = x + xnoise
y = b**2 * np.sin(theta)
yn = y + ynoise
def get_residuals(prms, x, y):
""" """
return 1 - ((x/prms[0])**2 + (y/prms[1])**2)
def f_error(prms, x, y):
""" """
resid = get_residuals(prms, x, y)
return np.sum(resid)
result = minimize(f_error, x0=initial_parameter_guess, args=(xn, yn))
# print(result)
The minimize algorithm via scipy does not find the optimal parameters; the following output is shown with print(result):
fun: -4243.611573066522
hess_inv: array([[41.44400248, 39.68101343],
[39.68101343, 37.99343048]])
jac: array([-1496.81719971, 2166.68896484])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 174
nit: 1
njev: 42
status: 2
success: False
x: array([-1.51640333, 2.93879038])
I have seen another solution to this problem, which use the matrix formulation of least squares, but the example is a bit difficult for me to follow. Almost all approaches I've seen are based on the approach in the posted link. I'd prefer using minimize unless the linear algebra approach was better for a reason that I am currently unaware of.
Anyways, can my approach above be adapted/tweaked in a way that will work and can be generalized for higher dimensions?
Two issues with the code . Instead of minimizing the sum of residuals , minimize the sum of squares of residuals. Secondly , elliptic equation should be defined as x=a*np.cos(theta) and y=b*np.sin(theta)
npoints = 75
theta = np.random.uniform(0, 2*np.pi, size=npoints)
a, b = (3, 5)
initial_parameter_guess = (2.5, 6)
xnoise = np.random.uniform(-1, 1, size=theta.size)
ynoise = np.random.uniform(-1, 1, size=theta.size)
x = a * np.cos(theta)
xn = x + xnoise
y = b * np.sin(theta)
yn = y + ynoise
def get_residuals(prms, x, y):
""" """
return 1 - ((x/prms[0])**2 + (y/prms[1])**2)
def f_error(prms, x, y):
""" """
resid = get_residuals(prms, x, y)
return np.sum(np.square(resid))
result = minimize(f_error, x0=initial_parameter_guess,args=(xn, yn))
result
fun: 5.85099318913613
hess_inv: array([[ 0.07025572, -0.02902779],
[-0.02902779, 0.12040811]])
jac: array([-5.96046448e-08, 1.19209290e-07])
message: 'Optimization terminated successfully.'
nfev: 48
nit: 10
njev: 12
status: 0
success: True
x: array([3.35248219, 5.13728323])
I am getting an optimize warning:
OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)
when trying to fit my piecewise function to my data using scipy.optimize.curve_fit. Meaning no fitting is happening. I can easily fit a parabola to my data, and I'm supplying curve_fit with what I feel are good initial parameters. Full code sample below. Does anyone know why curve_fit might not be getting along with np.piecewise? Or am I making a different mistake?
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt
def piecewise_linear(x, x0, y0, k1, k2):
y = np.piecewise(x, [x < x0, x >= x0],
[lambda x:k1*x + y0-k1*x0, lambda x:k2*x + y0-k2*x0])
return y
def parabola(x, a, b):
y = a * x**2 + b
return y
x = np.array([-3, -2, -1, 0, 1, 2, 3])
y = np.array([9.15, 5.68, 2.32, 0.00, 2.05, 5.29, 8.62])
popt_piecewise, pcov = curve_fit(piecewise_linear, x, y, p0=[0.1, 0.1, -5, 5])
popt_parabola, pcov = curve_fit(parabola, x, y, p0=[1, 1])
new_x = np.linspace(x.min(), x.max(), 61)
fig, ax = plt.subplots()
ax.plot(x, y, 'o', ls='')
ax.plot(new_x, piecewise_linear(new_x, *popt_piecewise))
ax.plot(new_x, parabola(new_x, *popt_parabola))
ax.set_xlim(-4, 4)
ax.set_ylim(-2, 16)
It is a problem with types, you have to change the following line, so that the x is given as floats:
x = np.array([-3, -2, -1, 0, 1, 2, 3]).astype(np.float)
otherwise the piecewise_linear will might end up casting the types.
Just to be on the safe side you could also make the initial points float here:
popt_piecewise, pcov = curve_fit(piecewise_linear, x, y, p0=[0.1, 0.1, -5., 5.])
For completeness, I'll point out that fitting a piecewise linear function does not require np.piecewise: any such function can be constructed out of absolute values, using a multiple of np.abs(x-x0) for each bend. The following produces a good fit to the data:
def pl(x, x0, a, b, c):
y = a*np.abs(x-x0) + b*x + c
return y
popt_pl, pcov = curve_fit(pl, x, y, p0=[0, 0, 0, 0])
print(pl(x, *popt_pl))
Output is close to original y-values:
[ 8.90899998 5.828 2.74700002 -0.33399996 2.03499998 5.32
8.60500002]
I have a set of data and I want to compare which line describes it best (polynomials of different orders, exponential or logarithmic).
I use Python and Numpy and for polynomial fitting there is a function polyfit(). But I found no such functions for exponential and logarithmic fitting.
Are there any? Or how to solve it otherwise?
For fitting y = A + B log x, just fit y against (log x).
>>> x = numpy.array([1, 7, 20, 50, 79])
>>> y = numpy.array([10, 19, 30, 35, 51])
>>> numpy.polyfit(numpy.log(x), y, 1)
array([ 8.46295607, 6.61867463])
# y ≈ 8.46 log(x) + 6.62
For fitting y = AeBx, take the logarithm of both side gives log y = log A + Bx. So fit (log y) against x.
Note that fitting (log y) as if it is linear will emphasize small values of y, causing large deviation for large y. This is because polyfit (linear regression) works by minimizing ∑i (ΔY)2 = ∑i (Yi − Ŷi)2. When Yi = log yi, the residues ΔYi = Δ(log yi) ≈ Δyi / |yi|. So even if polyfit makes a very bad decision for large y, the "divide-by-|y|" factor will compensate for it, causing polyfit favors small values.
This could be alleviated by giving each entry a "weight" proportional to y. polyfit supports weighted-least-squares via the w keyword argument.
>>> x = numpy.array([10, 19, 30, 35, 51])
>>> y = numpy.array([1, 7, 20, 50, 79])
>>> numpy.polyfit(x, numpy.log(y), 1)
array([ 0.10502711, -0.40116352])
# y ≈ exp(-0.401) * exp(0.105 * x) = 0.670 * exp(0.105 * x)
# (^ biased towards small values)
>>> numpy.polyfit(x, numpy.log(y), 1, w=numpy.sqrt(y))
array([ 0.06009446, 1.41648096])
# y ≈ exp(1.42) * exp(0.0601 * x) = 4.12 * exp(0.0601 * x)
# (^ not so biased)
Note that Excel, LibreOffice and most scientific calculators typically use the unweighted (biased) formula for the exponential regression / trend lines. If you want your results to be compatible with these platforms, do not include the weights even if it provides better results.
Now, if you can use scipy, you could use scipy.optimize.curve_fit to fit any model without transformations.
For y = A + B log x the result is the same as the transformation method:
>>> x = numpy.array([1, 7, 20, 50, 79])
>>> y = numpy.array([10, 19, 30, 35, 51])
>>> scipy.optimize.curve_fit(lambda t,a,b: a+b*numpy.log(t), x, y)
(array([ 6.61867467, 8.46295606]),
array([[ 28.15948002, -7.89609542],
[ -7.89609542, 2.9857172 ]]))
# y ≈ 6.62 + 8.46 log(x)
For y = AeBx, however, we can get a better fit since it computes Δ(log y) directly. But we need to provide an initialize guess so curve_fit can reach the desired local minimum.
>>> x = numpy.array([10, 19, 30, 35, 51])
>>> y = numpy.array([1, 7, 20, 50, 79])
>>> scipy.optimize.curve_fit(lambda t,a,b: a*numpy.exp(b*t), x, y)
(array([ 5.60728326e-21, 9.99993501e-01]),
array([[ 4.14809412e-27, -1.45078961e-08],
[ -1.45078961e-08, 5.07411462e+10]]))
# oops, definitely wrong.
>>> scipy.optimize.curve_fit(lambda t,a,b: a*numpy.exp(b*t), x, y, p0=(4, 0.1))
(array([ 4.88003249, 0.05531256]),
array([[ 1.01261314e+01, -4.31940132e-02],
[ -4.31940132e-02, 1.91188656e-04]]))
# y ≈ 4.88 exp(0.0553 x). much better.
You can also fit a set of a data to whatever function you like using curve_fit from scipy.optimize. For example if you want to fit an exponential function (from the documentation):
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.exp(-b * x) + c
x = np.linspace(0,4,50)
y = func(x, 2.5, 1.3, 0.5)
yn = y + 0.2*np.random.normal(size=len(x))
popt, pcov = curve_fit(func, x, yn)
And then if you want to plot, you could do:
plt.figure()
plt.plot(x, yn, 'ko', label="Original Noised Data")
plt.plot(x, func(x, *popt), 'r-', label="Fitted Curve")
plt.legend()
plt.show()
(Note: the * in front of popt when you plot will expand out the terms into the a, b, and c that func is expecting.)
I was having some trouble with this so let me be very explicit so noobs like me can understand.
Lets say that we have a data file or something like that
# -*- coding: utf-8 -*-
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
import numpy as np
import sympy as sym
"""
Generate some data, let's imagine that you already have this.
"""
x = np.linspace(0, 3, 50)
y = np.exp(x)
"""
Plot your data
"""
plt.plot(x, y, 'ro',label="Original Data")
"""
brutal force to avoid errors
"""
x = np.array(x, dtype=float) #transform your data in a numpy array of floats
y = np.array(y, dtype=float) #so the curve_fit can work
"""
create a function to fit with your data. a, b, c and d are the coefficients
that curve_fit will calculate for you.
In this part you need to guess and/or use mathematical knowledge to find
a function that resembles your data
"""
def func(x, a, b, c, d):
return a*x**3 + b*x**2 +c*x + d
"""
make the curve_fit
"""
popt, pcov = curve_fit(func, x, y)
"""
The result is:
popt[0] = a , popt[1] = b, popt[2] = c and popt[3] = d of the function,
so f(x) = popt[0]*x**3 + popt[1]*x**2 + popt[2]*x + popt[3].
"""
print "a = %s , b = %s, c = %s, d = %s" % (popt[0], popt[1], popt[2], popt[3])
"""
Use sympy to generate the latex sintax of the function
"""
xs = sym.Symbol('\lambda')
tex = sym.latex(func(xs,*popt)).replace('$', '')
plt.title(r'$f(\lambda)= %s$' %(tex),fontsize=16)
"""
Print the coefficients and plot the funcion.
"""
plt.plot(x, func(x, *popt), label="Fitted Curve") #same as line above \/
#plt.plot(x, popt[0]*x**3 + popt[1]*x**2 + popt[2]*x + popt[3], label="Fitted Curve")
plt.legend(loc='upper left')
plt.show()
the result is:
a = 0.849195983017 , b = -1.18101681765, c = 2.24061176543, d = 0.816643894816
Here's a linearization option on simple data that uses tools from scikit learn.
Given
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import FunctionTransformer
np.random.seed(123)
# General Functions
def func_exp(x, a, b, c):
"""Return values from a general exponential function."""
return a * np.exp(b * x) + c
def func_log(x, a, b, c):
"""Return values from a general log function."""
return a * np.log(b * x) + c
# Helper
def generate_data(func, *args, jitter=0):
"""Return a tuple of arrays with random data along a general function."""
xs = np.linspace(1, 5, 50)
ys = func(xs, *args)
noise = jitter * np.random.normal(size=len(xs)) + jitter
xs = xs.reshape(-1, 1) # xs[:, np.newaxis]
ys = (ys + noise).reshape(-1, 1)
return xs, ys
transformer = FunctionTransformer(np.log, validate=True)
Code
Fit exponential data
# Data
x_samp, y_samp = generate_data(func_exp, 2.5, 1.2, 0.7, jitter=3)
y_trans = transformer.fit_transform(y_samp) # 1
# Regression
regressor = LinearRegression()
results = regressor.fit(x_samp, y_trans) # 2
model = results.predict
y_fit = model(x_samp)
# Visualization
plt.scatter(x_samp, y_samp)
plt.plot(x_samp, np.exp(y_fit), "k--", label="Fit") # 3
plt.title("Exponential Fit")
Fit log data
# Data
x_samp, y_samp = generate_data(func_log, 2.5, 1.2, 0.7, jitter=0.15)
x_trans = transformer.fit_transform(x_samp) # 1
# Regression
regressor = LinearRegression()
results = regressor.fit(x_trans, y_samp) # 2
model = results.predict
y_fit = model(x_trans)
# Visualization
plt.scatter(x_samp, y_samp)
plt.plot(x_samp, y_fit, "k--", label="Fit") # 3
plt.title("Logarithmic Fit")
Details
General Steps
Apply a log operation to data values (x, y or both)
Regress the data to a linearized model
Plot by "reversing" any log operations (with np.exp()) and fit to original data
Assuming our data follows an exponential trend, a general equation+ may be:
We can linearize the latter equation (e.g. y = intercept + slope * x) by taking the log:
Given a linearized equation++ and the regression parameters, we could calculate:
A via intercept (ln(A))
B via slope (B)
Summary of Linearization Techniques
Relationship | Example | General Eqn. | Altered Var. | Linearized Eqn.
-------------|------------|----------------------|----------------|------------------------------------------
Linear | x | y = B * x + C | - | y = C + B * x
Logarithmic | log(x) | y = A * log(B*x) + C | log(x) | y = C + A * (log(B) + log(x))
Exponential | 2**x, e**x | y = A * exp(B*x) + C | log(y) | log(y-C) = log(A) + B * x
Power | x**2 | y = B * x**N + C | log(x), log(y) | log(y-C) = log(B) + N * log(x)
+Note: linearizing exponential functions works best when the noise is small and C=0. Use with caution.
++Note: while altering x data helps linearize exponential data, altering y data helps linearize log data.
Well I guess you can always use:
np.log --> natural log
np.log10 --> base 10
np.log2 --> base 2
Slightly modifying IanVS's answer:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
#return a * np.exp(-b * x) + c
return a * np.log(b * x) + c
x = np.linspace(1,5,50) # changed boundary conditions to avoid division by 0
y = func(x, 2.5, 1.3, 0.5)
yn = y + 0.2*np.random.normal(size=len(x))
popt, pcov = curve_fit(func, x, yn)
plt.figure()
plt.plot(x, yn, 'ko', label="Original Noised Data")
plt.plot(x, func(x, *popt), 'r-', label="Fitted Curve")
plt.legend()
plt.show()
This results in the following graph:
We demonstrate features of lmfit while solving both problems.
Given
import lmfit
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
np.random.seed(123)
# General Functions
def func_log(x, a, b, c):
"""Return values from a general log function."""
return a * np.log(b * x) + c
# Data
x_samp = np.linspace(1, 5, 50)
_noise = np.random.normal(size=len(x_samp), scale=0.06)
y_samp = 2.5 * np.exp(1.2 * x_samp) + 0.7 + _noise
y_samp2 = 2.5 * np.log(1.2 * x_samp) + 0.7 + _noise
Code
Approach 1 - lmfit Model
Fit exponential data
regressor = lmfit.models.ExponentialModel() # 1
initial_guess = dict(amplitude=1, decay=-1) # 2
results = regressor.fit(y_samp, x=x_samp, **initial_guess)
y_fit = results.best_fit
plt.plot(x_samp, y_samp, "o", label="Data")
plt.plot(x_samp, y_fit, "k--", label="Fit")
plt.legend()
Approach 2 - Custom Model
Fit log data
regressor = lmfit.Model(func_log) # 1
initial_guess = dict(a=1, b=.1, c=.1) # 2
results = regressor.fit(y_samp2, x=x_samp, **initial_guess)
y_fit = results.best_fit
plt.plot(x_samp, y_samp2, "o", label="Data")
plt.plot(x_samp, y_fit, "k--", label="Fit")
plt.legend()
Details
Choose a regression class
Supply named, initial guesses that respect the function's domain
You can determine the inferred parameters from the regressor object. Example:
regressor.param_names
# ['decay', 'amplitude']
To make predictions, use the ModelResult.eval() method.
model = results.eval
y_pred = model(x=np.array([1.5]))
Note: the ExponentialModel() follows a decay function, which accepts two parameters, one of which is negative.
See also ExponentialGaussianModel(), which accepts more parameters.
Install the library via > pip install lmfit.
Wolfram has a closed form solution for fitting an exponential. They also have similar solutions for fitting a logarithmic and power law.
I found this to work better than scipy's curve_fit. Especially when you don't have data "near zero". Here is an example:
import numpy as np
import matplotlib.pyplot as plt
# Fit the function y = A * exp(B * x) to the data
# returns (A, B)
# From: https://mathworld.wolfram.com/LeastSquaresFittingExponential.html
def fit_exp(xs, ys):
S_x2_y = 0.0
S_y_lny = 0.0
S_x_y = 0.0
S_x_y_lny = 0.0
S_y = 0.0
for (x,y) in zip(xs, ys):
S_x2_y += x * x * y
S_y_lny += y * np.log(y)
S_x_y += x * y
S_x_y_lny += x * y * np.log(y)
S_y += y
#end
a = (S_x2_y * S_y_lny - S_x_y * S_x_y_lny) / (S_y * S_x2_y - S_x_y * S_x_y)
b = (S_y * S_x_y_lny - S_x_y * S_y_lny) / (S_y * S_x2_y - S_x_y * S_x_y)
return (np.exp(a), b)
xs = [33, 34, 35, 36, 37, 38, 39, 40, 41, 42]
ys = [3187, 3545, 4045, 4447, 4872, 5660, 5983, 6254, 6681, 7206]
(A, B) = fit_exp(xs, ys)
plt.figure()
plt.plot(xs, ys, 'o-', label='Raw Data')
plt.plot(xs, [A * np.exp(B *x) for x in xs], 'o-', label='Fit')
plt.title('Exponential Fit Test')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend(loc='best')
plt.tight_layout()
plt.show()
My code bellow produces a polyfit of the points in my graph, but I want this fit to always pass through zero, how do I do this?
import pylab as pl
import numpy as np
y=(abs((UX2-UY2)+(2*UXY)))
a=np.mean(y)
y=y-a
x=(abs((X2-Y2)+(2*XY)))
b=np.mean(x)
x=x-b
ax=pl.subplot(1,4,4) #plot XY
fit=pl.polyfit(x,y,1)
slope4, fit_fn=pl.poly1d(fit)
print slope4
fit_fn=pl.poly1d(fit)
x_min=-2
x_max=5
n=10000
x_fit = pl.linspace(x_min, x_max, n)
y_fit = fit_fn(x_fit)
q=z=[-2,5]
scat=pl.plot(x,y, 'o', x_fit,y_fit, '-r', z, q, 'g' )
When you fit an n-degree polynomial p(x) = a0 + a1*x + a2*x**2 + ... + an*x**n to a set of data points (x0, y0), (x1, y1), ..., (xm, y_m), a call to np.lstsq is made with a coefficient matrix that looks like:
[1 x0 x0**2 ... x0**n]
[1 x1 x1**2 ... x1**n]
...
[1 xm xm**2 ... xm**n]
If you remove the j-th column from that matrix, you are effectively setting that coefficient in the polynomial to 0. So to get rid of the a0 coefficient you could do the following:
def fit_poly_through_origin(x, y, n=1):
a = x[:, np.newaxis] ** np.arange(1, n+1)
coeff = np.linalg.lstsq(a, y)[0]
return np.concatenate(([0], coeff))
n = 1000
x = np.random.rand(n)
y = 1 + 3*x - 4*x**2 + np.random.rand(n)*0.25
c0 = np.polynomial.polynomial.polyfit(x, y, 2)
c1 = fit_poly_through_origin(x, y, 2)
p0 = np.polynomial.Polynomial(c0)
p1 = np.polynomial.Polynomial(c1)
plt.plot(x, y, 'kx')
xx = np.linspace(0, 1, 1000)
plt.plot(xx, p0(xx), 'r-', )
plt.plot(xx, p1(xx), 'b-', )
As was mentioned, you can't really do it explicitly with polyfit (but you can write your own function).
However, if you want to still use polyfit() you can try this math hack: add a point at zero, and then use the w flag (weights) in polyfit() to give it a high weight while all other points get a low weight. This will have the effect of forcing the polynomial to pass at zero or very close.