Related
I am trying to find a function to model my curved dataset using scipy's curve fit function which gets me a line and spits out error message "OptimizeWarning: Covariance of the parameters could not be estimated"
This is my data:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.exp(-b * x) + c
x = [6529,6530,6531,6532,6533,6534,6535,6536,6537,6538,6539,6540,6541,6542,6543,6544,6545,6546,6547,6548,6549,6550,6551,6552,6553,6554,6555,6556,6557,6558,6559,6560,6561,6562,6563,6564,6565,6566,6567,6568,6569,6570,6571,6572,6573,6574,6575,6576,6577,6578,6579,6580,6581,6582,6583,6584,6585,6586,6587,6588,6589,6590,6591,6592,6593,6594,6595,6596,6597,6598,6599,6600,6601,6602,6603,6604,6605,6606,6607,6608,6609,6610,6611,6612,6613,6614,6615,6616,6617,6618,6619,6620,6621,6622,6623,6624,6625,6626,6627,6628,6629]
xdata = np.array(x)
xdata
y =[0.547409184506862,0.548439334507089,0.548707663683029,0.549517457040911,0.549928046272648,0.552133076225856,0.553680404362389,0.554546610865359,0.556183559635298,0.55702393264449,0.558127603440548,0.56005701419833,0.567268696243044,0.567959158455945,0.56888383955428,0.570002946460703,0.571738099634388,0.571781066314887,0.572492904162659,0.573363360104158,0.575401312020501,0.579556244995718,0.581790310757611,0.583406644934125,0.583431253838386,0.584646128487338,0.591634693424932,0.593621181449664,0.596918227952608,0.597537299010122,0.597822020010253,0.598891783912097,0.599584877929425,0.600869677067233,0.605499427101361,0.60658392374002,0.607408603951367,0.608003672935112,0.612816417541406,0.614176393253985,0.615691612727725,0.617134841882831,0.624502603183639,0.627504005062751,0.62811483139368,0.630923224681103,0.631913519350306,0.632861774084856,0.633396927216081,0.634723100574364,0.636823036518848,0.637335872514631,0.641989703420432,0.645848889736627,0.65711344945379,0.657128729116295,0.663572015593525,0.663936607768306,0.664261916284895,0.665497047151865,0.675768594810369,0.676207425367557,0.67770213122942,0.684362066388147,0.686091459831405,0.68923449405901,0.696555953074396,0.699523852803358,0.700114629853266,0.700439968032363,0.70422912133774,0.709054418203755,0.718323675919034,0.720874110082631,0.731420313805414,0.740339970645214,0.743941832408661,0.744098264483215,0.755637715557576,0.770607412727517,0.772170144147719,0.77386595160397,0.782346429315201,0.793536926034814,0.799814727138212,0.811806170429752,0.820389958104833,0.823442687233217,0.836431157138314,0.837230603981482,0.84457830759536,0.849472002016762,0.853064650303987,0.864143132487712,0.875848713252245,0.879457500784825,0.906226501241044,0.938124902423877,0.957251151492073,0.984140865987764,1]
ydata = np.array(y)
ydata
plt.plot(xdata, ydata, 'b.', label='data')
I tried the curve_fit function
popt, pcov = curve_fit(func, xdata, ydata)
popt
plt.plot(xdata, func(xdata, *popt), 'r-',
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
but it just plots a horizontal line.
The closest I've gotten to this was using a linear regression model in R
(something along the lines of model <- lm(log(y)~x, data=df) that looks like this:
But I want it to look more like this:
So that I can take the derivative of the curve afterwards. Sorry if my code is lacking, I'm new to python.
You can apply a simple transformation to xdata:
max_xdata = xdata.max()
xdata_t = max_xdata - xdata
popt, pcov = curve_fit(func, xdata_t, ydata)
print(popt)
fix, ax = plt.subplots(1, 1)
ax.plot(xdata_t, ydata, 'b.', label='data')
ax.plot(xdata_t, func(xdata_t, *popt), 'r-',
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
Which gets you:
Or, to get the fit to the original dataset:
fix, ax = plt.subplots(1, 1)
ax.plot(xdata, ydata, 'b.', label='data')
ax.plot(xdata, func(max_xdata - xdata, *popt), 'r-',
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
You can use linear regression if you transform your data before you do the fit.
The function you want looks like this:
y = a*exp(b*x)
Take the natural log of both sides:
ln(y) = ln(a) + b*x
You can see this looks like a linear regression where the y-intercept is ln(a) and the slope is b.
You'll need to take the natural log of each term in the y-array before you do the linear regression.
When the calculation is done, you'll get the value for a by exp(ln(a)).
I'm trying to plot a histogram in python by importing data from an excel file.
Also, the histogram needs to be fitted with an exponential function.
How can I do this plotting and fitting procedure?
For plotting just use plt.hist and your data
import random
import matplotlib.pyplot as plt
# data for test
data = [random.randint(1,20) for i in range(20)]
n, x, _ = plt.hist(data)
bin_centers = 0.5*(x[1:]+x[:-1])
plt.plot(bin_centers,n);
for fitting you can extract bins centers and try to fit it with curve_fit:
from scipy.optimize import curve_fit
# some exponential function
def func(x, a, b, c):
return a * np.exp(-b * x) + c
popt, pcov = curve_fit(func, bin_centers, n, bounds=(0, [3., 1., 0.5]))
# bounds are variable, so you can change them as you wish
plt.plot(bin_centers, n, label='data')
plt.plot(bin_centers, func(bin_centers, *popt), label='fit')
plt.legend()
I'm trying to optimize a exponential fitting with scipy.optimize.curve_fit. But the result is no good . My code is :
def func(x, a, b, c):
return a * np.exp(-b * x) + c
# xdata and data is obtain from another dataframe and their type is nparray
xdata =[36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70 ,71,72]
ydata = [4,4,4,6,6,13,22,22,26,28,38,48,55,65,65,92,112,134,171,210,267,307,353,436,669,669,818,1029,1219,1405,1617,1791,2032,2032,2182,2298,2389]
popt, pcov = curve_fit(func, xdata, ydata)
plt.plot(xdata, func(xdata, *popt), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
plt.scatter(xdata, ydata, s=1)
plt.show()
Then I got the result like this:
enter image description here
the result showed that :
pcov = [[inf inf inf] [inf inf inf] [inf inf inf]]
popt = [1 1 611.83784]
I don't know how to make my curve fit well. Can you helo me? Thank you!
Fitting against exponential functions is exceedingly tough because tiny variations in the exponent can make large differences in the result. The optimizer is optimizing across many orders of magnitude, and errors near the origin are not equally weighted compared to errors higher up the curve.
The simplest way to handle this is to convert your exponential data to a line using a transformation:
y' = np.log(y)
Then instead of needing to use the fancier (and slower) curve_fit, you can simply use numpy's polyfit function and fit a line. If you wish, you can transform the data back into linear space for analysis. Here, I've edited your code to do the fit with np.polyfit, and you can see the fit is sensible.
import numpy as np
import matplotlib.pyplot as plt
# from scipy.optimize import curve_fit
# def func(x, a, b, c):
# return a * np.exp(-b * x) + c
# xdata and data is obtain from another dataframe and their type is nparray
xdata = np.array([36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70 ,71,72])
ydata = np.array([4,4,4,6,6,13,22,22,26,28,38,48,55,65,65,92,112,134,171,210,267,307,353,436,669,669,818,1029,1219,1405,1617,1791,2032,2032,2182,2298,2389])
# popt, pcov = curve_fit(func, xdata, ydata)
# plt.plot(xdata, func(xdata, *popt), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
# Fit a line (deg=1)
P, pcov = np.polyfit(xdata, np.log(ydata), deg=1, cov=True)
print(pcov)
plt.scatter(xdata, ydata, s=1)
plt.plot(xdata, np.exp(P[0]*xdata + P[1]), 'r-')
plt.legend()
plt.show()
The method is not finding the optimal point. One thing to try is changing the initial guess so that b starts negative, because it looks from your data that b must be negative so that the func fits it decently. Also, from the docs of curve_fit, the initial guess is 1 by default if not specified. A good initial guess is:
popt, pcov = curve_fit(func, xdata, ydata, p0=[1, -0.05, 1])
which gives
popt
array([ 1.90782987e+00, -1.01639857e-01, -1.73633728e+02])
pcov
array([[ 1.08960274e+00, 7.93580944e-03, -5.24526701e+01],
[ 7.93580944e-03, 5.79450721e-05, -3.74693994e-01],
[-5.24526701e+01, -3.74693994e-01, 3.34388178e+03]])
And the plot
I would like to find and plot a function f that represents a curve fitted on some number of set points that I already know, x and y.
After some research I started experimenting with scipy.optimize and curve_fit but on the reference guide I found that the program uses a function to fit the data instead and it assumes ydata = f(xdata, *params) + eps.
So my question is this: What do I have to change in my code to use the curve_fit or any other library to find the function of the curve using my set points? (note: I want to know the function as well so I can integrate later for my project and plot it). I know that its going to be a decaying exponencial function but don't know the exact parameters. This is what I tried in my program:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.exp(-b * x) + c
xdata = np.array([0.2, 0.5, 0.8, 1])
ydata = np.array([6, 1, 0.5, 0.2])
plt.plot(xdata, ydata, 'b-', label='data')
popt, pcov = curve_fit(func, xdata, ydata)
plt.plot(xdata, func(xdata, *popt), 'r-', label='fit')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
Am currently developing this project on a Raspberry Pi, if it changes anything. And would like to use least squares method since is great and precise, but any other method that works well is welcome.
Again, this is based on the reference guide of scipy library. Also, I get the following graph, which is not even a curve: Graph and curve based on set points
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.exp(-b * x) + c
#c is a constant so taking the derivative makes it go to zero
def deriv(x, a, b, c):
return -a * b * np.exp(-b * x)
#Integrating gives you another c coefficient (offset) let's call it c1 and set it equal to zero by default
def integ(x, a, b, c, c1 = 0):
return -a/b * np.exp(-b * x) + c*x + c1
#There are only 4 (x,y) points here
xdata = np.array([0.2, 0.5, 0.8, 1])
ydata = np.array([6, 1, 0.5, 0.2])
#curve_fit already uses "non-linear least squares to fit a function, f, to data"
popt, pcov = curve_fit(func, xdata, ydata)
a,b,c = popt #these are the optimal parameters for fitting your 4 data points
#Now get more x values to plot the curve along so it looks like a curve
step = 0.01
fit_xs = np.arange(min(xdata),max(xdata),step)
#Plot the results
plt.plot(xdata, ydata, 'bx', label='data')
plt.plot(fit_xs, func(fit_xs,a,b,c), 'r-', label='fit')
plt.plot(fit_xs, deriv(fit_xs,a,b,c), 'g-', label='deriv')
plt.plot(fit_xs, integ(fit_xs,a,b,c), 'm-', label='integ')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
I'm trying to fit my data to (cos(x))^n. The vale of n in theory is 2, but my data should give me around 1.7. When I define my fitting function and I try curve_fit, I get an error
def f(x,a,b,c):
return a+b*np.power(np.cos(x),c)
param, extras = curve_fit(f, x, y)
This is my data
x y error
90 3.3888756187 1.8408898986
60 2.7662844365 1.6632150903
45 2.137309503 1.4619540017
30 1.5256883339 1.2351875703
0 1.4665463518 1.2110104672
The error looks like this:
/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:4:
RuntimeWarning: invalid value encountered in power after removing
the cwd from sys.path.
/usr/lib/python3/dist-packages/scipy/optimize/minpack.py:690:
OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)
The problem is that cos(x) can become negative and then cos(x) ^ n can be undefined. Illustration:
np.cos(90)
-0.44807361612917013
and e.g.
np.cos(90) ** 1.7
nan
That causes the two error messages you receive.
It works fine, if you modify your model, e.g. to a + b * np.cos(c * x + d). Then the plot looks as follows:
The code can be found below with some inline comments:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def f(x, a, b, c, d):
return a + b * np.cos(c * x + d)
# your data
xdata = [90, 60, 45, 30, 0]
ydata = [3.3888756187, 2.7662844365, 2.137309503, 1.5256883339, 1.4665463518]
# plot data
plt.plot(xdata, ydata, 'bo', label='data')
# fit the data
popt, pcov = curve_fit(f, xdata, ydata, p0=[3., .5, 0.1, 10.])
# plot the result
xdata_new = np.linspace(0, 100, 200)
plt.plot(xdata_new, f(xdata_new, *popt), 'r-', label='fit')
plt.legend(loc='best')
plt.show()