I would like to find and plot a function f that represents a curve fitted on some number of set points that I already know, x and y.
After some research I started experimenting with scipy.optimize and curve_fit but on the reference guide I found that the program uses a function to fit the data instead and it assumes ydata = f(xdata, *params) + eps.
So my question is this: What do I have to change in my code to use the curve_fit or any other library to find the function of the curve using my set points? (note: I want to know the function as well so I can integrate later for my project and plot it). I know that its going to be a decaying exponencial function but don't know the exact parameters. This is what I tried in my program:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.exp(-b * x) + c
xdata = np.array([0.2, 0.5, 0.8, 1])
ydata = np.array([6, 1, 0.5, 0.2])
plt.plot(xdata, ydata, 'b-', label='data')
popt, pcov = curve_fit(func, xdata, ydata)
plt.plot(xdata, func(xdata, *popt), 'r-', label='fit')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
Am currently developing this project on a Raspberry Pi, if it changes anything. And would like to use least squares method since is great and precise, but any other method that works well is welcome.
Again, this is based on the reference guide of scipy library. Also, I get the following graph, which is not even a curve: Graph and curve based on set points
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.exp(-b * x) + c
#c is a constant so taking the derivative makes it go to zero
def deriv(x, a, b, c):
return -a * b * np.exp(-b * x)
#Integrating gives you another c coefficient (offset) let's call it c1 and set it equal to zero by default
def integ(x, a, b, c, c1 = 0):
return -a/b * np.exp(-b * x) + c*x + c1
#There are only 4 (x,y) points here
xdata = np.array([0.2, 0.5, 0.8, 1])
ydata = np.array([6, 1, 0.5, 0.2])
#curve_fit already uses "non-linear least squares to fit a function, f, to data"
popt, pcov = curve_fit(func, xdata, ydata)
a,b,c = popt #these are the optimal parameters for fitting your 4 data points
#Now get more x values to plot the curve along so it looks like a curve
step = 0.01
fit_xs = np.arange(min(xdata),max(xdata),step)
#Plot the results
plt.plot(xdata, ydata, 'bx', label='data')
plt.plot(fit_xs, func(fit_xs,a,b,c), 'r-', label='fit')
plt.plot(fit_xs, deriv(fit_xs,a,b,c), 'g-', label='deriv')
plt.plot(fit_xs, integ(fit_xs,a,b,c), 'm-', label='integ')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
Related
I'm trying to optimize a exponential fitting with scipy.optimize.curve_fit. But the result is no good . My code is :
def func(x, a, b, c):
return a * np.exp(-b * x) + c
# xdata and data is obtain from another dataframe and their type is nparray
xdata =[36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70 ,71,72]
ydata = [4,4,4,6,6,13,22,22,26,28,38,48,55,65,65,92,112,134,171,210,267,307,353,436,669,669,818,1029,1219,1405,1617,1791,2032,2032,2182,2298,2389]
popt, pcov = curve_fit(func, xdata, ydata)
plt.plot(xdata, func(xdata, *popt), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
plt.scatter(xdata, ydata, s=1)
plt.show()
Then I got the result like this:
enter image description here
the result showed that :
pcov = [[inf inf inf] [inf inf inf] [inf inf inf]]
popt = [1 1 611.83784]
I don't know how to make my curve fit well. Can you helo me? Thank you!
Fitting against exponential functions is exceedingly tough because tiny variations in the exponent can make large differences in the result. The optimizer is optimizing across many orders of magnitude, and errors near the origin are not equally weighted compared to errors higher up the curve.
The simplest way to handle this is to convert your exponential data to a line using a transformation:
y' = np.log(y)
Then instead of needing to use the fancier (and slower) curve_fit, you can simply use numpy's polyfit function and fit a line. If you wish, you can transform the data back into linear space for analysis. Here, I've edited your code to do the fit with np.polyfit, and you can see the fit is sensible.
import numpy as np
import matplotlib.pyplot as plt
# from scipy.optimize import curve_fit
# def func(x, a, b, c):
# return a * np.exp(-b * x) + c
# xdata and data is obtain from another dataframe and their type is nparray
xdata = np.array([36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70 ,71,72])
ydata = np.array([4,4,4,6,6,13,22,22,26,28,38,48,55,65,65,92,112,134,171,210,267,307,353,436,669,669,818,1029,1219,1405,1617,1791,2032,2032,2182,2298,2389])
# popt, pcov = curve_fit(func, xdata, ydata)
# plt.plot(xdata, func(xdata, *popt), 'r-', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
# Fit a line (deg=1)
P, pcov = np.polyfit(xdata, np.log(ydata), deg=1, cov=True)
print(pcov)
plt.scatter(xdata, ydata, s=1)
plt.plot(xdata, np.exp(P[0]*xdata + P[1]), 'r-')
plt.legend()
plt.show()
The method is not finding the optimal point. One thing to try is changing the initial guess so that b starts negative, because it looks from your data that b must be negative so that the func fits it decently. Also, from the docs of curve_fit, the initial guess is 1 by default if not specified. A good initial guess is:
popt, pcov = curve_fit(func, xdata, ydata, p0=[1, -0.05, 1])
which gives
popt
array([ 1.90782987e+00, -1.01639857e-01, -1.73633728e+02])
pcov
array([[ 1.08960274e+00, 7.93580944e-03, -5.24526701e+01],
[ 7.93580944e-03, 5.79450721e-05, -3.74693994e-01],
[-5.24526701e+01, -3.74693994e-01, 3.34388178e+03]])
And the plot
I am trying to predict the trend of an internet post.
I have available the number of comments and votes the post has after 2 minutes of being posted (can change, but it should be enough).
Currently I use this formula:
predicted_votes = (votes_per_minute + n_comments * 60 * h) * k
And then I find k experimentally. I get the post data, wait an hour, do
k = (older_k + actual_votes/predicted_votes) / 2
And so on. This kind of works. The accuracy is pretty low (40 - 50%), but it gives me a rough idea on how the post is going to react.
I was wondering if I could employ a more complex equation, something like:
predicted_votes = ((votes_per_minute * x + n_comments * y) * 60 * hour) * k # Hour stands for 'how many hours to predict'
And then optimize the parameters to approximate a bit better.
I would assume that I could use Machine Learning, although I don't have a GPU available (that's right, I'm running on integrated graphics, blame Mojave), so I am trying this approach instead.
So the question boils down to, how do I optimize those parameters (k,x,y) to get a better accuracy?
EDIT:
I tried following what #Alexis said, and this is where I am at right now:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
initial_votes_list = [1.41, 0.9, 0.94, 0.47, 0]
initial_comment_list = [0, 3, 0, 1, 64]
def func(x, k, t, s):
votes_per_minute = x[0]
n_comments = x[1]
return ((votes_per_minute * t + n_comments * s) * 60) * k
xdata = [1.41,0]
y = func(xdata, 2.5, 1.3, 0.5)
np.random.seed(1729)
ydata = y + 5
plt.plot(xdata, ydata, 'b-', label='data')
popt, pcov = curve_fit(func, xdata, ydata)
plt.plot(xdata, func(xdata, *popt), 'g--',
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
plt.xlabel('Time')
plt.ylabel('Score')
plt.legend()
plt.show()
I am not sure how to feed the data I have (votes_per_minute, n_comments), nor how I could tell the algorithm that y axis is actually time.
EDIT 2:
Tried doing what #Alexis told me, but I am unsure what to use as actual_score, a number doesn't work, a list neither.. Also, I want to predict the 'score' not the number of comments.
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
initial_votes_list = [1.41, 0.9, 0.94, 0.47, 0]
initial_comment_list = [0, 3, 0, 1, 64]
final_score = [26,12,13,14,229]
def func(x,k,t,s):
return ((x[0]*k+x[1]*t)*60*x[2])*s
X = [[a,b,c] for a,b,c in zip(initial_votes_list,initial_comment_list,[i for i in range(len(initial_votes_list))])]
y = actual_votes # What is this?
popt, pcov = curve_fit(func, X, y)
plt.plot(xdata, func(xdata, *popt), 'g--',
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
plt.xlabel('Time')
plt.ylabel('Score')
plt.legend()
plt.show()
you don't need ML to do so (overkill i think here). Scipy provides a nice and easy way to fit a curve to the observations you have.
scipy.optimize.curve_fit allows you to fit a function with unknown parameters to your observation. As you already know the general form of the function, optimizing the hyper parameters is a well known stat problem and thus scipy should be enough.
We can take a small example to demonstrate this:
first we generate the datas
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> from scipy.optimize import curve_fit
>>>
>>> def func(x, a, b, c):
... return a * np.exp(-b * x) + c
Define the data to be fit with some noise:
>>> xdata = np.linspace(0, 4, 50)
>>> y = func(xdata, 2.5, 1.3, 0.5)
>>> np.random.seed(1729)
>>> y_noise = 0.2 * np.random.normal(size=xdata.size)
>>> ydata = y + y_noise
>>> plt.plot(xdata, ydata, 'b-', label='data')
then we fit the function (ax+b=y) to the data using scipy:
popt, pcov = curve_fit(func, xdata, ydata)
you could add constraints to this, but for your problem it is not necessary.
By the way, this example is at the end of the link i provided. Everything you should know to use the curve fit is available on this page.
Edit
it seems you have a hard time figuring out how to use this. Let's go slowly and analytically to make sure we are ok every step of the way:
you want to predict the number of comment, this is your y. It is known. not calculated
you have in entry three parameters: the votes_per_minute , the n_comments and the hour h
and last but not least, you have three parameters to a function (x,y,k)
so X[i] (one sample) should look like this: [votes_per_minute,n_comments,h]
and with your formula y = ((votes_per_minute * k + n_comments * t) * 60 * h) * s, by replacing the names:
def func(x,k,t,s):
return ((x[0]*k+x[1]*t)*60*x[2])*s
X = np.array([[a,b,c] for a,b,c in zip(initial_votes_list,initial_comment_list,[i for i in range(len(initial_votes_list))])]).T
y = score
and then:
popt, pcov = curve_fit(func, X, y)
(if i understand your issue...if not, i don't see where the problem is)
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
initial_votes_list = [1.41, 0.9, 0.94, 0.47, 0]
initial_comment_list = [0, 3, 0, 1, 64]
final_score = [26,12,13,14,229]
def func(x,k,t,s):
return ((x[0]*k+x[1]*t)*60*x[2])*s
X = np.array([[a,b,c] for a,b,c in zip(initial_votes_list,initial_comment_list,[i for i in range(len(initial_votes_list))])]).T
y = [0.12,0.20,0.5,0.9,1]
popt, pcov = curve_fit(func, X, y)
print(popt)
>>>[-6.65969099e+00 -6.99241803e-02 -9.33412000e-04]
I am trying to fit a curve to a set of data points:
def func(x, a, b, c):
return a * np.log(-b * x) + c
#return a * np.exp(-b * x) + c
xdata = np.array(X)
ydata = np.array(y)
popt, pcov = curve_fit(func, xdata, ydata)
plt.scatter(xdata, ydata, label='data')
plt.plot(xdata, func(xdata, *popt), 'r', label='fit')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
plt.show()
The data set contains about 50 points. 2 of them could be outliers. I generated two plots of the data and fitted curve: the first one contains the outliers and the other plot excludes the outliers:
However, both curve fits contain many NaN values, which is why the red fitted line is so small. I got values of 1 for each variable in popt. I tried both the log and exp fits as seen in the code above
Are there better curves than exponential or log fits that I can try?
EDIT: definition of func comes before the curve_fit call
I'm trying to fit my data to (cos(x))^n. The vale of n in theory is 2, but my data should give me around 1.7. When I define my fitting function and I try curve_fit, I get an error
def f(x,a,b,c):
return a+b*np.power(np.cos(x),c)
param, extras = curve_fit(f, x, y)
This is my data
x y error
90 3.3888756187 1.8408898986
60 2.7662844365 1.6632150903
45 2.137309503 1.4619540017
30 1.5256883339 1.2351875703
0 1.4665463518 1.2110104672
The error looks like this:
/usr/local/lib/python3.5/dist-packages/ipykernel_launcher.py:4:
RuntimeWarning: invalid value encountered in power after removing
the cwd from sys.path.
/usr/lib/python3/dist-packages/scipy/optimize/minpack.py:690:
OptimizeWarning: Covariance of the parameters could not be estimated
category=OptimizeWarning)
The problem is that cos(x) can become negative and then cos(x) ^ n can be undefined. Illustration:
np.cos(90)
-0.44807361612917013
and e.g.
np.cos(90) ** 1.7
nan
That causes the two error messages you receive.
It works fine, if you modify your model, e.g. to a + b * np.cos(c * x + d). Then the plot looks as follows:
The code can be found below with some inline comments:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def f(x, a, b, c, d):
return a + b * np.cos(c * x + d)
# your data
xdata = [90, 60, 45, 30, 0]
ydata = [3.3888756187, 2.7662844365, 2.137309503, 1.5256883339, 1.4665463518]
# plot data
plt.plot(xdata, ydata, 'bo', label='data')
# fit the data
popt, pcov = curve_fit(f, xdata, ydata, p0=[3., .5, 0.1, 10.])
# plot the result
xdata_new = np.linspace(0, 100, 200)
plt.plot(xdata_new, f(xdata_new, *popt), 'r-', label='fit')
plt.legend(loc='best')
plt.show()
I'm trying to optimize a logarithmic fit to a data set with scipy.optimize.curve_fit. Before trying it on an actual data set, I wrote code to run on a dummy data set.
def do_fitting():
x = np.linspace(0, 4, 100)
y = func(x, 1.1, .4, 5)
y2 = y + 0.2 * np.random.normal(size=len(x))
popt, pcov = curve_fit(func, x, y2, p0=np.array([2, 0.5, 1]))
plt.figure()
plt.plot(x, y, 'bo', label="Clean Data")
plt.plot(x, y2, 'ko', label="Fuzzed Data")
plt.plot(x, func(x, *popt), 'r-', label="Fitted Curve")
plt.legend()
plt.show()
Of course, do_fitting() relies on func(), which it passes to curve_fit. Here's the problem. When I pass a func() that contains np.log, i.e. the function that I actually want to fit to, curve_fit declares that p0 (the initial condition) is the optimal solution and returns immediately with an infinite covariance.
Here's what happens if I run do_fitting() with a non-logarithmic func():
def func(x, a, b, c):
return a * np.exp(x*b) + c
popt = [ 0.90894173 0.44279212 5.19928151]
pcov = [[ 0.02044817 -0.00471525 -0.02601574]
[-0.00471525 0.00109879 0.00592502]
[-0.02601574 0.00592502 0.0339901 ]]
Here's what happens when I run do_fitting() with a logarithmic func():
def func(x, a, b, c):
return a * np.log(x*b) + c
popt = [ 2. 0.5 1. ]
pcov = inf
You'll notice that the logarithmic solution for popt is equal to the value I gave curve_fit for p0 in the above do_fitting(). This is true, and pcov is infinite, for every value of p0 I have tried.
What am I doing wrong here?
The problem is very simple - since the first value in your x array is 0, you are taking the log of 0, which is equal to -inf:
x = np.linspace(0, 4, 100)
p0 = np.array([2, 0.5, 1])
print(func(x, *p0).min())
# -inf
I was able to fit a logarithmic function just fine using the following code (hardly modified from your original):
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
def func(x, a, b, c):
return a * np.log(x+b) + c
def do_fitting():
x = np.linspace(0, 4, 100)
y = func(x, 1.1, .4, 5)
y2 = y + 0.2 * np.random.normal(size=len(x))
popt, pcov = curve_fit(func, x, y2, p0=np.array([2, 0.5, 1]))
plt.figure()
plt.plot(x, y, 'bo', label="Clean Data")
plt.plot(x, y2, 'ko', label="Fuzzed Data")
plt.plot(x, func(x, *popt), 'r-', label="Fitted Curve")
plt.legend()
plt.show()
do_fitting()
(Unfortunately I can't post a picture of the final fit, but it agrees quite nicely with the clean data).
Likely your problem is not the logarithm itself, but some difficulty curve_fit is having with the specific function you're trying to fit. Can you edit your question to provide an example of the exact logarithmic function you're trying to fit?
EDIT: The function you provided is not well-defined for x=0, and produces a RuntimeWarning upon execution. curve_fit is not good at handling NaNs, and will not be able to fit the function in this case. If you change x to
x = np.linspace(1, 4, 100)
curve_fit performs just fine.