I am trying to fit gaussian to a spectrum and the y values are on the order of 10^(-19). Curve_fit gives me poor fitting result, both before and after I multiply my whole data by 10^(-19). Attached is my code, it is fairly simple set of data except that the values are very small. If I want to keep my original values, how would I get a reasonable gaussian fit that would give me the correct parameters?
#get fits data
aaa=pyfits.getdata('p1.cal.fits')
aaa=np.matrix(aaa)
nrow=np.shape(aaa)[0]
ncol=np.shape(aaa)[1]
ylo=79
yhi=90
xlo=0
xhi=1023
glo=430
ghi=470
#sum all the rows to get spectrum
ysum=[]
for x in range(xlo,xhi):
sum=np.sum(aaa[ylo:yhi,x])
ysum.append(sum)
wavelen_pix=range(xhi-xlo)
max=np.max(ysum)
print "maximum is at x=", np.where(ysum==max)
##fit gaussian
#fit only part of my data in the chosen range [glo:ghi]
x=wavelen_pix[glo:ghi]
y=ysum[glo:ghi]
def func(x, a, x0, sigma):
return a*np.exp(-(x-x0)**2/float((2*sigma**2)))
sig=np.std(ysum[500:1000]) #std of background noise
popt, pcov = curve_fit(func, x, sig)
print popt
#this gives me [1.,1.,1.], which is obviously wrong
gaus=func(x,popt[0],popt[1],popt[2])
aaa is a 153 by 1024 image matrix, partly looks like this:
matrix([[ -8.99793629e-20, 8.57133275e-21, 4.83523386e-20, ...,
-1.54811004e-20, 5.22941515e-20, 1.71179195e-20],
[ 2.75769318e-20, 1.03177243e-20, -3.19634928e-21, ...,
1.66583803e-20, -9.88712568e-22, -2.56897725e-20],
[ 2.88121935e-20, 8.57964252e-21, -2.60784327e-20, ...,
1.72335180e-20, -7.61189937e-21, -3.45333075e-20],
...,
[ 1.04006903e-20, 1.61200683e-20, 7.04195205e-20, ...,
1.72459645e-20, 4.29404029e-20, 1.99889374e-20],
[ 3.22315752e-21, -5.61394194e-21, 3.28763096e-20, ...,
1.99063583e-20, 2.12989880e-20, -1.23250648e-21],
[ 3.66591810e-20, -8.08647455e-22, -6.22773168e-20, ...,
-4.06145681e-21, 4.92453132e-21, 4.23689309e-20]], dtype=float32)
You are calling curve_fit incorrectly, here is the usage
curve_fit(f, xdata, ydata, p0=None, sigma=None, absolute_sigma=False, check_finite=True, **kw)
f is your function whose first arg is an array of independent variables, and whose subsequent args are the function parameters (such as amplitude, center, etc)
xdata are the independent variables
ydata are the dependedent variable
p0 is an initial guess at the function parameters (for Guassian this is amplitude, width, center)
By default p0 is set to a list of ones [1,1,...], which is probably why you get that as a result, the fit just never executed because you called it incorrectly.
Try estimating the amplitude, center, and width from the data, then make a p0 object (see below for details)
init_guess = ( a_i, x0_i, sig_i) # same order as they are supplied to your function
popt, pcov = curve_fit(func, xdata=x,ydata=y,p0=init_guess)
Here is a short example
xdata = np.linspace(0, 4, 50)
mygauss = ( 10,2,0.5) #( amp, center, width)
y = func(xdata, *mygauss ) # using your func defined above
ydata = y + 2*(np.random.random(50)- 0.5) # add some noise to create fake data
Now I can guess the fit params
ai = np.max( ydata) # guess the amplitude
xi = xdata[ np.argmax( ydata)] # guess the position of center
Guessing the width is tricky, I would first find where the half max is located (there are two, but you only need to find one, as the Gaussian is symmetric):
pos_half = argmin( np.abs( ydata-ao/2 ) ) # subtract half the amplitude and find the minimum
Now evaluate how far this is from the center of the gaussian (xi) :
sig_i = np.abs( xi - xdata[ pos_half] ) # estimate the width
Now you can make make the initial guess
init_guess = (ai, xi sig_i)
and fit
params, variance = curve_fit( func, xdata=xdata, ydata=ydata, p0=init_guess)
print params
#array([ 9.99457443, 2.01992858, 0.49599629])
which is very close to mygauss. Hope it helps.
Forget about rescaling, or making linear changes, or using the p0 parameter, which usually don't work! Try using the bounds parameter in the curve_fit for n parameters like this:
a0=np.array([a01,...,a0n])
af=np.array([af1,...,afn])
method="trf",bounds=(a0,af)
Hope it works!
;)
Related
I have been trying to fit a function(the function is given in the code under the name concave_func) onto data points in python but have had very little to no success. I have 7 parameters(C_1, C_2, alpha_one, alpha_two, I_x, nu_t, T_e) in the function that I have to estimate, and only 6 data points. I have tried 2 methods to fit the curve and estimate the parameters,
1). scipy.optimize.minimize
2). scipy.optimize.curve_fit.
However, I'm not obtaining the desired results i.e the curve is not fitting the data points.
I have attached my code below.
frequency = np.array([22,45,150,408,1420,23000]) #x_values
b_temp = [2.55080863e+04, 4.90777800e+03, 2.28984753e+02, 2.10842949e+01, 3.58631166e+00, 5.68716056e-04] #y_values
#Defining the function that I want to fit
def concave_func(x, C_1, C_2, alpha_one, alpha_two, I_x, nu_t, T_e):
one = x**(-alpha_one)
two = (C_2/C_1)*(x**(-alpha_two))
three = I_x*(x**-2.1)
expo = np.exp(-1*((nu_t/x)**2.1))
eqn_one = C_1*(one + two + three)*expo
eqn_two = T_e*(1 - expo)
return eqn_one + eqn_two
#Defining chi_square function
def chisq(params, xobs, yobs):
ynew = concave_func(xobs, *params)
#yerr = np.sum((ynew- yobs)**2)
yerr = np.sum(((yobs- ynew)/ynew)**2)
print(yerr)
return yerr
result = minimize(chisq, [1,2,2,2,1,1e6,8000], args = (frequency,b_temp), method = 'Nelder-Mead', options = {'disp' : True, 'maxiter': 10000})
x = np.linspace(-300,24000,1000)
plt.yscale("log")
plt.xscale("log")
plt.plot(x,concave_func(x, *result.x))
print(result.x)
print(result)
plt.plot(frequency, b_temp, 'r*' )
plt.xlabel("log Frequency[MHz]")
plt.ylabel("log Temp[K]")
plt.title('log Temparature vs log Frequency')
plt.grid()
plt.savefig('the_plot_2060.png')
I have attached the plot that I obtained below.
The plot clearly does not fit the data, and something is definitely wrong. I would also want my parameters alpha_one and alpha_two to be constrained to lie between 2 and 3. I also do not want my parameter T_e to exceed 10,000. Any thoughts?
I have been struggling for apparently no reason trying to fit a sin function to a small dataset that resembles a sinusoid. I've looked at many other questions and tried different libraries and can't seem to find any glaring mistake in my code. Also in many answers people are fitting a function onto data where y = f(x); but I'm retrieving both of my lists independently from stellar spectra.
These are the lists for reference:
time = np.array([2454294.5084288 , 2454298.37039515, 2454298.6022165 ,
2454299.34790096, 2454299.60750029, 2454300.35176022,
2454300.61361622, 2454301.36130122, 2454301.57111912,
2454301.57540159, 2454301.57978822, 2454301.5842906 ,
2454301.58873511, 2454302.38635047, 2454302.59553152,
2454303.41548415, 2454303.56765036, 2454303.61479213,
2454304.38528718, 2454305.54043812, 2454306.36761011,
2454306.58025083, 2454306.60772791, 2454307.36686591,
2454307.49460991, 2454307.58258509, 2454308.3698358 ,
2454308.59468672, 2454309.40004997, 2454309.51208756,
2454310.43078368, 2454310.6091061 , 2454311.40121502,
2454311.5702085 , 2454312.39758274, 2454312.54580053,
2454313.52984047, 2454313.61734047, 2454314.37609003,
2454315.56721061, 2454316.39218499, 2454316.5672538 ,
2454317.49410168, 2454317.6280825 , 2454318.32944441,
2454318.56913047])
velocities = np.array([-2.08468951, -2.26117398, -2.44703149, -2.10149768, -2.09835213,
-2.20540079, -2.4221183 , -2.1394637 , -2.0841663 , -2.2458154 ,
-2.06177386, -2.47993416, -2.13462117, -2.26602791, -2.47359571,
-2.19834895, -2.17976339, -2.37745005, -2.48849617, -2.15875901,
-2.27674409, -2.39054554, -2.34029665, -2.09267843, -2.20338104,
-2.49483926, -2.08860222, -2.26816951, -2.08516229, -2.34925637,
-2.09381667, -2.21849357, -2.43438148, -2.28439031, -2.43506056,
-2.16953358, -2.24405359, -2.10093237, -2.33155007, -2.37739938,
-2.42468714, -2.19635302, -2.368558 , -2.45959665, -2.13392004,
-2.25268181]
These are radial velocities of a star observed at different times. When plotted they look like this:
Plotted Data
This is then the code I'm using to fit a test sine on the data:
x = time
y = velocities
def sin_fit(x, A, w):
return A * np.sin(w * x)
popt, pcov = curve_fit(sin_fit,x,y) #try to calculate exoplanet parameters with these data
xfit = np.arange(min(x),max(x),0.1)
fit = sin_fit(xfit,*popt)
mod = plt.figure()
plt.xlabel("Time (G. Days)")
plt.ylabel("Radial Velocity")
plt.scatter(x,[i for i in y],color="b",label="Data")
plt.plot(x,[i for i in y],color="b",alpha=0.2)
plt.plot(xfit,fit,color="r",label="Model Fit")
plt.legend()
mod.savefig("Data with sin fit.png")
plt.show()
I thought this was right, and it seems right by looking at other answers, but then this is what I get:
Data with model sine
What am I doing wrong?
Thank you in advanceee
I guess it's due the sin_fit function is not able to fit the data at all. The sin function per default whirls around y=0 while your data whirls somewhere around y=-2.3.
I tried your code and extended the sin_fit with an offset, yielding way better results (althought looking not too perfect):
def sin_fit(x, A, w, offset):
return A * np.sin(w * x) + offset
with this the function has at least a chance to fit
I'm trying to fit a lorentzian to one of the peaks in my dataset.
We were given the fit for a gaussian, and aside from the actual fit equation, the code is very similar, so I'm not sure where I am going wrong. I don't see why there is an issue with the dimensions when I'm using curve_fit.
Here are the relevant pieces of my code for a better idea of what I'm talking about.
Reading the CSV file in and trimming it
import csv
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from matplotlib.ticker import StrMethodFormatter
#reading in the csv file
with open("Data-Oscilloscope.csv") as csv_file:
csv_reader = csv.reader(csv_file, delimiter=",")
time =[]
voltage_raw = []
for row in csv_reader:
time.append(float(row[3]))
voltage_raw.append(float(row[4]))
print("voltage:", row[4])
#trimming the data
trim_lower_index = 980
trim_upper_index = 1170
time_trim = time[trim_lower_index:trim_upper_index]
voltage_trim = voltage_raw[trim_lower_index:trim_upper_index]
The Gaussian fit given
#fitting the gaussian function
def gauss_function(x, a, x0, sigma):
return a*np.exp(-(x-x0)**2/(2*sigma**2))
popt, pcov = curve_fit(gauss_function, time_trim, voltage_trim, p0=[1,.4,0.1])
perr = np.sqrt(np.diag(pcov))
#plot of the gaussian fit
plt.figure(2)
plt.plot(time_trim, gauss_function(time_trim, *popt), label = "fit")
plt.plot(time_trim, voltage_trim, "-b")
plt.show()
My attempted Lorentzian fit
#x is just the x values, a is the amplitude, x0 is the central value, and f is the full width at half max
def lorentz_function(x, a, x0,f):
w = f/2 #half width at half max
return a*w/ [(x-x0)**2+w**2]
popt, pcov = curve_fit(lorentz_function, time_trim, voltage_trim, p0=[1,.4,0.1])
I get an error running this that states:
in leastsq raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))
TypeError: Improper input: N=3 must not exceed M=1
I'm probably missing something very obvious but just can't see it.
Thanks in advance!
EDIT: I have taken a look at other, similar questions and went through their explanations, but can't see how those fit in with my code because the number of parameters and dimensions of my inputs should be fine, given that they worked for the gaussian fit.
You didn't show the full traceback/error so I can only guess where it's happening. It's probably looking at the result returned by lorentz_function, and finding the dimensions to be wrong. So while the error is produced by your function, the testing is in its caller (in this case a level or two down).
def optimize.curve_fit(
f,
xdata,
ydata,
p0=None,... # p0=[1,.4,0.1]
...
res = leastsq(func, p0,
...
curve_fit passes the task to leastsq, which starts as:
def optimize.leastsq(
func,
x0,
args=(), ...
x0 = asarray(x0).flatten()
n = len(x0)
if not isinstance(args, tuple):
args = (args,)
shape, dtype = _check_func('leastsq', 'func', func, x0, args, n)
m = shape[0]
if n > m:
raise TypeError('Improper input: N=%s must not exceed M=%s' % (n, m))
I'm guessing _check_func does
res = func(x0, *args) # call your func with initial values
and returning the shape of res. The error says there's a mismatch between what it expects based on the shape of x0 and the result of your func.
I'm guessing that with a 3 element p0, it's complaining that your function returned a 1 element result (due to the []).
lorentz is your function. You don't test the output shape so it can't raise this error.
I had a similar problem yielding this error message.
In that case, the array of data passed to the optimize.leastsq was in the matrix form. the data seems to have to be a 1 row array.
For example, the sentence calling the leastsq was
result = optimize.leastsq(fit_func, param0, args=(xdata, ydata, zdata))
xdata, ydata, zdata was in the [ 1 x num ] matrix form. It was
[[200. .... 350.]]
It should be
[200. .... 350.]
So, I had to add the sentence for conversion
xdata = xdata[0, :]
So do ydata and zdata.
I hope this be a help for you.
in the framework of my bachelor's thesis, I need to evaluate my data with python. Unfortunately there's no suiting script of my fellow students yet and I'm quite new to programming.
I have this data set and I'm trying to fit it with a gaussian by using scipy.optimize.curve_fit. Since there are a lot of unusable counts especially at the end of the axis, I'd like to confine the part that is to be fitted.
Picture raw data
This is what I have so far:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
x=np.arange(5120)
y=array([ 0.81434599, 1.17054264, 0.85279188, ..., 1. ,
1. , 13.56291391]) #most of the data isn't interesting
#to me, part of interest see below
def Gauss(x, a, x0, sigma):
return a * np.exp(-(x - x0)**2 / (2 * sigma**2))
mean = sum(x * y) / sum(y)
sigma = np.sqrt(sum(y * (x - mean)**2) / sum(y))
popt,pcov = curve_fit(Gauss, x, y, p0=[max(y), mean, sigma],
maxfev=360000)
plt.plot(x,y,label='data')
plt.plot(x,Gauss(x, *popt), 'r-',label='fit')
On docs.scipy.org I've found a general description about curve_fit
If I try using
bounds=([2400,-np.inf, -np.inf],[2600, np.inf, np.inf]),
I'm getting the ValueError: x0 is infeasible. What is the problem here?
I also tried to confine it with
popt,pcov = curve_fit(Gauss, x[2400:2600], y[2400:2600], p0=[max(y), mean, sigma], maxfev=360000)
as suggested in a comment on this question: "Error when obtaining gaussian fit for graph" at stackoverflow
In this case I only get a straight line though.
Picture: Confinement with x[2400:2600],y[2400:2600] as arguments of curve_fit
I really hope you can help me out here. I only need a way to fit a small part of my data. Thanks in advance!
interesting data:
y=array([ 0.93396226, 1.00884956, 1.15457413, 1.07590759,
0.88915094, 1.07142857, 1.10714286, 1.14171123, 1.06666667,
0.84975369, 0.95480226, 0.99388379, 1.01675978, 0.83967391,
0.9771987 , 1.02402402, 1.04531722, 1.07492795, 0.97135417,
0.99714286, 1.0248139 , 1.26223776, 1.1533101 , 0.99099099,
1.18867925, 1.15772871, 0.95076923, 1.03313253, 1.02278481,
0.93265993, 1.06705539, 1.00265252, 1.02023121, 0.92076503,
0.99728997, 1.03353659, 1.15116279, 1.04336043, 0.95076923,
1.05515588, 0.92571429, 0.93448276, 1.02702703, 0.90056818,
0.96068796, 1.08493151, 1.13584906, 1.1212938 , 1.0739645 ,
0.98972603, 0.94594595, 1.07913669, 0.98425197, 0.87762238,
0.96811594, 1.02710843, 0.99392097, 0.91384615, 1.09809264,
1.00630915, 0.93175074, 0.87572254, 1.00651466, 0.78772379,
1.12244898, 1.2248062 , 0.97109827, 0.94607843, 0.97900262,
0.97527473, 1.01212121, 1.16422287, 1.20634921, 0.97275204,
1.01090909, 0.99404762, 1.00561798, 1.01146132, 1.08695652,
0.97214485, 1.03525641, 0.99096386, 1.05135952, 1.16451613,
0.90462428, 0.76876877, 0.47701149, 0.27607362, 0.21580547,
0.20598007, 0.16766467, 0.15533981, 0.19745223, 0.15407855,
0.18925831, 0.26997245, 0.47603834, 0.596875 , 0.85126582, 0.96
, 1.06578947, 1.08761329, 0.89548023, 0.99705882, 1.07142857,
0.95677233, 0.86119874, 1.02857143, 0.98250729, 0.94214876,
1.04166667, 0.96024465, 1.07022472, 1.10344828, 1.04859335,
0.96655518, 1.06424581, 1.01754386, 1.03492063, 1.18627451,
0.91036415, 1.03355705, 1.09116809, 0.96083551, 1.01298701,
1.03691275, 1.02923977, 1.11612903, 1.01457726, 1.06285714,
0.98186528, 1.16470588, 0.86645963, 1.07317073, 1.09615385,
1.21192053, 0.94385027, 0.94244604, 0.88390501, 0.95718654,
0.9691358 , 1.01729107, 1.01119403, 1.20350877, 1.12890625,
1.06940063, 0.90410959, 1.14662757, 0.97093023, 1.03021148,
1.10629921, 0.97118156, 1.10693642, 1.07917889, 0.9484127 ,
1.07581227, 0.98006645, 0.98986486, 0.90066225, 0.90066225,
0.86779661, 0.86779661, 0.96996997, 1.01438849, 0.91186441,
0.91290323, 1.03745318, 1.0615942 , 0.97202797, 1.16608997,
0.94182825, 1.08333333, 0.9076087 , 1.18181818, 1.20618557,
1.01273885, 0.93606138, 0.87457627, 0.90575916, 1.09756098,
0.99115044, 1.13380282, 1.04333333, 1.04026846, 1.0297619 ,
1.04334365, 1.03395062, 0.92553191, 0.98198198, 1. ,
0.9439528 , 1.02684564, 1.1372549 , 0.96676737, 0.99649123,
1.07051282, 1.10367893, 1.0866426 , 1.15384615, 0.99667774])
You might find the lmfit module (https://lmfit.github.io/lmfit-py/) useful for this. It is designed to make curve fitting very easy, has built-in models for common peaks like Gaussian, and has many useful features such as allowing you to set bounds on parameters. A fit to your data with lmfit might look like this:
import numpy as np
import matplotlib.pyplot as plt
from lmfit.models import GaussianModel, ConstantModel
y = np.array([.....]) # uses your shorter data range
x = np.arange(len(y))
# make a model that is a Gaussian + a constant:
model = GaussianModel(prefix='peak_') + ConstantModel()
# make parameters with starting values:
params = model.make_params(c=1.0, peak_center=90,
peak_sigma=5, peak_amplitude=-5)
# it's not really needed for this data, but you can put bounds on
# parameters like this (or set .vary=False to fix a parameter)
params['peak_sigma'].min = 0 # sigma > 0
params['peak_amplitude'].max = 0 # amplitude < 0
params['peak_center'].min = 80
params['peak_center'].max = 100
# run fit
result = model.fit(y, params, x=x)
# print, plot results
print(result.fit_report())
plt.plot(x, y)
plt.plot(x, result.best_fit)
plt.show()
This will print out
[[Model]]
(Model(gaussian, prefix='peak_') + Model(constant))
[[Fit Statistics]]
# function evals = 54
# data points = 200
# variables = 4
chi-square = 1.616
reduced chi-square = 0.008
Akaike info crit = -955.625
Bayesian info crit = -942.432
[[Variables]]
peak_sigma: 4.03660814 +/- 0.204240 (5.06%) (init= 5)
peak_center: 91.2246614 +/- 0.200267 (0.22%) (init= 90)
peak_amplitude: -9.79111362 +/- 0.445273 (4.55%) (init=-5)
c: 1.02138228 +/- 0.006796 (0.67%) (init= 1)
peak_fwhm: 9.50548558 +/- 0.480950 (5.06%) == '2.3548200*peak_sigma'
peak_height: -0.96766623 +/- 0.041854 (4.33%) == '0.3989423*peak_amplitude/max(1.e-15, peak_sigma)'
[[Correlations]] (unreported correlations are < 0.100)
C(peak_sigma, peak_amplitude) = -0.599
C(peak_amplitude, c) = -0.328
C(peak_sigma, c) = 0.196
and make a plot like this:
I am trying to fit this x data: [0.4,0.165,0.165,0.585,0.585], this y data: [.45, .22, .63, .22, .63], and this z data: [1, 0.99, 0.98,0.97,0.96] to a paraboloid. I am using scipy's curve_fit tool. Here is my code:
doex = [0.4,0.165,0.165,0.585,0.585]
doey = [.45, .22, .63, .22, .63]
doez = np.array([1, .99, .98,.97,.96])
def paraBolEqn(data,a,b,c,d):
if b < .16 or b > .58 or c < .22 or c >.63:
return 1e6
else:
return ((data[0,:]-b)**2/(a**2)+(data[1,:]-c)**2/(a**2))
data = np.vstack((doex,doey))
zdata = doez
opt.curve_fit(paraBolEqn,data,zdata)
I am trying to center the paraboloid between .16 and .58 (x axis) and between .22 and .63 (y axis). I am doing this by returning a large value if b or c are outside of this range.
Unfortunately the fit is wayyy off and my popt values are all 1, and my pcov is inf.
Any help would be great.
Thank you
Rather than forcing high return values for out-of range regions you need to provide a good initial guess. In addition, the mode lacks an offset parameter and the paraboloid has the wrong sign. Change the model to:
def paraBolEqn(data,a,b,c,d):
x,y = data
return -(((x-b)/a)**2+((y-d)/c)**2)+1.0
I fixed the offset to 1.0 because if it were added as fit parameter the system would be underdetermined (fewer or equal number of data points than fit parameters).
Call curve_fit with an initial guess like this:
popt,pcov=opt.curve_fit(paraBolEqn,np.vstack((doex,doey)),doez,p0=[1.5,0.4,1.5,0.4])
This yields:
[ 1.68293045 0.31074135 2.38822062 0.36205424]
and a nice nice match to the data: