We Keep Coding

Numpy Polynomial class not printing correct coefficients from fit

As recommended in the numpy documentation, I am trying to move from the old polyfit and polyval functions to the Polynomial class. Below is a minimal example that demonstrates my confusion with this.
import numpy as np
x = np.array([-2.3, -2.8, -2.9, -3.1])
y = np.array([2.4, 3.1, 3.3, 3.5])
poly = np.polynomial.Polynomial.fit(x, y, 1)
print(poly)
print(poly.coef)
print(poly(0))
print(poly(1) - poly(0))
which gives the output, running numpy 1.23.3,
2.9697841726618703 - 0.5611510791366912·x¹
[ 2.96978417 -0.56115108]
-0.8179856115107942
-1.4028776978417268
Judging from the first two lines, the fitted polynomial is something like 2.969 - 0.561x. But evaluating it at x=0 gives -0.817, and evaluating the slope by f(1) - f(0) gives -1.40. The latter is what I would expect given the points that I am fitting, but what's going on with the first two lines of output?

If you check out the documentation you'll see that the polynomial object has two more fields: poly.domain and poly.window.
To get better numerical properties the range independent variable of the input to fit() will get renormalized to [-1, 1] (by default at least, this is the poly.window you can set yourself), so the coefficients you get from poly.coef are valid on that renormalized domain. To get back the "original" coefficients you have to undo that normalization, just as I've done in the following snippet. You can also go in the other direction an normalize the x-values yourself which I also included in this snippet:
import matplotlib.pyplot as plt
plt.plot(t, c[0] + c[1]*t) # line in the normalized domain
#x_normalized = (x - d[0])/(d[1] - d[0]) * 2 - 1 # points in the normalized domain
a = 2/(d[1]-d[0])
b = - 2*d[0]/(d[1]-d[0]) - 1
x_normalized = a*x + b
print('original coefficients')
print(a*c[1], c[1]*b + c[0])
plt.plot(x_normalized, y, 'o')
plt.show()

Sine fitting using scipy is not returning good fit

trying to fit some sine wave to data i collected. But Amplitude and Frequency are way off. Any suggestions?
x=[0,1,3,4,5,6,7,11,12,13,14,15,16,18,20,21,22,24,26,28,29,30,31,32,35,37,38,40,41,42,43,44,45,48,49,50,51,52,53,54,55,57,58,60,61,62,63,65,66,67,68,69,70,71,73,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,112,114,115,116,117,120,122,123,124,125,128,129,130,131,132,136,137,138,139,140,143,145,147,148,150,151,153,154,155,156,160,163,164,165,167,168,169,171,172,173,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,199,201,202,203,204,205,207,209,210,215,217,218,223,224,225,226,228,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,254,255,256,257,258,259,260,261,262,263,264,265,266,267,269,270,271,272,273,274,275,276,279,280,281,282,286,287,288,292,294,295,296,298,301,302,303,310,311,312,313,315,316,317,318,319,320,321,323,324,325,326,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,348,349,350,351,352,354,356,357,358,359,362,363,365,366,367,371,372,373,374,375,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,404,405,406,407,408,411,412,413,417,418,419,420,421,422,428,429,431,435,436,437,443,444,445,446,450,451,452,453,454,455,456,459,460,461,462,464,465,466,467,468,469,470,471,472,473,474,475,476,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,495,496,497,498,499,500,501,505,506,507,512,513,514,515,516,517,519,521,522,523,524,525,526,528,529,530,531,532,533,535,537,538,539,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,559,560,561,562,563,564,566,567,568,569,570,571,572,573,574,575,577,578,579,584,585,586,588,591,592,593,594,596,598,600,601,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,642,643,644,646,647,648,650,652,653,654,655,656,660,661,662,663,665,666,667,668,669,670,671,672,673,676,677,678,679,680,681,682,684,685,687,688,690,691,692,693,694,695,696,697,698,701,702,703,704,707,708,709,710,712,713,714,715,717,718,719,721,722,723 ]
y=[53.66666667,53.5,51,53.66666667,54.33333333,55.5,57,59,56.5,57.33333333,56,56,57,58,58.66666667,59.5,57,59,58,61.5,60,61,62.5,67,60.66666667,62.5,64.33333333,64,64,65,65,65.66666667,68,70.5,67,67.5,71.5,65,70.5,73.33333333,72,67,76,73.5,72.83333333,75,73,74,73,71,70.5,73.16666667,70,75,69,71,68.33333333,68.5,66.75,62,63.5,63,62.5,61,53.5,61.25,55,57.5,62,54.75,56.5,52.33333333,52.33333333,49,47.66666667,47.5,45,44,42.5,41,37,37.2,34.5,33.4,33.2,34,26,28.6,25,25.5,27,22.66666667,21.66666667,21.5,22.5,22,19.8,19.66666667,20,20,17,26,22.6,19,28,26.33333333,24.25,27,28.5,30,24,33,31,41,38,22,31.66666667,30,39,26,33.5,40,40.5,38,44,47,48,43,42.5,44,43,51.5,48,49.66666667,51.5,47,56,50,50,58,51,58,58.5,57.33333333,57.5,64,57,59,56.5,65.5,60,63.66666667,62,62,65.33333333,66.5,65,66,65,68,65.5,65.83333333,60,65.5,70,68,64,65.42857143,62,68,63.25,62,63.33333333,60.4,59,52.5,52.6,55.16666667,50,51,45.33333333,48.33333333,39.4,38.25,34.33333333,43.25,31.33333333,29.5,29.5,29,27,26,27,25.5,24.5,23,22,22.5,19.5,20,20,18,18.5,17,16,16,15,14,14.5,13,12.5,11.5,11,11,11,10.5,10.5,9,9,10,10,10.5,9,10,10,11,11,11,10,10.66666667,12,12,12.5,13,13,14,14,14.5,16,16,18,16.5,20.5,21.5,21,25,28,22,29,29,28.66666667,36,42,36.75,43.5,48,44.75,50.66666667,53.75,51,57.33333333,58.5,58.66666667,60,60.25,61.75,60,58.5,63,61,60.33333333,62,63,63,60,61.5,62.33333333,62.66666667,61,63.5,61,61.66666667,62,59,60,57.5,56,57,58.5,52.5,50.5,47.5,49.66666667,49.66666667,54.66666667,45.66666667,41,44,33.16666667,49,45,29.5,39.5,29,20.5,23.5,23,19,18.66666667,17,16.75,15.5,15,16,17,13.5,12.2,12,14,13,11,11.5,11.5,11,11.5,11,11.5,11.5,12,13,13,13,13,13.5,14,14,14,15,17,15,16,16,17,18,17,18,18.5,19.5,20.5,20,21.5,20,22,22,23,23,25,26,28,29,36.25,31,37.75,41.33333333,43.6,37.5,46.5,38,47.33333333,46.75,47,50.5,48.5,58,50.5,48.75,54.33333333,56,49,55.5,60,56.5,56,60,56.5,52.75,54,56,57,56,52.66666667,52,52.66666667,53,47.66666667,44,48,50.5,45,46.66666667,48,44.66666667,42.33333333,46.5,43,36.75,41,28,35,36.5,36,37.33333333,24,30.5,29,29.33333333,32.5,20,25.5,27.5,18,33,25.75,26,19.5,16,15.5,18,13,21,12,12.25,11,5,9,10,7.5,5,7.5,4,4.5,5.666666667,3.5,6.5,5,7,7.333333333,7,9,7.5,9,9.5,11,9,10,12,11.5,12.5,13,14,13.5,13,14,15,15,16,16.5,17.5,19.66666667,19.33333333,20.5,23.66666667,25.5,28.75,31,32.66666667,33.66666667,29,32.33333333,37.6,31,39.5,49,44.14285714,41,42.16666667,45,47.66666667,50.2,52.66666667,52,50,54,53.33333333,54.66666667,54.5,54,56,54,53.5,53,53,52,51.5,51.5,52,48,53,48,50,49.5,48.5,46,45,47,49,48,44,42,42,43,43,42.5,41.5,39.5,46,36,37.5,39,39,38,43,40,38,32.5,34,35.33333333,35,35,30.5,30,31.33333333,33,26,30,27,24,30,28,25,29,25.33333333]
from scipy.optimize import curve_fit
from numpy import sin
def fitting(x, a, b, c):
return a * sin(b*x + c)
constants = curve_fit(fitting, x, y)
a_fit= constants[0][0]
b_fit= constants[0][1]
c_fit = constants[0][2]
fit_y=[]
for i in x:
fit_y.append(fitting(i, a_fit, b_fit, c_fit))
plt.plot(x,fit_y, '--', color='red')
plt.scatter(x,y)

You should add an offset to your fitting function, as your data clearly has an offset around 40.
And then you need a proper initial estimate parameter p0 so that the fit converges to the ideal solution. This will do the job :
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from numpy import sin
def fitting(x, a,b,c,d):
return a * sin(b*x + c) + d
p0 = [ (np.max(y)-np.min(y))/2, 6/150, 0, np.mean(y)]
constants = curve_fit(fitting, x, y , p0=p0 )
guess_y = [ fitting(i, *p0) for i in x]
fit_y = [ fitting(i, *constants[0]) for i in x]
plt.plot(x,guess_y, '--', color='green',label='guess')
plt.plot(x,fit_y, '--', color='red',label='fit')
plt.scatter(x,y,label='data') plt.legend()
plt.legend()
If you feel like it, you could even add a linear offset (a*x+b)
Note : thanks for the edit jonsca

I would add this as a comment, but I can't. Fundamentally, a * sin(b*x + c) isn't going to fit well to your data, you don't have an average value of zero so you'd have to try a*sin(b*x +c) + d, but even then I don't think you'll get a great fit. You could try:
Give it some initial values to work with using the p0 input argument https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html . It never hurts to help the minimizer out..
Try a different function, what you have here looks like a sin wave, with offset 'a0' and maybe a decaying amplitude.
But you really need to just look at your data before trying to force a function to fit to it.

Fitting parameters of a data set from a model

I am trying to fit the parameters of a transit light curve.
I have observed transit light curve data and I am using a .py in python that through 4 parameters (period, a(semi-major axis), inclination, planet radius) returns a model transit light curve. I would like to minimize the residual between these two light curves. This is what I am trying to do: First - Estimate a max likelihood using method = "L-BFGS-B" and then apply the mcmc using emcee to estimate the uncertainties.
The code:
p = lmfit.Parameters()
p.add_many(('per', 2.), ('inc', 90.), ('a', 5.), ('rp', 0.1))
per_b = [1., 3.]
a_b = [4., 6.]
inc_b = [88., 90.]
rp_b = [0.1, 0.3]
bounds = [(per_b[0], per_b[1]), (inc_b[0], inc_b[1]), (a_b[0], a_b[1]), (rp_b[0], rp_b[1])]
def residual(p):
v = p.valuesdict()
eclipse.criarEclipse(v['per'], v['a'], v['inc'], v['rp'])
lc0 = numpy.array(eclipse.getCurvaLuz()) (observed flux data)
ts0 = numpy.array(eclipse.getTempoHoras()) (observed time data)
c = numpy.linspace(min(time_phased[bb]),max(time_phased[bb]),len(time_phased[bb]),endpoint=True)
nn = interpolate.interp1d(ts0,lc0)
return nn(c) - smoothed_LC[bb] (residual between the model and the data)
Inside def residual(p) I make sure that both the observed data (time_phased[bb] and smoothed_LC[bb]) have the same size of the model transit light curve. I want it to give me the best fit values for the parameters (v['per'], v['a'], v['inc'], v['rp']).
I need your help and I appreciate your time and your attention. Kindest regards, Yuri.

Your example is incomplete, with many partial concepts and some invalid Python. This makes it slightly hard to understand your intention. If the answer below is not sufficient, update your question with a complete example.
It seems pretty clear that you want to model your data smoothed_LC[bb] (not sure what bb is) with a model for some effect of an eclipse. With that assumption, I would recommend using the lmfit.Model approach. Start by writing a function that models the data, just so you check and plot your model. I'm not entirely sure I understand everything you're doing, but this model function might look like this:
import numpy
from scipy import interpolate
from lmfit import Model
# import eclipse from somewhere....
def eclipse_lc(c, per, a, inc, p):
eclipse.criarEclipse(per, a, inc, rp)
lc0 = numpy.array(eclipse.getCurvaLuz()) # observed flux data
ts0 = numpy.array(eclipse.getTempoHoras()) # observed time data
return interpolate.interp1d(ts0,lc0)(c)
With this model function, you can build a Model:
lc_model = Model(eclipse_lc)
and then build parameters for your model. This will automatically name them after the argument names of your model function. Here, you can also give them initial values:
params = lc_model.make_params(per=2, inc=90, a=5, rp=0.1)
You wanted to place upper and lower bounds on these parameters. This is done by setting min and max parameters, not making an ordered array of bounds:
params['per'].min = 1.0
params['per'].max = 3.0
and so on. But also: setting such tight bounds is usually a bad idea. Set bounds to avoid unphysical parameter values or when it becomes evident that you need to place them.
Now, you can fit your data with this model. Well, first you need to get the data you want to model. This seems less clear from your example, but perhaps:
c_data = numpy.linspace(min(time_phased[bb]), max(time_phased[bb]),
len(time_phased[bb]), endpoint=True)
lc_data = smoothed_LC[bb]
Well: why do you need to make this c_data? Why not just use time_phased as the independent variable? Anyway, now you can fit your data to your model with your parameters:
result = lc_model(lc_data, params, c=c_data)
At this point, you can print out a report of the results and/or view or get the best-fit arrays:
print(result.fit_report())
for p in result.params.items(): print(p)
import matplotlib.pyplot as plt
plt.plot(c_data, lc_data, label='data')
plt.plot(c_data. result.best_fit, label='fit')
plt.legend()
plt.show()
Hope that helps...

fitting data to a custom model in python

Hi So I am fairly used to python but this is the first time I am using python for data analysis and I was wondering if you could shed some light on an issue I am having.
I need to fit about 4000 different plots to the following function :
b+e*A*(1.19104*(10**-16))*((x*(10**-9))**-5)*((-1+np.exp(0.0143878/(T*x*(10**-9))))**-1)
in this function I want to restrict b, e and A to certain values for each plot and the value of the variable T to shift according to the data.
I tried using scipy optimize but I couldn't figure out how to hold parameters with that one.
I tried using pyAstronomy funcfit but for one it is extremely inefficient (or maybe my code is idk) and I wasn't getting data that looked what I had approximated the data should look like.
Finally I am currently trying to use lmfit but my code here seems to have all of the parameters stay the same as the guess and not vary at all. I am confused on how to proceed.
my current code looks like this...
def bbnm(x,b,e,T,A):
y = b+e*A*(1.19104*(10**-16))*((x*(10**-9))**-5)*((-1+np.exp(0.0143878/(T*x*(10**-9))))**-1)
return y
params = bbmodel.make_params(b=0,e=.99,T=5100,A=wgeometry)
params['b'].vary = False
params['e'].vary = False
params['A'].vary = False
params['T'].vary = True
bb = bbmodel.fit(power[1465:2510],params,x=wavelength[1465:2510])

Here is code to restrict scipy's curve_fit parameters to within specified bounds. In this example, the first parameter's bounds are +/- infinity (unbounded), the second parameter's bounds are +/- 100 but the fitted parameter is within the bounds and fitted normally, and the third parameter is restricted by its bounds.
import numpy
import matplotlib
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
xData = numpy.array([5.0, 6.1, 7.2, 8.3, 9.4])
yData = numpy.array([ 10.0, 18.4, 20.8, 23.2, 35.0])
def standardFunc(data, a, b, c):
return a * data + b * data**2 + c
# some initial parameter values - must be within bounds
initialParameters = numpy.array([1.0, 1.0, 1.0])
# bounds on parameters - initial parameters must be within these
lowerBounds = (-numpy.Inf, -100.0, -5.0)
upperBounds = (numpy.Inf, 100.0, 5.0)
parameterBounds = [lowerBounds, upperBounds]
fittedParameters, pcov = curve_fit(standardFunc, xData, yData, initialParameters, bounds = parameterBounds)
# values for display of fitted function
a, b, c = fittedParameters
# for plotting the fitting results
xPlotData = numpy.linspace(min(xData), max(xData), 50)
y_plot = standardFunc(xPlotData, a, b, c)
plt.plot(xData, yData, 'D') # plot the raw data as a scatterplot
plt.plot(xPlotData, y_plot) # plot the equation using the fitted parameters
plt.show()
print('fitted parameters:', fittedParameters)

How to do Scipy curve fitting with error bars and obtain standard errors on fitting parameters?

I am trying to fit my data points. It looks like the fitting without errors are not that optimistic, therefore now I am trying to fit the data implementing the errors at each point. My fit function is below:
def fit_func(x,a,b,c):
return np.log10(a*x**b + c)
then my data points are below:
r = [ 0.00528039,0.00721161,0.00873037,0.01108928,0.01413011,0.01790143,0.02263833, 0.02886089,0.03663713,0.04659512,0.05921978,0.07540126,0.09593949, 0.12190075,0.15501736,0.19713563,0.25041524,0.3185025,0.40514023,0.51507869, 0.65489938,0.83278859,1.05865016,1.34624082]
logf = [-1.1020581079659384, -1.3966927245616112, -1.4571368537041418, -1.5032694247562564, -1.8534775558300272, -2.2715812166948304, -2.2627690390113862, -2.5275290780299331, -3.3798813619309365, -6.0, -2.6270989211307034, -2.6549656159564918, -2.9366845162570079, -3.0955026428779604, -3.2649261507250289, -3.2837123017838366, -3.0493752067042856, -3.3133647996463229, -3.0865051494299243, -3.1347499415910169, -3.1433062918466632, -3.1747394718538979, -3.1797597345585245, -3.1913094832146616]
Because my data is in log scale, logf, then the error bar for each data point is not symmetric. The upper error bar and lower error bar are below:
upper = [0.070648916083227764, 0.44346256268274886, 0.11928131794776076, 0.094260899008089094, 0.14357124858039971, 0.27236750587684311, 0.18877122991380402, 0.28707938182603066, 0.72011863806906318, 0, 0.16813325716948757, 0.13624929595316049, 0.21847915642008875, 0.25456116079315372, 0.31078368240910148, 0.23178227464741452, 0.09158189214515966, 0.14020538489677881, 0.059482730164901909, 0.051786777740678414, 0.041126467609954531, 0.034394612910981337, 0.027206248503368613, 0.021847333685597548]
lower = [0.06074797748043137, 0.21479225959441428, 0.093479845697059583, 0.077406149968278104, 0.1077175009766278, 0.16610073183912188, 0.13114254113054535, 0.17133966123838595, 0.57498950902908286, 2.9786837094190934, 0.12090437578535695, 0.10355760401838676, 0.14467588244034646, 0.15942693835964539, 0.17929440903034921, 0.15031667827534712, 0.075592499975030591, 0.10581886912443572, 0.05230849287772843, 0.04626422871423852, 0.03756658820680725, 0.03186944137872727, 0.025601929615431285, 0.02080073540367966]
I have the fitting as:
popt, pcov = optimize.curve_fit(fit_func, r, logf,sigma=[lower,upper])
logf_fit = fit_func(r,*popt)
But this is wrong, how can I implement the curve fitting from scipy to include the upper and lower errors? How could I get the fitting errors of the fitting parameters a, b, c?

You can use scipy.optimize.leastsq with custom weights:
import scipy.optimize as optimize
import numpy as np
# redefine lists as array
x=np.array(r)
y=np.array(logf)
errup=np.array(upper)
errlow=np.array(lower)
# error function
def fit_func(x,a,b,c):
return np.log10(a*x**b + c)
def my_error(V):
a,b,c=V
yfit=fit_func(x,a,b,c)
weight=np.ones_like(yfit)
weight[yfit>y]=errup[yfit>y] # if the fit point is above the measure, use upper weight
weight[yfit<=y]=errlow[yfit<=y] # else use lower weight
return (yfit-y)**2/weight**2
answer=optimize.leastsq(my_error,x0=[0.0001,-1,0.0006])
a,b,c=answer[0]
print(a,b,c)
It works, but is very sensitive to initial values, since there is a log which can go in wrong domain (negative numbers) and then it fails. Here I find a=9.14464745425e-06 b=-1.75179880756 c=0.00066720486385which is pretty close to data.