We Keep Coding

Super Simple Data cannot be fitted. Why?

So I have data from an experiment. All I want is to fit a log function and if possible to keep the base as a variable parameter for the best possible fit. I don't think that's to much to ask.
So here are the two x and y lists of data I have (they are numpy arrays):
optxlst = np.arange( 1, 92 )
optylst = [
1.96892911, 3.2887339, 4.39055229, 5.27383471, 6.06991106, 6.87452902,
7.50480744, 7.9697223, 8.46885029, 8.94166484, 9.40262963, 9.78905132,
10.13815734, 10.45350799, 10.72270686, 10.9475476, 11.19546712, 11.40757756,
11.60946643, 11.84196217, 11.99959217, 12.15661292, 12.30413587, 12.38961966,
12.5699555, 12.76926629, 12.90418109, 13.03114574, 13.13427797, 13.22579756,
13.33522992, 13.42566857, 13.53639778, 13.59552165, 13.67813624, 13.76984529,
13.8265399, 13.89902442, 13.96015279, 14.01978188, 14.06700569, 14.11720781,
14.07497814, 14.17468682, 14.21045489, 14.2578857, 14.30824306, 14.34443375,
14.37487868, 14.40496269, 14.42757652, 14.47123075, 14.49446011, 14.52862657,
14.55842334, 14.59823539, 14.61842774, 14.66413474, 14.74780246, 14.75705207,
14.80998592, 14.87927607, 14.90345136, 14.93509437, 14.9453211, 14.96428837,
14.96461419, 15.01489101, 15.06245498, 15.0761481, 15.11197484 ,15.16265455,
15.21534841, 15.25283963, 15.24408824, 15.27259487, 15.29698918, 15.3260376,
15.33598051, 15.377234, 15.3913971, 15.41035688, 15.41644309, 15.43849198,
15.46378739, 15.48653162, 15.49380553, 15.49286959, 15.49551899, 15.5068906,
15.50743637]
If plotted they result in the following plot:
Now all I want is to create a nice log function fit. But I get the error message:
>> RuntimeError: Optimal parameters not found: Number of calls to function has reached maxfev = 600.
At this point I am out of ideas. Here is the relevant part of the code:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
optxlst = [...]
optylst = [...]
def logf( x, b, d ): #b is supposed to be the base of the log. Using a math formula here.
return np.log( x ) / np.log( b ) + d
######################## Fitting ####################
popt, pcov = curve_fit( logf, optxlst, optylst) #<-this causes the error
print( popt )
######################## Plotting ###################
xticks=[]
for a in range( 0, 91, 10 ):
xticks.append( a )
plt.plot( optxlst,optylst, 'bo' , markersize=3, label="Datepoints" )
plt.xticks( xticks )
plt.legend()
plt.title( 'Curve fitting of Stab.check', fontsize='16')
plt.xlabel( 'Iteration', fontsize='16')
plt.ylabel( 'Xcoord[um]', fontsize='18')
plt.show()

why is it so bad that using acceptance-rejection-sampling for computing integral

I want to compute $\int_1^2 x^2 dx$ using the acceptance-rejection-sampling method, but it performs not well as my expectation.
import random
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
sns.set_style('darkgrid')
plt.rcParams['figure.figsize'] = (12,8)
def acceptRejectSampling(number,p,c):
sample=[]
while len(sample)<number:
y = random.uniform(2, 4)
u = random.uniform(0, c)
if u <= p(y):
sample.append(y)
return sample
def f(x):#the domain of x is [1,2]
return x**2
def p(y):# pdf of y, the domain of y is [2,4]
return 0.5/(y**(0.5))
S=7/3.0
X=np.random.uniform(1,2,1000)
print("area：",S)
print("monte calro estimator： ",np.mean([f(i) for i in X]))
#acceptance rejection sampling
y=np.linspace(2,4,200)
c=0.5/np.power(2,0.5)
cc=[c for i in range(200)]
samples=acceptRejectSampling(10000,p,c)
print("acceptance-rejection-sampling estimator： ", np.mean(samples))
plt.plot(y,cc, '--',label='c*g(x)')
py=[p(i) for i in y]
plt.plot(y, py,label='p(y)')
plt.hist(samples, bins=50, density=True,label='sampling')
plt.legend()
plt.show()
the output of code is:
aera： 2.3333333333333335
monte calro estimator： 2.391469143847661
acceptance-rejection-sampling estimator： 2.938527759943371
I am confused that why acceptance-rejection-sampling performs so bad and I want to consult you guys on this problem. please help me, thank you very much!

How to fit data logarithmic in python?

I've been trying to fit some data I have gained from some simulations. From the curve, I guess a logarithmic fit would be ideal. However, the curve comes looking out quite funky. I've also tried higher order polynomials and np.polyfit, but I couldn't get either to work.
Any help would be appreciated!
from scipy.optimize import curve_fit
import numpy as np
import matplotlib.pyplot as plt
xdata=[9.24104360013e-06, 4.72619458107e-06, 4.03957328857e-06, 9.78301182748e-06, 1.36994566431e-05, 1.16294573409e-05, 7.70899546232e-06, 2.72587766232e-06, 2.19089955631e-06, 5.34851640035e-06, 7.84434545123e-06, 7.6524185787e-06, 1.00592536363e-05, 6.08711035578e-07, 4.08259572135e-07, 5.74424798328e-07, 6.20036326494e-07, 4.34755225756e-06, 4.72832211908e-06, 1.25156011417e-06, 1.44996714816e-05, 3.79992166335e-06, 4.45935911838e-06, 6.6307841155e-06, 2.38540191336e-06, 9.4649801666e-07, 9.11518608157e-06, 3.1944675219e-06, 5.32674287313e-06, 1.48463901861e-05, 3.41127723277e-06, 3.40027150288e-06, 3.33064781566e-06, 2.12828505238e-06, 7.22565690506e-06, 7.86527964811e-06, 2.25791582571e-06, 1.94875869207e-05, 1.54712884424e-05, 5.82300791075e-06, 9.5783833758e-06, 1.89519143607e-05, 1.03731970283e-05, 2.53090894753e-05, 9.26047056658e-06, 1.05428610146e-05, 2.89162870493e-05, 4.78624726782e-05, 1.00005855557e-05, 6.88617910928e-05]
ydata=[0.00281616449359, 0.00257023004939, 0.00250030932407, 0.00284317789756, 0.00300158447316, 0.00291690879783, 0.00274898865728, 0.0023625485679, 0.0023018015629, 0.00259860025555, 0.00269155777824, 0.00265941197135, 0.0028073724168, 0.00192920496041, 0.00182900945464, 0.00191452746379, 0.00193227563253, 0.00253266811688, 0.00255961306471, 0.00212426145702, 0.00285906942634, 0.00247877245272, 0.0025348504727, 0.00269881922057, 0.00232270371493, 0.00204672286703, 0.00281306442303, 0.00241938445736, 0.00261083321385, 0.00287440363274, 0.00244324770882, 0.00244364989768, 0.00244593671433, 0.00228714406931, 0.00263301289418, 0.00269385915315, 0.0022968948347, 0.00313898537645, 0.00305650121575, 0.00265291893623, 0.00278748794063, 0.00312801724905, 0.00289450806538, 0.00313176225397, 0.00284010926578, 0.0028957865422, 0.00335438183977, 0.00360421739757, 0.00270734995952, 0.00377301191882]
plt.plot(xdata,ydata,'o')
x = np.array(xdata, dtype=float) #transform your data in a numpy array of floats
y = np.array(ydata, dtype=float) #so the curve_fit can work
#def func(x,a,b,c):
# return a*x**2+ b*x +c
def func(x,a,b):
return a*np.log(x)+ b
popt, pcov = curve_fit(func, x, y)
plt.plot(x, func(x, *popt), label="Fitted Curve")
plt.show()

Sort x before plotting
x_sorted = np.sort(x)
plt.plot(x_sorted, func(x_sorted, *popt), label="Fitted Curve")
plt.show()

curve fitting with a known function numpy

I have a x and y one-dimension numpy array and I would like to reproduce y with a known function to obtain "beta". Here is the code I am using:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
y = array([ 0.04022493, 0.04287536, 0.03983657, 0.0393201 , 0.03810298,
0.0363814 , 0.0331144 , 0.03074823, 0.02795767, 0.02413816,
0.02180802, 0.01861309, 0.01632699, 0.01368056, 0.01124232,
0.01005323, 0.00867196, 0.00940864, 0.00961282, 0.00892419,
0.01048963, 0.01199101, 0.01533408, 0.01855704, 0.02163586,
0.02630014, 0.02971127, 0.03511223, 0.03941218, 0.04280329,
0.04689105, 0.04960554, 0.05232003, 0.05487037, 0.05843364,
0.05120701])
x= array([ 0., 0.08975979, 0.17951958, 0.26927937, 0.35903916,
0.44879895, 0.53855874, 0.62831853, 0.71807832, 0.80783811,
0.8975979 , 0.98735769, 1.07711748, 1.16687727, 1.25663706,
1.34639685, 1.43615664, 1.52591643, 1.61567622, 1.70543601,
1.7951958 , 1.88495559, 1.97471538, 2.06447517, 2.15423496,
2.24399475, 2.33375454, 2.42351433, 2.51327412, 2.60303391,
2.6927937 , 2.78255349, 2.87231328, 2.96207307, 3.05183286,
3.14159265])
def func(x,beta):
return 1.0/(4.0*np.pi)*(1+beta*(3.0/2*np.cos(x)**2-1.0/2))
guesses = [20]
popt,pcov = curve_fit(func,x,y,p0=guesses)
y_fit = 1/(4.0*np.pi)*(1+popt[0]*(3.0/2*np.cos(x)**2-1.0/2))
plt.figure(1)
plt.plot(x,y,'ro',x,y_fit,'k-')
plt.show()
The code works but the fitting is completely off (see picture). Any idea why?
It looks like the formula to use contains an additional parameter, i.e. p
def func(x,beta,p):
return p/(4.0*np.pi)*(1+beta*(3.0/2*np.cos(x)**2-1.0/2))
guesses = [20,5]
popt,pcov = curve_fit(func,x,y,p0=guesses)
y_fit = func(angle_plot,*popt)
plt.figure(2)
plt.plot(x,y,'ro',x,y_fit,'k-')
plt.show()
print popt # [ 1.23341604 0.27362069]
In the popt which one is beta and which one is p?

This is perhaps not what you want but, if you are just trying to get a good fit to the data, you could use np.polyfit:
fit = np.polyfit(x,y,4)
fit_fn = np.poly1d(fit)
plt.scatter(x,y,label='data',color='r')
plt.plot(x,fit_fn(x),color='b',label='fit')
plt.legend(loc='upper left')
Note that fit gives the coefficient values of, in this case, a 4th order polynomial:
>>> fit
array([-0.00877534, 0.05561778, -0.09494909, 0.02634183, 0.03936857])

This is going to be as good as you can get (assuming you get the equation right as #mdurant suggested), an additional intercept term is required to further improve the fit:
def func(x,beta, icpt):
return 1.0/(4.0*np.pi)*(1+beta*(3.0/2*np.cos(x)**2-1.0/2))+icpt
guesses = [20, 0]
popt,pcov = curve_fit(func,x,y,p0=guesses)
y_fit = func(x, *popt)
plt.figure(1)
plt.plot(x,y,'ro', x,y_fit,'k-')
print popt #[ 0.33748816 -0.05780343]

Finding the intersection of a curve from polyfit

This seems simple but I can't quite figure it out. I have a curve calculated from x,y data. Then I have a line. I want to find the x, y values for where the two intersect.
Here is what I've got so far. It's super confusing and doesn't give the correct result. I can look at the graph and find the intersection x value and calculate the correct y value. I'd like to remove this human step.
import numpy as np
import matplotlib.pyplot as plt
from pylab import *
from scipy import linalg
import sys
import scipy.interpolate as interpolate
import scipy.optimize as optimize
w = np.array([0.0, 11.11111111111111, 22.22222222222222, 33.333333333333336, 44.44444444444444, 55.55555555555556, 66.66666666666667, 77.77777777777777, 88.88888888888889, 100.0])
v = np.array([0.0, 8.333333333333332, 16.666666666666664, 25.0, 36.11111111111111, 47.22222222222222, 58.333333333333336, 72.22222222222221, 86.11111111111111, 100.0])
z = np.polyfit(w, v, 2)
print (z)
p=np.poly1d(z)
g = np.polyval(z,w)
print (g)
N=100
a=arange(N)
b=(w,v)
b=np.array(b)
c=(w,g)
c=np.array(c)
print(c)
d=-a+99
e=(a,d)
print (e)
p1=interpolate.PiecewisePolynomial(w,v[:,np.newaxis])
p2=interpolate.PiecewisePolynomial(w,d[:,np.newaxis])
def pdiff(x):
return p1(x)-p2(x)
xs=np.r_[w,w]
xs.sort()
x_min=xs.min()
x_max=xs.max()
x_mid=xs[:-1]+np.diff(xs)/2
roots=set()
for val in x_mid:
root,infodict,ier,mesg = optimize.fsolve(pdiff,val,full_output=True)
# ier==1 indicates a root has been found
if ier==1 and x_min<root<x_max:
roots.add(root[0])
roots=list(roots)
print(np.column_stack((roots,p1(roots),p2(roots))))
plt.plot(w,v, 'r', a, -a+99, 'b-')
plt.show()
q=input("what is the intersection value? ")
print (p(q))
Any ideas to get this to work?
Thanks

I don't think I fully understand what you are trying to do in your code, but what you described in English can be done with
from __future__ import division
import numpy as np
import matplotlib.pyplot as plt
w = np.array([0.0, 11.11111111111111, 22.22222222222222, 33.333333333333336,
44.44444444444444, 55.55555555555556, 66.66666666666667,
77.77777777777777, 88.88888888888889, 100.0])
v = np.array([0.0, 8.333333333333332, 16.666666666666664, 25.0,
36.11111111111111, 47.22222222222222, 58.333333333333336,
72.22222222222221, 86.11111111111111, 100.0])
poly_coeff = np.polynomial.polynomial.polyfit(w, v, 2)
poly = np.polynomial.polynomial.Polynomial(poly_coeff)
roots = np.polynomial.polynomial.polyroots(poly_coeff - [99, -1, 0])
x = np.linspace(np.min(roots) - 50, np.max(roots) + 50, num=1000)
plt.plot(x, poly(x), 'r-')
plt.plot(x, 99 - x, 'b-')
for root in roots:
plt.plot(root, 99 - root, 'ro')