The issue I am having is that when I use the code below to find the norm-1 of my error. Firstly, when I plot the error against step-size h, the error values are quite small, in the range of 10^-14 to 10^-16. Secondly, underneath, you can see my attempt to apply the np.polyfit to my graph, which when run, won't fit a characteristic but will output values. The value of p[0] is not perfect, so I believe something is wrong, but it is "close" to the desired output of 3. Is this a matter of just the wrong input or bad data?
def rk3(A,bvector,y0,interval,N):
x0=interval[0]
x_end=interval[1]
x=np.linspace(x0,x_end,N+1)
h=(x_end-x0)/N
y=np.zeros((N+1,len(y0)))
y[0, :] = y0
for n in range(N):
y_1=y[n,:]+h*(np.dot(A,y[n,:])+bvector(x[n]))
y_2=(3/4)*y[n,:]+(1/4)*y_1+(1/4)*h*(np.dot(A,y_1)+bvector(x[n]+h))
y[n+1,:]=(1/3)*y[n,:]+(2/3)*y_2+(2/3)*h*(np.dot(A,y_2)+bvector(x[n]+(1/2)*h))
return x,y
err_vals = []
h_vals = []
for k in range(2,11): #for the range of N=40k, where k=1,...,10
N=40*k
x, y = rk3(A,bvector,y0,[0,0.1],N)
yc = y[-1,:]
h = (x[-1]-x[0])/N
h_vals.append(h)
yvals.append(yc)
yn = y[:,1]
abs_err = np.zeros(N)
print("The value of y at k=",k," is ",yc)
for j in range(1,N):
y_exact=np.array([np.exp(-1000*x[j]), (1000/999)*(np.exp(-x[j])-np.exp(-1000*x[j]))])
y_exact_2 = y_exact[1]
abs_err[j] = np.abs((y[j, 1] - y_exact_2)/y_exact_2)
Error = h*np.sum(abs_err[j])
err_vals.append(Error)
p = np.polyfit(np.log(h_vals), np.log(err_vals), 1)
pyplot.loglog(h_vals,err_vals,"kx")
pyplot.xlabel("h")
pyplot.ylabel("Error")
pyplot.loglog(h,np.exp(p[1])*h**(p[0]), 'r--')
print("Best fit line slope ",format(p[0]))
My evolution of your code below gives a completely straight line with slope close to 3 for the integration over the interval [0,0.01].
For the given interval [0,0.1] the slope value is about 1/3 larger. The error profiles, that is, the absolute error divided by the expected global error power of the step size, gives a converging pattern, confirming the convergence of order 3 of the method.
The error bound 2e7*h^3 is rather large, showing why the combination of problem and method can become very problematic for larger step sizes.
The error is computed via the L1 norms of the function difference and exact solution,
Error = sum(abs((y-y_exact(x))[:,1]))/sum(abs(y[:,1]))
giving a mathematically sound quantity. The summation of the local relative errors can lead to distortions of the total error where the exact solution has a root or small values. But still, even using your computation method of integrating the local relative error leaving out the first data point which is zero,
Error = sum(abs((y[1:,1]/y_exact(x)[1:,1]-1)))*h
gives a similar linear plot, with the range shifted down to 1e-7..1e-9, the slope staying at 3.0293
Note that if you want to use the list h_vals in a computation like the one to plot the fitted line, you have to convert in into a numpy array first.
h=np.asarray(h_vals)
complete code
def rk3(A,bvector,y0,interval,N):
"""Solves an IVP y'=f(x, y(x)) on x \in [0, x_end] with y(0) = y0 using N points, using Runge-Kutta method."""
x=np.linspace(*interval,N+1)
h=x[1]-x[0]
y=np.zeros((N+1,len(y0)))
y[0, :] = y0
for n in range(N):
y_1=y[n]+h*(np.dot(A,y[n])+bvector(x[n]))
y_2=(3/4)*y[n,:]+(1/4)*y_1+(1/4)*h*(np.dot(A,y_1)+bvector(x[n]+h))
y[n+1]=(1/3)*y[n]+(2/3)*y_2+(2/3)*h*(np.dot(A,y_2)+bvector(x[n]+0.5*h))
return x,y
A = np.array([[-1000.0,0.0],[1000.0,-1.0]]);
bvector = lambda x: 0
y_exact = lambda x: np.array([np.exp(-1000*x), (1000/999)*(np.exp(-x)-np.exp(-1000*x))]).T
y0 = y_exact(0)
plt.figure(figsize=(6,3));
h_vals, y_vals, err_vals = [],[],[]
for k in range(2,11): #for the range of N=40k, where k=1,...,10
N=40*k
x, y = rk3(A,bvector,y0,[0,0.01],N)
yc = y[-1,:]
h = x[1]-x[0];
plt.plot(x,(y-y_exact(x))[:,1]/h**3)
h_vals.append(h)
y_vals.append(yc)
yn = y[:,1]
print("The value of y at k=",k," is ",yc)
Error = sum(abs((y-y_exact(x))[:,1]))/sum(abs(y[:,1]))
err_vals.append(Error)
plt.grid(); plt.show()
p = np.polyfit(np.log(h_vals), np.log(err_vals), 1)
plt.figure(figsize=(6,4))
plt.loglog(h_vals,err_vals,"kx")
h=np.asarray(h_vals)
plt.plot(h,np.exp(p[1])*h**(p[0]), '--r', lw=0.5)
plt.xlabel("h")
plt.ylabel("Error")
plt.grid(); plt.show()
print("Best fit line slope ",format(p[0]))
Related
I have written a Monte-Carlo simulation to fit 49 data points with asymmetric error bars. Since the errors are asymmetric on both axis, I cannot simply use scipy.optimize.curve_fit module. This is my basic approach:
Generate a list of 1000 random numbers from within the confidence level (error range) using triangular probability distribution distribution with maximum probability at a certain data point. Now I have a list of dimensions [49*1000].
Convert this list from [491000] to [100049]. I did this so that I have a data set of 1000 samples of 49 points which are within the error range.
Use scipy's curve_fit to fit these 1000 samples seperately and find the free parameters in the function y=m*x+c (c is free parameter and I already know the slope m.)
Find mean squared error in each of these 1000 samples using sklearn.metrics.mean_squared_error module.
Find the index with least Mean squared and use the popt value (c parameter) at this index to plot the fit.
Working Code: This is my code:
trials=int(1e4)
#xdata: most probable x value
#ydata: most probable y value
#xerr6low: lower bound on x error
#xerr6up: upper bound on x error
#yerr6low: lower bound on y error
#yerr6up: upper bound on y error
#generating random number using triangular distribution weighted with highest probability at xdata/ydata:
xarray1=[0]*len(obs6xr)
yarray1=[0]*len(obs6xr)
for i in range(0,len(obs6xr)):
xarray1[i]=np.random.triangular(xdata[i]-xerr6low[i],xdata[i],xdata[i]+xerr6up[i],trials)
yarray1[i]=np.random.triangular(ydata[i]-yerr6low[i],ydata[i],ydata[i]+yerr6up[i],trials)
xarray=[list(x) for x in zip(*xarray1)]
yarray=[list(x) for x in zip(*yarray1)]
def func(x, c):
#return (np.log10(a)+(b*(x-12))+np.log10(10**(8+c)))
#return ((x/1e12)**a)*(b)*(10**(8+c))
m = 1.65
mx = [element * m for element in x]
y = [j+c for j in mx]
return y
#Fit for the parameters a, b, c of the function func:
popt=np.zeros(trials)
pcov=np.zeros(trials)
print(len(xarray),len(yarray),len(popt))
for i in tqdm(range (trials),desc='Optimizing'):
popt[i], pcov[i] = curve_fit(func, xarray[i], np.array(yarray[i]))
MSE=np.zeros(trials)
for i in tqdm(range (trials),desc='Calculating MSE'):
MSE[i]=mean_squared_error(np.array(yarray[i]),func(np.array(xarray[i]),popt[i]))
minimum_MSE=np.amin(MSE)
index_MSE=np.where(MSE == np.amin(MSE))
print('minimum MSE = ',minimum_MSE,'at index = ', index_MSE)
print(popt[index_MSE])
def MbhthShimasakupropc(x,c):
Mbhth = (10**(c))*(x**1.65)
return Mbhth
plt.figure(figsize=(5,5))
x=np.logspace(11,14,trials)
plt.loglog(obs6x,MbhthShimasakuprop(obs6x,popt[index_MSE]))
plt.xscale('log')
plt.yscale('log')
m=np.logspace(np.log10(4e10),np.log10(3e14),1000)
plt.errorbar(obs0x, obs0y, xerr=asymmetric_errorx0, yerr=asymmetric_errory0, fmt='o',color='black', markersize='2.5', ecolor='black',capsize=2, elinewidth=1)
plt.errorbar(obs6x, obs6y, xerr=asymmetric_errorx6, yerr=asymmetric_errory6, fmt='o',color='red', markersize='2.5', ecolor='red',capsize=2, elinewidth=1)
plt.loglog(m,MbhthShimasaku(m,0),color='black',linestyle=':',label='Shimasaku-Ferrarese z=0')
plt.xlim(4e10,3e14)
plt.ylim(1e6,1e11)
plt.legend(['Monte-Carlo Fitting','local relation','z=0','z~6'])
plt.show()
Result:
As we can see, this code works perfectly fine. But if I change function to 2 parameters (a&b),
func(x,a,b) and apply the same code with some small tweaks, the code fails miserably.
Not Working Code:
trials=int(1e3)
#generating random number:
xarray1=[0]*len(obs6xr)
yarray1=[0]*len(obs6xr)
for i in range(0,len(obs6xr)):
xarray1[i]=np.random.triangular(xdata[i]-xerr6low[i],xdata[i],xdata[i]+xerr6up[i],trials)
yarray1[i]=np.random.triangular(ydata[i]-yerr6low[i],ydata[i],ydata[i]+yerr6up[i],trials)
xarray=[list(x) for x in zip(*xarray1)]
yarray=[list(x) for x in zip(*yarray1)]
def func(x, a, b):
y=a*x+b
return y
#Fit for the parameters a, b, c of the function func:
popt=[0]*(trials)
pcov=[0]*(trials)
print(len(xarray),len(yarray),len(popt))
for i in tqdm(range (trials)):
popt[i], pcov[i] = curve_fit(func, np.array(xarray[i]), np.array(yarray[i]))
MSE=np.zeros(trials)
for i in tqdm(range (trials)):
MSE[i]=mean_squared_error(np.array(yarray[i]),func(np.array(xarray[i]),np.array(popt[i])[0],np.array(popt[i])[1]))
minimum_MSE=np.amin(MSE)
index_MSE=np.where(MSE == np.amin(MSE))
print('minimum MSE = ',minimum_MSE,'at index = ', index_MSE)
def MbhthShimasakupropmc(x,m,c):
Mbhth = (10**(c))*(x**m)
return Mbhth
plt.figure(figsize=(5,5))
x=np.logspace(11,14,trials)
plt.loglog(obs6x,MbhthShimasakupropmc(obs6x,*popt[[index_MSE][0][0][0]]))
plt.xscale('log')
plt.yscale('log')
m=np.logspace(np.log10(4e10),np.log10(3e14),1000)
plt.errorbar(obs0x, obs0y, xerr=asymmetric_errorx0, yerr=asymmetric_errory0, fmt='o',color='black', markersize='2.5', ecolor='black',capsize=2, elinewidth=1)
plt.errorbar(obs6x, obs6y, xerr=asymmetric_errorx6, yerr=asymmetric_errory6, fmt='o',color='red', markersize='2.5', ecolor='red',capsize=2, elinewidth=1)
plt.loglog(m,MbhthShimasaku(m,0),color='black',linestyle=':',label='Shimasaku-Ferrarese z=0')
plt.legend(['Monte-Carlo Fitting','local relation','z=0','z~6'])
plt.show()
Result:
I don't know what I am doing wrong. Can someone help me debug the issue.
I want to approximate the solutions of dy/dx = -x +1, with eulers method on the interval from 0 to 2. I'm using this code
def f(x):
return -x+1 # insert any function here
x0 = 1 # Initial slope #
dt = 0.1 # time step
T = 2 # ...from...to T
t = np.linspace(0, T, int(T/dt) + 1) # divide the interval from 0 to 2 by dt
x = np.zeros(len(t))
x[0] = x0 # value at 1 is the initial slope
for i in range(1, len(t)): # apply euler method
x[i] = x[i-1] + f(x[i-1])*dt
plt.figure() # plot the result
plt.plot(t,x,color='blue')
plt.xlabel('t')
plt.ylabel('y(t)')
plt.show()
Can I use this code to approximate the solutions of any function on any interval? It's hard to see whether this actually works, because I don't know how to plot the actual solution ( -1/2x^2 + x ) along side the approximation.
It would probably help if you consistently used the same variable names for the same role. Per your output, the solution is y(t). Thus your differential equation should be dy(t)/dt = f(t,y(t)). This would then give an implementation for the slope function and its exact solution
def f(t,y): return 1-t
def exact_y(t,t0,y0): return y0+0.5*(1-t0)**2-0.5*(1-t)**2
Then implement the Euler loop also as a separate function, keeping out problem specific details as much as possible
def Eulerint(f,t0,y0,te,dt):
t = np.arange(t0,te+dt,dt)
y = np.zeros(len(t))
y[0] = y0
for i in range(1, len(t)): # apply euler method
y[i] = y[i-1] + f(t[i-1],y[i-1])*dt
return t,y
Then plot the solutions as
y0,T,dt = 1,2,0.1
t,y = Eulerint(f,0,y0,T,dt)
plt.plot(t,y,color='blue')
plt.plot(t,exact_y(t,0,y0),color='orange')
You can just plot the actual solution by using:
def F(x):
return -0.5*x+x
# some code lines
plt.plot(t,x,color='blue')
plt.plot(t,F(t),color='orange')
But please note that the actual solution (-1/2x+x = 1/2x) does not correspond to your slope f(x) and will show a different solution.
The *real slope f(x) of the actual solution (-1/2x+x = 1/2x) is just simply f(x)=1/2
From https://stackoverflow.com/a/30460089/2202107, we can generate CDF of a normal distribution:
import numpy as np
import matplotlib.pyplot as plt
N = 100
Z = np.random.normal(size = N)
# method 1
H,X1 = np.histogram( Z, bins = 10, normed = True )
dx = X1[1] - X1[0]
F1 = np.cumsum(H)*dx
#method 2
X2 = np.sort(Z)
F2 = np.array(range(N))/float(N)
# plt.plot(X1[1:], F1)
plt.plot(X2, F2)
plt.show()
Question: How do we generate the "original" normal distribution, given only x (eg X2) and y (eg F2) coordinates?
My first thought was plt.plot(x,np.gradient(y)), but gradient of y was all zero (data points are evenly spaced in y, but not in x) These kind of data is often met in percentile calculations. The key is to get the data evenly space in x and not in y, using interpolation:
x=X2
y=F2
num_points=10
xinterp = np.linspace(-2,2,num_points)
yinterp = np.interp(xinterp, x, y)
# for normalizing that sum of all bars equals to 1.0
tot_val=1.0
normalization_factor = tot_val/np.trapz(np.ones(len(xinterp)),yinterp)
plt.bar(xinterp, normalization_factor * np.gradient(yinterp), width=0.2)
plt.show()
output looks good to me:
I put my approach here for examination. Let me know if my logic is flawed.
One issue is: when num_points is large, the plot looks bad, but it's a issue in discretization, not sure how to avoid it.
Related posts:
I failed to understand why the answer was so complicated in https://stats.stackexchange.com/a/6065/131632
I also didn't understand why my approach was different than Generate distribution given percentile ranks
I am trying to implement simple optimization problem in Python. I am currently struggling with the following error:
Value Error: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
As far as I understand, this means that I am somewhere trying to plug in an array where only single value can be accepted. Nevertheless, I haven't managed to come up with a solution, nor have I discovered where is the problem.
My code follows
def validationCurve(X, y, Xval, yval):
#[lmbda_vec, error_train, error_val] =
# VALIDATIONCURVE(X, y, Xval, yval) returns the train and
# validation errors (in error_train, error_val) for different
# values of lmbda. Given the training set (X,y) and validation
# set (Xval, yval).
lmbda_vec = [0, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1];
m = len(y);
X = numpy.concatenate((numpy.ones((m,1)), X), axis = 1);
n = len(Xval);
Xval = numpy.concatenate((numpy.ones((n,1)), Xval), axis = 1);
error_train = numpy.zeros((len(lmbda_vec), 1));
error_val = numpy.zeros((len(lmbda_vec), 1));
for i in range(0,len(lmbda_vec)):
lmbda = lmbda_vec[i];
theta = trainLinearReg(X, y, lmbda);
error_train[i] = linearRegCostFunction(X, y, theta, lmbda);
error_val[i] = linearRegCostFunction(Xval, yval, theta, lmbda);
return lmbda_vec, error_train, error_val
def trainLinearReg(X, y, lmbda):
#[theta] = TRAINLINEARREG (X, y, lmbda) trains linear
# regression usingthe dataset (X, y) and regularization
# parameter lmbda. Returns the trained parameters theta.
alpha = 1 # learning rate
num_iters = 200 # number of iterations
initial_theta = (numpy.zeros((len(X[0,:]),1))) #initial guess
#Create "short hand" for the cost function to be minimized
costFunction = lambda t: linearRegCostFunction(X, y, t, lmbda);
#Minimize using Conjugate Gradient
theta = minimize(costFunction, initial_theta, method = 'Newton-CG',
jac = True, options = {'maxiter': 200})
return theta
def linearRegCostFunction(X, y, theta, lmbda):
# [J, grad] = LINEARREGCOSTFUNCTION(X, y, theta, lmbda)
# computes the cost of using theta as the parameter for
# linear regression to fit the data points in X and y.
# Returns the cost in J and the gradient in grad.
# Initialize some useful values
m, n = X.shape; # number of training examples
J = 0;
grad = numpy.zeros((n ,1))
J = numpy.dot((y- X # theta).T, (y-X # theta)) +
lmbda*(theta[1:].T # theta[1:])
J = J/m
grad = (X.T # (y - X # theta))/m
grad [1:] += (lmbda*theta[1:])/m
grad = grad[:];
return grad
I am trying to obtain an optimal regularization parameter by computing cost function and minimizing with respect to theta.
My input values are:
X.shape = (100,25), y.shape = (100,1)
Xval.shape = (55,25), yval.shape = (55,1)
Outputted errors are:
--> 129 lmbda_vec , error_train, error_val = validationCurve(Xtrain, ytrain, Xva
lid, yvalid )
---> 33 theta = trainLinearReg(X, y, lmbda);
---> 49 theta = minimize(costFunction, initial_theta,
method = 'Newton-CG', jac = True, options = {'maxiter': 200})
Later I won't to use the optimized model to predict y on new X.
Could you please advice me where is the problem in my code?
Also, if you observe any points for improvement in my code, please let me know. I will be glad to hear and improve.
Thank you!
Nimitz14 hit the general explanation: you've supplied an array where a scalar is required. This caused a run-time error. The problem is how to fix it from here.
First, try slapping your favourite debugger on the problem, so you can stop the program at useful spots and figure out exactly what array is causing the problem. This should help you determine where it originated.
Failing that, place some strategic print statements along the call route, printing each argument just before the function call. Then examine the signature (call sequence) of each function, and see where you might have given an array in place of a scalar. I don't see it, but ...
Is it possible that you've somehow redefined True as an array?
The code below is giving me a flat line for the line of best fit rather than a nice curve along the model of e^(-x) that would fit the data. Can anyone show me how to fix the code below so that it fits my data?
import numpy as np
import matplotlib.pyplot as plt
import scipy.optimize
def _eNegX_(p,x):
x0,y0,c,k=p
y = (c * np.exp(-k*(x-x0))) + y0
return y
def _eNegX_residuals(p,x,y):
return y - _eNegX_(p,x)
def Get_eNegX_Coefficients(x,y):
print 'x is: ',x
print 'y is: ',y
# Calculate p_guess for the vectors x,y. Note that p_guess is the
# starting estimate for the minimization.
p_guess=(np.median(x),np.min(y),np.max(y),.01)
# Calls the leastsq() function, which calls the residuals function with an initial
# guess for the parameters and with the x and y vectors. Note that the residuals
# function also calls the _eNegX_ function. This will return the parameters p that
# minimize the least squares error of the _eNegX_ function with respect to the original
# x and y coordinate vectors that are sent to it.
p, cov, infodict, mesg, ier = scipy.optimize.leastsq(
_eNegX_residuals,p_guess,args=(x,y),full_output=1,warning=True)
# Define the optimal values for each element of p that were returned by the leastsq() function.
x0,y0,c,k=p
print('''Reference data:\
x0 = {x0}
y0 = {y0}
c = {c}
k = {k}
'''.format(x0=x0,y0=y0,c=c,k=k))
print 'x.min() is: ',x.min()
print 'x.max() is: ',x.max()
# Create a numpy array of x-values
numPoints = np.floor((x.max()-x.min())*100)
xp = np.linspace(x.min(), x.max(), numPoints)
print 'numPoints is: ',numPoints
print 'xp is: ',xp
print 'p is: ',p
pxp=_eNegX_(p,xp)
print 'pxp is: ',pxp
# Plot the results
plt.plot(x, y, '>', xp, pxp, 'g-')
plt.xlabel('BPM%Rest')
plt.ylabel('LVET/BPM',rotation='vertical')
plt.xlim(0,3)
plt.ylim(0,4)
plt.grid(True)
plt.show()
return p
# Declare raw data for use in creating regression equation
x = np.array([1,1.425,1.736,2.178,2.518],dtype='float')
y = np.array([3.489,2.256,1.640,1.043,0.853],dtype='float')
p=Get_eNegX_Coefficients(x,y)
It looks like it's a problem with your initial guesses; something like (1, 1, 1, 1) works fine:
You have
p_guess=(np.median(x),np.min(y),np.max(y),.01)
for the function
def _eNegX_(p,x):
x0,y0,c,k=p
y = (c * np.exp(-k*(x-x0))) + y0
return y
So that's test_data_maxe^( -.01(x - test_data_median)) + test_data_min
I don't know much about the art of choosing good starting parameters, but I can say a few things. leastsq is finding a local minimum here - the key in choosing these values is to find the right mountain to climb, not to try to cut down on the work that the minimization algorithm has to do. Your initial guess looks like this (green):
(1.736, 0.85299999999999998, 3.4889999999999999, 0.01)
which results in your flat line (blue):
(-59.20295956, 1.8562 , 1.03477144, 0.69483784)
Greater gains were made in adjusting the height of the line than in increasing the k value. If you know you're fitting to this kind of data, use a larger k. If you don't know, I guess you could try to find a decent k value by sampling your data, or working back from the slope between an average of the first half and the second half, but I wouldn't know how to go about that.
Edit: You could also start with several guesses, run the minimization several times, and take the line with the lowest residuals.