Related
Does anyone have an idea how to fit a curve to another curve, simply by shifting it to the right. For example, in this plot, I want to shift the orange curve to the right (no vertical shift!) in order that the curves overlap eachother. Can anyone help me to do this?
Data of the curves:
y1 = [1.2324, 1.4397, 1.5141, 1.7329, 1.9082, 2.2884, 2.166, 2.8175, 3.1014, 2.8893, 3.673, 4.3875, 4.9817, 5.6906, 6.3667, 7.2854, 8.2703, 9.3432, 10.591, 11.963, 13.579, 15.36, 17.306, 19.508, 21.976, 24.666, 27.692, 31.026, 34.724, 38.702]
y2 = [1.6231, 1.6974, 1.8145, 2.4805, 2.5643, 2.6176, 2.9332, 3.4379, 4.0154, 4.2258, 4.6837, 5.9837, 6.4408, 7.2903, 8.2283, 9.4134, 10.537, 11.947, 13.344, 15.202, 17.073, 19.211, 21.598, 24.216, 27.06, 30.31, 33.933, 37.882, 42.201, 46.978]
x = [0.1, 0.127, 0.161, 0.204, 0.259, 0.329, 0.418, 0.53, 0.672, 0.853, 1.08, 1.37, 1.74, 2.21, 2.81, 3.56, 4.52, 5.74, 7.28, 9.24, 11.7, 14.9, 18.9, 24.0, 30.4, 38.6, 48.9, 62.1, 78.8, 100.0]
This question has a few problems that needed some fiddling to resolve. This is not the ideal solution I'm sure, but it provided a close enough value to what was expected by the manual method (around 1.5 and 1.6).
The first roadblock is that when you shift the X values, you don't get matching y values, so calculating the residual can be tricky. I brute-forced my way through this problem by creating a huge new x array with 1000 points, then interpolated the original 2 y values on this new x array (this will come later). Therefore, when calculating the residual between the two curves, the x values will be off, but not by much.
reference_y = y1
to_shift_y = y2
expanded_x = np.logspace(np.log10(x[0]), np.log10(x[-1]), num=1000)
expanded_y_reference = np.interp(expanded_x, x, reference_y)
expanded_y_to_shift = np.interp(expanded_x, x, to_shift_y)
Then, when you shift x by some constant, you'll get two regions where there won't be equivalent x values.
original x: -------------------------------xxxx
shifted x: xxxx-------------------------------
I created a new x array with the shift parameter, hor_shift set so some value greater than 1. Then, I found the indices where the original and shifted stop matching.
start = np.argmax(expanded_x >= expanded_x_shifted[0])
end = np.argmin(expanded_x_shifted <= expanded_x[-1])
Since these arrays are [False, False, True, True ...] and [True, True, ..., True, False, False], argmax and argmin will return the first instance where you have a different value.
Now, we have to slice our original and shifted x arrays so they have the same size, and values in common, and the same with the expanded y arrays. Pardon the long names, it's just so I don't get confused.
expanded_x_original_in_common_with_shifted = expanded_x[start:]
expanded_x_shifted_in_common_with_original = expanded_x_shifted[:end]
sliced_expanded_y_reference = expanded_y_reference[start:]
sliced_expanded_y_to_shift = expanded_y_to_shift[:end]
And last, and most importantly, we can calculate a distance between the two curves, assuming the x values are aligned.
residual = ((sliced_expanded_y_reference - sliced_expanded_y_to_shift) ** 2).sum()
By minimizing this, we can get the ideal shift.
We can compare our curves. Here, I used two values for the shift, 1.3 and 1.56, to illustrate good and bad shift values (these were found by testing different values). The vertical lines show the region in common.
Now, we can transform this process into a function and use some minimization method to find the ideal shift value. Here's what I got.
from lmfit import Parameters, minimize
par = Parameters()
# If the shift parameter is 1, you get an error
par.add('shift', value=1.1, min=1)
def min_function(par, x, reference_y, to_shift_y):
hor_shift = par['shift'].value
# print(hor_shift) # <- in case you want to follow the process
expanded_x = np.logspace(np.log10(x[0]), np.log10(x[-1]), num=1000)
expanded_x_shifted = expanded_x * hor_shift
start = np.argmax(expanded_x >= expanded_x_shifted[0])
end = np.argmin(expanded_x_shifted <= expanded_x[-1])
expanded_x_original_in_common_with_shifted = expanded_x[start:]
expanded_x_shifted_in_common_with_original = expanded_x_shifted[:end]
expanded_y_reference = np.interp(expanded_x, x, reference_y)
expanded_y_to_shift = np.interp(expanded_x, x, to_shift_y)
sliced_expanded_y_reference = expanded_y_reference[start:]
sliced_expanded_y_to_shift = expanded_y_to_shift[:end]
residual = ((sliced_expanded_y_reference - sliced_expanded_y_to_shift) ** 2).sum()
return residual
minimize(min_function, par, method='nelder', args=(x, reference_y, to_shift_y))
This results in an ideal shift parameter of 1.555, confirming the initial guess. Note that you have to change the residual expression to (sliced_expanded_y_reference - sliced_expanded_y_to_shift) if you want your chisquared to match the one in the graphs.
The notations are changed in order to make more clear the matrix equations below :
y(x)=y1(x)
z(x)=y2(x)
A translation of value=c on the x logarithmic scale is equivalent to an expansion of value=b on the x linear scale because log(x)+c=log(b x) with c=log(b).
The inverse function x=f(y) have to approximately coincide with bx=f(z). So we consider the sum of residuals [f(y)-x]^2+[f(z)-b x]^2. This leads to the regression calculus below. The function f(y) is approximated with a polynomial of degree m.
With the given data the shape of the curve of f(y) is rather smooth. This suggest that a low degree m might be sufficient.
For example with m=2 the result is :
The black curve is the horizontaly translated blue curve from c=0.180 on the logarithmic scale.
Of course one can use a polynomial of higher degree. For example with m=3 we get b=1.536 and c=0.186
This numerical example is a favourable case because the curve x=f(y) has a simple shape. In case of more complicated shapes probably a bigger value of m should be necessary, with the risk of unreliable regression calculus.
I am experimenting with Python to fit curves to a series of data-points, summary below:
From the below, it would seem that polynomials of order greater than 2 are the best fit, followed by linear and finally exponential which has the overall worst outcome.
While I appreciate this might not be exponential growth, I just wanted to know whether you would expect the exponential function perform so badly (basically the coefficient of x,b, has been set to 0 and an arbitrary point has been picked on the curve to intersect) or if I have somehow done something wrong in my code to fit.
The code I'm using to fit is as follows:
# Fitting
def exponenial_func(x,a,b,c):
return a*np.exp(-b*x)+c
def linear(x,m,c):
return m*x+c
def quadratic(x,a,b,c):
return a*x**2 + b*x+c
def cubic(x,a,b,c,d):
return a*x**3 + b*x**2 + c*x + d
x = np.array(x)
yZero = np.array(cancerSizeMean['levelZero'].values)[start:]
print len(x)
print len(yZero)
popt, pcov = curve_fit(exponenial_func,x, yZero, p0=(1,1,1))
expZeroFit = exponenial_func(x, *popt)
plt.plot(x, expZeroFit, label='Control, Exponential Fit')
popt, pcov = curve_fit(linear, x, yZero, p0=(1,1))
linearZeroFit = linear(x, *popt)
plt.plot(x, linearZeroFit, label = 'Control, Linear')
popt, pcov = curve_fit(quadratic, x, yZero, p0=(1,1,1))
quadraticZeroFit = quadratic(x, *popt)
plt.plot(x, quadraticZeroFit, label = 'Control, Quadratic')
popt, pcov = curve_fit(cubic, x, yZero, p0=(1,1,1,1))
cubicZeroFit = cubic(x, *popt)
plt.plot(x, cubicZeroFit, label = 'Control, Cubic')
*Edit: curve_fit is imported from the scipy.optimize package
from scipy.optimize import curve_fit
curve_fit tends to perform poorly if you give it a poor initial guess with functions like the exponential that could end up with very large numbers. You could try altering the maxfev input so that it runs more iterations. otherwise, I would suggest trying with with something like:
p0=(1000,-.005,0)
-.01, since it ~doubles from 300 to 500 and you have -b in your eqn, 100 0 since it is ~3000 at 300 (1.5 doublings from 0). See how that turns out
As for why the initial exponential doesn't work at all, your initial guess is b=1, and x is in range of (300,1000) or range. This means python is calculating exp(-300) which either throws an exception or is set to 0. At this point, whether b is increased or decreased, the exponential is going to still be set to 0 for any value in the general vicinity of the initial estimate.
Basically, python uses a numerical method with limited precision, and the exponential estimate went outside of the range of values it can handle
I'm not sure how you're fitting the curves -- are you using polynomial least squares? In that case, you'd expect the fit to improve with each additional degree of flexibility, and you choose the power based on diminishing marginal improvement / outside theory.
The improving fit should look something like this.
I actually wrote some code to do Polynomial Least Squares in python for a class a while back, which you can find here on Github. It's a bit hacky though and loosely commented since I was just using it to solve exercises. Hope it's helpful.
I attempted to test my gradient descent program on rosenbrock function. But no matter how I adjusted my learning rate (step argument), precision (precision argument) and number of iterations (iteration argument), I couldn't get a very close result.
import numpy as np
def minimize(f, f_grad, x, step=1e-3, iterations=1e3, precision=1e-3):
count = 0
while True:
last_x = x
x = x - step * f_grad(x)
count += 1
if count > iterations or np.linalg.norm(x - last_x) < precision:
break
return x
def rosenbrock(x):
"""The Rosenbrock function"""
return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1])**2.0)
def rosenbrock_grad(x):
"""Gradient of Rosenbrock function"""
xm = x[1:-1]
xm_m1 = x[:-2]
xm_p1 = x[2:]
der = np.zeros_like(x)
der[1:-1] = 200*(xm-xm_m1**2) - 400*(xm_p1 - xm**2)*xm - 2*(1-xm)
der[0] = -400*x[0]*(x[1]-x[0]**2) - 2*(1-x[0])
der[-1] = 200*(x[-1]-x[-2]**2)
return der
x0 = np.array([1.3, 0.7, 0.8, 1.9, 1.2])
minimize(rosenbrock, rosenbrock_grad, x0, step=1e-6, iterations=1e4, precision=1e-6)
For example, code like above gives me array([ 1.01723267, 1.03694999, 1.07870143, 1.16693184, 1.36404334]). But if I use any built-in optimization methods in scipy.optimize, I can get very close answer or exactly equal array([ 1., 1., 1., 1., 1.]) (this is the true answer).
However, if I use very small step, precision and very large iterations in my program, the calculation just takes forever on my computer.
I wonder if this is due to
any bugs in my program
or just because
gradient descent is inefficient here and demands very small
step, precision and very large iterations to yield a very close
solution
or because
I need to do some special feature scaling.
(Ps. I also tried to plot two-dimensional plot where value of function is on y axis and the number of iterations is on x axis to "debug" gradient descent, but even I get a nice-looking downsloping graph, the solution is still not very close.)
To quote the Rosenbrock Wikipedia page:
The global minimum is inside a long, narrow, parabolic shaped flat valley. To find the valley is trivial. To converge to the global minimum, however, is difficult.
Gradient descent is a simple algorithm, so it is probably no surprise that it cannot find the minimum. Let's see what happens in 2D for different starting points:
Just as Wikipedia says: it easily finds the valley but then fails to converge further. The gradient along the valley is very flat compared to the rest of the function.
I would conclude that your implementation works correctly but perhaps the Rosenbrock function is not the most appropriate function to test it.
Contrary to other answers, I further argue that the step size is too small rather than too large. The problem is not overshooting but that the algorithm gets stuck. If I set the the step size to 1e-3 without changing other settings the algorithm converges to the maximum within two digits. This happens despite overshooting the valley from some starting positions in the 2D case - but you need the speed not to get stuck later on, so to say.
Here is the modified code to reproduce above figure:
import numpy as np
import matplotlib.pyplot as plt
def minimize(f, f_grad, x, step=1e-3, iterations=1e3, precision=1e-3):
count = 0
while True:
last_x = x
x_hist.append(x)
x = x - step * f_grad(x)
count += 1
if count > iterations or np.linalg.norm(x - last_x) < precision:
x_hist.append(x)
break
return x
def rosenbrock(x):
"""The Rosenbrock function"""
return sum(100.0*(x[1:]-x[:-1]**2.0)**2.0 + (1-x[:-1])**2.0)
def rosenbrock_grad(x):
"""Gradient of Rosenbrock function"""
xm = x[1:-1]
xm_m1 = x[:-2]
xm_p1 = x[2:]
der = np.zeros_like(x)
der[1:-1] = 200*(xm-xm_m1**2) - 400*(xm_p1 - xm**2)*xm - 2*(1-xm)
der[0] = -400*x[0]*(x[1]-x[0]**2) - 2*(1-x[0])
der[-1] = 200*(x[-1]-x[-2]**2)
return der
k = np.linspace(0, 2, 101)
f = np.empty((k.shape[0], k.shape[0]))
for i, y in enumerate(k):
for j, x in enumerate(k):
f[i, j] = rosenbrock(np.array([x, y]))
plt.imshow(np.log10(f), extent=[k[0], k[-1], k[-1], k[0]], cmap='autumn')
for start in [[0.5, 0.5], [1.0, 0.5], [1.5, 0.5],
[0.5, 1.0], [1.0, 1.0], [1.5, 1.0],
[0.5, 1.5], [1.0, 1.5], [1.5, 1.5]]:
x0 = np.array(start)
x_hist = []
minimize(rosenbrock, rosenbrock_grad, x0, step=1e-6, iterations=1e4, precision=1e-9)
x_hist = np.array(x_hist)
plt.plot(x_hist[:, 0], x_hist[:, 1], 'k')
plt.plot(x0[0], x0[1], 'ok')
Your method is vulnerable to overshoot. In a case with instantaneously high gradient, your solution will jump very far. It is often appropriate in optimization to refuse to take a step when it fails to reduce cost.
Linesearch
Once you have chosen a direction by computing the gradient, search along that direction until you reduce cost by some fraction of the norm of the gradient.
I.e. Start with x[n+1]= x - α * gradient
And vary α from 1.0 to 0.0, accepting a value for x if has reduced the cost by some fraction of the norm of gradient. This is a nice convergence rule termed the Armijo rule.
Other advice
Consider optimizing the 2D Rosenbrock function first, and plotting your path over that cost field.
Consider numerically verifying that your gradient implementation is correct. More often than not, this is the problem.
Imagine you're hiking along a
knife-edge
mountain path that's getting narrower and narrower.
A constant step-size will take you over the edge, aieeeee;
you want to take shorter, more careful steps as you climb.
Similarly, to follow a Rosenbrock valley, a program must
take shorter, more careful steps as the valley narrows.
Decreasing step-sizes as step0 / t^0.5 or 0.25
helps GD on Rosenbrock a bit,
but is still very sensitive to step0.
Real step-sizes aka learning rates must adapt to the problem terrain, e.g.
line search for smooth problems, Ada* for
SGD .
By the way, the Rosenbrock function is a sum of squares,
and there are powerful methods for that; see
scipy.optimize.least_squares .
First question:
I'm trying to fit experimental datas with function of the following form:
f(x) = m_o*(1-exp(-t_o*x)) + ... + m_j*(1-exp(-t_j*x))
Currently, I don't find a way to have an undetermined number of parameters m_j, t_j, I'm forced to do something like this:
def fitting_function(x, m_1, t_1, m_2, t_2):
return m_1*(1.-numpy.exp(-t_1*x)) + m_2*(1.-numpy.exp(-t_2*x))
parameters, covariance = curve_fit(fitting_function, xExp, yExp, maxfev = 100000)
(xExp and yExp are my experimental points)
Is there a way to write my fitting function like this:
def fitting_function(x, li):
res = 0.
for el in range(len(li) / 2):
res += li[2*idx]*(1-numpy.exp(-li[2*idx+1]*x))
return res
where li is the list of fitting parameters and then do a curve_fitting? I don't know how to tell to curve_fitting what is the number of fitting parameters.
When I try this kind of form for fitting_function, I have errors like "ValueError: Unable to determine number of fit parameters."
Second question:
Is there any way to force my fitting parameters to be positive?
Any help appreciated :)
See my question and answer here. I've also made a minimal working example demonstrating how it could be done for your application. I make no claims that this is the best way - I am muddling through all this myself, so any critiques or simplifications are appreciated.
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as pl
def wrapper(x, *args): #take a list of arguments and break it down into two lists for the fit function to understand
N = len(args)/2
amplitudes = list(args[0:N])
timeconstants = list(args[N:2*N])
return fit_func(x, amplitudes, timeconstants)
def fit_func(x, amplitudes, timeconstants): #the actual fit function
fit = np.zeros(len(x))
for m,t in zip(amplitudes, timeconstants):
fit += m*(1.0-np.exp(-t*x))
return fit
def gen_data(x, amplitudes, timeconstants, noise=0.1): #generate some fake data
y = np.zeros(len(x))
for m,t in zip(amplitudes, timeconstants):
y += m*(1.0-np.exp(-t*x))
if noise:
y += np.random.normal(0, noise, size=len(x))
return y
def main():
x = np.arange(0,100)
amplitudes = [1, 2, 3]
timeconstants = [0.5, 0.2, 0.1]
y = gen_data(x, amplitudes, timeconstants, noise=0.01)
p0 = [1, 2, 3, 0.5, 0.2, 0.1]
popt, pcov = curve_fit(lambda x, *p0: wrapper(x, *p0), x, y, p0=p0) #call with lambda function
yfit = gen_data(x, popt[0:3], popt[3:6], noise=0)
pl.plot(x,y,x,yfit)
pl.show()
print popt
print pcov
if __name__=="__main__":
main()
A word of warning, though. A linear sum of exponentials is going to make the fit EXTREMELY sensitive to any noise, particularly for a large number of parameters. You can test that by adding even a small amount of noise to the data generated in the script - even small deviations cause it to get the wrong answer entirely while the fit still looks perfectly valid by eye (test with noise=0, 0.01, and 0.1). Be very careful interpreting your results even if the fit looks good. It's also a form that allows for variable swapping: the best fit solution is the same even if you swap any pairs of (m_i, t_i) with (m_j, t_j), meaning your chi-square has multiple identical local minima that might mean your variables get swapped around during fitting, depending on your initial conditions. This is unlikely to be a numeriaclly robust way to extract these parameters.
To your second question, yes, you can, by defining your exponentials like so:
m_0**2*(1.0-np.exp(-t_0**2*x)+...
Basically, square them all in your actual fit function, fit them, and then square the results (which could be negative or positive) to get your actual parameters. You can also define variables to be between a certain range by using different proxy forms.
I wanted to compute the volume of the intersect of a sphere and infinite cylinder at some distance b, and i figured i would do it using a quick and dirty python script. My requirements are a <1s computation with >3 significant digits.
My thinking was as such:
We place the sphere, with radius R, such that its center is at the origin, and we place the cylinder, with radius R', such that its axis is spanned in z from (b,0,0). We integrate over the sphere, using a step function that returns 1 if we are inside the cylinder, and 0 if not, thus integrating 1 over the set constrained by being inside both sphere and cylinder, i.e. the intersect.
I tried this using scipy.intigrate.tplquad. It did not work out. I think its because of the discontinuity of the step function as i get warnings such the following. Of course, i might just be doing this wrong. Assuming i have not made some stupid mistake, I could attempt to formulate the ranges of the intersect, thus removing the need for the step function, but i figured i might try and get some feedback first. Can anyone spot any mistake, or point towards some simple solution.
Warning: The maximum number of
subdivisions (50) has been achieved.
If increasing the limit yields no
improvement it is advised to analyze
the integrand in order to determine
the difficulties. If the position of
a local difficulty can be
determined (singularity,
discontinuity) one will probably
gain from splitting up the interval
and calling the integrator on the
subranges. Perhaps a special-purpose
integrator should be used.
Code:
from scipy.integrate import tplquad
from math import sqrt
def integrand(z, y, x):
if Rprim >= (x - b)**2 + y**2:
return 1.
else:
return 0.
def integral():
return tplquad(integrand, -R, R,
lambda x: -sqrt(R**2 - x**2), # lower y
lambda x: sqrt(R**2 - x**2), # upper y
lambda x,y: -sqrt(R**2 - x**2 - y**2), # lower z
lambda x,y: sqrt(R**2 - x**2 - y**2), # upper z
epsabs=1.e-01, epsrel=1.e-01
)
R=1
Rprim=1
b=0.5
print integral()
Assuming you are able to translate and scale your data such a way that the origin of the sphere is in [0, 0, 0] and its radius is 1, then a simple stochastic approximation may give you a reasonable answer fast enough. So, something along the lines could be a good starting point:
import numpy as np
def in_sphere(p, r= 1.):
return np.sqrt((p** 2).sum(0))<= r
def in_cylinder(p, c, r= 1.):
m= np.mean(c, 1)[:, None]
pm= p- m
d= np.diff(c- m)
d= d/ np.sqrt(d** 2).sum()
pp= np.dot(np.dot(d, d.T), pm)
return np.sqrt(((pp- pm)** 2).sum(0))<= r
def in_sac(p, c, r_c):
return np.logical_and(in_sphere(p), in_cylinder(p, c, r_c))
if __name__ == '__main__':
n, c= 1e6, [[0, 1], [0, 1], [0, 1]]
p= 2* np.random.rand(3, n)- 2
print (in_sac(p, c, 1).sum()/ n)* 2** 3
Performing a triple adaptive numerical integrations on a discontinuous function that is constant over two domains is a terribly poor idea, especially if you wish to see either speed or accuracy.
I would suggest a far better idea is to reduce the problem analytically.
Align the cylinder with an axis, by transformation. This translates the sphere to some point that is not at the origin.
Now, find the limits of intersection of the sphere with the cylinder along that axis.
Integrate over that axis variable. The area of intersection at any fixed value along the axis is simply the area of intersection of two circles, which in turn is simply computable using trigonometry and a little effort.
In the end, you will have an exact result, with almost no computation time needed.
I solved it using a simple MC integration, as suggested by eat, but my implementation was to slow. My requirements had increased. I therefore reformulated the problem mathematically, as suggested by woodchips.
Basically i formulated the limits of x as a function of z and y, and y as a function of z. Then i, in essence, integrated f(z,y,z)=1 over the intersection, using the limits. I did this because of the speed increase, allowing me to plot volume vs b, and because it allows me to integrate more complex functions with relative minor modification.
I include my code in case anyone is interested.
from scipy.integrate import quad
from math import sqrt
from math import pi
def x_max(y,r):
return sqrt(r**2-y**2)
def x_min(y,r):
return max(-sqrt(r**2 - y**2), -sqrt(R**2 - y**2) + b)
def y_max(r):
if (R<b and b-R<r) or (R>b and b-R>r):
return sqrt( R**2 - (R**2-r**2+b**2)**2/(4.*b**2) )
elif r+R<b:
return 0.
else: #r+b<R
return r
def z_max():
if R>b:
return R
else:
return sqrt(2.*b*R - b**2)
def delta_x(y, r):
return x_max(y,r) - x_min(y,r)
def int_xy(z):
r = sqrt(R**2 - z**2)
return quad(delta_x, 0., y_max(r), args=(r))
def int_xyz():
return quad(lambda z: int_xy(z)[0], 0., z_max())
R=1.
Rprim=1.
b=0.5
print 4*int_xyz()[0]
First off: You can calculate the volume of the intersection by hand. If you don't want to (or can't) do that, here's an alternative:
I'd generate a tetrahedral mesh for the domain and then add up the cell volumes. An example with pygalmesh and meshplex (both authored by myself):
import pygalmesh
import meshplex
import numpy
ball = pygalmesh.Ball([0, 0, 0], 1.0)
cyl = pygalmesh.Cylinder(-1, 1, 0.7, 0.1)
u = pygalmesh.Intersection([ball, cyl])
mesh = pygalmesh.generate_mesh(u, cell_size=0.05, edge_size=0.1)
points = mesh.points
cells = mesh.cells["tetra"]
# kick out unused vertices
uvertices, uidx = numpy.unique(cells, return_inverse=True)
cells = uidx.reshape(cells.shape)
points = points[uvertices]
mp = meshplex.MeshTetra(points, cells)
print(sum(mp.cell_volumes))
This gives you
and prints 2.6567890958740463 as volume. Decrease cell or edge sizes for higher precision.