Solving non-linear equations in python - python
I have 4 non-linear equations with three unknowns X, Y, and Z that I want to solve for. The equations are of the form:
F(m) = X^2 + a(m)Y^2 + b(m)XYcosZ + c(m)XYsinZ
...where a, b and c are constants which are dependent on each value of F in the four equations.
What is the best way to go about solving this?
There are two ways to do this.
Use a non-linear solver
Linearize the problem and solve it in the least-squares sense
Setup
So, as I understand your question, you know F, a, b, and c at 4 different points, and you want to invert for the model parameters X, Y, and Z. We have 3 unknowns and 4 observed data points, so the problem is overdetermined. Therefore, we'll be solving in the least-squares sense.
It's more common to use the opposite terminology in this case, so let's flip your equation around. Instead of:
F_i = X^2 + a_i Y^2 + b_i X Y cosZ + c_i X Y sinZ
Let's write:
F_i = a^2 + X_i b^2 + Y_i a b cos(c) + Z_i a b sin(c)
Where we know F, X, Y, and Z at 4 different points (e.g. F_0, F_1, ... F_i).
We're just changing the names of the variables, not the equation itself. (This is more for my ease of thinking than anything else.)
Linear Solution
It's actually possible to linearize this equation. You can easily solve for a^2, b^2, a b cos(c), and a b sin(c). To make this a bit easier, let's relabel things yet again:
d = a^2
e = b^2
f = a b cos(c)
g = a b sin(c)
Now the equation is a lot simpler: F_i = d + e X_i + f Y_i + g Z_i. It's easy to do a least-squares linear inversion for d, e, f, and g. We can then get a, b, and c from:
a = sqrt(d)
b = sqrt(e)
c = arctan(g/f)
Okay, let's write this up in matrix form. We're going to translate 4 observations of (the code we'll write will take any number of observations, but let's keep it concrete at the moment):
F_i = d + e X_i + f Y_i + g Z_i
Into:
|F_0| |1, X_0, Y_0, Z_0| |d|
|F_1| = |1, X_1, Y_1, Z_1| * |e|
|F_2| |1, X_2, Y_2, Z_2| |f|
|F_3| |1, X_3, Y_3, Z_3| |g|
Or: F = G * m (I'm a geophysist, so we use G for "Green's Functions" and m for "Model Parameters". Usually we'd use d for "data" instead of F, as well.)
In python, this would translate to:
def invert(f, x, y, z):
G = np.vstack([np.ones_like(x), x, y, z]).T
m, _, _, _ = np.linalg.lstsq(G, f)
d, e, f, g = m
a = np.sqrt(d)
b = np.sqrt(e)
c = np.arctan2(g, f) # Note that `c` will be in radians, not degrees
return a, b, c
Non-linear Solution
You could also solve this using scipy.optimize, as #Joe suggested. The most accessible function in scipy.optimize is scipy.optimize.curve_fit which uses a Levenberg-Marquardt method by default.
Levenberg-Marquardt is a "hill climbing" algorithm (well, it goes downhill, in this case, but the term is used anyway). In a sense, you make an initial guess of the model parameters (all ones, by default in scipy.optimize) and follow the slope of observed - predicted in your parameter space downhill to the bottom.
Caveat: Picking the right non-linear inversion method, initial guess, and tuning the parameters of the method is very much a "dark art". You only learn it by doing it, and there are a lot of situations where things won't work properly. Levenberg-Marquardt is a good general method if your parameter space is fairly smooth (this one should be). There are a lot of others (including genetic algorithms, neural nets, etc in addition to more common methods like simulated annealing) that are better in other situations. I'm not going to delve into that part here.
There is one common gotcha that some optimization toolkits try to correct for that scipy.optimize doesn't try to handle. If your model parameters have different magnitudes (e.g. a=1, b=1000, c=1e-8), you'll need to rescale things so that they're similar in magnitude. Otherwise scipy.optimize's "hill climbing" algorithms (like LM) won't accurately calculate the estimate the local gradient, and will give wildly inaccurate results. For now, I'm assuming that a, b, and c have relatively similar magnitudes. Also, be aware that essentially all non-linear methods require you to make an initial guess, and are sensitive to that guess. I'm leaving it out below (just pass it in as the p0 kwarg to curve_fit) because the default a, b, c = 1, 1, 1 is a fairly accurate guess for a, b, c = 3, 2, 1.
With the caveats out of the way, curve_fit expects to be passed a function, a set of points where the observations were made (as a single ndim x npoints array), and the observed values.
So, if we write the function like this:
def func(x, y, z, a, b, c):
f = (a**2
+ x * b**2
+ y * a * b * np.cos(c)
+ z * a * b * np.sin(c))
return f
We'll need to wrap it to accept slightly different arguments before passing it to curve_fit.
In a nutshell:
def nonlinear_invert(f, x, y, z):
def wrapped_func(observation_points, a, b, c):
x, y, z = observation_points
return func(x, y, z, a, b, c)
xdata = np.vstack([x, y, z])
model, cov = opt.curve_fit(wrapped_func, xdata, f)
return model
Stand-alone Example of the two methods:
To give you a full implementation, here's an example that
generates randomly distributed points to evaluate the function on,
evaluates the function on those points (using set model parameters),
adds noise to the results,
and then inverts for the model parameters using both the linear and non-linear methods described above.
import numpy as np
import scipy.optimize as opt
def main():
nobservations = 4
a, b, c = 3.0, 2.0, 1.0
f, x, y, z = generate_data(nobservations, a, b, c)
print 'Linear results (should be {}, {}, {}):'.format(a, b, c)
print linear_invert(f, x, y, z)
print 'Non-linear results (should be {}, {}, {}):'.format(a, b, c)
print nonlinear_invert(f, x, y, z)
def generate_data(nobservations, a, b, c, noise_level=0.01):
x, y, z = np.random.random((3, nobservations))
noise = noise_level * np.random.normal(0, noise_level, nobservations)
f = func(x, y, z, a, b, c) + noise
return f, x, y, z
def func(x, y, z, a, b, c):
f = (a**2
+ x * b**2
+ y * a * b * np.cos(c)
+ z * a * b * np.sin(c))
return f
def linear_invert(f, x, y, z):
G = np.vstack([np.ones_like(x), x, y, z]).T
m, _, _, _ = np.linalg.lstsq(G, f)
d, e, f, g = m
a = np.sqrt(d)
b = np.sqrt(e)
c = np.arctan2(g, f) # Note that `c` will be in radians, not degrees
return a, b, c
def nonlinear_invert(f, x, y, z):
# "curve_fit" expects the function to take a slightly different form...
def wrapped_func(observation_points, a, b, c):
x, y, z = observation_points
return func(x, y, z, a, b, c)
xdata = np.vstack([x, y, z])
model, cov = opt.curve_fit(wrapped_func, xdata, f)
return model
main()
You probably want to be using scipy's nonlinear solvers, they're really easy: http://docs.scipy.org/doc/scipy/reference/optimize.nonlin.html
Related
Fitting a coned semi torus surface to a scatter plot in python
So maybe this is more of a mathematical question but I'm a little stuck on this one. I have a scatter plot and am trying to fit a surface to it. Using two Gaussian's is working fairly well but its not perfect, and since I know the original data comes from a torus I'd like to force the fit to use an ellipse with an intensity. What I have currently created the following fig: You can see it works kinda well but its not quite giving me that perfect Torus shape I desire. I now show you the fit functions I'm currently using and ones I've tried. To generate the plot: # Our function to fit is going to be a sum of two-dimensional Gaussians def gaussian(x, y, x0, y0, xalpha, yalpha, A): return A * np.exp( -((x-x0)/xalpha)**2 -((y-y0)/yalpha)**2) # This is the callable that is passed to curve_fit. M is a (2,N) array # where N is the total number of data points in Z, which will be ravelled # to one dimension. def _gaussian(M, *args): x, y = M arr = np.zeros(x.shape) for i in range(len(args)//5): arr += gaussian(x, y, *args[i*5:i*5+5]) return arr And others I've tried: def distancefromellips(x, y, h, a, k, b): xfact = (x-h)**2/a**2 yfact = (x-k)**2/b**2 return abs(xfact-yfact-1) def distanceandgauss(x,y,h,a,k,b,x0, y0, xalpha, yalpha, A): return distancefromellips(x, y, h, a, k, b) * gaussian(x, y, x0, y0, xalpha, yalpha, A) def _ellipseG(M, *args): x, y = M arr = np.zeros(x.shape) for i in range(len(args)//9): arr += distanceandgauss(x, y, *args[i*9:i*9+9]) return arr Let me know what you think. Maybe the solution is just more Gaussian's? TIA.
The solution is to use a function in polar coordinates. A "coned torus" is just a Gaussian over R with some variance on theta. A function such as: def polarf(x, y, a, b, c, d, A, B, C, lamb): r = np.sqrt(x**2 + y**2) #print(r) theta = np.arctan(y/x) #print(theta.min()) return a * np.cos(theta*A) * np.exp(-(((r-b+B*np.cos(theta))**2)/c)) * (C*(np.exp(-lamb)*(lamb**r))/scipy.special.gamma(r+1)) #Standard best fit so far Will work well giving such an image:
How to minimise a sum of two functions allowing one of the arguments to vary over both functions?
I have two functions f(x,y,z) and g(x,y,z). I want to minimise the sum
h(x,y,z) = f(x,y,z) + g(x,y,z), allowing x to be variable over both functions f and g.
I can minimise both these functions separately or together using scipy.optimise.minimise, which basically calculates the values of f + g (or h) at a bunch of x, y and z values, and then returns me the values (x, y, z) for which f + g is minimum.
What happens here is that : both f and g are evaluated at same values of (x, y, z), but I want one of the arguments (say x) to vary over f and g.
This is a rough outline of what I am trying:
def f(x,y,z):
return scalar
def g(x,y,z):
return another_scalar
def h(theta):
x, y, z = theta
return f(x,y,z) + g(x,y,z)
def bestfit(guess, method='Nelder-Mead'):
result = op.minimize(h,
guess,
method=method,
options={'maxfev': 5000,
'disp': False})
if not result.success:
print('Optimisation did not converge.')
return result
g = [x0, y0, z0]
bf = bestfit(g, method='Nelder-Mead')
print(bf)
I am not sure if I can do it using scipy.optimise. Can I? Or is there some other python module I can use?
My first thought would be to define new functions, say a and b, with fixed values of y and z, such that your new functions are a(x) = f(x, y0, z0) and b(x) = g(x, y0, z0) and then minimize these functions.
Scipy ValueError: object too deep for desired array with optimize.leastsq
I am trying to fit my 3D data with linear 3D function Z = ax+by+c. I import the data with pandas:
dataframe = pd.read_csv('3d_data.csv',names=['x','y','z'],header=0)
print(dataframe)
x y z
0 52.830740 7.812507 0.000000
1 44.647931 61.031381 8.827942
2 38.725318 0.707952 52.857968
3 0.000000 31.026271 17.743218
4 57.137854 51.291656 61.546131
5 46.341341 3.394429 26.462564
6 3.440893 46.333864 70.440650
I have done some digging and found that the best way to fit 3D data it is to use optimize from scipy with the model equation and residual function:
def model_calc(parameter, x, y):
a, b, c = parameter
return a*x + b*y + c
def residual(parameter, data, x, y):
res = []
for _x in x:
for _y in y:
res.append(data-model_calc(parameter,x,y))
return res
I fit the data with:
params0 = [0.1, -0.2,1.]
result = scipy.optimize.leastsq(residual,params0,(dataframe['z'],dataframe['x'],dataframe['y']))
fittedParams = result[0]
But the result is a ValueError:
ValueError: object too deep for desired array [...]
minpack.error: Result from function call is not a proper array of floats.
I was trying to minimize the residual function to give only single value or single np.array but it didn't help. I don't know where is the problem and if maybe the search space for parameters it is not too complex. I would be very grateful for some hints!
If you are fitting parameters to a function, you can use curve_fit. Here's an implementation: from scipy.optimize import curve_fit def model_calc(X, a, b, c): x, y = X return a*x + b*y + c p0 = [0.1, -0.2, 1.] popt, pcov = curve_fit(model_calc, (dataframe.x, dataframe.y), dataframe.z, p0) #popt is the fit, pcov is the covariance matrix (see the docs) Note that your sintax must be if the form f(X, a, b, c), where X can be a 2D vector (See this post). (Another approach) If you know your fit is going to be linear, you can use numpy.linalg.lstsq. See here. Example solution: import numpy as np from numpy.linalg import lstsq A = np.vstack((dataframe.x, dataframe.y, np.ones_like(dataframe.y))).T B = dataframe.z a, b, c = lstsq(A, B)[0]
Why doesn't sympy simplify the Fourier Transform of a derivative?
We know that the Fourier Transform of a derivative is
where k is the fourier variable. Explanation here
My question is, why doesn't sympy use this knowledge? For example:
from sympy import Function, symbols, fourier_transform, Derivative
f = Function('f')
x, k= symbols('x, k')
G = fourier_transform(Derivative(f(x), x, x) + f(x), x, k)
print(G)
This prints
FourierTransform(f(x), x, k) + FourierTransform(Derivative(f(x), x, x), x, k)
But I expected it to print (up to some factors of 2 pi i)
FourierTransform(f(x), x, k) + k**2 FourierTransform(f(x), x, k)
Is there a way to tell sympy it's save to make this simplification because I expect f(x) -> 0 as x goes to infinity?
If not, what would be the cleanest way to make the substitution?
The simple reason Sympy doesn't do this is that it's not implemented yet. As a workaround for now, you can manually replace the FourierTransform of the derivative with a multiplication:
from sympy import Wild, FourierTransform, Derivative
a, b, c = symbols('a b c', cls=Wild)
G.replace(
FourierTransform(Derivative(a, b, b), b, c),
c**2 * FourierTransform(a, b, c)
)
As far as I know, Sympy doesn't offer a pattern that matches an arbitrary number of arguments, so you can't have a single pattern that matches Derivative(f(x), x), Derivative(f(x), x, x), Derivative(f(x), x, x, x), and so on. You could get around that by using the function-function form of replace(), but if you know what order of derivative you're dealing with, it's probably simpler to just put in that many bs explicitly, as I did in the example.
Forcing data to fit points with curve_fit
I have a little problem using the curve_fit function included in Scipy. Here is the function I would like to fit : def funclog(x, a, b, c, d): return a * np.log(b * x + c) + d The problem I have is that I would like the fit function to have a specific value on some points (y(min)=0 and y(max)=1). How can I force these points with curve_fit ? Thank you
The requirement of the fit having specific values at x=0, x=1, implies that the parameters a, b, c, d are constrained according to the set of two equations: funclog(0, a, b, c, d) = 0, funclog(1, a, b, c, d) = 1 For the form of funclog you are considering, you can solve this system of equations with respect to a and d resulting in the (unique) solution a = 1/(-log(c) + log(b + c)) and d=log(c)/(log(c) - log(b + c)) (assuming that b and c are such that the denominators are not equal to zero). Replacing these expressions for a and d in funclog results in a new fitting function, namely, (log(c) - log(b*x + c))/(log(c) - log(b + c)), which by default satisfies the constraints. The values of b and c can be found by curve_fit.
You may try use bounds: bounds = ([amin, bmin, cmin, dmin], [amax, bmax, cmax, dmax]) (or np.inf -np.inf if limes of param is in infininty) next popt1, pcov1 = curve_fit(funclog, x, y, bounds=bounds)