Related
For instance if my numbers were: A=[1,2,3] B=[4,5,6] and C=[5,10,11] then the closest function I can get to would be 2A+B. Any idea about what kind of optimization pattern would be needed for something like this?
Assuming you want to find real values p and q such that the distance between pA + qB and C is minimal.
pA + qB = pi + 2pj + 3pk + 4qi + 5qj + 6qk = i(p + 4q) + j(2p + 5q) + k(3p + 6q)
The distance is given by:
f(p,q) = √((p + 4q - 5)² + (2p + 5q - 10)² + (3p + 6q - 11)²)
Take its partial derivatives and find the local minimum.
This can be set up as a system of linear equations and solved using numpy
import numpy as np
A=[1,2,3]
B=[4,5,6]
C=[5,10,11]
# Set up as a system of linear equations to solve with
# min mean square error, i.e. find x, y such that:
# Solve Ax + By = C
# As linear equations become
# [A; B] [x y] = C
# With M = [A; B] we have (i.e. A, B are the column vectors of M)
M = np.matrix((list(zip(A, B))))
# Find Least Square Error Solution
sol = np.linalg.lstsq(M, C, rcond=None)[0] # Solve for [x y]
print("Solution (Least Min Square Error\n", sol) # [2.11111111 0.88888889]
approx = np.dot(M, sol)
print("approximation\n", approx) # [[ 5.66666667 8.66666667 11.66666667]]
I'm currently working on an algorithm that determines the cost of a wind turbine support structure. The algorithm I'm writing needs to optimize the weight of an initial input support structure so that the stress levels will not exceed but be near to the failure criteria of the properties of the materials used. Another requirement is that the natural frequency of the structure needs to be bounded between 2 values. To optimize the structure 4 variables can be altered.
Can I use a function from the Scipy.Optimize library that optimizes the weight of this structure using several design parameters, but keeps into account the natural frequency and the maximum stress value in the support structure?
The function I'm optimizing looks like this:
def func(self, x):
self.properties.D_mp = x[0] # Set a new diameter for the monopile
self.properties.Dtrat_tower = x[1] # Set a new thickness ratio for the tower
self.properties.Dtrat_tp = x[2] # Set a new thickness ratio for the transition piece
self.properties.Dtrat_mud = x[3] # Set a new thickness ratio for the mudline region of the monopile
self.UpdateAll() # Update the support structure based on the changes in variables above
eig = self.GetEigenFrequency() # Get the natural frequency
maxUtil = self.GetMaximumUtilisationFactor() # Get the maximum utilisation ratio on the structure (more than 1 means stress is higher than maximum allowed)
# Natural frequency of 0.25 and utilisation ratio of 1 are ideal
# Create some penalty...
penalty = (100000 * abs((eig - 0.25)))
penalty += (100000 * abs(maxUtil - 1))
return self.GetTotalMass() + penalty
Thanks in advance!
It is probably easiest to make this a single-value optimization problem by folding in the frequency and stress constraints as penalties to the overall fitness, something like
LOW_COST = 10.
MID_COST = 150.
HIGH_COST = 400.
def weight(a, b, c, d):
return "calculated weight of structure"
def frequency(a, b, c, d):
return "calculated resonant frequency"
def freq_penalty(freq):
# Example linear piecewise penalty function -
# increasing cost for frequencies below 205 or above 395
if freq < 205:
return MID_COST * (205 - freq)
elif freq < 395:
return 0.
else:
return MID_COST * (freq - 395)
def stress_fraction(a, b, c, d):
return "calculated stress / failure criteria"
def stress_penalty(stress_frac):
# Example linear piecewise penalty function -
# low extra cost for stress fraction below 0.85,
# high extra cost for stress fraction over 0.98
if stress_frac < 0.85:
return LOW_COST * (0.85 - stress_frac)
elif stress_frac < 0.98:
return 0.
else:
return HIGH_COST * (stress_frac - 0.98)
def overall_fitness(parameter_vector):
a, b, c, d = parameter_vector
return (
# D'oh! it took me a while to get this right -
# we want _minimum_ weight and _minimum_ penalty
# to get _maximum_ fitness.
-weight(a, b, c, d)
- freq_penalty(frequency(a, b, c, d))
- stress_penalty(stress_fraction(a, b, c, d)
)
... you will of course want to find more suitable penalty functions and play with the relative weighting, but this should give you the general idea. Then you can maximize it like
from scipy.optimize import fmin
initial_guess = [29., 45., 8., 0.06]
result = fmin(lambda x: -overall_fitness(x), initial_guess, maxfun=100000, full_output=True)
(using the lambda lets fmin (a minimizer) find a maximum value for overall_fitness).
Alternatively, fmin allows a callback function which is applied after each iteration; you could use this to put hard limits on frequency IF you know how to tweak a,b,c,d appropriately - something like
def callback(x):
a, b, c, d = x # unpack parameter vector
freq = frequency(a, b, c, d)
if freq < 205:
# apply appropriate correction to put frequency back in bounds
return [a, b, c + 0.2, d]
elif freq < 395:
return x
else:
return [a, b, c - 0.2, d]
You can use the leastsq function of spicy.optimize.
In my case it was to fit an exponential function with two variables :
def func_exp(p, x, z):
# exponential function with multiple parameters
a, b, c, d, t, t2 = p[0], p[1], p[2], p[3], p[4], p[5]
return a*np.exp(b + pow(x,c)*t + pow(z,d)*t2)
But to use the leastsq function you need to create an error function, this this one you'll optimize.
def err(p, x,z, y):
# error function compare the previous to the estimate to
return func_exp(p, x,z) - y
# minimise the residuals
To use it :
p0=[1e4,-1e-3,1,1,-1e-2, -1e-6]
# First parameters
pfit_fin, pcov, infodict, errmsg, success = leastsq(err, p0, args=(X,Y,Z), full_output=1, epsfcn=0.000001)
So that's return the best parameters, to generate the results :
Y_2= func_exp(pfit_fin, X,Y)
I hope that will help you,
Chris.
We know that the Fourier Transform of a derivative is
where k is the fourier variable. Explanation here
My question is, why doesn't sympy use this knowledge? For example:
from sympy import Function, symbols, fourier_transform, Derivative
f = Function('f')
x, k= symbols('x, k')
G = fourier_transform(Derivative(f(x), x, x) + f(x), x, k)
print(G)
This prints
FourierTransform(f(x), x, k) + FourierTransform(Derivative(f(x), x, x), x, k)
But I expected it to print (up to some factors of 2 pi i)
FourierTransform(f(x), x, k) + k**2 FourierTransform(f(x), x, k)
Is there a way to tell sympy it's save to make this simplification because I expect f(x) -> 0 as x goes to infinity?
If not, what would be the cleanest way to make the substitution?
The simple reason Sympy doesn't do this is that it's not implemented yet. As a workaround for now, you can manually replace the FourierTransform of the derivative with a multiplication:
from sympy import Wild, FourierTransform, Derivative
a, b, c = symbols('a b c', cls=Wild)
G.replace(
FourierTransform(Derivative(a, b, b), b, c),
c**2 * FourierTransform(a, b, c)
)
As far as I know, Sympy doesn't offer a pattern that matches an arbitrary number of arguments, so you can't have a single pattern that matches Derivative(f(x), x), Derivative(f(x), x, x), Derivative(f(x), x, x, x), and so on. You could get around that by using the function-function form of replace(), but if you know what order of derivative you're dealing with, it's probably simpler to just put in that many bs explicitly, as I did in the example.
I have a simple x,y data set to fit, at least at first glance. The issue is that scipy.optimize.curve_fit gives back a very large value for one of the parameters fitted and I don't know if this is mathematically correct or if there's something wrong with how I'm fitting the data.
The figure below shows both the data points and the best fit obtained in blue. The curve used (func in the MWE below) has four parameters a, b, c, d being fitted:
a gives approximately the x value where the curve attains it's half-maximum.
b represents the x value where the curve stabilizes. This func value is given by the d parameter, ie: func(b) = d
c is related to the maximum value of the curve at the origin: func(0) = c*constant + d
d is where the curve stabilizes (black line in the figure).
The b parameter is the one I'm having issues with (see end of question), it's also the one I'm most interested in assigning a reasonable value.
The MWE shows the function being fitted and the results:
import numpy as np
from scipy.optimize import curve_fit
import matplotlib.pyplot as plt
# Function to be fitted.
def func(x, a, b, c, d):
return c * (1 / np.sqrt(1 + (np.asarray(x) / a) ** 2) -
1 / np.sqrt(1 + (b / a) ** 2)) ** 2 + d
# Define x,y data.
x_list = [12.5, 37.5, 62.5, 87.5, 112.5, 137.5, 162.5, 187.5, 212.5, 237.5,
262.5, 287.5, 312.5, 337.5, 362.5, 387.5, 412.5, 437.5, 462.5, 487.5,
512.5]
y_list = [0.008, 0.0048, 0.0032, 0.00327, 0.0023, 0.00212, 0.00187,
0.00086, 0.00070, 0.00100, 0.00056, 0.00076, 0.00052, 0.00077, 0.00067,
0.00048, 0.00078, 0.00067, 0.00069, 0.00061, 0.00047]
# Initial guess for the 4 parameters.
guess = (50., 200., 80. / 10000., 6. / 10000.)
# Fit curve to x,y data.
f_prof, f_err = curve_fit(func, x_list, y_list, guess)
# Values for the a,b,c,d fitted parameters.
print f_prof
# Errors (standard deviations) for the fitted parameters.
print np.sqrt(f_err[0][0]), np.sqrt(f_err[1][1]), np.sqrt(f_err[2][2]),\
np.sqrt(f_err[3][3])
# Generate plot.
plt.scatter(x_list, y_list)
plt.plot(x_list, func(x_list, f_prof[0], f_prof[1], f_prof[2], f_prof[3]))
plt.hlines(y=f_prof[3], xmin=0., xmax=max(x_list))
plt.show()
The results I get are:
# a, b, c, d
52.74, 2.52e+09, 7.46e-03, 5.69e-04
# errors
11.52, 1.53e+16, 0.0028, 0.00042
The b parameter has a huge value and also a huge error. By looking at the data plotted in the figure, one would estimate by eye that the value for b (ie: the x value where the data set stabilizes) should be around x=300. Why am I getting such a large value for b and its error?
I don't know if this is intentional or a mistake but it looks to me like 'b' will be correlated strongly with 'a' and 'd' and has no "interaction" with the independent variable 'x'. If b/a is large enough, you could approximate 1/np.sqrt(1 + (b / a) ** 2)) ** 2 as a/b, so that your function becomes
c * function_of(x, a) - a/b + d
Your 'a' and 'x' values are large enough that this becomes very nearly c*a/x - a/b + d.
As pointed by behzad.nouri, curve_fit can be slightly unstable compared to other minimizers, and always minimizes the least-squares. But it does return the full covariance matrix including correlations between variables (off-diagonal elements of your f_err). Use these!!
If you're sure 'b' has a value around 300, or are interested in easily switching between fmin and the levenberg-marquardt algorithm, you might find the lmfit package (http://lmfit.github.io/lmfit-py/) useful. It allows you to put bounds on parameters, easily switch between fitting algorithms, and also to do a more brute-force exploration of the confidence intervals for the parameters.
you can use a penalty value for the norm of parameters, and use fmin:
from scipy.optimize import fmin
def func(x, a, b, c, d):
return c * (1 / np.sqrt(1 + (x / a) ** 2) - 1 / np.sqrt(1 + (b / a) ** 2)) ** 2 + d
def errfn(params, xs, ys, lm, ord=1):
'''
lm: penalty maltiplier
ord: order in norm calculation
'''
from numpy.linalg import norm
a, b, c, d = params
err = func(xs, a, b, c, d) - ys
return norm(err) + lm * norm(params, ord)
params = fmin(errfn, guess, args=(xs, ys, 1e-6, 2))
above I am using a small penalty of 1e-6 and the fit result are
[6.257e+01 3.956e+02 9.926e-03 7.550e-04]
with a decent fit:
edit: playing with the penalty function and the norm order, it gives a very good fit at
params = [ 1.479e+01 -3.344e+00 -8.781e-03 8.347e-03]
From a quick look, it seems that a large b will eliminate the second term of func():
When b/a goes to infinity, 1 / np.sqrt(1 + (b / a) ** 2)) ** 2 goes to zero.
This suggests to me that this part of the function is not needed in the model and is doing more damage than good.
Just set func to be:
c * (1 / np.sqrt(1 + (np.asarray(x) / a) ** 2) + d
I have 4 non-linear equations with three unknowns X, Y, and Z that I want to solve for. The equations are of the form:
F(m) = X^2 + a(m)Y^2 + b(m)XYcosZ + c(m)XYsinZ
...where a, b and c are constants which are dependent on each value of F in the four equations.
What is the best way to go about solving this?
There are two ways to do this.
Use a non-linear solver
Linearize the problem and solve it in the least-squares sense
Setup
So, as I understand your question, you know F, a, b, and c at 4 different points, and you want to invert for the model parameters X, Y, and Z. We have 3 unknowns and 4 observed data points, so the problem is overdetermined. Therefore, we'll be solving in the least-squares sense.
It's more common to use the opposite terminology in this case, so let's flip your equation around. Instead of:
F_i = X^2 + a_i Y^2 + b_i X Y cosZ + c_i X Y sinZ
Let's write:
F_i = a^2 + X_i b^2 + Y_i a b cos(c) + Z_i a b sin(c)
Where we know F, X, Y, and Z at 4 different points (e.g. F_0, F_1, ... F_i).
We're just changing the names of the variables, not the equation itself. (This is more for my ease of thinking than anything else.)
Linear Solution
It's actually possible to linearize this equation. You can easily solve for a^2, b^2, a b cos(c), and a b sin(c). To make this a bit easier, let's relabel things yet again:
d = a^2
e = b^2
f = a b cos(c)
g = a b sin(c)
Now the equation is a lot simpler: F_i = d + e X_i + f Y_i + g Z_i. It's easy to do a least-squares linear inversion for d, e, f, and g. We can then get a, b, and c from:
a = sqrt(d)
b = sqrt(e)
c = arctan(g/f)
Okay, let's write this up in matrix form. We're going to translate 4 observations of (the code we'll write will take any number of observations, but let's keep it concrete at the moment):
F_i = d + e X_i + f Y_i + g Z_i
Into:
|F_0| |1, X_0, Y_0, Z_0| |d|
|F_1| = |1, X_1, Y_1, Z_1| * |e|
|F_2| |1, X_2, Y_2, Z_2| |f|
|F_3| |1, X_3, Y_3, Z_3| |g|
Or: F = G * m (I'm a geophysist, so we use G for "Green's Functions" and m for "Model Parameters". Usually we'd use d for "data" instead of F, as well.)
In python, this would translate to:
def invert(f, x, y, z):
G = np.vstack([np.ones_like(x), x, y, z]).T
m, _, _, _ = np.linalg.lstsq(G, f)
d, e, f, g = m
a = np.sqrt(d)
b = np.sqrt(e)
c = np.arctan2(g, f) # Note that `c` will be in radians, not degrees
return a, b, c
Non-linear Solution
You could also solve this using scipy.optimize, as #Joe suggested. The most accessible function in scipy.optimize is scipy.optimize.curve_fit which uses a Levenberg-Marquardt method by default.
Levenberg-Marquardt is a "hill climbing" algorithm (well, it goes downhill, in this case, but the term is used anyway). In a sense, you make an initial guess of the model parameters (all ones, by default in scipy.optimize) and follow the slope of observed - predicted in your parameter space downhill to the bottom.
Caveat: Picking the right non-linear inversion method, initial guess, and tuning the parameters of the method is very much a "dark art". You only learn it by doing it, and there are a lot of situations where things won't work properly. Levenberg-Marquardt is a good general method if your parameter space is fairly smooth (this one should be). There are a lot of others (including genetic algorithms, neural nets, etc in addition to more common methods like simulated annealing) that are better in other situations. I'm not going to delve into that part here.
There is one common gotcha that some optimization toolkits try to correct for that scipy.optimize doesn't try to handle. If your model parameters have different magnitudes (e.g. a=1, b=1000, c=1e-8), you'll need to rescale things so that they're similar in magnitude. Otherwise scipy.optimize's "hill climbing" algorithms (like LM) won't accurately calculate the estimate the local gradient, and will give wildly inaccurate results. For now, I'm assuming that a, b, and c have relatively similar magnitudes. Also, be aware that essentially all non-linear methods require you to make an initial guess, and are sensitive to that guess. I'm leaving it out below (just pass it in as the p0 kwarg to curve_fit) because the default a, b, c = 1, 1, 1 is a fairly accurate guess for a, b, c = 3, 2, 1.
With the caveats out of the way, curve_fit expects to be passed a function, a set of points where the observations were made (as a single ndim x npoints array), and the observed values.
So, if we write the function like this:
def func(x, y, z, a, b, c):
f = (a**2
+ x * b**2
+ y * a * b * np.cos(c)
+ z * a * b * np.sin(c))
return f
We'll need to wrap it to accept slightly different arguments before passing it to curve_fit.
In a nutshell:
def nonlinear_invert(f, x, y, z):
def wrapped_func(observation_points, a, b, c):
x, y, z = observation_points
return func(x, y, z, a, b, c)
xdata = np.vstack([x, y, z])
model, cov = opt.curve_fit(wrapped_func, xdata, f)
return model
Stand-alone Example of the two methods:
To give you a full implementation, here's an example that
generates randomly distributed points to evaluate the function on,
evaluates the function on those points (using set model parameters),
adds noise to the results,
and then inverts for the model parameters using both the linear and non-linear methods described above.
import numpy as np
import scipy.optimize as opt
def main():
nobservations = 4
a, b, c = 3.0, 2.0, 1.0
f, x, y, z = generate_data(nobservations, a, b, c)
print 'Linear results (should be {}, {}, {}):'.format(a, b, c)
print linear_invert(f, x, y, z)
print 'Non-linear results (should be {}, {}, {}):'.format(a, b, c)
print nonlinear_invert(f, x, y, z)
def generate_data(nobservations, a, b, c, noise_level=0.01):
x, y, z = np.random.random((3, nobservations))
noise = noise_level * np.random.normal(0, noise_level, nobservations)
f = func(x, y, z, a, b, c) + noise
return f, x, y, z
def func(x, y, z, a, b, c):
f = (a**2
+ x * b**2
+ y * a * b * np.cos(c)
+ z * a * b * np.sin(c))
return f
def linear_invert(f, x, y, z):
G = np.vstack([np.ones_like(x), x, y, z]).T
m, _, _, _ = np.linalg.lstsq(G, f)
d, e, f, g = m
a = np.sqrt(d)
b = np.sqrt(e)
c = np.arctan2(g, f) # Note that `c` will be in radians, not degrees
return a, b, c
def nonlinear_invert(f, x, y, z):
# "curve_fit" expects the function to take a slightly different form...
def wrapped_func(observation_points, a, b, c):
x, y, z = observation_points
return func(x, y, z, a, b, c)
xdata = np.vstack([x, y, z])
model, cov = opt.curve_fit(wrapped_func, xdata, f)
return model
main()
You probably want to be using scipy's nonlinear solvers, they're really easy: http://docs.scipy.org/doc/scipy/reference/optimize.nonlin.html