We Keep Coding

scipy.optimize.minimize does not converge in multivariable optimization

I want to find the values of board_trim and lm that will give me the lowest (closest to 0) value for Board_Moments.
For this I use scipy.optimize.minimize, but it does not converge. I really can't figure it out.
with Parameters:
displacement = 70
b = 6.5
deadrise = 20
LCG = 10
Vs_ms = 23.15 #ms
rho = 1025
mu = 1.19e-6
def Board_Moments(params):
board_trim, lm = params
displacement_N = displacement * 9.81 #kN
lp = Lp(Vs_ms, b, lm)
N = displacement_N * cos(d2r(board_trim)) #Drag Forces Perpendicular to the keel
#Taking moments about transom at height of CG
deltaM = (displacement_N * LCG) - (N * lp) #equilibrium condition
return deltaM
where lp:
def Lp(Vs_ms, b, lm):
cv = Cv(Vs_ms, b)
Lambda = Lambda_(lm, b)
Cp = 0.75 - (1 / (5.21 * (cv / Lambda)**2 + 2.39))
lp = Cp * lm
return lp
and
def Cv(Vs_ms, b):
cv = Vs_ms / (9.81 * b)**0.5
return cv
and
def Lambda_(lm, b):
lambda_ = lm / b
return lambda_
the optimization is done with:
board_trim = 2 #initial estimate
lm = 17.754 #initial estimate
x0 = [board_trim, lm]
Deltam = minimize(Board_Moments, x0, method = 'Nelder-Mead')
print(Deltam)
The error I get:
final_simplex: (array([[ 1.36119237e+01, 3.45635965e+23],
[-1.36046725e+01, 3.08439110e+23],
[ 2.07268577e+01, 2.59841956e+23]]), array([-7.64916992e+25,
-6.82618616e+25, -5.53373709e+25]))
fun: -7.649169916342451e+25
message: 'Maximum number of function evaluations has been exceeded.'
nfev: 401
nit: 220
status: 1
success: False
x: array([1.36119237e+01, 3.45635965e+23])
Any help would be much appreciated, thanks

You mention
that will give me the lowest (closest to 0) value for Board_Moments.
But minimize will search the absolute minimum. If you print the intermediate values for deltaM (which you should have done to debug your problem), you'll find they just get smaller and smaller, below zero (so -10, -100, -500 etc. That kind of progression).
To get as close to zero as possible, the solution is simply: return the absolute value of deltaM from Board_Moments:
def Board_Moments(params):
# code as before ...
deltaM = (displacement_N * LCG) - (N * lp) #equilibrium condition
# This print function would have shown the problem immediately
#print(deltaM)
# Use absolute (the built-in `abs` or `np.abs`;
# doesn't really matter for a single value)
# to get close to zero
return np.abs(deltaM)
For this particular case and fix, the result I get is:
final_simplex: (array([[ 2.32386388, 15.3390523 ],
[ 2.32394414, 15.33905343],
[ 2.32390145, 15.33905283]]), array([5.33445927e-08, 7.27723091e-08, 1.09428584e-07]))
fun: 5.334459274308756e-08
message: 'Optimization terminated successfully.'
nfev: 107
nit: 59
status: 0
success: True
x: array([ 2.32386388, 15.3390523 ])
(and if you comment out the print function, you'll see it easily converge towards zero.)

Combining Sympy with scipy.optimize.least_squares

I'm trying to calculate the value of the variables D and theta for which the function E attains its minimum value.
E = sqrt(x1-Dcos(theta)^2 + (y1-Dsin(theta)^2)) + sqrt(x2-2Dcos(theta)^2 + (y2-2Dsin(theta)^2)) + sqrt(x3-3Dcos(theta)^2 + (y3-3Dsin(theta)^2)).
Here, (x1,y1), (x2,y2), (x3,y3) are known. Now I calculate the partial derivatives of E wrt to D and theta and set them to zero. Now I have 2 equations and 2 unknowns, so theoretically this system should be exactly solvable. The only issue here is that this is highly non-linear. So analytical solutions are out of the question. I'm using Sympy here to calculate the partial derivatives and generate the equations to be used in least_squares from scipy.optimize. I do get a solution for D and theta but it doesn't make any physical sense. Furthermore, the cost value of least_squares is ~17, so my solutions are not very reliable, right? Could someone help me out here? Here's the code:
import sympy as sym
D, theta = sym.symbols("D, theta")
x1,x2,x3 = 9.0,22.0,24.0
y1,y2,y3 = 14.0,14.0,14.0
E = ((x1-D*sym.cos(theta))**2 + (y1-D*sym.sin(theta))**2)**0.5 + ((x2-2*D*sym.cos(theta))**2 + (y2-2*D*sym.sin(theta))**2)**0.5 + ((x3-3*D*sym.cos(theta))**2 + (y3-3*D*sym.sin(theta))**2)**0.5
gradient = sym.derive_by_array(E, (D, theta))
from scipy.optimize import least_squares as ls
grad = sym.lambdify((D, theta), gradient)
A = ls(lambda v: grad(v[0],v[1]), (8,0), bounds=([0, -22.5*np.pi/180], [12, 22.5*np.pi/180])) #theta should be between -22.5 degree and 22.5 degree and D should be between 0 and 12 but ideally not 0.
D2, theta2 = A.x # 0.04561884938833529 -0.3926990816987241
I should also mention that calculating D and theta is part of a problem that involves fitting a line through (x1,y1), (x2,y2), (x3,y3), and (13,2). D is the distance between (13,2) and the first point closest to (x1,y1) on the fitted line, 2D is similarly the distance between (13,2) and the second point closest to (x2,y2), and so on. This analysis has to be done over all the gridpoints on a lat-lon grid of size (21,69). All alternate suggestions to solve this problem are also welcome. Thanks in advance!

You are looking to approximate the sequence 9, 22, 24 with an equidistant sequence like 11,18,25. This does not have a good fit, so a large residual value is reasonable.
Doing the least square sum manually gives
(a-d-9)^2 + (a-22)^2 + (a+d-24)^2 = 3*a^2 +2*d^2 - 110*a - 30*d +const
so the optimum is at a = 55/3 = 18+1/3 and d = 15/2 = 7+1/2
On a second glance, you try to minimize the sum of norms, that is, the sum of distances from points on a line parallel to the x axis to points equidistant on a line through the origin. You can do that without doing any least-squares gradient optimization (which is a strange idea anyway, why not just find the zero location of the gradient with fsolve?)
x = np.array([ 9.0,22.0,24.0])
y = np.array([14.0,14.0,14.0])
K = 1+np.arange(len(x))
target = lambda X,Y: sum(np.hypot(x-K*X, y-K*Y))
from scipy.optimize import fmin, minimize
#U = fmin(lambda u: target(*u), [4,4])
U = minimize(lambda u: target(*u), [4,4])
print(U)
X,Y = U.x
print(X,Y, np.hypot(X,Y), np.arctan2(Y,X))
K = np.arange(1+len(x))
plt.plot(x,y,'o', ms=8); plt.plot(K*X,K*Y, '-s', ms=4); plt.grid(); plt.show()
with the results
fmin:
-----
Optimization terminated successfully.
Current function value: 16.987937
Iterations: 49
Function evaluations: 92
minimize:
---------
fun: 16.987921401556633
hess_inv: array([[2.72893764e-08, 6.01803467e-08],
[6.01803467e-08, 1.53257534e-07]])
jac: array([ 1.29665709, -0.08849001])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 740
nit: 29
njev: 182
status: 2
success: False
x: array([8. , 4.66666667])
X: 8.000000, Y: 4.666667,
D: 9.261629, theta: 0.5280744

scipy.optimize.minimize returns ABNORMAL_TERMINATION_IN_LNSRCH using L-BFGS-B

I'm using scipy.optimize.minimize to solve/calculate 3768 variables in 314 affine transformations to transform one point cloud to another eval_fun, firstly I tried using scipy.optimize.fmin_l_bfgs_b with approx_grad to avoid implementing gradient but it kept crashing with ABNORMAL_TERMINATION_IN_LNSRCH. Then I implemented gradient func and started using minimize and it still crashes with the same error.
res = minimize(eval_func,
x0=np.array(M),
method='L-BFGS-B',
args=(scan_pts, scan_fce, scan_nrm, scan_mar, temp_pts, temp_fce, temp_nrm, temp_mar, alfa, beta, gama,edges),
options={'iprint' : 99, 'maxiter' : 100}, jac=True)
This Is complete output:
This problem is unconstrained.
Line search cannot locate an adequate point after 20 function
and gradient evaluations. Previous x, f and g restored.
Possible causes: 1 error in function or gradient evaluation;
2 rounding error dominate computation.
RUNNING THE L-BFGS-B CODE
* * *
Machine precision = 2.220D-16
N = 3768 M = 10
At X0 0 variables are exactly at the bounds
At iterate 0 f= 1.88129D-01 |proj g|= 9.98119D-01
ITERATION 1
---------------- CAUCHY entered-------------------
There are 0 breakpoints
GCP found in this segment
Piece 1 --f1, f2 at start point -9.3847D+02 9.3847D+02
Distance to the stationary point = 1.0000D+00
---------------- exit CAUCHY----------------------
3768 variables are free at GCP 1
* * *
Tit = total number of iterations
Tnf = total number of function evaluations
Tnint = total number of segments explored during Cauchy searches
Skip = number of BFGS updates skipped
Nact = number of active bounds at final generalized Cauchy point
Projg = norm of the final projected gradient
F = final function value
* * *
N Tit Tnf Tnint Skip Nact Projg F
3768 1 21 1 0 0 9.981D-01 1.881D-01
F = 0.18812870968000006
ABNORMAL_TERMINATION_IN_LNSRCH
I tried changing all values as said here:
scipy.optimize.fmin_l_bfgs_b returns 'ABNORMAL_TERMINATION_IN_LNSRCH' but none of them work
What can I do?

Gradient-Based Optimizations in Python

I am trying to solve a couple minimization problems using Python but the setup with constraints is difficult for me to understand. I have:
minimize: x+y+2z^2
subject to: x = 1 and x^2+y^2 = 1
This is very easy obviously and I know the solution is x=1,y=0,z=0. I tried to use scipy.optimize.L-BFGS-B but had issues.
I also have:
minimize: 2x1^2+x2^2
subject to: x1+x2=1
I need to use a gradient based optimizer so I chose scipy.optimizer.COBYLA but had issues using an equality constraint as it only takes inequality constraints. The code for this is:
def objective(x):
x1 = x[0]
x2 = x[1]
return 2*(x1**2)+ x2
def constraint1(x):
return x[0]+x[1]-1
#Try an initial condition of x1=1 and x2=0
#Our initial condition satisfies the constraint already
x0 = [0.3,0.7]
print(objective(x0))
xnew = [0.25,0.75]
print(objective(xnew))
#Since we have already calculated on paper we know that x1 and x2 fall between 0 and 1
#We can set our bounds for both variables as being between 0 and 1
b = (0,1)
bounds = (b,b)
#Lets make note of the type of constraint we have for out optimizer
con1 = {'type': 'eq', 'fun':constraint1}
cons = [con1]
sol_gradient = minimize(objective,x0,method='COBYLA',bounds=bounds, constraints=cons)
Then I get error about using equality constraints with this optimizer.

A few things:
Your objective function does not match with the description you have provided. Should it be this: 2*(x1**2) + x2**2?
From the docs scipy.optimize.minimize you can see that COBYLA does not support eq as a constraint. From the page:
Note that COBYLA only supports inequality constraints.
Since you said you want to use a Gradient based optimizer, one option could be to use the Sequential Least Squares Programming (SLSQP) optimizer.
Below is the code replacing 'COBYLA' with 'SLSQP' and changing the objective function according to 1:
def objective(x):
x1 = x[0]
x2 = x[1]
return 2*(x1**2)+ x2**2
def constraint1(x):
return x[0]+x[1]-1
#Try an initial condition of x1=1 and x2=0
#Our initial condition satisfies the constraint already
x0 = [0.3,0.7]
print(objective(x0))
xnew = [0.25,0.75]
print(objective(xnew))
#Since we have already calculated on paper we know that x1 and x2 fall between 0 and 1
#We can set our bounds for both variables as being between 0 and 1
b = (0,1)
bounds = (b,b)
#Lets make note of the type of constraint we have for out optimizer
con1 = {'type': 'eq', 'fun':constraint1}
cons = [con1]
sol_gradient = minimize(objective,x0,method='SLSQP',bounds=bounds, constraints=cons)
print(sol_gradient)
Which gives the final answer as:
fun: 0.6666666666666665
jac: array([1.33333336, 1.33333335])
message: 'Optimization terminated successfully'
nfev: 7
nit: 2
njev: 2
status: 0
success: True
x: array([0.33333333, 0.66666667])

Minimize quadratic function subject to linear equality constraints with SciPy

I have a reasonably simple constrained optimization problem but get different answers depending on how I do it. Let's get the import and a pretty print function out of the way first:
import numpy as np
from scipy.optimize import minimize, LinearConstraint, NonlinearConstraint, SR1
def print_res( res, label ):
print("\n\n ***** ", label, " ***** \n")
print(res.message)
print("obj func value at solution", obj_func(res.x))
print("starting values: ", x0)
print("ending values: ", res.x.astype(int) )
print("% diff", (100.*(res.x-x0)/x0).astype(int) )
print("target achieved?",target,res.x.sum())
The sample data is very simple:
n = 5
x0 = np.arange(1,6) * 10_000
target = x0.sum() + 5_000 # increase sum from 15,000 to 20,000
Here's the constrained optimization (including jacobians). In words, the objective function I want to minimize is just the sum of squared percentage changes from the initial values to final values. The linear equality constraint is simply requiring x.sum() to equal a constant.
def obj_func(x):
return ( ( ( x - x0 ) / x0 ) ** 2 ).sum()
def obj_jac(x):
return 2. * ( x - x0 ) / x0 ** 2
def constr_func(x):
return x.sum() - target
def constr_jac(x):
return np.ones(n)
And for comparison, I've re-factored as an unconstrained minimization by using the equality constraint to replace x[0] with a function of x[1:]. Note that the unconstrained function is passed x0[1:] whereas the constrained function is passed x0.
def unconstr_func(x):
x_one = target - x.sum()
first_term = ( ( x_one - x0[0] ) / x0[0] ) ** 2
second_term = ( ( ( x - x0[1:] ) / x0[1:] ) ** 2 ).sum()
return first_term + second_term
I then try to minimize in three ways:
Unconstrained with 'Nelder-Mead'
Constrained with 'trust-constr' (w/ & w/o jacobian)
Constrained with 'SLSQP' (w/ & w/o jacobian)
Code:
##### (1) unconstrained
res0 = minimize( unconstr_func, x0[1:], method='Nelder-Mead') # OK, but weird note
res0.x = np.hstack( [target - res0.x.sum(), res0.x] )
print_res( res0, 'unconstrained' )
##### (2a) constrained -- trust-constr w/ jacobian
nonlin_con = NonlinearConstraint( constr_func, 0., 0., constr_jac )
resTCjac = minimize( obj_func, x0, method='trust-constr',
jac='2-point', hess=SR1(), constraints = nonlin_con )
print_res( resTCjac, 'trust-const w/ jacobian' )
##### (2b) constrained -- trust-constr w/o jacobian
nonlin_con = NonlinearConstraint( constr_func, 0., 0. )
resTC = minimize( obj_func, x0, method='trust-constr',
jac='2-point', hess=SR1(), constraints = nonlin_con )
print_res( resTC, 'trust-const w/o jacobian' )
##### (3a) constrained -- SLSQP w/ jacobian
eq_cons = { 'type': 'eq', 'fun' : constr_func, 'jac' : constr_jac }
resSQjac = minimize( obj_func, x0, method='SLSQP',
jac = obj_jac, constraints = eq_cons )
print_res( resSQjac, 'SLSQP w/ jacobian' )
##### (3b) constrained -- SLSQP w/o jacobian
eq_cons = { 'type': 'eq', 'fun' : constr_func }
resSQ = minimize( obj_func, x0, method='SLSQP',
jac = obj_jac, constraints = eq_cons )
print_res( resSQ, 'SLSQP w/o jacobian' )
Here is some simplified output (and of course you can run the code to get the full output):
starting values: [10000 20000 30000 40000 50000]
***** (1) unconstrained *****
Optimization terminated successfully.
obj func value at solution 0.0045454545454545305
ending values: [10090 20363 30818 41454 52272]
***** (2a) trust-const w/ jacobian *****
The maximum number of function evaluations is exceeded.
obj func value at solution 0.014635854609684874
ending values: [10999 21000 31000 41000 51000]
***** (2b) trust-const w/o jacobian *****
`gtol` termination condition is satisfied.
obj func value at solution 0.0045454545462939935
ending values: [10090 20363 30818 41454 52272]
***** (3a) SLSQP w/ jacobian *****
Optimization terminated successfully.
obj func value at solution 0.014636111111111114
ending values: [11000 21000 31000 41000 51000]
***** (3b) SLSQP w/o jacobian *****
Optimization terminated successfully.
obj func value at solution 0.014636111111111114
ending values: [11000 21000 31000 41000 51000]
Notes:
(1) & (2b) are plausible solutions in that they achieve significantly lower objective function values and intuitively we'd expect the variables with larger starting values to move more (both absolutely and in percentage terms) than the smaller ones.
Adding the jacobian to 'trust-const' causes it to get the wrong answer (or at least a worse answer) and also to exceed max iterations. Maybe the jacobian is wrong, but the function is so simple that I'm pretty sure it's correct (?)
'SLSQP' doesn't seem to work w/ or w/o the jacobian supplied, but works very fast and claims to terminate successfully. This seems very worrisome in that getting the wrong answer and claiming to have terminated successfully is pretty much the worst possible outcome.
Initially I used very small starting values and targets (just 1/1,000 of what I have above) and in that case all 5 approaches above work fine and give the same answers. My sample data is still extremely small, and it seems kinda bizarre for it to handle 1,2,..,5 but not 1000,2000,..5000.
FWIW, note that the 3 incorrect results all hit the target by adding 1,000 to each initial value -- this satisfies the constraint but comes nowhere near minimizing the objective function (b/c variables with higher initial values should be increased more than lower ones to minimize the sum of squared percentage differences).
So my question is really just what is happening here and why do only (1) and (2b) seem to work?
More generally, I'd like to find a good python-based approach to this and similar optimization problems and will consider answers using other packages besides scipy although the best answer would ideally also address what is going on with scipy here (e.g. is this user error or a bug I should post to github?).

Here is how this problem could be solved using nlopt which is a library for nonlinear optimization which I've been pretty impressed with.
First, the objective function and gradient are both defined using the same function:
def obj_func(x, grad):
if grad.size > 0:
grad[:] = obj_jac(x)
return ( ( ( x/x0 - 1 )) ** 2 ).sum()
def obj_jac(x):
return 2. * ( x - x0 ) / x0 ** 2
def constr_func(x, grad):
if grad.size > 0:
grad[:] = constr_jac(x)
return x.sum() - target
def constr_jac(x):
return np.ones(n)
Then, to run the minimization using Nelder-Mead and SLSQP:
opt = nlopt.opt(nlopt.LN_NELDERMEAD,len(x0)-1)
opt.set_min_objective(unconstr_func)
opt.set_ftol_abs(1e-15)
xopt = opt.optimize(x0[1:].copy())
xopt = np.hstack([target - xopt.sum(), xopt])
fval = opt.last_optimum_value()
print_res(xopt,fval,"Nelder-Mead");
opt = nlopt.opt(nlopt.LD_SLSQP,len(x0))
opt.set_min_objective(obj_func)
opt.add_equality_constraint(constr_func)
opt.set_ftol_abs(1e-15)
xopt = opt.optimize(x0.copy())
fval = opt.last_optimum_value()
print_res(xopt,fval,"SLSQP w/ jacobian");
And here are the results:
***** Nelder-Mead *****
obj func value at solution 0.00454545454546
result: 3
starting values: [ 10000. 20000. 30000. 40000. 50000.]
ending values: [10090 20363 30818 41454 52272]
% diff [0 1 2 3 4]
target achieved? 155000.0 155000.0
***** SLSQP w/ jacobian *****
obj func value at solution 0.00454545454545
result: 3
starting values: [ 10000. 20000. 30000. 40000. 50000.]
ending values: [10090 20363 30818 41454 52272]
% diff [0 1 2 3 4]
target achieved? 155000.0 155000.0
When testing this out, I think I discovered what the issue with the original attempt was. If I set the absolute tolerance on the function to 1e-8 which is what the scipy functions default to I get:
***** Nelder-Mead *****
obj func value at solution 0.0045454580693
result: 3
starting values: [ 10000. 20000. 30000. 40000. 50000.]
ending values: [10090 20363 30816 41454 52274]
% diff [0 1 2 3 4]
target achieved? 155000.0 155000.0
***** SLSQP w/ jacobian *****
obj func value at solution 0.0146361108503
result: 3
starting values: [ 10000. 20000. 30000. 40000. 50000.]
ending values: [10999 21000 31000 41000 51000]
% diff [9 5 3 2 2]
target achieved? 155000.0 155000.0
which is exactly what you were seeing. So my guess is that the minimizer ends up somewhere in the likelihood space during SLSQP where the next jump is less than 1e-8 from the last place.

This is a partial answer to the question that I'm putting here to keep the question from getting even bigger, but I'd still love to see a more comprehensive and explanatory answer. These answers are based on comments from two others, but neither of them fully wrote out the code, and I thought it would make sense to make that explicit so here it is:
Fixing 2a (trust-constr with jacobian)
It's seems that the key here with regard to the Jacobian and Hessian is to specify neither or both (but not the jacobian only). #SubhaneilLahiri commented to this effect and there was also an error message to this effect that I initially failed to notice:
UserWarning: delta_grad == 0.0. Check if the approximated function is linear. If the function is linear better results can be obtained by defining the Hessian as zero instead of using quasi-Newton approximations.
So I fixed it by defining the hessian function:
def constr_hess(x,v):
return np.zeros([n,n])
and adding it to the constraint
nonlin_con = NonlinearConstraint( constr_func, 0., 0., constr_jac, constr_hess )
Fixing 3a & 3b (SLSQP)
This just seemed to be a matter of making the tolerance smaller as suggested by #user545424. So I just added options={'ftol':1e-15} to the minimization:
resSQjac = minimize( obj_func, x0, method='SLSQP',
options={'ftol':1e-15},
jac = obj_jac, constraints = eq_cons )