I'm trying to find the shortest path between two points, (0,0) and (1000,-100). The path is to be defined by a 7th order polynomial function:
p(x) = a0 + a1*x + a2*x^2 + ... + a7*x^7
To do so I tried to minimize the function that calculates the total path length from the polynomial function:
length = int from 0 to 1000 of { sqrt(1 + (dp(x)/dx)^2 ) }
Obviously the correct solution will be a linear line, however later on I want to add constraints to the problem. This one was supposed to be a first approach.
The code I implemented was:
import numpy as np
import matplotlib.pyplot as plt
import math
import sys
import scipy
def path_tracer(a,x):
return a[0] + a[1]*x + a[2]*x**2 + a[3]*x**3 + a[4]*x**4 + a[5]*x**5 + a[6]*x**6 + a[7]*x**7
def lof(a):
upper_lim = a[8]
L = lambda x: np.sqrt(1 + (a[1] + 2*a[2]*x + 3*a[3]*x**2 + 4*a[4]*x**3 + 5*a[5]*x**4 + 6*a[6]*x**5 + 7*a[7]*x**6)**2)
length_of_path = scipy.integrate.quad(L,0,upper_lim)
return length_of_path[0]
a = np.array([-4E-11, -.4146,.0003,-7e-8,0,0,0,0,1000]) # [polynomial parameters, x end point]
xx = np.linspace(0,1200,1200)
y = [path_tracer(a,x) for x in xx]
cons = ({'type': 'eq', 'fun': lambda x:path_tracer(a,a[8])+50})
c = scipy.optimize.minimize(lof, a, constraints = cons)
print(c)
When I ran it however the minimization routine fails and returns the initial parameters unchanged. The output is:
fun: 1022.9651540965604
jac: array([ 0.00000000e+00, -1.78130722e+02, -1.17327499e+05,
-7.62458172e+07, 9.42803815e+11, 9.99924786e+14,
9.99999921e+17, 1.00000000e+21, 1.00029755e+00])
message: 'Singular matrix C in LSQ subproblem'
nfev: 11
nit: 1
njev: 1
status: 6
success: False
x: array([ -4.00000000e-11, -4.14600000e-01, 3.00000000e-04,
-7.00000000e-08, 0.00000000e+00, 0.00000000e+00,
0.00000000e+00, 0.00000000e+00, 1.00000000e+03])
Am I doing something wrong or is the routine just not appropriate to solve this kind of problems? If so, is there an alternative in Python?
You can use this routine, but there are some problems with your approach:
The domain of the polynomial should be normalized to something reasonable, like [0, 1]. This makes the optimization much easier. You can revert this after you are done with the optimization
You could simplify the code by using polyval and related functions
The optimal solution to this is quite obviously -0.1 x, so I'm not sure why you feel the need to optimize.
A solution that works is
import numpy as np
import scipy.optimize
x = np.linspace(0, 1, 1000)
def obj_fun(p):
deriv = np.polyval(np.polyder(p), x)
return np.sum(np.sqrt(1 + deriv ** 2))
cons = ({'type': 'eq', 'fun': lambda p: np.polyval(p, [0, 1]) - [0, -100]})
p0 = np.zeros(8)
c = scipy.optimize.minimize(obj_fun, p0, constraints = cons)
Where we can plot the result
import matplotlib.pyplot as plt
plt.plot(np.polyval(c.x, x), label='result')
plt.plot(-100 * x, label='optimal')
plt.legend()
Related
I would like to get an optimal solution for following equation set:
x_w * 1010 + x_m * d_m = 1017
x_w + x_m = 1
my code is as follows:
from scipy.optimize import minimize
import numpy as np
def f1(p):
x_w, x_m, d_m = p
return (x_w*1010 + x_m*d_m) - 1017.7
def f2(p):
x_w, x_m, d_m = p
return x_w + x_m - 1
bounds =[(0,1), (0,1), (1000, 10000)]
x0 = np.array([0.5, 0.5, 1500])
res = minimize(lambda p: f1(p)+f2(p), x0=x0, bounds=bounds)
However, all I get back (res.x) are the initial values (x0).
How do I make it work? Is there a better approach? There are just these two equations for the three variables.
In general, you can't solve the equation system by minimizing f1(p) + f2(p) since the minimum of this objective is no solution of the equation system. However, you have to minimize the sum of squared errors of each equation, i.e. you minimize f1(p)**2 + f2(p)**2:
minimize(lambda p: f1(p)**2 + f2(p)**2, x0=x0, bounds=bounds)
Alternatively, you could use scipy.optimize.fsolve which doesn't support bounds, unfortunately.
I'm trying to solve a set of equations using scipy.minimize, however I'm not getting satisfatory results, so maybe I'm gettting something wrong.
I want to solve the following system of equations.
12.25 * (x + y * 2.2 + z * 4.84) - 8.17437483750257 = 0
12.25 * (x + y * 3.1 + z * 9.61) - 21.9317236606432 = 0
12.25 * (x + y * 4 + z * 16) - 107.574834524443 = 0
Using Wolfram Alpha I get the answers
x=22.626570068753, y=-17.950683342597, z=3.6223614029055
Which indeed solve the system of equations, giving a residual error of
9.407585821463726e-12
Now using scipy.minimize I do:
import numpy as np
from scipy.optimize import fsolve
from scipy.optimize import minimize
def my_func(p):
points = [8.17437483750257, 21.9317236606432, 107.574834524443]
h1 = abs(12.25 * (p[0] + p[1] * 2.2 + p[2] * 4.84) - points[0])
h2 = abs(12.25 * (p[0] + p[1] * 3.1 + p[2] * 9.61) - points[1])
h3 = abs(12.25 * (p[0] + p[1] * 4 + p[2] * 16) - points[2])
return h1 + h2 + h3
ini = np.array([22, -15, 5]) # Initial points close to solution
res = minimize(my_func, ini)
print(res)
fun: 1.4196640741924451
hess_inv: array([[ 20.79329103, -14.63447889, 2.36145776],
[-14.63447889, 10.30037625, -1.66214485],
[ 2.36145776, -1.66214485, 0.26822135]])
jac: array([ 12.25 , 60.02499545, 254.43249989])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 261
nit: 8
njev: 64
status: 2
success: False
x: array([ 21.39197235, -17.08623345, 3.48344393])
First, It says success=False and second it finds solutions that are not optimal.
Why having initial values that are close to the optimal solution it fails to find those solutions.
Is it something wrong with the definition of the optimizer?
Tried running it giving initial values of [0,0,0] and it just gives awful results
ini = np.array([0, 0, 0]) # Initial points close to solution
res = minimize(my_func, ini)
print(res)
fun: 73.66496363902732
hess_inv: array([[ 0.98461683, -0.04223651, -0.1207056 ],
[-0.04223651, 0.88596592, -0.31885642],
[-0.1207056 , -0.31885642, 0.13448927]])
jac: array([ 12.25 , 15.92499924, -18.98750019])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 164
nit: 1
njev: 40
status: 2
success: False
x: array([0.02901304, 0.08994042, 0.29448233])
Note: I don't want to use fsolve to find the solutions, but minimize.
The reason is that my real problem involves having more equations than unknowns, so at the end I want a solution that minimizes the errors of all this equations.
However since it was not giving good results, I wanted to first test on an easy problem for which an exact solution exists. But even in this case it doesn't work.
Once I make it work for this problem I'll expand it adding more equations.
...my real problem involves having more equations than unknowns, so at the end I want a solution that minimizes the errors of all this equations
This sounds a lot like the problem solved in the generalized method of moments (GMM), where you also have more equations than unknowns.
Problems of this kind are usually solved using least squares. Suppose your whole system looks like this:
h1(x, y, z) = 0
h2(x, y, z) = 0
h3(x, y, z) = 0
h4(x, y, z) = 0
It has 3 unknowns and 4 equations. Then your objective function will be:
F(x, y, z) = H(x, y, z)' * W * H(x, y, z)
H(x, y, z) is the vector of all hj(x, y, z) above
H(x, y, z)' is its transpose
W is the weighting matrix
If W is the identity matrix, you get the least squares objective function. Then, F(x, y, z) is a quadratic form (basically a parabola in multiple dimensions), which should be simple to optimize because it's convex and smooth.
Your code uses absolute values like h1 = abs(12.25 * (p[0] + p[1] * 2.2 + p[2] * 4.84) - points[0]), but abs can be difficult to differentiate near the origin, yet that's exactly where your optimum lies, since you essentially want h1 to equal zero.
You can approximate the absolute value function by squaring the error:
h1 =(12.25 * (p[0] + p[1] * 2.2 + p[2] * 4.84) - points[0])**2
This results in basically the same approach as the GMM (or least squares) and gives you a function that's easy to optimize since the square is smooth near the origin.
Optimisation problems (and solvers) usually benefit from a well behaved (smooth) "optimisation surface". When you use the abs function, it creates a "choppy" surface, with points were the derivatives are not continuous.
If instead of using abs you use a quadratic function (that has the same effect), you get a solution close to what you expect. Just change my_func to:
def my_func(p):
points = [8.17437483750257, 21.9317236606432, 107.574834524443]
h1 = (12.25 * (p[0] + p[1] * 2.2 + p[2] * 4.84) - points[0])**2
h2 = (12.25 * (p[0] + p[1] * 3.1 + p[2] * 9.61) - points[1])**2
h3 = (12.25 * (p[0] + p[1] * 4 + p[2] * 16) - points[2])**2
return h1 + h2 + h3
What I got is:
fun: 8.437863292878727e-10
hess_inv: array([[ 0.64753863, -0.43474506, 0.06909179],
[-0.43474506, 0.29487798, -0.04722923],
[ 0.06909179, -0.04722923, 0.00761762]])
jac: array([-6.26698693e-12, 6.22490927e-10, -5.11716516e-10])
message: 'Optimization terminated successfully.'
nfev: 55
nit: 7
njev: 11
status: 0
success: True
x: array([ 22.62653789, -17.95066124, 3.62235782])
I want to optimise my portfolio using Markowitz theory (risk minimization by the Markowitz method for a given income = 15%) and Scipy.minimize
I have risk function
def objective(x):
x1=x[0];x2=x[1];x3=x[2]; x4=x[3]
return 1547.87020*x1**2 + 125.26258*x1*x2 + 1194.3433*x1*x3 + 63.6533*x1*x4 \
+ 27.3176649*x2**2 + 163.28848*x2*x3 + 4.829816*x2*x4 \
+ 392.11819*x3**2 + 56.50518*x3*x4 \
+ 34.484063*x4**2
Sum of parts of stocks(in %) = 1
def constraint1(x):
return (x[0]+x[1]+x[2]+x[3]-1.0)
Income function with restriction
def constraint2(x):
return (-1.37458*x[0] + 0.92042*x[1] + 5.06189*x[2] + 0.35974*x[3] - 15.0)
And I test it using:
x0=[0,1,1,0] #Initial value
b=(0.0,1.0)
bnds=(b,b,b,b)
con1={'type':'ineq','fun':constraint1}
con2={'type':'eq','fun':constraint2}
cons=[con1,con2]
sol=minimize(objective,x0,method='SLSQP',\
bounds=bnds,constraints=cons)
And my result is:
fun: 678.5433939
jac: array([1383.25920868, 222.75363159, 1004.03005219, 130.30312347])
message: 'Positive directional derivative for linesearch'
nfev: 216
nit: 20
njev: 16
status: 8
success: False
x: array([0., 1., 1., 1.])
But how? Sum of parts of portfolio cant be more than 1(now parts of stock 2=stock3=stock4=100%). Its constraint1. Where is problem?
The output says "success: False"
So it is telling you that it failed to find a solution to the problem.
Also, why did you put
con1={'type':'ineq','fun':constraint1}
Don't you want
con1={'type':'eq','fun':constraint1}
I got success using method='BFGS'
Your code is returning values that do not respect your constraint due to false definition of the first constraint (a-b >= 0 => a>b) so in your case a=1(the order in an inequality is important). On the other hand your x0 must also respect your constraints and sum([0,1,1,0]) = 2 > 1.
I slightly improved your code and fixed the aforementioned issues, but I still think that you need to review your second constraint:
import numpy as np
from scipy.optimize import minimize
def objective(x):
x1, x2, x3, x4 = x[0], x[1], x[2], x[3]
coefficients = np.array([1547.87020, 125.26258, 1194.3433, 63.6533, 27.3176649, 163.28848, 4.829816, 392.11819, 56.50518, 34.484063])
xs = np.array([ x1**2, x1*x2, x1*x3, x1*x4, x2**2, x2*x3, x2*x4, x3**2, x3*x4, x4**2])
return np.dot(xs, coefficients)
const1 = lambda x: 1 - sum(x)
const2 = lambda x: np.dot(np.array([-1.37458, 0.92042, 5.06189, 0.35974]), x) - 15.0
x0 = [0, 0, 0, 0] #Initial value
b = (0.0, 1.0)
bnds = (b, b, b, b)
cons = [{'type':'ineq','fun':const1}, {'type':'eq', 'fun':const2}]
# minimize
sol = minimize(objective,
x0,
method = 'SLSQP',
bounds = bnds,
constraints = cons)
print(sol)
output:
fun: 392.1181900000138
jac: array([1194.34332275, 163.28847885, 784.23638535, 56.50518036])
message: 'Positive directional derivative for linesearch'
nfev: 92
nit: 11
njev: 7
status: 8
success: False
x: array([0.00000000e+00, 5.56638069e-14, 1.00000000e+00, 8.29371293e-14])
how can I fit the differential function of the followint scipy tutorial
Scipy Differential Equation Tutorial?
In the end I want to fit some datapoints that follow a set of two differential equations with six parameters in total but I'd like to start with an easy example. So far I tried the functions scipy.optimize.curve_fit and scipy.optimize.leastsq but I did not get anywhere.
So this is how far I came:
import numpy as np
import scipy.optimize as scopt
import scipy.integrate as scint
import scipy.optimize as scopt
def pend(y, t, b, c):
theta, omega = y
dydt = [omega, -b*omega - c*np.sin(theta)]
return dydt
def test_pend(y, t, b, c):
theta, omega = y
dydt = [omega, -b*omega - c*np.sin(theta)]
return dydt
b = 0.25
c = 5.0
y0 = [np.pi - 0.1, 0.0]
guess = [0.5, 4]
t = np.linspace(0, 1, 11)
sol = scint.odeint(pend, y0, t, args=(b, c))
popt, pcov = scopt.curve_fit(test_pend, guess, t, sol)
with the following error message:
ValueError: too many values to unpack (expected 2)
And I'm sorry as this is assumingly a pretty simple question but I don't get it to work. So thanks in advance.
You need to provide a function f(t,b,c) that given an argument or a list of arguments in t returns the value of the function at the argument(s). This requires some work, either by determining the type of t or by using a construct that works either way:
def f(t,b,c):
tspan = np.hstack([[0],np.hstack([t])])
return scint.odeint(pend, y0, tspan, args=(b,c))[1:,0]
popt, pcov = scopt.curve_fit(f, t, sol[:,0], p0=guess)
which returns popt = array([ 0.25, 5. ]).
This can be extended to fit even more parameters,
def f(t, a0,a1, b,c):
tspan = np.hstack([[0],np.hstack([t])])
return scint.odeint(pend, [a0,a1], tspan, args=(b,c))[1:,0]
popt, pcov = scopt.curve_fit(f, t, sol[:,0], p0=guess)
which results in popt = [ 3.04159267e+00, -2.38543640e-07, 2.49993362e-01, 4.99998795e+00].
Another possibility is to explicitly compute the square norm of the differences to the target solution and apply minimization to the so-defined scalar function.
def f(param):
b,c = param
t_sol = scint.odeint(pend, y0, t, args=(b,c))
return np.linalg.norm(t_sol[:,0]-sol[:,0]);
res = scopt.minimize(f, np.array(guess))
which returns in res
fun: 1.572327981969186e-08
hess_inv: array([[ 0.00031325, 0.00033478],
[ 0.00033478, 0.00035841]])
jac: array([ 0.06129361, -0.04859557])
message: 'Desired error not necessarily achieved due to precision loss.'
nfev: 518
nit: 27
njev: 127
status: 2
success: False
x: array([ 0.24999905, 4.99999884])
I have a least squares minimization problem subject to inequality constraints which I am trying to solve using scipy.optimize.minimize. It seems that there are two options for inequality constraints: COBYLA and SLSQP.
I first tried SLSQP since it allow for explicit partial derivatives of the function to be minimized. Depending on the scaling of the problem, it fails with error:
Positive directional derivative for linesearch (Exit mode 8)
whenever interval or more general inequality constraints are imposed.
This has been observed previously e.g., here. Manual scaling of the function to be minimized (along with the associated partial derivatives) seems to get rid of the problem, but I cannot achieve the same effect by changing ftol in the options.
Overall, this whole thing is causing me to have doubts about the routine working in a robust manner. Here's a simplified example:
import numpy as np
import scipy.optimize as sp_optimize
def cost(x, A, y):
e = y - A.dot(x)
rss = np.sum(e ** 2)
return rss
def cost_deriv(x, A, y):
e = y - A.dot(x)
deriv0 = -2 * e.dot(A[:,0])
deriv1 = -2 * e.dot(A[:,1])
deriv = np.array([deriv0, deriv1])
return deriv
A = np.ones((10,2)); A[:,0] = np.linspace(-5,5, 10)
x_true = np.array([2, 2/20])
y = A.dot(x_true)
x_guess = x_true / 2
prm_bounds = ((0, 3), (0,1))
cons_SLSQP = ({'type': 'ineq', 'fun' : lambda x: np.array([x[0] - x[1]]),
'jac' : lambda x: np.array([1.0, -1.0])})
# works correctly
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(A, y), jac=cost_deriv, bounds=prm_bounds, method='SLSQP', constraints=cons_SLSQP, options={'disp': True})
print(min_res_SLSQP)
# fails
A = 100 * A
y = A.dot(x_true)
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(A, y), jac=cost_deriv, bounds=prm_bounds, method='SLSQP', constraints=cons_SLSQP, options={'disp': True})
print(min_res_SLSQP)
# works if bounds and inequality constraints removed
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(A, y), jac=cost_deriv,
method='SLSQP', options={'disp': True})
print(min_res_SLSQP)
How should ftol be set to avoid failure? More generally, can a similar problem arise with COBYLA? Is COBYLA a better choice for this type of inequality constrained least squares optimization problem?
Using a square root in the cost function was found to improve performance. However, for a non-linear re-paramterization of the problem (simpler but closer to what I need to do in practice), it fails again. Here are the details:
import numpy as np
import scipy.optimize as sp_optimize
def cost(x, y, g):
e = ((y - x[1]) / x[0]) - g
rss = np.sqrt(np.sum(e ** 2))
return rss
def cost_deriv(x, y, g):
e = ((y- x[1]) / x[0]) - g
factor = 0.5 / np.sqrt(e.dot(e))
deriv0 = -2 * factor * e.dot(y - x[1]) / (x[0]**2)
deriv1 = -2 * factor * np.sum(e) / x[0]
deriv = np.array([deriv0, deriv1])
return deriv
x_true = np.array([1/300, .1])
N = 20
t = 20 * np.arange(N)
g = 100 * np.cos(2 * np.pi * 1e-3 * (t - t[-1] / 2))
y = g * x_true[0] + x_true[1]
x_guess = x_true / 2
prm_bounds = ((1e-4, 1e-2), (0, .4))
# check derivatives
delta = 1e-9
C0 = cost(x_guess, y, g)
C1 = cost(x_guess + np.array([delta, 0]), y, g)
approx_deriv0 = (C1 - C0) / delta
C1 = cost(x_guess + np.array([0, delta]), y, g)
approx_deriv1 = (C1 - C0) / delta
approx_deriv = np.array([approx_deriv0, approx_deriv1])
deriv = cost_deriv(x_guess, y, g)
# fails
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(y, g), jac=cost_deriv,
bounds=prm_bounds, method='SLSQP', options={'disp': True})
print(min_res_SLSQP)
Instead of minimizing np.sum(e ** 2), minimize sqrt(np.sum(e ** 2)), or better (in terms of calculation): np.linalg.norm(e)!
This modification:
does not change your solution in regards to x
will need post-processing if the original objective is needed (probably not)
is much more robust
With this change, all cases work, even using numerical-differentiation (i was too lazy to modify the gradient, which needs to reflect this!).
Example output (number of func-evals gives away num-diff):
Optimization terminated successfully. (Exit mode 0)
Current function value: 3.815547437029837e-06
Iterations: 16
Function evaluations: 88
Gradient evaluations: 16
fun: 3.815547437029837e-06
jac: array([-6.09663382, -2.48862544])
message: 'Optimization terminated successfully.'
nfev: 88
nit: 16
njev: 16
status: 0
success: True
x: array([ 2.00000037, 0.10000018])
Optimization terminated successfully. (Exit mode 0)
Current function value: 0.0002354577991007501
Iterations: 23
Function evaluations: 114
Gradient evaluations: 23
fun: 0.0002354577991007501
jac: array([ 435.97259208, 288.7483819 ])
message: 'Optimization terminated successfully.'
nfev: 114
nit: 23
njev: 23
status: 0
success: True
x: array([ 1.99999977, 0.10000014])
Optimization terminated successfully. (Exit mode 0)
Current function value: 0.0003392807206384532
Iterations: 21
Function evaluations: 112
Gradient evaluations: 21
fun: 0.0003392807206384532
jac: array([ 996.57340243, 51.19298764])
message: 'Optimization terminated successfully.'
nfev: 112
nit: 21
njev: 21
status: 0
success: True
x: array([ 2.00000008, 0.10000104])
While there are probably some problems with SLSQP, it's still one of the most tested and robust codes given that broad application-spectrum!
I would also expect SLSQP to be much better here compared to COBYLA, as the latter is based heavily on linearizations. (but just take it as a guess; it's easy to try given the minimize-interface!)
Alternative
In general, an Interior-point based solver for Convex Quadratic Programming will be the best approach here. But for this, you need to leave scipy. (or maybe an SOCP-solver would be better... i'm not sure).
cvxpy brings a nice-modelling system and a good open-source solver (ECOS; although technically a conic-solver -> more general and less robust; but should beat SLSQP).
Using cvxpy and ECOS, this looks like:
import numpy as np
import cvxpy as cvx
""" Problem data """
A = np.ones((10,2)); A[:,0] = np.linspace(-5,5, 10)
x_true = np.array([2, 2/20])
y = A.dot(x_true)
x_guess = x_true / 2
prm_bounds = ((0, 3), (0,1))
# problematic case
A = 100 * A
y = A.dot(x_true)
""" Solve """
x = cvx.Variable(len(x_true))
constraints = [x[0] >= x[1]]
for ind, (lb, ub) in enumerate(prm_bounds): # ineffecient -> matrix-based expr better!
constraints.append(x[ind] >= lb)
constraints.append(x[ind] <= ub)
objective = cvx.Minimize(cvx.norm(A*x - y))
problem = cvx.Problem(objective, constraints)
problem.solve(solver=cvx.ECOS, verbose=False)
print(problem.status)
print(problem.value)
print(x.value.T)
# optimal
# -6.67593652593801e-10
# [[ 2. 0.1]]