We Keep Coding

Fit an equation to data by minimizing mean squared error by minimizing sum of the squared error

I am trying the estimate fitting parameters that can minimize sum of the squared error between the predicted and test values.
To do this, I am using scipy.optimize.minimize
Below is my code:
import numpy as np
import pandas as pd
import math
import scipy.optimize as spo
# Initial guess of the parameters
LOGGMIN = 120 # variable 1
LOGGMAX = 104 # variable 2
BETA = 0 # variable 3
GAMMA = 50 #variable 4
C1E_O = -504 #variable 5
C2E_O = 2012 #variable 6
ReferenceTemperatureE_O = 21
x0 = [LOGGMIN, LOGGMAX, BETA, GAMMA, C1E_O, C2E_O] # Initial guess as a list
# Importing the data from Excel file
df = pd.read_excel(r'C:\Users\Mahmoud Khadijeh\Desktop\MasterCurve.xlsx', header=None)
K = ((np.dot(-C1E_O,df[0]) + C1E_O*ReferenceTemperatureE_O)/(C2E_O+(df[0]-ReferenceTemperatureE_O)))
df['log f'] = np.log10(df[1])
df['Test LogG'] = np.log10(df[2])
df['WLF Shift Factor Log alphaT'] = K
df['WLF Reduced Frequency log(fr)'] =df['log f'] + df['WLF Shift Factor Log alphaT']
df['Predicted logG*'] = LOGGMIN + ((LOGGMAX - LOGGMIN)/(1+np.exp(BETA + GAMMA *(df['log f']+df['WLF Shift Factor Log alphaT']))))
df['Predicted Error'] = ((df['Test LogG']-df['Predicted logG*'])/(df['Test LogG']))**2
# Defining an objective function (Sum of the square error),, The function that we have to minimize
def objective(SE):
global Sum_Error
LOGGMIN = SE[0] #Variable 1 that we have to optimize
LOGGMAX = SE[1] #Variable 2 that we have to optimize
BETA = SE[2] #Variable 3 that we have to optimize
GAMMA = SE[3] #Variable 4 that we have to optimize
C1E_O = SE[4] #Variable 5 that we have to optimize
C2E_O = SE[5] #Variable 6 that we have to optimize
Sum_Error = df['Predicted Error'].sum()
return Sum_Error
sol = spo.minimize(objective, x0, method='SLSQP', options={'disp': True})
print(df)
print('sum= ', Sum_Error)
print ('Solver', sol)
I want to optimize the variables [LOGGMIN, LOGGMAX, BETA, GAMMA, C1E_O, C2E_O] so I can obtain the minimue value of the function (some of the squared error)
However, the output that I got is :
[60 rows x 9 columns]
sum= 16837173.60030545
Solver fun: 16837173.60030545
jac: array([0., 0., 0., 0., 0., 0.])
message: 'Optimization terminated successfully'
nfev: 7
nit: 1
njev: 1
status: 0
success: True
x: array([ 120., 104., 0., 50., -504., 2012.])
As you can see, that the output of the parameters is similiar to the initial guess and the value of the function is very big. It supposed to be close to 0.
Any Help!
Thanks in advance

You are not recalculating your objective actually.
The code should be something like this.
import numpy as np
import pandas as pd
import math
import scipy.optimize as spo
# Initial guess of the parameters
LOGGMIN = 120 # variable 1
LOGGMAX = 104 # variable 2
BETA = 0 # variable 3
GAMMA = 50 # variable 4
C1E_O = -504 # variable 5
C2E_O = 2012 # variable 6
ReferenceTemperatureE_O = 21
x0 = np.array([LOGGMIN, LOGGMAX, BETA, GAMMA, C1E_O, C2E_O]) # Initial guess as a list
# Importing the data from Excel file
df = pd.read_excel(r'C:\Users\Mahmoud Khadijeh\Desktop\MasterCurve.xlsx', header=None)
# Defining an objective function (Sum of the square error),, The function that we have to minimize
def objective(SE):
LOGGMIN = SE[0] # Variable 1 that we have to optimize
LOGGMAX = SE[1] # Variable 2 that we have to optimize
BETA = SE[2] # Variable 3 that we have to optimize
GAMMA = SE[3] # Variable 4 that we have to optimize
C1E_O = SE[4] # Variable 5 that we have to optimize
C2E_O = SE[5] # Variable 6 that we have to optimize
K = ((np.dot(-C1E_O, df[0]) + C1E_O * ReferenceTemperatureE_O) / (C2E_O + (df[0] - ReferenceTemperatureE_O)))
new_df = pd.DataFrame()
new_df['log f'] = np.log10(df[1])
new_df['Test LogG'] = np.log10(df[2])
new_df['WLF Shift Factor Log alphaT'] = K
new_df['WLF Reduced Frequency log(fr)'] = new_df['log f'] + new_df['WLF Shift Factor Log alphaT']
new_df['Predicted logG*'] = LOGGMIN + (
(LOGGMAX - LOGGMIN) / (1 + np.exp(BETA + GAMMA * (new_df['log f'] + new_df['WLF Shift Factor Log alphaT']))))
new_df['Predicted Error'] = ((new_df['Test LogG'] - new_df['Predicted logG*']) / (new_df['Test LogG'])) ** 2
Sum_Error = new_df['Predicted Error'].sum()
return Sum_Error
sol = spo.minimize(objective, x0, method='SLSQP', options={'disp': True})
print(df)
print('Solver', sol)

Scipy.optimize minimize 'inequality constraints incompatible'

I am trying to minimize variance across a portfolio of 100 securities.
def portvol(w, x):
return np.dot(w.T, np.dot(x, w))*252
covmat = annreturn.cov()
w0 = np.ones(len(covmat)) * (1 / len(covmat)) #equal weighting initially
bounds = ((0,1),) * len(covmat)
constraints = {'fun': lambda i: np.sum(i)-1.0, 'type': 'eq'}
optweights = minimize(portvol, w0, args = (covmat), method = 'SLSQP', bounds = bounds, constraints =
constraints)
annreturn.cov() is a 100x100 DataFrame. The output is the same .01 even weightings I started with and this failure message:
message: 'Inequality constraints incompatible'
nfev: 102
nit: 1
njev: 1
status: 4
success: False

This is how I calculated annualized returns...
annreturn = data.pct_change() #again, assuming percentage change
annreturn = annreturn.iloc[1:]
annreturn = (annreturn+1)**252-1
If you don't notice anything off the bat, it's ok. It took me 2 days to realize I didn't divide my PCT_CHANGE() result by 100. Time well spent. I was getting correlations to the powers of like 15+. Here is what the last line should have looked like, and the minimize function from the original question works fine.
annreturn = (annreturn/100+1)**252-1
Sorry if anyone took time on this without the above piece!

Is it normal in scipy.optimise?

I want to optimise my portfolio using Markowitz theory (risk minimization by the Markowitz method for a given income = 15%) and Scipy.minimize
I have risk function
def objective(x):
x1=x[0];x2=x[1];x3=x[2]; x4=x[3]
return 1547.87020*x1**2 + 125.26258*x1*x2 + 1194.3433*x1*x3 + 63.6533*x1*x4 \
+ 27.3176649*x2**2 + 163.28848*x2*x3 + 4.829816*x2*x4 \
+ 392.11819*x3**2 + 56.50518*x3*x4 \
+ 34.484063*x4**2
Sum of parts of stocks(in %) = 1
def constraint1(x):
return (x[0]+x[1]+x[2]+x[3]-1.0)
Income function with restriction
def constraint2(x):
return (-1.37458*x[0] + 0.92042*x[1] + 5.06189*x[2] + 0.35974*x[3] - 15.0)
And I test it using:
x0=[0,1,1,0] #Initial value
b=(0.0,1.0)
bnds=(b,b,b,b)
con1={'type':'ineq','fun':constraint1}
con2={'type':'eq','fun':constraint2}
cons=[con1,con2]
sol=minimize(objective,x0,method='SLSQP',\
bounds=bnds,constraints=cons)
And my result is:
fun: 678.5433939
jac: array([1383.25920868, 222.75363159, 1004.03005219, 130.30312347])
message: 'Positive directional derivative for linesearch'
nfev: 216
nit: 20
njev: 16
status: 8
success: False
x: array([0., 1., 1., 1.])
But how? Sum of parts of portfolio cant be more than 1(now parts of stock 2=stock3=stock4=100%). Its constraint1. Where is problem?

The output says "success: False"
So it is telling you that it failed to find a solution to the problem.
Also, why did you put
con1={'type':'ineq','fun':constraint1}
Don't you want
con1={'type':'eq','fun':constraint1}
I got success using method='BFGS'

Your code is returning values that do not respect your constraint due to false definition of the first constraint (a-b >= 0 => a>b) so in your case a=1(the order in an inequality is important). On the other hand your x0 must also respect your constraints and sum([0,1,1,0]) = 2 > 1.
I slightly improved your code and fixed the aforementioned issues, but I still think that you need to review your second constraint:
import numpy as np
from scipy.optimize import minimize
def objective(x):
x1, x2, x3, x4 = x[0], x[1], x[2], x[3]
coefficients = np.array([1547.87020, 125.26258, 1194.3433, 63.6533, 27.3176649, 163.28848, 4.829816, 392.11819, 56.50518, 34.484063])
xs = np.array([ x1**2, x1*x2, x1*x3, x1*x4, x2**2, x2*x3, x2*x4, x3**2, x3*x4, x4**2])
return np.dot(xs, coefficients)
const1 = lambda x: 1 - sum(x)
const2 = lambda x: np.dot(np.array([-1.37458, 0.92042, 5.06189, 0.35974]), x) - 15.0
x0 = [0, 0, 0, 0] #Initial value
b = (0.0, 1.0)
bnds = (b, b, b, b)
cons = [{'type':'ineq','fun':const1}, {'type':'eq', 'fun':const2}]
# minimize
sol = minimize(objective,
x0,
method = 'SLSQP',
bounds = bnds,
constraints = cons)
print(sol)
output:
fun: 392.1181900000138
jac: array([1194.34332275, 163.28847885, 784.23638535, 56.50518036])
message: 'Positive directional derivative for linesearch'
nfev: 92
nit: 11
njev: 7
status: 8
success: False
x: array([0.00000000e+00, 5.56638069e-14, 1.00000000e+00, 8.29371293e-14])

fmin_slsqp returns initial guess finding the minimum of cubic spline

I am trying to find the minimum of a natural cubic spline. I have written the following code to find the natural cubic spline. (I have been given test data and have confirmed this method is correct.) Now I can not figure out how to find the minimum of this function.
This is the data
xdata = np.linspace(0.25, 2, 8)
ydata = 10**(-12) * np.array([1,2,1,2,3,1,1,2])
This is the function
import scipy as sp
import numpy as np
import math
from numpy.linalg import inv
from scipy.optimize import fmin_slsqp
from scipy.optimize import minimize, rosen, rosen_der
def phi(x, xd,yd):
n = len(xd)
h = np.array(xd[1:n] - xd[0:n-1])
f = np.divide(yd[1:n] - yd[0:(n-1)],h)
q = [0]*(n-2)
for i in range(n-2):
q[i] = 3*(f[i+1] - f[i])
A = np.zeros(((n-2),(n-2)))
#define A for j=0
A[0,0] = 2*(h[0] + h[1])
A[0,1] = h[1]
#define A for j = n-2
A[-1,-2] = h[-2]
A[-1,-1] = 2*(h[-2] + h[-1])
#define A for in the middle
for j in range(1,(n-3)):
A[j,j-1] = h[j]
A[j,j] = 2*(h[j] + h[j+1])
A[j,j+1] = h[j+1]
Ainv = inv(A)
B = Ainv.dot(q)
b = (n)*[0]
b[1:(n-1)] = B
# now we find a, b, c and d
a = [0]*(n-1)
c = [0]*(n-1)
d = [0]*(n-1)
s = [0]*(n-1)
for r in range(n-1):
a[r] = 1/(3*h[r]) * (b[r + 1] - b[r])
c[r] = f[r] - h[r]*((2*b[r] + b[r+1])/3)
d[r] = yd[r]
#solution 1 start
for m in range(n-1):
if xd[m] <= x <= xd[m+1]:
s = a[m]*(x - xd[m])**3 + b[m]*(x-xd[m])**2 + c[m]*(x-xd[m]) + d[m]
return(s)
#solution 1 end
I want to find the minimum on the domain of my xdata, so a fmin didn't work as you can not define bounds there. I tried both fmin_slsqp and minimize. They are not compatible with the phi function I wrote so I rewrote phi(x, xd,yd) and added an extra variable such that phi is phi(x, xd,yd, m). M indicates in which subfunction of the spline we are calculating a solution (from x_m to x_m+1). In the code we replaced #solution 1 by the following
# solution 2 start
return(a[m]*(x - xd[m])**3 + b[m]*(x-xd[m])**2 + c[m]*(x-xd[m]) + d[m])
# solution 2 end
To find the minimum in a domain x_m to x_(m+1) we use the following code: (we use an instance where m=0, so x from 0.25 to 0.5. The initial guess is 0.3)
fmin_slsqp(phi, x0 = 0.3, bounds=([(0.25,0.5)]), args=(xdata, ydata, 0))
What I would then do (I know it's crude), is iterate this with a for loop to find the minimum on all subdomains and then take the overall minimum. However, the function fmin_slsqp constantly returns the initial guess as the minimum. So there is something wrong, which I do not know how to fix. If you could help me this would be greatly appreciated. Thanks for reading this far.

When I plot your function phi and the data you feed in, I see that its range is of the order of 1e-12. However, fmin_slsqp is unable to handle that level of precision and fails to find any change in your objective.
The solution I propose is scaling the return of your objective by the same order of precision like so:
return(s*1e12)
Then you get good results.
>>> sol = fmin_slsqp(phi, x0=0.3, bounds=([(0.25, 0.5)]), args=(xdata, ydata))
>>> print(sol)
Optimization terminated successfully. (Exit mode 0)
Current function value: 1.0
Iterations: 2
Function evaluations: 6
Gradient evaluations: 2
[ 0.25]

Python's scipy.optimize.minimize with SLSQP fails with "Positive directional derivative for linesearch"

I have a least squares minimization problem subject to inequality constraints which I am trying to solve using scipy.optimize.minimize. It seems that there are two options for inequality constraints: COBYLA and SLSQP.
I first tried SLSQP since it allow for explicit partial derivatives of the function to be minimized. Depending on the scaling of the problem, it fails with error:
Positive directional derivative for linesearch (Exit mode 8)
whenever interval or more general inequality constraints are imposed.
This has been observed previously e.g., here. Manual scaling of the function to be minimized (along with the associated partial derivatives) seems to get rid of the problem, but I cannot achieve the same effect by changing ftol in the options.
Overall, this whole thing is causing me to have doubts about the routine working in a robust manner. Here's a simplified example:
import numpy as np
import scipy.optimize as sp_optimize
def cost(x, A, y):
e = y - A.dot(x)
rss = np.sum(e ** 2)
return rss
def cost_deriv(x, A, y):
e = y - A.dot(x)
deriv0 = -2 * e.dot(A[:,0])
deriv1 = -2 * e.dot(A[:,1])
deriv = np.array([deriv0, deriv1])
return deriv
A = np.ones((10,2)); A[:,0] = np.linspace(-5,5, 10)
x_true = np.array([2, 2/20])
y = A.dot(x_true)
x_guess = x_true / 2
prm_bounds = ((0, 3), (0,1))
cons_SLSQP = ({'type': 'ineq', 'fun' : lambda x: np.array([x[0] - x[1]]),
'jac' : lambda x: np.array([1.0, -1.0])})
# works correctly
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(A, y), jac=cost_deriv, bounds=prm_bounds, method='SLSQP', constraints=cons_SLSQP, options={'disp': True})
print(min_res_SLSQP)
# fails
A = 100 * A
y = A.dot(x_true)
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(A, y), jac=cost_deriv, bounds=prm_bounds, method='SLSQP', constraints=cons_SLSQP, options={'disp': True})
print(min_res_SLSQP)
# works if bounds and inequality constraints removed
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(A, y), jac=cost_deriv,
method='SLSQP', options={'disp': True})
print(min_res_SLSQP)
How should ftol be set to avoid failure? More generally, can a similar problem arise with COBYLA? Is COBYLA a better choice for this type of inequality constrained least squares optimization problem?
Using a square root in the cost function was found to improve performance. However, for a non-linear re-paramterization of the problem (simpler but closer to what I need to do in practice), it fails again. Here are the details:
import numpy as np
import scipy.optimize as sp_optimize
def cost(x, y, g):
e = ((y - x[1]) / x[0]) - g
rss = np.sqrt(np.sum(e ** 2))
return rss
def cost_deriv(x, y, g):
e = ((y- x[1]) / x[0]) - g
factor = 0.5 / np.sqrt(e.dot(e))
deriv0 = -2 * factor * e.dot(y - x[1]) / (x[0]**2)
deriv1 = -2 * factor * np.sum(e) / x[0]
deriv = np.array([deriv0, deriv1])
return deriv
x_true = np.array([1/300, .1])
N = 20
t = 20 * np.arange(N)
g = 100 * np.cos(2 * np.pi * 1e-3 * (t - t[-1] / 2))
y = g * x_true[0] + x_true[1]
x_guess = x_true / 2
prm_bounds = ((1e-4, 1e-2), (0, .4))
# check derivatives
delta = 1e-9
C0 = cost(x_guess, y, g)
C1 = cost(x_guess + np.array([delta, 0]), y, g)
approx_deriv0 = (C1 - C0) / delta
C1 = cost(x_guess + np.array([0, delta]), y, g)
approx_deriv1 = (C1 - C0) / delta
approx_deriv = np.array([approx_deriv0, approx_deriv1])
deriv = cost_deriv(x_guess, y, g)
# fails
min_res_SLSQP = sp_optimize.minimize(cost, x_guess, args=(y, g), jac=cost_deriv,
bounds=prm_bounds, method='SLSQP', options={'disp': True})
print(min_res_SLSQP)

Instead of minimizing np.sum(e ** 2), minimize sqrt(np.sum(e ** 2)), or better (in terms of calculation): np.linalg.norm(e)!
This modification:
does not change your solution in regards to x
will need post-processing if the original objective is needed (probably not)
is much more robust
With this change, all cases work, even using numerical-differentiation (i was too lazy to modify the gradient, which needs to reflect this!).
Example output (number of func-evals gives away num-diff):
Optimization terminated successfully. (Exit mode 0)
Current function value: 3.815547437029837e-06
Iterations: 16
Function evaluations: 88
Gradient evaluations: 16
fun: 3.815547437029837e-06
jac: array([-6.09663382, -2.48862544])
message: 'Optimization terminated successfully.'
nfev: 88
nit: 16
njev: 16
status: 0
success: True
x: array([ 2.00000037, 0.10000018])
Optimization terminated successfully. (Exit mode 0)
Current function value: 0.0002354577991007501
Iterations: 23
Function evaluations: 114
Gradient evaluations: 23
fun: 0.0002354577991007501
jac: array([ 435.97259208, 288.7483819 ])
message: 'Optimization terminated successfully.'
nfev: 114
nit: 23
njev: 23
status: 0
success: True
x: array([ 1.99999977, 0.10000014])
Optimization terminated successfully. (Exit mode 0)
Current function value: 0.0003392807206384532
Iterations: 21
Function evaluations: 112
Gradient evaluations: 21
fun: 0.0003392807206384532
jac: array([ 996.57340243, 51.19298764])
message: 'Optimization terminated successfully.'
nfev: 112
nit: 21
njev: 21
status: 0
success: True
x: array([ 2.00000008, 0.10000104])
While there are probably some problems with SLSQP, it's still one of the most tested and robust codes given that broad application-spectrum!
I would also expect SLSQP to be much better here compared to COBYLA, as the latter is based heavily on linearizations. (but just take it as a guess; it's easy to try given the minimize-interface!)
Alternative
In general, an Interior-point based solver for Convex Quadratic Programming will be the best approach here. But for this, you need to leave scipy. (or maybe an SOCP-solver would be better... i'm not sure).
cvxpy brings a nice-modelling system and a good open-source solver (ECOS; although technically a conic-solver -> more general and less robust; but should beat SLSQP).
Using cvxpy and ECOS, this looks like:
import numpy as np
import cvxpy as cvx
""" Problem data """
A = np.ones((10,2)); A[:,0] = np.linspace(-5,5, 10)
x_true = np.array([2, 2/20])
y = A.dot(x_true)
x_guess = x_true / 2
prm_bounds = ((0, 3), (0,1))
# problematic case
A = 100 * A
y = A.dot(x_true)
""" Solve """
x = cvx.Variable(len(x_true))
constraints = [x[0] >= x[1]]
for ind, (lb, ub) in enumerate(prm_bounds): # ineffecient -> matrix-based expr better!
constraints.append(x[ind] >= lb)
constraints.append(x[ind] <= ub)
objective = cvx.Minimize(cvx.norm(A*x - y))
problem = cvx.Problem(objective, constraints)
problem.solve(solver=cvx.ECOS, verbose=False)
print(problem.status)
print(problem.value)
print(x.value.T)
# optimal
# -6.67593652593801e-10
# [[ 2. 0.1]]