We Keep Coding

Gekko: Problem with obtaining the solution (offline, Solver=1, 2, or 3) or in a reasonable time (online, remote=True)

The simple optimization model below (a support vector machine, see https://www.supplychaindataanalytics.com/creating-a-support-vector-machine-using-gekko-in-python/ for further information) is an NLP with T=86 and U=6 (dataset generated for this minimal working example).
import numpy as np
import gekko as op
import itertools as it
a = np.random.rand(86, 6)
b = np.random.randint(0,6, size=(86))
C = range(len(set(b))) #Set of classes
U = range(len(a[0])) #Set of input features
T = range(len(b)) #Set of the training points
def model (C,U,T,a,b,solve="y"):
save_b = tuple(b)
alpha_c=[None for j in C]
z_c=[None for j in C]
for j in C:
for t in T:
if b[t] == j:
b[t] = +1
else:
b[t] = -1
print(b)
m = op.GEKKO(remote=False, name='SupportVectorMachine')
alpha = {t: m.Var(lb=0, ub=None) for t in T}
n_a = {(t,i): a[t][i] for t,i in it.product(T,U)}
n_b = {t: b[t] for t in T}
objs = {0: m.sum([alpha[t] for t in T]) - 0.5*m.sum([alpha[t]*alpha[tt] * n_b[t]*n_b[tt] * m.sum([n_a[(t,i)]*n_a[(tt,i)] for i in U]) for t,tt in it.product(T,T)])}
cons = {0: {0: ( m.sum([alpha[t]*n_b[t] for t in T]) == 0) for t in T}}
m.Maximize(objs[0])
for keys1 in cons:
for keys2 in cons[keys1]: m.Equation(cons[keys1][keys2])
if solve == "y":
m.options.SOLVER=1
m.solve(disp=False)
for keys in alpha:
alpha[keys] = alpha[keys].value[0]
print(f"alpha[{keys}]", alpha[keys])
x = [None for i in U]
for i in U:
x[i]=sum(alpha[t]*b[t]*n_a[(t,i)] for t in T)
for t in T:
if alpha[t]>0:
z=b[t] - sum(x[i]*n_a[(t,i)] for i in U)
break
b = list(save_b)
alpha_c[j]=alpha
z_c[j]=z
return m,z,alpha
m, z, alpha = model(C,U,T,a,b) #Model and solve the problem
With m.options.SOLVER=1, the code exits with:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmpj66p0g5qsupportvectormachine\\options.json'
With m.options.SOLVER=2, the code exits with:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmpgat29b25supportvectormachine\\options.json'
With m.options.SOLVER=3, the code exits with:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USERNAME\\AppData\\Local\\Temp\\tmpgat29b25supportvectormachine\\options.json'
With m = op.GEKKO(remote=True, name='SupportVectorMachine') the code seems to take too much time to run and no output is reported.
I wondered why such a case occurs and how I could troubleshoot the code? Do we need to feed the solver's algorithm with an initial guess every time? (I am using gekko (1.21.5).)
Thanks in advance.
ٍEdit:
On Google's Colaboratory, it exits with:
Exception: #error: Solution Not Found

Try IMODE=2 (regression mode) where the model is defined only once and automatically applied for each data point. This greatly improves the speed of the model compilation. Below is a minimal example of IMODE=2. There is also a comparison to Scipy Minimize for this same example and a nonlinear regression example.
import numpy as np
from gekko import GEKKO
# load data
xm = np.array([18.3447,79.86538,85.09788,10.5211,44.4556, \
69.567,8.960,86.197,66.857,16.875, \
52.2697,93.917,24.35,5.118,25.126, \
34.037,61.4445,42.704,39.531,29.988])
ym = np.array([5.072,7.1588,7.263,4.255,6.282, \
6.9118,4.044,7.2595,6.898,4.8744, \
6.5179,7.3434,5.4316,3.38,5.464, \
5.90,6.80,6.193,6.070,5.737])
# define model
m = GEKKO()
# one value across all data
a,b,c = m.Array(m.FV,3,value=0)
c.LOWER = -100; c.UPPER = 100
# load data
x = m.Param(value=xm)
ymeas = m.Param(value=ym)
# predicted value
ypred = m.Var()
a.STATUS = 1; b.STATUS = 1; c.STATUS = 1
m.Equation(ypred == a + b/x + c*m.log(x))
m.Minimize(((ypred-ymeas)/ymeas)**2)
m.options.IMODE = 2 # regression mode
m.solve() # remote=False for local solve
print('Final SSE Objective: ' + str(m.options.objfcnval))
There are also a couple things in the code that improve the performance.
The m.sum([n_a[(t,i)]*n_a[(tt,i)] for i in U]) in the objective function can be replaced with a regular summation as sum([n_a[(t,i)]*n_a[(tt,i)] for i in U]). When all of the arguments are floating point numbers, this allows evaluation of the value instead of the Gekko symbolic form of the summation.
Can the cons be defined once (dimension T) instead of the same constraint repeated (dimension T*T)?
cons = {0: m.sum([alpha[t]*n_b[t] for t in T]) == 0}
m.Maximize(objs[0])
for keys1 in cons:
m.Equation(cons[keys1])
Thanks for posting the SVM code. If it can be transitioned into IMODE=2 then the solution efficiency can be greatly improved. If it is not possible (stay with IMODE=3 default) then hopefully the above suggestions help.
We're working on a new module in Gekko to import sklearn (sklearn example), tensorflow, gpflow and other models into Gekko. The potential is to use those specially crafted regression packages and export models to the more general optimization gekko package. We've found that a general gradient-based solver may be doing too much work in the fitting by providing automatic differentiation of constraints and objective with sparse 1st and 2nd derivatives when only 1st derivatives are required. The specifically designed optimizers for each regression method may be more efficient, especially with large data sets that train on GPUs. More documentation and results are expected in the next release of Gekko v1.1+ around Sept 2022.

GEKKO Error: "Equation without an equality (=) or inequality (>,<)" when calling functions within constraint and objective

I have been getting the below error for my code and I am at a complete loss as to the source of the error:
#error: Equation Definition
Equation without an equality (=) or inequality (>,<)
true
STOPPING...
I am seeking to identify the solution 'x' that minimises the result of the function 'was_constraint' subject to meeting the constraint set by 'warf_moodys_constraint'. The functions return a float value and when i just pass the initial starting vector 'x' to each function separately I don't receive any errors originating from those functions. Can anyone please advise where I might be going wrong?
def was_constraint(sol_g, df, orig):
sol = gekko_to_numpy(sol_g)
x1 = orig.loc["Denominator","WAS"]*orig.loc["Current","WAS"]
x2 = (sol*df["All-In Rate"]).sum()/100
y1 = orig.loc["Denominator","WAS"]+sum(sol)
return y1/(x1+x2)
def warf_moodys_constraint(sol_g, df, orig):
sol = gekko_to_numpy(sol_g)
x1 = orig.loc["Denominator","Moodys WARF"]*orig.loc["Current","Moodys WARF"]
x2 = sum(np.where(sol > 0, sol*df["Moody's WARF"], 0))
y1 = orig.loc["Denominator","Moodys WARF"] +sum(np.where(sol > 0, sol, 0))
return 3000 - (x1+x2)/y1
def gekko_to_numpy(sol_g):
res = np.zeros(len(sol_g))
for i in range(len(sol_g)):
res[i] = sol_g[i].value.value
return res
clo_data = pd.read_excel('CLO.xlsx', sheet_name='CLO')
m = GEKKO()
x = [m.Var() for i in range(len(clo_data["Holdings"]))]
for i in range(len(clo_data["Lower Bound"])):
x[i].lower = 0
x[i].upper = 1000000
m.Equation(warf_moodys_constraint(x, clo_data, metrics)>=0)
m.Obj(was_constraint(x, clo_data, metrics))
m.options.IMODE = 3 #steady state optimization
m.solve()

You need to define the equations in terms of Gekko variables. The approach to translate Gekko variables into a Numpy array won't work to define the equations because Gekko doesn't do call-backs into the Python functions.
def gekko_to_numpy(sol_g):
res = np.zeros(len(sol_g))
for i in range(len(sol_g)):
res[i] = sol_g[i].value.value
return res
Gekko builds the gk_model0.apm model in the run folder that you can see with m.open_folder(). When you solve with m.solve() Gekko compiles the model into byte-code and solves it with sparse nonlinear solvers such as IPOPT or APOPT. If you can't use Gekko variables then maybe the scipy.opitimize.minimize() function would be a better choice. Here is a tutorial with that optimizer.

Does SciPy's fsolve support different solvers algorithms?

I am using fsolve function in SciPy to solve certain non linear equation system, and I have noticed that comparing the results with MATLAB's fsolve for the exact input and initial condition I am getting different outcomes. Therefore, I was wondering how I could try to achieve same results with SciPy's fsolve as those in MATLAB.
Currently I have configured fsolve in Matlab like this
options = optimoptions('fsolve','Display','iter-detailed','PlotFcn',#optimplotfirstorderopt);
options.StepTolerance = 1e-14;
%options.OptimalityTolerance = 1e-14
options.FunctionTolerance = 1e-14;
options.MaxIterations = 100000;
options.MaxFunctionEvaluations = 400;
% options.Diagnostics = 'on'
options.Algorithm = 'levenberg-marquardt';%'trust-region'%
fun= #solveTransmissionLineForGAndC;
Whereas I have configure fsolve in SciPy like this
fsolve(self.CandGTransmissionLineDistributedE,[g0,c0],Yin,xtol=1e-14,maxfev= 100000,epsfcn=1e-14)
So, as an example, the output I get from both, MATLAB and SciPy, for same four inputs is
SciPy: [0.0263397261951065,0.0255996001756125,0.0255996001756125,0.0248335970729323]
MATLAB:
[0.036897213114754,0.036897213114754,0.036897213114754,0.036897213114754,0.036897213114754]
How can I select the solver algorithm (or how) in SciPy's fsolve?
EDIT: adding MWE
def WandG(GW,Zin):
G,W = GW # define real variables to be used for the function
R = (G**2)*(Zin.real - 0.1164) +(( 70e-12 * ( W/100 ) )**2)*( 2.9036155068866304e+16*(Zin.real - 0.1164) ) - G
I = (G**2)*( Zin.imag - 18.743998408378143 * (1-W/200) ) + (((W/100)*70e-12)**2)*( 2.9036155068866304e+16*(Zin.imag - 18.743998408378143 * (1-W/200)) ) + 170399985.53071037*(70e-12*(W/100) )
return R,I
sol = optimize.root(WandG, [0.136879496, 47.04],(12.652884410928804+14.632724423763523j), jac=False, method='lm',tol=1e-14)
The solution using Scipy would be
sol.x[0] = 0.0795283512113496 # G
sol.x[1] = 36.53727146377749 # W
Whereas with MATLAB for the same initial conditions, solver and Zin, it returns, which I trust more since it resembles better to the final result of W which is W=50
G = 0.0794
W = 44.5234

MLE for a Polya Distribution

I'm working on programming a MLE for the Polya distribution using scipy. The Nelder-Mead method is working, however I get a "Desired error not necessarily achieved due to precision loss." error when running BFGS. The Nelder-Mead method seems like it's too slow for my needs (I have a lot of fairly big data, say 1000 tables in some cases as big as 10x10000). I've tried using the check_grad function and the result is smallish on the example below (order 10^-2), so I'm not sure if that means there's a bug in the gradient of the log likelihood or the likelihood is just very strongly peaked. For what it's worth, I've stared quite hard at my code and I can't see the issue. Here's some example code to recreate the problem
#setup some data
from numpy.random import dirichlet, multinomial
from scipy.optimize import check_grad
alpha = [10,30,50]
p = pd.DataFrame(dirichlet(alpha,200))
data = p.apply(lambda x: multinomial(500,x),1)
a = np.array(data.mean(0))
#optimize
result = minimize(lambda a: -1*llike(data,exp(a)),
x0=np.log(a),
method='Nelder-Mead')
x0=result.x
result = minimize(lambda a: -1*llike(data,exp(a)),
x0=x0,
jac=lambda a: -1*gradient_llike(data,np.exp(a)),
method='BFGS')
exp(result.x) #should be close to alpha
#uhoh, let's check that this is right.
check_grad(func=lambda a: -1*llike(data,a),grad=lambda a: -1*gradient_llike(data,a),x0=alpha)
Here's the code for my functions
def log_polya(Z,alpha):
"""
Z is a vector of counts
https://en.wikipedia.org/wiki/Dirichlet-multinomial_distribution
http://mimno.infosci.cornell.edu/info6150/exercises/polya.pdf
"""
if not isinstance(alpha,np.ndarray):
alpha = np.array(alpha)
if not isinstance(Z,np.ndarray):
Z = np.array(Z)
#Concentration Parameter
A = sum(alpha)
#Number of Datapoints
N = sum(Z)
return gammaln(A) - gammaln(N+A) + sum(gammaln(Z+alpha) - gammaln(alpha))
def llike(data,alpha):
return sum(data.apply(log_polya,1,alpha=alpha))
def log_polya_derivative(Z,alpha):
if not isinstance(alpha,np.ndarray):
alpha = np.array(alpha)
if not isinstance(Z,np.ndarray):
Z = np.array(Z)
if 0. in Z+alpha:
Warning("invalid prior parameter,nans should be produced")
#Concentration Parameter
A = sum(alpha)
#Number of Datapoints
N = sum(Z)
K = len(Z)
return np.array([psi(A) - psi(N+A) + psi(Z[i]+alpha[i]) - psi(alpha[i]) for i in xrange(K)])
def gradient_llike(data,alpha):
return np.array(data.apply(log_polya_derivative,1,alpha=alpha).sum(0))
UPDATE: Still curious about this, but for those interested in a working implementation for this problem, the following code for implementing the Minka Fixed Point Algorithm seems to work well (i.e. recovers quickly values that are close to the true dirichlet parameter).
def minka_mle_polya(data):
"""
http://research.microsoft.com/en-us/um/people/minka/papers/dirichlet/minka-dirichlet.pdf
"""
data = np.array(data)
K = np.shape(data)[1]
alpha = np.array(data.mean(0))
alpha_new = np.ndarray((K))
precision = 10
while precision > 10**-5:
for k in range(K):
A = sum(alpha)
N = data.sum(1)
numerator = sum(
psi(data[:,k]+alpha[k])-psi(alpha[k])
)
denominator = sum(
psi(N+A)-psi(A)
)
alpha_new[k] = alpha[k]*numerator/denominator
precision = sum(abs(alpha_new - alpha))
alpha_old = np.array(alpha)
alpha = np.array(alpha_new)
print "Gap", precision

ValueError in CVXPY minization function (Minimum Variance Optimization)

I'm a bit of a beginner and in the process of moving an algorithm that works with minimum variance optimization from scipy.minimize.optimize (which didn't perform properly) to CVXPY.
R are the expected returns, C the coveriances and rf the risk-free rate. w are the optimal weights and r various means along the Efficient Frontier for which the weights are calculated.
When I run the code below I get:
ValueError: setting an array element with a sequence.
I believe var is at fault here, but I don't know how else to structure it. Insight much appreciated. On top of that, the rest of the code could have additional errors so if you spot any please do point them out!
def solve_frontier(R, C, rf, context):
frontier_mean, frontier_var, frontier_weights = [], [], []
n = len(R)
w = cvx.Variable(n)
r = cvx.Parameter(sign='positive')
mean_1 = sum(R*w)
var = dot(dot(w, C), w)
penalty = (1/100)*abs(mean_1-r)
prob = cvx.Problem(cvx.Minimize(var + penalty),
[sum(w)-context.allowableMargin == 0])
r_vals = linspace(max(min(R), rf), max(R), num=20)
for i in range(20):
r.value = r_vals[i]
prob.solve()
frontier_mean.append(r)
frontier_var.append(compute_var(prob.value, C))
frontier_weights.append(prob.value)
print "status:", prob.status
return array(frontier_mean), array(frontier_var), frontier_weights

The problem was in frontier_mean.append(r), which should have been frontier_mean.append(r.value).