I want to compare Gurobi and Scipy's linear programming tools, such as linprog. Scipy requires to specify problems in a matrix-list-vector-form while Gurobi works like here such that
m = Model()
m.addVar(...) %for variables
m.addConstr(..>) %for constraints
m.update() %for updating the model
m.optimize % for optimizing the model
m.params %for getting parameters
m._vars %for getting variables
in comparison Scipy
Minimize: c^T * x
Subject to: A_ub * x <= b_ub
A_eq * x == b_eq
c : array_like
Coefficients of the linear objective function to be minimized.
A_ub : array_like, optional
2-D array which, when matrix-multiplied by x, gives the values of the upper-bound inequality constraints at x.
b_ub : array_like, optional
1-D array of values representing the upper-bound of each inequality constraint (row) in A_ub.
A_eq : array_like, optional
2-D array which, when matrix-multiplied by x, gives the values of the equality constraints at x.
b_eq : array_like, optional
1-D array of values representing the RHS of each equality constraint (row) in A_eq.
bounds : sequence, optional
My goal is to write the code in only one method and still benchmark the results with both solvers. In order to speed up comparing the solvers:
Does there exist Gurobi-style model construction of LP problems for Scipy?
Does there exist some package to make the two methods interchangeable (I could write scipy-style for Gurobi or in Gurobi-style for Scipy)?
Does scipy provide any other interface to specify linear programming problems?
That sounds like a lot of work to show the obvious:
Commercial solvers like Gurobi are much faster and much more robust than non-commercial ones
There are also high-quality benchmarks showing this by H. Mittelmann (CLP and GLPK beeing the most popular non-commercial ones)
While scipy's linprog is ok, it's much worse than the open-source competition including CBC/CLP, GLPK, lpSolve...
Speed and robustness!
Also: scipy's linprog really seems unmaintained open issues
There are some ways you could do that:
A) Use linprog's way of problem-definition and convert this to Gurobi-style
very easy to convert-matrix form to Gurobi's modelling
B) Use cvxpy as modelling-tool, grab the standard-form and write a wrapper for Gurobi (actually: there is one) and linprog (again easy). This would allow a very powerful modelling-language to be used by both
Disadvantage: Intransparent transformations according to the problem (e.g. abs(some_vector) might introduce auxiliary variables)
C) Write some MPS-reader / or take one from other tools to model you problems within Gurobi, output these and read & solve within linprog
Candidate tools: cvxopt's mps-reader (well-hidden in the docs), some GLPK-interface or even some CBC-interface
(Maybe hidden transformations)
No matter what you do, solution-process analysis will be a big part of your code as linprog might fail a lot. It's also not able to handle big sparse models.
Remarks based on your gurobi-example
Your example (TSP) is a MIP, not an LP
For MIP, everything said above get's worse (especially performance differences between commerical and open-source)
There is no MIP-solver within scipy!
Related
I am trying to solve the set of linear equations:
min || Ax - B ||^2
for t in [0,1]
such that the coefficients x in this equation satisfy the linear equation:
C x = D
This system attempts to fit a set of Polynomials to approximate a function F(t) over the range of t.
A is a matrix, representing the map of the set of polynomials over the range of t values
x is a vector of coefficients (what I want) corresponding to a weight applied to each polynomial in A
B is a vector representing the F(t) values,
C is a matrix and D a vector, which together represent the boundary conditions on the coefficients of this system
This is a case of solving linear equations using the constraint of ordinary least squares.
While there are known closed form solutions e.g. Karush-Kuhn-Tucker I'm looking for a routing in scipy / numpy that can be used to solve this.
Research has shown the scipy.optimize module, which includes functions such as:
scipy.optimize.least_squares .
scipy.optimize.nnls .
scipy.optimize.lsq_linear .
The above is suggested both from this question and this question.
But these do not have conditions that work for a constraint of some other linear equation. What can I use in scipy and numpy to do this?
For this you may use scipy.minimize(method='SLSQP'). The documentation is here.
You can define the equality constraint as a callable function with the signature:
def cons1(x):
return sum(D - C*x)
SLSQP essential uses the constraint then to drive your optimisation problem. Note that this is a gradient based decent, so you will most likely find a local minima to high dimensional problems.
Another option is scipy.minimize(method=’trust-constr’), the documentation is here. These methods are natively implemented in python so the source code and modifications are accessible through.
If you have a smooth monotonic or context function, in my experience SLSQP should suffice.
I have a non-linear minimization problem that takes a combination of continuous and binary variables as input. Think of it as a network flow problem with valves, for which the throughput can be controlled, and with pumps, for which you can change the direction.
A "natural," minimalistic formulation could be:
arg( min( f(x1,y2,y3) )) s.t.
x1 \in [0,1] //a continuous variable
y2,y3 \in {0,1} //two binary variables
The objective function is deterministic, but expensive to solve. If I leave away the binary variables, Scipy's differential evolution algorithm turns out to be a useful solution approach for my problem (converging faster than basin hopping).
There is some evidence available already with regard to the inclusion of integer variables in a differential evolution-based minimization problem. The suggested approaches turn y2,y3 into continuous variables x2,x3 \in [0,1], and then modify the objective function as follows:
(i) f(x1, round(x2), round(x3))
(ii) f(x1,x2,x3) + K( (x2-round(x2))^2 + (x3-round(x3))^2 )
with K a tuning parameter
A third, and probably naive approach would be to combine the binary variables into a single continuous variable z \in [0,1], and thereby to reduce the number of optimization variables.
For instance,
if z<0.25: y2=y3=0
elif z<0.5: y2=1, y3=0
elif z<0.75: y2=0, y3=1
else: y2=y3=1.
Which one of the above should be preferred, and why? I'd be very curious to hear how binary variables can be integrated in a continuous differential evolution algorithm (such as Scipy's) in a smart way.
PS. I'm aware that there's some literature available that proposes dedicated mixed-integer evolutionary algorithms. For now, I'd like to stay with Scipy.
I'd be very curious to hear how binary variables can be integrated in a continuous differential evolution algorithm
wrapdisc is a package that is a thin wrapper which will let you optimize binary variables alongside floats with various scipy.optimize optimizers. There is a usage example in its readme. With it, you don't have to adapt your objective function at all.
As of v2.0.0, it has two possible encodings for binary:
ChoiceVar: This uses one-hot max encoding. Two floats are used to represent the binary variable.
GridVar: This uses rounding. One float is used to represent the binary variable.
Although neither of these two variable types were made for binary, they both can support it just the same. On average, GridVar requires fewer function evaluations because it uses one less float than ChoiceVar.
When scipy 1.9 is released the differential_evolution function will gain an integrality parameter that will allow the user to indicate which parameters should be considered as integers. For binary selection one would use bounds of (0,1) for an integer parameter.
I am preconditioning a matrix using spilu, however, to pass this preconditioner into cg (the built in conjugate gradient method) it is necessary to use the LinearOperator function, can someone explain to me the parameter matvec, and why I need to use it. Below is my current code
Ainv=scla.spilu(A,drop_tol= 1e-7)
Ainv=scla.LinearOperator(Ainv.shape,matvec=Ainv)
scla.cg(A,b,maxiter=maxIterations, M = Ainv)
However this doesnt work and I am given the error TypeError: 'SuperLU' object is not callable. I have played around and tried
Ainv=scla.LinearOperator(Ainv.shape,matvec=Ainv.solve)
instead. This seems to work but I want to know why matvec needs Ainv.solve rather than just Ainv, and is it the right thing to feed LinearOperator?
Thanks for your time
Without having much experience with this part of scipy, some comments:
According to the docs you don't have to use LinearOperator, but you might do
M : {sparse matrix, dense matrix, LinearOperator}, so you can use explicit matrices too!
The idea/advantage of the LinearOperator:
Many iterative methods (e.g. cg, gmres) do not need to know the individual entries of a matrix to solve a linear system A*x=b. Such solvers only require the computation of matrix vector products docs
Depending on the task, sometimes even matrix-free approaches are available which can be much more efficient
The working approach you presented is indeed the correct one (some other source doing it similarily, and some course-materials doing it like that)
The idea of not using the inverse matrix, but using solve() here is not to form the inverse explicitly (which might be very costly)
A similar idea is very common in BFGS-based optimization algorithms although wiki might not give much insight here
scipy has an extra LinearOperator for this not forming the inverse explicitly! (although i think it's only used for statistics / completing/finishing some optimization; but i successfully build some LBFGS-based optimizers with this one)
Source # scicomp.stackexchange discussing this without touching scipy
And because of that i would assume spilu is completely going for this too (returning an object with a solve-method)
I am looking for a MINLP optimizer to solve portfolio optimisation problem which minimizes x'.S.x where x is a vector S is a given matrix. There are integer constraints which x elements depend on ex; x[i] = g[i].K[i] where g[i] is an integer and K[i] is a given vector, thus we need to find g[i]s while minimizing objective.
I am considering using AMPL or gams. Main program is in python. I am not sure if these are the best MINLP out there, but anyways there seems to be some examples on both websites. In terms of matrix multiplication for the minimization objective, I am not clear if there is a simple way of writing this in AMPL, do I need to write it as an algebraic expansion? Can you provide an example of x'.S.x operation in AMPL language?
In terms of gams, I see the package is free only for limited usage of a number of variables. Therefore I was considering AMPL, but maybe for smaller problems gams might be the solution if I cannot figure out AMPL notation for matrix vector multiplications
The AMPL syntax is very straightforward:
sum{i in I, j in I} x[i]*S[i,j]*x[j]
Note that many portfolio models do not need a full-blown MINLP solver but can be solved with the quadratic (and SOCP) capabilities present in systems like Cplex and Gurobi. Your question is somewhat difficult to parse (at least for me) but I believe your model falls in this category.
I'm trying to run an optimization with scipy.optimize.differential_evolution. The code calls for bounds for each variable in x. But I want to a solution where parts of x must be integers, while others can range freely as floats. The relevant part of my code looks like
bounds = [(0,3),(0,3),(0,3),???,???]
result = differential_evolution(func, bounds)
What do I replace the ???'s with to force those variables to be ints in a given range?
As noted in the comments there isn't direct support for a "integer constraint".
You could however minimize a modified objective function, e.g.:
def func1(x):
return func(x) + K * (x[3] - round(x[3]))**2
and this will force x[3] towards an integer value (unfortunately you have to tune the K parameter).
An alternative is to round (some of) the real-valued parameters before evaluating the objective function:
def func1(x):
z = x;
z[3] = round(z[3])
return func(z)
Both are common techniques to approach a discrete optimization problem using Differential Evolution and they work quite well.
Differential evolution can support integer constraint but the current scipy implementation would need to be changed.
From the scipy source code it appears that their DE is based Storn, R and Price, K, Differential Evolution - a Simple and Efficient Heuristic for Global Optimization over Continuous Spaces, Journal of Global Optimization, 1997
However there has been progress in this field as pointed out by this review paper Recent advances in differential evolution – An updated survey
There are a few papers which introduce changes to the algorithm so that it can handle ints. I have not had time to look at all the options but perhaps this paper could help.
An Improved Differential Evolution Algorithm for Mixed Integer Programming Problems
What do I replace the ???'s with to force those variables to be ints in a given range?
wrapdisc is a package that is a thin wrapper which will let you optimize bounded integer variables alongside floats with various scipy.optimize optimizers. There is a usage example in its readme. With it, you don't have to adapt your objective function at all. It internally uses rounding in order to support integers, although this detail is hidden from the user.