I have a large problem defined in CVXPY modelling language. I want to solve series of this problems - still the same format but with different parameters (constants).
I found out that after calling problem.solve() internal problem generating takes 20 s and main optimization runtime takes 0.2 s. It's a lot of time when I want to solve dozens of similar problems.
Is there any tool for CVXPY like YALMIP optimizer or any possibility to reduce problem generating time?
Yes there is. And it's even explained in the official docs.
Parameters
Parameters are symbolic representations of constants. The purpose of parameters is to change the value of a constant in a problem without reconstructing the entire problem.
Example straight from the docs (modified):
from cvxpy import *
import numpy
# Problem data.
n = 15
m = 10
numpy.random.seed(1)
A = numpy.random.randn(n, m)
b = numpy.random.randn(n, 1)
# gamma must be positive due to DCP rules.
gamma = Parameter(sign="positive") # !!!
# Construct the problem.
x = Variable(m)
error = sum_squares(A*x - b)
obj = Minimize(error + gamma*norm(x, 1))
prob = Problem(obj) # !!!
# Construct a trade-off curve of ||Ax-b||^2 vs. ||x||_1
sq_penalty = []
l1_penalty = []
x_values = []
gamma_vals = numpy.logspace(-4, 6)
for val in gamma_vals:
gamma.value = val # !!!
prob.solve() # !!!
# Use expr.value to get the numerical value of
# an expression in the problem.
sq_penalty.append(error.value)
l1_penalty.append(norm(x, 1).value)
x_values.append(x.value)
So what does it do
As you noticed, the setup of your optimization-problem might take some time, because it's following the DCP-approach (which proves convexity by construction).
Using parameter, this DCP-processing is only done one time! Every new solve willl only change some small parts within the problem. It's important to describe your parameter as precise as possible, so that DCP can work. Example: Parameter(sign="positive").
Is there more you can do
Maybe. Depending on the solver, you can also use warm-starting, if you think a special guess (e.g. solution-vector of your last iteration) is a good start for the new problem.
This would replace: prob.solve() with prob.solve(warm_start=True), resulting in reusing the previous solution as a start (explained here). Manually defining this vector does not seem to be possible (from cvxpy).
Sadly, the only solver supporting this (within cvxpy) is SCS as far as i know (and others will ignore it without crashing)!
Related
I am trying to parallelize an interaction with a Python object that is computationally expensive. I would like to use Ray to do this but so far my best efforts have failed.
The object is a CPLEX model object and I'm trying to add a set of constraints for a list of conditions.
Here's my setup:
import numpy as np
import docplex.mp.model as cpx
import ray
m = cpx.Model(name="mymodel")
def mask_array(arr, mask_val):
array_mask = np.argwhere(arr == mask_val)
arg_slice = [i[0] for i in array_mask]
return arg_slice
weeks = [1,3,7,8,9]
const = 1.5
r = rate = np.array(df['r'].tolist(), dtype=np.float)
x1 = m.integer_var_list(data_indices, lb=lower_bound, ub=upper_bound)
x2 = m.dot(x1, r)
#ray.remote
def add_model_constraint(m, x2, x2sum, const):
m.add_constraint(x2sum <= x2*const)
return m
x2sums = []
for w in weeks:
arg_slice = mask_array(x2, w)
x2sum = m.dot([x2[i] for i in arg_slice], r[arg_slice])
x2sums.append(x2sum)
#: this is the expensive part
for x2sum in x2sums:
add_model_constraint.remote(m, x2, x2sum, const)
In a nutshell, what I'm doing is creating a model object, some variables, and then looping over a set of weeks in order to build a constraint. I subset my variable, compute some dot products and apply the constraint. I would like to be able to create the constraint in parallel because it takes a while but so far my code just hangs and I'm not sure why.
I don't know if I should return the model object in my function because by default the m.add_constraint method modifies the object in place. But at the same time I know Ray returns references to the remote value so yea, not sure what's supposed to happen there.
Is this at all a valid use of ray? It it reasonable to expect to be able to modify a CPLEX object in this way (or any other arbitrary python object)?
I am new to Ray so I may be structuring this all wrong, or maybe this will never work for X, Y, and Z reason which would also be good to know.
The Model object is not designed to be used in parallel. You cannot add constraints from multiple threads at the same time. This will result in undefined behavior. You will have to at least a lock to make sure only thread at a time adds constraints.
Note that parallel model building may not be a good idea at all: the order of constraints will be more or less random. On the other hand, behavior of the solver may depend on the order of constraints (this is called performance variability). So you may have a hard time reproducing certain results/behavior.
I understand the primary issue was the performance of module building.
From the code you sent, I have two suggestions to address this:
post constraints in batches, that is store constraints in a list and add them once using Model.add_constraints(), this should be more efficient than adding them one at a time.
experiment with Model.dotf() (functional-style scalar product). It avoids building auxiliary lists, passing instead a function of the key , returning the coefficient.
This method is new in Docplex version 2.12.
For example, assuming a list of 3 variables:
abc = m.integer_var_list(3, name=["a", "b", "c"]) m.dotf(abc, lambda
k: k+2)
docplex.mp.LinearExpression(a+2b+3c)
Model.dotf() is usually faster than Model.dot()
I am trying to implement an R algortihm dealing with non-negative ODE Systems. I need something like ode45 in MATLAB to define states which have to be none-negative.
I discussed about that already 3 years ago but with no real solution. deSolve is still not the way to go. I found some python code which looks very promising. Maybe this is possible in R as well. In the end I have to define a function wraper, as functools in python. What it does is pretty simple. Here is the code the of the python wraper:
def wrap(f):
#wraps(f)
def wrapper(t, y, *args, **kwargs):
low = y < 0
y = np.maximum(y, np.ones(np.shape(y))*0)
result = f(t, y, *args, **kwargs)
result[too_low] = np.maximum(result[low], np.ones(low.sum())*0)
return result
return wrapper
return wrap
I mean in python this is straight forward. The wraper will be used in each step of the integration called by
solver = scipy.integrate.odeint(f, y0)
solution = solver.solve()
Is the same possible in R? I know there is a functools package and functools function, as well. But I have no clue if this really works. Can I use events in deSolve for that?
I am working now on this project for 5 years and I am out of ideas. I used an MATLAB, C++ and Python interface but all this is to slow, I need it in R. Thank you very much for your help!
deSolve does not support automatic non-negativity constraints for good reasons. We had such questions several times in the past, but it turned out in all these cases, that the reason of the negative value was an incomplete model specification. The typical case is that something is exported from an empty pool. Because unwanted negative values are usually an indicator of an inadequate model specification, we do (currently) not consider to add a "non-negative" constraint in the future.
Example: in the following equation, X can become negative by model design:
dX/dt = -k
whereas the following cannot:
dX/dt = -k * X
If you need a linear decrease "most of the time" that reduces to zero shortly before X becomes zero, you can use a Monod-type safeguard (or something similar):
dX/dt = -k*X / (k2 + X)
The selection of k2 is relatively uncritical. It should be small enough not to influence the overall behavior and not too small, compared to the numerical accuracy of the solver.
Another method to avoid negative values is to work in log-transformed space. Here are some related threads:
https://stat.ethz.ch/pipermail/r-sig-dynamic-models/2010q2/000028.html
https://stat.ethz.ch/pipermail/r-sig-dynamic-models/2013q3/000222.html
https://stat.ethz.ch/pipermail/r-sig-dynamic-models/2016/000437.html
In addition, it is of course also possible to write an own wrapper in R.
Hope it helps
EDIT: looks like this was already answered before here
It didn't show up in my searches because I didn't know the right nomenclature. I'll leave the question here for now in case someone arrives here because of the constraints.
I'm trying to optimize a function which is flat on almost all points ("steps function", but in a higher dimension).
The objective is to optimize a set of weights, that must sum to one, and are the parameters of a function which I need to minimize.
The problem is that, as the function is flat at most points, gradient techniques fail because they immediately converge on the starting "guess".
My hypothesis is that this could be solved with (a) Annealing or (b) Genetic Algos. Scipy sends me to basinhopping. However, I cannot find any way to use the constraint (the weights must sum to 1) or ranges (weights must be between 0 and 1) using scipy.
Actual question: How can I solve a minimization problem without gradients, and also use constraints and ranges for the input variables?
The following is a toy example (evidently this one could be solved using the gradient):
# import minimize
from scipy.optimize import minimize
# define a toy function to minimize
def my_small_func(g):
x = g[0]
y = g[1]
return x**2 - 2*y + 1
# define the starting guess
start_guess = [.5,.5]
# define the acceptable ranges (for [g1, g2] repectively)
my_ranges = ((0,1),(0,1))
# define the constraint (they must always sum to 1)
def constraint(g):
return g[0] + g[1] - 1
cons = {'type':'eq', 'fun': constraint}
# minimize
minimize(my_small_func, x0=start_guess, method='SLSQP',
bounds=rranges, constraints=cons)
I usually use R so maybe this is a bad answer, but anyway here goes.
You can solve optimization problems like the using a global optimizer. An example of this is Differential Evolution. The linked method does not use gradients. As for constraints, I usually build them manually. That looks something like this:
# some dummy function to minimize
def objective.function(a, b)
if a + b != 1 # if some constraint is not met
# return a very high value, indicating a very bad fit
return(10^90)
else
# do actual stuff of interest
return(fit.value)
Then you simply feed this function to the differential evolution package function and that should do the trick. Methods like differential evolution are made to solve in particular very high dimensional problems. However the constraint you mentioned can be a problem as it will likely result in very many invalid parameter configurations. This is not necessarily a problem for the algorithm, but is simply means you need to do a lot of tweaking and need to expect a lot of waiting time. Depending on your problem, you could try optimizing weights/ parameters in blocks. That means, optimize parameters given a set of weights, then optimize weights given the previous set of parameters and repeat that many times.
Hope this helps :)
Currently I'm using PuLP to solve a maximization problem. It works fine, but I'd like to be able to get the N-best solutions instead of just one. Is there a way to do this in PuLP or any other free/Python solution? I toyed with the idea of just randomly picking some of the variables from the optimal solution and throwing them out and re-running, but this seems like a total hack.
If your problem is fast to solve, you can try to limit the objective from above step by step. For examle, if the objective value of the optimal solution is X, try to re-run the problem with an additional constraint:
problem += objective <= X - eps, ""
where the reduction step eps depends on your knowledge of the problem.
Of course, if you just pick some eps blindly and get a solution, you don't know if the solution is the 2nd best, 10th best or 1000-th best ... But you can do some systematic search (binary, grid) on the eps parameter (if the problem is really fast to solve).
So I figured out how (by RTFM) to get multiple soutions. In my code I essentially have:
number_unique = 1 # The number of variables that should be unique between runs
model += objective
model += constraint1
model += constraint2
model += constraint3
for i in range(1,5):
model.solve()
selected_vars = []
for p in vars:
if p_vars[p].value() != 0:
selected_vars.append(p)
print_results()
# Add a new constraint that the sum of all of the variables should
# not total up to what I'm looking for (effectively making unique solutions)
model += sum([p_vars[p] for p in selected_vars]) <= 10 - number_unique
This works great, but I've realized that I really do need to go the random route. I've got 10 different variables and by only throwing out a couple of them, my solutions tend to have the same heavy weighted vars in all the permutations (which is to be expected).
I am having trouble sovling the optical bloch equation, which is a first order ODE system with complex values. I have found scipy may solve such system, but their webpage offers too little information and I can hardly understand it.
I have 8 coupled first order ODEs, and I should generate a function like:
def derv(y):
compute the time dervative of elements in y
return answers as an array
then do complex_ode(derv）
My questions are:
my y is not a list but a matrix, how can i give a corrent output
fits into complex_ode()?
complex_ode() needs a jacobian, I have no idea how to start constructing one
and what type it should be?
Where should I put the initial conditions like in the normal ode and
time linspace?
this is scipy's complex_ode link:
http://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.complex_ode.html
Could anyone provide me with more infomation so that I can learn a bit more.
I think we can at least point you in the right direction. The optical
bloch equation is a problem which is well understood in the scientific
community, although not by me :-), so there are already solutions on the internet
to this particular problem.
http://massey.dur.ac.uk/jdp/code.html
However, to address your needs, you spoke of using complex_ode, which I suppose
is fine, but I think just plain scipy.integrate.ode will work just fine as well
according to their documentation:
from scipy import eye
from scipy.integrate import ode
y0, t0 = [1.0j, 2.0], 0
def f(t, y, arg1):
return [1j*arg1*y[0] + y[1], -arg1*y[1]**2]
def jac(t, y, arg1):
return [[1j*arg1, 1], [0, -arg1*2*y[1]]]
r = ode(f, jac).set_integrator('zvode', method='bdf', with_jacobian=True)
r.set_initial_value(y0, t0).set_f_params(2.0).set_jac_params(2.0)
t1 = 10
dt = 1
while r.successful() and r.t < t1:
r.integrate(r.t+dt)
print r.t, r.y
You also have the added benefit of an older more established and better
documented function. I am surprised you have 8 and not 9 coupled ODE's, but I'm
sure you understand this better than I. Yes, you are correct, your function
should be of the form ydot = f(t,y), which you call def derv() but you're
going to need to make sure your function takes at least two parameters
like derv(t,y). If your y is in matrix, no problem! Just "reshape" it in
the derv(t,y) function like so:
Y = numpy.reshape(y,(num_rows,num_cols));
As long as num_rows*num_cols = 8, your number of ODE's you should be fine. Then
use the matrix in your computations. When you're all done, just be sure to return
a vector and not a matrix like:
out = numpy.reshape(Y,(8,1));
The Jacobian is not required, but it will likely allow the computation to proceed
much more quickly. If you do not know how to compute this you may want to consult
wikipedia or a calculus text book. It's pretty simple, but can be time consuming.
As far as initial conditions, you should probably already know what those should
be, whether it's complex or real valued. As long as you select values that are
within reason, it shouldn't matter much.