We Keep Coding

How do I integrate a logical boolean constraint into the standard matrix form of a Mixed Integer Linear Problem in scipy.optimize.milp / linprog?

I want to implement boolean logic and dependent variables into a Mixed-Integer Linear Program with scipy.optimize.milp using a highs solver.
How do I set the actual matrices and vectors c, A_ub, b_ub, A_eq, b_eq to fit these exemplary Boolean operations of the exemplary MILP:
Boolean variables: a, b, c, d, e, f, g, h, i, j, k, l
Minimize 1a+1b+...+1l
such that:
a OR b
c AND d
e XOR f
g NAND h
i != j
k == l
a,b,...,l are set to integers via the integrality parameter:
integrality=np.repeat(3, 12+amount_of_helper_variables)
And the lower and upper bounds are set to match boolean values 1 or 0 only:
Forall x in {a,b,...,l}: 0 <= x <= 1
I figured this CS post might help a lot as a general building guide, especially for solvers taking arbitrary formula input formats, but didn't get far myself with the conversion to standard matrix form until now.
I'm asking for a generalizable approach for conversion that basically can be used as a helper method for array creation and doesn't just apply to the stated problem but all boolean formula conversions for standard matrix form MILP using np.arrays to juggle the variables and helpers around.

Disclaimer
Generalization is fine, but sometimes we lose exploitable substructures in mathematical-optimization. Sometimes this is bad!
Recommendation
That being said, i recommend the following.
Intermediate language: Conjunctive normal form
It's well known, that we can express any boolean function with it
It's the form a SAT-solver would expect: DIMACS CNF -> some empirical proof that it's a good pick
There is lots of well-understood tooling
There is a natural MILP-formulation
Transformation: CNF -> MILP
Helper-function
Input: CNF defined on boolean variables (integral and bounded by [0, 1])
Output:
Set of constraints aka rows in constraint matrix A_ub
Set of constants aka scalars in b_ub
No matter what kind of input you have:
You might go through one joint CNF or decompose into many CNFs. And by definition you can concatenate them and their "conjunction." Meaning: A_ub and b_ub are stacking those outputs.
The transformation is simple:
for each c in cnf:
for each disjunction in c:
add constraint:
---------------
sum of positive literals - sum of negative literals >= 1 - |negative literals|
Wiki: Literal:
A positive literal is just an atom (e.g. x).
A negative literal is the negation of an atom (e.g. not x).
Example for a given clause = disjunction in some cnf:
x1 or x2 or !x3
->
x1 + x2 + (1-x3) >= 1 easier to understand
<->
x1 + x2 - x3 >= 1 - 1 as proposed above
<->
x1 + x2 - x3 >= 0
(i left one step open -> we need to multiply our constraints with -1 to follow scipys standard-form; but well... you get the idea)
Tooling
CNF
SymPy has a boolean algebra module which could help (e.g. transform to cnf)
pyeda can achieve similar things (and is actually more targeting use-cases like that)
Remarks
There is tons of other potentiall relevant stuff, especially around CNF-creation.
These things are often important in the real-world, e.g. Tseitin-transformation (for cases where a native cnf-creation would result in exponential-size). pyeda also knows about tseitin if i remember correctly.
But well... it's just a Stack-Overflow answer ;-)
References
If you need some reading material, i recommend:
Hooker, John N. Integrated methods for optimization. Vol. 170. New York: Springer, 2012.

I would approach this in two steps:
Write things down equation based
Convert (painfully) into matrix format
So we have:
x OR y. I.e. x=1 OR y=1. That is x+y>=1.
x AND y. I.e. x=1 AND y=1. That means just fixing both variables to 1.
x XOR y. I.e. x=1 XOR y=1. That is x+y=1.
x NAND y. I.e. not (x=1 AND y=1). So x+y<=1.
x <> y. This different notation for x XOR y. We handled that already.
x=y.This equation is ready as is. Maybe write as x-y=0.
Step 2, can usually be done in block format using a (large) piece of paper. Each column is a variable (or block of variables) and each row is a constraint. Here all matrix entries (coefficients) are 0, -1 or 1. E.g. x-y=0 means: create a row with a coefficient of 1 in the x column and a -1 in the y column. See: How to implement Linear Programming problem in scipy with complex objective for an example. It is often better to automate this and let a program do this for you. Python tools that do this for you are e.g. PuLP and Pyomo.

Compare decision variables to floating values

I'm currently working on a trajectory optimization problem that involves binary actuators. In order to avoid solving an MINLP I do not simply optimize over the states and control inputs but instead, assume that each of the binary actuators alternates between the states "on" and "off" and optimize over the duration of those intervals. I will call the array containing the decision variable h (an N by 2 matrix in the particular case below).
A minimal example using a double integrator that has two control inputs that enact a positive or negative force on the system respectively would be:
Here I model the state trajectory as some train of 3rd order polynomial.
I particularly do not want to merge these actuators into one with the states -1,0,1 since the more general system I'd like to apply this to also has more binary actuators. I add some additional constraints such as connecting the polynomials continuously and differentiably; enforce that the sum of all intervals is equal to the desired final time; enforce initial and final state constraints and finally enforce the dynamics of the system.
My initial idea was to then enforce the dynamics at constant intervals, i.e.:
However, the issue here is that any of the actuators could be in any arbitrary interval for some time t. Since the intervals can shrink to duration zero one actuator might be in the last interval while the other one remains in the first. I.e. the value of a decision variable (time duration) changes which decision variables are dependent on each other. This is more or less manifested in drake by the fact that I cannot do a comparison such as Tau < t if Tau is a drake expression and t is some number. The code snippet is:
# Enforce dynamics at the end of each control interval
for t in np.arange(0, Tf, dt_dyn):
# Find the index of the interval that is active for each actuator
t_ctrl = np.cumsum(h, axis=0)
intervals = (t_ctrl < t)
idxs = np.sum(intervals, axis=0)
# If the idx is even the actuator is off, otherwise its on
prog.AddConstraint(eq(qdd(q_a, t, dt_state),
continuous_dynamics(q(q_a, t, dt_state),
qd(q_a, t, dt_state),
[idxs[0] % 2, idxs[1] % 2])))
and the resulting error message:
Traceback (most recent call last):
File "test.py", line 92, in <module>
intervals = (t_ctrl < t)
RuntimeError: You should not call `__bool__` / `__nonzero__` on `Formula`. If you are trying to make a map with `Variable`, `Expression`, or `Polynomial` as keys (and then access the map in Python), please use pydrake.common.containers.EqualToDict`.
In the end, my question is more of a conceptual than technical nature: Does drake support this dependence of the "dependence of decision variables on the values" in some other way? Or is there a different way I can transcribe my problem to avoid the ambiguity in which interval my actuators are for some given time?
I've also linked to the overall script that implements the minimal example here

The immediate problem is that intervals = (t_ctrl < t) is a vector of dtype Formula, not a vector of dtype Variable(type=BINARY), so you can't actually sum it up. To do an arithmetic sum, you'd need to change that line to an np.vectorize-wrapped function that calls something like if_then_else(t_argument < t_constant, 0.0, 1.0) in order to have it be an Expression-valued vector, at which which would be sum-able.
That won't actually help you, though, since you cannot do % (modular) arithmetic on symbolic expressions anyway, so the % 2.0 == 0.0 stuff will raise an exception once you make it that far.
I suspect that you'll need a different approach to encoding the problem into variables, but unfortunately an answer there is beyond my skill level at the moment.

Problem with roots of a non-linear equation

I have a hyperbolic function and i need to find the 0 of it. I have tried various classical methods (bisection, newton and so on).
Second derivatives are continuous but not accessible analytically, so i have to exclude methods using them.
For the purpose of my application Newton method is the only one providing sufficient speed but it's relatively unstable if I'm not close enough to the actual zero. Here is a simple screenshot:
The zero is somewhere around 0.05. and since the function diverges at 0, if i take a initial guess value greater then the minimum location of a certain extent, then i obviously have problems with the asymptote.
Is there a more stable method in this case that would eventually offer speeds comparable to Newton?
I also thought of transforming the function in an equivalent better function with the same zero and only then applying Newton but I don't really know which transformations I can do.
Any help would be appreciated.

Dekker's or Brent's method should be almost as fast as Newton. If you want something simple to implement yourself, the Illinois variant of the regula-falsi method is also reasonably fast. These are all bracketing methods, so should not leave the domain if the initial interval is inside the domain.
def illinois(f,a,b,tol=1e-8):
'''regula falsi resp. false postion method with
the Illinois anti-stalling variation'''
fa = f(a)
fb = f(b)
if abs(fa)<abs(fb): a,fa,b,fb = b,fb,a,fa
while(np.abs(b-a)>tol):
c = (a*fb-b*fa)/(fb-fa)
fc = f(c)
if fa*fc < 0:
fa *= 0.5
else:
a, fa = b, fb
b, fb = c, fc
return b, fb

How about using log(x) instead of x?

For your case, #sams-studio's answer might work, and I would try that first. In a similar situation - also in multi-variate context - I used Newton-homotopy methods.
Basically, you limit the Newton step until the absolute value of y is descending.
The cheapest way to implement is that you half the Newton step if y increases from the last step. After a few steps, you're back at Newton with full second order convergence.
Disclamer: If you can bound your solution (you know a maximal x), the answer from #Lutz Lehmann would also be my first choice.

Can I make a Min Z = max(a,b,c) in PuLP

I was wondering if i could make a multiple objective function in PuLP, by doing this Can I make a Min Z = max(a,b,c) in PuLP, however when using this code
ilp_prob = pulp.LpProblem("Miniimize Problem", pulp.LpMinimize)
x = []
if m >3:
return 1,1
for i in range(m):
temp = []
for j in range(len(jobs)):
temp += [pulp.LpVariable("x_%s_%s" %((i+1),(j+1)),0,1, cat = 'Binary')]
x+= [temp]
ilp_prob += max([pulp.lpSum([jobs[j]*x[i][j] for j in range(len(jobs))] for i in range(m))])
for i in range(len(jobs)):
ilp_prob += pulp.lpSum([x[j][i] for j in range(m)])==1
ilp_prob.solve()
It just returns all 1 in x[0], and all 0 in x[0].

I'm pretty sure you can't just use python's (!) max on pulp's internal expressions! Those solvers are working on a very specific problem-specification, LP standard form, where is no concept for that!
The exception would be if pulp would overload this max-function for it's data-structures (don't know if that's possible at all in python), but i'm pretty sure pulp does not support re-formulations like that (there is some needed; as again: the target is the Standard-form).
cvxpy for example does not overload, but introduces customized max-functions, which internally transform your problem.
That being said: i'm surprised your code runs without a critical error. But i'm too lazy to check pulps sources here.
Have a look at the usual LP/IP formulation-guides.
A first idea would be:
target: min (max(a,b,c))
reformulation:
introduce a new variable z
add constraints:
z >= a
z >= b
z >= c
assumption: the objective somehow want's to minimize z (maximizing will get you in trouble as the problem will get unbounded!)
this is the case here, as the final objective for our target would look like:
min(z)
Remark: One has to be careful that the problem will still be linear/convex (depending on the solver). In this case (our simple example; i did not check your whole model) i don't see a problem, but in more complex cases, min(max(complex_expression)) subjective to complex constraints, this might introduce non-convexity (and can't be solved by Conic solvers incl. LP-solvers).
And just throwing a keyword in the ring: your approach/objective sounds a bit like robust-optimization, where usually some worst-case scenario is optimized. Not all multi-objective optimization problems are treating multiple objective-components like that.

How to access solution for dual simplex solver?

I have an objective function with several hundreds of quadratic terms which I would like to minimize; in this case I try to minimize the absolute distance between several variables. So the structure of my problem looks like this (highly simplified):
Minimize
obj: [ a^2 - 2 a * b + b^2 ] / 2
Subject To
c1: a + b >= 10
c2: a <= 100
End
I use the Python API to solve the problem in the following way:
import cplex
cpx = cplex.Cplex()
cpx.read('quadratic_obj_so.lp')
# use the dual simplex
cpx.parameters.lpmethod.set(cpx.parameters.lpmethod.values.dual)
cpx.solve()
print cpx.solution.get_values()[0:15]
print cpx.solution.status[cpx.solution.get_status()]
print cpx.solution.get_objective_value()
And for the above example I then receive (showing only iterations 16-18):
Itn Primal Obj Dual Obj Prim Inf Upper Inf Dual Inf
16 1.4492800e-19 -1.0579911e-07 3.81e-14 7.11e-15 5.17e-25
17 9.0580247e-21 -2.6449779e-08 1.91e-14 3.55e-15 2.33e-25
18 5.6612645e-22 -6.6124446e-09 5.45e-14 7.11e-15 6.46e-27
[73.11695794600045, 73.11695794603409]
optimal
0.0
so a and b are equal which makes sense since I try to minimize their distance and the constrains are clearly fulfilled.
However, my actual problem is far more complex and I receive:
Itn Primal Obj Dual Obj Prim Inf Upper Inf Dual Inf
92 1.4468496e+06 1.2138985e+06 1.80e+02 2.64e-12 5.17e-02
93 1.4468523e+06 1.2138969e+06 2.23e+02 2.17e-12 1.08e-02
94 1.4468541e+06 1.2138945e+06 2.93e+02 2.31e-12 5.62e-02
* 1.4457132e+06 1.2138598e+06 7.75e+00 7.61e-09 2.76e-02
num_best
1445714.46525
I have now several questions regarding the output which are closely connected:
1) Clearly, it is not the objective value for the dual simplex printed. Why is that, since I set the solver to be the dual simplex?!
2) How do I now access the results for the dual simplex? As the objective value is smaller I would be more interested in these results.
3) Does the num_best status guarantee that all the constrains are met i.e. is the solution valid but just not guaranteed to be optimal?
4) Primal Obj and Dual Obj differ quite a lot. Is there any strategy to minimize their difference?

To the best of my knowledge, get_objective_value always returns the best primal bound (regardless of the lpmethod).
Information about the dual solution can be retrieved with get_dual_values.
The num_best solution status means that a solution is available, but there is no proof of optimality (see here). This is probably the most important point with regards to the rest of the questions here.
You could try turning on the numerical emphasis parameter to see if that helps. There are also various tolerances you can adjust (e.g., optimality tolerance).
Note that all of the links I've used above are for the C Callable Library (which the Python API calls internally) for CPLEX 12.6.3.