We Keep Coding

Adding soft constraints to a scheduling problem in or-tools python

tools and im trying to use it to generate a timeTable for a highschool.
Variables are Lesson, rooms, and timeslots and the goal of course to assign all the lessons to a certain room and a timeslot while respecting the given constraints.
The problem is in the documentation i don't see it talking about soft and hard constraint and all the constraint i've added are surely hard ones, is there a way to implement soft constraint for this example just a simple one.
Thanks in advance.

Probably a duplicate of Do Google Optimization Tools support Soft Constraints?, but I'll add some examples with CP-SAT.
Here's a simple soft limit example:
from ortools.sat.python import cp_model
model = cp_model.CpModel()
solver = cp_model.CpSolver()
x = [model.NewBoolVar("") for i in range(10)]
# hard constraint: number of 1's >= 4
model.Add(sum(x) >= 4)
# soft constraint: number of 1's <= 5
delta = model.NewIntVar(-5, 5, "")
excess = model.NewIntVar(0, 5, "")
model.Add(delta == sum(x) - 5)
model.AddMaxEquality(excess, [delta, 0])
model.Minimize(excess)
solver.Solve(model)
print([solver.Value(i) for i in x])
print(solver.Value(excess))
See a more complex example here
And here's one concerning fullfiled requests:
from ortools.sat.python import cp_model
model = cp_model.CpModel()
solver = cp_model.CpSolver()
x = [model.NewIntVar(0, 10, "") for i in range(10)]
# request: sum() <= 10
req1 = model.NewBoolVar("")
model.Add(sum(x) <= 10).OnlyEnforceIf(req1)
# request: sum() >= 5
req2 = model.NewBoolVar("")
model.Add(sum(x) >= 5).OnlyEnforceIf(req2)
# request: sum() >= 100
req3 = model.NewBoolVar("")
model.Add(sum(x) >= 100).OnlyEnforceIf(req3)
model.Maximize(req1 + req2 + req3)
solver.Solve(model)
print(solver.Value(sum(x)))
print(solver.ObjectiveValue())

issue with converting rate equation to python code

I am trying to convert these rate equations to python code, I have made I lot of research but can't seem to get any clear path to follow to achieve this, please any help will be appreciated
This is a newly updated code....i wrote using the quide from Tom10.....please what do you think?
import numpy as np
# import numpy as sum # not necessary, just for convenience, and replaces the builtin
# set N_core value
N_CORE = 0
# set the initial conditions appropriately (you need to set these correctly)
N = np.ones(8)
r = np.ones((8, 8))
dN = np.zeros(8) # the value here is not important for your equations
# set constant for equation 1
R_P1abs37 = 20
F_P1 = 20
R_P1abs47 = 40
W_3317 = 1.0
# set constant for equation 2
W_6142 = 90
W_5362 = 80
# Set you constants appropriately for equation 3
R_P2abs35 = 30
F_P2 = 40
R_L2se34 = 50
F_L2 = 90
# equation 4 constants
W_2214 = 20
#equation 5 constants
R_P1abs13 = 30
R_L2se32 = 20
F_L1 = 10
# equation 1 formular
dN[7] =sum(r[7,:]*N[7]) + (R_P1abs37*F_P1) + (R_P1abs47*F_P1) + (W_3317*N[3]**2)
# equation 2 formular
dN[6] = (r[7,6]*N[7]) - sum(r[6,:]*N[6]) - (W_6142*N[6]*N[1]) + (W_5362*N[5]*N[3])
#equation 3 formular
dN[5] = sum(r[:,5]*N) - sum(r[5,:]*N[5]) + R_P2abs35*F_P2 - R_L2se34*F_L2 - W_5362*N[5]*N[3]
# equation 4 formular
dN[4] = sum(r[:,4]*N) - sum(r[4,:]*N[4]) - (R_P1abs47*F_P1) + (R_L2se34*F_L2) + (W_2214*N[2]**2)+ (W_6142*N[6]*N[1])
#equation 5 formular
dN[3] = sum(r[:,3]*N) - sum(r[3,:]*N[3]) + (R_P1abs13*F_P1) - (R_P1abs37*F_P1) - (R_P2abs35*F_P2)
-(R_L2se32*F_L1) - ((2*W_3317)*N[3]**2) - (W_5362*N[5]*N[3])
#equation 6 formular
dN[2] = sum(r[:,2]*N) - (r[2,1]*N[2]) + (R_L2se32*F_L1) - ((2*W_2214)*N[2]**2) + (W_6142*N[6]*N[1])+(W_5362*N[5]*N[3])
#equation 7 formular
dN[1] = sum(r[:,1] * N) - (R_P1abs13*F_P1) + (W_2214*N[2]**2) + (W_3317+N[3]**2) - (W_6142+N[6]*N[1])
#equation for N CORE
N_CORE = sum(dN)
print(N_CORE)

Here is list of relevant issues based on your question and comments:
Usually if the summation is over i, then everything without an i subscript is constant for that sum. (Mathematically these constant terms can just be brought out of the sum; so the first equation is a bit odd where the N_7 could be moved out of the sum but I think they're keeping it in to show the symmetry with the other equations which all have an r*N term).
The capitol sigma symbol (Σ) means you need to do a sum, which you can do in a loop, but both Python list and numpy have a sum function. Numpy has the additional advantage that multiplication is interpreted as multiplication of the individual elements, making the expression easier. So for a[0]*[b0] + a[1]*b[1] + a[2]*b[2] and numpy arrays is simply sum(a*b) and for Python lists it's sum([a[i]*b[i] for in range(len(a))]
Therefore using numpy, the setup and your third equation would look like:
import numpy as np
import numpy.sum as sum # not necessary, just for convenience, and replaces the builtin
# set the initial conditions appropriately (you need to set these correctly)
N = np.ones(7, dtype=np.float)
# r seems to be a coupling matrix, and should be set according to your system
r = np.ones((7, 7), dtype = np.float)
# the values for dN are not important for your equations because dN only appears on the left side of the equations, so we just make a place to store the results
dN = np.zeros(7, dtype=np.float)
# Set you constants appropriate.y
R_P2abs35 = 1.0
F_P2 = 1.0
R_L2se34 = 1.0
F_L2 = 1.0
W_5362 = 1.0
dN[5] = sum(r[:,5]*N) - sum(r[5,:]*N[5]) + R_P2abs35*F_P2 - R_L2se34*F_L2 - W_5362*N[5]*N[3]
Note that although the expressions in the sums look similar, the first is essentially a dot product between two vectors and the second is a scalar times a vector so N[5] could be taken out of the sum (but I left it there to match the equation).
Final note: I see you're new to S.O. so I thought it would be helpful if I answered this question for you. In the future, please show some attempt at the code -- it really helps a lot.

Follow up question: GEKKO optimization in matrix form

this is a follow up question to the one I posted earlier:
GEKKO - optimization in matrix form
I need to add one more constraint that tracks "inventory" ("Inv"), which tracks the sum(q[i,:] - q[:,i]). "Inv" will be a 4X1 column vector. I tried the following:
m = GEKKO(remote=False)
q = m.Array(m.Var,(4,4),lb=0,ub=10)
for i in range(4):
for j in range(4):
if j<=i:
q[i,j].upper=0 # set upper bound = 0
def profit(q):
profit = np.sum(q.flatten() * pmx.flatten())
return profit
Inv[0]=0
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
m.Equation(I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i]))) # New Line 1a inventory
Inv[i] = Inv[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])) # New Line 1b inventory. Keep either 1a or 1b
m.Equation(Inv[i] <= 15) # New Line 2 inventory constraint
m.Equation(Inv[4] = 0) # New Line 3 ending inventory should be equal to starting inventory
m.Maximize(profit(q))
m.solve()
print(q)
qr = np.array([[q[i,j].value[0] for j in range(4)] for i in range(4)])
Ir = np.array([Inv[i].value[0] for i in range(4)]) #New Line 4
Errors :
1a. Adding New Line 1a: "keyword can't be an expression"
1b. Replacing New Line 1a with 1b: no issues (but, I'm not sure if GEKKO will keep track of I or not.Also, I need to define "I", the way "q" was done...not sure how). Replacing = comment out 1a, then run the code with 1b.
New Line 2: Error = "object of type 'int' has no len()"; but type(I) shows as ndarray. (Kept New Lines 1a and 1b, and then added New Line 2)
New Line 3:Error = "keyword can't be an expression" (Kept Line 32and then added Line 3)
New Line 4: Error "'numpy.ndarray' object has no attribute 'value'" [removed Lines 3 and 4. This makes sense as if I can't capture "Inv" in the model, then it won't have the value attribute)
Questions:
1. Am I defining inventory correctly?
If yes, can it be done in the current model specification, or will it need an entirely different formulation? If so, could you please guide on what that would be?
Of the various videos posted on the GEKKO website, is there a specific one I should look at for more information? I was thinking the DO video, but I don't believe this is quite a dynamic optimization problem (as I'm not trying to optimize the best path)....
Thanks again for all your help,
----UPDATE 5/10
Also tried:
Inv = m.SV()
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
#m.Equation(I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])))
m.Equation(m.Inv.dt() == m.Inv + (np.sum(q[i,:]) - np.sum(q[:,i])))
#I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])
m.Equation(m.Inv <= 15)
#m.Equation(I[4] = 0)
m.Maximize(profit(q))
New Error: 'GEKKO' object has no attribute 'Inv'

One way to do this is to start with zero inventory with Inv[0]=0 and then track the inventory amount with gekko variables Inv[1:4]. A couple tips on building the model:
Use double equal signs for equality constraints
You can define a temporary variable such as I = Inv[2]+Inv[3] but it won't be a gekko variable
You may also want to look at Intermediate variables for those that are explicitly calculated. This can speed up the calculation.
I recommend this tutorial in the Dynamic Optimization course
import numpy as np
import scipy.optimize as opt
from gekko import GEKKO
p= np.array([4, 5, 6.65, 12]) #p = prices
pmx = np.triu(p - p[:, np.newaxis]) #pmx = price matrix, upper triangular
m = GEKKO(remote=False)
q = m.Array(m.Var,(4,4),lb=0,ub=10)
# only upper triangular can change
for i in range(4):
for j in range(4):
if j<=i:
q[i,j].upper=0 # set upper bound = 0
def profit(q):
profit = np.sum(q.flatten() * pmx.flatten())
return profit
Inv = m.Array(m.Var,5,lb=0,ub=15)
Inv[0].upper = 0 # start with 0 inventory
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
# track inventory
m.Equation(Inv[i+1]==Inv[i] + (m.sum(q[i,:])-m.sum(q[:,i])))
m.Equation(Inv[4] == 0) # Use double == sign, not needed in loop
m.Maximize(profit(q))
m.solve()
print(q)
# convert to matrix form
qr = np.array([[q[i,j].value[0] for j in range(4)] for i in range(4)])
for i in range(4):
rs = qr[i,:].sum()
print('Row sum ' + str(i) + ' = ' + str(rs))
cs = qr[:,i].sum()
print('Col sum ' + str(i) + ' = ' + str(cs))
Ir = np.array([Inv[i].value[0] for i in range(4)])
print(Ir)

Why is Z3 slow for tiny search space?

I'm trying to make a Z3 program (in Python) that generates boolean circuits that do certain tasks (e.g. adding two n-bit numbers) but the performance is terrible to the point where a brute-force search of the entire solution space would be faster. This is my first time using Z3 so I could be doing something that impacts my performance, but my code seems fine.
The following is copied from my code here:
from z3 import *
BITLEN = 1 # Number of bits in input
STEPS = 1 # How many steps to take (e.g. time)
WIDTH = 2 # How many operations/values can be stored in parallel, has to be at least BITLEN * #inputs
# Input variables
x = BitVec('x', BITLEN)
y = BitVec('y', BITLEN)
# Define operations used
op_list = [BitVecRef.__and__, BitVecRef.__or__, BitVecRef.__xor__, BitVecRef.__xor__]
unary_op_list = [BitVecRef.__invert__]
for uop in unary_op_list:
op_list.append(lambda x, y : uop(x))
# Chooses a function to use by setting all others to 0
def chooseFunc(i, x, y):
res = 0
for ind, op in enumerate(op_list):
res = res + (ind == i) * op(x, y)
return res
s = Solver()
steps = []
# First step is just the bits of the input padded with constants
firststep = Array("firststep", IntSort(), BitVecSort(1))
for i in range(BITLEN):
firststep = Store(firststep, i * 2, Extract(i, i, x))
firststep = Store(firststep, i * 2 + 1, Extract(i, i, y))
for i in range(BITLEN * 2, WIDTH):
firststep = Store(firststep, i, BitVec("const_0_%d" % i, 1))
steps.append(firststep)
# Generate remaining steps
for i in range(1, STEPS + 1):
this_step = Array("step_%d" % i, IntSort(), BitVecSort(1))
last_step = steps[-1]
for j in range(WIDTH):
func_ind = Int("func_%d_%d" % (i,j))
s.add(func_ind >= 0, func_ind < len(op_list))
x_ind = Int("x_%d_%d" % (i,j))
s.add(x_ind >= 0, x_ind < WIDTH)
y_ind = Int("y_%d_%d" % (i,j))
s.add(y_ind >= 0, y_ind < WIDTH)
node = chooseFunc(func_ind, Select(last_step, x_ind), Select(last_step, y_ind))
this_step = Store(this_step, j, node)
steps.append(this_step)
# Set the result to the first BITLEN bits of the last step
if BITLEN == 1:
result = Select(steps[-1], 0)
else:
result = Concat(*[Select(steps[-1], i) for i in range(BITLEN)])
# Set goal
goal = x | y
s.add(ForAll([x, y], goal == result))
print(s)
print(s.check())
print(s.model())
The code basically lays out the inputs as individual bits, then at each "step" one of 5 boolean functions can operate on the values from the previous step, where the final step represents the end result.
In this example, I generate a circuit to calculate the boolean OR of two 1-bit inputs, and an OR function is available in the circuit, so the solution is trivial.
I have a solution space of only 5*5*2*2*2*2=400:
5 Possible functions (two function nodes)
2 Inputs for each function, each of which has two possible values
This code takes a few seconds to run and provides a correct answer, but I feel like it should run instantaneously as there are only 400 possible solutions, of which quite a few are valid. If I increase the inputs to be two bits long, the solution space has a size of (5^4)*(4^8)=40,960,000 and never finishes on my computer, though I feel this should be easily doable with Z3.
I also tried effectively the same code but substituted Arrays/Store/Select for Python lists and "selected" the variables by using the same trick I used in chooseFunc(). The code is here and it runs in around the same time the original code does, so no speedup.
Am I doing something that would drastically slow down the solver? Thanks!

You have a duplicated __xor__ in your op_list; but that's not really the major problem. The slowdown is inevitable as you increase bit-size, but on a first look you can (and should) avoid mixing integer reasoning with booleans here. I'd code your chooseFunc as follows:
def chooseFunc(i, x, y):
res = False;
for ind, op in enumerate(op_list):
res = If(ind == i, op (x, y), res)
return res
See if that improves run-times in any meaningful way. If not, the next thing to do would be to get rid of arrays as much as possible.

Python speed up large nested array processing

I'm wondering if there is a faster way to do this.
"""
Structure
-data[]
-data[0]
-data[number, number, number, number, number, number, number]
- ... ect X 12000
-data[1]
-data[number, number, number, number, number, number, number]
- ... ect X 12000
-data[2]
-data[number, number, number, number, number, number, number]
- ... ect X 12000
-data[3]
-data[number, number, number, number, number, number, number]
- ... ect X 12000
x and y are the first two numbers in each data array.
"""
I need to scan each item in layers 1,2,3 against each item in the first layer (0) looking to see if they fall within a given search radius. This takes a while.
for i in range (len(data[0])):
x = data[0][i][0]
y = data[0][i][1]
for x in range (len(data[1])):
x1 = data[1][x][0]
y1 = data[1][x][1]
if( math.pow((x1 -x),2) + math.pow((y1 - y),2) < somevalue):
matches1.append(data[0][i])
matches2.append(data[1][x])
continue
else:
continue
Thanks for any assistance!

First you should write more readable python code:
for x,y in data[0]:
for x1, y1 in data[1]:
if (x1 - x)**2 + (y1 - y)**2 < somevalue:
matches1.append((x,y))
matches2.append((x1,y1))
The you can vectorize the inner loop with numpy:
for x,y in data[0]:
x1, y1 = data[1].T
indices = (x1 - x)**2 + (y1 - y)**2 < somevalue
matches.append(((x,y), data[1][indices]))

For this specific problem scipy.spatial.KDTree or rather its Cython workalike scipy.spatial.cKDTree would appear to be taylor-made:
import numpy as np
from scipy.spatial import cKDTree
# create some random data
data = np.random.random((4, 12000, 7))
# in each record discard all but x and y
data_xy = data[..., :2]
# build trees
trees = [cKDTree(d) for d in data_xy]
somevalue = 0.001
# find all close pairs between reference layer and other layers
pairs = []
for tree in trees[1:]:
pairs.append(trees[0].query_ball_tree(tree, np.sqrt(somevalue)))
This example takes less than a second. Please note that the output format is different to the one your script produces. For each of the three non-reference layers it is a list of lists, where the inner list at index k contains the indices of the points that are close to point k in the reference list.

I would suggest creating a function out of this and using the numba libray with decorator #jit(nopython=True).
also as suggested you should use numpy arrays as numba is focusing on utilizing numpy operations.
from numba import jit
#jit(nopython=True)
def search(data):
matches1 = []
matches2 = []
for i in range (len(data[0])):
x = data[0][i][0]
y = data[0][i][1]
for x in range (len(data1[1])):
x1 = data[1][x][0]
y1 = data[1][x][1]
if( math.pow((x1 -x),2) + math.pow((y1 - y),2) < somevalue):
matches1.append(data[0][i])
matches2.append(data[1][x])
continue
else:
continue
return matches1, matches2
if __name__ == '__main__':
# Initialize
# import your data however.
m1, m2 = search(data)
The key is to make sure to only use the allowed functions supported by numba.
I have seen speed increases from 100x faster to ~300x faster.

This could also be a good place to use GPGPU computation. From python you have pycuda and pyopencl depending on your underlying hardware. Opencl can also use some of the SIMD instructions on the CPU if you don't have a gpu.
If you don't want to go down the GPGPU road then numpy or numba would also be useful as mentioned before.