We Keep Coding

Convert string representation of a comparison formula back to formula

Python newbie here.
I have a library function that is expecting a formula object;
solver.Add(x + y < 5)
However my formula is dynamic and being provided from another system as a string "x + y < 5"
The library function doesn't not accept a string, my question is there a way to create a function that can evaluate my string and return the formula as an object? eg
solver.Add(evaluateFormula("x + y < 5"))
or
solver.Add(evaluateFormula("x + y") < 5)
Using the built in eval() does not work for this type of formula since that tries to execute the formula which is not what I want to do.
Here's some full sample code;
from ortools.linear_solver import pywraplp
def main():
# Create the MIP solver BOP/SAT/CBC
solver = pywraplp.Solver.CreateSolver('BOP')
# Sample Input Data
req_json = {
"Variables": [
{
"Name": "x",
"Max": 10
},
{
"Name": "y",
"Max": 5
}
],
"Constraints": [
{
"Name": "c",
"Formula": "x + y < 5"
}
]
}
# Create the variables
v = {}
for i in range(len(req_json['Variables'])):
v[i] = solver.IntVar(0, req_json['Variables'][i]['Max'], req_json['Variables'][i]['Name'])
# Create the constraints
solver.Add(v[0] + v[1] < 5)
# solver.Add(evaluateFormula(req_json['Constraints'][0]['Formula']))
# Maximize
solver.Maximize(v[0] + v[1])
# Solve
status = solver.Solve()
if status == pywraplp.Solver.OPTIMAL:
print('Solution:')
print('Objective value =', solver.Objective().Value())
print('x =', v[0].solution_value())
print('y =', v[1].solution_value())

The idea is that x and y are variables, e.g.,
x = solver.IntVar(0, xmax, 'x')
y = solver.IntVar(0, ymax, 'y')
so that when you call solver.Add(x + y < 5), python will look up the names x and y and substitute them for the objects they are. That is, IntVar + IntVar < 5, which will be a constraint object.
When you do eval('x + y < 5'), it's as if you're doing x + y < 5, so you just need to ensure that the names x and y exist in the current scope. There are two ways of achieving that, namely
var = req_json['Variables'][0]
exec('{0} = IntVar(0, {1}, "{0}")'.format(var['Name'], var['Max'])) # first
locals()[var['Name']] = IntVar(0, var['Max'], var['Name']) # second
The first one creates the string 'x = IntVar(0, 10, "x")', which it executes as a literal python statement. While the second one creates the IntVar programmatically and then stores it in the name x. locals()['x'] = 1 is the equivalent of x = 1.
All in all, the solution could be
# You don't need to manually store the variables, as they're added in `solver.variables()`
for var in req_json['Variables']:
name = var['Name']
locals()[name] = solver.IntVar(0, var['Max'], name)
for constraint in req_json['Constraints']:
solver.Add(eval(constraint['Formula']), constraint['Name'])
# However you decide what the expression is to maximise. Whether it's another key
# in your `req_json` dict, or the LHS of the constraint formula. I'll just hardcode this.
solver.Maximize(x + y)
status = solver.Solve()
if status == pywraplp.Solver.OPTIMAL:
print('Solution:')
print('Objective value =', solver.Objective().Value())
for v in solver.variables():
print('{} = {}'.format(v.name(), v.solution_value()))
THIS ASSUMES THAT eval(constraint['Formula']) WILL NEVER DO ANYTHING MALICIOUS, which you say is your case. If you can't guarantee that, your other option is to parse the string manually for the variable names, operations and relations and build a safe string which can then be evaluated.
Finally, if you run this as is, you'll get an error saying
ValueError: Operators "<" and ">" not supported with the linear solver
But if you change the constraint formula to 'x + y <= 5', it'll work just fine.

Follow up question: GEKKO optimization in matrix form

this is a follow up question to the one I posted earlier:
GEKKO - optimization in matrix form
I need to add one more constraint that tracks "inventory" ("Inv"), which tracks the sum(q[i,:] - q[:,i]). "Inv" will be a 4X1 column vector. I tried the following:
m = GEKKO(remote=False)
q = m.Array(m.Var,(4,4),lb=0,ub=10)
for i in range(4):
for j in range(4):
if j<=i:
q[i,j].upper=0 # set upper bound = 0
def profit(q):
profit = np.sum(q.flatten() * pmx.flatten())
return profit
Inv[0]=0
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
m.Equation(I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i]))) # New Line 1a inventory
Inv[i] = Inv[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])) # New Line 1b inventory. Keep either 1a or 1b
m.Equation(Inv[i] <= 15) # New Line 2 inventory constraint
m.Equation(Inv[4] = 0) # New Line 3 ending inventory should be equal to starting inventory
m.Maximize(profit(q))
m.solve()
print(q)
qr = np.array([[q[i,j].value[0] for j in range(4)] for i in range(4)])
Ir = np.array([Inv[i].value[0] for i in range(4)]) #New Line 4
Errors :
1a. Adding New Line 1a: "keyword can't be an expression"
1b. Replacing New Line 1a with 1b: no issues (but, I'm not sure if GEKKO will keep track of I or not.Also, I need to define "I", the way "q" was done...not sure how). Replacing = comment out 1a, then run the code with 1b.
New Line 2: Error = "object of type 'int' has no len()"; but type(I) shows as ndarray. (Kept New Lines 1a and 1b, and then added New Line 2)
New Line 3:Error = "keyword can't be an expression" (Kept Line 32and then added Line 3)
New Line 4: Error "'numpy.ndarray' object has no attribute 'value'" [removed Lines 3 and 4. This makes sense as if I can't capture "Inv" in the model, then it won't have the value attribute)
Questions:
1. Am I defining inventory correctly?
If yes, can it be done in the current model specification, or will it need an entirely different formulation? If so, could you please guide on what that would be?
Of the various videos posted on the GEKKO website, is there a specific one I should look at for more information? I was thinking the DO video, but I don't believe this is quite a dynamic optimization problem (as I'm not trying to optimize the best path)....
Thanks again for all your help,
----UPDATE 5/10
Also tried:
Inv = m.SV()
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
#m.Equation(I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])))
m.Equation(m.Inv.dt() == m.Inv + (np.sum(q[i,:]) - np.sum(q[:,i])))
#I[i] = I[i-1] + (np.sum(q[i,:]) - np.sum(q[:,i])
m.Equation(m.Inv <= 15)
#m.Equation(I[4] = 0)
m.Maximize(profit(q))
New Error: 'GEKKO' object has no attribute 'Inv'

One way to do this is to start with zero inventory with Inv[0]=0 and then track the inventory amount with gekko variables Inv[1:4]. A couple tips on building the model:
Use double equal signs for equality constraints
You can define a temporary variable such as I = Inv[2]+Inv[3] but it won't be a gekko variable
You may also want to look at Intermediate variables for those that are explicitly calculated. This can speed up the calculation.
I recommend this tutorial in the Dynamic Optimization course
import numpy as np
import scipy.optimize as opt
from gekko import GEKKO
p= np.array([4, 5, 6.65, 12]) #p = prices
pmx = np.triu(p - p[:, np.newaxis]) #pmx = price matrix, upper triangular
m = GEKKO(remote=False)
q = m.Array(m.Var,(4,4),lb=0,ub=10)
# only upper triangular can change
for i in range(4):
for j in range(4):
if j<=i:
q[i,j].upper=0 # set upper bound = 0
def profit(q):
profit = np.sum(q.flatten() * pmx.flatten())
return profit
Inv = m.Array(m.Var,5,lb=0,ub=15)
Inv[0].upper = 0 # start with 0 inventory
for i in range(4):
m.Equation(np.sum(q[i,:])<=10)
m.Equation(np.sum(q[:,i])<=8)
# track inventory
m.Equation(Inv[i+1]==Inv[i] + (m.sum(q[i,:])-m.sum(q[:,i])))
m.Equation(Inv[4] == 0) # Use double == sign, not needed in loop
m.Maximize(profit(q))
m.solve()
print(q)
# convert to matrix form
qr = np.array([[q[i,j].value[0] for j in range(4)] for i in range(4)])
for i in range(4):
rs = qr[i,:].sum()
print('Row sum ' + str(i) + ' = ' + str(rs))
cs = qr[:,i].sum()
print('Col sum ' + str(i) + ' = ' + str(cs))
Ir = np.array([Inv[i].value[0] for i in range(4)])
print(Ir)

Why is CVXPy or DMCP returning a "NoneType" when simple constraints are added?

I recently ran into an interesting package/paper called dmcp (disciplined multi-convex programming). You can find a link to the repo here and a link to the paper here.
I wanted to use this to solve for equilibria in a model I have been studying in my research (the problem is actually convex, but I've been struggling to get it into DCP form so I figured this would be nice to have as a back up).
We have proved that equilibria always exist and I have been testing the code below on a toy example that I have solved by hand, but for some reason DMCP fails after a few iterations since one of the CVX variable.value is becoming a NoneType (I think).
After looking through the DMCP repo's code, I believe that the error is happening somewhere when they use CVXPy but I don't know enough to figure out whether the error is in DMCP or in CVXPy.
The code I want to get working has parameters:
num_agents = 2
num_goods = 2
adjacency_matrix = np.eye(2)
endowment = np.eye(2)
utilities = np.abs(np.eye(2)-1)
resale = np.ones(num_agents)
Variables:
prices = cp.Variable((num_agents,num_goods), nonneg=True)
consumption = cp.Variable((num_agents*num_agents,num_goods), nonneg=True)
resale = cp.Variable((num_agents*num_agents,num_goods), nonneg=True)
sum_x_ij = cp.Variable((num_agents,num_goods), nonneg=True)
sum_x_ji = cp.Variable((num_agents,num_goods), nonneg=True)
sum_y_ij = cp.Variable((num_agents,num_goods), nonneg=True)
sum_y_ji = cp.Variable((num_agents,num_goods), nonneg=True)
Initial values:
prices.value = np.random.uniform(low=.1,high=1.1,size=(num_agents,num_goods))
consumption.value = np.ones((num_agents*num_agents,num_goods))
resale.value = np.ones((num_agents*num_agents,num_goods))
A helper function:
def map_to_idx(agent,neighbor):
return (agent*num_agents)+neighbor
Objective:
obj = 0#cp.sum(cp.multiply(utilities[:,:-1],sum_x_ij)) #+ cp.sum(cp.sum(cp.multiply(prices,sum_y_ij),axis=1))
for agent in range(num_agents):
if utilities[agent,-1] != 1:
obj += cp.pnorm(cp.multiply(cp.power(utilities[agent,:-1],1.0/utilities[agent,-1]),sum_x_ij[agent,:]),utilities[agent,-1])
else:
obj += utilities[agent,:-1]*sum_x_ij[agent,:]
obj += prices[agent,:]*sum_y_ij[agent,:]
for neighbor in range(num_agents):
obj -= prices[neighbor,:]*resale[map_to_idx(agent,neighbor),:]
Constraints:
constraints = [prices >= 0.1, consumption >= 0, resale >= 0, sum_x_ij >= 0, sum_x_ji >= 0, sum_y_ij >= 0, sum_y_ji >= 0]
for agent in range(num_agents):
constraints += [sum_x_ji[agent,:] + sum_y_ji[agent,:] == endowments[agent,:] + sum_y_ij[agent,:]]
constraints += [cp.sum(sum_y_ij[agent,:]) <= resale_vect[agent]]
lhs_budget_const = 0
rhs_budget_const = prices[agent,:]*endowments[agent,:] + prices[agent,:]*sum_y_ij[agent,:]
lhs_sum_xij = 0
lhs_sum_yij = 0
lhs_sum_xji = 0
lhs_sum_yji = 0
for neighbor in range(num_agents):
lhs_budget_const += prices[neighbor,:]*consumption[map_to_idx(agent,neighbor),:]
rhs_budget_const -= prices[neighbor,:]*resale[map_to_idx(agent,neighbor),:]
if adjacency_matrix[agent,neighbor] == 0:
constraints += [consumption[map_to_idx(agent,neighbor),:] == 0]
constraints += [resale[map_to_idx(agent,neighbor),:] == 0]
lhs_sum_xij += consumption[map_to_idx(agent,neighbor),:]
lhs_sum_yij += resale[map_to_idx(agent,neighbor),:]
lhs_sum_xji += consumption[map_to_idx(neighbor,agent),:]
lhs_sum_yji += resale[map_to_idx(neighbor,agent),:]
constraints += [lhs_budget_const == rhs_budget_const]
constraints += [lhs_sum_xij == sum_x_ij[agent,:]]
constraints += [lhs_sum_xji == sum_x_ji[agent,:]]
constraints += [lhs_sum_yij == sum_y_ij[agent,:]]
constraints += [lhs_sum_yji == sum_y_ji[agent,:]]
constraints += [resale[map_to_idx(agent,agent),:] == 0]
constraints += [cp.sum(prices) == 1]
constraints += [cp.sum(consumption,axis=0) == cp.sum(endowments,axis=0)]
and finally:
prob = cp.Problem(cp.Maximize(obj), constraints)
print()
print("minimal sets:", dmcp.find_minimal_sets(prob))
print("problem is DMCP:", dmcp.is_dmcp(prob))
print()
result = prob.solve(method='bcd',max_iter = 100)
print(result)
print()
print("p = " + str(prices.value))
print("x = " + str(sum_x_ij.value))
print("y = " + str(sum_y_ij.value))
I know that this program has a solution. When I comment out the part of the objective where I am multiplying prices with sum_y_ij and resale, then I things run smoothly. When that is present in the objective I get the following output/errors:
Defining variables
Setting some initial guesses
Defining objective
Defining constraints
minimal sets: [[0], [1, 2, 5]]
problem is DMCP: True
max abs slack = 11.257764306622317 mu = 0.005 original objective value = 41.569236735359254 fixed objective value = 20.693015614921645 status= optimal
max abs slack = 21.155476427136296 mu = 0.005 original objective value = 81.25842921844962 fixed objective value = 60.55171440353868 status= optimal
max abs slack = 30.996381672274627 mu = 0.0075 original objective value = 120.77145211345528 fixed objective value = 99.41459375605088 status= optimal
max abs slack = 40.85265643546646 mu = 0.0075 original objective value = 160.97795623707566 fixed objective value = 138.83652995015788 status= optimal
max abs slack = 50.630481600263245 mu = 0.01125 original objective value = 200.88955474035853 fixed objective value = 177.10923936689406 status= optimal
max abs slack = 200.65501019663174 mu = 0.01125 original objective value = 442.07817550005626 fixed objective value = 319.96444147471675 status= optimal
max abs slack = 388.4237602479294 mu = 0.016875 original objective value = 671.3475469742531 fixed objective value = 540.2441656255529 status= optimal
max abs slack = 53335.2974424018 mu = 0.016875 original objective value = 54331.04746214825 fixed objective value = 35106.94351475132 status= optimal_inaccurate
max abs slack = 215936.17724686957 mu = 0.0253125 original objective value = 219254.45219690344 fixed objective value = 40873.564352143076 status= optimal_inaccurate
max abs slack = 27527746.923883904 mu = 0.0253125 original objective value = 23405815.216764793 fixed objective value = 7009847.781686075 status= optimal_inaccurate
Traceback (most recent call last):
File "driver.py", line 64, in <module>
solv.find_multi_convex_equilibrium(num_agents, num_goods, adjacency_matrix, endowments, utilities, resale_vect, initial_guess=None, verbose=True)
File "/home/gabe/Graphical_Econs_with_Resale/Jain_Convex_Program/solver.py", line 123, in find_multi_convex_equilibrium
result = prob.solve(method='bcd',max_iter = 100)
File "/usr/local/lib/python3.5/dist-packages/cvxpy/problems/problem.py", line 289, in solve
return solve_func(self, *args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/dmcp-1.0.0-py3.5.egg/dmcp/bcd.py", line 72, in bcd
result = _bcd(prob, fix_sets, max_iter, solver, mu, rho, mu_max, ep, lambd, linearize, proximal)
File "/usr/local/lib/python3.5/dist-packages/dmcp-1.0.0-py3.5.egg/dmcp/bcd.py", line 119, in _bcd
max_slack = np.max([np.max(np.abs(var.value)) for var in var_slack])
File "/usr/local/lib/python3.5/dist-packages/dmcp-1.0.0-py3.5.egg/dmcp/bcd.py", line 119, in <listcomp>
max_slack = np.max([np.max(np.abs(var.value)) for var in var_slack])
TypeError: bad operand type for abs(): 'NoneType'
This error even happens if I even include a single prices[0,0]*resale[0,0] in the objective.
Any advice would be super helpful! Thank you!

If I take a user supplied value as a function parameter, how do I make it a global variable?

I'm writing a program that can riffle shuffle a given sequence (list l), m times. My function takes in the list l and the number m as inputs but Ive defined the shuffle itself for one shuffle and then used a for loop to do it m times. However, now the for loop does not take the user assigned value of m.
I'm a Python noob so it's likely I'm missing a simple thing. Ive tried using global m, to (re)define m within my function but either I dont know how to do it, or it doesn't seem to work.
def riffle_shuffle(l, global m):
#global m goes here?
r = np.random.binomial(len(l),0.5)
sd1 = l[:r]
d2 = l[r:]
fl = []
c = [sd2,sd1]
l2 = sd2+sd1
for i in range(1,len(l) + 1):
x = [sd2,sd1]
y = [(len(sd2))/(len(l) - i+1),(len(sd1))/(len(l) - i+1)]
a = choices(x,y)
a1 = a[0][0]
fl.append(a1)
#Deck Split is c
#Sub decks are',c
#Probabilities are',y
#Deck chosen is',a
#fl
if a1 in sd1:
sd1.remove(a1)
elif a1 in sd2:
sd2.remove(a1)
return fl,m
for j in range(1,m+1):
fl = riffle_shuffle(fl)
return fl
I've gotten errors that say m is not defined, invalid syntax, the following error message. I don't know what this last one means.
'maximum recursion depth exceeded in comparison'
Any help is much appreciated, thanks!
EDIT: I missed the for loop I'd mentioned in the description. It's up now sorry.

So... You want method that do riffle shuffle m times, right?
There is some problems with your code :
First, return is outside of the function.
Second, you call your function in your function, without breaking condition : So the function will call the function, and that will call the function again, and again, and so on, until error occurs. That is maximum recursion depth exceeded in comparison.
Third, you have use np.random.choice like this : np.random.choice(x, p=y). Otherwise, python don't know y is probabilities, and it will interpret it as second argument : size of the output. So error occurs here.
This might be the code you want to write :
import numpy as np
def riffle_shuffle(l, m):
if m == 0:
return l
else:
fl = []
r = np.random.binomial(len(l), 0.5)
sd1 = l[:r]
sd2 = l[r:]
c = [sd2,sd1]
l2 = sd2+sd1
for i in range(1,len(l) + 1):
x = [sd2,sd1]
y = [(len(sd2))/(len(l) - i+1), (len(sd1))/(len(l) - i+1)]
a = np.random.choice(2, p=y)
a = x[a]
a1 = a[0]
fl.append(a1)
#Deck Split is c
#Sub decks are',c
#Probabilities are',y
#Deck chosen is',a
#fl
if a1 in sd1:
sd1.remove(a1)
elif a1 in sd2:
sd2.remove(a1)
fl = riffle_shuffle(fl, m - 1)
return fl
a = riffle_shuffle([1, 2, 3, 4, 5, 6, 7, 8], 3)
print(a)
#output : [5, 6, 1, 7, 4, 8, 2, 3] (can be changed)
As you did, I called the function 'recursively' - call function in function - with break condition.
In this way, you don't have to use global variable - using global variable is not good idea in most situation.
And, about your question(How can I make user-supplied value to global variable), you can do something like this.
a = 0
def foo(m):
global a
a = m
#and your code here...

Use concentration of a blending as a constraint on a linear optimization

I have the following table, from which a have to create a recipe with a specific value for protein and carbohydrates.
And using or-tools to solve this problem, so far I have:
The formated data
data = [
['f1', 10, 15, 17, 10],
['f2', 2, 11, 12, 14],
['f3', 6.5, 17, 16, 13],
['f4', 8, 12, 8, 16]
]
The constraines for the nutriends:
nutrients = [
["protein",15.5],
["carbohydrates",12.3]]
The objective function, where the upper bound "datai" is the stock of that particular element.
food = [[]] * len(data)
# Objective: minimize the sum of (price-normalized) foods.
objective = solver.Objective()
for i in range(0, len(data)):
food[i] = solver.NumVar(0.0, data[i][1], data[i][0])
objective.SetCoefficient(food[i], 4)
objective.SetMinimization()
I also have the as constrain the required value of each nutrient:
constraints = [0] * (len(nutrients))
for i in range(0, len(nutrients)):
constraints[i] = solver.Constraint(nutrients[i][1], solver.infinity())
for j in range(0, len(data)):
constraints[i].SetCoefficient(food[j], data[j][i+3])
And finally the solver:
status = solver.Solve()
if status == solver.OPTIMAL:
# Display the amounts (in dollars) to purchase of each food.
price = 0
num_nutrients = len(data[i]) - 3
nutrients = [0] * (len(data[i]) - 3)
for i in range(0, len(data)):
price += food[i].solution_value()
for nutrient in range(0, num_nutrients):
nutrients[nutrient] += data[i][nutrient+3] * food[i].solution_value()
if food[i].solution_value() > 0:
print ("%s = %f" % (data[i][0], food[i].solution_value()))
print ('Optimal price: $%.2f' % (price))
else: # No optimal solution was found.
if status == solver.FEASIBLE:
print ('A potentially suboptimal solution was found.')
else:
print ('The solver could not solve the problem.')
Which up to this part is working fine, the result I get is the following:
f1 = 0.077049
f3 = 0.886885
Optimal price: $0.96
Know I need to add as well the constraints of how many kg I will make, which have to satisfy the previous constraints as well.
My first guess was to add a multiplier to the nutrients requirement
factor = 10
nutrients = [
["protein",15.5*factor],
["carbohydrates",12.3*factor]]
Tjis way I will have 10 times more food, but then I realized that this is not correct, since what I need is a concentración E.G.
I need 10kg with 15.5 protein/kg and 12.3 carbohydrates/kg
the constrain I need is something like this:
(f1*W + f2*X + f3*Y + f4*Z)/(W+X+Y+Z) = 10kg with 15.5 protein/kg and 12.3 carbohydrates/kg
Where W, X, Y and Z are the kg of each food
How can I add this constrain to solver?

(f1*W + f2*X + f3*Y + f4*Z)/(W+X+Y+Z) = 10
is the same as
f1*W + f2*X + f3*Y + f4*Z = 10*(W+X+Y+Z)
This is now linear.
And, in case we missed some math classes, we can write this as a standard LP constraint:
(f1-10)*W + (f2-10)*X + (f3-10)*Y + (f4-10)*Z = 0