I am trying to solve a blending problem with a system of 3 equations and I have 3 objectives to reach for, or try to get the values as close as posible for the three of them:
The equations are:
def sat (c,s,a,f):
return (100*c)/(2.8*s+1.18*a+0.65*f) #For this I need sat = 98.5
def ms (s,a,f):
return s/(a+f) #For this I need ms = 2.5
def ma (a,f):
return (a/f) #For this I need ms = 1.3
#The total mix ratio:
r1+r2+r3+r4+r5+r6 = 1
material_1:
c = 51.29
s = 4.16
a = 0.97
f = 0.38
material_2:
c = 51.42
s = 4.16
a = 0.95
f = 0.37
material_3:
c = 6.88
s = 63.36
a = 13.58
f = 3.06
material_4:
c = 32.05
s = 1.94
a = 0.0
f = 0.0
material_5:
c = 4.56
s = 21.43
a = 3.82
f = 52.28
material_6:
c = 0.19
s = 7.45
a = 4.58
f = 0.42
#The aproximate values I am trying to find are around:
0.300 <= r1 <= 0.370
0.300 <= r2 <= 0.370
0.070 <= r3 <= 0.130
0.005 <= r4 <= 0.015
0.010 <= r5 <= 0.030
0.110 <= r6 <= 0.130
So how can I calculate the value for every ratio "r" in order to get the closets values to the objectives for the 3 equations?
I looked on some optimizers but as I am new with them I still can not understand how to set up the problem, the equations and constraints into them.
I guess I made it, of course the code is awful but I will try to make it look better later.
I added the cost of the components so I can give a function to "minimize", of course this is becouse I know the aproximated material ratio so it guide the solver to it.
I will post the code for it:
c1 = 51.42
c2 = 51.42
c3 = 6.88
c5 = 32.05
c6 = 4.56
c7 = 0.19
s1 = 4.16
s2 = 4.16
s3 = 63.36
s5 = 1.94
s6 = 21.43
s7 = 7.45
a1 = 0.97
a2 = 0.95
a3 = 13.58
a5 = 0.0
a6 = 3.82
a7 = 4.58
f1 = 0.38
f2 = 0.37
f3 = 3.06
f5 = 0.0
f6 = 52.28
f7 = 0.42
r7 = 0.125
r1 = cp.Variable()
r2 = cp.Variable()
r3 = cp.Variable()
r5 = cp.Variable()
r6 = cp.Variable()
#Costos
caliza = 10
arcilla = 20
hierro = 170
yeso = 80
objective = cp.Minimize(r1*caliza+r2*caliza+r3*arcilla+r5*yeso+r6*hierro)
constraints = [
r1-r2 == 0,
r1>= 0.20,
r1<= 0.40,
r3<=0.14,
r3>=0.06,
r5>=0.001,
r5<=0.008,
r6>=0.01,
r6<=0.03,
2.5*((r1*a1+r2*a2+r3*a3+r5*a5+r6*a6+r7*a7)+(f1*r1+f2*r2+f3*r3+f5*r5+f6*r6+f7*r7))-(r1*s1+r2*s2+r3*s3+r5*s5+r6*s6+r7*s7)==0,
(98.5*(2.8*(r1*s1+r2*s2+r3*s3+r5*s5+r6*s6+r7*s7)+1.18*(r1*a1+r2*a2+r3*a3+r5*a5+r6*a6+r7*a7)+0.65*(f1*r1+f2*r2+f3*r3+f5*r5+f6*r6+f7*r7))-100*(r1*c1+r2*c2+r3*c3+r5*c5+r6*c6+r7*c7)) == 0,
#1.3*(f1*r1+f2*r2+f3*r3+f5*r5+f6*r6+f7*r7)-(r1*a1+r2*a2+r3*a3+r5*a5+r6*a6+r7*a7) == 0,
r1+r2+r3+r5+r6+r7 == 1]
problem = cp.Problem(objective,constraints)
problem.solve()
print(r1.value,r2.value,r3.value,r5.value,r6.value)
print(problem.status)
This gives me the result:
0.3644382497863931 0.3644382497863931 0.12287226775076901 0.0009999999955268117 0.022251232680917873
optimal
Anyways, the only way to make a feasible result is to only consider 2 of the three constraints functions, becouse the components cant reach the 3 of them and this indicates that I need to check de material components before I try to reach the 3 constraints (wich were the sat, ma and ms).
Now I will try to make the code better using pandas so i can get the material components with somekind of for loop and laso use it for the ratios.
Thank you so much for your help👍.
So this is a simple/trivial example to show the intent that was mentioned in comment to minimize the square of errors... Instead of using a constraint to pin a value to an exact outcome, we let the solver find the best outcome that minimizes the square of the error where error = value - target. I think what I've written below is fairly clear. CVXPY likes to work in the linear algebra realm, and I'm sure this could be converted into vector / matrix format, but the concept is to remove constraints and let the solver figure out the best combo. Obviously, if there are hard constraints, those need to be added, but note that I've just made an example with 2 of your 3 targets (with some trivial math) and moved it into the objective.
Your problem with 3 constraints that aren't simultaneously satisfiable is probably a candidate for a conversion like this...
import cvxpy as cp
r1 = cp.Variable()
r2 = cp.Variable()
ma = 2.5
ms = 3.4
delta_1 = (r1 + r2 - ma)**2 # diff from r1 + r2 and ma
delta_2 = (3*r1 + 2*r2 - ms)**2 # diff from 3r1 + 2r2 and ms
prob = cp.Problem(cp.Minimize(delta_1 + delta_2))
prob.solve()
print(prob.value)
print(r1.value, r2.value)
Output
9.860761315262648e-31
-1.6000000000000014 4.100000000000002
Ok this is what i have done and works fine:
#I call the values from a pandas DF:
c1 = df.at[0, 'MAX']
c2 = df.at[4, 'MAX']
c3 = df.at[8, 'MAX']
c5 = df.at[12, 'MAX']
c6 = df.at[16, 'MAX']
c7 = df.at[20, 'MAX']
s1 = df.at[1, 'MAX']
s2 = df.at[5, 'MAX']
s3 = df.at[9, 'MAX']
s5 = df.at[13, 'MAX']
s6 = df.at[17, 'MAX']
s7 = df.at[21, 'MAX']
a1 = df.at[2, 'MAX']
a2 = df.at[6, 'MAX']
a3 = df.at[10, 'MAX']
a5 = df.at[14, 'MAX']
a6 = df.at[18, 'MAX']
a7 = df.at[22, 'MAX']
f1 = df.at[3, 'MAX']
f2 = df.at[7, 'MAX']
f3 = df.at[11, 'MAX']
f5 = df.at[15, 'MAX']
f6 = df.at[19, 'MAX']
f7 = df.at[23, 'MAX']
r1 = cp.Variable()
r2 = cp.Variable()
r3 = cp.Variable()
r5 = cp.Variable()
r6 = cp.Variable()
r7 = 12.5
#Objectives
ma = 1.3
ms = 2.50
lsf = 98.5
delta1 =(ms*((r1*a1+r2*a2+r3*a3+r5*a5+r6*a6+r7*a7)+(f1*r1+f2*r2+f3*r3+f5*r5+f6*r6+f7*r7))-(r1*s1+r2*s2+r3*s3+r5*s5+r6*s6+r7*s7))**2
delta2 =(ma*(f1*r1+f2*r2+f3*r3+f5*r5+f6*r6+f7*r7)-(r1*a1+r2*a2+r3*a3+r5*a5+r6*a6+r7*a7))**2
delta3 =((lsf*(2.8*(r1*s1+r2*s2+r3*s3+r5*s5+r6*s6+r7*s7)+1.18*(r1*a1+r2*a2+r3*a3+r5*a5+r6*a6+r7*a7)+0.65*(f1*r1+f2*r2+f3*r3+f5*r5+f6*r6+f7*r7))-100*(r1*c1+r2*c2+r3*c3+r5*c5+r6*c6+r7*c7)))**2
objective = cp.Minimize(delta1+delta2+delta3)
constraints = [r1-r2 == 0, #I added this to make r1=r2.
r1>= 0.20,
r3>=0, #I added these to make it non negative.
r5>=0,
r5<=0.008,
r6>=0,
r1+r2+r3+r5+r6+r7 == 1]
problem = cp.Problem(objective,constraints)
problem.solve()
print(r1.value,r2.value,r3.value,r5.value,r6.value)
print(problem.status)
Once again i want to thank you for your help guys.
Maybe you know how I can improve the code for get the variable values, maybe there is and example of using a for loop to get the values instead of put it directly from the DF for each one, the DF looks like this:
DATO MAX
0 c1 51.95000
1 s1 3.07000
2 a1 0.83000
3 f1 0.31000
4 c2 52.26000
5 s2 2.82000
6 a2 0.75000
...
Related
I am trying to get the mode of models at column level and the find the error associated with that model
if two or more more modes are received then we select the model with least error out of these two modes
import pandas as pd
data1 = {'Iteration1': ["M2",'M1',"M3","M5","M4","M6"],
'Iteration1_error': [96,98,34,19,22,9],
'Iteration2': ["M3",'M1',"M1","M5","M6","M4"],
'Iteration2_error': [76,88,54,12,92,19],
'Iteration3': ["M3",'M1',"M1","M5","M6","M4"],
'Iteration3_error': [66,68,84,52,72,89]}
Input1 = pd.DataFrame(data1,
columns=['Iteration1','Iteration1_error','Iteration2','Iteration2_error','Iteration3','Iteration3_error'],
index=['I1', 'I2','I3','I4','I5','I6'])
print(Input1)
data2 = {'Iteration1': ["M2",'M1',"M3","M5","M4","M6"],
'Iteration1_error': [96,98,34,19,22,9],
'Iteration2': ["M3",'M1',"M1","M5","M6","M4"],
'Iteration2_error': [76,88,54,12,92,19],
'Iteration3': ["M3",'M1',"M1","M5","M6","M4"],
'Iteration3_error': [66,68,84,52,72,89],
'Mode of model name in all iterations':['M3','M1','M1','M5','M6','M4'],
'Best model error':[66,68,54,12,72,19]
}
Output1 = pd.DataFrame(data2,
columns=['Iteration1','Iteration1_error','Iteration2','Iteration2_error','Iteration3','Iteration3_error','Mode of model name in all iterations','Best model error'],
index=['I1', 'I2','I3','I4','I5','I6'])
print(Output1)
Question: So we are expecting an output with two etc columns at the end , one tells us about mode at column level second tells about the error of that mode, first 6 columns are input dataframe, incase two modes or more are received example ("M1","M2","M3") all three values are different so technically it will have 3 modes so model with least accuracy will be selected
What I tried: I was able to get the mode at column level by using .mode(numeric_only=False) but what issue I am getting how can I get that modes error from 2nd, 4th and 6th column, there I am stuck at
Use:
#filter only columns by Iteration with number
df = Input1.filter(regex='Iteration\d+$')
#get first mode
s = df.mode(axis=1).iloc[:, 0]
#compare df for all possible modes, add suffix for match errors columns,
#last filter original with min
s1 = Input1.where(df.eq(s, axis=0).add_suffix('_error')).min(axis=1)
#add new columns
Output1 = Input1.assign(best_mode = s, best_error=s1)
print (Output1)
Iteration1 Iteration1_error Iteration2 Iteration2_error Iteration3 \
I1 M2 96 M3 76 M3
I2 M1 98 M1 88 M1
I3 M3 34 M1 54 M1
I4 M5 19 M5 12 M5
I5 M4 22 M6 92 M6
I6 M6 9 M4 19 M4
Iteration3_error best_mode best_error
I1 66 M3 66.0
I2 68 M1 68.0
I3 84 M1 54.0
I4 52 M5 12.0
I5 72 M6 72.0
I6 89 M4 19.0
Another idea if is possible use pair and unpairs columns (in data has to exist all pairs ech other, sorted):
df = Input1.iloc[:, ::2]
s = df.mode(axis=1).iloc[:, 0]
s1 = Input1.iloc[:, 1::2].where(df.eq(s, axis=0).to_numpy()).min(axis=1)
Output1 = Input1.assign(best_mode = s, best_error=s1)
I am a new Pyomo/Python user. In the process of learning pyomo, I need to solve some optimization problems.
I need to make a balance between the total generation and demand (800 MW).
If I want to find the minimum value of 'Sum(ap^2+bp+c))'.And this is the mathematic model:mathematics model of this problem
How can I construct an abstract model that I can choose the value of a, b, c from the same row in the table below? If I set the 'sets' individually, then the value of abc will come from the different row which cannot satisfy the formula. And how do I set a random value between the Pmin and Pmax? Use two constraints to limit the value?
It really makes me confused.
from pyomo.environ import *
import matplotlib.pyplot as plt
import numpy as np
import matplotlib.pyplot as plt
import random
model = AbstractModel()
model.J = Set()
model.A = Param(model.J)
model.B = Param(model.J)
model.C = Param(model.J)
model.P_min = Param(model.J)
model.P_max= Param(model.J)
model.P = Var(model.J, domain=NonNegativeReals)
def obj_expression(model):
return sum(model.A* model.P**2+model.B*model.P+model.C for j in model.J)
model.OBJ = Objective(rule=obj_expression, sense=minimize)
# Upper bounds rule
def upper_bounds_rule(model,j):
return model.P[j] <= model.P_max[j]
model.upper = Constraint(model.J,rule=upper_bounds_rule)
# Lower bounds rule
def lower_bounds_rule(model, j):
return model.P[j] >= model.P_min[j]
model.lower = Constraint(model.J, rule=lower_bounds_rule)
def rule_eq1(model,j):
return sum(model.P[j] >= 800;
model.eq1 = Constraint(model.J, rule=rule_eq1)
opt = SolverFactory('Ipopt')
instance = model.create_instance("E:/pycharm_project/PD/pd.dat")
results = opt.solve(instance) # solves and updates instance
#data file
# Creating the set J:
set J := g1 g2 g3 g4 g5 g6 g7 g8 g9 g10 ;
# Creating Parameters A, B, C, P_min, P_max:
param : A B C P_min P_max:=
g1 0.0148 12.1 82 80 200
g2 0.0289 12.6 49 120 320
g3 0.0135 13.2 100 50 150
g4 0.0127 13.9 105 250 520
g5 0.0261 13.5 72 80 280
g6 0.0212 15.4 29 50 150
g7 0.0382 14.0 32 30 120
g8 0.0393 13.5 40 30 110
g9 0.0396 15.0 25 20 80
g10 0.0510 14.3 15 20 60
;
Can you help me with this? Thanks!
Vivi
I think you should structure your .dat files in accordance with what is described in the Pyomo documentation here.
I believe something like this would work for you for parameters A, B, C:
# Creating the set J:
set J := g1 g2 g3 g4 g5 g6 g7 g8 g9 g10 ;
# Creating Parameters A, B, C:
param : A B C :=
g1 0.0148 12.1 82
g2 0.0289 12.6 49
g3 0.0135 13.2 100
g4 0.0127 13.9 105
g5 0.0261 13.5 72
g6 0.0212 15.4 29
g7 0.0382 14.0 32
g8 0.0393 13.5 40
g9 0.0396 15.0 25
g10 0.0510 14.3 15
;
Now I am not sure about Pmin and Pmax, since in your model they seem to be 2-dimensional, while in your data, they seem to only be 1-dimensional. But generally, you can follow the instructions in the link above to create your .dat files.
As for your second question, I am not sure I understand you correctly, but you refer to the P_min <= P <= P_max constraint in the mathematical model description, correct?
Then, for the P_min <= P part you need to slightly change your current constraint:
def lower_bounds_rule(model, i, j):
return model.P[i,j] >= model.P_min[i,j]
model.lower = Constraint(model.i,model.j,rule=lower_bounds_rule)
For example
from sympy import *
v11, v12, v21, v22 = symbols('v11, v12, v21, v22')
w11, w12, w21, w22 = symbols('w11, w12, w21, w22')
x11, x12, x21, x22 = symbols('x11, x12, x21, x22')
y11, y12, y21, y22 = symbols('y11, y12, y21, y22')
z11, z12, z21, z22 = symbols('z11, z12, z21, z22')
r11, r12, r21, r22 = symbols('r11, r12, r21, r22')
a1, a2, a3, a4, a5, a6, a7, a8, a9, a10, a11, \
a12, a13, a14, a15, a16, a17, a18 = symbols('a1:19')
# want to solve
b1, b2 = symbols('b1, b2')
# system of equations
eqn1 = a1*v11 + a2*v12 - b1 # a1 = 1
eqn2 = a1*v21 + a2*v22 - a3
eqn3 = a3*r11 + a4*r12 - a2
eqn4 = a3*r21 + a4*r22 - a5
eqn5 = a5*w11 + a6*w12 - a4
eqn6 = a5*w21 + a6*w22 - a7
eqn7 = a7*r11 + a8*r12 - a6
eqn8 = a7*r21 + a8*r22 - a9
eqn9 = a9*x11 + a10*x12 - a8
eqn10 = a9*x21 + a10*x22 - a11
eqn11 = a11*r11 + a12*r12 - a10
eqn12 = a11*r21 + a12*r22 - a13
eqn13 = a13*y11 + a14*y12 - a12
eqn14 = a13*y21 + a14*y22 - a15
eqn15 = a15*r11 + a16*r12 - a14
eqn16 = a15*r21 + a16*r22 - a17
eqn17 = a17*z11 + a18*z12 - a16 # a18 = 0
eqn18 = a17*z21 + a18*z22 - b2
print('begin')
result = solve([eqn1, eqn2, eqn3, eqn4, eqn5, eqn6, eqn7, \
eqn8, eqn9, eqn10, eqn11, eqn12, eqn13, \
eqn14, eqn15, eqn16, eqn17, eqn18], \
[a2, a3, a4, a5, a6, a7, a8, a9, a10, \
a11, a12, a13, a14, a15, a16, a17, b1, b2], \
simplify=False, rational=False, manual=True)
b1_ans = result[b1]
b2_ans = result[b2]
print(b1_ans)
to solve these equations need MATLAB only in seconds, but by using sympy I cannot get the result.
Is there any way to speed up the solver? If not can you recommend another way to solve system of equations using python?
Thanks in advance
Since you have many more symbols than equations you will have to eliminate those that you are not interested in. Presumably you want to eliminate want = set(var('a1:19')) given the constraints of eqs = [Eq(a1, 1), Eq(a18, 0), eqn1, eqn2, eqn3, eqn4, eqn5, eqn6, eqn7, eqn8, eqn9, eqn10, eqn11, eqn12, eqn13, eqn14, eqn15, eqn16, eqn17, eqn18].
After recursively eliminating those (solving for a variable that appears in only 1 equation then removing that variable from want, the following relationships are obtained:
# recursively eliminate equations that have only one symbol of interest
sol = {}
while 1:
was = len(sol)
for ix, i in enumerate(eqs):
if len(i.free_symbols & want) == 1:
new = solve(i, *want, dict=True)
assert len(new) == 1 # only doing this for single-solution equations
sol.update(new[0]) # store in known values
if len(sol) == was:
break
order of equations
0 a1 - 1
1 a18
2 a1*v11 + a2*v12 - b1
3 a1*v21 + a2*v22 - a3
4 -a2 + a3*r11 + a4*r12
5 a3*r21 + a4*r22 - a5
6 -a4 + a5*w11 + a6*w12
7 a5*w21 + a6*w22 - a7
8 -a6 + a7*r11 + a8*r12
9 a7*r21 + a8*r22 - a9
10 a10*x12 - a8 + a9*x11
11 a10*x22 - a11 + a9*x21
12 -a10 + a11*r11 + a12*r12
13 a11*r21 + a12*r22 - a13
14 -a12 + a13*y11 + a14*y12
15 a13*y21 + a14*y22 - a15
16 -a14 + a15*r11 + a16*r12
17 a15*r21 + a16*r22 - a17
18 -a16 + a17*z11 + a18*z12
19 a17*z21 + a18*z22 - b2
solutions
{a1: 1, a18: 0, a2: (-a1*v11 + b1)/v12, a3: a1*v21 + a2*v22, a4: (a2 - a3*r11)/r12, a5: a3*r21 + a4*r22, a6: (a4 - a5*w11)/w12, a7: a5*w21 + a6*w22, a8: (a6 - a7*r11)/r12, a9: a7*r21 + a8*r22, a10: (a8 - a9*x11)/x12, a11: a10*x22 + a9*x21, a12: (a10 - a11*r11)/r12, a13: a11*r21 + a12*r22, a14: (a12 - a13*y11)/y12, a15: a13*y21 + a14*y22, a16: (a14 - a15*r11)/r12, a17: a15*r21 + a16*r22}
equations not solved/used
{Eq(-a16 + a17*z11 + a18*z12, 0), Eq(a17*z21 + a18*z22 - b2, 0)}
The solution for a2 can be re-arranged for b1 and the 2nd equation in the unsolved set can be solved for b2.
>>> solve(Eq(a17*z21 + a18*z22 - b2, 0), b2, dict=True)
[{b2: a17*z21 + a18*z22}]
>>> solve(s1[0][a2],b1, dict=True)
[{b1: a1*v11}]
The remaining equation represents a condition which must be satisfied for there to be a solution: Eq(-a16 + a17*z11 + a18*z12, 0).
It would be interesting to see what the result from MATLAB is.
I am trying to generate a matrix of numbers with 7 rows and 4 columns. Each row must sum to 100 and each column must have an even spread (if permitted) between a min and max range (specified below).
Goal:
C1 C2 C3 C4 sum range
1 low 100 ^
2 .. |
3 .. |
4 .. |
5 .. |
6 .. |
7 high _
c1_high = 98
c1_low = 75
c2_high = 15
c2_low = 6
c3_high = 8
c3_low = 2
c4_low = 0.05
c4_high =0.5
In addition to this, i need the spread of each row to be as linear as possible, though a line fitted to the data with a second order polynomial would suffice (with an r^2 value of >0.98).
I am currently trying to do this using the following sudocode:
generate random number between ranges for c1,c2,c3 and c4.
repeat this 7 times
check correlation between each generated c1 value and a range of numbers from 1-7. For example:
repeat step 3 for c2,c3 and c4.
Break loop when step 3 and 4 are successful
This has proven to be too burdensome in terms of the number of iterations required and as a result, the solution is never reached.
Is there a more efficient way of achieving this solution?
So far:
import pandas as pd
import numpy as np
from sklearn.utils import shuffle
c1_high = 98
c1_low = 75
c2_high = 15
c2_low = 6
c3_high = 8
c3_low = 2
c4_low = 0.05
c4_high =0.5
def matrix_gen(): #generates matrix within min and max values
container =[]
d={}
offset = np.linspace(0.05,1,9)
c1= np.linspace(c1_low, c1_high, 7)
c2= np.linspace(c2_low, c2_high, 7)
c3= np.linspace(c3_low, c3_high, 7)
c4= np.linspace(c4_low, c4_high, 7)
for i in np.arange(7):
d["row{0}".format(i)]=[item[i] for item in [c1,c2,c3,c4]]
df =pd.DataFrame(d)
df.loc[4,:] = df.iloc[0,:][::-1].values
df1 = df.drop(0)
df1.loc[5,:] = df1.sum(axis=0)
new_name = df1.index[-1]
df1 = df1.rename(index={new_name: 'sum'})
return df1
m = matrix_gen()
print(m)
out:
row0 row1 row2 row3 row4 row5 row6
1 6.00 7.500000 9.000000 10.500 12.000000 13.500000 15.0
2 2.00 3.000000 4.000000 5.000 6.000000 7.000000 8.0
3 0.05 0.125000 0.200000 0.275 0.350000 0.425000 0.5
4 98.00 94.166667 90.333333 86.500 82.666667 78.833333 75.0
sum 106.05 104.791667 103.533333 102.275 101.016667 99.758333 98.5
next function:
def shuf(): # attempts at shuffling the values around such that the 'sum' row is as close to 100 as possible.
df = matrix_gen()
df1 = df[1:4]
count =0
while True:
df1 = shuffle(df1)
df1.loc[5,:] = df1.sum(axis=0)
for i in df1.loc[5].values:
if 98<= i <=100:
print('solution')
return df1
else:
count+=1
print(count)
continue
opt = shuf()
print(opt)
next function will need to apply a deviation to each number to provide a sum of each row equal to 100. Optimization should include minimizing deviations.
I think an interesting approach would be to use an optimization model.
Ordered values
Let x(i,j) be the matrix your want to fill. Then we have:
sum(j, x(i,j)) = 100 ∀i
L(j) ≤ x(i,j) ≤ U(j) ∀i,j
x(i,j) = x(i-1,j) + step(j) + deviation(i,j)
special cases:
x(1,j) = L(j) + deviation(1,j)
and x(m,j) = U(j) + deviation(m,j)
step(j) ≥ 0
minimize sum((i,j), deviation(i,j)^2 )
This is a quadratic programming problem. It is possible to absolute deviations instead of squared ones. In that case you have an LP.
The model can be refined to minimize squared relative errors.
This is a little bit related to what is called matrix balancing (a statistical technique often used in economic modeling).
Unordered values
In the above I assumed the values had to be ordered. Now I understand this is not the case. I adapted the model to handle this as follows. First an overview of the results.
The input data is:
---- 17 PARAMETER LO
c1 80.000, c2 5.000, c3 0.500, c4 0.050
---- 17 PARAMETER UP
c1 94.000, c2 14.000, c3 5.000, c4 0.500
Warning: Note that this data has been changed by the poster. My answer is using the original LO and UP values before they were changed.
The model operates in three steps:
(1) populate a perfectly organized matrix without obeying the row sum constraints. This can be done outside the model. I generated simply:
---- 53 PARAMETER init initial matrix
c1 c2 c3 c4 rowsum
r1 80.000 5.000 0.500 0.050 85.550
r2 82.333 6.500 1.250 0.125 90.208
r3 84.667 8.000 2.000 0.200 94.867
r4 87.000 9.500 2.750 0.275 99.525
r5 89.333 11.000 3.500 0.350 104.183
r6 91.667 12.500 4.250 0.425 108.842
r7 94.000 14.000 5.000 0.500 113.500
I.e. from lo(j) to up(j) with equal steps.
(2) The second step is to permute the values within a column to achieve a solution that has a close match to the row sums. This gives:
---- 53 VARIABLE y.L after permutation
c1 c2 c3 c4 rowsum
r1 94.000 5.000 0.500 0.125 99.625
r2 82.333 12.500 4.250 0.500 99.583
r3 89.333 8.000 2.000 0.200 99.533
r4 87.000 9.500 2.750 0.275 99.525
r5 84.667 11.000 3.500 0.350 99.517
r6 91.667 6.500 1.250 0.050 99.467
r7 80.000 14.000 5.000 0.425 99.425
This is already very close and maintains "perfect" spread.
(3) Change the values a little bit by adding a deviation such that the row sums are exactly 100. Minimize the sum of the squared relative deviations. This gives:
---- 53 VARIABLE x.L final values
c1 c2 c3 c4 rowsum
r1 94.374 5.001 0.500 0.125 100.000
r2 82.747 12.503 4.250 0.500 100.000
r3 89.796 8.004 2.000 0.200 100.000
r4 87.469 9.506 2.750 0.275 100.000
r5 85.142 11.007 3.501 0.350 100.000
r6 92.189 6.510 1.251 0.050 100.000
r7 80.561 14.012 5.002 0.425 100.000
---- 53 VARIABLE d.L deviations
c1 c2 c3 c4
r1 0.374 0.001 1.459087E-5 1.459087E-7
r2 0.414 0.003 9.542419E-5 9.542419E-7
r3 0.462 0.004 2.579521E-4 2.579521E-6
r4 0.469 0.006 4.685327E-4 4.685327E-6
r5 0.475 0.007 7.297223E-4 7.297223E-6
r6 0.522 0.010 0.001 1.123123E-5
r7 0.561 0.012 0.002 1.587126E-5
Steps (2) and (3) have to be inside the optimization model: they have to be executed simultaneously to achieve proven optimal solutions.
The mathematical model can look like:
The model solves within a few seconds to proven global optimality using a solver like Cplex or Gurobi.
I think this is pretty cute model (ok, that is really nerdy, I know). The permutation is modeled with a permutation matrix P (binary values). This makes the model a MIQP (Mixed Integer Quadratic Programming) model. It can be linearized fairly easily: use absolute values instead of squares in the objective. After proper reformulation, we end up with a linear MIP model. There is lots of software available to handle this. This includes libraries and packages callable from Python.
Note: I probably should not divide by init(i,j) in the objective, but rather by the column means in the init matrix. Dividing by y(i,j) would be the best, but that leads to another non-linearity.
Your numbers are small enough for a smart brute force approach.
I use two methods to quantify and minimize deviations from the "clean" equidistant values (linspace(low, high, 7)). "abserr" for squared difference and "relerr" for squared error divided by squared clean value. I also check corrcoefs in the very end but I've never seen anything below 99.8%
The following code first finds the shuffle od the clean values with the smallest error. This takes just a few seconds, because we use the following tricks:
split the 4 columns into two pairs
each pair has 7! relative arrangements, a mangeable number even when squared (one factor for each pair)
compute these (7!)^2 shuffles and sum over pairs
to not have to iterate over all relative shuffles between the pairs we observe that the total error is minimized if the the two sets of pair sums are arranged in opposite order this is true for "abserr" and "relerr"
In the end the values are corrected to make rows sum to 100. Here again we use the fact that the summed error is minimized when evenly spread.
The code below contains two variants a legacy one solve which contains a small inaccuracy when minimizing relerr and a corrected version improved_solve. They frequently find different solutions but in more than 100 random problems only one led to a very slightly smaller error with improved_solve.
Answers to a few examples:
OP's example:
((75, 98), (6, 15), (2, 8), (0.05, 0.5))
solve relerr improved_solve relerr
table: table:
76.14213 15.22843 8.12183 0.50761 76.14213 15.22843 8.12183 0.50761
79.02431 13.53270 7.01696 0.42603 79.02431 13.53270 7.01696 0.42603
81.83468 11.87923 5.93961 0.34648 81.83468 11.87923 5.93961 0.34648
84.57590 10.26644 4.88878 0.26888 84.57590 10.26644 4.88878 0.26888
87.25048 8.69285 3.86349 0.19317 87.25048 8.69285 3.86349 0.19317
89.86083 7.15706 2.86282 0.11928 89.86083 7.15706 2.86282 0.11928
92.40924 5.65771 1.88590 0.04715 92.40924 5.65771 1.88590 0.04715
avgerr: avgerr:
0.03239 0.03239
corrcoefs: corrcoefs:
0.99977 0.99977 0.99977 0.99977 0.99977 0.99977 0.99977 0.99977
An example where sorting some colums ascending some descending is not optimal:
((11, 41), (4, 34), (37, 49), (0.01, 23.99))
Note that the solvers find different solutions, but the error is the same.
solve relerr improved_solve relerr
table: table:
10.89217 18.81374 46.53926 23.75483 11.00037 24.00080 49.00163 15.99720
26.00087 9.00030 49.00163 15.99720 16.00107 19.00127 45.00300 19.99467
31.00207 4.00027 45.00300 19.99467 25.74512 13.86276 36.63729 23.75483
16.00000 29.00000 43.00000 12.00000 35.99880 8.99970 46.99843 8.00307
20.99860 33.99773 40.99727 4.00640 41.00000 4.00000 43.00000 12.00000
40.99863 13.99953 36.99877 8.00307 20.99860 33.99773 40.99727 4.00640
36.35996 24.23998 39.38996 0.01010 31.30997 29.28997 39.38996 0.01010
avgerr: avgerr:
0.00529 0.00529
corrcoefs: corrcoefs:
0.99993 0.99994 0.99876 0.99997 0.99989 0.99994 0.99877 0.99997
This is the problem where improved_solve actually beats legacy solve:
((36.787862883725872, 43.967159949544317),
(40.522239654303483, 47.625869880574164),
(19.760537036548321, 49.183056694462799),
(45.701873101046154, 48.051424087501672))
solve relerr improved_solve relerr
table: table:
21.36407 23.53276 28.56241 26.54076 20.25226 26.21874 27.07599 26.45301
22.33545 24.52391 26.03695 27.10370 21.53733 26.33278 25.10656 27.02333
23.33149 25.54022 23.44736 27.68093 22.90176 26.45386 23.01550 27.62888
24.35314 26.58266 20.79119 28.27301 24.35314 26.58266 20.79119 28.27301
25.40141 27.65226 18.06583 28.88050 25.90005 26.71994 18.42047 28.95953
26.47734 28.75009 15.26854 29.50403 27.55225 26.86656 15.88840 29.69279
27.58205 29.87728 12.39644 30.14424 29.32086 27.02351 13.17793 30.47771
avgerr: avgerr:
0.39677 0.39630
corrcoefs: corrcoefs:
0.99975 0.99975 0.99975 0.99975 0.99847 0.99847 0.99847 0.99847
Code:
import numpy as np
import itertools
import math
N_CHUNKS = 3
def improved_solve(LH, errtype='relerr'):
N = math.factorial(7)
# accept anything that looks like a 2d array
LH = np.asanyarray(LH)
# build equidistant columns
C = np.array([np.linspace(l, h, 7) for l, h in LH])
# subtract offset; it's cheaper now than later
c0, c1, c2, c3 = C - 25
# list all permutiations of a single column
p = np.array(list(itertools.permutations(range(7))))
# split into left and right halves, compute all relative permutiations
# and sort them by their sums of corresponding elements.
# Left pairs in ascending, right pairs in descending order.
L = np.sort(c0 + c1[p], axis=1)
R = np.sort(c2 + c3[p], axis=1)[:, ::-1]
# For each pair of permutations l in L, r in R compute the smallest
# possible error (sum of squared deviations.)
if errtype == 'relerr':
err = np.empty((N, N))
split = np.linspace(0, N, N_CHUNKS+1, dtype=int)[1:-1]
for LCH, ECH in zip(np.split(L, split, axis=0),
np.split(err, split, axis=0)):
dev = LCH[:, None] + R[None, :]
((dev / (100+dev))**2).sum(axis=-1, out=ECH)
del dev
elif errtype == 'abserr':
err = (np.add.outer(np.einsum('ij,ij->i', L, L),
np.einsum('ij,ij->i', R, R))
+ np.einsum('ik, jk->ij', 2*L, R))
else:
raise ValueError
# find pair of pairs with smallest error
i = np.argmin(err.ravel())
i1, i3 = np.unravel_index(i, (N, N))
# recreate shuffled table
c0, c1, c2, c3 = C
lidx = np.argsort(c0 + c1[p[i1]])
ridx = np.argsort(c2 + c3[p[i3]])[::-1]
C = np.array([c0[lidx], c1[p[i1]][lidx], c2[ridx], c3[p[i3]][ridx]])
# correct rowsums, calculate error and corrcoef and return
if errtype == 'relerr':
result = C * (100.0 / C.sum(axis=0, keepdims=True))
err = math.sqrt((((result-C)/C)**2).mean())
else:
result = C + (25 - C.mean(axis=0, keepdims=True))
err = math.sqrt(((result-C)**2).mean())
rs = np.sort(result, axis=1)
cc = tuple(np.corrcoef(ri, range(7))[0, 1] for ri in rs)
return dict(table=result.T, avgerr=err, corrcoefs=cc)
def solve(LH, errtype='relerr'):
LH = np.asanyarray(LH)
if errtype=='relerr':
err1 = 200 / LH.sum()
diff = np.diff(LH * err1, axis=1).ravel()
elif errtype=='abserr':
err1 = 25 - LH.mean()
diff = np.diff(LH, axis=1).ravel()
else:
raise ValueError
C = np.array([np.linspace(-d/2, d/2, 7) for d in diff])
c0, c1, c2, c3 = C
p = np.array(list(itertools.permutations(range(7))))
L = np.sort(c0 + c1[p], axis=1)
R = np.sort(c2 + c3[p], axis=1)[:, ::-1]
err = (np.add.outer(np.einsum('ij,ij->i', L, L),
np.einsum('ij,ij->i', R, R))
+ np.einsum('ik, jk->ij', 2*L, R)).ravel()
i = np.argmin(err)
i1, i3 = np.unravel_index(i, (math.factorial(7), math.factorial(7)))
L = np.argsort(c0 + c1[p[i1]])
R = np.argsort(c2 + c3[p[i3]])[::-1]
ref = [np.linspace(l, h, 7) for l, h in LH]
if errtype=='relerr':
c0, c1, c2, c3 = [np.linspace(l, h, 7) for l, h in LH * err1]
C = np.array([c0[L], c1[p[i1]][L], c2[R], c3[p[i3]][R]])
err2 = 100 / np.sum(C, axis=0)
C *= err2
cs = list(map(sorted, C))
err = math.sqrt(sum((c/r-1)**2 for ci, ri in zip(cs, ref) for c, r in zip(ci, ri)) / 28)
elif errtype=='abserr':
c0, c1, c2, c3 = [np.linspace(l, h, 7) for l, h in LH + err1]
C = np.array([c0[L], c1[p[i1]][L], c2[R], c3[p[i3]][R]])
err2 = 25 - np.mean(C, axis=0)
C += err2
cs = list(map(sorted, C))
err = math.sqrt(sum((c-r)**2 for ci, ri in zip(cs, ref) for c, r in zip(ci, ri)) / 28)
else:
raise ValueError
cc = tuple(np.corrcoef(ci, range(7))[0, 1] for ci in cs)
return dict(table=C.T, avgerr=err, corrcoefs=cc)
for problem in [((75, 98), (6, 15), (2, 8), (0.05, 0.5)),
((11, 41), (4, 34), (37, 49), (0.01, 23.99)),
((80, 94), (5, 14), (0.5, 5), (0.05, 0.5)),
((36.787862883725872, 43.967159949544317),
(40.522239654303483, 47.625869880574164),
(19.760537036548321, 49.183056694462799),
(45.701873101046154, 48.051424087501672))]:
for errtype in ('relerr', 'abserr'):
print()
columns = []
for solver in (solve, improved_solve):
sol = solver(problem, errtype)
column = [[' '.join((solver.__name__, errtype))]] + \
[[k + ':'] + [' '.join([f'{e:8.5f}' for e in r])
for r in np.atleast_2d(v)]
for k, v in sol.items()]
column = (line for block in column for line in block)
columns.append(column)
for l, r in zip(*columns):
print(f"{l:39s} {r:39s}")
problems = []
for i in range(0):
problem = np.sort(np.random.random((4, 2)), axis=1) * 50
for errtype in ('relerr', 'abserr'):
sol0 = solve(problem, errtype)
sol1 = improved_solve(problem, errtype)
if not np.allclose(sol0['table'], sol1['table']):
print(i, end= " ")
if np.abs((sol0['avgerr']-sol1['avgerr'])
/(sol0['avgerr']+sol1['avgerr']))>1e-6:
print(problem)
problems.append(problem)
columns = []
for sol, name in [(sol0, 'old '), (sol1, 'improved ')]:
column = [[name + errtype]] + \
[[k + ':'] + [' '.join([f'{e:8.5f}' for e in r])
for r in np.atleast_2d(v)]
for k, v in sol.items()]
column = (line for block in column for line in block)
columns.append(column)
for l, r in zip(*columns):
print(f"{l:39s} {r:39s}")
I have a large catalog that I am selecting data from according to the following criteria:
columns = ["System", "rp", "mp", "logg"]
catalog = pd.read_csv('data.txt', skiprows=1, sep ='\s+', names=columns)
# CUTS
i = (catalog.rp != -1) & (catalog.mp != -1)
new_catalog = pd.DataFrame(catalog[i])
print("{0} targets after cuts".format(len(new_catalog)))
When I perform the above cuts the code is working fine. Next, I want to add one more cut: I want to select all the targets that have 4.0 < logg < 5.0. However, some of the targets have logg = -1 (which stands for the fact that the value is not available). Luckily, I can calculate logg from the other available parameters. So here is my updated cuts:
# CUTS
i = (catalog.rp != -1) & (catalog.mp != -1)
if catalog.logg[i] == -1:
catalog.logg[i] = catalog.mp[i] / catalog.rp[i]
i &= (4 <= catalog.logg) & (catalog.logg <= 5)
However, I am receiving an error:
if catalog.logg[i] == -1:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Can someone please explain what I am doing wrong and how I can fix it. Thank you
Edit 1
My dataframe looks like the following:
Data columns:
System 477 non-null values
rp 477 non-null values
mp 477 non-null values
logg 477 non-null values
dtypes: float64(37), int64(3), object(3)None
Edit 2
System rp mp logg FeH FeHu FeHl Mstar Mstaru Mstarl
0 target-01 5196 24 24 0.31 0.04 0.04 0.905 0.015 0.015
1 target-02 5950 150 150 -0.30 0.25 0.25 0.950 0.110 0.110
2 target-03 5598 50 50 0.04 0.05 0.05 0.997 0.049 0.049
3 target-04 6558 44 -1 0.14 0.04 0.04 1.403 0.061 0.061
4 target-05 6190 60 60 0.05 0.07 0.07 1.194 0.049 0.050
....
[5 rows x 43 columns]
Edit 3
My code in a format that I understand should be:
for row in range(len(catalog)):
parameter = catalog['logg'][row]
if parameter == -1:
parameter = catalog['mp'][row] / catalog['rp'][row]
if parameter > 4.0 and parameter < 5.0:
# select this row for further analysis
However, I am trying to write my code in a more simple and professional way. I don't want to use the for loop. How can I do it?
EDIT 4
Consider the following small example:
System rp mp logg
target-01 2 -1 2 # will NOT be selected since mp = -1
target-02 -1 3 4 # will NOT be selected since rp = -1
target-03 7 6 4.3 # will be selected since mp != -1, rp != -1, and 4 < logg <5
target-04 3.2 15 -1 # will be selected since mp != -1, rp != -1, logg = mp / rp = 15/3.2 = 4.68 (which is between 4 and 5)
you get the error because catalog.logg[i] is not a scalar,but a series,so you should turn to vectorized manipulation:
catalog.loc[i,'logg'] = catalog.loc[i,'mp']/catalog.loc[i,'rp']
which would modify the logg column inplace
As for edit 3:
rows=catalog.loc[(catalog.logg > 4) & (catalog.logg < 5)]
which will select rows that satisfy the condition
Instead of that code:
if catalog.logg[i] == -1:
catalog.logg[i] = catalog.mp[i] / catalog.rp[i]
You could use following:
i &= df.logg == -1
df.loc[i, 'logg'] = df.loc[i, 'mp'] / df.loc[i, 'rp']
# or
df.ix[i, 'logg'] = df.ix[i, 'mp'] / df.ix[i, 'rp']
For your edit 3 you need to add that line:
your_rows = df[(df.logg > 4) & (df.logg < 5)]
Full code:
i = (catalog.rp != -1) & (catalog.mp != -1)
i &= df.logg == -1
df.ix[i, 'logg'] = df.ix[i, 'mp'] / df.ix[i, 'rp']
your_rows = df[(df.logg > 4) & (df.logg < 5)]
EDIT
Probably I still don't understand what you want, but I get your desired output:
import pandas as pd
from io import StringIO
data = """
System rp mp logg
target-01 2 -1 2
target-02 -1 3 4
target-03 7 6 4.3
target-04 3.2 15 -1
"""
catalog = pd.read_csv(StringIO(data), sep='\s+')
i = (catalog.rp != -1) & (catalog.mp != -1)
i &= catalog.logg == -1
catalog.ix[i, 'logg'] = catalog.ix[i, 'mp'] / catalog.ix[i, 'rp']
your_rows = catalog[(catalog.logg > 4) & (catalog.logg < 5)]
In [7]: your_rows
Out[7]:
System rp mp logg
2 target-03 7.0 6 4.3000
3 target-04 3.2 15 4.6875
Am I still wrong?