We Keep Coding

Converting from AMPL to Pyomo

I'm trying to convert an AMPL model to Pyomo (something I have no experience with using). I'm finding the syntax hard to adapt to, especially the constraint and objective sections. I've already linked my computer together with python, anaconda, Pyomo, and GLPK, and just need to get the actual code down. I'm a beginner coder so forgive me if my code is poorly written. Still trying to get the hang of this!
Here is the data from the AMPL code:
set PROD := 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30;
set PROD1:= 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30;
ProdCost 414 3 46 519 876 146 827 996 922 308 568 176 58 13 20 974 121 751 130 844 280 123 275 843 717 694 72 413 65 631
HoldingCost 606 308 757 851 148 867 336 44 364 960 69 428 778 485 285 938 980 932 199 175 625 513 536 965 366 950 632 88 698 744
Demand 105 70 135 67 102 25 147 69 23 84 32 41 81 133 180 22 174 80 24 156 28 125 23 137 180 151 39 138 196 69
And here is the model:
set PROD; # set of production amounts
set PROD1; # set of holding amounts
param ProdCost {PROD} >= 0; # parameter set of production costs
param Demand {PROD} >= 0; # parameter set of demand at each time
param HoldingCost {PROD} >= 0; # parameter set of holding costs
var Inventory {PROD1} >= 0; # variable that sets inventory amount at each time
var Make {p in PROD} >= 0; # variable of amount produced at each time
minimize Total_Cost: sum {p in PROD} ((ProdCost[p] * Make[p]) + (Inventory[p] * HoldingCost[p]));
# Objective: minimize total cost from all production and holding cost
subject to InventoryConstraint {p in PROD}: Inventory[p] = Inventory[p-1] + Make[p] - Demand[p];
# excess production transfers to inventory
subject to MeetDemandConstraint {p in PROD}: Make[p] >= Demand[p] - Inventory[p-1];
# Constraint: holding and production must exceed demand
subject to InitialInventoryConstraint: Inventory[0] = 0;
# Constraint: Inventory must start at 0
Here's what I have so far. Not sure if it's right or not:
from pyomo.environ import *
demand=[105,70,135,67,102,25,147,69,23,84,32,41,81,133,180,22,174,80,24,156,28,125,23,137,180,151,39,138,196,69]
holdingcost=[606,308,757,851,148,867,336,44,364,960,69,428,778,485,285,938,980,932,199,175,625,513,536,965,366,950,632,88,698,744]
productioncost=[414,3,46,519,876,146,827,996,922,308,568,176,58,13,20,974,121,751,130,844,280,123,275,843,717,694,72,413,65,631]
model=ConcreteModel()
model.I=RangeSet(1,30)
model.J=RangeSet(0,30)
model.x=Var(model.I, within=NonNegativeIntegers)
model.y=Var(model.J, within=NonNegativeIntegers)
model.obj = Objective(expr = sum(model.x[i]*productioncost[i]+model.y[i]*holdingcost[i] for i in model.I))
def InventoryConstraint(model, i):
return model.y[i-1] + model.x[i] - demand[i] <= model.y[i]
InvCont = Constraint(model, rule=InventoryConstraint)
def MeetDemandConstraint(model, i):
return model.x[i] >= demand[i] - model.y[i-1]
DemCont = Constraint(model, rule=MeetDemandConstraint)
def Initial(model):
return model.y[0] == 0
model.Init = Constraint(rule=Initial)
opt = SolverFactory('glpk')
results = opt.solve(model,load_solutions=True)
model.solutions.store_to(results)
results.write()
Thanks!

The only issues I see are in some of your constraint declarations. You need to attach the constraints to the model and the first argument passed in should be the indexing set (which I'm assuming should be model.I).
def InventoryConstraint(model, i):
return model.y[i-1] + model.x[i] - demand[i] <= model.y[i]
model.InvCont = Constraint(model.I, rule=InventoryConstraint)
def MeetDemandConstraint(model, i):
return model.x[i] >= demand[i] - model.y[i-1]
model.DemCont = Constraint(model.I, rule=MeetDemandConstraint)
The syntax that you're using to solve the model is a little out-dated but should work. Another option would be:
opt = SolverFactory('glpk')
opt.solve(model,tee=True) # The 'tee' option prints the solver output to the screen
model.display() # This will print a summary of the model solution
Another command that is useful for debugging is model.pprint(). This will display the entire model including the expressions for Constraints and Objectives.

Python: TransportableException: TransportableException

I am using a package from online called Pyrcca. I am running through this tutorial: https://github.com/gallantlab/pyrcca/blob/master/Pyrcca_usage_example.ipynb but using my own input code:
# Split into training and validation data
TCIA_train, TCIA_test = train_test_split(TCIA_reduced, test_size=0.2)
TCGA_train, TCGA_test = train_test_split(TCGA, test_size=0.2)
# Initialize a cca object as an instantiation of the CCACrossValidate class.
ccaCV = rcca.CCACrossValidate(kernelcca=False, numCCs = [5,10], regs = [0.8, 0.5, 0.1, 1e2])
# Use the train() and validate() methods to run the analysis and perform cross-dataset prediction.
ccaCV.train([TCIA_train, TCGA_train])
testcorrsCV = ccaCV.validate([TCIA_test, TCGA_test])
Based on this, I encounter an error that I have never seen before and that I am unable to debug. I am hoping for some help. Thanks!
It says
"TransportableException: TransportableException"
Error log:
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-113-c44a925b202d> in <module>()
3
4 # Use the train() and validate() methods to run the analysis and perform cross-dataset prediction.
----> 5 ccaCV.train([TCIA_train, TCGA_train], False)
6 testcorrsCV = ccaCV.validate([TCIA_test, TCGA_test])
~/PycharmProjects/thesis_code/Thesis_code/Packages/rcca.py in train(self, data, parallel)
171 kernelcca=self.kernelcca, ktype=self.ktype,
172 gausigma=self.gausigma, degree=self.degree,
--> 173 cutoff=self.cutoff, selection=selection)
174 running_corr_mean_sum += fold_corr_mean
175
~/PycharmProjects/thesis_code/Thesis_code/Packages/rcca.py in train_cvfold(data, reg, numCC, kernelcca, ktype, gausigma, degree, cutoff, selection)
202 for ind in chunk]
203 notheldinds = list(set(range(nT)) - set(heldinds))
--> 204 comps = kcca([d[notheldinds] for d in data], reg, numCC,
205 kernelcca=kernelcca, ktype=ktype,
206 gausigma=gausigma, degree=degree)
~/PycharmProjects/thesis_code/Thesis_code/Packages/rcca.py in <listcomp>(.0)
202 for ind in chunk]
203 notheldinds = list(set(range(nT)) - set(heldinds))
--> 204 comps = kcca([d[notheldinds] for d in data], reg, numCC,
205 kernelcca=kernelcca, ktype=ktype,
206 gausigma=gausigma, degree=degree)
~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
2131 if isinstance(key, (Series, np.ndarray, Index, list)):
2132 # either boolean or fancy integer index
-> 2133 return self._getitem_array(key)
2134 elif isinstance(key, DataFrame):
2135 return self._getitem_frame(key)
~/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in _getitem_array(self, key)
2175 return self._take(indexer, axis=0, convert=False)
2176 else:
-> 2177 indexer = self.loc._convert_to_indexer(key, axis=1)
2178 return self._take(indexer, axis=1, convert=True)
2179
~/anaconda3/lib/python3.6/site-packages/pandas/core/indexing.py in _convert_to_indexer(self, obj, axis, is_setter)
1267 if mask.any():
1268 raise KeyError('{mask} not in index'
-> 1269 .format(mask=objarr[mask]))
1270
1271 return _values_from_object(indexer)
KeyError: '[ 0 1 2 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 22 23 24 25 26 27\n 28 30 34 35 36 37 39 40 41 42 43 44] not in index'
I am wondering whether the "\n" is a problem?

What is the difference between the type of those two files?

I am trying to test correlation power analysis attack with two different files of ciphertexts: The first file Ciphertexts.mat was already converted from numpy to matlab by using this line of codes:
import scipy.io
import numpy as np
tab_Obs = np.load('C:\\Users\\My_Test_Traces\\Ciphertexts.npy')
scipy.io.savemat('C:\\Users\\My_Test_Traces\\Ciphertexts.mat', {
"tab_Obs":tab_Obs}
)
The result is:
load 'C:/Users/cpa/data/Ciphertexts.mat';
S =
{
ciph_dec =
163 20 11 228 7 53 249 241 134 90 166 177 179 43 86 103
35 22 125 217 16 82 174 101 197 242 118 33 214 232 86 162
77 116 29 212 76 7 155 18 255 101 126 86 235 155 46 11
...........
}
The second file is parsed_cipher_0cm.mat:
load 'C:/Users/cpa/data/parsed_cipher_0cm.mat';
S =
{
ciph_dec =
67 70 185 254 55 71 60 118 165 27 247 120 31 106 154 24
24 51 124 37 190 187 208 55 32 224 134 214 49 173 224 209
192 86 229 54 24 216 91 9 136 132 131 82 44 170 234 33
.......
}
At first, I think that I have the same two files with the same type, after that, when I try to execute the second file gives me the best solution but the execution via the second file gives me this result:
error: binary operator `*' not implemented for `int32 matrix' by `matrix' operations
error: evaluating binary operator `*' near line 57, column 10
My error is the type of the first file, the error is in the calculation of the h1.
I try by this code in matlab:
load 'C:/Users/cpa/data/Ciphertexts.mat';
%load 'C:/Users/cpa/data/parsed_cipher_0cm.mat';
% truncate measurements
n_measures = 999
tab_Obs = tab_Obs(1:n_measures,:);
ciph_dec = ciph_dec(1:n_measures,:);
K=0:255;
disp (length(K));
Y_i = ciph_dec(:,sbox_n)
F = ones(1,length(K))
sbox_n = 2
disp (size(Y_i))
disp (size(F))
h1=(Y_i*ones(1,length(K)))
disp (size(h1))
I did this test to know the type of each file:
I find that:
Ciphertexts.mat---------> type: int32
parsed_cipher_0cm.mat---> type: float64
I need to have the file Ciphertexts.mat' s type= float64, how to resolve the problem please?

MATLAB does not generally allow for matrix multiplication involving integer data types unless one of the operands is a scalar. We can see this with a simple example:
ones(2, 'int8')*ones(2, 'int8')
Which throws an error:
Error using *
MTIMES is not fully supported for integer classes. At least one input must be scalar.
To compute elementwise TIMES, use TIMES (.*) instead.
This likely for integer overflow safety, though there may be other reasons I'm not familiar with. Though the error messages are not exact, the issue is presumably related.
The immediate MATLAB fix is to cast ciph_dec as a double, which should resolve the multiplication issue:
load 'C:/Users/cpa/data/Ciphertexts.mat';
% load 'C:/Users/cpa/data/parsed_cipher_0cm.mat';
% truncate measurements
n_measures = 999;
tab_Obs = tab_Obs(1:n_measures, :);
ciph_dec = double(ciph_dec(1:n_measures, :)); % Force floating point
K = 0:255;
disp(length(K));
Y_i = ciph_dec(:, sbox_n);
F = ones(1, length(K));
sbox_n = 2;
disp(size(Y_i))
disp(size(F))
h1 = (Y_i*ones(1, length(K)));
disp (size(h1))

Try simply converting to double - ciph_dec = double(ciph_dec). Matlab prefers those and as long as you are within int32, double has all these numbers, so you won't be losing precision.

How to solve Index out of bounds error?

I'm a bit confused on an error I keep running into. I didn't have it before, but at the same time my data was wrong so I had to re-write the code.
Running the following:
plt.figure(figsize=(20,10))
x = np.arange(1416, 1426, 0.009766)
gaverage = np.empty((21,1024), dtype = np.float64)
calibdata = open(pathc + 'calib_5m.dat').readlines()
#print(np.size(calibdata)) ||| Yields: 624
#print(np.size(calibdata)//16) ||| Yields: 39
calib = np.empty(shape=(np.size(calibdata)//16,1024), dtype=np.float64)
for i in range(0, np.size(calibdata)//4):
calib[i] = calibdata[i*4+3].split()
caverage = np.average(calib[i] ,axis = 0)
Yields this:
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-25-87f3f4739851> in <module>()
11 calib = np.empty(shape=(np.size(calibdata)//16,1024), dtype=np.float64)
12 for i in range(0, np.size(calibdata)//4):
---> 13 calib[i] = calibdata[i*4+3].split()
14 caverage = np.average(calib[i] ,axis = 0)
15
IndexError: index 39 is out of bounds for axis 0 with size 39
Now what I'm trying to do here is basically take every 4th line in the file read in calibdata and write it to a new array, calib[i]. If the indices are the same size how are they out of bounds? I think there's some fundamentally flawed logic here on my part so if anyone can point out where I'm falling short, that would be great.

calib is initialized to size (39,n). But i iterator goes well beyond that:
In [243]: for i in range(np.size(calibdata)//4):
...: print(i, i*4+3)
...:
0 3
1 7
2 11
3 15
4 19
5 23
6 27
7 31
8 35
....
147 591
148 595
149 599
150 603
151 607
152 611
153 615
154 619
155 623
In [244]: calib=np.zeros((np.size(calibdata)//16),int)
In [245]: calib.shape
Out[245]: (39,)

How to use list as a parameter for function

My first function creates a list from my input file. I'm trying to use the list I created as a parameter for my second function. How would I do this? I understand that each function has its own namespace so the way I'm doing it wrong. I'm assuming I need to assign this variable in the global namespace.
def get_data(file_object):
while True:
try:
file_object=input("Enter the name of the input file: ")
input_file=open(file_object, "r")
break
except FileNotFoundError:
print("Error: file not found\n:")
student_db=[]
for line in input_file:
fields=(line.split())
name=int(fields[0])
exam1=int(fields[1])
exam2=int(fields[2])
exam3=int(fields[3])
in_class=int(fields[4])
projects=int(fields[5])
exercises=int(fields[6])
record=[name,exam1,exam2,exam3,in_class,projects,exercises]
student_db.append(record)
student_db.sort()
return student_db
#def calculate_grade(a_list):
# print(a_list)
#how do I use student_db as a parameter??
def main():
# a_list=student_db
# b=calculate_grade(a_list)
# print(b)
a=get_data("data.tiny.txt")
print(a)
Here is the input file I am using
031 97 108 113 48 217 14
032 97 124 147 45 355 15
033 140 145 175 50 446 14
034 133 123 115 46 430 15
035 107 92 136 45 278 13
036 98 115 130 37 387 15
037 117 69 131 34 238 12
038 134 125 132 50 434 15
039 125 116 178 50 433 15
040 125 142 156 50 363 15
041 77 51 68 45 219 15
042 122 142 182 50 447 15
043 103 123 102 46 320 15
044 106 100 127 50 362 15
045 125 110 140 50 396 15
046 120 98 129 48 325 13
047 89 70 80 46 302 14
048 99 130 103 50 436 15
049 100 87 148 17 408 13
050 104 47 91 37 50 9

Your main (and commented out code) should look like:
def calculate_grade(a_list):
print(a_list)
def main():
a_list=get_data("data.tiny.txt")
calculate_grade(a_list)
main()
Remember this:
If your function returns a value. then you would assign it to a variable in the global namespace and use it at different points. If it has a print statement then you do not need to use print again when you are calling it

In your example, the student_db is (as I understood) stored in the variable a. You can just pass that variable to the second function, so just add calculate_grade(a) to your main function (after defining it, obviously).

The function get_data() returning a list which can be assigned to local variable and passed to other functions. like you are doing it now.
a=get_data("data.tiny.txt")
calculate_grade(a)
we can't directly use student_db[] because it is local to get_data(). it can declared/used as global but ultimately they make your program less flexible and more likely to contain errors that will be difficult to spot.
other approach would be using methods (object oriented mechanism).
class FileList :
def get_data(self, file_object):
while True:
try:
file_object=input("Enter the name of the input file: ")
input_file=open(file_object, "r")
break
except FileNotFoundError:
print("Error: file not found\n:")
self.student_db=[]
for line in input_file:
fields=(line.split())
name=int(fields[0])
exam1=int(fields[1])
exam2=int(fields[2])
exam3=int(fields[3])
in_class=int(fields[4])
projects=int(fields[5])
exercises=int(fields[6])
record=[name,exam1,exam2,exam3,in_class,projects,exercises]
self.student_db.append(record)
self.student_db.sort()
def print_object(self):
print self.student_db
def main():
myobj=FileList()
myobj.get_data("data.tiny.txt")
myobj.print_object()