I am having trouble creating a simple model in pyomo. I want to define the following abstract model:
An attempt at creating an abstract model
I define
m.V = pyo.Set()
m.C = pyo.Set() # I first wanted to make this an indexed set in m.V, but this does not work as I cannot create variables with indexed sets (in next line)
m.Components = pyo.Var(m.V*m.C, domain=Binary)
Now I have no idea how to add the constraint. Just adding
Def constr(m,v):
return sum([m.Components[v,c] for c in m.C]) == 2
m.Constraint = Constraint(m.V, rule= constr)
will lead to the model also summing over components in m.C that should not fall under m.V (eg if I pass m.V = ['Cars', 'Boats'], and one of the 'Boats' components I want to pass is ‘New sails’; the above constraint will also put a constraint on m.Components[‘Cars’,’New sails’], which does not make much sense.
Trying to work out a concrete example
Now if I try to work through this problem in a concrete way and follow e.g. Variable indexed by an indexed Set with Pyomo, I still get an issue with the constraint. E.g. say I want to create a model that has this structure:
set_dict = {‘Car’:[ ‘New wheels’, ’New gearbox’, ’New seats’],’Boat’: [’New seats’, ‘New sail’, ‘New rudder‘]}
I then create these sets and variables:
m.V = pyo.Set(initialize=[‘Car’,’Boat’])
m.C = pyo.Set(initialize=[‘New wheels’, ’New gearbox’, ’New seats’, ‘New sail’, ‘New rudder‘])
m.VxC = pyo.Set(m.V*m.C, within = set_dict)
m.Components = pyo.Var(m.VxC, domain=Binary)
But now I still dont see a way to add the constraint in a pyomo native way. I cannot define a function that sums just over m.C as then it will sum over values that are not allowed again (e.g. as above, ‘New sail’ for the ‘Cars’ vehicle type). It seems the only way to do this is to refer back to the set_dict and loop & sum over that?
I need to create an abstract model, so I want to be able to write out this model in a pyomo native way, not relying on additional dictionaries and other objects to pass the right dimensions/sets into the model.
Any idea how I could do this?
You didn't say what form your data is in, but some variation of below should work. I'm not a huge fan of AbstractModels, but each format for the data should have some accommodation to build sparse sets which is what you want to do to represent the legal combinations of V x C.
By adding a membership test within your constraint(s), you can still sum across either V or C as needed.
import pyomo.environ as pyo
m = pyo.AbstractModel()
### SETS
m.V = pyo.Set()
m.C = pyo.Set()
m.VC = pyo.Set(within = m.V*m.C)
### VARS
m.select = pyo.Var(m.VC, domain=pyo.Binary)
### CONSTRAINTS
def constr(m,v):
return sum(m.select[v,c] for c in m.C if (v,c) in m.VC) == 2
m.Constraint = pyo.Constraint(m.V, rule= constr)
Related
I am trying to change Model names in Abaqus with respect to the values in an array list. At first, I created two array lists and divided them but it is not a good idea as I will have 100 values later on in Beam_h and Beam_w and the values will repeat.. What can I do if I want to have my model names be: model20-10, model30-10, model50-10? Also, the loop I used so far gives me model0, model1, model2. What to write in the loop to get desired model names?
#Here is the code,
Beam_h = [20, 30, 50] #Beam Height mm
Beam_w = [10, 10, 10] #Beam width mm
divide_list = [x/y for x, y in zip(Beam_h, Beam_w)] ##I do not want to divide.It's wrong
for values in divide_list: ##looping not right
mdb.Model(name='model'+str(values))
BeamModel = mdb.Model(name='model'+str(values))
I think, you just need to figure out string concatenation.
You need to check for duplicate model names as well. Because Abaqus replaces the already existing model if you create a model with duplicate name.
To address this issue, you can use dictionary object in following way:
dup_Models = dict() # To store the model names and repeating count
for x,y in zip(Beam_h,Beam_w):
modelName = 'model%d-%d'%(x,y) # String concatenation - You can use other ways also!
if modelName not in dup_Models:
dup_Models[modelName] = 1
umodelName = modelName # Unique model name (No Duplication)
else:
dup_Models[modelName] += 1
umodelName = '%s_%d'%(modelName,dup_Models[modelName])
BeamModel = mdb.Model(name=umodelName)
One last thing, you cannot use .(dot) in model name(Abaqus throws an error). So, if Beam_h and Beam_w has values other than integers, then you have to name the model some other way, for ex. w=10.5 and w=20.5 --> model10pt5-20pt5.
Using gurobi and python I am trying to solve a water balance(similar to the classic transportation problem) linear programming problem in the form of:
minimize c'x subject to:
Ax=b
lb<=x<=ub
A, L are sparse crs scipy matrices, c,b,lb,ub are vectors.
My problem should be updated for a number of steps and some elements are updated with new values. Specifically A is fixed, and all other elements get new values at each step. The following snippet works perfectly and is the basis I used so far (ignore the "self", as the model is embedded in a solver class, while "water_network is the graph object holding values and properties for each step):
### Snippet 1: Formulating/initializing the problem
# unitC is the c vector
# Bounds holds both lb and ub values for each x
self.model = gurobipy.Model()
rows, cols = len(self.water_network.node_list), len(self.water_network.edge_name_list)
self.x1 = []
for j in range(cols):
self.x1.append(self.model.addVar(lb=self.water_network.Bounds[j,0], ub=self.water_network.Bounds[j,1],obj=self.water_network.unitC[j]))
self.model.update()
self.EqualityConstraintA=[]
for i in range(rows):
start = self.water_network.A_sparse.indptr[i]
end = self.water_network.A_sparse.indptr[i+1]
variables = [self.x1[j] for j in self.water_network.A_sparse.indices[start:end]]
coeff = self.water_network.A_sparse.data[start:end]
expr = gurobipy.LinExpr(coeff, variables)
self.EqualityConstraintA.append(self.model.addConstr(lhs=expr, sense=gurobipy.GRB.EQUAL, rhs=self.water_network.b [i],name='A'+str(i)))
self.model.update()
self.model.ModelSense = 1
self.model.optimize()
The following simple snippet is used to update the problem at each step. Note i use the getConstrs function:
#### Snippet 2: Updating the constraints, working ok for every step.
self.model.setAttr("LB",self.model.getVars(), self.water_network.Bounds[:,0])
self.model.setAttr("UB", self.model.getVars(), self.water_network.Bounds[:,1])
self.model.setAttr("OBJ", self.model.getVars(), self.water_network.unitC)
self.model.setAttr("RHS", self.model.getConstrs(),self.water_network.b)
The problem arised when a new set of constraints should be added to the problem, in the form of:
Lx=0 where L is a sparse matrix that is updated every step! Now in the formulation I add the following just after the snippet 1:
self.EqualityConstraintL=[]
leakrows= len(self.water_network.ZeroVector)
for i in range(leakrows):
start = self.water_network.L_sparse.indptr[i]
end=self.water_network.L_sparse.indptr[i+1]
variables=[self.x1[j] for j in self.water_network.L_sparse.indices[start:end]]
coeff=self.water_network.L_sparse.data[start:end]
expr = gurobipy.LinExpr(coeff, variables)
self.EqualityConstraintL.append(self.model.addConstr(lhs=expr, sense=gurobipy.GRB.EQUAL, rhs=self.water_network.ZeroVector[i],name='L'+str(i)))
However, I can no longer use the getConstrs to update all constraints at once, as some need only the RHS changed and others need only the LHS changed. So I did the following for the update (Snippet 3):
self.model.setAttr("LB",self.model.getVars(), self.water_network.Bounds[:,0])
self.model.setAttr("UB", self.model.getVars(), self.water_network.Bounds[:,1])
self.model.setAttr("OBJ", self.model.getVars(), self.water_network.unitC)
# Update A rhs...
for i in range(len(self.water_network.edge_name_list)):
self.model.setAttr("RHS", self.model.getConstrs()[i],self.water_network.b[i])
# Update L expr...
x1=self.model.getVars()
n=len(self.water_network.node_list) # because there are n rows in the A constrains, and L constraints are added after
# Now i rebuild the LHS expressions
for i in range(len(self.water_network.ZeroVector)):
start = self.water_network.L_sparse.indptr[i]
end=self.water_network.L_sparse.indptr[i+1]
variables=[x1[j] for j in self.water_network.L_sparse.indices[start:end]]
coeff=self.water_network.L_sparse.data[start:end]
expr = gurobipy.LinExpr(coeff, variables)
self.model.setAttr("LHS",self.model.getConstrs()[n+i],expr)
self.model.update()
self.model.optimize()
When I run the problem, it initializes fine, but at the second step it returns this error:
File "model.pxi", line 1709, in gurobipy.Model.setAttr
TypeError: object of type 'Constr' has no len()
and the offending line is:
self.model.setAttr("RHS", self.model.getConstrs()[i],self.water_network.b[i])
Two questions: 1) why is that happening? replacing getConstrs()[i] with getConstrByName('A'+str(i)) also fails with the exact same error. How to update the RHS/LHS of a specific constraint?
2) Is there a way to more efficiently update the RHS on the constraints contained in the self.EqualityConstraintA list and then the LHS on the other constraints contained in the self.EqualityConstraintL list ?
Many thanks in advance!
Di
The setAttr function on the model object is for
setting attributes globally on the model
setting attributes for a list of variables
setting attributes for a list of constraints
The individual constraint and variable objects have their own setAttr functions to set attributes on single variables and constraints. In your case,
for i in range(len(self.water_network.edge_name_list)):
self.model.getConstrs()[i].setAttr('RHS', self.water_network.b[i])
Which could be replaced by the more pythonic (and likely more efficient)
m = self.model
constrs = m.getConstrs()[:len(self.water_network.edge_name_list)]
m.setAttr('RHS', constrs, self.water_network.b)
I am trying to design an optimizer that chooses which products to sell based on some pre-defined parameters. The only restrictions would be the maximal amount of products to sell and some dependencies between products (If you sell product B, you have to sell product D f.e.). I am having problems defining the latter constraint.
What follows is a simplified version of the problem:
import numpy as np
from pyomo import environ as pe
## define articles
article_list=('A','B','C','D')
## and their values ("sales")
alphas=(0,1,2,3)
alphas_input=dict(zip(article_list,alphas))
## generate compatibility matrix, 1 means article pair is dependant
compatibilities=dict(
((article_a,article_b),0)
for article_a in article_list
for article_b in article_list
)
## manually assign compatibilities so that
## every product is dependant on itself and D and B are dependant on each other
comp_arr=[1,0,0,0,0,1,0,1,0,0,1,0,0,1,0,1]
compatibilities=dict(zip(compatibilities.keys(),comp_arr))
## indices: articles
model_exp.article_list = pe.Set(
initialize=article_list)
Defining model
## create model
model_exp = pe.ConcreteModel()
## parameters: fixed values
model_exp.alphas=pe.Param(
model_exp.article_list,
initialize=alphas_input,
within=pe.Reals)
model_exp.compatibilities=pe.Param(
model_exp.article_list*model_exp.article_list,
initialize=compatibilities,
within=pe.Binary
)
## variables: selected articles -> 0/1 values
model_exp.assignments=pe.Var(
model_exp.article_list,
domain=pe.Binary
)
## objective function
model_exp.objective=pe.Objective(
expr=pe.summation(model_exp.alphas,model_exp.assignments),
sense=pe.maximize
)
Defining constraints
def limit_number_articles(model):
n_products_assigned=sum(
model_exp.assignments[article]
for article in model.article_list
)
return n_products_assigned<=2
model_exp.limit_number_articles=pe.Constraint(
rule=limit_number_articles
)
Now to the problematic constraint. Without this constraint the optimizer would choose C and D as the two articles since they have the higher alphas. But since I have defined D and B as dependant on each other, I need the optimizer to either choose both of them or none of them (since they have higher alphas than A and C, the optimal solution would be to choose them).
This is the closest I've got to defining the constraint I need:
def control_compatibilities(model,article_A):
sum_list=[]
#loopo over article pairs
for article_loop in model_exp.article_list:
# check whether the article pair is dependant
if model_exp.compatibilities[article_A,article_loop]==1:
# sum the amount of articles among the pair that are included
# if none are (0) or both are (2) return True
sum_list.append(sum([model_exp.assignments[article_A]==1,
model_exp.assignments[article_loop]==1]) in [0,2])
else:
#if they are not dependant, no restruction applies
sum_list.append(True)
sum_assignments=sum(sum_list)
return sum_assignments==4
model_exp.control_compatibilities=pe.Constraint(
model_exp.article_list,
rule=control_compatibilities
)
The above contraint returns the following error:
Invalid constraint expression. The constraint expression resolved to a
trivial Boolean (True) instead of a Pyomo object. Please modify your rule to
return Constraint.Feasible instead of True.
Any ideas on to how to define the constraint would be very helpful.
Look at some resources on modeling logic constraints and implications with binary variables. A quick google search produced the resources below which should guide you to a valid formulation:
http://ben-israel.rutgers.edu/386/Logic.pdf
https://laurentlessard.com/teaching/cs524/slides/20%20-%20logic%20constraints%20and%20integer%20variables.pdf
I solved it substracting the selection of one item from the other (0-0=0 and 1-1=0) and iterating over all dependant product pairs.
def control_compatibilities(model,article_A):
compatible_pairs=[k for k,v in compatibilities.items() if v==1]
compatible_pairs_filt=[a for a in compatible_pairs if a[0]==article_A]
sum_assignments=sum(model_exp.assignments[a[0]]-model_exp.assignments[a[1]]
for a in compatible_pairs_filt)
return sum_assignments==0
model_exp.control_compatibilities=pe.Constraint(
model_exp.article_list,
rule=control_compatibilities
)
I give a lot of information on the methods that I used to write my code. If you just want to read my question, skip to the quotes at the end.
I'm working on a project that has a goal of detecting sub populations in a group of patients. I thought this sounded like the perfect opportunity to use association rule mining as I'm currently taking a class on the subject.
I there are 42 variables in total. Of those, 20 are continuous and had to be discretized. For each variable, I used the Freedman-Diaconis rule to determine how many categories to divide a group into.
def Freedman_Diaconis(column_values):
#sort the list first
column_values[1].sort()
first_quartile = int(len(column_values[1]) * .25)
third_quartile = int(len(column_values[1]) * .75)
fq_value = column_values[1][first_quartile]
tq_value = column_values[1][third_quartile]
iqr = tq_value - fq_value
n_to_pow = len(column_values[1])**(-1/3)
h = 2 * iqr * n_to_pow
retval = (column_values[1][-1] - column_values[1][1])/h
test = int(retval+1)
return test
From there I used min-max normalization
def min_max_transform(column_of_data, num_bins):
min_max_normalizer = preprocessing.MinMaxScaler(feature_range=(1, num_bins))
data_min_max = min_max_normalizer.fit_transform(column_of_data[1])
data_min_max_ints = take_int(data_min_max)
return data_min_max_ints
to transform my data and then I simply took the interger portion to get the final categorization.
def take_int(list_of_float):
ints = []
for flt in list_of_float:
asint = int(flt)
ints.append(asint)
return ints
I then also wrote a function that I used to combine this value with the variable name.
def string_transform(prefix, column, index):
transformed_list = []
transformed = ""
if index < 4:
for entry in column[1]:
transformed = prefix+str(entry)
transformed_list.append(transformed)
else:
prefix_num = prefix.split('x')
for entry in column[1]:
transformed = str(prefix_num[1])+'x'+str(entry)
transformed_list.append(transformed)
return transformed_list
This was done to differentiate variables that have the same value, but appear in different columns. For example, having a value of 1 for variable x14 means something different from getting a value of 1 in variable x20. The string transform function would create 14x1 and 20x1 for the previously mentioned examples.
After this, I wrote everything to a file in basket format
def create_basket(list_of_lists, headers):
#for filename in os.listdir("."):
# if filename.e
if not os.path.exists('baskets'):
os.makedirs('baskets')
down_length = len(list_of_lists[0])
with open('baskets/dataset.basket', 'w') as basketfile:
basket_writer = csv.DictWriter(basketfile, fieldnames=headers)
for i in range(0, down_length):
basket_writer.writerow({"trt": list_of_lists[0][i], "y": list_of_lists[1][i], "x1": list_of_lists[2][i],
"x2": list_of_lists[3][i], "x3": list_of_lists[4][i], "x4": list_of_lists[5][i],
"x5": list_of_lists[6][i], "x6": list_of_lists[7][i], "x7": list_of_lists[8][i],
"x8": list_of_lists[9][i], "x9": list_of_lists[10][i], "x10": list_of_lists[11][i],
"x11": list_of_lists[12][i], "x12":list_of_lists[13][i], "x13": list_of_lists[14][i],
"x14": list_of_lists[15][i], "x15": list_of_lists[16][i], "x16": list_of_lists[17][i],
"x17": list_of_lists[18][i], "x18": list_of_lists[19][i], "x19": list_of_lists[20][i],
"x20": list_of_lists[21][i], "x21": list_of_lists[22][i], "x22": list_of_lists[23][i],
"x23": list_of_lists[24][i], "x24": list_of_lists[25][i], "x25": list_of_lists[26][i],
"x26": list_of_lists[27][i], "x27": list_of_lists[28][i], "x28": list_of_lists[29][i],
"x29": list_of_lists[30][i], "x30": list_of_lists[31][i], "x31": list_of_lists[32][i],
"x32": list_of_lists[33][i], "x33": list_of_lists[34][i], "x34": list_of_lists[35][i],
"x35": list_of_lists[36][i], "x36": list_of_lists[37][i], "x37": list_of_lists[38][i],
"x38": list_of_lists[39][i], "x39": list_of_lists[40][i], "x40": list_of_lists[41][i]})
and I used the apriori package in Orange to see if there were any association rules.
rules = Orange.associate.AssociationRulesSparseInducer(patient_basket, support=0.3, confidence=0.3)
print "%4s %4s %s" % ("Supp", "Conf", "Rule")
for r in rules:
my_rule = str(r)
split_rule = my_rule.split("->")
if 'trt' in split_rule[1]:
print 'treatment rule'
print "%4.1f %4.1f %s" % (r.support, r.confidence, r)
Using this, technique I found quite a few association rules with my testing data.
THIS IS WHERE I HAVE A PROBLEM
When I read the notes for the training data, there is this note
...That is, the only
reason for the differences among observed responses to the same treatment across patients is
random noise. Hence, there is NO meaningful subgroup for this dataset...
My question is,
why do I get multiple association rules that would imply that there are subgroups, when according to the notes I shouldn't see anything?
I'm getting lift numbers that are above 2 as opposed to the 1 that you should expect if everything was random like the notes state.
Supp Conf Rule
0.3 0.7 6x0 -> trt1
Even though my code runs, I'm not getting results anywhere close to what should be expected. This leads me to believe that I messed something up, but I'm not sure what it is.
After some research, I realized that my sample size is too small for the number of variables that I have. I would need a way larger sample size in order to really use the method that I was using. In fact, the method that I tried to use was developed with the assumption that it would be run on databases with hundreds of thousands or millions of rows.
I am stuck in this issue:
I have two models:
Location and Rate.
each location has its rate, possibly multiple rates.
i want to get locations ordered by its rates, ascendingly.
obvouisly, order_by and distinct() dont work together:
locations = Location.objects.filter(**s_kwargs).order_by('locations_rate__rate').distinct('id')
then i read the docs and came to annotate(). but i am not sure whether i have to use a function between annotate.
if i do this:
locations = Location.objects.filter(**s_kwargs).annotate(rate=Count('locations_rate__rate')).order_by('rate')
but this counts the rates and orders by the sum. i want to get locations with its rates ordered by the value of those rates.
my model definitions are:
class Location(models.Model):
name = models.TextField()
adres = models.TextField()
class Rate(models.Model):
location = models.ForeignKey(Location,related_name='locations_rate')
rate = models.IntegerField(max_length=2)
price_rate = models.IntegerField(max_length=2) #<--- added now
datum = models.DateTimeField(auto_now_add=True,blank=True) #<--- added now
Well the issue is not how to make query in Django for the problem you described. It's that your problem is either incorrect or not property thought through. Let me explained with an example:
Suppose you have two Location objects, l1 and l2. l1 has two Rate objects related to it, r1 and r3, such that r1.rate = 1 and r3.rate = 3; And l2 has one rate object related to it, r2, such that r2.rate = 2. Now what should be the order of your query's result l1 followed l2 or l2 followed by l1?? As one of l1's rate is less than l2's rate and the other one is greater than l2's rate.
Try this:
from django.db.models import Count, Sum
# if you want to annotate by count of rates
locations = Location.objects.filter(**s_kwargs) \
.annotate(rate_count = Count('locations_rate')) \
.order_by('rate_count')
# if you want to annotate on values of rate e.g. Sum
locations = Location.objects.filter(**s_kwargs) \
.annotate(rate_count = Sum('locations_rate')) \
.order_by('rate_count')
Possibly you want something like this:
locations = (Location.objects.filter(**s_kwargs)
.values('locations_rate__rate')
.annotate(Count('locations_rate__rate'))
.order_by('locations_rate__rate'))
You need the Count() since you actually need a GROUP BY query, and GROUP BY only works with aggregate functions like COUNT or SUM.
Anyway I think your problem can be solved with normal distinct():
locations = (Location.objects.filter(**s_kwargs)
.order_by('locations_rate__rate')
.distinct('locations_rate__rate'))
Why would you want to use annotate() instead?
I haven't tested both but hope it helps.
annotate(*args, **kwargs),Annotates each object in the QuerySet with the provided list of aggregate values (averages, sums, etc) that have
been computed over the objects that are related to the objects in the QuerySet.
So if you want only to get locations ordered by its rates, ascendingly you dont have to use annotate()
you can try this :
loc = Location.objects.all()
rate = Rate.objects.filter(loc=rate__location).order_by('-rate')