We Keep Coding

Why I can't get the shadow price from Gurobipy while the model is feasible?

I'm trying to solve peer to peer market optimization problem. Since the price of each trade between different participants can be different, I want to draw the price of each trade. And the problem should be a continuous model, because the trade amount can be at any level.
The model already works fine without problem, I can get the trade amount of each trade using like model.variables.trade[t,k,y].x.
But the problem is when I try to get the shadow price from the related constraints, it shows : GurobiError: Unable to retrieve attribute 'Pi'.
I can't find out how comes this error since the model is already feasible, and what I do is get an attribute which already exists.
The related part of my code shows as follows:
def _build_constraints(self):
m = self.model
P_nm3 = self.variables.P_nm3
agents = self.data.agents
timewindow = self.data.timewindow
windowinterval = self.data.windowinterval
budget = self.data.budget
P_n = self.variables.P_n
self.constraints.pnmmatch = {}
......
for t in self.data.interval:
for i in range(len(P_nm3[t])):
for j in range(len(P_nm3[t][0])):
self.constraints.pnmmatch[t,i,j] = m.addConstr(
P_nm3[t][i][j],
gb.GRB.EQUAL,
-P_nm3[t][j][i]
)
.......
poolmarket = marketclear()
poolmarket.optimize()
print(poolmarket.constraints.pnmmatch[0,1,6])
print(type(poolmarket.constraints.pnmmatch))
print(poolmarket.constraints.pnmmatch[0,1,6].pi)
shadow_price = poolmarket.model.getAttr(gb.GRB.Attr.Pi)
The result is:
<gurobi.Constr R1001>
<class 'dict'>
AttributeError: Unable to retrieve attribute 'pi'
both poolmarket.constraints.pnmmatch[0,1,6].pi or poolmarket.model.getAttr(gb.GRB.Attr.Pi) doesn't work.
If I comment on the last two lines about pi, the models works pretty fine.
But in a simpler similar model where the constraint is:
for t in self.data.interval:
for i in range(len(P_nm3[t])):
for j in range(len(P_nm3[t][0])):
self.constraints.pnmmatch[t] = m.addConstr(
P_nm3[t][i][j],
gb.GRB.EQUAL,
-P_nm3[t][j][i]
)
Then I'm able to draw the pi value with poolmarket.constraints.pnmmatch[0].pi command.
How can I solve this?
Thanks in advance!

Python route finder for aviation

I made an app that besides all other things try to find a valid route between 2 airports.
I have all the required data in a sqlite3 database witch i query and plot in a basemap embeded in PyQt5 with signals live.
My problem is i can't find the algorithm to make all possible variations(with disqualify some, as all possibilies are enormous ) and store them to output the final valid routes.
Dijkstra's algorithm i think can't implemented as any time a route can reach a dead end.
My main problem is the algorithm and its implementation and not the data so don't hesitate and write any data required for any possible algorithm.
The algorith hints are:
I have a starting waypoint.
I find all routes that includes this starting point(disqualifying oposite headings)(each route have various waypoints).
Find next waypoint for each route.
Now this waypoint can be connected to other route and so on.
Routes then tested and disquilyfied by various variants or reaching dead ends.
Continue until you reach to the final(target waypoint).
Output the somehow stored route(s)
What i got so far with stack issues:
##finding base direction##
base_radians = math.atan2(self.dest_coord[0]-self.dep_coord[0], self.dest_coord[1]-self.dep_coord[1])
base_degrees = math.degrees(base_radians)
print(base_degrees)
if base_degrees < 0 :
base_heading = 'W'
else:
base_heading = 'E'
### finding all routes connected to first waypoint###
self.cursor.execute("select DISTINCT ats_ident,seq_num from dafif_ats where wpt1_ident = ? AND ats_icao = ? AND direction = ? ORDER BY ats_ident,seq_num ASC",('ATV','LGGG',base_heading))
sub_ats_idents = self.cursor.fetchall()
#### for each route find next waypoints###
for i in sub_ats_idents:
self.cursor.execute("select wpt1_ident,wpt2_ident from dafif_ats where ats_ident = ? and ats_icao = ? and direction = ? and seq_num >= ? ORDER BY seq_num ASC",(i[0],'LGGG',base_heading,i[1]))
each_wpt_combo = self.cursor.fetchall()
#### for each next waypoint find possible routes###
for x in each_wpt_combo:
self.cursor.execute("select DISTINCT ats_ident,seq_num from dafif_ats where wpt1_ident = ? AND ats_icao = ? AND direction = ? ORDER BY ats_ident,seq_num ASC",(x[0],'LGGG',base_heading))
each_ats = self.cursor.fetchall()
print(each_ats)
#### for each subroute plot waypoints###
for z in each_ats:
self.cursor.execute("select wpt1_dlon,wpt1_dlat,wpt2_dlon,wpt2_dlat from dafif_ats where wpt1_ident = ? AND ats_icao = ? AND direction = ? ORDER BY ats_ident,seq_num ASC",(x[0],'LGGG',base_heading))
plot_var = self.cursor.fetchall()
self.route_sender.emit(plot_var)
time.sleep(0.1)
Any material or example to read will be super.
Thx in advance.

For future readers , A* algorithm with hieristics is the solution for these kind of problems.

google tensor flow crash course. Issues with REPRESENTATION:Programming exercises Task 2: Make Better Use of Latitude

Hi got into another roadblock in tensorflow crashcourse...at the representation programming excercises at this page.
https://developers.google.com/…/repres…/programming-exercise
I'm at Task 2: Make Better Use of Latitude
seems I narrowed the issue to when I convert the raw latitude data into "buckets" or ranges which will be represented as 1 or zero in my feature. The actual code and issue I have is in the paste bin. Any advice would be great! thanks!
https://pastebin.com/xvV2A9Ac
this is to convert the raw latitude data in my pandas dictionary into "buckets" or ranges as google calls them.
LATITUDE_RANGES = zip(xrange(32, 44), xrange(33, 45))
the above code I changed and replaced xrange with just range since xrange is already deprecated python3.
could this be the problem? using range instead of xrange? see below for my conundrum.
def select_and_transform_features(source_df):
selected_examples = pd.DataFrame()
selected_examples["median_income"] = source_df["median_income"]
for r in LATITUDE_RANGES:
selected_examples["latitude_%d_to_%d" % r] = source_df["latitude"].apply(
lambda l: 1.0 if l >= r[0] and l < r[1] else 0.0)
return selected_examples
The next two are to run the above function and convert may exiting training and validation data sets into ranges or buckets for latitude
selected_training_examples = select_and_transform_features(training_examples)
selected_validation_examples = select_and_transform_features(validation_examples)
this is the training model
_ = train_model(
learning_rate=0.01,
steps=500,
batch_size=5,
training_examples=selected_training_examples,
training_targets=training_targets,
validation_examples=selected_validation_examples,
validation_targets=validation_targets)
THE PROBLEM:
oki so here is how I understand the problem. When I run the training model it throws this error
ValueError: Feature latitude_32_to_33 is not in features dictionary.
So I called selected_training_examples and selected_validation_examples
here's what I found. If I run
selected_training_examples = select_and_transform_features(training_examples)
then I get the proper data set when I call selected_training_examples which yields all the feature "buckets" including Feature #latitude_32_to_33
but when I run the next function
selected_validation_examples = select_and_transform_features(validation_examples)
it yields no buckets or ranges resulting in the
`ValueError: Feature latitude_32_to_33 is not in features dictionary.`
so I next tried disabling the first function
selected_training_examples = select_and_transform_features(training_examples)
and I just ran the second function
selected_validation_examples = select_and_transform_features(validation_examples)
If I do this, I then get the desired dataset for
selected_validation_examples .
The problem now is running the first function no longer gives me the "buckets" and I'm back to where I began? I guess my question is how are the two functions affecting each other? and preventing the other from giving me the datasets I need? If I run them together?
Thanks in advance!

a python developer gave me the solution so just wanted to share. LATITUDE_RANGES = zip(xrange(32, 44), xrange(33, 45)) can only be used once the way it was written so I placed it inside the succeding def select_and_transform_features(source_df) function which solved the issues. Thanks again everyone.

Why does my association model find subgroups in a dataset when there shouldn't any?

I give a lot of information on the methods that I used to write my code. If you just want to read my question, skip to the quotes at the end.
I'm working on a project that has a goal of detecting sub populations in a group of patients. I thought this sounded like the perfect opportunity to use association rule mining as I'm currently taking a class on the subject.
I there are 42 variables in total. Of those, 20 are continuous and had to be discretized. For each variable, I used the Freedman-Diaconis rule to determine how many categories to divide a group into.
def Freedman_Diaconis(column_values):
#sort the list first
column_values[1].sort()
first_quartile = int(len(column_values[1]) * .25)
third_quartile = int(len(column_values[1]) * .75)
fq_value = column_values[1][first_quartile]
tq_value = column_values[1][third_quartile]
iqr = tq_value - fq_value
n_to_pow = len(column_values[1])**(-1/3)
h = 2 * iqr * n_to_pow
retval = (column_values[1][-1] - column_values[1][1])/h
test = int(retval+1)
return test
From there I used min-max normalization
def min_max_transform(column_of_data, num_bins):
min_max_normalizer = preprocessing.MinMaxScaler(feature_range=(1, num_bins))
data_min_max = min_max_normalizer.fit_transform(column_of_data[1])
data_min_max_ints = take_int(data_min_max)
return data_min_max_ints
to transform my data and then I simply took the interger portion to get the final categorization.
def take_int(list_of_float):
ints = []
for flt in list_of_float:
asint = int(flt)
ints.append(asint)
return ints
I then also wrote a function that I used to combine this value with the variable name.
def string_transform(prefix, column, index):
transformed_list = []
transformed = ""
if index < 4:
for entry in column[1]:
transformed = prefix+str(entry)
transformed_list.append(transformed)
else:
prefix_num = prefix.split('x')
for entry in column[1]:
transformed = str(prefix_num[1])+'x'+str(entry)
transformed_list.append(transformed)
return transformed_list
This was done to differentiate variables that have the same value, but appear in different columns. For example, having a value of 1 for variable x14 means something different from getting a value of 1 in variable x20. The string transform function would create 14x1 and 20x1 for the previously mentioned examples.
After this, I wrote everything to a file in basket format
def create_basket(list_of_lists, headers):
#for filename in os.listdir("."):
# if filename.e
if not os.path.exists('baskets'):
os.makedirs('baskets')
down_length = len(list_of_lists[0])
with open('baskets/dataset.basket', 'w') as basketfile:
basket_writer = csv.DictWriter(basketfile, fieldnames=headers)
for i in range(0, down_length):
basket_writer.writerow({"trt": list_of_lists[0][i], "y": list_of_lists[1][i], "x1": list_of_lists[2][i],
"x2": list_of_lists[3][i], "x3": list_of_lists[4][i], "x4": list_of_lists[5][i],
"x5": list_of_lists[6][i], "x6": list_of_lists[7][i], "x7": list_of_lists[8][i],
"x8": list_of_lists[9][i], "x9": list_of_lists[10][i], "x10": list_of_lists[11][i],
"x11": list_of_lists[12][i], "x12":list_of_lists[13][i], "x13": list_of_lists[14][i],
"x14": list_of_lists[15][i], "x15": list_of_lists[16][i], "x16": list_of_lists[17][i],
"x17": list_of_lists[18][i], "x18": list_of_lists[19][i], "x19": list_of_lists[20][i],
"x20": list_of_lists[21][i], "x21": list_of_lists[22][i], "x22": list_of_lists[23][i],
"x23": list_of_lists[24][i], "x24": list_of_lists[25][i], "x25": list_of_lists[26][i],
"x26": list_of_lists[27][i], "x27": list_of_lists[28][i], "x28": list_of_lists[29][i],
"x29": list_of_lists[30][i], "x30": list_of_lists[31][i], "x31": list_of_lists[32][i],
"x32": list_of_lists[33][i], "x33": list_of_lists[34][i], "x34": list_of_lists[35][i],
"x35": list_of_lists[36][i], "x36": list_of_lists[37][i], "x37": list_of_lists[38][i],
"x38": list_of_lists[39][i], "x39": list_of_lists[40][i], "x40": list_of_lists[41][i]})
and I used the apriori package in Orange to see if there were any association rules.
rules = Orange.associate.AssociationRulesSparseInducer(patient_basket, support=0.3, confidence=0.3)
print "%4s %4s %s" % ("Supp", "Conf", "Rule")
for r in rules:
my_rule = str(r)
split_rule = my_rule.split("->")
if 'trt' in split_rule[1]:
print 'treatment rule'
print "%4.1f %4.1f %s" % (r.support, r.confidence, r)
Using this, technique I found quite a few association rules with my testing data.
THIS IS WHERE I HAVE A PROBLEM
When I read the notes for the training data, there is this note
...That is, the only
reason for the differences among observed responses to the same treatment across patients is
random noise. Hence, there is NO meaningful subgroup for this dataset...
My question is,
why do I get multiple association rules that would imply that there are subgroups, when according to the notes I shouldn't see anything?
I'm getting lift numbers that are above 2 as opposed to the 1 that you should expect if everything was random like the notes state.
Supp Conf Rule
0.3 0.7 6x0 -> trt1
Even though my code runs, I'm not getting results anywhere close to what should be expected. This leads me to believe that I messed something up, but I'm not sure what it is.

After some research, I realized that my sample size is too small for the number of variables that I have. I would need a way larger sample size in order to really use the method that I was using. In fact, the method that I tried to use was developed with the assumption that it would be run on databases with hundreds of thousands or millions of rows.

Back-propagation in Tensorflow

I am constructing a time delayed recurrent model and I need to know about how and when TensorFlow computes its backwards step.
Consider the following model and pseudo-code:
unit_1 = LSTM(unit_size)
unit_2 = LSTM(unit_size)
unit_3 = LSTM(unit_size)
unit_4 = LSTM(unit_size)
ip_W = Variable([4 * unit_size, output_size])
ip_b = Variable([output_size])
prev_1 = tf.zeros([unit_size])
prev_2 = tf.zeros([unit_size])
prev_3 = tf.zeros([unit_size])
prev_4 = tf.zeros([unit_size])
for t, input in enumerate(input_data):
if t%1==0:
prev_1 = unit_1([input])
if t%2==0:
prev_2 = unit_2([input])
if t%3==0:
prev_3 = unit_3([input])
if t%4==0:
prev_4 = unit_4([input])
concat = tf.concat(0,[prev_1, prev_2, prev_3, prev_4])
output[t] = tf.matmul(concat, ip_W) + ip_B
Here is a gist to a usable version of this code, based on tensorflow/python/ops/rnn.py
My Question:
For time steps where cells are not called (i.e. at T=1, unit_0 is called, while all the rest are not) are their weights updated? I'm torn as to whether it would be a good idea to have them update at every state or not. The cells themselves haven't been exposed to any new data in the steps, so I afraid that back propagating may result in over-correction. Would appreciate other's insights into this.
Let me know if any clarification is necessary.