Related
Please help me on my periodic capacitated vehicle routing problem.
Find the same question here on google-groups.
What I am trying to model:
1 depot
multiple customers (let's assume 4)
multiple days (let's assume 3)
Multiple vehicle types with price per km and specific capacity
Every customer has: delivery frequency and demand per delivery
Every customer has to be assigned to a delivery pattern. Multiple delivery patterns are possible for each frequency. The pattern indicates if a delivery is possible on that day or not. For a frequency of 1 and 3 working days: [[1,0,0],[0,1,0],[0,0,1]], a frequency of 2 [[1,1,0],[1,0,1],[1,1,0]], a frequency of 3: [1,1,1]these are the possible patterns.
The demand of each customer per delivery is the same, for every day a delivery is performed
So we have a set of customers that have to be delivered by one depo. Multiple Vehicle Types are available for the job. The day of service is flexible because only the frequency is fixed to a customer, but there are multiple possibilities to fulfill the demand.
What I did so far:
I copied every node by the number of working days.
I restrict the start and end to the depo nodes of each day.
I multiply the vehicles by the number of days.
Furthermore, I handle the days as different kinds of cargo. I do assign a capacity of zero to the vehicle when it should not be used on that day.
[[10, 15, 15, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 10, 15, 15, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 10, 15, 15]]
The first 3 vehicles are for day one, the second three are day two
and the last 3 are day 3.
For the demand matrix I use the same trick. In a case of 4 customers + 1 depot the demand matrix could look like this
[[0, 3, 0, 5, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 3, 4, 5, 2, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 2]]
The second entry in the first list (3) and the 7th entry in the second list (3) are for the same customer just on two different days. The nodes were copied.
This works good so far. But it's not what I want to do.
I do assign a demand for every customer and day by hand, but I want to assign the demand by a chosen delivery pattern for each customer.
The model could assign pattern [1,0,1] to one customer with a frequency of 2 and a pattern of [0,0,1] to another customer. This would result in the possibility to plan a tour on day 3 with both demands combined.
I need help to define a dimension that manages the assignment of patterns to customers which results in "flexible" demand matrix.
Thank you for your help.
Full Code:
"""Periodic Capacited Vehicles Routing Problem (PCVRP)."""
from ortools.constraint_solver import routing_enums_pb2
from ortools.constraint_solver import pywrapcp
import numpy as np
def dublicate_nodes(base_distance_matrix, num_days):
matrix = []
for i in range(num_days):
day_element = []
for node in range(len(base_distance_matrix)):
node_element = []
for day in range(num_days):
node_element.append(base_distance_matrix[node])
day_element.append(np.concatenate(node_element).tolist())
matrix.extend(day_element)
#print(matrix)
return matrix
def depo_nodes(num_days, num_verhicles, num_nodes):
depo_array = []
for day in range(num_days):
for veh in range(int(num_verhicles/num_days)):
depo_array.append(day*num_nodes)
#print(depo_array)
return depo_array
def veh_types_costs(cost_array, num_per_type, num_days):
cost_array_km = []
for day in range(num_days):
for i in range(len(cost_array)):
for num in range(num_per_type[i]):
cost_array_km.append(cost_array[i])
#print(cost_array_km)
return(cost_array_km)
def veh_capacity_matrix(num_vehicle_per_type, vehicle_capacity_type, num_days):
matrix= []
for index in range(num_days):
matrix_element = []
for day in range(num_days):
for i, num in enumerate(num_vehicle_per_type):
for times in range(num):
if day == index:
matrix_element.append(vehicle_capacity_type[i]);
else:
matrix_element.append(0)
matrix.append(matrix_element)
#print(matrix)
return matrix
def create_data_model():
data = {}
data["num_days"] = 3 #Anzahl Tage
### Definition der Fahrezugtypen
data["num_vehicle_per_type"] = [1, 2] #Anzahl Fahrzeuge pro Typ
data["vehicle_costs_type_per_km"] = [1, 0.01] #Kosten pro km pro Fahrzeugtyp
data["vehicle_costs_type_per_stop"] = [1, 1000] #Kosten pro Stop pro Fahrzeugtyp
data['price_per_km'] = veh_types_costs(data["vehicle_costs_type_per_km"], data["num_vehicle_per_type"], data["num_days"]) #Matrix für price_per_km je Fahrzeug ertsellen
data["price_per_stop"] = veh_types_costs(data["vehicle_costs_type_per_stop"], data["num_vehicle_per_type"], data["num_days"])
data["vehicle_capacity_type"] = [10, 15] # Kapaität pro Fahrzeugtyp
data['vehicle_capacities_matrix'] = veh_capacity_matrix(data["num_vehicle_per_type"], data["vehicle_capacity_type"], data["num_days"]) #Kapazitäten pro Fahrzeugs pro Tag
print('vehicle_capacities_matrix')
print(data['vehicle_capacities_matrix'])
data["num_vehicles_per_day"] = sum(data["num_vehicle_per_type"]) #Gesamtanzahl der Fahrzeuge pro Tag
data['num_vehicles'] = data["num_days"] * data["num_vehicles_per_day"]# Gesamtanzahl der Fahrzeuge in gesamten Zeitraum
###Distanzmatrix bestimmen
data["base_distance_matrix"] = [
[
0, 548, 776, 696, 582
],
[
548, 0, 684, 308, 194
],
[
776, 684, 0, 992, 878
],
[
696, 308, 992, 0, 114
],
[
582, 194, 878, 114, 0
]
] #Distanzmatrix mit allen Kunden einzeln
data['distance_matrix'] = dublicate_nodes(data["base_distance_matrix"], data["num_days"]) # Distanzmatrix mit mehrfachen Nodes pro Kunden, je Tag ein Node
###Start und Ende festlegen
data["num_nodes"] = len(data["base_distance_matrix"]) # Anzahl Kunden
data['starts'] = depo_nodes(data["num_days"], data['num_vehicles'], data["num_nodes"]) #Deponodes (Start) für die einzelnen Fahrzeuge (Tage)
data['ends'] = depo_nodes(data["num_days"], data['num_vehicles'], data["num_nodes"]) #Deponodes (Ende) für die einzelnen Fahrzeuge (Tage)
###Demand pro Kunde
data['demands_matrix'] = [[0, 3, 0, 5, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],[0, 0, 0, 0, 0, 0, 3, 4, 5, 2, 0, 0, 0, 0, 0],[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 2]] #Demandmatrix mit je einer list pro Tag
return data
def print_solution(data, manager, routing, solution):
"""Prints solution on console."""
total_distance = 0
total_load = 0
for vehicle_id in range(data['num_vehicles']):
if vehicle_id % data["num_vehicles_per_day"] == 0:
print(("------Tag {}------").format(int(vehicle_id/data["num_vehicles_per_day"])))
if routing.IsVehicleUsed(assignment= solution, vehicle=vehicle_id) == False:
continue
index = routing.Start(vehicle_id)
if vehicle_id >= data["num_vehicles_per_day"]:
plan_output = 'Route for vehicle {}:\n'.format(abs(vehicle_id-data["num_vehicles_per_day"]*int(vehicle_id/data["num_vehicles_per_day"])))
else:
plan_output = 'Route for vehicle {}:\n'.format(vehicle_id)
route_costs = 0
route_load = 0
while not routing.IsEnd(index):
node_index = manager.IndexToNode(index)
capacity_ID = int(vehicle_id/data["num_vehicles_per_day"])
route_load += data['demands_matrix'][capacity_ID][node_index]
plan_output += ' {0} Load({1}) -> '.format(node_index, route_load)
previous_index = index
index = solution.Value(routing.NextVar(index))
route_costs += routing.GetArcCostForVehicle(
previous_index, index, vehicle_id)
plan_output += ' {0} Load({1})\n'.format(manager.IndexToNode(index),
route_load)
plan_output += 'Costs of the route: {}€\n'.format(route_costs)
plan_output += 'Load of the route: {}\n'.format(route_load)
print(plan_output)
total_distance += route_costs
total_load += route_load
print('Total costs of all routes: {}€'.format(total_distance))
print('Total load of all routes: {}'.format(total_load))
def main():
"""Periodic CVRP problem."""
# Instantiate the data problem.
data = create_data_model()
# Create the routing index manager.
manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']), data['num_vehicles'], data['starts'],
data['ends'])
# Create Routing Model.
routing = pywrapcp.RoutingModel(manager)
### Kostenfunktion je Fahrzeug festlegen ###
def create_cost_callback(dist_matrix, km_costs, stop_costs):
# Create a callback to calculate distances between cities.
def distance_callback(from_index, to_index):
from_node = manager.IndexToNode(from_index)
to_node = manager.IndexToNode(to_index)
return int(dist_matrix[from_node][to_node]) * (km_costs) + (stop_costs)
return distance_callback
for i in range(data['num_vehicles']):
cost_callback = create_cost_callback(data['distance_matrix'], data["price_per_km"][i],
data["price_per_stop"][i]) # Callbackfunktion erstellen
cost_callback_index = routing.RegisterTransitCallback(cost_callback) # registrieren
routing.SetArcCostEvaluatorOfVehicle(cost_callback_index, i) # Vehicle zuordnen
#Define Capacities for Vehicles for every Day
def create_demand_callback(demand_matrix, demand_index):
# Create a callback to calculate capacity usage.
def demand_callback(from_index):
#Returns the demand of the node.
# Convert from routing variable Index to demands NodeIndex.
from_node = manager.IndexToNode(from_index)
return demand_matrix[demand_index][from_node]
return demand_callback
for i in range(data["num_days"]): #Jedes Fahrzeug hat pro Tag eine andere Kapazität
demand_callback = create_demand_callback(data['demands_matrix'], i)
demand_callback_index = routing.RegisterUnaryTransitCallback(demand_callback)
dimension_name = 'Capacity_day_{}'.format(i)
routing.AddDimensionWithVehicleCapacity(
demand_callback_index,
0, # null capacity slack
data['vehicle_capacities_matrix'][i], # vehicle maximum capacities
True, # start cumul to zero
dimension_name)
# Setting first solution heuristic.
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (
routing_enums_pb2.FirstSolutionStrategy.AUTOMATIC)
search_parameters.local_search_metaheuristic = (
routing_enums_pb2.LocalSearchMetaheuristic.AUTOMATIC)
search_parameters.time_limit.FromSeconds(1)
# Solve the problem.
solution = routing.SolveWithParameters(search_parameters)
# Print solution on console.
if solution:
print_solution(data, manager, routing, solution)
return solution
if __name__ == '__main__':
solution_array = []
solution = main()
So I found a solution for my problem.
The solving time for problems greater than 40 customers could be better.
I would appreciate improvement suggestions.
"""Periodic Capacited Vehicles Routing Problem with delivery pattern(PCVRP)."""
from ortools.constraint_solver import routing_enums_pb2
from ortools.constraint_solver import pywrapcp
import numpy as np
import random
import math
import pandas as pd
def dublicate_nodes(base_distance_matrix, num_days):
matrix = []
for i in range(num_days):
day_element = []
for node in range(len(base_distance_matrix)):
node_element = []
for day in range(num_days):
node_element.append(base_distance_matrix[node])
day_element.append(np.concatenate(node_element).tolist())
matrix.extend(day_element)
#print(matrix)
return matrix
def depo_nodes(num_days, num_verhicles, num_nodes):
depo_array = []
for day in range(num_days):
for veh in range(int(num_verhicles/num_days)):
depo_array.append(day*num_nodes)
#print(depo_array)
return depo_array
def veh_types_costs(cost_array, num_per_type, num_days):
cost_array_km = []
for day in range(num_days):
for i in range(len(cost_array)):
for num in range(num_per_type[i]):
cost_array_km.append(cost_array[i])
#print(cost_array_km)
return(cost_array_km)
def veh_capacity_matrix(num_vehicle_per_type, vehicle_capacity_type, num_days):
matrix= []
for index in range(num_days):
matrix_element = []
for day in range(num_days):
for i, num in enumerate(num_vehicle_per_type):
for times in range(num):
if day == index:
matrix_element.append(vehicle_capacity_type[i]);
else:
matrix_element.append(0)
matrix.append(matrix_element)
#print(matrix)
return matrix
def dist_matrix(koordianten):
matrix =[]
for k_1 in koordianten:
matrix_element = []
for k_2 in koordianten:
matrix_element.append(np.linalg.norm(k_1-k_2))
matrix.append(matrix_element)
#print(matrix)
return matrix
def demand_matrix(demand_matrix_day,num_days):
matrix = []
for index in range(num_days):
matrix_element = []
for day in range(num_days):
if day == index:
matrix_element = matrix_element +demand_matrix_day
else:
matrix_element = matrix_element +(list(np.zeros(len(demand_matrix_day))))
matrix.append(matrix_element)
return matrix
def kunden_indizes(ID_list, num_days):
indizes = {}
for ID in ID_list:
indizes[ID] = []
for index, key in enumerate(indizes):
for i in range(len(ID_list)*num_days):
if (i % len(ID_list)) == index:
indizes[key].append(i)
#print(indizes)
return indizes
def create_data_model(node_num):
data = {}
data["num_days"] = 5 #Anzahl Tage
###Kundenset
num_IDs = node_num
base = [0]
data["IDs"] = list(range(0, num_IDs))
#data["IDs"] = [0, 1, 2, 3, 4]
print(data["IDs"])
data["demand"] = [random.randint(1,30) for i in range(0,num_IDs)]
#data["demand"] =[0, 16, 8, 1, 7]
data["demand"][0] = 0
print("demand", data["demand"])
data["frequenz"] = [random.randint(1,5) for i in range(0,num_IDs)]
#data["frequenz"] =[0, 3, 5, 2, 1]
data["frequenz"][0] = 0
print("freq ",data["frequenz"])
summe_dem =0
for index, demand in enumerate(data["demand"]):
summe_dem += data["demand"][index] * data["frequenz"][index]
print("DEMANDSUMME: ",summe_dem)
data["koordianten"] = np.random.rand(num_IDs, 2)*1000
#data["koordianten"] = np.array([(0, 0), (4, 0), (4, 4), (0, 4), (3, 3)])*100
data["koordianten"][0] = (0,0)
#print("koord ", data["koordianten"])
data["PAT"] = {
5:
[[1, 1, 1, 1, 1]],
4:
[[0, 1, 1, 1, 1],
[1, 0, 1, 1, 1],
[1, 1, 0, 1, 1],
[1, 1, 1, 0, 1],
[1, 1, 1, 1, 0]],
3: [[0, 1, 0, 1, 1],
[1, 0, 1, 0, 1],
[1, 1, 0, 1, 0],
[0, 1, 1, 0, 1],
[1, 0, 1, 1, 0],
],
2: [[1, 0, 1, 0, 0],
[0, 1, 0, 1, 0],
[0, 0, 1, 0, 1],
[1, 0, 0, 1, 0],
[0, 1, 0, 0, 1]],
1: [[1, 0, 0, 0, 0],
[0, 1, 0, 0, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 1, 0],
[0, 0, 0, 0, 1]],
}
### Definition der Fahrezugtypen
data["vehicle_costs_type_per_km"] = [1, 0.01] #Kosten pro km pro Fahrzeugtyp
data["vehicle_costs_type_per_stop"] = [1, 1000] #Kosten pro Stop pro Fahrzeugtyp
data["vehicle_capacity_type"] = [100, 200] # Kapaität pro Fahrzeugtyp
data["num_vehicle_per_type"] = [math.ceil(summe_dem /data["vehicle_capacity_type"][0]), math.ceil(summe_dem /data["vehicle_capacity_type"][1])] # Anzahl Fahrzeuge pro Typ
data['price_per_km'] = veh_types_costs(data["vehicle_costs_type_per_km"], data["num_vehicle_per_type"], data["num_days"]) #Matrix für price_per_km je Fahrzeug ertsellen
data["price_per_stop"] = veh_types_costs(data["vehicle_costs_type_per_stop"], data["num_vehicle_per_type"], data["num_days"])
data['vehicle_capacities_matrix'] = veh_capacity_matrix(data["num_vehicle_per_type"], data["vehicle_capacity_type"], data["num_days"]) #Kapazitäten pro Fahrzeugs pro Tag
#print('vehicle_capacities_matrix')
#print(data['vehicle_capacities_matrix'])
data["num_vehicles_per_day"] = sum(data["num_vehicle_per_type"]) #Gesamtanzahl der Fahrzeuge pro Tag
data['num_vehicles'] = data["num_days"] * data["num_vehicles_per_day"]# Gesamtanzahl der Fahrzeuge in gesamten Zeitraum
###Distanzmatrix bestimmen
data["base_distance_matrix"] = dist_matrix(data["koordianten"])
data['distance_matrix'] = dublicate_nodes(data["base_distance_matrix"], data["num_days"]) # Distanzmatrix mit mehrfachen Nodes pro Kunden, je Tag ein Node
###Start und Ende festlegen
data["num_nodes"] = len(data["base_distance_matrix"]) # Anzahl Kunden
data['starts'] = depo_nodes(data["num_days"], data['num_vehicles'], data["num_nodes"]) #Deponodes (Start) für die einzelnen Fahrzeuge (Tage)
data['ends'] = depo_nodes(data["num_days"], data['num_vehicles'], data["num_nodes"]) #Deponodes (Ende) für die einzelnen Fahrzeuge (Tage)
###Demand pro Kunde
data["kunden_indizes"] = kunden_indizes(data["IDs"], data["num_days"])
data["demands_matrix"] = demand_matrix(demand_matrix_day=data["demand"], num_days=data["num_days"])
#print(data["demands_matrix"])
return data
def print_solution(data, manager, routing, solution):
"""Prints solution on console."""
total_distance = 0
total_load = 0
for vehicle_id in range(data['num_vehicles']):
if vehicle_id % data["num_vehicles_per_day"] == 0:
print(("------Tag {}------").format(int(vehicle_id/data["num_vehicles_per_day"])))
if routing.IsVehicleUsed(assignment= solution, vehicle=vehicle_id) == False:
continue
index = routing.Start(vehicle_id)
if vehicle_id >= data["num_vehicles_per_day"]:
plan_output = 'Route for vehicle {}:\n'.format(abs(vehicle_id-data["num_vehicles_per_day"]*int(vehicle_id/data["num_vehicles_per_day"])))
else:
plan_output = 'Route for vehicle {}:\n'.format(vehicle_id)
route_costs = 0
route_load = 0
while not routing.IsEnd(index):
node_index = manager.IndexToNode(index)
capacity_ID = int(vehicle_id/data["num_vehicles_per_day"])
route_load += data['demands_matrix'][capacity_ID][node_index]
plan_output += ' {0} Load({1}) -> '.format(node_index, route_load)
previous_index = index
index = solution.Value(routing.NextVar(index))
route_costs += routing.GetArcCostForVehicle(
previous_index, index, vehicle_id)
plan_output += ' {0} Load({1})\n'.format(manager.IndexToNode(index),
route_load)
plan_output += 'Costs of the route: {}€\n'.format(route_costs)
plan_output += 'Load of the route: {}\n'.format(route_load)
print(plan_output)
total_distance += route_costs
total_load += route_load
print('Total costs of all routes: {}€'.format(total_distance))
print('Total load of all routes: {}'.format(total_load))
def main(node_num):
"""Periodic CVRP problem."""
# Instantiate the data problem.
data = create_data_model(node_num)
# Create the routing index manager.
manager = pywrapcp.RoutingIndexManager(len(data['distance_matrix']), data['num_vehicles'], data['starts'],
data['ends'])
# Create Routing Model.
routing = pywrapcp.RoutingModel(manager)
### Kostenfunktion je Fahrzeug festlegen ###
def create_cost_callback(dist_matrix, km_costs, stop_costs):
# Create a callback to calculate distances between cities.
def distance_callback(from_index, to_index):
from_node = manager.IndexToNode(from_index)
to_node = manager.IndexToNode(to_index)
return int(dist_matrix[from_node][to_node]) * (km_costs) + (stop_costs)
return distance_callback
for i in range(data['num_vehicles']):
cost_callback = create_cost_callback(data['distance_matrix'], data["price_per_km"][i],
data["price_per_stop"][i]) # Callbackfunktion erstellen
cost_callback_index = routing.RegisterTransitCallback(cost_callback) # registrieren
routing.SetArcCostEvaluatorOfVehicle(cost_callback_index, i) # Vehicle zuordnen
#Define Capacities for Vehicles for every Day
def create_demand_callback(demand_matrix, demand_index):
# Create a callback to calculate capacity usage.
def demand_callback(from_index):
#Returns the demand of the node.
# Convert from routing variable Index to demands NodeIndex.
from_node = manager.IndexToNode(from_index)
return demand_matrix[demand_index][from_node]
return demand_callback
for i in range(data["num_days"]): #Jedes Fahrzeug hat pro Tag eine andere Kapazität
demand_callback = create_demand_callback(data['demands_matrix'], i)
demand_callback_index = routing.RegisterUnaryTransitCallback(demand_callback)
dimension_name = 'Capacity_day_{}'.format(i)
routing.AddDimensionWithVehicleCapacity(
demand_callback_index,
0, # null capacity slack
data['vehicle_capacities_matrix'][i], # vehicle maximum capacities
True, # start cumul to zero
dimension_name)
#Drop visits that would exceed the frequency
for index, key in enumerate(data["kunden_indizes"]):
if index == 0:
continue
#routing.solver().Add(sum(routing.ActiveVar(manager.NodeToIndex(i)) for i in data["kunden_indizes"][key]) == data["frequenz"][index])
bool_array = []
for index, freq in enumerate(data["frequenz"]):
if index == 0:
continue
bool_array_part =[]
for index_pat, pat in enumerate(data["PAT"][freq]):
#print(index,freq,index_pat)
bool_name = str(index) +str(freq) + str(index_pat)
bool_array_part.append(routing.solver().BoolVar(bool_name))
bool_array.append(bool_array_part)
#print(bool_array)
for i in bool_array:
routing.solver().Add(sum(i) == 1)
node_array = []
for index, freq in enumerate(data["frequenz"]):
if index == 0:
continue
node_array_part = []
for index_pat, pat in enumerate(data["PAT"][freq]):
node_array_sub_part = []
for index_day, day_value in enumerate(pat):
if day_value == 1:
node_array_sub_part.append(data["kunden_indizes"][index][index_day])
#print(node_array_part)
node_array_part.append(node_array_sub_part)
node_array.append(node_array_part)
#print(node_array)
for index, bool in enumerate(bool_array):
for i in range(len(bool)):
#print(node_array[index][i])
#print(bool_array[index][i])
routing.solver().Add(sum(routing.ActiveVar(manager.NodeToIndex(i)) for i in node_array[index][i]) * bool_array[index][i] == bool_array[index][i] * len(node_array[index][i]))
#routing.solver().Add(routing.ActiveVar(manager.NodeToIndex(5)) == 1)
penalty = 0
for node in range(0, len(data['distance_matrix'])):
if node in data["kunden_indizes"][0]:
continue
else:
routing.AddDisjunction([manager.NodeToIndex(node)], penalty)
# Setting first solution heuristic.
search_parameters = pywrapcp.DefaultRoutingSearchParameters()
search_parameters.first_solution_strategy = (
routing_enums_pb2.FirstSolutionStrategy.AUTOMATIC)
search_parameters.local_search_metaheuristic = (
routing_enums_pb2.LocalSearchMetaheuristic.AUTOMATIC)
#search_parameters.time_limit.FromSeconds(300)
# Solve the problem.
solution = routing.SolveWithParameters(search_parameters)
# Print solution on console.
if solution:
print_solution(data, manager, routing, solution)
print()
print("Statistics")
print(' - wall time : %f s' % (int(routing.solver().WallTime())/1000))
solvingtime = int(routing.solver().WallTime())/1000
return solvingtime
if __name__ == '__main__':
node_num_array = [10,20,30,40,50,60,70,80,90,100]
solvingtime_array = []
num_array =[]
for i in node_num_array:
for j in range(0,4):
solvingtime = main(i)
solvingtime_array.append(solvingtime)
num_array.append(i)
print(i, "-->",solvingtime)
print(solvingtime_array)
print(num_array)
df_results = pd.DataFrame(data= {"num_nodes":num_array, "solvingtime":solvingtime_array})
#encoding="latin_1", sep=";")
print(df_results)
I want to realize the bert model.
So I built a class with __getitem__ in it.
I can print something like test[0], but when I assign a value, like data = test[0], a KeyError occurs.
import random
"""
corpus_file = 'vocab'
vocab_size = 6
vocab_freq = 1
save_path = 'obj/'
max_sentence = 16
corpus -> org_line -> ope_line
corpus -> org_line -> token_list -> idx_to_token + token_to_idx
"""
class vocab():
def __init__(self, corpus_file, vocab_size, vocab_freq,save_path,max_sentence):
self.max_sentence = max_sentence
self.special_labels = ['PAD', 'UNK', 'SEP', 'CLS', 'MASK']
# output
self.data = []
self.idx_to_token = []
self.token_to_idx = {}
# ope
self.pre_ope(corpus_file,vocab_size,vocab_freq)
#self.save_data(save_path)
#self.print_data()
def pre_ope(self,corpus_file,vocab_size,vocab_freq):
token_list = {}
with open(corpus_file, 'r') as f:
while 1:
new_org_line = f.readline()
if new_org_line != '':
new_org_line = new_org_line.strip('\n')
new_sentence = new_org_line.split('\t')
sentence = []
for tmp in new_sentence:
token_sentence = tmp.split()
sentence.append(token_sentence)
for token in token_sentence:
if token_list.get(token):
token_list[token] += 1
else:
new_token = {token: 1}
token_list.update(new_token)
self.data.append(sentence)
else:
break
f.close()
token_list = sorted(token_list.items(), key=lambda i: (-i[1], i[0]))
self.build_dictionary(token_list,vocab_freq,vocab_size)
'''
Special labels:
PAD
UNK
SEP sentence separator
CLS classifier token
MASK
'''
def build_dictionary(self,token_list,vocab_freq,vocab_size):
for idx, label in enumerate(self.special_labels):
self.idx_to_token.append(label)
self.token_to_idx[label] = idx
for idx, (token, freq) in enumerate(token_list):
if freq >= vocab_freq :
self.idx_to_token.append(token)
self.token_to_idx[token] = idx + len(self.special_labels)
if len(self.idx_to_token) >= vocab_size + len(self.special_labels) and vocab_size != 0 :
break
def __len__(self):
return len(self.data)
def print_data(self):
print(self.data)
print(self.idx_to_token)
print(self.token_to_idx)
def __getitem__(self, item):
s1,s2,is_next_sentence = self.get_random_next_sentence(item)
s1,s1_label = self.get_random_sentence(s1)
s2,s2_label = self.get_random_sentence(s2)
sentence = [self.token_to_idx['CLS']] +s1 +[self.token_to_idx['SEP']] +s2 +[self.token_to_idx['SEP']]
label = [-1] +s1_label +[-1] +s2_label +[-1]
if len(sentence) > self.max_sentence :
print('sentence is greater than the setting of max sentence')
for pos in range(len(sentence),self.max_sentence):
sentence.append(self.token_to_idx['PAD'])
label.append(-1)
return {
'token' : sentence,
'label' : label,
'is_next' : is_next_sentence
}
def get_random_next_sentence(self,item):
s1 = self.data[item][0]
s2 = self.data[item][1]
if random.random() < 0.5 :
is_next = 0
s2 = self.data[self.get_random_line(item)][1]
else:
is_next = 1
return s1,s2,is_next
def get_random_line(self,item):
rand = random.randint(0,len(self.data)-1)
while rand == item :
rand = random.randint(0,len(self.data)-1)
return rand
def get_random_sentence(self,sentence):
label = []
for idx,token in enumerate(sentence):
rand = random.random()
if rand < 0.15:
rand = rand/0.15
if rand < 0.8: #mask
sentence[idx] = self.token_to_idx['MASK']
elif rand < 0.9: #rand
sentence[idx] = random.randint(len(self.special_labels),len(self.token_to_idx)-1)
else: # still
sentence[idx] = self.token_to_idx[token]
label.append(self.token_to_idx[token])
else:
sentence[idx] = self.token_to_idx[token]
label.append(-1)
return sentence,label
if __name__ == '__main__':
test = vocab('vocab', 0, 1,'obj/',16)
print(len(test))
print(test[0])
print(test[1])
data = test[0]
Result:
2
{'token': [3, 4, 18, 12, 15, 11, 2, 7, 9, 13, 2, 0, 0, 0, 0, 0], 'label': [-1, 10, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], 'is_next': 0}
{'token': [3, 6, 4, 5, 8, 5, 17, 2, 16, 5, 14, 20, 2, 0, 0, 0], 'label': [-1, -1, 19, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], 'is_next': 0}
Traceback (most recent call last):
File "vocab.py", line 146, in <module>
data = test[0]
File "vocab.py", line 90, in ```__getitem__```
s1,s1_label = self.get_random_sentence(s1)
File "vocab.py", line 136, in get_random_sentence
sentence[idx] = self.token_to_idx[token]
KeyError: 4
vocab file:
hello this is my home nice to meet you
I want to go to school and have lunch
change the code:
def get_random_next_sentence(self,item):
s1 = self.data[item][0]
s2 = self.data[item][1]
if random.random() < 0.5 :
is_next = 0
s2 = self.data[self.get_random_line(item)][1]
else:
is_next = 1
return s1,s2,is_next
to:
def get_random_next_sentence(self,item):
s1 = copy.deepcopy(self.data[item][0])
s2 = copy.deepcopy(self.data[item][1])
if random.random() < 0.5 :
is_next = 0
s2 = copy.deepcopy(self.data[self.get_random_line(item)][1])
print(s2)
else:
is_next = 1
return s1,s2,is_next
I am working on setting up a controls dashboard for operators. whenever a heartbeat stops incrementing, after 5 seconds, the LED changes to red. This also enables the acknowledge buttons. When I click them, the LED chages to yellow and I would like to disable the button again. The issue is that it only works for some of the buttons in my code for some reason. any idea why that might be? It only works for the buttons on the left side and not the right side. I am sure that the error is happening at "heartbeats[string][5].setEnabled(False)"
def ack_thread(string):
ack = threading.Thread(target=acknowledge, args=(string,))
ack.start()
def acknowledge(string):
global heartbeats
rowcount = ui.tableWidget.rowCount()
for x in range(0, rowcount, 1):
if ui.tableWidget.item(x,0).text() == string:
print(ui.tableWidget.item(x,0).text())
if ui.tableWidget.item(x,4).text() == 'False':
ui.tableWidget.setItem(x, 4, QtWidgets.QTableWidgetItem("True"))
ui.tableWidget.viewport().update()
heartbeats[string][3].setPixmap(ui.yellowled)
print(heartbeats[string][5])
# heartbeats[string][5].setEnabled(True)
heartbeats[string][5].setEnabled(False)
def gettime():
import datetime
currentDT = datetime.datetime.now()
return currentDT.strftime("%Y-%m-%d %H:%M:%S")
def alarm_handler(source):
rowPosition = ui.tableWidget.rowCount()
ui.tableWidget.insertRow(rowPosition)
ui.tableWidget.setItem(rowPosition, 0, QtWidgets.QTableWidgetItem(source))
ui.tableWidget.setItem(rowPosition, 1, QtWidgets.QTableWidgetItem("text1"))
ui.tableWidget.setItem(rowPosition, 2, QtWidgets.QTableWidgetItem("text2"))
ui.tableWidget.setItem(rowPosition, 3, QtWidgets.QTableWidgetItem(gettime()))
ui.tableWidget.setItem(rowPosition, 4, QtWidgets.QTableWidgetItem("False"))
def heartbeat():
import mhs2
from time import sleep
global heartbeats
mhs2.DB_CONFIG['host'] = '192.168.1.79'
# old value, new value, number of failures, LED object, checkbutton object, Acknowledge object,acknowledged
heartbeats = {
'pd1_hb': [0, 0, 0, ui.label_26, ui.checkBox_7, ui.pushButton_11, True],
'pd9_hb': [0, 0, 0, ui.label_20, ui.checkBox_12, ui.pushButton_13, True],
'pd11_hb': [0, 0, 0, ui.label_21, ui.checkBox_11, ui.pushButton_12, True],
'pd13_hb': [0, 0, 0, ui.label_22, ui.checkBox_13, ui.pushButton_2, True],
'irr1_hb': [0, 0, 0, ui.label_25, ui.checkBox_8, ui.pushButton_5, True],
'irr2_hb': [0, 0, 0, ui.label_24, ui.checkBox_9, ui.pushButton_4, True],
'irr5_hb': [0, 0, 0, ui.label_23, ui.checkBox_10, ui.pushButton_3, True],
'kyledata_hb': [0, 0, 0, ui.label_14, ui.checkBox, ui.pushButton, True],
'standata_hb': [0, 0, 0, ui.label_19, ui.checkBox_3, ui.pushButton_10, True],
'kennydata_hb': [0, 0, 0, ui.label_18, ui.checkBox_4, ui.pushButton_9, True],
'cartmandata_hb': [0, 0, 0, ui.label_17, ui.checkBox_6, ui.pushButton_8, True],
'chefdata_hb': [0, 0, 0, ui.label_16, ui.checkBox_5, ui.pushButton_6, True],
'buttarsdata_hb': [0, 0, 0, ui.label_15, ui.checkBox_2, ui.pushButton_7, True]
}
for item in heartbeats:
heartbeats[item][3].setPixmap(ui.offled)
query = "SELECT * FROM Heartbeat"
enable = mhs2.exec_query(query)
for row in enable:
if row['enable'] == 1:
heartbeats[row['parameter']][4].setChecked(True)
else:
heartbeats[row['parameter']][4].setChecked(False)
while 1:
command = "SELECT * FROM Heartbeat"
hb = mhs2.exec_query(command)
for row in hb:
if heartbeats[row['parameter']][4].isChecked():
heartbeats[row['parameter']][1] = row['value']
if heartbeats[row['parameter']][1] != heartbeats[row['parameter']][0]:
heartbeats[row['parameter']][2] = 0
heartbeats[row['parameter']][3].setPixmap(ui.greenled)
heartbeats[row['parameter']][6] = True
else:
heartbeats[row['parameter']][2] += 1
if heartbeats[row['parameter']][2] > 5 and heartbeats[row['parameter']][6]:
heartbeats[row['parameter']][3].setPixmap(ui.redled)
heartbeats[row['parameter']][5].setEnabled(True)
heartbeats[row['parameter']][6] = False
alarm = threading.Thread(target=alarm_handler,args=(row['parameter'],))
alarm.start()
heartbeats[row['parameter']][0] = heartbeats[row['parameter']][1]
else:
heartbeats[row['parameter']][3].setPixmap(ui.offled)
sleep(1)
if __name__ == "__main__":
import sys
app = QtWidgets.QApplication(sys.argv)
Dialog = QtWidgets.QDialog()
ui = Ui_Dialog()
ui.setupUi(Dialog)
Dialog.show()
hb = threading.Thread(target=heartbeat)
hb.start()
sys.exit(app.exec_())
So I have a little program which makes a DB call and then converts it into a PDF.
I have most of them working but this last one is returning a Key Error on me and I cannot figure out why.
Here is an example of the data being returned by the DB:
((None, 0, 0, 0, 0, 0, 0, 0), (0, 0, 0, 0, 0, 26, 0, 26), (1, 1, 0, 0, 0, 17, 0, 18), (2, 0, 0, 0, 0, 15, 0, 16))
The Traceback:
Traceback (most recent call last):
File "C:\Users\Ace\AppData\Local\Programs\Python\Python36\lib\tkinter\__init__.py", line 1699, in __call__
return self.func(*args)
File "C:/Users/Ace/Desktop/IPNV/KP_App/FML/firstapp.py", line 232, in hrday_in
hourday_filter(noodle, dest, drange)
File "C:\Users\Ace\Desktop\IPNV\KP_App\FML\dataIN.py", line 187, in hourday_filter
doc.export(pths, drange)
File "C:\Users\Ace\Desktop\IPNV\KP_App\FML\calazan.py", line 58, in export
reverse=reverse_order)
KeyError: 'h'
Im not even sure where the 'h' comes from.
Here is the function that I run the data through to prepare for the pdf generation:
def hourday_filter(tuna, pth, drange):
data = []
for hr, number, local, chicken, alligator, ace, lola, chunk in tuna:
data.append({'hour': hr,
'number': number,
'local': local,
'long': chicken,
'inter': alligator,
'income': ace,
'tandem': lola,
'total': chunk})
fields = (
('hour', 'Hour of Day'),
('number', 'Internal Calls '),
('local', 'Local Calls'),
('long', 'Long Distance Calls'),
('inter', 'International Calls '),
('income', 'Incoming Calls'),
('tandem', 'Tandem Calls'),
('total', 'Total Calls'),
)
pths = pth + '/HourofDay.pdf'
doc = DataToPdf(fields, data, sort_by='hr',
title='Hour of Day Report')
doc.export(pths, drange)
And From there the data is passed to this function to actually convert it too pdf.
class DataToPdf:
"""
Export a list of dictionaries to a table in a PDF file.
"""
def __init__(self, fields, data, sort_by=None, title=None):
"""
Arguments:
fields - A tuple of tuples ((fieldname/key, display_name))
specifying the fieldname/key and corresponding display
name for the table header.
data - The data to insert to the table formatted as a list of
dictionaries.
sort_by - A tuple (sort_key, sort_order) specifying which field
to sort by and the sort order ('ASC', 'DESC').
title - The title to display at the beginning of the document.
"""
self.fields = fields
self.data = data
self.title = title
self.sort_by = sort_by
def export(self, filename, drange, data_align='LEFT', table_halign='LEFT'):
doc = SimpleDocTemplate(filename, pagesize=letter)
styles = getSampleStyleSheet()
styleH = styles['Heading1']
styleD = styles['Heading4']
date = time.strftime("%m/%d/%Y")
date2 = 'Ran on: ' + date
date3 = ' For the period ' + str(drange[0]) + ' to ' + str(drange[1]) # Edit here to display report date range
story = []
if self.title:
story.append(Paragraph(self.title, styleH))
story.append(Spacer(1, 0.25 * inch))
story.append(Paragraph(date2, styleD))
story.append(Spacer(1, 0.015 * inch))
story.append(Paragraph(date3, styleD))
if self.sort_by:
reverse_order = False
if str(self.sort_by[1]).upper() == 'DESC':
reverse_order = False
self.data = sorted(self.data,
key=itemgetter(self.sort_by[0]),
reverse=reverse_order)
converted_data = self.__convert_data()
table = Table(converted_data, hAlign=table_halign)
table.setStyle(TableStyle([
('FONT', (0, 0), (-1, 0), 'Helvetica-Bold'),
('ALIGN', (0, 0), (-1, 0), 'CENTER'),
('ALIGN', (0, 0), (0, -1), data_align),
('INNERGRID', (0, 0), (-1, -1), 0.50, colors.black),
('BOX', (0, 0), (-1, -1), 0.25, colors.black),
]))
data_len = len(converted_data)
for each in range(data_len):
if each % 2 == 0:
bg_color = colors.whitesmoke
else:
bg_color = colors.lightgrey
table.setStyle(TableStyle([('BACKGROUND', (0, each), (-1, each), bg_color)]))
story.append(table)
doc.build(story)
def __convert_data(self):
"""
Convert the list of dictionaries to a list of list to create
the PDF table.
"""
# Create 2 separate lists in the same order: one for the
# list of keys and the other for the names to display in the
# table header.
keys, names = zip(*[[k, n] for k, n in self.fields])
new_data = [names]
for d in self.data:
new_data.append([d[k] for k in keys])
return new_data
Is it possible that first result of the db (the null one) is causing this? I've made about a dozen of these reports now with no problems, not sure where I am messing up here.
So thanks to FamousJameous I realized that it was indeed the first field killing my filter. The sort_by call did not know how to deal with the NULL value. I managed to fix it by removing that first field from the tuple.
The database returned the results into the variable result
From there:
new_result = result[1:]
This line removed the NULL line and stopped the error
I'm reading xls file using xlrd. The problem is, when xlrd reading value like this "12/09/2012", i get result like this "xldate:41252.0". When I use xlrd.xldate_as_tuple, i get this result:
(2016, 12, 10, 0, 0, 0)
My code:
curr_row = -1
while curr_row < num_rows:
curr_row += 1
row = worksheet.row(curr_row)
for x in xrange(num_cols):
field_type = worksheet.cell_type(curr_row, x)
if field_type == 3: # this is date
field_value = worksheet.cell_value(curr_row, x)
print worksheet.cell(curr_row, x).value
print xlrd.xldate_as_tuple(field_value, 1)
Result:
41252.0
(2016, 12, 10, 0, 0, 0)
Both results are wrong for me. How can i get original cell value "12/09/2012" using xlrd ?
According to the docstring, you should pass your workbook's datemode to xldate_as_tuple as a second parameter:
from datetime import datetime
import xlrd
book = xlrd.open_workbook("test.xls")
sheet = book.sheet_by_index(0)
a1 = sheet.cell_value(rowx=0, colx=0)
print a1 # prints 41252.0
print xlrd.xldate_as_tuple(a1, 1) # prints (2016, 12, 10, 0, 0, 0)
a1_tuple = xlrd.xldate_as_tuple(a1, book.datemode)
print a1_tuple # prints (2012, 12, 9, 0, 0, 0)
a1_datetime = datetime(*a1_tuple)
print a1_datetime.strftime("%m/%d/%Y") # prints 12/09/2012