We Keep Coding

Include Only Unique Values in a PuLP Optimization Solution

This post is a related question that spun off of this question.
My goal is to generate an optimal fantasy football lineup given a set of constraints. Due to the nature of the fantasy football league, it is possible that I have the same player multiple times in my inventory, although I can only play one instance of that player in my lineup. I have created an optimization problem using PuLP to generate the lineup, but I need to add a constraint to ensure that a player does not appear in that lineup more than once.
Below is my current model. Any suggestions on how to ensure no duplicate players in my solution are appreciated!
import pandas as pd
import pulp
print('--- (1/4) Defining the problem ---')
# Read csv
raw_data = pd.read_csv('./csv/fantasypros.csv')
# create new columns that has binary numbers if player == a specific position
encoded = pd.get_dummies(raw_data['Pos. Parent']) # <-- One-Hote Encoding
raw_data = raw_data.join(encoded) # <-- joining it to the raw_data table
raw_data["salary"] = raw_data["Point Cost"].astype(float)
model = pulp.LpProblem("NFTdraft", pulp.LpMaximize)
total_points = {}
cost = {}
qb = {}
rb = {}
wr = {}
te = {}
k = {}
dst = {}
dk = {}
num_players = {}
vars_list = []
# i = row index, player = player attributes
for i, player in raw_data.iterrows():
var_name = 'x' + str(i) # Create variable name
decision_var = pulp.LpVariable(var_name, cat='Binary') # Initialize Variables
total_points[decision_var] = player["FPTS"] # Create FPTS Dictionary
cost[decision_var] = player["salary"] # Create Cost Dictionary
# Create Dictionary for Player Types
qb[decision_var] = player["QB"]
rb[decision_var] = player["RB"]
wr[decision_var] = player["WR"]
te[decision_var] = player["TE"]
k[decision_var] = player["K"]
dst[decision_var] = player["DST"]
dk[decision_var] = player["DK"]
num_players[decision_var] = 1.0
objective_function = pulp.LpAffineExpression(total_points)
model += objective_function
total_cost = pulp.LpAffineExpression(cost)
model += (total_cost <= 135)
print('--- (2/4) Defining the constraints ---')
QB_constraint = pulp.LpAffineExpression(qb)
RB_constraint = pulp.LpAffineExpression(rb)
WR_constraint = pulp.LpAffineExpression(wr)
TE_constraint = pulp.LpAffineExpression(te)
K_constraint = pulp.LpAffineExpression(k)
DST_constraint = pulp.LpAffineExpression(dst)
DK_constraint = pulp.LpAffineExpression(dk)
total_players = pulp.LpAffineExpression(num_players)
model += (QB_constraint >= 1)
model += (QB_constraint <= 2)
model += (RB_constraint <= 8)
model += (WR_constraint <= 8)
model += (TE_constraint <= 8)
model += (K_constraint <= 1)
model += (DST_constraint <= 1)
model += (DK_constraint <= 2)
model += (total_players == 10)
print('--- (3/4) Solving the problem ---')
model.solve()
print('--- (4/4) Formatting the results ---')
raw_data["is_drafted"] = 0.0
for var in model.variables():
raw_data.loc[int(var.name[1:]), 'is_drafted'] = var.varValue # <--- CHANGED HERE
my_team = raw_data[raw_data["is_drafted"] == 1.0]
my_team = my_team[["Asset Name", "Player", "Pos. Parent", "Rarity", "Point Cost", "FPTS"]]
print(my_team)
print("Total used amount of salary cap: {}".format(my_team["Point Cost"].sum()))
print("Projected points: {}".format(my_team["FPTS"].sum().round(1)))
print('--- Completed ---')

To set the restriction to only allow single player (from your dataframe of duplicate players) you'll need to setup something similar to the other restrictions). Here we'll just one-hot encode on "Player", then loop through that to create the restriction that you have to have 1 or less of each "Player" name.
import pandas as pd
import pulp
print('--- (1/4) Defining the problem ---')
# Read csv
raw_data = pd.read_csv('./csv/fantasypros.csv').drop_duplicates().reset_index(drop=True)
# create new columns that has binary numbers if player == a specific position
encoded = pd.get_dummies(raw_data['Pos. Parent']) #<-- One-Hote Encoding
raw_data = raw_data.join(encoded) #<-- joining it to the raw_data table
# Will be used to create the constraint of not using same player name in lineup
encoded = pd.get_dummies(raw_data['Player']) #<-- One-Hote Encoding
raw_data = raw_data.join(encoded) #<-- joining it to the raw_data table
raw_data["salary"] = raw_data["Point Cost"].astype(float)
model = pulp.LpProblem("NFTdraft", pulp.LpMaximize)
total_points = {}
cost = {}
qb = {}
rb = {}
wr = {}
te = {}
k = {}
dst = {}
dk = {}
num_players = {}
# Here I created a list of all the possible players in the dataframe
# This is used later to construct the dictionary of players, then
# to add each of those into the model
players_list = list(raw_data['Player'].unique())
players_list.sort()
pLAYER_dict = {}
for player in players_list:
pLAYER_dict[player] ={}
vars_list = []
# i = row index, player = player attributes
for i, player in raw_data.iterrows():
#print('Processing row: %s of %s' %((i+1),len(raw_data)))
var_name = 'x' + str(i) # Create variable name
decision_var = pulp.LpVariable(var_name, cat='Binary') # Initialize Variables
total_points[decision_var] = player["FPTS"] # Create FPTS Dictionary
cost[decision_var] = player["salary"] # Create Cost Dictionary
# Create Dictionary for Player Types
qb[decision_var] = player["QB"]
rb[decision_var] = player["RB"]
wr[decision_var] = player["WR"]
te[decision_var] = player["TE"]
k[decision_var] = player["K"]
dst[decision_var] = player["DST"]
dk[decision_var] = player["DK"]
num_players[decision_var] = 1.0
# Here is where I store each value for each player name for the player
for key, v in PLAYER_dict.items():
PLAYER_dict[key].update({decision_var:player[key]})
objective_function = pulp.LpAffineExpression(total_points)
model += objective_function
total_cost = pulp.LpAffineExpression(cost)
model += (total_cost <= 135)
print('--- (2/4) Defining the constraints ---')
QB_constraint = pulp.LpAffineExpression(qb)
RB_constraint = pulp.LpAffineExpression(rb)
WR_constraint = pulp.LpAffineExpression(wr)
TE_constraint = pulp.LpAffineExpression(te)
K_constraint = pulp.LpAffineExpression(k)
DST_constraint = pulp.LpAffineExpression(dst)
DK_constraint = pulp.LpAffineExpression(dk)
total_players = pulp.LpAffineExpression(num_players)
model += (QB_constraint >= 1)
model += (QB_constraint <= 2)
model += (RB_constraint <= 8)
model += (WR_constraint <= 8)
model += (TE_constraint <= 8)
model += (K_constraint <= 1)
model += (DST_constraint <= 1)
model += (DK_constraint <= 2)
model += (total_players == 10)
for k1, v1 in PLAYER_dict.items():
player_constraint = pulp.LpAffineExpression(v1)
model += (player_constraint <= 1)
print('--- (3/4) Solving the problem ---')
model.solve()
print('--- (4/4) Formatting the results ---')
raw_data["is_drafted"] = 0.0
for var in model.variables():
raw_data.loc[int(var.name[1:]), 'is_drafted'] = var.varValue # <--- CHANGED HERE
my_team = raw_data[raw_data["is_drafted"] == 1.0]
my_team = my_team[["Asset Name", "Player", "Pos. Parent", "Rarity", "Point Cost", "FPTS"]]
print(my_team)
print("Total used amount of salary cap: {}".format(my_team["Point Cost"].sum()))
print("Projected points: {}".format(my_team["FPTS"].sum().round(1)))
print('--- Completed ---')
Output:
Asset Name Player ... Point Cost FPTS
59 Arizona WR #1 DeAndre Hopkins ... 19 20.301
375 Carolina RB #1 Christian McCaffrey ... 20 26.500
582 Cincinnati WR #3 Tyler Boyd ... 10 13.000
803 Denver RB #2 Javonte Williams ... 8 11.100
1011 Green Bay WR #4 Randall Cobb ... 5 8.800
1170 Indianapolis QB #2 Jacob Eason ... 5 11.400
1301 Kansas City QB #1 Patrick Mahomes II ... 20 23.900
1349 Kansas City WR #1 Tyreek Hill ... 20 21.100
1658 Minnesota RB #1 Dalvin Cook ... 20 22.500
2729 Washington WR #2 Curtis Samuel ... 8 11.700
[10 rows x 6 columns]
Total used amount of salary cap: 135
Projected points: 170.3
--- Completed ---

Movie File parse file into a dictionary of the form

1.6. Recommend a Movie
Create a function that counts how many keywords are similar in a set of movie reviews
and recommend the movie with the most similar number of keywords.
The solution to this task will require the use of dictionaries.
The film reviews & keywords are in a file called film_reviews.txt, separated by commas.
The first term is the movie name, the remaining terms are the film’s keyword tags (i.e.,
“amazing", “poetic", “scary", etc.).
Function name: similar_movie()
Parameters/arguments: name of a movie
Returns: a list of movies similar to the movie passed as an argument
film_reviews.txt -
7 Days in Entebbe,fun,foreign,sad,boring,slow,romance
12 Strong,war,violence,foreign,sad,action,romance,bloody
A Fantastic Woman,fun,foreign,sad,romance
A Wrinkle in Time,book,witty,historical,boring,slow,romance
Acts of Violence,war,violence,historical,action
Annihilation,fun,war,violence,gore,action
Armed,foreign,sad,war,violence,cgi,fancy,action,bloody
Black '47,fun,clever,witty,boring,slow,action,bloody
Black Panther,war,violence,comicbook,expensive,action,bloody

I think this could work for you
film_data = {'films': {}}
with open('film_reviews.txt', 'r') as f:
for line in f.readlines():
data = line.split(',')
data[-1] = data[-1].strip() # removing new line character
film_data['films'][data[0].lower()] = data[1:]
def get_smilar_movie(name):
if name.lower() in film_data['films'].keys():
original_review = film_data['films'][name.lower()]
similarities = dict()
for key in film_data['films']:
if key == name.lower():
continue
else:
similar_movie_review = set(film_data['films'][key])
overlap = set(original_review) & similar_movie_review
universe = set(original_review) | similar_movie_review
# % of overlap compared to the first movie = output1
output1 = float(len(overlap)) / len(set(original_review)) * 100
# % of overlap compared to the second movie = output2
output2 = float(len(overlap)) / len(similar_movie_review) * 100
# % of overlap compared to universe
output3 = float(len(overlap)) / len(universe) * 100
similarities[output1 + output2 + output3] = dict()
similarities[output1 + output2 + output3]['reviews'] = film_data['films'][key]
similarities[output1 + output2 + output3]['movie'] = key
max_similarity = max(similarities.keys())
movie2 = similarities[max_similarity]
print(name,' reviews ',film_data['films'][name.lower()])
print('similar movie ',movie2)
print('Similarity = {0:.2f}/100'.format(max_similarity/3))
return movie2['movie']
return None
The get_similar_movie function will return the most similar movie from the film_data dict. The function will take a movie name as argument.

How can I optimise in term of time this python code

I write this code but I find it very slow and I don't know how to really improve it in term of time. data is a json object with approximately 70 000 key in it. I think the slowest part is the actors part because i'm iterating on a list (which contain at most 3 elements).
genres_number = {}
actors_number = {}
for movie in data:
for genre in data[movie]["genres"]:
if data[movie]["actors"] != None:
for actor in data[movie]["actors"]:
if actor not in actors_number.keys():
actors_number[actor] = 1
else:
actors_number[actor] = actors_number[actor] + 1
if genre not in genres_number.keys():
genres_number[genre] = 1
else:
genres_number[genre] = genres_number[genre] + 1
res = []
res.append(genres_number)
res.append(actors_number)
return res

How does this work for you
from collections import defaultdict
def get_stats(data):
genres_number = defaultdict(int)
actors_number = defaultdict(int)
for movie in data:
actors = movie.get('actors')
if actors:
for actor in actors:
actors_number[actor] += 1
genres = movie.get('genres')
for genre in genres:
genres_number[actor] += 1
res = []
res.append(dict(genres_number))
res.append(dict(actors_number))
return res

Finding total and average salaries from 1000 lines of text

I need to write a python program that reads 1000 lines of text and take the total and average of the salaries of each professor category (assistant, associate, full, all faculty). Here is the problem:
A university posts its employee salary at
http://cs.armstrong.edu/liang/data/Salary.txt. Each line in the file
consists of faculty first name, last name, rank, and salary (see
Exercise 13.16). Write a program to display the total salary for
assistant professors, associate professors, full professors, and all
faculty, respectively, and display the average salary for assistant
professors, associate professors, full professors, and all faculty,
respectively.
This is the code I have so far:
import urllib.request
def main():
infile = urllib.request.urlopen('http://cs.armstrong.edu/liang/data/Salary.txt')
s = infile.read().decode().splitlines()
asstTotal = 0
asstCount = 0
asscTotal = 0
asscCount = 0
fullTotal = 0
fullCount = 0
for line in s:
if "assistant" in line:
asstCount += 1
asstTotal += int(float(s[3]))
if "associate" in line:
asscCount += 1
asscTotal += int(float(s[3]))
if "full" in line:
fullCount += 1
fullTotal += int(float(s[3]))
allFacTotal = asstTotal + asscTotal + fullTotal
avgAsst = int(asstTotal / asstCount)
avgAssc = int(asscTotal / asscCount)
avgFull = int(fullTotal / fullCount)
avgAll = int(allFacTotal / 1000)
print("Total assistant professor salary is", asstTotal)
print("Total associate professor salary is", asscTotal)
print("Total full professor salary is", fullTotal)
print("Total faculty salary is", allFacTotal)
print("Average assistant professor salary is", avgAsst)
print("Average associate professor salary is", avgAssc)
print("Average full professor salary is", avgFull)
print("Average faculty salary is", avgAll)
main()
I'm not sure how to get the salaries from the respective professor ranks and totaling and averaging them. Any input would be much appreciated!
Result:
Total assistant professor salary is 24306418
Total associate professor salary is 27235856
Total full professor salary is 27631726
Total faculty salary is 79174000
Average assistant professor salary is 79174
Average associate professor salary is 79174
Average full professor salary is 79174
Average faculty salary is 79174

This can be done to account for any type of rank, by building up a list of pay for each position in a dictionary, then simply dividing the sum by the length.
My example:
import urllib.request
def main():
infile = urllib.request.urlopen('http://cs.armstrong.edu/liang/data/Salary.txt')
s = infile.read().decode().splitlines()
pays = {}
for line in s:
*name, rank, pay = line.split()
pays[rank] = pays.get(rank, []) + [float(pay)]
for rank, sals in pays.items():
avg = sum(sals) / len(sals)
print("{:>10} ({:3}): ${:,.2f}".format(rank.title(), len(sals), avg))
if __name__ == '__main__':
main()
And the output:
Associate (344): $83,849.26
Full (349): $102,229.37
Assistant (307): $65,949.55

import urllib.request
def main():
infile = urllib.request.urlopen("http://cs.armstrong.edu/liang/data/Salary.txt")
asstSalTotal = 0
asstTotal = 0
asscSalTotal = 0
asscTotal = 0
fullSalTotal = 0
fullTotal = 0
for line in infile.readlines():
data = line.decode().split()
designation = data[2]
if designation == "assistant":
asstSalTotal += float(data[3])
asstTotal += 1
elif designation == "associate":
asscSalTotal += float(data[3])
asscTotal += 1
elif designation == "full":
fullSalTotal += float(data[3])
fullTotal += 1
allSalTotal = asstSalTotal + asscSalTotal + fullSalTotal
allTotal = asstTotal + asscTotal + fullTotal
print("Total Salary:\n\tAssistant:\t%d\n\tAssociate:\t%d\n\tFull:\t%d\n\tAll:\t%d\n\nAvg Salary:\n\tAssistant:\t%d\n\tAssociate:\t%d\n\tFull:\t%d\n\tAll:\t%d\n" % (asstSalTotal, asscSalTotal, fullSalTotal, allSalTotal, asstTotal, asscTotal, fullTotal, allTotal))
main()

In python2.7, use urllib2.urlopen and group salaries together by title with a table.
In python3, use urllib.request.urlopen and change print .. to print(..)
import urllib2
rs = urllib2.urlopen('http://cs.armstrong.edu/liang/data/Salary.txt')
title_salaries = {}
for line in rs.read().split("\n"):
_, _, title, salary = line.split(' ')
if not title_salaries.has_key(title):
title_salaries[title] = []
title_salaries[title].append(float(salary))
for title, salaries in title_salaries.items():
print title
print 'avg:', sum(salaries) / len(salaries)
print 'total:', sum(salaries)
print

Independent arrays interfered with each other?

There's a given data set of two columns: EmployeeCode and Surname.
The format is like:
EmployeeCode[1] = "L001"
Surname[1] = "Pollard"
EmployeeCode[2] = "L002"
Surname[2] = "Wills"
...
What I was trying to do is to sort according to lexicographic order for each column so as to facilitate implementation of binary search later on.
This is my code:
#data set
EmployeeCode, Surname = [0]*33, [0]*33
EmployeeCode[1] = "L001"
Surname[1] = "Pollard"
EmployeeCode[2] = "L002"
Surname[2] = "Wills"
EmployeeCode[3] = "L007"
Surname[3] = "Singh"
EmployeeCode[4] = "L008"
Surname[4] = "Yallop"
EmployeeCode[5] = "L009"
Surname[5] = "Adams"
EmployeeCode[6] = "L013"
Surname[6] = "Davies"
EmployeeCode[7] = "L014"
Surname[7] = "Patel"
EmployeeCode[8] = "L021"
Surname[8] = "Kelly"
EmployeeCode[9] = "S001"
Surname[9] = "Ong"
EmployeeCode[10] = "S002"
Surname[10] = "Goh"
EmployeeCode[11] = "S003"
Surname[11] = "Ong"
EmployeeCode[12] = "S004"
Surname[12] = "Ang"
EmployeeCode[13] = "S005"
Surname[13] = "Wong"
EmployeeCode[14] = "S006"
Surname[14] = "Teo"
EmployeeCode[15] = "S007"
Surname[15] = "Ho"
EmployeeCode[16] = "S008"
Surname[16] = "Chong"
EmployeeCode[17] = "S009"
Surname[17] = "Low"
EmployeeCode[18] = "S010"
Surname[18] = "Sim"
EmployeeCode[19] = "S011"
Surname[19] = "Tay"
EmployeeCode[20] = "S012"
Surname[20] = "Tay"
EmployeeCode[21] = "S013"
Surname[21] = "Chia"
EmployeeCode[22] = "S014"
Surname[22] = "Tan"
EmployeeCode[23] = "S015"
Surname[23] = "Yeo"
EmployeeCode[24] = "S016"
Surname[24] = "Lim"
EmployeeCode[25] = "S017"
Surname[25] = "Tan"
EmployeeCode[26] = "S018"
Surname[26] = "Ng"
EmployeeCode[27] = "S018"
Surname[27] = "Lim"
EmployeeCode[28] = "S019"
Surname[28] = "Toh"
EmployeeCode[29] = "N011"
Surname[29] = "Morris"
EmployeeCode[30] = "N013"
Surname[30] = "Williams"
EmployeeCode[31] = "N016"
Surname[31] = "Chua"
EmployeeCode[32] = "N023"
Surname[32] = "Wong"
#sort based on value of main array
def bubble_sort(main, second):
sort = True
passed = len(main)-1
while sort:
sort = False
i = 2
while i<= passed:
#print(main[i],main[i-1],i)
if main[i] < main[i-1]:
main[i], main[i-1] = main[i-1], main[i]
second[i], second[i-1] = second[i-1], second[i]
sort = True
i += 1
passed -= 1
return main,second
#main
#prepare sorted array for binary search
#for search by surname, sort according to surname
sName,sCode = bubble_sort(Surname,EmployeeCode)
print("**BEFORE******")
for k in range(0,33):
print(sName[k],sCode[k])
print("*BEFORE*******")
#for search by ECode, sort according to ECode
cCode,cName = bubble_sort(EmployeeCode, Surname)
print("**AFTER******")
for k in range(0,33):
print(sName[k],sCode[k])
print("**AFTER******")
However, after the 2nd time sorting, the 1st time sorting result in sName and sCode just changed by themselves. I've never manually changed it.
BEFORE(1st sorting)
**BEFORE******
0 0
Adams L009
Ang S004
Chia S013
Chong S008
Chua N016
Davies L013
Goh S002
Ho S007
Kelly L021
Lim S016
Lim S018
Low S009
Morris N011
Ng S018
Ong S001
Ong S003
Patel L014
Pollard L001
Sim S010
Singh L007
Tan S014
Tan S017
Tay S011
Tay S012
Teo S006
Toh S019
Williams N013
Wills L002
Wong S005
Wong N023
Yallop L008
Yeo S015
*BEFORE*******
AFTER(2nd sorting, see last 4 items)
**AFTER******
0 0
Pollard L001
Wills L002
Singh L007
Yallop L008
Adams L009
Davies L013
Patel L014
Kelly L021
Morris N011
Williams N013
Chua N016
Wong N023
Ong S001
Goh S002
Ong S003
Ang S004
Wong S005
Teo S006
Ho S007
Chong S008
Low S009
Sim S010
Tay S011
Tay S012
Chia S013
Tan S014
Yeo S015
Lim S016
Tan S017
Lim S018
Ng S018
Toh S019
Can anyone tell me how could this happened?

Assignments and argument passing in Python won't ever create copies of objects. When you bubblesort your lists, the lists you pass into the bubble sort are the same exact list objects in memory as the lists the bubble sort passes back out. Surname, sName, and cName are the exact same objects, and when you do the second bubble sort, you modify sName and sCode instead of creating independent, sorted lists.
If you want to copy a list, you have to do so explicitly. This is a shallow copy:
new_list = original[:]
new_list will be a new list containing the same objects original contained.
This is a deep copy:
import copy
new_list = copy.deepcopy(original)
new_list will be a new list containing deep copies of the objects original contained. (This is sometimes too deep; for example, if you have a list of lists, you sometimes don't want to copy the objects inside the inner lists.)
Finally, I'd like to note that your initialization code is painfully verbose. You can use a list literal instead of creating a list full of zeros and assigning each element separately:
EmployeeCode = [
'Pollard',
'Wills',
...
]