We Keep Coding

Define the reward function to minimize costs

I have a problem with my reinforcement learning model. I am trying to simulate an electric battery storage.
The battery charges when the electricity prices are low and discharge ONLY to the user at fixed hours during the day, every day.
Therefore, the only cost for the user is power of charge * electricity price at the hour.
The reward function is set as the opposite of the cumulative sum of the cost.
Is it a correct approach? How to properly define it so that the overall cost of the purchased electricity is at minimum at the end of the year?
The problem that I have is that the battery will always near the maximum capacity and never fully take advantage of the full range of MWh available.
1. Define a dataframe where to store fictitious electricity prices for 365 days
df=pd.DataFrame(np.random.randint(0,500,size=(24, 365)))
2. Define the main parameters
Lookback_window_size=7
Current_day=Lookback_window_size
P_charge=2 #MW
P_discharge=3 #MW
3. Define the class Battery(Env)
class Battery(Env):
metadata = {'render.modes': ['human']}
def __init__(self, df):
#Import the dataframe
self.df = df
# The action space is a 1D array of shape (24,). Since we are simulating day-ahead market, the action space returns
# the overall daily charge / no charge scenario
# action = 1 means that we charge our battery, action = 0 means that we don't charge
self.action_space= spaces.MultiBinary(24)
# The observation space is a 1D array. Given a lookback window size of 1 day, then The first 48 columns represent
# the electricity prices for the current day + all the days before included in the lookback window size.
# The last two columns store SOC (state of charge) at the end of the day and overall cost
# (how much we paid for electricity).
self.observation_shape=(int((Lookback_window_size+1)*24+2),)
self.observation_space = spaces.Box(low = 0, high=np.inf, shape=self.observation_shape, dtype=np.float64)
def _next_observation(self):
# Add the prices of the last days to the monitor matrix
prices=[]
for i in range(self.Current_day - Lookback_window_size,self.Current_day + 1):
prices=np.concatenate([prices,self.df.iloc[0:,i].values])
# Add extra values to monitor such as SOC, cost and day of the week (Monday=1,Tuesday=2,etc.)
extra = [self.SOC, self.Cost]
obs=np.concatenate([prices,extra])
return obs
def _take_action(self, action):
# Being the action space an array, the for loop will check the action at every hour (action[i]) and update the
# cost and the state of charge
self.capacity=200 #MWh
i=0
for x in action:
#When action = 1 then we charge our battery, if action = 0 then we don't charge
if x == 1:
# The cost increase based on the price of the electricity at that hour
self.Cost+=self.df[self.Current_day][i]*P_charge
# If we charge, then the state of charge (SOC) increases as well
self.SOC+=P_charge
# Everyday we discharge the battery always at the same hours
if (i in range(8,14)):
self.SOC-=P_discharge
# if the battery is depleted, then we directly buy electricity from the grid
if self.SOC<0:
self.Cost+=self.df[self.Current_day][i+1]*(-self.SOC)
self.SOC=0
#the battery cannot charge above the capacity threshold.
if self.capacity is not None:
if self.SOC > self.capacity:
# We subtract the latest cost. Since it could not have happened being the SOC above the maximum.
self.Cost-=self.df[self.Current_day][i]*P_charge
# The capacity needs to be set to the threshold
self.SOC = min(self.SOC, self.capacity)
i+=1
def step(self, action):
# Execute one time step within the environment
self._take_action(action)
self.Current_day += 1
# Maximizing the reward means to minimize the costs
reward = - self.Cost
# Stop at the end of the dataframe
done = self.Current_day >= len(self.df.columns)-1
obs = self._next_observation()
return obs, reward, done, {}
def render(self, mode='human', close=False):
print(f'Day: {self.Current_day}')
print(f'SOC: {self.SOC}')
print(f'Cost: {self.Cost}')
print(f'Actions: {action}')
def reset(self):
self.Current_day = Lookback_window_size
# Give an initial SOC value
self.SOC = 50
# Cost at day 0 is null
self.Cost = 0
return self._next_observation()

MIP using PULP not approaching result

I am trying to solve a MIP problem. I am trying to find the number of exams to be done by each tech on a date for a week by minimizing the total number of techs used.
I have demand, time taken by each tech, list of techs etc. in separate dataframes.
Initially, I was using the cost function as minimizing the total time used to finish demand which #kabdulla helped me solve, linkhere!
Now, with the new cost function, the script gets stuck and doesn't seem to converge and I am not able to identify the reason.
Below is my code so far:
# Instantiate problem class
model = pulp.LpProblem("Time minimizing problem", pulp.LpMinimize)
capacity = pulp.LpVariable.dicts("capacity",
((examdate , techname, region) for examdate, techname, region in tech_data_new.index),
lowBound=0,
cat='Integer')
tech_used = pulp.LpVariable.dicts("techs",
((examdate,techname) for examdate,techname,region in tech_data_new.index.unique()),
cat='Binary')
model += pulp.lpSum(tech_used[examdate, techname] for examdate,techname in date_techname_index.index.unique())
for date in demand_data.index.get_level_values('Exam Date').unique():
for i in demand_data.loc[date].index.tolist():
model += pulp.lpSum([capacity[examdate,techname,region] for examdate, techname, region in tech_data_new.index if (date == examdate and i == region)]) == demand_data.loc[(demand_data.index.get_level_values('Exam Date') == date) & (demand_data.index.get_level_values('Body Region') == i), shiftname].item()
for examdate, techname,region in tech_data_new.index:
model += (capacity[examdate, techname, region]) <= tech_data_new.loc[(examdate,techname,region), 'Max Capacity']*tech_used[examdate, techname]
# Number of techs used in a day should be less than 8
for examdate in tech_data_new.index.get_level_values('Exam Date').unique():
model += pulp.lpSum(tech_used[examdate, techname] for techname in tech_data_new.index.get_level_values('Technologist Name').unique()) <=8
# Max time each tech should work in a day should be less than 8 hours(28800 secs)
for date in tech_data_new.index.get_level_values('Exam Date').unique():
for name in tech_data_new.loc[date].index.get_level_values('Technologist Name').unique():
#print(name)
model += pulp.lpSum(capacity[examdate,techname,region] * tech_data_new.loc[(examdate,techname,region), 'Time taken'] for examdate, techname, region in tech_data_new.index if (date == examdate and name == techname)) <= 28800
The last condition seems to be the problem, if I remove it, the problem converges. However, I am not able to understand the problem.
Please let me know, what I am missing in my understanding. Thanks.

Post Processing Aggregates

I have a question for a data analysis question where I need to separate data into two groups: months and states.
The task involves as:
In any given month, the minimum monthly rainfall for each state is the lowest rainfall recording from any weather station in that state during that month.
For example, suppose NSW had recordings of 1, 10, 5, and 7 for January; then its minimum monthly rainfall for January would be , the lowest of those recordings.
If there are several correct answers, your program can output any of the correct answers.
On the given data, the program's output should look like this:
Month: 11
State: QLD
What I have tried is:
minimum_monthly_rainfall = {}
is_first_line = True
for row in open("climate_data_2017.csv"):
if is_first_line:
is_first_line = False
else:
values = row.split(",")
month = values[0].split("-")[1]
rainfall = float(values[6])
state = values [1]
if month in minimum_monthly_rainfall:
minimum_monthly_rainfall[month].append(rainfall)
else:
minimum_monthly_rainfall[month] = [rainfall]
print(minimum_monthly_rainfall)
I am not sure as what to do now. I have skipped the first line of the data file which i will show below, and I have discriminated all the monthly rainfall into corresponding months. However I am not sure how I will separate this back into months? I know I shoul
The data file is:
Date,State,City,Station Code,Minimum temperature (C),Maximum temperature (C),Rainfall (mm),Evaporation (mm),Sunshine (hours),Direction of maximum wind gust,Speed of maximum wind gust (km/h),9am Temperature (C),9am relative humidity (%),3pm Temperature (C),3pm relative humidity (%)
2017-06-03,VIC,Melbourne,86338,9.9,14.3,0,,,SSW,19,10.8,76,13.1,73
2017-06-04,NSW,Sydney,66062,10.5,19.7,1.4,0.6,6.8,W,28,12.2,77,19,57
2017-06-04,QLD,Brisbane,40913,12.8,22.6,0.6,1.8,9.1,ENE,31,16.3,77,22.5,52
2017-06-04,VIC,Melbourne,86338,4.4,15.8,0,,,NE,13,5.7,100,15.5,53
2017-06-05,NSW,Sydney,66062,9.4,19.6,0.5,2.8,9.4,W,28,11.5,78,19.2,41
2017-06-05,QLD,Brisbane,40913,10.1,25.3,0.6,1.8,9.2,ENE,31,15,71,24.9,30
2017-06-05,VIC,Melbourne,86338,5.6,17.5,0,,,NW,28,8.7,82,16.8,57
2017-06-06,NSW,Sydney,66062,9.1,17.9,0.5,2.2,7.2,WSW,59,11.2,58,14.7,43
2017-06-06,QLD,Brisbane,40913,9.2,25.2,0.6,2.4,7.7,N,41,17.9,38,24.5,13
2017-06-06,VIC,Melbourne,86338,8.7,13.9,6.4,,,SSW,43,11.8,78,12.6,68
2017-06-07,NSW,Sydney,66062,8.8,15.5,54.6,2.6,0,SSW,56,13.3,82,14.2,85
2017-06-07,QLD,Brisbane,40913,5,22.8,0.6,4.4,9.1,SE,31,12.4,68,22.5,39
2017-06-07,VIC,Melbourne,86338,5.9,14.3,0,,,SE,28,8.3,91,14,54
2017-06-08,NSW,Sydney,66062,12.4,18.1,61,1.8,1.3,SE,44,13.3,91,16.9,84
2017-06-08,QLD,Brisbane,40913,9.9,22.5,0.6,2.2,9.2,ENE,48,13,75,22.1,25
2017-06-08,VIC,Melbourne,86338,3.6,15,0,,,SSW,30,6.6,86,14,56
2017-06-09,NSW,Sydney,66062,10.4,18.9,3.4,1.6,4.8,ESE,52,12.5,90,18.2,64
2017-06-09,QLD,Brisbane,40913,7.7,24.1,0.6,4.8,9.3,NE,37,15.8,27,23.8,14
2017-06-09,VIC,Melbourne,86338,6.6,15,1.8,,,S,22,11,98,13.9,66
2017-06-10,NSW,Sydney,66062,12.5,18.1,38.8,3.6,0.2,SSE,50,15.8,85,17.5,70
2017-06-10,QLD,Brisbane,40913,7.7,20.8,0.6,3.8,1.9,NNE,33,14.7,33,20.4,25
2017-06-10,VIC,Melbourne,86338,7,15.8,0,,,SSW,17,7.5,100,14,68
2017-06-11,NSW,Sydney,66062,12.5,17.2,2.8,1.6,0.2,WNW,20,13.1,90,16,85
2017-06-11,QLD,Brisbane,40913,11.4,23.5,0.6,3.2,4.5,ENE,41,13.6,33,23.4,14
2017-06-11,VIC,Melbourne,86338,3.4,15.5,0.2,,,SSW,22,6.1,96,13.6,67
2017-06-12,NSW,Sydney,66062,11.3,20.4,1.2,0.8,7.5,SSE,26,11.6,92,19.1,59
2017-06-12,QLD,Brisbane,40913,10.1,24,0.6,3.8,7.6,NNE,39,16.7,26,23.7,22
2017-06-12,VIC,Melbourne,86338,6,15.9,0,,,SSW,19,10,85,15,77
2017-06-13,NSW,Sydney,66062,9.5,18.7,0.5,2.2,9.1,SSE,37,11.2,82,18.2,62
2017-06-13,QLD,Brisbane,40913,10.8,24.8,0.6,3.4,5.6,NE,28,15,35,22.4,24
2017-06-13,VIC,Melbourne,86338,8.8,15.1,0,,,WSW,20,10.7,88,13.4,67
2017-06-14,NSW,Sydney,66062,11.1,17.8,2.6,1.8,2.9,WNW,24,13.3,82,17.4,78
2017-06-14,QLD,Brisbane,40913,12.1,20.4,14.6,3.4,4.3,E,37,14.2,83,19.4,63
2017-06-14,VIC,Melbourne,86338,4,17.4,0,,,N,28,7.8,97,16.7,56
2017-06-15,NSW,Sydney,66062,10.7,20.1,0.6,1.2,6.4,W,22,11.9,89,18.7,69
2017-06-15,QLD,Brisbane,40913,9.9,19.3,0.6,3.2,6.5,SSW,24,14.7,87,18.1,65
2017-06-15,VIC,Melbourne,86338,7.7,19,0,,,N,20,9.8,87,17.4,53
2017-06-16,NSW,Sydney,66062,11.9,17.3,0.5,1.6,0.2,WNW,26,13.3,79,17.2,69
2017-06-16,QLD,Brisbane,40913,6.6,20.8,0.6,1,8.3,S,22,12.1,85,20.2,43
2017-06-16,VIC,Melbourne,86338,6.6,17.1,0,,,NNE,15,8.4,86,15.9,65
2017-06-17,NSW,Sydney,66062,13.2,19.1,0.5,1,0.2,SSW,26,14.6,81,17.4,67
2017-06-17,QLD,Brisbane,40913,5,21.7,0.6,2.4,9,E,39,13.1,78,21.5,47
2017-06-17,VIC,Melbourne,86338,4.2,15.5,0,,,NE,15,5.4,95,14.9,66
2017-06-18,NSW,Sydney,66062,11.3,18,1.8,2,6.3,S,52,12.9,83,17.6,62
2017-07-02,NSW,Sydney,66062,5.4,17.6,0.5,2,9.5,W,28,7.6,78,16.1,44
2017-07-02,QLD,Brisbane,40913,3.9,15,20,1.6,4.2,S,35,7.3,90,14.2,46
2017-07-02,VIC,Melbourne,86338,0.8,14.3,0,,,N,41,3.8,75,13.9,36
2017-07-03,NSW,Sydney,66062,5.6,15.8,0.5,2.6,2.7,N,31,7.9,81,15.4,57
2017-07-03,QLD,Brisbane,40913,7,16.6,0.6,2,4.6,NNW,31,8.9,82,15.2,58
2017-07-03,VIC,Melbourne,86338,3.8,13.9,0,,,NNE,39,9,60,11.2,92
2017-07-04,NSW,Sydney,66062,7.9,21.8,0.2,0.6,9.3,,,13.2,76,21.5,23
2017-07-04,QLD,Brisbane,40913,5.5,14.4,2,1.4,1.2,N,19,8.3,94,13.9,73
2017-07-04,VIC,Melbourne,86338,9,16.2,3,,,N,33,11,78,15.6,54
2017-07-05,NSW,Sydney,66062,12.2,20.8,0.5,5.4,9.2,W,52,14.4,51,18.7,36
2017-07-05,QLD,Brisbane,40913,8,13.9,3.4,1.8,2.8,SW,59,11.9,92,10.7,83
2017-07-05,VIC,Melbourne,86338,10.1,15.8,0,,,SSW,24,12.4,75,13.4,65
2017-07-06,NSW,Sydney,66062,9.1,18.2,0.5,4.8,9.7,WSW,37,11.5,62,17.4,36
2017-07-06,QLD,Brisbane,40913,4.7,16,10.4,4.2,5.5,WNW,30,7.4,88,14.4,60
2017-07-06,VIC,Melbourne,86338,4.8,15,0,,,N,28,8.1,75,14.1,46
2017-07-07,NSW,Sydney,66062,6.6,19.1,0.5,3.6,9.4,WNW,35,8.8,73,18.6,24
2017-07-07,QLD,Brisbane,40913,7.4,17.6,9.8,1,6.9,SW,33,11.6,97,16.2,59
2017-07-07,VIC,Melbourne,86338,8,14.2,0,,,NNW,37,9,79,13.4,56
2017-07-08,NSW,Sydney,66062,7,18.1,0.5,3.2,9.3,W,37,9,66,17,36
2017-07-08,QLD,Brisbane,40913,3.2,17.2,0.6,1.4,9,S,28,10.2,83,16.5,41
2017-07-08,VIC,Melbourne,86338,5.3,13.5,0,,,NW,30,9,84,13.2,58
2017-07-09,NSW,Sydney,66062,7.4,18,0.5,2,9.5,W,33,9.2,69,17.6,33
2017-07-09,QLD,Brisbane,40913,7.1,17.8,0.6,2.4,7.4,NE,22,10.4,68,17.4,42
2017-07-09,VIC,Melbourne,86338,7.3,14.6,0.8,,,NNW,39,9.7,70,13.3,49
2017-07-10,NSW,Sydney,66062,7,17.3,0.5,3.6,9.4,W,39,9.4,67,16.7,36
2017-07-10,QLD,Brisbane,40913,5.9,17.9,0.6,2.4,4,NNE,39,12.5,52,17.6,40
2017-07-10,VIC,Melbourne,86338,7,16,0,,,W,28,9.2,74,15.2,48
2017-07-11,NSW,Sydney,66062,6.6,18.4,0.5,2.8,9.2,SSW,39,9,63,17.7,46
2017-07-11,QLD,Brisbane,40913,12.5,19.7,0.6,2.6,4.2,WNW,52,15.6,52,18.5,78
2017-07-11,VIC,Melbourne,86338,6.9,14.3,0.4,,,NW,19,9.8,79,13.8,57
2017-07-12,NSW,Sydney,66062,7.8,14.8,1.6,3.4,0.5,W,31,8.9,76,13,79
2017-07-12,QLD,Brisbane,40913,14.8,20.4,3.4,2.4,8,NW,61,18,69,19.2,59
2017-07-12,VIC,Melbourne,86338,3.2,15.1,0,,,N,30,4.8,89,14.7,49
2017-07-13,NSW,Sydney,66062,8,17.4,10.4,1,9.3,ENE,30,9.6,80,17.2,49
2017-07-13,QLD,Brisbane,40913,13.2,18.6,7.2,3.6,3,SW,33,15.1,96,18.1,60
2017-07-13,VIC,Melbourne,86338,4.8,13.3,0,,,N,56,10.9,56,13,54
2017-07-14,NSW,Sydney,66062,9.6,20.8,0.5,4,6.1,NNW,30,12.2,67,19.3,41
2017-07-14,QLD,Brisbane,40913,12.3,19.2,0.4,2.8,2.3,NE,24,14.6,76,18.1,62
2017-07-14,VIC,Melbourne,86338,9.9,15.9,2,,,N,50,10.5,76,15.2,41
2017-07-15,NSW,Sydney,66062,12.1,19.8,0.4,4,6.1,SW,30,13,71,17.9,39
2017-07-15,QLD,Brisbane,40913,11.4,19.9,0.6,2.6,1.3,WNW,50,14,96,16.9,82
2017-07-15,VIC,Melbourne,86338,4.5,12,0,,,NW,22,7.8,71,11.6,58
2017-07-16,NSW,Sydney,66062,7.7,18.4,0.5,1.8,9.4,WSW,31,9.4,61,16.8,32
2017-07-16,QLD,Brisbane,40913,13.7,19.2,4.6,2,3.6,SW,37,15.8,66,18.1,45
2017-07-16,VIC,Melbourne,86338,3,12.5,0,,,N,52,5.7,75,11.1,54
2017-07-17,NSW,Sydney,66062,7.9,19.3,0.5,4.2,6.5,W,24,9.8,75,17.3,44
2017-07-17,QLD,Brisbane,40913,8.7,19.2,0.6,3.4,9.1,NE,33,12.4,64,18.5,41
2017-07-17,VIC,Melbourne,86338,5.7,15.3,0,,,N,52,11.8,55,14.6,50
2017-07-18,NSW,Sydney,66062,9.1,22.9,0.5,1.2,9.8,NW,54,14.1,62,22.6,30
2017-07-18,QLD,Brisbane,40913,11.2,24.6,0.6,3.2,9.2,N,46,14.8,52,24.1,35
2017-07-18,VIC,Melbourne,86338,9.7,14.5,0.2,,,NNW,41,10.6,75,13.8,55
2017-07-19,NSW,Sydney,66062,10.6,17.5,0.5,6.6,9.1,WSW,57,12,53,16.9,33
2017-07-19,QLD,Brisbane,40913,13.2,18.7,8.4,3.8,3,N,61,14,96,14.1,89
2017-07-19,VIC,Melbourne,86338,8.9,13.2,5,,,SSW,37,10.8,99,11.6,81
2017-07-20,NSW,Sydney,66062,9.3,17.5,0.5,4.4,9.6,WSW,63,11.4,55,15.8,26
2017-07-20,QLD,Brisbane,40913,11.5,18.2,18.4,3.6,7.7,WSW,56,15.3,69,16.4,71
2017-07-20,VIC,Melbourne,86338,7.9,10.7,0.6,,,SSW,39,9.4,82,9.7,77
2017-07-21,NSW,Sydney,66062,7.5,18,0.5,4.2,9.2,W,46,9.6,53,17.4,37
2017-07-21,QLD,Brisbane,40913,10.6,20.1,5,0.6,7,NW,59,13.2,88,18.6,53
2017-07-21,VIC,Melbourne,86338,3.3,13.8,4.2,,,N,26,5.8,93,13.4,53
2017-07-22,NSW,Sydney,66062,6,19.1,0.5,3.6,9.8,W,31,7.9,64,18.7,27
2017-07-22,QLD,Brisbane,40913,10.7,19.3,6.1,,4.5,WSW,31,12.4,97,17.8,55
2017-07-22,VIC,Melbourne,86338,5.4,12.4,0,,,N,59,8.4,65,11.6,54
2017-07-23,NSW,Sydney,66062,6.5,22.1,0.5,3.6,9.9,NW,44,10.8,49,22.1,22
2017-07-23,QLD,Brisbane,40913,11.7,18.1,2.3,6.6,4.8,N,50,13.4,93,18,66
2017-07-23,VIC,Melbourne,86338,8.4,14.1,0,,,N,44,10.2,57,11.3,90
2017-07-24,NSW,Sydney,66062,10,21.3,0.5,7.2,9.6,WSW,31,12.7,51,20.9,24
2017-07-24,QLD,Brisbane,40913,10,18.6,16.8,4.2,3.9,W,39,12.4,95,17.1,50
2017-07-24,VIC,Melbourne,86338,8.1,16.1,7,,,WSW,28,9.9,77,15.2,53
2017-07-25,NSW,Sydney,66062,7.9,20.6,0.5,4.2,9.9,NW,30,11,63,19.7,32
2017-07-25,QLD,Brisbane,40913,10.4,19.6,3,0,3.4,NW,39,13,96,18.7,53
2017-07-25,VIC,Melbourne,86338,9.8,17.2,0,,,N,48,11.1,72,16.4,53
2017-07-26,NSW,Sydney,66062,11,21.3,0.5,2.8,9.9,W,70,18,39,20.2,27
2017-07-26,QLD,Brisbane,40913,11.7,19.2,0.4,2.2,3.1,SW,48,15.6,78,17.4,73
2017-07-26,VIC,Melbourne,86338,10.9,15,2.4,,,SW,37,12.5,70,12.7,71
2017-07-27,NSW,Sydney,66062,9.1,17.5,0.5,4.8,9.7,W,31,11.2,61,17.1,44
2017-07-27,QLD,Brisbane,40913,11.6,21.5,24,1.8,6.5,NNW,59,14.5,90,19.4,68
2017-07-27,VIC,Melbourne,86338,4.4,14.2,2,,,N,52,6.2,90,13.8,55
2017-07-28,NSW,Sydney,66062,11.2,19.1,0.5,4.2,8.3,WSW,67,13.1,51,18.9,25
2017-07-28,QLD,Brisbane,40913,14.5,17.7,10.8,3.4,,WNW,78,17,60,13.9,83
2017-07-28,VIC,Melbourne,86338,6.2,14.3,2.8,,,NNE,37,10.9,73,13.4,56
2017-07-29,NSW,Sydney,66062,8.2,21.2,0.5,4.6,9.9,W,31,10.7,52,20.9,23
2017-07-29,QLD,Brisbane,40913,9.7,15.1,11.2,6.8,5,WSW,72,12.1,77,12.3,86
2017-07-29,VIC,Melbourne,86338,9.7,17.9,0,,,N,81,12.7,51,16.9,41
2017-07-30,NSW,Sydney,66062,10.7,26.5,0.5,6,6.2,NNW,44,18.4,25,24.8,22
2017-07-30,QLD,Brisbane,40913,5.3,17.1,9.8,1.6,6.5,WSW,33,11.1,79,15.4,52
2017-07-30,VIC,Melbourne,86338,11.3,18.1,0,,,NW,31,13.7,64,17.5,34
2017-07-31,NSW,Sydney,66062,15.6,16.5,0.5,4.4,0,WSW,56,15.7,72,11.8,85
2017-07-31,QLD,Brisbane,40913,9.5,15.7,0.6,2.2,0.6,WSW,67,12.3,75,12.8,81
2017-07-31,VIC,Melbourne,86338,6.7,14.8,0,,,S,19,9,79,13.7,56
2017-08-01,NSW,Sydney,66062,8.6,18.1,6,2.4,9.8,SW,39,11.6,57,16.9,39
2017-08-01,QLD,Brisbane,40913,1.8,15.3,23.8,2.6,9,SSW,39,8.4,87,14.2,53
2017-08-01,VIC,Melbourne,86338,3.5,15.5,0,,,SSW,20,6.2,89,13,64
2017-08-02,NSW,Sydney,66062,7.2,18.2,0.5,3.2,8.8,S,41,9.6,62,16.8,51
2017-08-02,QLD,Brisbane,40913,1,14.9,0.6,0.6,9.7,E,26,8.7,81,14.3,47
2017-08-02,VIC,Melbourne,86338,3.4,12.9,0,,,NNE,13,5.6,100,12.8,61
2017-08-03,NSW,Sydney,66062,8.9,17.4,3,2.8,5.5,NNE,37,10.3,85,16.4,70
2017-08-03,QLD,Brisbane,40913,3.6,17.9,0.2,1.8,2.1,W,54,8.5,87,14.3,96
2017-08-03,VIC,Melbourne,86338,3.1,9,0,,,NNE,15,4.3,89,8.7,84
2017-08-04,NSW,Sydney,66062,10.3,17.3,13.8,1.8,10.2,W,50,12.8,60,16.6,41
2017-08-04,QLD,Brisbane,40913,7.9,15.5,1.6,2,1,WSW,30,10.3,77,14.3,64
2017-08-04,VIC,Melbourne,86338,4.3,12.1,6.6,,,N,50,8.9,87,11.5,66
2017-08-05,NSW,Sydney,66062,9.6,19.9,0.5,5.6,10.1,WNW,48,14,46,19.3,22
2017-08-05,QLD,Brisbane,40913,9.3,18.3,0.4,1.6,6.1,WSW,31,12.4,94,16.8,59
2017-08-05,VIC,Melbourne,86338,8.8,14.2,0,,,N,52,10.6,68,13.1,55
2017-08-06,NSW,Sydney,66062,12.1,21.3,0.5,6.8,9.9,NW,54,15,46,20.9,26
2017-08-06,QLD,Brisbane,40913,6.7,19.8,0.6,3.2,6.1,NW,37,11.8,81,19.4,41
2017-08-06,VIC,Melbourne,86338,9,13,0,,,NNE,54,10,76,11.7,54
2017-08-07,NSW,Sydney,66062,13.5,18.4,0.5,6.8,9.8,W,63,14.7,44,16.9,25
2017-08-07,QLD,Brisbane,40913,10.6,19.4,0.6,2.4,0.8,N,30,14.5,78,18.4,65
2017-08-07,VIC,Melbourne,86338,7.7,12.3,4.2,,,SW,41,9.4,70,11,79
2017-08-08,NSW,Sydney,66062,9.7,19.3,0.5,7.2,10.3,WSW,54,12.1,45,18.5,24
2017-08-08,QLD,Brisbane,40913,12.5,19.2,0.4,2,2.3,NW,57,15.9,82,14.8,93
2017-08-08,VIC,Melbourne,86338,7.9,15,2.2,,,NW,26,10.5,72,13.9,59
2017-08-09,NSW,Sydney,66062,9.2,20.9,0.5,5,10.4,W,33,12,53,20,32
2017-08-09,QLD,Brisbane,40913,8.5,14.7,43.6,4.4,3,SW,72,9.7,93,11.6,63
2017-08-09,VIC,Melbourne,86338,8.9,16.8,0,,,N,46,11.4,74,15.1,60
2017-08-10,NSW,Sydney,66062,9.2,24.2,0.5,4,9.4,NNW,31,11.8,64,23.5,24017-09-10,NSW,Sydney,66062,8,19.1,0.5,5,10.8,W,24,13.5,52,17.3,51
2017-09-10,QLD,Brisbane,40913,10.6,20.8,0.2,4.6,4.7,W,30,14.6,91,20.3,46
2017-09-10,VIC,Melbourne,86338,5.3,16.1,0.2,,,N,20,10.8,76,16,47
2017-09-11,NSW,Sydney,66062,8.7,22.7,0.5,3.8,10.1,NE,24,14.4,60,22.5,29
2017-09-11,QLD,Brisbane,40913,11.2,19.6,5.2,4.4,5.6,WSW,41,16.4,94,17.8,73
2017-09-11,VIC,Melbourne,86338,8.1,18.1,0,,,N,46,12.7,63,16.9,50
2017-09-12,NSW,Sydney,66062,11.3,27.2,0.5,5.8,2.3,E,26,18.2,43,23.4,34
2017-09-12,QLD,Brisbane,40913,13.2,19.4,1.4,4.2,7.9,SW,54,15.9,68,19,35
2017-09-12,VIC,Melbourne,86338,12.2,19.9,0.2,,,NNW,39,15.2,60,19.7,59
2017-09-13,NSW,Sydney,66062,18.2,33.8,0.5,6.2,9.6,NNW,70,24.9,19,32.5,10
2017-09-13,QLD,Brisbane,40913,7.7,21.6,0.6,6.4,11,E,43,13.3,50,21.6,27
2017-09-13,VIC,Melbourne,86338,13.3,16.4,2,,,WSW,50,14.9,62,11.7,69
2017-09-14,NSW,Sydney,66062,12,17.3,0.2,13,10.5,WSW,72,13.8,37,16.4,26
2017-09-14,QLD,Brisbane,40913,6.3,25.6,0.6,6,7,WSW,30,17.2,34,24.1,27
2017-09-14,VIC,Melbourne,86338,5.8,15.3,1.4,,,W,43,9.6,66,14.1,56
2017-09-15,NSW,Sydney,66062,10.5,22.8,0.5,7.8,10.6,W,52,16.6,37,22.2,24
2017-09-15,QLD,Brisbane,40913,14.5,27.8,0.6,4.6,8.5,ENE,37,17.6,43,27.6,26
2017-09-15,VIC,Melbourne,86338,9.6,16.7,0,,,WSW,35,13.5,58,14.8,82
2017-09-16,NSW,Sydney,66062,12.5,24.2,0.5,7,6.4,SSW,56,17.3,43,21.3,22
2017-09-16,QLD,Brisbane,40913,10.9,24.4,0.6,4.8,6.5,WSW,33,15,96,21.9,48
2017-09-16,VIC,Melbourne,86338,7.4,13.3,21.4,,,SSW,54,9.1,81,12.5,52
2017-09-17,NSW,Sydney,66062,8.1,18.6,0.5,4.6,10.8,ESE,28,13.6,48,16.4,37
2017-09-17,QLD,Brisbane,40913,14.6,21.7,0.4,4.4,9.6,WSW,33,18.5,72,21.1,47
2017-09-17,VIC,Melbourne,86338,4,18.1,0.2,,,N,43,10.5,58,17.8,35
2017-09-18,NSW,Sydney,66062,8.2,21,0.5,5.6,10.7,NE,39,14,55,20.2,55
2017-09-18,QLD,Brisbane,40913,7.6,26,0.6,4,10.4,E,37,15.8,71,25.7,34
2017-09-18,VIC,Melbourne,86338,10.5,22,0,,,NNW,63,15.7,32,20.9,30
2017-09-19,NSW,Sydney,66062,14,25.2,0.5,6.4,10.8,SSW,54,20.4,30,23.1,16
2017-09-19,QLD,Brisbane,40913,8.9,22.4,0.6,7,8.5,WSW,31,17.9,73,21,67
2017-09-19,VIC,Melbourne,86338,9,13.9,0.2,,,WNW,31,11.6,58,12.4,58
2017-09-20,NSW,Sydney,66062,10.7,19,0.5,9.2,7.7,WSW,33,14.6,44,17.7,56
2017-09-20,QLD,Brisbane,40913,14.7,21.4,5.4,4.6,6.9,W,48,18,57,20.6,52
2017-09-20,VIC,Melbourne,86338,6,19.2,0,,,N,35,11.6,63,18.6,42
2017-09-21,NSW,Sydney,66062,10.4,21.3,0.5,3.8,10.1,NE,30,15.6,65,21.2,49
2017-09-21,QLD,Brisbane,40913,16.8,21.5,5.8,5,3.4,NW,81,19.8,72,19,61
2017-09-21,VIC,Melbourne,86338,11.6,23.6,0,,,N,54,14.8,46,22.4,33
2017-09-22,NSW,Sydney,66062,12.8,27.7,0.5,7.2,10.5,NNE,35,20.1,35,25.8,26
2017-09-22,QLD,Brisbane,40913,13,19.2,7.4,5.2,1.9,W,89,14.5,88,17.3,58
2017-09-22,VIC,Melbourne,86338,12.8,25.6,0,,,N,33,14.9,71,24.8,32
2017-09-23,NSW,Sydney,66062,15.5,32.2,0.5,8.6,5.7,NNE,50,23,25,29.1,25
2017-09-23,QLD,Brisbane,40913,11.4,17.1,9.8,5.4,5.3,SW,52,13.4,72,16.1,48
2017-09-23,VIC,Melbourne,86338,14.8,30.6,0,,,N,61,21.1,30,30,24
2017-09-24,NSW,Sydney,66062,23,29.2,0.5,12,6.5,W,54,27.1,20,27.8,15
2017-09-24,QLD,Brisbane,40913,10.1,17.3,5.2,4.4,4.5,NNW,46,14.8,75,14,87
2017-09-24,VIC,Melbourne,86338,12.9,19.9,0.4,,,NNW,56,14.9,56,18.4,40
2017-09-25,NSW,Sydney,66062,16,26.7,0.5,10.8,10.8,WNW,57,22.3,21,23.3,19
2017-09-25,QLD,Brisbane,40913,9.3,17,10.4,3.2,4.6,NE,35,11.8,81,14.9,67
2017-09-25,VIC,Melbourne,86338,10.3,14.2,1.6,,,SSW,35,12.9,64,12.8,68
2017-09-26,NSW,Sydney,66062,12.3,22.2,0.5,9.6,11,WSW,41,19.5,29,19,47
2017-09-26,QLD,Brisbane,40913,6,18.2,3.4,4.4,9.3,WSW,41,13.1,69,17.5,40
2017-09-26,VIC,Melbourne,86338,5,15.1,0.4,,,S,24,11.3,64,14,46
2017-09-27,NSW,Sydney,66062,15.4,22.7,0.5,6.6,8.2,ENE,50,18.1,58,21.1,62
2017-09-27,QLD,Brisbane,40913,5,16.7,0.6,6.8,5.6,SW,41,13.1,58,14.7,56
2017-09-27,VIC,Melbourne,86338,9.6,24.6,0,,,NNW,50,12.5,64,24.5,30
2017-09-28,NSW,Sydney,66062,18,25.7,0.5,6.2,3.8,WNW,59,22.7,47,24.1,26
2017-09-28,QLD,Brisbane,40913,6.9,16.6,1,3.4,3.5,WSW,33,12.7,72,13.5,76
2017-09-28,VIC,Melbourne,86338,10.4,18.5,0,,,SW,35,14.1,66,16.8,45 2017-10-01,VIC,Melbourne,86338,9.7,17.2,0,,,S,24,12.4,65,16.4,58
2017-10-02,NSW,Sydney,66062,12,22.9,0.5,6.2,10.1,NE,44,16.7,55,21.3,60
2017-10-02,QLD,Brisbane,40913,7.7,24,0.6,4.8,11.3,WSW,39,19.6,43,22.1,45
2017-10-02,VIC,Melbourne,86338,9.5,16.4,0,,,SSW,24,12.1,75,15.2,58
2017-10-15,NSW,Sydney,66062,16,22.4,0.5,3.8,2.4,ESE,28,22.3,52,20.1,58
2017-10-15,QLD,Brisbane,40913,16.4,25.4,0.6,7.2,11.6,W,33,20.4,61,22.4,53
2017-10-15,VIC,Melbourne,86338,6.8,17.6,0,,,SSW,31,12.4,67,16.7,67
2017-10-16,NSW,Sydney,66062,15.9,23.3,0.5,5,10.4,E,30,22.4,55,22.3,50
2017-10-16,QLD,Brisbane,40913,15.3,19.9,3,5.8,6.5,NW,91,15.8,91,18.2,51
2017-10-16,VIC,Melbourne,86338,11.3,28.6,0,,,NNE,28,12.9,91,26.2,40
2017-10-17,NSW,Sydney,66062,16.9,23.5,0.5,7.6,9.9,ENE,39,21.2,57,22.9,47
2017-10-17,QLD,Brisbane,40913,13.9,20.1,7.6,5.4,5.8,WNW,52,17,71,19.8,50
2017-10-17,VIC,Melbourne,86338,12.8,30.2,0,,,N,37,21,50,30.1,24
2017-10-18,NSW,Sydney,66062,18.8,23.2,0.5,9,10.4,NNE,50,20.9,64,22.5,56
2017-10-18,QLD,Brisbane,40913,12,19.4,5,5,11.7,SW,44,16.7,46,18.2,37
2017-10-18,VIC,Melbourne,86338,14.5,30.9,0,,,NNW,48,21.7,43,29.9,31
2017-10-19,NSW,Sydney,66062,17.1,25.3,0.5,10,11.4,NNE,52,22.1,52,24.6,47
2017-10-19,QLD,Brisbane,40913,6.8,24.7,0.6,6.4,12.1,E,56,16.8,44,24.4,24
2017-10-19,VIC,Melbourne,86338,21.7,25.6,0,,,SW,43,23.8,45,23.5,71
2017-10-20,NSW,Sydney,66062,19,19.6,7.8,9.2,0,S,56,19.6,87,15.3,85
2017-10-20,QLD,Brisbane,40913,12.1,29.9,0.6,6,12.1,E,61,22.3,31,29.4,17
2017-10-20,VIC,Melbourne,86338,11.3,16.2,2.6,,,S,41,13.4,69,15.3,56
2017-10-21,NSW,Sydney,66062,13,20,16.4,4.4,11.3,SSW,50,16.8,52,18.9,53
2017-10-21,QLD,Brisbane,40913,15.9,29.5,0.6,9.2,4.9,ENE,65,23.2,32,28.2,26
2017-10-21,VIC,Melbourne,86338,11,15.4,0,,,S,28,11.8,71,14.8,63
2017-10-22,NSW,Sydney,66062,13.4,21.7,0.2,4.8,5.5,ESE,30,16,75,18.7,60
2017-10-22,QLD,Brisbane,40913,18.5,28.4,1.2,7.2,11.7,E,70,23.1,56,27.1,46
2017-10-22,VIC,Melbourne,86338,11.1,16,0,,,SSW,28,12.7,78,15.2,76
2017-10-23,NSW,Sydney,66062,12.7,23,3.4,3.6,12.1,S,31,18,60,22.1,43
2017-10-23,QLD,Brisbane,40913,14.3,23.6,0.6,5.6,3.8,WSW,37,18.2,80,22.4,57
2017-10-23,VIC,Melbourne,86338,11.9,19,0,,,SSW,31,13.6,72,17.3,61
2017-10-24,NSW,Sydney,66062,13.9,24.3,0.5,6.8,10.5,NE,39,19.9,57,22.6,55
2017-10-24,QLD,Brisbane,40913,14.3,22.6,0.6,6.2,11,W,37,18.2,63,20.3,40
2017-10-24,VIC,Melbourne,86338,12.1,20.5,0.2,,,SSW,26,15,74,17.4,61
2017-10-25,NSW,Sydney,66062,18,26.9,0.5,8,5,S,59,21,67,23.3,56
2017-10-25,QLD,Brisbane,40913,9.6,22.7,0.6,7,7.3,W,44,19,69,21.7,54
2017-10-25,VIC,Melbourne,86338,13.3,18.5,11.4,,,SSW,24,14.5,94,17.3,67
2017-10-26,NSW,Sydney,66062,17.2,24.6,0.4,7,6.5,WSW,63,19.7,73,22.6,63
2017-10-26,QLD,Brisbane,40913,13.3,20.9,1.2,4.6,5.3,WNW,67,17.6,77,16.4,66
2017-10-26,VIC,Melbourne,86338,12.9,16.8,0.2,,,SSW,39,13.8,83,15.5,73
2017-10-27,NSW,Sydney,66062,14.8,24.2,34.2,7.2,3.9,WSW,61,18.1,85,22.3,65
2017-10-27,QLD,Brisbane,40913,8.4,19.3,2,6.2,4.8,WSW,44,16.7,48,17.5,51
2017-10-27,VIC,Melbourne,86338,10.3,27.4,0,,,N,48,15.1,78,25.2,37
2017-10-28,NSW,Sydney,66062,17.5,25.1,0.2,3.2,7.7,ENE,35,21.6,66,22.3,65
2017-10-28,QLD,Brisbane,40913,14,20.5,1,4.2,6.2,WSW,52,18.4,54,19.8,54
2017-10-28,VIC,Melbourne,86338,15,21.4,0.4,,,N,39,15.4,72,19.7,46
2017-10-29,NSW,Sydney,66062,20.2,29.6,0.2,4.6,10.6,NNE,39,24.4,49,27.7,35
2017-10-29,QLD,Brisbane,40913,12.6,20.6,0.8,4.2,11.6,SW,54,16.7,46,19.9,41
2017-10-29,VIC,Melbourne,86338,11.5,28.6,0,,,N,61,19,63,27.4,25
2017-10-30,NSW,Sydney,66062,20.3,35.4,0.5,11.4,9.3,SSW,69,28.5,37,34.9,13
2017-10-30,QLD,Brisbane,40913,7.9,27.5,0.6,6.8,12.5,WSW,35,19.1,44,25.9,33
2017-10-30,VIC,Melbourne,86338,8.8,17,4.4,,,W,46,12.6,62,14.4,55
2017-10-31,NSW,Sydney,66062,13,20.5,0.5,12,10.1,SSE,57,15.8,43,19.7,36
2017-10-31,QLD,Brisbane,40913,8.7,29.3,0.6,10.4,12.2,SW,39,20.2,54,27.1,35
2017-10-31,VIC,Melbourne,86338,8.1,16.4,6,,,S,28,12,62,15.4,48
2017-11-01,NSW,Sydney,66062,13,22.2,0.5,9.6,4.2,SSW,54,15.4,39,21,32
2017-11-01,QLD,Brisbane,40913,15,30.5,0.6,10.4,10.4,SE,44,21.5,48,29,39
2017-11-01,VIC,Melbourne,86338,11.9,18,0.4,,,SSW,26,14.7,78,15.8,66
2017-11-02,NSW,Sydney,66062,15.3,22.6,0.5,5.6,9.9,ESE,31,20.1,56,21.8,46
2017-11-02,QLD,Brisbane,40913,14.7,27.6,0.6,10.2,11.9,E,46,19.4,51,27,35
2017-11-02,VIC,Melbourne,86338,11.3,17.4,1.4,,,SSE,26,12.7,79,15.9,56
2017-11-03,NSW,Sydney,66062,16.9,26.9,0.5,6.6,5.9,SSW,57,20.6,58,25.7,48
2017-11-03,QLD,Brisbane,40913,13.6,26.9,0.6,9.8,12.3,ESE,43,19.5,49,25.3,41
2017-11-03,VIC,Melbourne,86338,9.4,16.6,1.4,,,S,46,12,58,15.3,45
2017-11-04,NSW,Sydney,66062,16.2,17.1,1.6,7.4,0,SSE,39,17,73,15.8,78
2017-11-04,QLD,Brisbane,40913,14.8,30.7,0.6,8,12.6,E,69,23.1,43,30.3,20
2017-11-04,VIC,Melbourne,86338,8.3,15.8,0.4,,,SSW,35,10.2,72,14.5,47
2017-11-05,NSW,Sydney,66062,14.1,19.5,31.8,2.6,0,ENE,39,14.9,85,17.7,59
2017-11-05,QLD,Brisbane,40913,15.2,31.1,0.6,10.2,12.6,E,67,21.6,38,31,20
2017-11-05,VIC,Melbourne,86338,6.4,17.6,0,,,SSW,39,13.4,59,15.6,49
2017-11-06,NSW,Sydney,66062,14.8,26.3,4.4,1.8,5.7,SSW,67,19.4,81,23.6,57
2017-11-06,QLD,Brisbane,40913,14.9,32.7,0.6,10.2,12.5,E,67,23.5,33,32.7,11
2017-11-06,VIC,Melbourne,86338,10.9,16.1,0,,,S,43,13.8,66,14.8,60
2017-11-07,NSW,Sydney,66062,13.8,21.8,5.6,8.8,11.3,SSE,54,18.9,40,20,40
2017-11-07,QLD,Brisbane,40913,17.3,36.4,0.6,12,12.5,E,59,27.1,21,35.4,14
2017-11-07,VIC,Melbourne,86338,10,15.8,3.4,,,S,43,12.2,76,14.4,52
2017-11-08,NSW,Sydney,66062,14.6,21.7,0.5,9.6,10,SSE,44,16.8,54,20.5,44
2017-11-08,QLD,Brisbane,40913,20,35.6,0.6,11.2,10.8,ENE,37,28,30,33.1,24
2017-11-08,VIC,Melbourne,86338,12.1,16.9,0,,,S,30,13.9,59,15.6,64
2017-11-09,NSW,Sydney,66062,12.3,22.1,0.5,6.8,11.6,E,28,17.9,52,21.2,50
2017-11-09,QLD,Brisbane,40913,15.2,31.6,0.6,7.8,12.8,SW,37,26.9,42,29.7,31
2017-11-09,VIC,Melbourne,86338,8.7,21.8,0,,,SSW,28,14.6,68,18.1,66
2017-11-10,NSW,Sydney,66062,14.2,23.4,0.5,6.8,11.6,E,26,19.2,59,22.7,50
2017-11-10,QLD,Brisbane,40913,14.1,32,0.6,8.4,12.6,W,33,26.4,41,30.1,37
2017-11-10,VIC,Melbourne,86338,12.1,27.2,0,,,SE,20,15.7,78,22.8,63
2017-11-11,NSW,Sydney,66062,14.7,23.7,0.5,7.4,8.6,ENE,33,19.9,58,20.9,56
2017-11-11,QLD,Brisbane,40913,16.2,38,0.6,9,10,W,41,30.4,32,33.5,27
2017-11-11,VIC,Melbourne,86338,14.8,24.7,0,,,SSW,26,17.8,79,22,65
2017-11-12,NSW,Sydney,66062,17.1,22.5,0.5,7.4,10.7,E,31,19.7,49,21.7,47
2017-11-12,QLD,Brisbane,40913,22.4,32.1,0.6,11.8,11.3,S,39,28.7,39,28.8,46
2017-11-12,VIC,Melbourne,86338,13.2,23.3,0,,,SSW,28,16.1,86,21.6,69
2017-11-13,NSW,Sydney,66062,17.5,22.3,0.5,6.4,9.1,E,33,19.1,48,21.2,42
2017-11-13,QLD,Brisbane,40913,17.5,30.1,0.6,10.2,12.7,SW,50,25.2,65,27.9,45
2017-11-13,VIC,Melbourne,86338,13.7,33,0,,,NE,24,19.3,73,32,20
2017-11-14,NSW,Sydney,66062,16,22.9,0.5,7.6,8.9,ENE,37,20,53,22.3,46
2017-11-14,QLD,Brisbane,40913,14.5,29.7,0.6,11.8,12.7,SE,54,20.2,47,29.1,19
2017-11-14,VIC,Melbourne,86338,18.2,33.9,0,,,N,48,25.3,38,33.1,27

Python - path dependent simulation

I've setup a simulation example below.
Setup:
I have weekly data, say 6 years of data each week of around 1000 stocks some weeks more other weeks less than 1000. I randomly chose 75 stocks at time t0. At t1 some stocks dies (probability p, goes out of fashion) or leave the index (structural such as merging). I need to simulate stocks so that every week I've exactly 75 stocks. Every week some stocks dies (between 0 and 75) and I pick new ones not from the existing 75. I also check if the stock leaves do to structural reasons. Every week I calculate the returns of the 75 stocks.
Questions: Is there an obvious why to improve the speed. I started with Pandas objects (group sort) which was to slow. I haven't tried to parallel the loop. I'm more interesting to hear if I should use numba (but it doesn't have the np.in1d function) or if there is a faster way to shuffle (I actually only need to shuffle the ones). I've also think about creating a fixed array with all stocks id using NaN, the problem here is that I need 75 names so I still need to filter out these NaN every week.
Maybe this is to detailed problem for this forum, I apologize if that's the case
Code:
from timeit import default_timer
import numpy as np
# Create dataset
n_weeks = 312 # Approximately 6 years of weekly data
n_stocks = np.random.normal(1000, 5, n_weeks).astype(dtype=np.uint16) # Around 1000 stocks every week but not fixed
idx_new_week = np.cumsum(np.hstack((0, n_stocks)))
# We give each stock a stock idea
n_obs = n_stocks.sum()
stock_id = np.ones([n_obs], dtype=np.uint16)
for j in range(1, n_weeks+1):
stock_id[idx_new_week[j-1]:idx_new_week[j]] = np.cumsum(np.ones(n_stocks[j-1]))
stock_rtn = np.random.normal(0, 0.25/np.sqrt(52), n_obs) # Simulated forward (one week ahead) return for each stock
# Simulation part
# Week 0 pick randomly 75 stocks
# Week n >=1 a stock dies for two reasons
# 1) randomness (probability 'p')
# 2) structural event (could be merger, fall out of index).
# We cannot assume that it is always the high stockid which dies for structural reasons (as it looks like here)
# If a stock dies we randomely pick a stock from the "deak" stock dataset (not included the ones which dies this week)
n_sim = 100 # I want this to be 1 mill
n_stock_cand = 75 # For this example we pick 75 stocks
p_survial = 0.90
# The weekly periodcal returns
pf_rtn = np.zeros([n_weeks, n_sim])
start = default_timer()
for k in range(0, n_sim):
# Randomely choice n_stock_cand at time zero
boolean_list = np.array([False] * (n_stocks[0] - n_stock_cand) + [True] * n_stock_cand)
np.random.shuffle(boolean_list) # Shuffle the list
stock_id_this_week = stock_id[idx_new_week[0]:idx_new_week[1]][boolean_list]
stock_rtn_this_week = stock_rtn[idx_new_week[0]:idx_new_week[1]][boolean_list]
# This part only simulate the Buzz portfolio names - later we simulate returns and from specific holdings of the 75 names
for j in range(1, n_weeks):
pf_rtn[j-1, k] = stock_rtn_this_week.mean()
# Find the number of stocks to keep
boolean_keep_stocks = np.random.rand(n_stock_cand) < p_survial
# Next we need to check if a stock is still part of the universe next period
stock_cand_temp = stock_id[idx_new_week[j-1]:idx_new_week[j]]
stock_rtn_temp = stock_rtn[idx_new_week[j-1]:idx_new_week[j]]
boolean_keep_stocks = (boolean_keep_stocks) & (np.in1d(stock_id_this_week, stock_cand_temp, assume_unique=True))
n_stocks_to_replace = n_stock_cand - boolean_keep_stocks.sum() # Number of new stocks to pick this week
if n_stocks_to_replace > 0:
# We have to pick from stocks which is not part of the portfolio already
boolean_cand = np.in1d(stock_cand_temp, stock_id_this_week, assume_unique=True, invert=True)
n_stocks_to_pick_from = boolean_cand.sum()
boolean_list = np.array([False] * (n_stocks_to_pick_from - n_stocks_to_replace) + [True] * n_stocks_to_replace)
np.random.shuffle(boolean_list) # Shuffle the list
# First avoid picking the same stock twich, next pick from the unique candidate list
stock_id_new = stock_cand_temp[boolean_cand][boolean_list] # The new stocks
stock_rtn_new = stock_rtn_temp[boolean_cand][boolean_list] # and their returns
stock_id_this_week = np.hstack((stock_id_this_week[boolean_keep_stocks], stock_id_new))
stock_rtn_this_week = np.hstack((stock_rtn_this_week[boolean_keep_stocks], stock_rtn_new))
else:
# No replacement of stocks / all surview but order might differ
boolean_cand = np.in1d(stock_cand_temp, stock_id_this_week, assume_unique=True, invert=False)
stock_id_this_week = stock_cand_temp[boolean_cand]
stock_rtn_this_week = stock_rtn_temp[boolean_cand]
# PnL last period
pf_rtn[n_weeks-1, k] = stock_rtn_this_week.mean()
print(default_timer() - start)

Trying to create a Planning algorithm

I have a list representing the year which is filled with a sub-list for each day of the year.
year = []
for i in range(0,52*7):
day = [i,0] #[day number, 0 = empty, 1 = something is planned]
year.append(day)
I also have a variable list of activities created by a class.
class Activities:
def __init__(self,name,weeks,weekends):
self.name = name
self.weeks = weeks
self.weekends = weekends
def __repr__(self):
return repr((self.name,self.weeks,self.weekends))
def activityMaker(activityList):
a= []
for i in range(0, len(activityList)):
a.append(Activities(activityList[i][0], activityList[i][1], activityList[i][2]))
a = sorted(a, key=lambda Activities: Activities.weeks)
activityList = a
return activityList
As an example;
>>> activityList = [['Tennis', 3, 0], ['Baseball', 4, 0], ['Swimming', 2, 0]]
>>>activities= activityMaker(activityList)
Which returns 'activities', sorted on Activities.weeks:
>>>activities[0].name
activities[0].week
activities[0].weekend
>>> 'Swimming' # activity name
2 #"i want to do this activity once every x weeks
0 # 0 = no preferance, 1 = not in weekends
Now here my dilemma. I wish to create an algorithm to fill year with the activities with as much rhythm as possible.
My current approach is not working properly. What I'm doing now is as follows.
for y in range(0,len(year), int(7*activities[0].weeks)):
year[y][1] = activities[i].name
Now the first activity is planned for each y. If I had two activities, which I each want planned once a week, I could plan the first on 0th, 7th, 14th etc, and the second on 3rd, 10th, 17th etcetera.
The problem with this approach is exemplified if activities[0] and activities[1] are respectively 2 and 3. If I apply the previous approach, activities[0] would be planned on the 0th, 14th, 28th etc, which is fine on its own. Between the 0th and 14th, the 2nd activity would be ideally placed on the 7th position, meaning the next time would be the 28th. On the 28 the first activity is already planned, however. This means that's there's nothing planned for two weeks and then suddenly 2 activities in a day. The second activity could be pushed to the 27th or 29th, but that would still mean that now activities are planned on the 0th, 7th, 14th, 28th, 29th. Aka, there's still 14 days between the 14th and the 28th, and then only 1 between the 28th and 29th.
In what way can I make sure that all activities are planned with as much average time in between activities?

Your problem is that unless the number of weeks is the same for all activities (so they all have the same rhythm, there will be some weeks with lots of activities and some weeks with none.
What I would suggest instead is this: As you walk through the weeks of the year, simply choose an activity (or two) at random for each week. That way, every week will have a modest amount of activity planned. Here's some example code:
import random
activities = ["Baseball", "Tennis", "Swimming", ... ]
skip_days = 3
year = {}
for y in range(0, 52*7, skip_days):
year[y] = random.choose(activities)
print year[0]
>>> "Swimming" (perhaps)
print year[15]
>>> "Baseball"
print year[17]
>>> None
If you want more activity, make skip_days smaller. If you want less, make it bigger. If you want a fixed amount of activity in each week, you could do something like
for y in range(0, 52*7, 7):
year[y] = random.choose(activities)
year[y+3] = random.choose(activities)
This would plan two days a week.