We Keep Coding

Post Processing Aggregates - python

I have a question for a data analysis question where I need to separate data into two groups: months and states.
The task involves as:
In any given month, the minimum monthly rainfall for each state is the lowest rainfall recording from any weather station in that state during that month.
For example, suppose NSW had recordings of 1, 10, 5, and 7 for January; then its minimum monthly rainfall for January would be , the lowest of those recordings.
If there are several correct answers, your program can output any of the correct answers.
On the given data, the program's output should look like this:
Month: 11
State: QLD
What I have tried is:
minimum_monthly_rainfall = {}
is_first_line = True
for row in open("climate_data_2017.csv"):
if is_first_line:
is_first_line = False
else:
values = row.split(",")
month = values[0].split("-")[1]
rainfall = float(values[6])
state = values [1]
if month in minimum_monthly_rainfall:
minimum_monthly_rainfall[month].append(rainfall)
else:
minimum_monthly_rainfall[month] = [rainfall]
print(minimum_monthly_rainfall)
I am not sure as what to do now. I have skipped the first line of the data file which i will show below, and I have discriminated all the monthly rainfall into corresponding months. However I am not sure how I will separate this back into months? I know I shoul
The data file is:
Date,State,City,Station Code,Minimum temperature (C),Maximum temperature (C),Rainfall (mm),Evaporation (mm),Sunshine (hours),Direction of maximum wind gust,Speed of maximum wind gust (km/h),9am Temperature (C),9am relative humidity (%),3pm Temperature (C),3pm relative humidity (%)
2017-06-03,VIC,Melbourne,86338,9.9,14.3,0,,,SSW,19,10.8,76,13.1,73
2017-06-04,NSW,Sydney,66062,10.5,19.7,1.4,0.6,6.8,W,28,12.2,77,19,57
2017-06-04,QLD,Brisbane,40913,12.8,22.6,0.6,1.8,9.1,ENE,31,16.3,77,22.5,52
2017-06-04,VIC,Melbourne,86338,4.4,15.8,0,,,NE,13,5.7,100,15.5,53
2017-06-05,NSW,Sydney,66062,9.4,19.6,0.5,2.8,9.4,W,28,11.5,78,19.2,41
2017-06-05,QLD,Brisbane,40913,10.1,25.3,0.6,1.8,9.2,ENE,31,15,71,24.9,30
2017-06-05,VIC,Melbourne,86338,5.6,17.5,0,,,NW,28,8.7,82,16.8,57
2017-06-06,NSW,Sydney,66062,9.1,17.9,0.5,2.2,7.2,WSW,59,11.2,58,14.7,43
2017-06-06,QLD,Brisbane,40913,9.2,25.2,0.6,2.4,7.7,N,41,17.9,38,24.5,13
2017-06-06,VIC,Melbourne,86338,8.7,13.9,6.4,,,SSW,43,11.8,78,12.6,68
2017-06-07,NSW,Sydney,66062,8.8,15.5,54.6,2.6,0,SSW,56,13.3,82,14.2,85
2017-06-07,QLD,Brisbane,40913,5,22.8,0.6,4.4,9.1,SE,31,12.4,68,22.5,39
2017-06-07,VIC,Melbourne,86338,5.9,14.3,0,,,SE,28,8.3,91,14,54
2017-06-08,NSW,Sydney,66062,12.4,18.1,61,1.8,1.3,SE,44,13.3,91,16.9,84
2017-06-08,QLD,Brisbane,40913,9.9,22.5,0.6,2.2,9.2,ENE,48,13,75,22.1,25
2017-06-08,VIC,Melbourne,86338,3.6,15,0,,,SSW,30,6.6,86,14,56
2017-06-09,NSW,Sydney,66062,10.4,18.9,3.4,1.6,4.8,ESE,52,12.5,90,18.2,64
2017-06-09,QLD,Brisbane,40913,7.7,24.1,0.6,4.8,9.3,NE,37,15.8,27,23.8,14
2017-06-09,VIC,Melbourne,86338,6.6,15,1.8,,,S,22,11,98,13.9,66
2017-06-10,NSW,Sydney,66062,12.5,18.1,38.8,3.6,0.2,SSE,50,15.8,85,17.5,70
2017-06-10,QLD,Brisbane,40913,7.7,20.8,0.6,3.8,1.9,NNE,33,14.7,33,20.4,25
2017-06-10,VIC,Melbourne,86338,7,15.8,0,,,SSW,17,7.5,100,14,68
2017-06-11,NSW,Sydney,66062,12.5,17.2,2.8,1.6,0.2,WNW,20,13.1,90,16,85
2017-06-11,QLD,Brisbane,40913,11.4,23.5,0.6,3.2,4.5,ENE,41,13.6,33,23.4,14
2017-06-11,VIC,Melbourne,86338,3.4,15.5,0.2,,,SSW,22,6.1,96,13.6,67
2017-06-12,NSW,Sydney,66062,11.3,20.4,1.2,0.8,7.5,SSE,26,11.6,92,19.1,59
2017-06-12,QLD,Brisbane,40913,10.1,24,0.6,3.8,7.6,NNE,39,16.7,26,23.7,22
2017-06-12,VIC,Melbourne,86338,6,15.9,0,,,SSW,19,10,85,15,77
2017-06-13,NSW,Sydney,66062,9.5,18.7,0.5,2.2,9.1,SSE,37,11.2,82,18.2,62
2017-06-13,QLD,Brisbane,40913,10.8,24.8,0.6,3.4,5.6,NE,28,15,35,22.4,24
2017-06-13,VIC,Melbourne,86338,8.8,15.1,0,,,WSW,20,10.7,88,13.4,67
2017-06-14,NSW,Sydney,66062,11.1,17.8,2.6,1.8,2.9,WNW,24,13.3,82,17.4,78
2017-06-14,QLD,Brisbane,40913,12.1,20.4,14.6,3.4,4.3,E,37,14.2,83,19.4,63
2017-06-14,VIC,Melbourne,86338,4,17.4,0,,,N,28,7.8,97,16.7,56
2017-06-15,NSW,Sydney,66062,10.7,20.1,0.6,1.2,6.4,W,22,11.9,89,18.7,69
2017-06-15,QLD,Brisbane,40913,9.9,19.3,0.6,3.2,6.5,SSW,24,14.7,87,18.1,65
2017-06-15,VIC,Melbourne,86338,7.7,19,0,,,N,20,9.8,87,17.4,53
2017-06-16,NSW,Sydney,66062,11.9,17.3,0.5,1.6,0.2,WNW,26,13.3,79,17.2,69
2017-06-16,QLD,Brisbane,40913,6.6,20.8,0.6,1,8.3,S,22,12.1,85,20.2,43
2017-06-16,VIC,Melbourne,86338,6.6,17.1,0,,,NNE,15,8.4,86,15.9,65
2017-06-17,NSW,Sydney,66062,13.2,19.1,0.5,1,0.2,SSW,26,14.6,81,17.4,67
2017-06-17,QLD,Brisbane,40913,5,21.7,0.6,2.4,9,E,39,13.1,78,21.5,47
2017-06-17,VIC,Melbourne,86338,4.2,15.5,0,,,NE,15,5.4,95,14.9,66
2017-06-18,NSW,Sydney,66062,11.3,18,1.8,2,6.3,S,52,12.9,83,17.6,62
2017-07-02,NSW,Sydney,66062,5.4,17.6,0.5,2,9.5,W,28,7.6,78,16.1,44
2017-07-02,QLD,Brisbane,40913,3.9,15,20,1.6,4.2,S,35,7.3,90,14.2,46
2017-07-02,VIC,Melbourne,86338,0.8,14.3,0,,,N,41,3.8,75,13.9,36
2017-07-03,NSW,Sydney,66062,5.6,15.8,0.5,2.6,2.7,N,31,7.9,81,15.4,57
2017-07-03,QLD,Brisbane,40913,7,16.6,0.6,2,4.6,NNW,31,8.9,82,15.2,58
2017-07-03,VIC,Melbourne,86338,3.8,13.9,0,,,NNE,39,9,60,11.2,92
2017-07-04,NSW,Sydney,66062,7.9,21.8,0.2,0.6,9.3,,,13.2,76,21.5,23
2017-07-04,QLD,Brisbane,40913,5.5,14.4,2,1.4,1.2,N,19,8.3,94,13.9,73
2017-07-04,VIC,Melbourne,86338,9,16.2,3,,,N,33,11,78,15.6,54
2017-07-05,NSW,Sydney,66062,12.2,20.8,0.5,5.4,9.2,W,52,14.4,51,18.7,36
2017-07-05,QLD,Brisbane,40913,8,13.9,3.4,1.8,2.8,SW,59,11.9,92,10.7,83
2017-07-05,VIC,Melbourne,86338,10.1,15.8,0,,,SSW,24,12.4,75,13.4,65
2017-07-06,NSW,Sydney,66062,9.1,18.2,0.5,4.8,9.7,WSW,37,11.5,62,17.4,36
2017-07-06,QLD,Brisbane,40913,4.7,16,10.4,4.2,5.5,WNW,30,7.4,88,14.4,60
2017-07-06,VIC,Melbourne,86338,4.8,15,0,,,N,28,8.1,75,14.1,46
2017-07-07,NSW,Sydney,66062,6.6,19.1,0.5,3.6,9.4,WNW,35,8.8,73,18.6,24
2017-07-07,QLD,Brisbane,40913,7.4,17.6,9.8,1,6.9,SW,33,11.6,97,16.2,59
2017-07-07,VIC,Melbourne,86338,8,14.2,0,,,NNW,37,9,79,13.4,56
2017-07-08,NSW,Sydney,66062,7,18.1,0.5,3.2,9.3,W,37,9,66,17,36
2017-07-08,QLD,Brisbane,40913,3.2,17.2,0.6,1.4,9,S,28,10.2,83,16.5,41
2017-07-08,VIC,Melbourne,86338,5.3,13.5,0,,,NW,30,9,84,13.2,58
2017-07-09,NSW,Sydney,66062,7.4,18,0.5,2,9.5,W,33,9.2,69,17.6,33
2017-07-09,QLD,Brisbane,40913,7.1,17.8,0.6,2.4,7.4,NE,22,10.4,68,17.4,42
2017-07-09,VIC,Melbourne,86338,7.3,14.6,0.8,,,NNW,39,9.7,70,13.3,49
2017-07-10,NSW,Sydney,66062,7,17.3,0.5,3.6,9.4,W,39,9.4,67,16.7,36
2017-07-10,QLD,Brisbane,40913,5.9,17.9,0.6,2.4,4,NNE,39,12.5,52,17.6,40
2017-07-10,VIC,Melbourne,86338,7,16,0,,,W,28,9.2,74,15.2,48
2017-07-11,NSW,Sydney,66062,6.6,18.4,0.5,2.8,9.2,SSW,39,9,63,17.7,46
2017-07-11,QLD,Brisbane,40913,12.5,19.7,0.6,2.6,4.2,WNW,52,15.6,52,18.5,78
2017-07-11,VIC,Melbourne,86338,6.9,14.3,0.4,,,NW,19,9.8,79,13.8,57
2017-07-12,NSW,Sydney,66062,7.8,14.8,1.6,3.4,0.5,W,31,8.9,76,13,79
2017-07-12,QLD,Brisbane,40913,14.8,20.4,3.4,2.4,8,NW,61,18,69,19.2,59
2017-07-12,VIC,Melbourne,86338,3.2,15.1,0,,,N,30,4.8,89,14.7,49
2017-07-13,NSW,Sydney,66062,8,17.4,10.4,1,9.3,ENE,30,9.6,80,17.2,49
2017-07-13,QLD,Brisbane,40913,13.2,18.6,7.2,3.6,3,SW,33,15.1,96,18.1,60
2017-07-13,VIC,Melbourne,86338,4.8,13.3,0,,,N,56,10.9,56,13,54
2017-07-14,NSW,Sydney,66062,9.6,20.8,0.5,4,6.1,NNW,30,12.2,67,19.3,41
2017-07-14,QLD,Brisbane,40913,12.3,19.2,0.4,2.8,2.3,NE,24,14.6,76,18.1,62
2017-07-14,VIC,Melbourne,86338,9.9,15.9,2,,,N,50,10.5,76,15.2,41
2017-07-15,NSW,Sydney,66062,12.1,19.8,0.4,4,6.1,SW,30,13,71,17.9,39
2017-07-15,QLD,Brisbane,40913,11.4,19.9,0.6,2.6,1.3,WNW,50,14,96,16.9,82
2017-07-15,VIC,Melbourne,86338,4.5,12,0,,,NW,22,7.8,71,11.6,58
2017-07-16,NSW,Sydney,66062,7.7,18.4,0.5,1.8,9.4,WSW,31,9.4,61,16.8,32
2017-07-16,QLD,Brisbane,40913,13.7,19.2,4.6,2,3.6,SW,37,15.8,66,18.1,45
2017-07-16,VIC,Melbourne,86338,3,12.5,0,,,N,52,5.7,75,11.1,54
2017-07-17,NSW,Sydney,66062,7.9,19.3,0.5,4.2,6.5,W,24,9.8,75,17.3,44
2017-07-17,QLD,Brisbane,40913,8.7,19.2,0.6,3.4,9.1,NE,33,12.4,64,18.5,41
2017-07-17,VIC,Melbourne,86338,5.7,15.3,0,,,N,52,11.8,55,14.6,50
2017-07-18,NSW,Sydney,66062,9.1,22.9,0.5,1.2,9.8,NW,54,14.1,62,22.6,30
2017-07-18,QLD,Brisbane,40913,11.2,24.6,0.6,3.2,9.2,N,46,14.8,52,24.1,35
2017-07-18,VIC,Melbourne,86338,9.7,14.5,0.2,,,NNW,41,10.6,75,13.8,55
2017-07-19,NSW,Sydney,66062,10.6,17.5,0.5,6.6,9.1,WSW,57,12,53,16.9,33
2017-07-19,QLD,Brisbane,40913,13.2,18.7,8.4,3.8,3,N,61,14,96,14.1,89
2017-07-19,VIC,Melbourne,86338,8.9,13.2,5,,,SSW,37,10.8,99,11.6,81
2017-07-20,NSW,Sydney,66062,9.3,17.5,0.5,4.4,9.6,WSW,63,11.4,55,15.8,26
2017-07-20,QLD,Brisbane,40913,11.5,18.2,18.4,3.6,7.7,WSW,56,15.3,69,16.4,71
2017-07-20,VIC,Melbourne,86338,7.9,10.7,0.6,,,SSW,39,9.4,82,9.7,77
2017-07-21,NSW,Sydney,66062,7.5,18,0.5,4.2,9.2,W,46,9.6,53,17.4,37
2017-07-21,QLD,Brisbane,40913,10.6,20.1,5,0.6,7,NW,59,13.2,88,18.6,53
2017-07-21,VIC,Melbourne,86338,3.3,13.8,4.2,,,N,26,5.8,93,13.4,53
2017-07-22,NSW,Sydney,66062,6,19.1,0.5,3.6,9.8,W,31,7.9,64,18.7,27
2017-07-22,QLD,Brisbane,40913,10.7,19.3,6.1,,4.5,WSW,31,12.4,97,17.8,55
2017-07-22,VIC,Melbourne,86338,5.4,12.4,0,,,N,59,8.4,65,11.6,54
2017-07-23,NSW,Sydney,66062,6.5,22.1,0.5,3.6,9.9,NW,44,10.8,49,22.1,22
2017-07-23,QLD,Brisbane,40913,11.7,18.1,2.3,6.6,4.8,N,50,13.4,93,18,66
2017-07-23,VIC,Melbourne,86338,8.4,14.1,0,,,N,44,10.2,57,11.3,90
2017-07-24,NSW,Sydney,66062,10,21.3,0.5,7.2,9.6,WSW,31,12.7,51,20.9,24
2017-07-24,QLD,Brisbane,40913,10,18.6,16.8,4.2,3.9,W,39,12.4,95,17.1,50
2017-07-24,VIC,Melbourne,86338,8.1,16.1,7,,,WSW,28,9.9,77,15.2,53
2017-07-25,NSW,Sydney,66062,7.9,20.6,0.5,4.2,9.9,NW,30,11,63,19.7,32
2017-07-25,QLD,Brisbane,40913,10.4,19.6,3,0,3.4,NW,39,13,96,18.7,53
2017-07-25,VIC,Melbourne,86338,9.8,17.2,0,,,N,48,11.1,72,16.4,53
2017-07-26,NSW,Sydney,66062,11,21.3,0.5,2.8,9.9,W,70,18,39,20.2,27
2017-07-26,QLD,Brisbane,40913,11.7,19.2,0.4,2.2,3.1,SW,48,15.6,78,17.4,73
2017-07-26,VIC,Melbourne,86338,10.9,15,2.4,,,SW,37,12.5,70,12.7,71
2017-07-27,NSW,Sydney,66062,9.1,17.5,0.5,4.8,9.7,W,31,11.2,61,17.1,44
2017-07-27,QLD,Brisbane,40913,11.6,21.5,24,1.8,6.5,NNW,59,14.5,90,19.4,68
2017-07-27,VIC,Melbourne,86338,4.4,14.2,2,,,N,52,6.2,90,13.8,55
2017-07-28,NSW,Sydney,66062,11.2,19.1,0.5,4.2,8.3,WSW,67,13.1,51,18.9,25
2017-07-28,QLD,Brisbane,40913,14.5,17.7,10.8,3.4,,WNW,78,17,60,13.9,83
2017-07-28,VIC,Melbourne,86338,6.2,14.3,2.8,,,NNE,37,10.9,73,13.4,56
2017-07-29,NSW,Sydney,66062,8.2,21.2,0.5,4.6,9.9,W,31,10.7,52,20.9,23
2017-07-29,QLD,Brisbane,40913,9.7,15.1,11.2,6.8,5,WSW,72,12.1,77,12.3,86
2017-07-29,VIC,Melbourne,86338,9.7,17.9,0,,,N,81,12.7,51,16.9,41
2017-07-30,NSW,Sydney,66062,10.7,26.5,0.5,6,6.2,NNW,44,18.4,25,24.8,22
2017-07-30,QLD,Brisbane,40913,5.3,17.1,9.8,1.6,6.5,WSW,33,11.1,79,15.4,52
2017-07-30,VIC,Melbourne,86338,11.3,18.1,0,,,NW,31,13.7,64,17.5,34
2017-07-31,NSW,Sydney,66062,15.6,16.5,0.5,4.4,0,WSW,56,15.7,72,11.8,85
2017-07-31,QLD,Brisbane,40913,9.5,15.7,0.6,2.2,0.6,WSW,67,12.3,75,12.8,81
2017-07-31,VIC,Melbourne,86338,6.7,14.8,0,,,S,19,9,79,13.7,56
2017-08-01,NSW,Sydney,66062,8.6,18.1,6,2.4,9.8,SW,39,11.6,57,16.9,39
2017-08-01,QLD,Brisbane,40913,1.8,15.3,23.8,2.6,9,SSW,39,8.4,87,14.2,53
2017-08-01,VIC,Melbourne,86338,3.5,15.5,0,,,SSW,20,6.2,89,13,64
2017-08-02,NSW,Sydney,66062,7.2,18.2,0.5,3.2,8.8,S,41,9.6,62,16.8,51
2017-08-02,QLD,Brisbane,40913,1,14.9,0.6,0.6,9.7,E,26,8.7,81,14.3,47
2017-08-02,VIC,Melbourne,86338,3.4,12.9,0,,,NNE,13,5.6,100,12.8,61
2017-08-03,NSW,Sydney,66062,8.9,17.4,3,2.8,5.5,NNE,37,10.3,85,16.4,70
2017-08-03,QLD,Brisbane,40913,3.6,17.9,0.2,1.8,2.1,W,54,8.5,87,14.3,96
2017-08-03,VIC,Melbourne,86338,3.1,9,0,,,NNE,15,4.3,89,8.7,84
2017-08-04,NSW,Sydney,66062,10.3,17.3,13.8,1.8,10.2,W,50,12.8,60,16.6,41
2017-08-04,QLD,Brisbane,40913,7.9,15.5,1.6,2,1,WSW,30,10.3,77,14.3,64
2017-08-04,VIC,Melbourne,86338,4.3,12.1,6.6,,,N,50,8.9,87,11.5,66
2017-08-05,NSW,Sydney,66062,9.6,19.9,0.5,5.6,10.1,WNW,48,14,46,19.3,22
2017-08-05,QLD,Brisbane,40913,9.3,18.3,0.4,1.6,6.1,WSW,31,12.4,94,16.8,59
2017-08-05,VIC,Melbourne,86338,8.8,14.2,0,,,N,52,10.6,68,13.1,55
2017-08-06,NSW,Sydney,66062,12.1,21.3,0.5,6.8,9.9,NW,54,15,46,20.9,26
2017-08-06,QLD,Brisbane,40913,6.7,19.8,0.6,3.2,6.1,NW,37,11.8,81,19.4,41
2017-08-06,VIC,Melbourne,86338,9,13,0,,,NNE,54,10,76,11.7,54
2017-08-07,NSW,Sydney,66062,13.5,18.4,0.5,6.8,9.8,W,63,14.7,44,16.9,25
2017-08-07,QLD,Brisbane,40913,10.6,19.4,0.6,2.4,0.8,N,30,14.5,78,18.4,65
2017-08-07,VIC,Melbourne,86338,7.7,12.3,4.2,,,SW,41,9.4,70,11,79
2017-08-08,NSW,Sydney,66062,9.7,19.3,0.5,7.2,10.3,WSW,54,12.1,45,18.5,24
2017-08-08,QLD,Brisbane,40913,12.5,19.2,0.4,2,2.3,NW,57,15.9,82,14.8,93
2017-08-08,VIC,Melbourne,86338,7.9,15,2.2,,,NW,26,10.5,72,13.9,59
2017-08-09,NSW,Sydney,66062,9.2,20.9,0.5,5,10.4,W,33,12,53,20,32
2017-08-09,QLD,Brisbane,40913,8.5,14.7,43.6,4.4,3,SW,72,9.7,93,11.6,63
2017-08-09,VIC,Melbourne,86338,8.9,16.8,0,,,N,46,11.4,74,15.1,60
2017-08-10,NSW,Sydney,66062,9.2,24.2,0.5,4,9.4,NNW,31,11.8,64,23.5,24017-09-10,NSW,Sydney,66062,8,19.1,0.5,5,10.8,W,24,13.5,52,17.3,51
2017-09-10,QLD,Brisbane,40913,10.6,20.8,0.2,4.6,4.7,W,30,14.6,91,20.3,46
2017-09-10,VIC,Melbourne,86338,5.3,16.1,0.2,,,N,20,10.8,76,16,47
2017-09-11,NSW,Sydney,66062,8.7,22.7,0.5,3.8,10.1,NE,24,14.4,60,22.5,29
2017-09-11,QLD,Brisbane,40913,11.2,19.6,5.2,4.4,5.6,WSW,41,16.4,94,17.8,73
2017-09-11,VIC,Melbourne,86338,8.1,18.1,0,,,N,46,12.7,63,16.9,50
2017-09-12,NSW,Sydney,66062,11.3,27.2,0.5,5.8,2.3,E,26,18.2,43,23.4,34
2017-09-12,QLD,Brisbane,40913,13.2,19.4,1.4,4.2,7.9,SW,54,15.9,68,19,35
2017-09-12,VIC,Melbourne,86338,12.2,19.9,0.2,,,NNW,39,15.2,60,19.7,59
2017-09-13,NSW,Sydney,66062,18.2,33.8,0.5,6.2,9.6,NNW,70,24.9,19,32.5,10
2017-09-13,QLD,Brisbane,40913,7.7,21.6,0.6,6.4,11,E,43,13.3,50,21.6,27
2017-09-13,VIC,Melbourne,86338,13.3,16.4,2,,,WSW,50,14.9,62,11.7,69
2017-09-14,NSW,Sydney,66062,12,17.3,0.2,13,10.5,WSW,72,13.8,37,16.4,26
2017-09-14,QLD,Brisbane,40913,6.3,25.6,0.6,6,7,WSW,30,17.2,34,24.1,27
2017-09-14,VIC,Melbourne,86338,5.8,15.3,1.4,,,W,43,9.6,66,14.1,56
2017-09-15,NSW,Sydney,66062,10.5,22.8,0.5,7.8,10.6,W,52,16.6,37,22.2,24
2017-09-15,QLD,Brisbane,40913,14.5,27.8,0.6,4.6,8.5,ENE,37,17.6,43,27.6,26
2017-09-15,VIC,Melbourne,86338,9.6,16.7,0,,,WSW,35,13.5,58,14.8,82
2017-09-16,NSW,Sydney,66062,12.5,24.2,0.5,7,6.4,SSW,56,17.3,43,21.3,22
2017-09-16,QLD,Brisbane,40913,10.9,24.4,0.6,4.8,6.5,WSW,33,15,96,21.9,48
2017-09-16,VIC,Melbourne,86338,7.4,13.3,21.4,,,SSW,54,9.1,81,12.5,52
2017-09-17,NSW,Sydney,66062,8.1,18.6,0.5,4.6,10.8,ESE,28,13.6,48,16.4,37
2017-09-17,QLD,Brisbane,40913,14.6,21.7,0.4,4.4,9.6,WSW,33,18.5,72,21.1,47
2017-09-17,VIC,Melbourne,86338,4,18.1,0.2,,,N,43,10.5,58,17.8,35
2017-09-18,NSW,Sydney,66062,8.2,21,0.5,5.6,10.7,NE,39,14,55,20.2,55
2017-09-18,QLD,Brisbane,40913,7.6,26,0.6,4,10.4,E,37,15.8,71,25.7,34
2017-09-18,VIC,Melbourne,86338,10.5,22,0,,,NNW,63,15.7,32,20.9,30
2017-09-19,NSW,Sydney,66062,14,25.2,0.5,6.4,10.8,SSW,54,20.4,30,23.1,16
2017-09-19,QLD,Brisbane,40913,8.9,22.4,0.6,7,8.5,WSW,31,17.9,73,21,67
2017-09-19,VIC,Melbourne,86338,9,13.9,0.2,,,WNW,31,11.6,58,12.4,58
2017-09-20,NSW,Sydney,66062,10.7,19,0.5,9.2,7.7,WSW,33,14.6,44,17.7,56
2017-09-20,QLD,Brisbane,40913,14.7,21.4,5.4,4.6,6.9,W,48,18,57,20.6,52
2017-09-20,VIC,Melbourne,86338,6,19.2,0,,,N,35,11.6,63,18.6,42
2017-09-21,NSW,Sydney,66062,10.4,21.3,0.5,3.8,10.1,NE,30,15.6,65,21.2,49
2017-09-21,QLD,Brisbane,40913,16.8,21.5,5.8,5,3.4,NW,81,19.8,72,19,61
2017-09-21,VIC,Melbourne,86338,11.6,23.6,0,,,N,54,14.8,46,22.4,33
2017-09-22,NSW,Sydney,66062,12.8,27.7,0.5,7.2,10.5,NNE,35,20.1,35,25.8,26
2017-09-22,QLD,Brisbane,40913,13,19.2,7.4,5.2,1.9,W,89,14.5,88,17.3,58
2017-09-22,VIC,Melbourne,86338,12.8,25.6,0,,,N,33,14.9,71,24.8,32
2017-09-23,NSW,Sydney,66062,15.5,32.2,0.5,8.6,5.7,NNE,50,23,25,29.1,25
2017-09-23,QLD,Brisbane,40913,11.4,17.1,9.8,5.4,5.3,SW,52,13.4,72,16.1,48
2017-09-23,VIC,Melbourne,86338,14.8,30.6,0,,,N,61,21.1,30,30,24
2017-09-24,NSW,Sydney,66062,23,29.2,0.5,12,6.5,W,54,27.1,20,27.8,15
2017-09-24,QLD,Brisbane,40913,10.1,17.3,5.2,4.4,4.5,NNW,46,14.8,75,14,87
2017-09-24,VIC,Melbourne,86338,12.9,19.9,0.4,,,NNW,56,14.9,56,18.4,40
2017-09-25,NSW,Sydney,66062,16,26.7,0.5,10.8,10.8,WNW,57,22.3,21,23.3,19
2017-09-25,QLD,Brisbane,40913,9.3,17,10.4,3.2,4.6,NE,35,11.8,81,14.9,67
2017-09-25,VIC,Melbourne,86338,10.3,14.2,1.6,,,SSW,35,12.9,64,12.8,68
2017-09-26,NSW,Sydney,66062,12.3,22.2,0.5,9.6,11,WSW,41,19.5,29,19,47
2017-09-26,QLD,Brisbane,40913,6,18.2,3.4,4.4,9.3,WSW,41,13.1,69,17.5,40
2017-09-26,VIC,Melbourne,86338,5,15.1,0.4,,,S,24,11.3,64,14,46
2017-09-27,NSW,Sydney,66062,15.4,22.7,0.5,6.6,8.2,ENE,50,18.1,58,21.1,62
2017-09-27,QLD,Brisbane,40913,5,16.7,0.6,6.8,5.6,SW,41,13.1,58,14.7,56
2017-09-27,VIC,Melbourne,86338,9.6,24.6,0,,,NNW,50,12.5,64,24.5,30
2017-09-28,NSW,Sydney,66062,18,25.7,0.5,6.2,3.8,WNW,59,22.7,47,24.1,26
2017-09-28,QLD,Brisbane,40913,6.9,16.6,1,3.4,3.5,WSW,33,12.7,72,13.5,76
2017-09-28,VIC,Melbourne,86338,10.4,18.5,0,,,SW,35,14.1,66,16.8,45 2017-10-01,VIC,Melbourne,86338,9.7,17.2,0,,,S,24,12.4,65,16.4,58
2017-10-02,NSW,Sydney,66062,12,22.9,0.5,6.2,10.1,NE,44,16.7,55,21.3,60
2017-10-02,QLD,Brisbane,40913,7.7,24,0.6,4.8,11.3,WSW,39,19.6,43,22.1,45
2017-10-02,VIC,Melbourne,86338,9.5,16.4,0,,,SSW,24,12.1,75,15.2,58
2017-10-15,NSW,Sydney,66062,16,22.4,0.5,3.8,2.4,ESE,28,22.3,52,20.1,58
2017-10-15,QLD,Brisbane,40913,16.4,25.4,0.6,7.2,11.6,W,33,20.4,61,22.4,53
2017-10-15,VIC,Melbourne,86338,6.8,17.6,0,,,SSW,31,12.4,67,16.7,67
2017-10-16,NSW,Sydney,66062,15.9,23.3,0.5,5,10.4,E,30,22.4,55,22.3,50
2017-10-16,QLD,Brisbane,40913,15.3,19.9,3,5.8,6.5,NW,91,15.8,91,18.2,51
2017-10-16,VIC,Melbourne,86338,11.3,28.6,0,,,NNE,28,12.9,91,26.2,40
2017-10-17,NSW,Sydney,66062,16.9,23.5,0.5,7.6,9.9,ENE,39,21.2,57,22.9,47
2017-10-17,QLD,Brisbane,40913,13.9,20.1,7.6,5.4,5.8,WNW,52,17,71,19.8,50
2017-10-17,VIC,Melbourne,86338,12.8,30.2,0,,,N,37,21,50,30.1,24
2017-10-18,NSW,Sydney,66062,18.8,23.2,0.5,9,10.4,NNE,50,20.9,64,22.5,56
2017-10-18,QLD,Brisbane,40913,12,19.4,5,5,11.7,SW,44,16.7,46,18.2,37
2017-10-18,VIC,Melbourne,86338,14.5,30.9,0,,,NNW,48,21.7,43,29.9,31
2017-10-19,NSW,Sydney,66062,17.1,25.3,0.5,10,11.4,NNE,52,22.1,52,24.6,47
2017-10-19,QLD,Brisbane,40913,6.8,24.7,0.6,6.4,12.1,E,56,16.8,44,24.4,24
2017-10-19,VIC,Melbourne,86338,21.7,25.6,0,,,SW,43,23.8,45,23.5,71
2017-10-20,NSW,Sydney,66062,19,19.6,7.8,9.2,0,S,56,19.6,87,15.3,85
2017-10-20,QLD,Brisbane,40913,12.1,29.9,0.6,6,12.1,E,61,22.3,31,29.4,17
2017-10-20,VIC,Melbourne,86338,11.3,16.2,2.6,,,S,41,13.4,69,15.3,56
2017-10-21,NSW,Sydney,66062,13,20,16.4,4.4,11.3,SSW,50,16.8,52,18.9,53
2017-10-21,QLD,Brisbane,40913,15.9,29.5,0.6,9.2,4.9,ENE,65,23.2,32,28.2,26
2017-10-21,VIC,Melbourne,86338,11,15.4,0,,,S,28,11.8,71,14.8,63
2017-10-22,NSW,Sydney,66062,13.4,21.7,0.2,4.8,5.5,ESE,30,16,75,18.7,60
2017-10-22,QLD,Brisbane,40913,18.5,28.4,1.2,7.2,11.7,E,70,23.1,56,27.1,46
2017-10-22,VIC,Melbourne,86338,11.1,16,0,,,SSW,28,12.7,78,15.2,76
2017-10-23,NSW,Sydney,66062,12.7,23,3.4,3.6,12.1,S,31,18,60,22.1,43
2017-10-23,QLD,Brisbane,40913,14.3,23.6,0.6,5.6,3.8,WSW,37,18.2,80,22.4,57
2017-10-23,VIC,Melbourne,86338,11.9,19,0,,,SSW,31,13.6,72,17.3,61
2017-10-24,NSW,Sydney,66062,13.9,24.3,0.5,6.8,10.5,NE,39,19.9,57,22.6,55
2017-10-24,QLD,Brisbane,40913,14.3,22.6,0.6,6.2,11,W,37,18.2,63,20.3,40
2017-10-24,VIC,Melbourne,86338,12.1,20.5,0.2,,,SSW,26,15,74,17.4,61
2017-10-25,NSW,Sydney,66062,18,26.9,0.5,8,5,S,59,21,67,23.3,56
2017-10-25,QLD,Brisbane,40913,9.6,22.7,0.6,7,7.3,W,44,19,69,21.7,54
2017-10-25,VIC,Melbourne,86338,13.3,18.5,11.4,,,SSW,24,14.5,94,17.3,67
2017-10-26,NSW,Sydney,66062,17.2,24.6,0.4,7,6.5,WSW,63,19.7,73,22.6,63
2017-10-26,QLD,Brisbane,40913,13.3,20.9,1.2,4.6,5.3,WNW,67,17.6,77,16.4,66
2017-10-26,VIC,Melbourne,86338,12.9,16.8,0.2,,,SSW,39,13.8,83,15.5,73
2017-10-27,NSW,Sydney,66062,14.8,24.2,34.2,7.2,3.9,WSW,61,18.1,85,22.3,65
2017-10-27,QLD,Brisbane,40913,8.4,19.3,2,6.2,4.8,WSW,44,16.7,48,17.5,51
2017-10-27,VIC,Melbourne,86338,10.3,27.4,0,,,N,48,15.1,78,25.2,37
2017-10-28,NSW,Sydney,66062,17.5,25.1,0.2,3.2,7.7,ENE,35,21.6,66,22.3,65
2017-10-28,QLD,Brisbane,40913,14,20.5,1,4.2,6.2,WSW,52,18.4,54,19.8,54
2017-10-28,VIC,Melbourne,86338,15,21.4,0.4,,,N,39,15.4,72,19.7,46
2017-10-29,NSW,Sydney,66062,20.2,29.6,0.2,4.6,10.6,NNE,39,24.4,49,27.7,35
2017-10-29,QLD,Brisbane,40913,12.6,20.6,0.8,4.2,11.6,SW,54,16.7,46,19.9,41
2017-10-29,VIC,Melbourne,86338,11.5,28.6,0,,,N,61,19,63,27.4,25
2017-10-30,NSW,Sydney,66062,20.3,35.4,0.5,11.4,9.3,SSW,69,28.5,37,34.9,13
2017-10-30,QLD,Brisbane,40913,7.9,27.5,0.6,6.8,12.5,WSW,35,19.1,44,25.9,33
2017-10-30,VIC,Melbourne,86338,8.8,17,4.4,,,W,46,12.6,62,14.4,55
2017-10-31,NSW,Sydney,66062,13,20.5,0.5,12,10.1,SSE,57,15.8,43,19.7,36
2017-10-31,QLD,Brisbane,40913,8.7,29.3,0.6,10.4,12.2,SW,39,20.2,54,27.1,35
2017-10-31,VIC,Melbourne,86338,8.1,16.4,6,,,S,28,12,62,15.4,48
2017-11-01,NSW,Sydney,66062,13,22.2,0.5,9.6,4.2,SSW,54,15.4,39,21,32
2017-11-01,QLD,Brisbane,40913,15,30.5,0.6,10.4,10.4,SE,44,21.5,48,29,39
2017-11-01,VIC,Melbourne,86338,11.9,18,0.4,,,SSW,26,14.7,78,15.8,66
2017-11-02,NSW,Sydney,66062,15.3,22.6,0.5,5.6,9.9,ESE,31,20.1,56,21.8,46
2017-11-02,QLD,Brisbane,40913,14.7,27.6,0.6,10.2,11.9,E,46,19.4,51,27,35
2017-11-02,VIC,Melbourne,86338,11.3,17.4,1.4,,,SSE,26,12.7,79,15.9,56
2017-11-03,NSW,Sydney,66062,16.9,26.9,0.5,6.6,5.9,SSW,57,20.6,58,25.7,48
2017-11-03,QLD,Brisbane,40913,13.6,26.9,0.6,9.8,12.3,ESE,43,19.5,49,25.3,41
2017-11-03,VIC,Melbourne,86338,9.4,16.6,1.4,,,S,46,12,58,15.3,45
2017-11-04,NSW,Sydney,66062,16.2,17.1,1.6,7.4,0,SSE,39,17,73,15.8,78
2017-11-04,QLD,Brisbane,40913,14.8,30.7,0.6,8,12.6,E,69,23.1,43,30.3,20
2017-11-04,VIC,Melbourne,86338,8.3,15.8,0.4,,,SSW,35,10.2,72,14.5,47
2017-11-05,NSW,Sydney,66062,14.1,19.5,31.8,2.6,0,ENE,39,14.9,85,17.7,59
2017-11-05,QLD,Brisbane,40913,15.2,31.1,0.6,10.2,12.6,E,67,21.6,38,31,20
2017-11-05,VIC,Melbourne,86338,6.4,17.6,0,,,SSW,39,13.4,59,15.6,49
2017-11-06,NSW,Sydney,66062,14.8,26.3,4.4,1.8,5.7,SSW,67,19.4,81,23.6,57
2017-11-06,QLD,Brisbane,40913,14.9,32.7,0.6,10.2,12.5,E,67,23.5,33,32.7,11
2017-11-06,VIC,Melbourne,86338,10.9,16.1,0,,,S,43,13.8,66,14.8,60
2017-11-07,NSW,Sydney,66062,13.8,21.8,5.6,8.8,11.3,SSE,54,18.9,40,20,40
2017-11-07,QLD,Brisbane,40913,17.3,36.4,0.6,12,12.5,E,59,27.1,21,35.4,14
2017-11-07,VIC,Melbourne,86338,10,15.8,3.4,,,S,43,12.2,76,14.4,52
2017-11-08,NSW,Sydney,66062,14.6,21.7,0.5,9.6,10,SSE,44,16.8,54,20.5,44
2017-11-08,QLD,Brisbane,40913,20,35.6,0.6,11.2,10.8,ENE,37,28,30,33.1,24
2017-11-08,VIC,Melbourne,86338,12.1,16.9,0,,,S,30,13.9,59,15.6,64
2017-11-09,NSW,Sydney,66062,12.3,22.1,0.5,6.8,11.6,E,28,17.9,52,21.2,50
2017-11-09,QLD,Brisbane,40913,15.2,31.6,0.6,7.8,12.8,SW,37,26.9,42,29.7,31
2017-11-09,VIC,Melbourne,86338,8.7,21.8,0,,,SSW,28,14.6,68,18.1,66
2017-11-10,NSW,Sydney,66062,14.2,23.4,0.5,6.8,11.6,E,26,19.2,59,22.7,50
2017-11-10,QLD,Brisbane,40913,14.1,32,0.6,8.4,12.6,W,33,26.4,41,30.1,37
2017-11-10,VIC,Melbourne,86338,12.1,27.2,0,,,SE,20,15.7,78,22.8,63
2017-11-11,NSW,Sydney,66062,14.7,23.7,0.5,7.4,8.6,ENE,33,19.9,58,20.9,56
2017-11-11,QLD,Brisbane,40913,16.2,38,0.6,9,10,W,41,30.4,32,33.5,27
2017-11-11,VIC,Melbourne,86338,14.8,24.7,0,,,SSW,26,17.8,79,22,65
2017-11-12,NSW,Sydney,66062,17.1,22.5,0.5,7.4,10.7,E,31,19.7,49,21.7,47
2017-11-12,QLD,Brisbane,40913,22.4,32.1,0.6,11.8,11.3,S,39,28.7,39,28.8,46
2017-11-12,VIC,Melbourne,86338,13.2,23.3,0,,,SSW,28,16.1,86,21.6,69
2017-11-13,NSW,Sydney,66062,17.5,22.3,0.5,6.4,9.1,E,33,19.1,48,21.2,42
2017-11-13,QLD,Brisbane,40913,17.5,30.1,0.6,10.2,12.7,SW,50,25.2,65,27.9,45
2017-11-13,VIC,Melbourne,86338,13.7,33,0,,,NE,24,19.3,73,32,20
2017-11-14,NSW,Sydney,66062,16,22.9,0.5,7.6,8.9,ENE,37,20,53,22.3,46
2017-11-14,QLD,Brisbane,40913,14.5,29.7,0.6,11.8,12.7,SE,54,20.2,47,29.1,19
2017-11-14,VIC,Melbourne,86338,18.2,33.9,0,,,N,48,25.3,38,33.1,27

Related

So I have an excel file containing data on a specific stock.
My excel file contains about 2 months of data, it monitors the Open price, Close price, High Price, Low Price and Volume of trades in 5 minute intervals, so there are about 3000 rows in my file.
I want to calculate the RSI (or EMA if it's easier) of a stock daily, I'm making a summary table that collects the daily data so it converts my table of 3000+ rows into a table with only about 60 rows (each row represents one day).
Essentially I want some sort of code that sorts the excel data by date then calculates the RSI as a single value for that day. RSI is given by: 100-(100/(1+RS)) where RS = average gain of up periods/average loss of down periods.
Note: My excel uses 'Datetime' so each row's 'Datetime' looks something like '2022-03-03 9:30-5:00' and the next row would be '2022-03-03 9:35-5:00', etc. So the code needs to just look at the date and ignore the time I guess.
Some code to maybe help understand what I'm looking for:
So here I'm calling my excel file, I want the code to take the called excel file, group data by date and then calculate the RSI of each day using the formula I wrote above.
dat = pd.read_csv('AMD_5m.csv',index_col='Datetime',parse_dates=['Datetime'],
date_parser=lambda x: pd.to_datetime(x, utc=True))
dates = backtest.get_dates(dat.index)
#create a summary table
cols = ['Num. Obs.', 'Num. Trade', 'PnL', 'Win. Ratio','RSI'] #add addtional fields if necessary
summary_table = pd.DataFrame(index = dates, columns=cols)
# loop backtest by dates
This is the code I used to fill out the other columns in my summary table, I'll put my SMA (simple moving average) function below.
for d in dates:
this_dat = dat.loc[dat.index.date==d]
#find the number of observations in date d
summary_table.loc[d]['Num. Obs.'] = this_dat.shape[0]
#get trading (i.e. position holding) signals
signals = backtest.SMA(this_dat['Close'].values, window=10)
#find the number of trades in date d
summary_table.loc[d]['Num. Trade'] = np.sum(np.diff(signals)==1)
#find PnLs for 100 shares
shares = 100
PnL = -shares*np.sum(this_dat['Close'].values[1:]*np.diff(signals))
if np.sum(np.diff(signals))>0:
#close position at market close
PnL += shares*this_dat['Close'].values[-1]
summary_table.loc[d]['PnL'] = PnL
#find the win ratio
ind_in = np.where(np.diff(signals)==1)[0]+1
ind_out = np.where(np.diff(signals)==-1)[0]+1
num_win = np.sum((this_dat['Close'].values[ind_out]-this_dat['Close'].values[ind_in])>0)
if summary_table.loc[d]['Num. Trade']!=0:
summary_table.loc[d]['Win. Ratio'] = 1. *num_win/summary_table.loc[d]['Num. Trade']
This is my function for calculating Simple Moving Average. I was told to try and adapt this for RSI or for EMA (Exponential Moving Average). Apparently adapting this for EMA isn't too troublesome but I can't figure it out.
def SMA(p,window=10,signal_type='buy only'):
#input price "p", look-back window "window",
#signal type = buy only (default) --gives long signals, sell only --gives sell signals, both --gives both long and short signals
#return a list of signals = 1 for long position and -1 for short position
signals = np.zeros(len(p))
if len(p)<window:
#no signal if no sufficient data
return signals
sma = list(np.zeros(window)+np.nan) #the first few prices does not give technical indicator values
sma += [np.average(p[k:k+window]) for k in np.arange(len(p)-window)]
for i in np.arange(len(p)-1):
if np.isnan(sma[i]):
continue #skip the open market time window
if sma[i]<p[i] and (signal_type=='buy only' or signal_type=='both'):
signals[i] = 1
elif sma[i]>p[i] and (signal_type=='sell only' or signal_type=='both'):
signals[i] = -1
return signals

I have two solutions to this. One is to loop through each group, then add the relevant data to the summary_table, the other is to calculate the whole series and set the RSI column as this.
I first recreated the data:
import yfinance
import pandas as pd
# initially created similar data through yfinance,
# then copied this to Excel and changed the Datetime column to match yours.
df = yfinance.download("AAPL", period="60d", interval="5m")
# copied it and read it as a dataframe
df = pd.read_clipboard(sep=r'\s{2,}', engine="python")
df.head()
# Datetime Open High Low Close Adj Close Volume
#0 2022-03-03 09:30-05:00 168.470001 168.910004 167.970001 168.199905 168.199905 5374241
#1 2022-03-03 09:35-05:00 168.199997 168.289993 167.550003 168.129898 168.129898 1936734
#2 2022-03-03 09:40-05:00 168.119995 168.250000 167.740005 167.770004 167.770004 1198687
#3 2022-03-03 09:45-05:00 167.770004 168.339996 167.589996 167.718094 167.718094 2128957
#4 2022-03-03 09:50-05:00 167.729996 167.970001 167.619995 167.710007 167.710007 968410
Then I formatted the data and created the summary_table:
df["date"] = pd.to_datetime(df["Datetime"].str[:16], format="%Y-%m-%d %H:%M").dt.date
# calculate percentage change from open and close of each row
df["gain"] = (df["Close"] / df["Open"]) - 1
# your summary table, slightly changing the index to use the dates above
cols = ['Num. Obs.', 'Num. Trade', 'PnL', 'Win. Ratio','RSI'] #add addtional fields if necessary
summary_table = pd.DataFrame(index=df["date"].unique(), columns=cols)
Option 1:
# loop through each group, calculate the average gain and loss, then RSI
for grp, data in df.groupby("date"):
# average gain for gain greater than 0
average_gain = data[data["gain"] > 0]["gain"].mean()
# average loss for gain less than 0
average_loss = data[data["gain"] < 0]["gain"].mean()
# add to relevant cell of summary_table
summary_table["RSI"].loc[grp] = 100 - (100 / (1 + (average_gain / average_loss)))
Option 2:
# define a function to apply in the groupby
def rsi_calc(series):
avg_gain = series[series > 0].mean()
avg_loss = series[series < 0].mean()
return 100 - (100 / (1 + (avg_gain / avg_loss)))
summary_table["RSI"] = df.groupby("date")["gain"].apply(lambda x: rsi_calc(x))
Output (same for each):
summary_table.head()
# Num. Obs. Num. Trade PnL Win. Ratio RSI
#2022-03-03 NaN NaN NaN NaN -981.214015
#2022-03-04 NaN NaN NaN NaN 501.950956
#2022-03-07 NaN NaN NaN NaN -228.379066
#2022-03-08 NaN NaN NaN NaN -2304.451654
#2022-03-09 NaN NaN NaN NaN -689.824739

I have monthly data which I split up by all months by each year, so each month (january, feb, etc.) has 25 points (25 years worth of monthly data).
The reason why I split up the data set per month is because I wanted to detrend each month separately.
Now the issue I have now is how do I rebuild the data set to get back to normal, monthly data?
Just to clarify, the monthly data set is split up in January, Febuary, March etc. with 25 points of all the Januaries for each year (25 total years).
This is my function to split the months up:
def split_months(monthly_data):
month_split = []
for month in range(12):
month_split.append(monthly_data[month::12])
month_split = np.array(month_split)
return month_split
The function applied to my time series is:
lsds = split_months(ob_monthly(fwf_tot_grnl,26,28))
print(lsds.shape)
(12, 25)
The original time series is basically this exact array of values (each number representing monthly means):
data = np.array(
[ 4.44 3.614 4.0460005 5.1229997 8.181001 23.88
48.895 26.740002 7.454 9.884 5.093 4.2770004
3.8999999 4.9040003 3.736 4.4030004 22.261 40.200005
62.825005 47. 20.904001 10.974 5.723 4.5070004
4.04 3.5040002 3.7760003 8.803 19.691 25.249998
64.494995 44.36 8.723998 5.4339995 4.6829996 4.1770005
3.8999999 3.4340003 3.726 11.383001 19.711 39.320004
81.924995 44.109993 15.584 6.5639997 9.043 5.3770003
4.9 5.644 5.426 9.943001 22.291 30.459997
45.245003 30.32 15.934 5.694 5.873 4.597
4.6400003 3.654 3.9460003 8.813 21.901 46.29
54.90499 33.729996 14.974 7.3840003 5.9030004 5.0770006
4.4700007 3.764 4.586 10.782999 19.921 59.17
77.784996 41.09 13.904 5.244 4.183 4.077
4.596 5.039 4.853 10.633999 11.36 25.538
66.06901 60.221 9.421 5.9580007 7.3010006 5.1370006
9.871 5.959001 6.2910004 11.011 15.582 41.586002
61.489 62.522 22.592005 8.76 7.284001 6.6680017
6.3420005 5.6429996 6.397001 13.169999 15.686999 35.1
69.087 61.676994 16.506002 8.993002 9.748 9.421001
7.5270014 6.089 6.802001 7.0059996 25.265001 52.129
71.71501 37.315002 23.323002 8.122001 6.8900003 6.7650013
7.5070014 5.7620006 6.369001 9.704 16.115002 61.615997
78.908 75.03501 24.828001 18.709 8.946001 7.6440015
7.036001 8.021001 6.745001 8.9609995 25.263998 53.342
66.784 60.457996 12.041001 8.851002 7.6539993 7.332002
8.109002 9.958001 10.364 9.103999 25.901 49.302998
77.47001 47.663006 17.101002 9.033002 12.889999 8.370002
7.4340014 6.643001 8.225 8.264002 27.967001 30.463001
71.04099 66.920006 15.218003 10.14 7.986001 7.931002
11.101 6.612001 7.8220005 8.818002 15.4159975 57.508
95.218994 76.62101 15.224001 9.743 9.98 7.506002
7.056001 6.377001 8.582001 11.452001 26.950003 48.557995
80.06599 46.355 13.2630005 13.066001 10.836 8.853002
7.9060006 6.7550006 7.1250005 8.478 18.976002 39.144997
74.204994 46.373 10.978003 8.085002 7.340002 7.4710016
8.144002 6.551 7.689001 11.208 36.002 66.902
102.02799 82.63899 26.937 11.016 9.987 8.977001
8.089001 6.610002 7.6930013 7.0730014 16.036 50.027996
103.17498 66.486 13.222001 8.791002 8.938003 7.6330013
7.270001 7.9689994 9.035001 15.478998 25.383999 85.61101
131.645 70.778 19.850004 15.265 12.019999 9.633
8.530001 7.2450004 9.307002 11.766 11.969999 38.693996
69.27799 44.055004 14.650001 11.418 9.868003 8.571002
8.055 7.061002 7.7850013 7.9510007 21.322002 45.706993
87.51201 66.734 16.962 9.555001 8.364002 8.183001
7.443001 7.8109994 6.9640007 6.980001 14.530002 38.425
78.09 38.747 18.515001 10.052002 8.556001 7.909002
7.576001 6.868001 7.470001 17.761002 23.598999 55.842
100.045 65.67799 10.5130005 7.649001 7.543002 6.8360014])
Then you can apply my function split_months(data) to split the data into separate months like January (25 values), February (25 values) etc.
How do I go back to monthly data after splitting each month separately? Basically how would I return to the original data array (called data)?

I would apply a nested loop in which the data gets stored like this: First value first row, first value second row,... first value last row. Then second value first row etc... This returns your original data. I defined it as a function as well:
def merge_months(split_data):
merged_months = []
for value in range(len(split_data[0])):
for month in range(len(split_data)):
merged_months.append(split_data[month][value])
return merged_months
Is that what you're looking for? Note that this only works if all months have an equal amount of data points. Otherwise you need to change len(split_data[0])): in the first line of the nested loop.

2 Daily files are placed in a folder, format being Items_20190102 and Items_20190102_orig.
I would like to pull the file via a loop without the 'orig' ending and perform aggregation using groupby on item level.
I would like to tally the daily aggregates into a monthly running total so as to produce a monthly dashboard throughout the year.
So far i have been able to group the data at item level and then compute running total for the day, on three summarized totals.
Uncalibrated spend
Calibrated spend.
Total Spend
import pandas as pd
l = ['Umbrella', 'Umbrella']
df = pd.DataFrame({'Date':['10/2/2011', '11/2/2011', '12/2/2011', '13/2/2011'],
'Product':['Umbrella', 'Umbrella', 'Badminton', 'Shuttle'],
'Last Price':[1200, 1500, 1600, 352],
'Updated Price':[1250, 1450, 1550, 400],
'Discount':[10, 10, 10, 10]})
Uncalibrated = df[df['Product'].isin(l)].groupby('Product')['Last Price'].sum()
Uncalibrated_abs = abs(uncalibrated)
Uncalibrated spend = Uncalibrated_abs.sum()
Calibrated = df[~df['Product'].isin(l)].groupby('Product')['Last Price'].tail(1)
Calibrated_Abs = abs(Calibrated)
Calibrated spend = Calibrated_Abs.sum()
Total = df.groupby(['Product'])['Last Price'].sum()
Total_Spend = abs(Total).sum()
The result of this code will be in the form of three summarized statistics:
Uncalibrated_Spend.
Calibrated Spend.
Total Spend.
Now, i would like to generate a loop so that every day that this code runs it appends the three statsitcs generated from that daily file to a new column with a date as header and also generates a running total for the month on another column.
Output
26/06/2019 27/06/2016 Cum Monthly Total
Uncalibrated_Spend. xxxxxx xxxx xxxx
Calibrated Spend. xxxxxx xxxx xxxx
Total Spend. xxxxxx xxxx xxxx

I have a list representing the year which is filled with a sub-list for each day of the year.
year = []
for i in range(0,52*7):
day = [i,0] #[day number, 0 = empty, 1 = something is planned]
year.append(day)
I also have a variable list of activities created by a class.
class Activities:
def __init__(self,name,weeks,weekends):
self.name = name
self.weeks = weeks
self.weekends = weekends
def __repr__(self):
return repr((self.name,self.weeks,self.weekends))
def activityMaker(activityList):
a= []
for i in range(0, len(activityList)):
a.append(Activities(activityList[i][0], activityList[i][1], activityList[i][2]))
a = sorted(a, key=lambda Activities: Activities.weeks)
activityList = a
return activityList
As an example;
>>> activityList = [['Tennis', 3, 0], ['Baseball', 4, 0], ['Swimming', 2, 0]]
>>>activities= activityMaker(activityList)
Which returns 'activities', sorted on Activities.weeks:
>>>activities[0].name
activities[0].week
activities[0].weekend
>>> 'Swimming' # activity name
2 #"i want to do this activity once every x weeks
0 # 0 = no preferance, 1 = not in weekends
Now here my dilemma. I wish to create an algorithm to fill year with the activities with as much rhythm as possible.
My current approach is not working properly. What I'm doing now is as follows.
for y in range(0,len(year), int(7*activities[0].weeks)):
year[y][1] = activities[i].name
Now the first activity is planned for each y. If I had two activities, which I each want planned once a week, I could plan the first on 0th, 7th, 14th etc, and the second on 3rd, 10th, 17th etcetera.
The problem with this approach is exemplified if activities[0] and activities[1] are respectively 2 and 3. If I apply the previous approach, activities[0] would be planned on the 0th, 14th, 28th etc, which is fine on its own. Between the 0th and 14th, the 2nd activity would be ideally placed on the 7th position, meaning the next time would be the 28th. On the 28 the first activity is already planned, however. This means that's there's nothing planned for two weeks and then suddenly 2 activities in a day. The second activity could be pushed to the 27th or 29th, but that would still mean that now activities are planned on the 0th, 7th, 14th, 28th, 29th. Aka, there's still 14 days between the 14th and the 28th, and then only 1 between the 28th and 29th.
In what way can I make sure that all activities are planned with as much average time in between activities?

Your problem is that unless the number of weeks is the same for all activities (so they all have the same rhythm, there will be some weeks with lots of activities and some weeks with none.
What I would suggest instead is this: As you walk through the weeks of the year, simply choose an activity (or two) at random for each week. That way, every week will have a modest amount of activity planned. Here's some example code:
import random
activities = ["Baseball", "Tennis", "Swimming", ... ]
skip_days = 3
year = {}
for y in range(0, 52*7, skip_days):
year[y] = random.choose(activities)
print year[0]
>>> "Swimming" (perhaps)
print year[15]
>>> "Baseball"
print year[17]
>>> None
If you want more activity, make skip_days smaller. If you want less, make it bigger. If you want a fixed amount of activity in each week, you could do something like
for y in range(0, 52*7, 7):
year[y] = random.choose(activities)
year[y+3] = random.choose(activities)
This would plan two days a week.

I have a data set recording different weeks and the new cases of dengue for that specific week and I am supposed to calculate the infection rate and recovery rate for each week. The infection rate can be calculated by dividing the number of newly infected patients by the susceptible population for that week while the recovery rate can be calculated by dividing the number of newly recovered patients by the infected population for that week. The infection rate is relatively simple but for the recovery rate I have to take into account that infected patients take exactly 2 weeks to recover and I'm stuck. Any help would be appreciated
t_pop = 4*10**6
s_pop = t_pop
i_pop = 0
r_pop = 0
weeks = 0
#Infection Rate
for index, row in data.iterrows():
new_i = row['New Cases']
s_pop -= new_i
weeks += 1
infection_rate = float(new_i)/float(s_pop)
print('Week', weeks, ':' ,infection_rate)
*Note: t_pop refers to total population which we assume to be 4million, s_pop refers to the population at risk of contracting dengue and i_pop refers to infected population

You could create a dictionary to store the data for each week, and then use it to refer back to when you need to calculate the recovery rate. For example:
dengue_dict = {}
dengue_dict["Week 1"] = {"Infection Rate": infection_rate, "Recovery Rate": None}
I use none at first, because there's no recovery rate until at least two weeks have gone by. Later, you can either update weeks or just add them right away. Here's an example for week 3:
recovery_rate = dengue_dict["Week 1"]["Infection Rate"]/infection_rate
And then update the entry in the dictionary:
dengue_dict["Week 3"]["Recovery Rate"] = recovery_rate