Upsampling hourly data to 5 minute data in pandas - python

I have the following data:
MTU (CET) Day-ahead Price [EUR/MWh]
0 09.10.2017 00:00 - 09.10.2017 01:00 43.13
1 09.10.2017 01:00 - 09.10.2017 02:00 34.80
2 09.10.2017 02:00 - 09.10.2017 03:00 33.31
3 09.10.2017 03:00 - 09.10.2017 04:00 32.24
.......
22 09.10.2017 22:00 - 09.10.2017 23:00 49.06
23 09.10.2017 23:00 - 10.10.2017 00:00 38.46
From which I would like to have data for every 5 minutes.
By using:
price = pd.read_csv(price_data)
price_x = price.set_index(pd.DatetimeIndex(price['MTU (CET)'].str[:-19]))
price2 = price_x.resample('300S').pad()
I get the following data:
2017-09-10 00:00:00 43.13
2017-09-10 00:05:00 43.13
2017-09-10 00:10:00 43.13
...
2017-09-10 22:45:00 49.06
2017-09-10 22:50:00 49.06
2017-09-10 22:55:00 49.06
2017-09-10 23:00:00 38.46
However, for the minutes between 23:00 and 00:00 the price should also be 38.46. Does anyone know how to help?


You need manually add last row with next hour and with data from last row seelcted by iloc:
price_x = price.set_index(pd.DatetimeIndex(price['MTU (CET)'].str[:-19]))
price_x.loc[price_x.index[-1] + pd.Timedelta(1, unit='h')] = price_x.iloc[-1]
print (price_x.tail(3))
Day-ahead Price [EUR/MWh]
MTU (CET)
2017-09-10 22:00:00 49.06
2017-09-10 23:00:00 38.46
2017-09-11 00:00:00 38.46
price2 = price_x.resample('300S').pad()
print (price2.tail(20))
Day-ahead Price [EUR/MWh]
MTU (CET)
2017-09-10 22:25:00 49.06
2017-09-10 22:30:00 49.06
2017-09-10 22:35:00 49.06
2017-09-10 22:40:00 49.06
2017-09-10 22:45:00 49.06
2017-09-10 22:50:00 49.06
2017-09-10 22:55:00 49.06
2017-09-10 23:00:00 38.46
2017-09-10 23:05:00 38.46
2017-09-10 23:10:00 38.46
2017-09-10 23:15:00 38.46
2017-09-10 23:20:00 38.46
2017-09-10 23:25:00 38.46
2017-09-10 23:30:00 38.46
2017-09-10 23:35:00 38.46
2017-09-10 23:40:00 38.46
2017-09-10 23:45:00 38.46
2017-09-10 23:50:00 38.46
2017-09-10 23:55:00 38.46
2017-09-11 00:00:00 38.46

Related

How to change the value of a day column based on time column?

I have a df
Time Samstag Sonntag Werktag
00:15:00 95.3 87.8 94.7
00:30:00 95.5 88.3 94.1
00:45:00 96.2 89.0 94.1
01:00:00 97.4 90.1 95.0
01:15:00 98.9 91.3 96.6
01:30:00 100.3 92.4 98.4
01:45:00 101.0 92.9 99.8
02:00:00 100.4 92.5 99.8
02:15:00 98.2 91.0 98.0
02:30:00 95.1 88.7 95.1
02:45:00 91.9 86.4 91.9
03:00:00 89.5 84.7 89.5
03:15:00 88.6 84.0 88.4
03:30:00 88.6 84.0 88.3
03:45:00 88.7 84.0 88.3
04:00:00 88.3 83.5 87.7
04:15:00 86.8 82.1 86.1
04:30:00 85.1 80.6 84.3
04:45:00 84.2 80.1 83.5
05:00:00 85.3 81.6 84.7
05:15:00 89.0 85.9 88.5
05:30:00 94.1 91.6 94.0
05:45:00 99.3 97.0 99.5
06:00:00 102.8 100.4 103.4
06:15:00 103.7 100.7 104.7
06:30:00 102.6 98.8 104.0
06:45:00 100.7 96.2 102.4
07:00:00 99.2 94.3 101.0
07:15:00 99.1 94.4 100.8
07:30:00 100.8 95.7 102.1
07:45:00 104.4 97.6 105.3
08:00:00 110.1 99.2 110.7
08:15:00 117.7 99.7 118.2
08:30:00 126.1 99.6 126.7
08:45:00 133.9 99.2 134.7
09:00:00 139.7 99.2 140.9
09:15:00 142.4 99.8 144.2
09:30:00 142.9 100.9 145.4
09:45:00 142.4 102.1 145.5
10:00:00 142.1 102.8 145.8
10:15:00 142.9 102.9 147.0
10:30:00 144.5 102.5 149.0
10:45:00 146.3 101.8 151.2
11:00:00 147.6 101.0 153.0
11:15:00 147.9 100.4 154.0
11:30:00 147.5 100.0 154.3
11:45:00 146.8 99.8 154.3
12:00:00 146.4 99.8 154.2
12:15:00 146.3 100.0 154.3
12:30:00 146.5 100.5 154.5
12:45:00 146.2 101.0 154.3
13:00:00 145.1 101.6 153.6
13:15:00 142.8 102.2 152.2
13:30:00 139.3 102.4 149.9
13:45:00 134.6 102.1 147.0
14:00:00 128.8 101.0 143.3
14:15:00 122.3 98.9 139.2
14:30:00 115.5 96.3 135.2
14:45:00 109.4 93.8 132.1
15:00:00 104.6 91.9 130.6
15:15:00 101.8 91.1 131.3
15:30:00 100.5 91.2 133.5
15:45:00 100.2 91.8 136.2
16:00:00 100.4 92.5 138.5
16:15:00 100.6 93.1 139.8
16:30:00 101.0 93.4 140.3
16:45:00 101.9 93.6 140.5
17:00:00 103.4 93.7 140.9
17:15:00 105.8 93.9 142.0
17:30:00 108.7 94.3 143.7
17:45:00 111.5 95.2 145.8
18:00:00 113.7 96.8 148.2
18:15:00 115.0 99.1 150.6
18:30:00 115.7 102.2 152.5
18:45:00 116.3 105.7 153.3
19:00:00 117.3 109.5 152.4
19:15:00 119.0 113.2 149.3
19:30:00 120.6 116.3 144.4
19:45:00 121.4 117.9 138.4
20:00:00 120.4 117.3 131.8
20:15:00 117.0 114.2 125.3
20:30:00 112.1 109.4 119.3
20:45:00 106.8 104.2 114.3
21:00:00 102.2 99.8 110.7
21:15:00 99.2 97.1 108.8
21:30:00 97.4 95.9 108.1
21:45:00 96.4 95.4 108.0
22:00:00 95.6 95.0 107.7
22:15:00 94.5 94.1 106.6
22:30:00 93.3 92.8 104.9
22:45:00 92.0 91.2 103.0
23:00:00 90.7 89.5 101.0
23:15:00 89.6 87.8 99.3
23:30:00 88.6 86.4 97.8
23:45:00 88.0 85.7 96.6
00:00:00 87.7 85.9 95.6
I did:
td = pd.to_timedelta(df['Time'].astype(str))
df1 = df.assign(Time=td.mask(td == pd.Timedelta(0),td + pd.Timedelta('1 days 00:00:00')), a=1)
df2 = pd.DataFrame({'dates': pd.date_range(
'01.01.2020', '31.12.2020'), 'a': 1})
df = df2.merge(df1, how='outer').drop('a', axis=1)
df['dates'] = df['dates'].add(df.pop('Time')).dt.strftime('%d.%m.%Y %H:%M')
df['dates'] = pd.to_datetime(df['dates'], dayfirst=True)
df['day'] = df['dates'].dt.day_name()
It gave the following output:
dates Samstag Sonntag Werktag day
2020-01-01 00:15:00 95.3 87.8 94.7 Wednesday
2020-01-01 00:30:00 95.5 88.3 94.1 Wednesday
2020-01-01 00:45:00 96.2 89.0 94.1 Wednesday
2020-01-01 01:00:00 97.4 90.1 95.0 Wednesday
2020-01-01 01:15:00 98.9 91.3 96.6 Wednesday
2020-01-01 01:30:00 100.3 92.4 98.4 Wednesday
2020-01-01 01:45:00 101.0 92.9 99.8 Wednesday
2020-01-01 02:00:00 100.4 92.5 99.8 Wednesday
2020-01-01 02:15:00 98.2 91.0 98.0 Wednesday
2020-01-01 02:30:00 95.1 88.7 95.1 Wednesday
2020-01-01 02:45:00 91.9 86.4 91.9 Wednesday
2020-01-01 03:00:00 89.5 84.7 89.5 Wednesday
2020-01-01 03:15:00 88.6 84.0 88.4 Wednesday
2020-01-01 03:30:00 88.6 84.0 88.3 Wednesday
2020-01-01 03:45:00 88.7 84.0 88.3 Wednesday
2020-01-01 04:00:00 88.3 83.5 87.7 Wednesday
2020-01-01 04:15:00 86.8 82.1 86.1 Wednesday
2020-01-01 04:30:00 85.1 80.6 84.3 Wednesday
2020-01-01 04:45:00 84.2 80.1 83.5 Wednesday
2020-01-01 05:00:00 85.3 81.6 84.7 Wednesday
2020-01-01 05:15:00 89.0 85.9 88.5 Wednesday
2020-01-01 05:30:00 94.1 91.6 94.0 Wednesday
2020-01-01 05:45:00 99.3 97.0 99.5 Wednesday
2020-01-01 06:00:00 102.8 100.4 103.4 Wednesday
2020-01-01 06:15:00 103.7 100.7 104.7 Wednesday
2020-01-01 06:30:00 102.6 98.8 104.0 Wednesday
2020-01-01 06:45:00 100.7 96.2 102.4 Wednesday
2020-01-01 07:00:00 99.2 94.3 101.0 Wednesday
2020-01-01 07:15:00 99.1 94.4 100.8 Wednesday
2020-01-01 07:30:00 100.8 95.7 102.1 Wednesday
2020-01-01 07:45:00 104.4 97.6 105.3 Wednesday
2020-01-01 08:00:00 110.1 99.2 110.7 Wednesday
2020-01-01 08:15:00 117.7 99.7 118.2 Wednesday
2020-01-01 08:30:00 126.1 99.6 126.7 Wednesday
2020-01-01 08:45:00 133.9 99.2 134.7 Wednesday
2020-01-01 09:00:00 139.7 99.2 140.9 Wednesday
2020-01-01 09:15:00 142.4 99.8 144.2 Wednesday
2020-01-01 09:30:00 142.9 100.9 145.4 Wednesday
2020-01-01 09:45:00 142.4 102.1 145.5 Wednesday
2020-01-01 10:00:00 142.1 102.8 145.8 Wednesday
2020-01-01 10:15:00 142.9 102.9 147.0 Wednesday
2020-01-01 10:30:00 144.5 102.5 149.0 Wednesday
2020-01-01 10:45:00 146.3 101.8 151.2 Wednesday
2020-01-01 11:00:00 147.6 101.0 153.0 Wednesday
2020-01-01 11:15:00 147.9 100.4 154.0 Wednesday
2020-01-01 11:30:00 147.5 100.0 154.3 Wednesday
2020-01-01 11:45:00 146.8 99.8 154.3 Wednesday
2020-01-01 12:00:00 146.4 99.8 154.2 Wednesday
2020-01-01 12:15:00 146.3 100.0 154.3 Wednesday
2020-01-01 12:30:00 146.5 100.5 154.5 Wednesday
2020-01-01 12:45:00 146.2 101.0 154.3 Wednesday
2020-01-01 13:00:00 145.1 101.6 153.6 Wednesday
2020-01-01 13:15:00 142.8 102.2 152.2 Wednesday
2020-01-01 13:30:00 139.3 102.4 149.9 Wednesday
2020-01-01 13:45:00 134.6 102.1 147.0 Wednesday
2020-01-01 14:00:00 128.8 101.0 143.3 Wednesday
2020-01-01 14:15:00 122.3 98.9 139.2 Wednesday
2020-01-01 14:30:00 115.5 96.3 135.2 Wednesday
2020-01-01 14:45:00 109.4 93.8 132.1 Wednesday
2020-01-01 15:00:00 104.6 91.9 130.6 Wednesday
2020-01-01 15:15:00 101.8 91.1 131.3 Wednesday
2020-01-01 15:30:00 100.5 91.2 133.5 Wednesday
2020-01-01 15:45:00 100.2 91.8 136.2 Wednesday
2020-01-01 16:00:00 100.4 92.5 138.5 Wednesday
2020-01-01 16:15:00 100.6 93.1 139.8 Wednesday
2020-01-01 16:30:00 101.0 93.4 140.3 Wednesday
2020-01-01 16:45:00 101.9 93.6 140.5 Wednesday
2020-01-01 17:00:00 103.4 93.7 140.9 Wednesday
2020-01-01 17:15:00 105.8 93.9 142.0 Wednesday
2020-01-01 17:30:00 108.7 94.3 143.7 Wednesday
2020-01-01 17:45:00 111.5 95.2 145.8 Wednesday
2020-01-01 18:00:00 113.7 96.8 148.2 Wednesday
2020-01-01 18:15:00 115.0 99.1 150.6 Wednesday
2020-01-01 18:30:00 115.7 102.2 152.5 Wednesday
2020-01-01 18:45:00 116.3 105.7 153.3 Wednesday
2020-01-01 19:00:00 117.3 109.5 152.4 Wednesday
2020-01-01 19:15:00 119.0 113.2 149.3 Wednesday
2020-01-01 19:30:00 120.6 116.3 144.4 Wednesday
2020-01-01 19:45:00 121.4 117.9 138.4 Wednesday
2020-01-01 20:00:00 120.4 117.3 131.8 Wednesday
2020-01-01 20:15:00 117.0 114.2 125.3 Wednesday
2020-01-01 20:30:00 112.1 109.4 119.3 Wednesday
2020-01-01 20:45:00 106.8 104.2 114.3 Wednesday
2020-01-01 21:00:00 102.2 99.8 110.7 Wednesday
2020-01-01 21:15:00 99.2 97.1 108.8 Wednesday
2020-01-01 21:30:00 97.4 95.9 108.1 Wednesday
2020-01-01 21:45:00 96.4 95.4 108.0 Wednesday
2020-01-01 22:00:00 95.6 95.0 107.7 Wednesday
2020-01-01 22:15:00 94.5 94.1 106.6 Wednesday
2020-01-01 22:30:00 93.3 92.8 104.9 Wednesday
2020-01-01 22:45:00 92.0 91.2 103.0 Wednesday
2020-01-01 23:00:00 90.7 89.5 101.0 Wednesday
2020-01-01 23:15:00 89.6 87.8 99.3 Wednesday
2020-01-01 23:30:00 88.6 86.4 97.8 Wednesday
2020-01-01 23:45:00 88.0 85.7 96.6 Wednesday
2020-01-02 00:00:00 87.7 85.9 95.6 Thursday
2020-01-02 00:15:00 95.3 87.8 94.7 Thursday
2020-01-02 00:30:00 95.5 88.3 94.1 Thursday
2020-01-02 00:45:00 96.2 89.0 94.1 Thursday
2020-01-02 01:00:00 97.4 90.1 95.0 Thursday
2020-01-02 01:15:00 98.9 91.3 96.6 Thursday
2020-01-02 01:30:00 100.3 92.4 98.4 Thursday
2020-01-02 01:45:00 101.0 92.9 99.8 Thursday
2020-01-02 02:00:00 100.4 92.5 99.8 Thursday
2020-01-02 02:15:00 98.2 91.0 98.0 Thursday
2020-01-02 02:30:00 95.1 88.7 95.1 Thursday
2020-01-02 02:45:00 91.9 86.4 91.9 Thursday
2020-01-02 03:00:00 89.5 84.7 89.5 Thursday
2020-01-02 03:15:00 88.6 84.0 88.4 Thursday
2020-01-02 03:30:00 88.6 84.0 88.3 Thursday
2020-01-02 03:45:00 88.7 84.0 88.3 Thursday
2020-01-02 04:00:00 88.3 83.5 87.7 Thursday
2020-01-02 04:15:00 86.8 82.1 86.1 Thursday
2020-01-02 04:30:00 85.1 80.6 84.3 Thursday
2020-01-02 04:45:00 84.2 80.1 83.5 Thursday
2020-01-02 05:00:00 85.3 81.6 84.7 Thursday
2020-01-02 05:15:00 89.0 85.9 88.5 Thursday
2020-01-02 05:30:00 94.1 91.6 94.0 Thursday
2020-01-02 05:45:00 99.3 97.0 99.5 Thursday
2020-01-02 06:00:00 102.8 100.4 103.4 Thursday
2020-01-02 06:15:00 103.7 100.7 104.7 Thursday
2020-01-02 06:30:00 102.6 98.8 104.0 Thursday
2020-01-02 06:45:00 100.7 96.2 102.4 Thursday
2020-01-02 07:00:00 99.2 94.3 101.0 Thursday
2020-01-02 07:15:00 99.1 94.4 100.8 Thursday
2020-01-02 07:30:00 100.8 95.7 102.1 Thursday
2020-01-02 07:45:00 104.4 97.6 105.3 Thursday
2020-01-02 08:00:00 110.1 99.2 110.7 Thursday
2020-01-02 08:15:00 117.7 99.7 118.2 Thursday
2020-01-02 08:30:00 126.1 99.6 126.7 Thursday
2020-01-02 08:45:00 133.9 99.2 134.7 Thursday
2020-01-02 09:00:00 139.7 99.2 140.9 Thursday
2020-01-02 09:15:00 142.4 99.8 144.2 Thursday
2020-01-02 09:30:00 142.9 100.9 145.4 Thursday
2020-01-02 09:45:00 142.4 102.1 145.5 Thursday
2020-01-02 10:00:00 142.1 102.8 145.8 Thursday
2020-01-02 10:15:00 142.9 102.9 147.0 Thursday
2020-01-02 10:30:00 144.5 102.5 149.0 Thursday
2020-01-02 10:45:00 146.3 101.8 151.2 Thursday
2020-01-02 11:00:00 147.6 101.0 153.0 Thursday
2020-01-02 11:15:00 147.9 100.4 154.0 Thursday
2020-01-02 11:30:00 147.5 100.0 154.3 Thursday
2020-01-02 11:45:00 146.8 99.8 154.3 Thursday
2020-01-02 12:00:00 146.4 99.8 154.2 Thursday
2020-01-02 12:15:00 146.3 100.0 154.3 Thursday
2020-01-02 12:30:00 146.5 100.5 154.5 Thursday
2020-01-02 12:45:00 146.2 101.0 154.3 Thursday
2020-01-02 13:00:00 145.1 101.6 153.6 Thursday
2020-01-02 13:15:00 142.8 102.2 152.2 Thursday
2020-01-02 13:30:00 139.3 102.4 149.9 Thursday
2020-01-02 13:45:00 134.6 102.1 147.0 Thursday
2020-01-02 14:00:00 128.8 101.0 143.3 Thursday
2020-01-02 14:15:00 122.3 98.9 139.2 Thursday
2020-01-02 14:30:00 115.5 96.3 135.2 Thursday
2020-01-02 14:45:00 109.4 93.8 132.1 Thursday
2020-01-02 15:00:00 104.6 91.9 130.6 Thursday
2020-01-02 15:15:00 101.8 91.1 131.3 Thursday
2020-01-02 15:30:00 100.5 91.2 133.5 Thursday
2020-01-02 15:45:00 100.2 91.8 136.2 Thursday
2020-01-02 16:00:00 100.4 92.5 138.5 Thursday
2020-01-02 16:15:00 100.6 93.1 139.8 Thursday
2020-01-02 16:30:00 101.0 93.4 140.3 Thursday
2020-01-02 16:45:00 101.9 93.6 140.5 Thursday
2020-01-02 17:00:00 103.4 93.7 140.9 Thursday
2020-01-02 17:15:00 105.8 93.9 142.0 Thursday
2020-01-02 17:30:00 108.7 94.3 143.7 Thursday
2020-01-02 17:45:00 111.5 95.2 145.8 Thursday
2020-01-02 18:00:00 113.7 96.8 148.2 Thursday
2020-01-02 18:15:00 115.0 99.1 150.6 Thursday
2020-01-02 18:30:00 115.7 102.2 152.5 Thursday
2020-01-02 18:45:00 116.3 105.7 153.3 Thursday
2020-01-02 19:00:00 117.3 109.5 152.4 Thursday
2020-01-02 19:15:00 119.0 113.2 149.3 Thursday
2020-01-02 19:30:00 120.6 116.3 144.4 Thursday
2020-01-02 19:45:00 121.4 117.9 138.4 Thursday
2020-01-02 20:00:00 120.4 117.3 131.8 Thursday
2020-01-02 20:15:00 117.0 114.2 125.3 Thursday
2020-01-02 20:30:00 112.1 109.4 119.3 Thursday
2020-01-02 20:45:00 106.8 104.2 114.3 Thursday
2020-01-02 21:00:00 102.2 99.8 110.7 Thursday
2020-01-02 21:15:00 99.2 97.1 108.8 Thursday
2020-01-02 21:30:00 97.4 95.9 108.1 Thursday
2020-01-02 21:45:00 96.4 95.4 108.0 Thursday
2020-01-02 22:00:00 95.6 95.0 107.7 Thursday
2020-01-02 22:15:00 94.5 94.1 106.6 Thursday
2020-01-02 22:30:00 93.3 92.8 104.9 Thursday
2020-01-02 22:45:00 92.0 91.2 103.0 Thursday
2020-01-02 23:00:00 90.7 89.5 101.0 Thursday
2020-01-02 23:15:00 89.6 87.8 99.3 Thursday
2020-01-02 23:30:00 88.6 86.4 97.8 Thursday
2020-01-02 23:45:00 88.0 85.7 96.6 Thursday
2020-01-03 00:00:00 87.7 85.9 95.6 Friday
2020-01-03 00:15:00 95.3 87.8 94.7 Friday
2020-01-03 00:30:00 95.5 88.3 94.1 Friday
2020-01-03 00:45:00 96.2 89.0 94.1 Friday
What I would like to do is to change the value of day at 2020-01-02 00:00:00 from Thursday to Wednesday, and similarly the value of day at 2020-01-03 00:00:00 from Friday to Thursday and so on.
In other words: The value of day for next day at 00:00:00 should be similar to the value of the previous day and from 00:15:00, a new day should begin.
Expected output
dates Samstag Sonntag Werktag day
2020-01-01 00:15:00 95.3 87.8 94.7 Wednesday
2020-01-01 00:30:00 95.5 88.3 94.1 Wednesday
2020-01-01 00:45:00 96.2 89.0 94.1 Wednesday
2020-01-01 01:00:00 97.4 90.1 95.0 Wednesday
2020-01-01 01:15:00 98.9 91.3 96.6 Wednesday
2020-01-01 01:30:00 100.3 92.4 98.4 Wednesday
2020-01-01 01:45:00 101.0 92.9 99.8 Wednesday
2020-01-01 02:00:00 100.4 92.5 99.8 Wednesday
2020-01-01 02:15:00 98.2 91.0 98.0 Wednesday
2020-01-01 02:30:00 95.1 88.7 95.1 Wednesday
2020-01-01 02:45:00 91.9 86.4 91.9 Wednesday
2020-01-01 03:00:00 89.5 84.7 89.5 Wednesday
2020-01-01 03:15:00 88.6 84.0 88.4 Wednesday
2020-01-01 03:30:00 88.6 84.0 88.3 Wednesday
2020-01-01 03:45:00 88.7 84.0 88.3 Wednesday
2020-01-01 04:00:00 88.3 83.5 87.7 Wednesday
2020-01-01 04:15:00 86.8 82.1 86.1 Wednesday
2020-01-01 04:30:00 85.1 80.6 84.3 Wednesday
2020-01-01 04:45:00 84.2 80.1 83.5 Wednesday
2020-01-01 05:00:00 85.3 81.6 84.7 Wednesday
2020-01-01 05:15:00 89.0 85.9 88.5 Wednesday
2020-01-01 05:30:00 94.1 91.6 94.0 Wednesday
2020-01-01 05:45:00 99.3 97.0 99.5 Wednesday
2020-01-01 06:00:00 102.8 100.4 103.4 Wednesday
2020-01-01 06:15:00 103.7 100.7 104.7 Wednesday
2020-01-01 06:30:00 102.6 98.8 104.0 Wednesday
2020-01-01 06:45:00 100.7 96.2 102.4 Wednesday
2020-01-01 07:00:00 99.2 94.3 101.0 Wednesday
2020-01-01 07:15:00 99.1 94.4 100.8 Wednesday
2020-01-01 07:30:00 100.8 95.7 102.1 Wednesday
2020-01-01 07:45:00 104.4 97.6 105.3 Wednesday
2020-01-01 08:00:00 110.1 99.2 110.7 Wednesday
2020-01-01 08:15:00 117.7 99.7 118.2 Wednesday
2020-01-01 08:30:00 126.1 99.6 126.7 Wednesday
2020-01-01 08:45:00 133.9 99.2 134.7 Wednesday
2020-01-01 09:00:00 139.7 99.2 140.9 Wednesday
2020-01-01 09:15:00 142.4 99.8 144.2 Wednesday
2020-01-01 09:30:00 142.9 100.9 145.4 Wednesday
2020-01-01 09:45:00 142.4 102.1 145.5 Wednesday
2020-01-01 10:00:00 142.1 102.8 145.8 Wednesday
2020-01-01 10:15:00 142.9 102.9 147.0 Wednesday
2020-01-01 10:30:00 144.5 102.5 149.0 Wednesday
2020-01-01 10:45:00 146.3 101.8 151.2 Wednesday
2020-01-01 11:00:00 147.6 101.0 153.0 Wednesday
2020-01-01 11:15:00 147.9 100.4 154.0 Wednesday
2020-01-01 11:30:00 147.5 100.0 154.3 Wednesday
2020-01-01 11:45:00 146.8 99.8 154.3 Wednesday
2020-01-01 12:00:00 146.4 99.8 154.2 Wednesday
2020-01-01 12:15:00 146.3 100.0 154.3 Wednesday
2020-01-01 12:30:00 146.5 100.5 154.5 Wednesday
2020-01-01 12:45:00 146.2 101.0 154.3 Wednesday
2020-01-01 13:00:00 145.1 101.6 153.6 Wednesday
2020-01-01 13:15:00 142.8 102.2 152.2 Wednesday
2020-01-01 13:30:00 139.3 102.4 149.9 Wednesday
2020-01-01 13:45:00 134.6 102.1 147.0 Wednesday
2020-01-01 14:00:00 128.8 101.0 143.3 Wednesday
2020-01-01 14:15:00 122.3 98.9 139.2 Wednesday
2020-01-01 14:30:00 115.5 96.3 135.2 Wednesday
2020-01-01 14:45:00 109.4 93.8 132.1 Wednesday
2020-01-01 15:00:00 104.6 91.9 130.6 Wednesday
2020-01-01 15:15:00 101.8 91.1 131.3 Wednesday
2020-01-01 15:30:00 100.5 91.2 133.5 Wednesday
2020-01-01 15:45:00 100.2 91.8 136.2 Wednesday
2020-01-01 16:00:00 100.4 92.5 138.5 Wednesday
2020-01-01 16:15:00 100.6 93.1 139.8 Wednesday
2020-01-01 16:30:00 101.0 93.4 140.3 Wednesday
2020-01-01 16:45:00 101.9 93.6 140.5 Wednesday
2020-01-01 17:00:00 103.4 93.7 140.9 Wednesday
2020-01-01 17:15:00 105.8 93.9 142.0 Wednesday
2020-01-01 17:30:00 108.7 94.3 143.7 Wednesday
2020-01-01 17:45:00 111.5 95.2 145.8 Wednesday
2020-01-01 18:00:00 113.7 96.8 148.2 Wednesday
2020-01-01 18:15:00 115.0 99.1 150.6 Wednesday
2020-01-01 18:30:00 115.7 102.2 152.5 Wednesday
2020-01-01 18:45:00 116.3 105.7 153.3 Wednesday
2020-01-01 19:00:00 117.3 109.5 152.4 Wednesday
2020-01-01 19:15:00 119.0 113.2 149.3 Wednesday
2020-01-01 19:30:00 120.6 116.3 144.4 Wednesday
2020-01-01 19:45:00 121.4 117.9 138.4 Wednesday
2020-01-01 20:00:00 120.4 117.3 131.8 Wednesday
2020-01-01 20:15:00 117.0 114.2 125.3 Wednesday
2020-01-01 20:30:00 112.1 109.4 119.3 Wednesday
2020-01-01 20:45:00 106.8 104.2 114.3 Wednesday
2020-01-01 21:00:00 102.2 99.8 110.7 Wednesday
2020-01-01 21:15:00 99.2 97.1 108.8 Wednesday
2020-01-01 21:30:00 97.4 95.9 108.1 Wednesday
2020-01-01 21:45:00 96.4 95.4 108.0 Wednesday
2020-01-01 22:00:00 95.6 95.0 107.7 Wednesday
2020-01-01 22:15:00 94.5 94.1 106.6 Wednesday
2020-01-01 22:30:00 93.3 92.8 104.9 Wednesday
2020-01-01 22:45:00 92.0 91.2 103.0 Wednesday
2020-01-01 23:00:00 90.7 89.5 101.0 Wednesday
2020-01-01 23:15:00 89.6 87.8 99.3 Wednesday
2020-01-01 23:30:00 88.6 86.4 97.8 Wednesday
2020-01-01 23:45:00 88.0 85.7 96.6 Wednesday
2020-01-02 00:00:00 87.7 85.9 95.6 Wednesday
2020-01-02 00:15:00 95.3 87.8 94.7 Thursday
2020-01-02 00:30:00 95.5 88.3 94.1 Thursday
2020-01-02 00:45:00 96.2 89.0 94.1 Thursday
2020-01-02 01:00:00 97.4 90.1 95.0 Thursday
2020-01-02 01:15:00 98.9 91.3 96.6 Thursday
2020-01-02 01:30:00 100.3 92.4 98.4 Thursday
2020-01-02 01:45:00 101.0 92.9 99.8 Thursday
2020-01-02 02:00:00 100.4 92.5 99.8 Thursday
2020-01-02 02:15:00 98.2 91.0 98.0 Thursday
2020-01-02 02:30:00 95.1 88.7 95.1 Thursday
2020-01-02 02:45:00 91.9 86.4 91.9 Thursday
2020-01-02 03:00:00 89.5 84.7 89.5 Thursday
2020-01-02 03:15:00 88.6 84.0 88.4 Thursday
2020-01-02 03:30:00 88.6 84.0 88.3 Thursday
2020-01-02 03:45:00 88.7 84.0 88.3 Thursday
2020-01-02 04:00:00 88.3 83.5 87.7 Thursday
2020-01-02 04:15:00 86.8 82.1 86.1 Thursday
2020-01-02 04:30:00 85.1 80.6 84.3 Thursday
2020-01-02 04:45:00 84.2 80.1 83.5 Thursday
2020-01-02 05:00:00 85.3 81.6 84.7 Thursday
2020-01-02 05:15:00 89.0 85.9 88.5 Thursday
2020-01-02 05:30:00 94.1 91.6 94.0 Thursday
2020-01-02 05:45:00 99.3 97.0 99.5 Thursday
2020-01-02 06:00:00 102.8 100.4 103.4 Thursday
2020-01-02 06:15:00 103.7 100.7 104.7 Thursday
2020-01-02 06:30:00 102.6 98.8 104.0 Thursday
2020-01-02 06:45:00 100.7 96.2 102.4 Thursday
2020-01-02 07:00:00 99.2 94.3 101.0 Thursday
2020-01-02 07:15:00 99.1 94.4 100.8 Thursday
2020-01-02 07:30:00 100.8 95.7 102.1 Thursday
2020-01-02 07:45:00 104.4 97.6 105.3 Thursday
2020-01-02 08:00:00 110.1 99.2 110.7 Thursday
2020-01-02 08:15:00 117.7 99.7 118.2 Thursday
2020-01-02 08:30:00 126.1 99.6 126.7 Thursday
2020-01-02 08:45:00 133.9 99.2 134.7 Thursday
2020-01-02 09:00:00 139.7 99.2 140.9 Thursday
2020-01-02 09:15:00 142.4 99.8 144.2 Thursday
2020-01-02 09:30:00 142.9 100.9 145.4 Thursday
2020-01-02 09:45:00 142.4 102.1 145.5 Thursday
2020-01-02 10:00:00 142.1 102.8 145.8 Thursday
2020-01-02 10:15:00 142.9 102.9 147.0 Thursday
2020-01-02 10:30:00 144.5 102.5 149.0 Thursday
2020-01-02 10:45:00 146.3 101.8 151.2 Thursday
2020-01-02 11:00:00 147.6 101.0 153.0 Thursday
2020-01-02 11:15:00 147.9 100.4 154.0 Thursday
2020-01-02 11:30:00 147.5 100.0 154.3 Thursday
2020-01-02 11:45:00 146.8 99.8 154.3 Thursday
2020-01-02 12:00:00 146.4 99.8 154.2 Thursday
2020-01-02 12:15:00 146.3 100.0 154.3 Thursday
2020-01-02 12:30:00 146.5 100.5 154.5 Thursday
2020-01-02 12:45:00 146.2 101.0 154.3 Thursday
2020-01-02 13:00:00 145.1 101.6 153.6 Thursday
2020-01-02 13:15:00 142.8 102.2 152.2 Thursday
2020-01-02 13:30:00 139.3 102.4 149.9 Thursday
2020-01-02 13:45:00 134.6 102.1 147.0 Thursday
2020-01-02 14:00:00 128.8 101.0 143.3 Thursday
2020-01-02 14:15:00 122.3 98.9 139.2 Thursday
2020-01-02 14:30:00 115.5 96.3 135.2 Thursday
2020-01-02 14:45:00 109.4 93.8 132.1 Thursday
2020-01-02 15:00:00 104.6 91.9 130.6 Thursday
2020-01-02 15:15:00 101.8 91.1 131.3 Thursday
2020-01-02 15:30:00 100.5 91.2 133.5 Thursday
2020-01-02 15:45:00 100.2 91.8 136.2 Thursday
2020-01-02 16:00:00 100.4 92.5 138.5 Thursday
2020-01-02 16:15:00 100.6 93.1 139.8 Thursday
2020-01-02 16:30:00 101.0 93.4 140.3 Thursday
2020-01-02 16:45:00 101.9 93.6 140.5 Thursday
2020-01-02 17:00:00 103.4 93.7 140.9 Thursday
2020-01-02 17:15:00 105.8 93.9 142.0 Thursday
2020-01-02 17:30:00 108.7 94.3 143.7 Thursday
2020-01-02 17:45:00 111.5 95.2 145.8 Thursday
2020-01-02 18:00:00 113.7 96.8 148.2 Thursday
2020-01-02 18:15:00 115.0 99.1 150.6 Thursday
2020-01-02 18:30:00 115.7 102.2 152.5 Thursday
2020-01-02 18:45:00 116.3 105.7 153.3 Thursday
2020-01-02 19:00:00 117.3 109.5 152.4 Thursday
2020-01-02 19:15:00 119.0 113.2 149.3 Thursday
2020-01-02 19:30:00 120.6 116.3 144.4 Thursday
2020-01-02 19:45:00 121.4 117.9 138.4 Thursday
2020-01-02 20:00:00 120.4 117.3 131.8 Thursday
2020-01-02 20:15:00 117.0 114.2 125.3 Thursday
2020-01-02 20:30:00 112.1 109.4 119.3 Thursday
2020-01-02 20:45:00 106.8 104.2 114.3 Thursday
2020-01-02 21:00:00 102.2 99.8 110.7 Thursday
2020-01-02 21:15:00 99.2 97.1 108.8 Thursday
2020-01-02 21:30:00 97.4 95.9 108.1 Thursday
2020-01-02 21:45:00 96.4 95.4 108.0 Thursday
2020-01-02 22:00:00 95.6 95.0 107.7 Thursday
2020-01-02 22:15:00 94.5 94.1 106.6 Thursday
2020-01-02 22:30:00 93.3 92.8 104.9 Thursday
2020-01-02 22:45:00 92.0 91.2 103.0 Thursday
2020-01-02 23:00:00 90.7 89.5 101.0 Thursday
2020-01-02 23:15:00 89.6 87.8 99.3 Thursday
2020-01-02 23:30:00 88.6 86.4 97.8 Thursday
2020-01-02 23:45:00 88.0 85.7 96.6 Thursday
2020-01-03 00:00:00 87.7 85.9 95.6 Thursday
2020-01-03 00:15:00 95.3 87.8 94.7 Friday
2020-01-03 00:30:00 95.5 88.3 94.1 Friday
2020-01-03 00:45:00 96.2 89.0 94.1 Friday
How can this be done??
Edit 1
import pandas as pd
df = pd.DataFrame({ 'dates': ['2020-01-01 22:15:00',
'2020-01-01 22:35:00',
'2020-01-01 22:45:00',
'2020-01-01 23:00:00',
'2020-01-01 23:15:00',
'2020-01-01 23:30:00',
'2020-01-01 23:45:00',
'2020-01-02 00:00:00',
'2020-01-02 22:15:00',
'2020-01-02 22:35:00',
'2020-01-02 22:45:00',
'2020-01-02 23:00:00',
'2020-01-02 23:15:00',
'2020-01-02 23:30:00',
'2020-01-02 23:45:00',
'2020-01-03 00:00:00'],
'expected_output':['Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Thursday',
'Thursday',
'Thursday',
'Thursday',
'Thursday','Thursday','Thursday','Thursday']})

Just check the minutes of Timestamp using apply.
# df = pd.DataFrame({'dates': ['2020-01-01 22:15:00', .....]}, )
# convert str date into Timestamp
df['dates'] = pd.to_datetime(df['dates'])
def calculate_day(x):
# get previous day
if x.hour == 0 and x.minute < 15:
return (x - pd.DateOffset(days=1)).day_name()
return x.day_name()
df['day'] = df['dates'].apply(calculate_day)
print(df)
# dates day
#0 2020-01-01 22:15:00 Wednesday
#...
JFYI: weekday_name deprecated. Use day_name().
Hope this helps.

select dataframe rows (datetimeindex) by a list of datetime.date

the df looks like this:
DateTime
2017-07-10 03:00:00 288.0
2017-07-10 04:00:00 306.0
2017-08-10 05:00:00 393.0
2017-08-10 06:00:00 522.0
2017-09-10 07:00:00 487.0
2017-09-10 08:00:00 523.0
2017-10-10 09:00:00 585.0
Question how to select row that in a list of dates:
['2017-07-10', '2017-09-10']
to have:
DateTime
2017-07-10 03:00:00 288.0
2017-07-10 04:00:00 306.0
2017-09-10 07:00:00 487.0
2017-09-10 08:00:00 523.0
Thanks

Given that the dates in your list contain up to the daily information, you could start by flooring (Series.dt.floor) the DatetimeIndex up to the daily level and indexing with the list of datetime objects using isin:
t = [pd.to_datetime('2017-07-10'), pd.to_datetime('2017-09-10')]
df.index= pd.to_datetime(df.index)
df[df.index.floor('d').isin(t)]
Output
DateTime
2017-07-10 03:00:00 288.0
2017-07-10 04:00:00 306.0
2017-09-10 07:00:00 487.0
2017-09-10 08:00:00 523.0

Assuming the Datetime is index, try with the below:
to_search=['2017-07-10', '2017-09-10']
df[df.index.to_series().dt.date.astype(str).isin(to_search)]
1
DateTime
2017-07-10 03:00:00 288.0
2017-07-10 04:00:00 306.0
2017-09-10 07:00:00 487.0
2017-09-10 08:00:00 523.0

Pandas DataFrame Calculate time difference between 2 columns on specific time range

I want to calculate time difference between two columns on specific time range.
I try df.between_time but it only works on index.
Ex. Time range: between 18.00 - 8.00
Data :
start stop
0 2018-07-16 16:00:00 2018-07-16 20:00:00
1 2018-07-11 08:03:00 2018-07-11 12:03:00
2 2018-07-13 17:54:00 2018-07-13 21:54:00
3 2018-07-14 13:09:00 2018-07-14 17:09:00
4 2018-07-20 00:21:00 2018-07-20 04:21:00
5 2018-07-20 17:00:00 2018-07-21 09:00:00
Expect Result :
start stop time_diff
0 2018-07-16 16:00:00 2018-07-16 20:00:00 02:00:00
1 2018-07-11 08:03:00 2018-07-11 12:03:00 0
2 2018-07-13 17:54:00 2018-07-13 21:54:00 03:54:00
3 2018-07-14 13:09:00 2018-07-14 17:09:00 0
4 2018-07-20 00:21:00 2018-07-20 04:21:00 04:00:00
5 2018-07-20 17:00:00 2018-07-21 09:00:00 14:00:00
Note: If time_diff > 1 days, I already deal with that case.
Question: Should I build a function to do this or there are pandas build-in function to do this? Any help or guide would be appreciated.

I think this can be a solution
tmp = pd.DataFrame({'time1': pd.to_datetime(['2018-07-16 16:00:00', '2018-07-11 08:03:00',
'2018-07-13 17:54:00', '2018-07-14 13:09:00',
'2018-07-20 00:21:00', '2018-07-20 17:00:00']),
'time2': pd.to_datetime(['2018-07-16 20:00:00', '2018-07-11 12:03:00',
'2018-07-13 21:54:00', '2018-07-14 17:09:00',
'2018-07-20 04:21:00', '2018-07-21 09:00:00'])})
time1_date = tmp.time1.dt.date.astype(str)
tmp['rule18'], tmp['rule08'] = pd.to_datetime(time1_date + ' 18:00:00'), pd.to_datetime(time1_date + ' 08:00:00')
# if stop exceeds 18:00:00, compute time difference from this hour
tmp['time_diff_rule1'] = np.where(tmp.time2 > tmp.rule18, (tmp.time2 - tmp.rule18), (tmp.time2 - tmp.time1))
# rearrange the dataframe with your second rule
tmp['time_diff_rule2'] = np.where((tmp.time2 < tmp.rule18) & (tmp.time1 > tmp.rule08), 0, tmp['time_diff_rule1'])
time_diff_rule1 time_diff_rule2
0 02:00:00 02:00:00
1 04:00:00 00:00:00
2 03:54:00 03:54:00
3 04:00:00 00:00:00
4 04:00:00 04:00:00
5 15:00:00 15:00:00

how to split dataframe vertically, having N columns in each resulting DF

I have the following dataframe:
Date 2017-12-05 2017-12-06 2017-12-15 2017-12-19 2017-12-20 2017-12-21 ....
time
00:00:00 19.94 21.19 21.88 20.76 21.26 21.26
00:15:00 20.29 21.07 21.71 21.79 21.95 21.52
00:30:00 21.03 21.25 21.80 22.15 22.26 21.62
00:45:00 22.20 21.56 22.77 22.20 22.33 21.91
01:00:00 23.25 22.15 23.71 22.31 22.69 21.99
01:15:00 23.78 23.33 24.53 22.29 22.82 22.58
Columns are dates, and I want to split the dataframe by every 3 date columns. How can I do this?
Illustration of the final result with one of the resultant dataframes following the split:
Date 2017-12-05 2017-12-06 2017-12-15 2
time
00:00:00 19.94 21.19 21.88
00:15:00 20.29 21.07 21.71
00:30:00 21.03 21.25 21.80
00:45:00 22.20 21.56 22.77
01:00:00 23.25 22.15 23.71
01:15:00 23.78 23.33 24.53

you can use np.split(..., axis=1):
In [157]: N = 3
In [158]: dfs = np.split(df, np.arange(N, len(df.columns), N), axis=1)
In [159]: dfs[0]
Out[159]:
2017-12-05 2017-12-06 2017-12-15
00:00:00 19.94 21.19 21.88
00:15:00 20.29 21.07 21.71
00:30:00 21.03 21.25 21.80
00:45:00 22.20 21.56 22.77
01:00:00 23.25 22.15 23.71
01:15:00 23.78 23.33 24.53
In [160]: dfs[1]
Out[160]:
2017-12-19 2017-12-20 2017-12-21
00:00:00 20.76 21.26 21.26
00:15:00 21.79 21.95 21.52
00:30:00 22.15 22.26 21.62
00:45:00 22.20 22.33 21.91
01:00:00 22.31 22.69 21.99
01:15:00 22.29 22.82 22.58

Something like this
ddf=[]
for i in range(len(df.columns)/3):
ddf.append(df[df.columns[3*i:3*i+3]])
to generate a list df of dataframes, each with 3 columns.
Add the rest (with less than 3 columns):
ddf.append(df[df.columns[3*(i+1):]])

How can I slice a dataframe by timestamp, when timestamp isn't classified as index?

How can I split my pandas dataframe by using the timestamp on it?
I got the following prices when I call df30m:
Timestamp Open High Low Close Volume
0 2016-05-01 19:30:00 449.80 450.13 449.80 449.90 74.1760
1 2016-05-01 20:00:00 449.90 450.27 449.90 450.07 63.5840
2 2016-05-01 20:30:00 450.12 451.00 450.02 450.51 64.1080
3 2016-05-01 21:00:00 450.51 452.05 450.50 451.22 75.7390
4 2016-05-01 21:30:00 451.21 451.64 450.81 450.87 71.1190
5 2016-05-01 22:00:00 450.87 452.05 450.87 451.07 73.8430
6 2016-05-01 22:30:00 451.09 451.70 450.91 450.91 68.1490
7 2016-05-01 23:00:00 450.91 450.98 449.97 450.61 84.5430
8 2016-05-01 23:30:00 450.61 451.50 450.55 451.45 111.2370
9 2016-05-02 00:00:00 451.47 452.31 450.69 451.19 190.0750
10 2016-05-02 00:30:00 451.20 451.68 450.45 450.82 186.0930
11 2016-05-02 01:00:00 450.83 451.64 450.65 450.73 112.4630
12 2016-05-02 01:30:00 450.73 451.10 450.31 450.56 137.7530
13 2016-05-02 02:00:00 450.56 452.01 449.98 450.27 151.6140
14 2016-05-02 02:30:00 450.27 451.30 450.23 451.11 99.5490
15 2016-05-02 03:00:00 451.29 451.29 450.17 450.33 178.9860
16 2016-05-02 03:30:00 450.44 451.20 450.44 450.75 65.1480
17 2016-05-02 04:00:00 450.79 451.20 450.75 451.00 78.0430
18 2016-05-02 04:30:00 451.00 451.11 450.85 451.11 64.7250
19 2016-05-02 05:00:00 451.11 451.64 451.00 451.12 73.4840
20 2016-05-02 05:30:00 451.12 451.83 450.67 451.33 94.1950
21 2016-05-02 06:00:00 451.35 451.37 450.17 450.18 227.7480
22 2016-05-02 06:30:00 450.18 450.43 450.17 450.17 83.0270
23 2016-05-02 07:00:00 450.17 450.43 448.90 449.41 170.4950
24 2016-05-02 07:30:00 449.38 450.00 448.56 448.56 243.0420
25 2016-05-02 08:00:00 448.67 448.67 446.21 448.00 525.7090
26 2016-05-02 08:30:00 448.12 448.49 445.00 445.00 673.5810
27 2016-05-02 09:00:00 445.00 445.51 440.11 444.20 1392.9049
28 2016-05-02 09:30:00 444.24 444.36 440.11 442.00 438.6860
29 2016-05-02 10:00:00 441.91 443.20 440.05 442.24 400.5850
... ... ... ... ... ... ...
1651 2016-06-05 05:00:00 578.74 579.00 577.92 578.39 93.6980
1652 2016-06-05 05:30:00 578.40 578.48 574.52 575.26 98.1580
1653 2016-06-05 06:00:00 575.24 576.02 572.47 574.06 126.8620
1654 2016-06-05 06:30:00 574.06 576.35 574.06 576.34 125.4120
1655 2016-06-05 07:00:00 576.34 576.34 574.73 575.83 34.8070
1656 2016-06-05 07:30:00 575.84 576.27 574.91 575.58 74.8180
1657 2016-06-05 08:00:00 575.58 578.57 575.58 578.36 123.2560
1658 2016-06-05 08:30:00 578.23 578.47 576.18 577.25 43.6590
1659 2016-06-05 09:00:00 577.20 578.85 576.70 577.27 95.3900
1660 2016-06-05 09:30:00 577.36 578.18 576.70 576.70 51.0250
1661 2016-06-05 10:00:00 576.70 576.70 574.55 575.39 101.0590
1662 2016-06-05 10:30:00 575.41 576.44 575.18 576.44 86.4340
1663 2016-06-05 11:00:00 576.50 577.89 576.50 577.80 113.0600
1664 2016-06-05 11:30:00 577.80 578.10 576.03 576.98 57.5050
1665 2016-06-05 12:00:00 576.98 577.55 576.59 577.54 56.1070
1666 2016-06-05 12:30:00 577.54 583.00 570.93 572.82 872.8200
1667 2016-06-05 13:00:00 572.94 573.19 569.64 572.50 310.0020
1668 2016-06-05 13:30:00 572.50 574.37 572.50 574.09 59.3410
1669 2016-06-05 14:00:00 574.09 574.19 571.51 572.98 155.4310
1670 2016-06-05 14:30:00 572.98 573.57 572.02 573.47 76.9270
1671 2016-06-05 15:00:00 573.62 575.10 572.97 573.37 59.1430
1672 2016-06-05 15:30:00 573.37 574.39 573.37 574.38 77.3270
1673 2016-06-05 16:00:00 574.39 575.59 574.38 575.59 52.0150
1674 2016-06-05 16:30:00 575.00 575.59 574.50 575.00 66.9300
1675 2016-06-05 17:00:00 575.00 576.83 574.38 576.60 50.2990
1676 2016-06-05 17:30:00 576.60 577.50 575.50 576.86 104.5200
1677 2016-06-05 18:00:00 576.86 577.21 575.44 575.80 55.3270
1678 2016-06-05 18:30:00 575.77 575.80 574.52 574.77 78.7760
1679 2016-06-05 19:00:00 574.73 575.18 572.52 574.47 126.4300
1680 2016-06-05 19:30:00 574.49 574.87 573.80 574.32 10.4930
As you can see, it contains the last 35 days grouped by intervals of 30 min.
I wanna manipulate this price history in different time windows.
So, as a beginner example, I would like to fetch only the info from the last 1 day.
How can I filter this dataframe to show the info from the last 1 day?
This is what I've tried:
import datetime
d0 = datetime.datetime.today()
d1 = datetime.datetime.today() - datetime.timedelta(days=1)
print d0
>>> 2016-06-05 17:10:37.633824
print d1
>>> 2016-06-04 17:10:37.633967
df_1d = df30m['Timestamp'] > d1
print df_1d
This returns me a pandas series filled with True or False
0 False
1 False
2 False
3 False
4 False
...
1676 True
1677 True
1678 True
1679 True
1680 True
Also I've tried to use the between_time() module.
df_1d = df30m.between_time(d0, d1)
But I got the following error message:
TypeError: Index must be DatetimeIndex
Please, can anyone show me a pythonic way to slice my dataframe?

You can use loc to index your data. Do you know if your timestamps at datetime.datetime formats or Pandas Timestamps?
df30m.loc[(df30m.Timestamp <= d0) & (df30m.Timestamp >= d1)]
You can set the index to the Timestamp column and then index as follows:
df.set_index('Timestamp', inplace=True)
df[d1:d0]

Categories