how to split dataframe vertically, having N columns in each resulting DF - python

I have the following dataframe:
Date 2017-12-05 2017-12-06 2017-12-15 2017-12-19 2017-12-20 2017-12-21 ....
time
00:00:00 19.94 21.19 21.88 20.76 21.26 21.26
00:15:00 20.29 21.07 21.71 21.79 21.95 21.52
00:30:00 21.03 21.25 21.80 22.15 22.26 21.62
00:45:00 22.20 21.56 22.77 22.20 22.33 21.91
01:00:00 23.25 22.15 23.71 22.31 22.69 21.99
01:15:00 23.78 23.33 24.53 22.29 22.82 22.58
Columns are dates, and I want to split the dataframe by every 3 date columns. How can I do this?
Illustration of the final result with one of the resultant dataframes following the split:
Date 2017-12-05 2017-12-06 2017-12-15 2
time
00:00:00 19.94 21.19 21.88
00:15:00 20.29 21.07 21.71
00:30:00 21.03 21.25 21.80
00:45:00 22.20 21.56 22.77
01:00:00 23.25 22.15 23.71
01:15:00 23.78 23.33 24.53


you can use np.split(..., axis=1):
In [157]: N = 3
In [158]: dfs = np.split(df, np.arange(N, len(df.columns), N), axis=1)
In [159]: dfs[0]
Out[159]:
2017-12-05 2017-12-06 2017-12-15
00:00:00 19.94 21.19 21.88
00:15:00 20.29 21.07 21.71
00:30:00 21.03 21.25 21.80
00:45:00 22.20 21.56 22.77
01:00:00 23.25 22.15 23.71
01:15:00 23.78 23.33 24.53
In [160]: dfs[1]
Out[160]:
2017-12-19 2017-12-20 2017-12-21
00:00:00 20.76 21.26 21.26
00:15:00 21.79 21.95 21.52
00:30:00 22.15 22.26 21.62
00:45:00 22.20 22.33 21.91
01:00:00 22.31 22.69 21.99
01:15:00 22.29 22.82 22.58


Something like this
ddf=[]
for i in range(len(df.columns)/3):
ddf.append(df[df.columns[3*i:3*i+3]])
to generate a list df of dataframes, each with 3 columns.
Add the rest (with less than 3 columns):
ddf.append(df[df.columns[3*(i+1):]])

Related

dataframe data transfer with selected values to another dataframe

My goal is selecting the column Sabah in dataframe prdt and entering every value to repeated rows called Sabah in dataframe prcal
prcal
Vakit Start_Date End_Date Start_Time End_Time
0 Sabah 2022-01-01 2022-01-01 NaN NaN
1 Güneş 2022-01-01 2022-01-01 NaN NaN
2 Öğle 2022-01-01 2022-01-01 NaN NaN
3 İkindi 2022-01-01 2022-01-01 NaN NaN
4 Akşam 2022-01-01 2022-01-01 NaN NaN
..........................................................
2184 Sabah 2022-12-31 2022-12-31 NaN NaN
2185 Güneş 2022-12-31 2022-12-31 NaN NaN
2186 Öğle 2022-12-31 2022-12-31 NaN NaN
2187 İkindi 2022-12-31 2022-12-31 NaN NaN
2188 Akşam 2022-12-31 2022-12-31 NaN NaN
2189 rows × 5 columns
prdt
Day Sabah Güneş Öğle İkindi Akşam Yatsı
0 2022-01-01 06:51:00 08:29:00 13:08:00 15:29:00 17:47:00 19:20:00
1 2022-01-02 06:51:00 08:29:00 13:09:00 15:30:00 17:48:00 19:21:00
2 2022-01-03 06:51:00 08:29:00 13:09:00 15:30:00 17:48:00 19:22:00
3 2022-01-04 06:51:00 08:29:00 13:09:00 15:31:00 17:49:00 19:22:00
4 2022-01-05 06:51:00 08:29:00 13:10:00 15:32:00 17:50:00 19:23:00
...........................................................................
360 2022-12-27 06:49:00 08:27:00 13:06:00 15:25:00 17:43:00 19:16:00
361 2022-12-28 06:50:00 08:28:00 13:06:00 15:26:00 17:43:00 19:17:00
362 2022-12-29 06:50:00 08:28:00 13:07:00 15:26:00 17:44:00 19:18:00
363 2022-12-30 06:50:00 08:28:00 13:07:00 15:27:00 17:45:00 19:18:00
364 2022-12-31 06:50:00 08:28:00 13:07:00 15:28:00 17:46:00 19:19:00
365 rows × 7 columns
Selected every row called sabah prcal.iloc[::6,:]
Made a list for prdt['Sabah'].
When integrating prcal.iloc[::6,:] = prdt['Sabah'][0:365] I get a value error:
ValueError: Must have equal len keys and value when setting with an iterable

Does pandas resample changes sort order of the datetime index? How can I avoid this?

Hi have a dataframe with datetimeindex showing latest first and oldest last.
When using .resample.agg the order of the index turns around. I couldn't read anything in the docs. Why is that and how can I avoid/change it? Thank you.
Here I create an example:
index = pd.date_range(start='2021-06-20',end='2021-07-31',freq='1min')
index = index.sort_values(ascending=False)
randval = np.random.randint(10,50,size=(len(index),2),dtype=np.uint8)
df = pd.DataFrame(randval,index=index,columns=['min','max'])
df_filt = df.between_time('10:00','12:00')
df_resampled = df_filt.resample('10T').agg({'min':'sum','max':'mean'})
df_resampled = df_resampled.dropna()
print('Numpy Version: ',np.version.version,)
print('Pandas Version: ',pd.__version__,'\n')
print('RAW DATAFRAME:\n ',df,'\n')
print('FILTERED DATAFRAME:\n',df_filt,'\n')
print('RESAMPLED DATAFRAME WITH CHANGED ORDER:\n',df_resampled,)
I get the output:
Numpy Version: 1.21.1
Pandas Version: 1.3.0
RAW DATAFRAME:
min max
2021-07-31 00:00:00 20 33
2021-07-30 23:59:00 23 29
2021-07-30 23:58:00 18 45
... ... ...
2021-06-20 00:02:00 48 28
2021-06-20 00:01:00 19 19
2021-06-20 00:00:00 33 16
[59041 rows x 2 columns]
FILTERED DATAFRAME:
min max
2021-07-30 10:30:00 14 38
2021-07-30 10:29:00 27 46
2021-07-30 10:28:00 41 17
... ... ...
2021-06-20 10:02:00 30 12
2021-06-20 10:01:00 43 49
2021-06-20 10:00:00 43 22
[1271 rows x 2 columns]
RESAMPLED DATAFRAME WITH CHANGED ORDER:
min max
2021-06-20 10:00:00 366.0 27.9
2021-06-20 10:10:00 264.0 29.9
2021-06-20 10:20:00 236.0 34.6
... ... ...
2021-07-30 10:10:00 310.0 32.0
2021-07-30 10:20:00 343.0 35.7
2021-07-30 10:30:00 14.0 38.0
[164 rows x 2 columns]
EDIT
I add a .sort_index to df_resample. This seems to work. For me still a bit weird to do it this way though.
df_resampled = df_resampled.sort_index(ascending=False)
# or all combined:
df_resampled = df_filt.resample('10T').agg({'min':'sum','max':'mean'}).dropna().sort_index(ascending=False)
gives this output:
RESAMPLED DATAFRAME:
min max
2021-07-30 10:30:00 34.0 45.0
2021-07-30 10:20:00 321.0 29.1
2021-07-30 10:10:00 315.0 33.6
2021-07-30 10:00:00 326.0 30.8
2021-07-29 10:30:00 13.0 24.0
2021-07-29 10:20:00 277.0 35.5
2021-07-29 10:10:00 350.0 25.2
2021-07-29 10:00:00 285.0 28.6
2021-07-28 10:30:00 23.0 12.0
2021-07-28 10:20:00 279.0 24.9
2021-07-28 10:10:00 254.0 30.3
2021-07-28 10:00:00 352.0 34.8
2021-07-27 10:30:00 33.0 14.0
2021-07-27 10:20:00 309.0 21.2
2021-07-27 10:10:00 273.0 30.5
2021-07-27 10:00:00 340.0 30.0
2021-07-26 10:30:00 34.0 14.0
2021-07-26 10:20:00 334.0 30.9
2021-07-26 10:10:00 261.0 20.1
2021-07-26 10:00:00 284.0 29.5

How to change the value of a day column based on time column?

I have a df
Time Samstag Sonntag Werktag
00:15:00 95.3 87.8 94.7
00:30:00 95.5 88.3 94.1
00:45:00 96.2 89.0 94.1
01:00:00 97.4 90.1 95.0
01:15:00 98.9 91.3 96.6
01:30:00 100.3 92.4 98.4
01:45:00 101.0 92.9 99.8
02:00:00 100.4 92.5 99.8
02:15:00 98.2 91.0 98.0
02:30:00 95.1 88.7 95.1
02:45:00 91.9 86.4 91.9
03:00:00 89.5 84.7 89.5
03:15:00 88.6 84.0 88.4
03:30:00 88.6 84.0 88.3
03:45:00 88.7 84.0 88.3
04:00:00 88.3 83.5 87.7
04:15:00 86.8 82.1 86.1
04:30:00 85.1 80.6 84.3
04:45:00 84.2 80.1 83.5
05:00:00 85.3 81.6 84.7
05:15:00 89.0 85.9 88.5
05:30:00 94.1 91.6 94.0
05:45:00 99.3 97.0 99.5
06:00:00 102.8 100.4 103.4
06:15:00 103.7 100.7 104.7
06:30:00 102.6 98.8 104.0
06:45:00 100.7 96.2 102.4
07:00:00 99.2 94.3 101.0
07:15:00 99.1 94.4 100.8
07:30:00 100.8 95.7 102.1
07:45:00 104.4 97.6 105.3
08:00:00 110.1 99.2 110.7
08:15:00 117.7 99.7 118.2
08:30:00 126.1 99.6 126.7
08:45:00 133.9 99.2 134.7
09:00:00 139.7 99.2 140.9
09:15:00 142.4 99.8 144.2
09:30:00 142.9 100.9 145.4
09:45:00 142.4 102.1 145.5
10:00:00 142.1 102.8 145.8
10:15:00 142.9 102.9 147.0
10:30:00 144.5 102.5 149.0
10:45:00 146.3 101.8 151.2
11:00:00 147.6 101.0 153.0
11:15:00 147.9 100.4 154.0
11:30:00 147.5 100.0 154.3
11:45:00 146.8 99.8 154.3
12:00:00 146.4 99.8 154.2
12:15:00 146.3 100.0 154.3
12:30:00 146.5 100.5 154.5
12:45:00 146.2 101.0 154.3
13:00:00 145.1 101.6 153.6
13:15:00 142.8 102.2 152.2
13:30:00 139.3 102.4 149.9
13:45:00 134.6 102.1 147.0
14:00:00 128.8 101.0 143.3
14:15:00 122.3 98.9 139.2
14:30:00 115.5 96.3 135.2
14:45:00 109.4 93.8 132.1
15:00:00 104.6 91.9 130.6
15:15:00 101.8 91.1 131.3
15:30:00 100.5 91.2 133.5
15:45:00 100.2 91.8 136.2
16:00:00 100.4 92.5 138.5
16:15:00 100.6 93.1 139.8
16:30:00 101.0 93.4 140.3
16:45:00 101.9 93.6 140.5
17:00:00 103.4 93.7 140.9
17:15:00 105.8 93.9 142.0
17:30:00 108.7 94.3 143.7
17:45:00 111.5 95.2 145.8
18:00:00 113.7 96.8 148.2
18:15:00 115.0 99.1 150.6
18:30:00 115.7 102.2 152.5
18:45:00 116.3 105.7 153.3
19:00:00 117.3 109.5 152.4
19:15:00 119.0 113.2 149.3
19:30:00 120.6 116.3 144.4
19:45:00 121.4 117.9 138.4
20:00:00 120.4 117.3 131.8
20:15:00 117.0 114.2 125.3
20:30:00 112.1 109.4 119.3
20:45:00 106.8 104.2 114.3
21:00:00 102.2 99.8 110.7
21:15:00 99.2 97.1 108.8
21:30:00 97.4 95.9 108.1
21:45:00 96.4 95.4 108.0
22:00:00 95.6 95.0 107.7
22:15:00 94.5 94.1 106.6
22:30:00 93.3 92.8 104.9
22:45:00 92.0 91.2 103.0
23:00:00 90.7 89.5 101.0
23:15:00 89.6 87.8 99.3
23:30:00 88.6 86.4 97.8
23:45:00 88.0 85.7 96.6
00:00:00 87.7 85.9 95.6
I did:
td = pd.to_timedelta(df['Time'].astype(str))
df1 = df.assign(Time=td.mask(td == pd.Timedelta(0),td + pd.Timedelta('1 days 00:00:00')), a=1)
df2 = pd.DataFrame({'dates': pd.date_range(
'01.01.2020', '31.12.2020'), 'a': 1})
df = df2.merge(df1, how='outer').drop('a', axis=1)
df['dates'] = df['dates'].add(df.pop('Time')).dt.strftime('%d.%m.%Y %H:%M')
df['dates'] = pd.to_datetime(df['dates'], dayfirst=True)
df['day'] = df['dates'].dt.day_name()
It gave the following output:
dates Samstag Sonntag Werktag day
2020-01-01 00:15:00 95.3 87.8 94.7 Wednesday
2020-01-01 00:30:00 95.5 88.3 94.1 Wednesday
2020-01-01 00:45:00 96.2 89.0 94.1 Wednesday
2020-01-01 01:00:00 97.4 90.1 95.0 Wednesday
2020-01-01 01:15:00 98.9 91.3 96.6 Wednesday
2020-01-01 01:30:00 100.3 92.4 98.4 Wednesday
2020-01-01 01:45:00 101.0 92.9 99.8 Wednesday
2020-01-01 02:00:00 100.4 92.5 99.8 Wednesday
2020-01-01 02:15:00 98.2 91.0 98.0 Wednesday
2020-01-01 02:30:00 95.1 88.7 95.1 Wednesday
2020-01-01 02:45:00 91.9 86.4 91.9 Wednesday
2020-01-01 03:00:00 89.5 84.7 89.5 Wednesday
2020-01-01 03:15:00 88.6 84.0 88.4 Wednesday
2020-01-01 03:30:00 88.6 84.0 88.3 Wednesday
2020-01-01 03:45:00 88.7 84.0 88.3 Wednesday
2020-01-01 04:00:00 88.3 83.5 87.7 Wednesday
2020-01-01 04:15:00 86.8 82.1 86.1 Wednesday
2020-01-01 04:30:00 85.1 80.6 84.3 Wednesday
2020-01-01 04:45:00 84.2 80.1 83.5 Wednesday
2020-01-01 05:00:00 85.3 81.6 84.7 Wednesday
2020-01-01 05:15:00 89.0 85.9 88.5 Wednesday
2020-01-01 05:30:00 94.1 91.6 94.0 Wednesday
2020-01-01 05:45:00 99.3 97.0 99.5 Wednesday
2020-01-01 06:00:00 102.8 100.4 103.4 Wednesday
2020-01-01 06:15:00 103.7 100.7 104.7 Wednesday
2020-01-01 06:30:00 102.6 98.8 104.0 Wednesday
2020-01-01 06:45:00 100.7 96.2 102.4 Wednesday
2020-01-01 07:00:00 99.2 94.3 101.0 Wednesday
2020-01-01 07:15:00 99.1 94.4 100.8 Wednesday
2020-01-01 07:30:00 100.8 95.7 102.1 Wednesday
2020-01-01 07:45:00 104.4 97.6 105.3 Wednesday
2020-01-01 08:00:00 110.1 99.2 110.7 Wednesday
2020-01-01 08:15:00 117.7 99.7 118.2 Wednesday
2020-01-01 08:30:00 126.1 99.6 126.7 Wednesday
2020-01-01 08:45:00 133.9 99.2 134.7 Wednesday
2020-01-01 09:00:00 139.7 99.2 140.9 Wednesday
2020-01-01 09:15:00 142.4 99.8 144.2 Wednesday
2020-01-01 09:30:00 142.9 100.9 145.4 Wednesday
2020-01-01 09:45:00 142.4 102.1 145.5 Wednesday
2020-01-01 10:00:00 142.1 102.8 145.8 Wednesday
2020-01-01 10:15:00 142.9 102.9 147.0 Wednesday
2020-01-01 10:30:00 144.5 102.5 149.0 Wednesday
2020-01-01 10:45:00 146.3 101.8 151.2 Wednesday
2020-01-01 11:00:00 147.6 101.0 153.0 Wednesday
2020-01-01 11:15:00 147.9 100.4 154.0 Wednesday
2020-01-01 11:30:00 147.5 100.0 154.3 Wednesday
2020-01-01 11:45:00 146.8 99.8 154.3 Wednesday
2020-01-01 12:00:00 146.4 99.8 154.2 Wednesday
2020-01-01 12:15:00 146.3 100.0 154.3 Wednesday
2020-01-01 12:30:00 146.5 100.5 154.5 Wednesday
2020-01-01 12:45:00 146.2 101.0 154.3 Wednesday
2020-01-01 13:00:00 145.1 101.6 153.6 Wednesday
2020-01-01 13:15:00 142.8 102.2 152.2 Wednesday
2020-01-01 13:30:00 139.3 102.4 149.9 Wednesday
2020-01-01 13:45:00 134.6 102.1 147.0 Wednesday
2020-01-01 14:00:00 128.8 101.0 143.3 Wednesday
2020-01-01 14:15:00 122.3 98.9 139.2 Wednesday
2020-01-01 14:30:00 115.5 96.3 135.2 Wednesday
2020-01-01 14:45:00 109.4 93.8 132.1 Wednesday
2020-01-01 15:00:00 104.6 91.9 130.6 Wednesday
2020-01-01 15:15:00 101.8 91.1 131.3 Wednesday
2020-01-01 15:30:00 100.5 91.2 133.5 Wednesday
2020-01-01 15:45:00 100.2 91.8 136.2 Wednesday
2020-01-01 16:00:00 100.4 92.5 138.5 Wednesday
2020-01-01 16:15:00 100.6 93.1 139.8 Wednesday
2020-01-01 16:30:00 101.0 93.4 140.3 Wednesday
2020-01-01 16:45:00 101.9 93.6 140.5 Wednesday
2020-01-01 17:00:00 103.4 93.7 140.9 Wednesday
2020-01-01 17:15:00 105.8 93.9 142.0 Wednesday
2020-01-01 17:30:00 108.7 94.3 143.7 Wednesday
2020-01-01 17:45:00 111.5 95.2 145.8 Wednesday
2020-01-01 18:00:00 113.7 96.8 148.2 Wednesday
2020-01-01 18:15:00 115.0 99.1 150.6 Wednesday
2020-01-01 18:30:00 115.7 102.2 152.5 Wednesday
2020-01-01 18:45:00 116.3 105.7 153.3 Wednesday
2020-01-01 19:00:00 117.3 109.5 152.4 Wednesday
2020-01-01 19:15:00 119.0 113.2 149.3 Wednesday
2020-01-01 19:30:00 120.6 116.3 144.4 Wednesday
2020-01-01 19:45:00 121.4 117.9 138.4 Wednesday
2020-01-01 20:00:00 120.4 117.3 131.8 Wednesday
2020-01-01 20:15:00 117.0 114.2 125.3 Wednesday
2020-01-01 20:30:00 112.1 109.4 119.3 Wednesday
2020-01-01 20:45:00 106.8 104.2 114.3 Wednesday
2020-01-01 21:00:00 102.2 99.8 110.7 Wednesday
2020-01-01 21:15:00 99.2 97.1 108.8 Wednesday
2020-01-01 21:30:00 97.4 95.9 108.1 Wednesday
2020-01-01 21:45:00 96.4 95.4 108.0 Wednesday
2020-01-01 22:00:00 95.6 95.0 107.7 Wednesday
2020-01-01 22:15:00 94.5 94.1 106.6 Wednesday
2020-01-01 22:30:00 93.3 92.8 104.9 Wednesday
2020-01-01 22:45:00 92.0 91.2 103.0 Wednesday
2020-01-01 23:00:00 90.7 89.5 101.0 Wednesday
2020-01-01 23:15:00 89.6 87.8 99.3 Wednesday
2020-01-01 23:30:00 88.6 86.4 97.8 Wednesday
2020-01-01 23:45:00 88.0 85.7 96.6 Wednesday
2020-01-02 00:00:00 87.7 85.9 95.6 Thursday
2020-01-02 00:15:00 95.3 87.8 94.7 Thursday
2020-01-02 00:30:00 95.5 88.3 94.1 Thursday
2020-01-02 00:45:00 96.2 89.0 94.1 Thursday
2020-01-02 01:00:00 97.4 90.1 95.0 Thursday
2020-01-02 01:15:00 98.9 91.3 96.6 Thursday
2020-01-02 01:30:00 100.3 92.4 98.4 Thursday
2020-01-02 01:45:00 101.0 92.9 99.8 Thursday
2020-01-02 02:00:00 100.4 92.5 99.8 Thursday
2020-01-02 02:15:00 98.2 91.0 98.0 Thursday
2020-01-02 02:30:00 95.1 88.7 95.1 Thursday
2020-01-02 02:45:00 91.9 86.4 91.9 Thursday
2020-01-02 03:00:00 89.5 84.7 89.5 Thursday
2020-01-02 03:15:00 88.6 84.0 88.4 Thursday
2020-01-02 03:30:00 88.6 84.0 88.3 Thursday
2020-01-02 03:45:00 88.7 84.0 88.3 Thursday
2020-01-02 04:00:00 88.3 83.5 87.7 Thursday
2020-01-02 04:15:00 86.8 82.1 86.1 Thursday
2020-01-02 04:30:00 85.1 80.6 84.3 Thursday
2020-01-02 04:45:00 84.2 80.1 83.5 Thursday
2020-01-02 05:00:00 85.3 81.6 84.7 Thursday
2020-01-02 05:15:00 89.0 85.9 88.5 Thursday
2020-01-02 05:30:00 94.1 91.6 94.0 Thursday
2020-01-02 05:45:00 99.3 97.0 99.5 Thursday
2020-01-02 06:00:00 102.8 100.4 103.4 Thursday
2020-01-02 06:15:00 103.7 100.7 104.7 Thursday
2020-01-02 06:30:00 102.6 98.8 104.0 Thursday
2020-01-02 06:45:00 100.7 96.2 102.4 Thursday
2020-01-02 07:00:00 99.2 94.3 101.0 Thursday
2020-01-02 07:15:00 99.1 94.4 100.8 Thursday
2020-01-02 07:30:00 100.8 95.7 102.1 Thursday
2020-01-02 07:45:00 104.4 97.6 105.3 Thursday
2020-01-02 08:00:00 110.1 99.2 110.7 Thursday
2020-01-02 08:15:00 117.7 99.7 118.2 Thursday
2020-01-02 08:30:00 126.1 99.6 126.7 Thursday
2020-01-02 08:45:00 133.9 99.2 134.7 Thursday
2020-01-02 09:00:00 139.7 99.2 140.9 Thursday
2020-01-02 09:15:00 142.4 99.8 144.2 Thursday
2020-01-02 09:30:00 142.9 100.9 145.4 Thursday
2020-01-02 09:45:00 142.4 102.1 145.5 Thursday
2020-01-02 10:00:00 142.1 102.8 145.8 Thursday
2020-01-02 10:15:00 142.9 102.9 147.0 Thursday
2020-01-02 10:30:00 144.5 102.5 149.0 Thursday
2020-01-02 10:45:00 146.3 101.8 151.2 Thursday
2020-01-02 11:00:00 147.6 101.0 153.0 Thursday
2020-01-02 11:15:00 147.9 100.4 154.0 Thursday
2020-01-02 11:30:00 147.5 100.0 154.3 Thursday
2020-01-02 11:45:00 146.8 99.8 154.3 Thursday
2020-01-02 12:00:00 146.4 99.8 154.2 Thursday
2020-01-02 12:15:00 146.3 100.0 154.3 Thursday
2020-01-02 12:30:00 146.5 100.5 154.5 Thursday
2020-01-02 12:45:00 146.2 101.0 154.3 Thursday
2020-01-02 13:00:00 145.1 101.6 153.6 Thursday
2020-01-02 13:15:00 142.8 102.2 152.2 Thursday
2020-01-02 13:30:00 139.3 102.4 149.9 Thursday
2020-01-02 13:45:00 134.6 102.1 147.0 Thursday
2020-01-02 14:00:00 128.8 101.0 143.3 Thursday
2020-01-02 14:15:00 122.3 98.9 139.2 Thursday
2020-01-02 14:30:00 115.5 96.3 135.2 Thursday
2020-01-02 14:45:00 109.4 93.8 132.1 Thursday
2020-01-02 15:00:00 104.6 91.9 130.6 Thursday
2020-01-02 15:15:00 101.8 91.1 131.3 Thursday
2020-01-02 15:30:00 100.5 91.2 133.5 Thursday
2020-01-02 15:45:00 100.2 91.8 136.2 Thursday
2020-01-02 16:00:00 100.4 92.5 138.5 Thursday
2020-01-02 16:15:00 100.6 93.1 139.8 Thursday
2020-01-02 16:30:00 101.0 93.4 140.3 Thursday
2020-01-02 16:45:00 101.9 93.6 140.5 Thursday
2020-01-02 17:00:00 103.4 93.7 140.9 Thursday
2020-01-02 17:15:00 105.8 93.9 142.0 Thursday
2020-01-02 17:30:00 108.7 94.3 143.7 Thursday
2020-01-02 17:45:00 111.5 95.2 145.8 Thursday
2020-01-02 18:00:00 113.7 96.8 148.2 Thursday
2020-01-02 18:15:00 115.0 99.1 150.6 Thursday
2020-01-02 18:30:00 115.7 102.2 152.5 Thursday
2020-01-02 18:45:00 116.3 105.7 153.3 Thursday
2020-01-02 19:00:00 117.3 109.5 152.4 Thursday
2020-01-02 19:15:00 119.0 113.2 149.3 Thursday
2020-01-02 19:30:00 120.6 116.3 144.4 Thursday
2020-01-02 19:45:00 121.4 117.9 138.4 Thursday
2020-01-02 20:00:00 120.4 117.3 131.8 Thursday
2020-01-02 20:15:00 117.0 114.2 125.3 Thursday
2020-01-02 20:30:00 112.1 109.4 119.3 Thursday
2020-01-02 20:45:00 106.8 104.2 114.3 Thursday
2020-01-02 21:00:00 102.2 99.8 110.7 Thursday
2020-01-02 21:15:00 99.2 97.1 108.8 Thursday
2020-01-02 21:30:00 97.4 95.9 108.1 Thursday
2020-01-02 21:45:00 96.4 95.4 108.0 Thursday
2020-01-02 22:00:00 95.6 95.0 107.7 Thursday
2020-01-02 22:15:00 94.5 94.1 106.6 Thursday
2020-01-02 22:30:00 93.3 92.8 104.9 Thursday
2020-01-02 22:45:00 92.0 91.2 103.0 Thursday
2020-01-02 23:00:00 90.7 89.5 101.0 Thursday
2020-01-02 23:15:00 89.6 87.8 99.3 Thursday
2020-01-02 23:30:00 88.6 86.4 97.8 Thursday
2020-01-02 23:45:00 88.0 85.7 96.6 Thursday
2020-01-03 00:00:00 87.7 85.9 95.6 Friday
2020-01-03 00:15:00 95.3 87.8 94.7 Friday
2020-01-03 00:30:00 95.5 88.3 94.1 Friday
2020-01-03 00:45:00 96.2 89.0 94.1 Friday
What I would like to do is to change the value of day at 2020-01-02 00:00:00 from Thursday to Wednesday, and similarly the value of day at 2020-01-03 00:00:00 from Friday to Thursday and so on.
In other words: The value of day for next day at 00:00:00 should be similar to the value of the previous day and from 00:15:00, a new day should begin.
Expected output
dates Samstag Sonntag Werktag day
2020-01-01 00:15:00 95.3 87.8 94.7 Wednesday
2020-01-01 00:30:00 95.5 88.3 94.1 Wednesday
2020-01-01 00:45:00 96.2 89.0 94.1 Wednesday
2020-01-01 01:00:00 97.4 90.1 95.0 Wednesday
2020-01-01 01:15:00 98.9 91.3 96.6 Wednesday
2020-01-01 01:30:00 100.3 92.4 98.4 Wednesday
2020-01-01 01:45:00 101.0 92.9 99.8 Wednesday
2020-01-01 02:00:00 100.4 92.5 99.8 Wednesday
2020-01-01 02:15:00 98.2 91.0 98.0 Wednesday
2020-01-01 02:30:00 95.1 88.7 95.1 Wednesday
2020-01-01 02:45:00 91.9 86.4 91.9 Wednesday
2020-01-01 03:00:00 89.5 84.7 89.5 Wednesday
2020-01-01 03:15:00 88.6 84.0 88.4 Wednesday
2020-01-01 03:30:00 88.6 84.0 88.3 Wednesday
2020-01-01 03:45:00 88.7 84.0 88.3 Wednesday
2020-01-01 04:00:00 88.3 83.5 87.7 Wednesday
2020-01-01 04:15:00 86.8 82.1 86.1 Wednesday
2020-01-01 04:30:00 85.1 80.6 84.3 Wednesday
2020-01-01 04:45:00 84.2 80.1 83.5 Wednesday
2020-01-01 05:00:00 85.3 81.6 84.7 Wednesday
2020-01-01 05:15:00 89.0 85.9 88.5 Wednesday
2020-01-01 05:30:00 94.1 91.6 94.0 Wednesday
2020-01-01 05:45:00 99.3 97.0 99.5 Wednesday
2020-01-01 06:00:00 102.8 100.4 103.4 Wednesday
2020-01-01 06:15:00 103.7 100.7 104.7 Wednesday
2020-01-01 06:30:00 102.6 98.8 104.0 Wednesday
2020-01-01 06:45:00 100.7 96.2 102.4 Wednesday
2020-01-01 07:00:00 99.2 94.3 101.0 Wednesday
2020-01-01 07:15:00 99.1 94.4 100.8 Wednesday
2020-01-01 07:30:00 100.8 95.7 102.1 Wednesday
2020-01-01 07:45:00 104.4 97.6 105.3 Wednesday
2020-01-01 08:00:00 110.1 99.2 110.7 Wednesday
2020-01-01 08:15:00 117.7 99.7 118.2 Wednesday
2020-01-01 08:30:00 126.1 99.6 126.7 Wednesday
2020-01-01 08:45:00 133.9 99.2 134.7 Wednesday
2020-01-01 09:00:00 139.7 99.2 140.9 Wednesday
2020-01-01 09:15:00 142.4 99.8 144.2 Wednesday
2020-01-01 09:30:00 142.9 100.9 145.4 Wednesday
2020-01-01 09:45:00 142.4 102.1 145.5 Wednesday
2020-01-01 10:00:00 142.1 102.8 145.8 Wednesday
2020-01-01 10:15:00 142.9 102.9 147.0 Wednesday
2020-01-01 10:30:00 144.5 102.5 149.0 Wednesday
2020-01-01 10:45:00 146.3 101.8 151.2 Wednesday
2020-01-01 11:00:00 147.6 101.0 153.0 Wednesday
2020-01-01 11:15:00 147.9 100.4 154.0 Wednesday
2020-01-01 11:30:00 147.5 100.0 154.3 Wednesday
2020-01-01 11:45:00 146.8 99.8 154.3 Wednesday
2020-01-01 12:00:00 146.4 99.8 154.2 Wednesday
2020-01-01 12:15:00 146.3 100.0 154.3 Wednesday
2020-01-01 12:30:00 146.5 100.5 154.5 Wednesday
2020-01-01 12:45:00 146.2 101.0 154.3 Wednesday
2020-01-01 13:00:00 145.1 101.6 153.6 Wednesday
2020-01-01 13:15:00 142.8 102.2 152.2 Wednesday
2020-01-01 13:30:00 139.3 102.4 149.9 Wednesday
2020-01-01 13:45:00 134.6 102.1 147.0 Wednesday
2020-01-01 14:00:00 128.8 101.0 143.3 Wednesday
2020-01-01 14:15:00 122.3 98.9 139.2 Wednesday
2020-01-01 14:30:00 115.5 96.3 135.2 Wednesday
2020-01-01 14:45:00 109.4 93.8 132.1 Wednesday
2020-01-01 15:00:00 104.6 91.9 130.6 Wednesday
2020-01-01 15:15:00 101.8 91.1 131.3 Wednesday
2020-01-01 15:30:00 100.5 91.2 133.5 Wednesday
2020-01-01 15:45:00 100.2 91.8 136.2 Wednesday
2020-01-01 16:00:00 100.4 92.5 138.5 Wednesday
2020-01-01 16:15:00 100.6 93.1 139.8 Wednesday
2020-01-01 16:30:00 101.0 93.4 140.3 Wednesday
2020-01-01 16:45:00 101.9 93.6 140.5 Wednesday
2020-01-01 17:00:00 103.4 93.7 140.9 Wednesday
2020-01-01 17:15:00 105.8 93.9 142.0 Wednesday
2020-01-01 17:30:00 108.7 94.3 143.7 Wednesday
2020-01-01 17:45:00 111.5 95.2 145.8 Wednesday
2020-01-01 18:00:00 113.7 96.8 148.2 Wednesday
2020-01-01 18:15:00 115.0 99.1 150.6 Wednesday
2020-01-01 18:30:00 115.7 102.2 152.5 Wednesday
2020-01-01 18:45:00 116.3 105.7 153.3 Wednesday
2020-01-01 19:00:00 117.3 109.5 152.4 Wednesday
2020-01-01 19:15:00 119.0 113.2 149.3 Wednesday
2020-01-01 19:30:00 120.6 116.3 144.4 Wednesday
2020-01-01 19:45:00 121.4 117.9 138.4 Wednesday
2020-01-01 20:00:00 120.4 117.3 131.8 Wednesday
2020-01-01 20:15:00 117.0 114.2 125.3 Wednesday
2020-01-01 20:30:00 112.1 109.4 119.3 Wednesday
2020-01-01 20:45:00 106.8 104.2 114.3 Wednesday
2020-01-01 21:00:00 102.2 99.8 110.7 Wednesday
2020-01-01 21:15:00 99.2 97.1 108.8 Wednesday
2020-01-01 21:30:00 97.4 95.9 108.1 Wednesday
2020-01-01 21:45:00 96.4 95.4 108.0 Wednesday
2020-01-01 22:00:00 95.6 95.0 107.7 Wednesday
2020-01-01 22:15:00 94.5 94.1 106.6 Wednesday
2020-01-01 22:30:00 93.3 92.8 104.9 Wednesday
2020-01-01 22:45:00 92.0 91.2 103.0 Wednesday
2020-01-01 23:00:00 90.7 89.5 101.0 Wednesday
2020-01-01 23:15:00 89.6 87.8 99.3 Wednesday
2020-01-01 23:30:00 88.6 86.4 97.8 Wednesday
2020-01-01 23:45:00 88.0 85.7 96.6 Wednesday
2020-01-02 00:00:00 87.7 85.9 95.6 Wednesday
2020-01-02 00:15:00 95.3 87.8 94.7 Thursday
2020-01-02 00:30:00 95.5 88.3 94.1 Thursday
2020-01-02 00:45:00 96.2 89.0 94.1 Thursday
2020-01-02 01:00:00 97.4 90.1 95.0 Thursday
2020-01-02 01:15:00 98.9 91.3 96.6 Thursday
2020-01-02 01:30:00 100.3 92.4 98.4 Thursday
2020-01-02 01:45:00 101.0 92.9 99.8 Thursday
2020-01-02 02:00:00 100.4 92.5 99.8 Thursday
2020-01-02 02:15:00 98.2 91.0 98.0 Thursday
2020-01-02 02:30:00 95.1 88.7 95.1 Thursday
2020-01-02 02:45:00 91.9 86.4 91.9 Thursday
2020-01-02 03:00:00 89.5 84.7 89.5 Thursday
2020-01-02 03:15:00 88.6 84.0 88.4 Thursday
2020-01-02 03:30:00 88.6 84.0 88.3 Thursday
2020-01-02 03:45:00 88.7 84.0 88.3 Thursday
2020-01-02 04:00:00 88.3 83.5 87.7 Thursday
2020-01-02 04:15:00 86.8 82.1 86.1 Thursday
2020-01-02 04:30:00 85.1 80.6 84.3 Thursday
2020-01-02 04:45:00 84.2 80.1 83.5 Thursday
2020-01-02 05:00:00 85.3 81.6 84.7 Thursday
2020-01-02 05:15:00 89.0 85.9 88.5 Thursday
2020-01-02 05:30:00 94.1 91.6 94.0 Thursday
2020-01-02 05:45:00 99.3 97.0 99.5 Thursday
2020-01-02 06:00:00 102.8 100.4 103.4 Thursday
2020-01-02 06:15:00 103.7 100.7 104.7 Thursday
2020-01-02 06:30:00 102.6 98.8 104.0 Thursday
2020-01-02 06:45:00 100.7 96.2 102.4 Thursday
2020-01-02 07:00:00 99.2 94.3 101.0 Thursday
2020-01-02 07:15:00 99.1 94.4 100.8 Thursday
2020-01-02 07:30:00 100.8 95.7 102.1 Thursday
2020-01-02 07:45:00 104.4 97.6 105.3 Thursday
2020-01-02 08:00:00 110.1 99.2 110.7 Thursday
2020-01-02 08:15:00 117.7 99.7 118.2 Thursday
2020-01-02 08:30:00 126.1 99.6 126.7 Thursday
2020-01-02 08:45:00 133.9 99.2 134.7 Thursday
2020-01-02 09:00:00 139.7 99.2 140.9 Thursday
2020-01-02 09:15:00 142.4 99.8 144.2 Thursday
2020-01-02 09:30:00 142.9 100.9 145.4 Thursday
2020-01-02 09:45:00 142.4 102.1 145.5 Thursday
2020-01-02 10:00:00 142.1 102.8 145.8 Thursday
2020-01-02 10:15:00 142.9 102.9 147.0 Thursday
2020-01-02 10:30:00 144.5 102.5 149.0 Thursday
2020-01-02 10:45:00 146.3 101.8 151.2 Thursday
2020-01-02 11:00:00 147.6 101.0 153.0 Thursday
2020-01-02 11:15:00 147.9 100.4 154.0 Thursday
2020-01-02 11:30:00 147.5 100.0 154.3 Thursday
2020-01-02 11:45:00 146.8 99.8 154.3 Thursday
2020-01-02 12:00:00 146.4 99.8 154.2 Thursday
2020-01-02 12:15:00 146.3 100.0 154.3 Thursday
2020-01-02 12:30:00 146.5 100.5 154.5 Thursday
2020-01-02 12:45:00 146.2 101.0 154.3 Thursday
2020-01-02 13:00:00 145.1 101.6 153.6 Thursday
2020-01-02 13:15:00 142.8 102.2 152.2 Thursday
2020-01-02 13:30:00 139.3 102.4 149.9 Thursday
2020-01-02 13:45:00 134.6 102.1 147.0 Thursday
2020-01-02 14:00:00 128.8 101.0 143.3 Thursday
2020-01-02 14:15:00 122.3 98.9 139.2 Thursday
2020-01-02 14:30:00 115.5 96.3 135.2 Thursday
2020-01-02 14:45:00 109.4 93.8 132.1 Thursday
2020-01-02 15:00:00 104.6 91.9 130.6 Thursday
2020-01-02 15:15:00 101.8 91.1 131.3 Thursday
2020-01-02 15:30:00 100.5 91.2 133.5 Thursday
2020-01-02 15:45:00 100.2 91.8 136.2 Thursday
2020-01-02 16:00:00 100.4 92.5 138.5 Thursday
2020-01-02 16:15:00 100.6 93.1 139.8 Thursday
2020-01-02 16:30:00 101.0 93.4 140.3 Thursday
2020-01-02 16:45:00 101.9 93.6 140.5 Thursday
2020-01-02 17:00:00 103.4 93.7 140.9 Thursday
2020-01-02 17:15:00 105.8 93.9 142.0 Thursday
2020-01-02 17:30:00 108.7 94.3 143.7 Thursday
2020-01-02 17:45:00 111.5 95.2 145.8 Thursday
2020-01-02 18:00:00 113.7 96.8 148.2 Thursday
2020-01-02 18:15:00 115.0 99.1 150.6 Thursday
2020-01-02 18:30:00 115.7 102.2 152.5 Thursday
2020-01-02 18:45:00 116.3 105.7 153.3 Thursday
2020-01-02 19:00:00 117.3 109.5 152.4 Thursday
2020-01-02 19:15:00 119.0 113.2 149.3 Thursday
2020-01-02 19:30:00 120.6 116.3 144.4 Thursday
2020-01-02 19:45:00 121.4 117.9 138.4 Thursday
2020-01-02 20:00:00 120.4 117.3 131.8 Thursday
2020-01-02 20:15:00 117.0 114.2 125.3 Thursday
2020-01-02 20:30:00 112.1 109.4 119.3 Thursday
2020-01-02 20:45:00 106.8 104.2 114.3 Thursday
2020-01-02 21:00:00 102.2 99.8 110.7 Thursday
2020-01-02 21:15:00 99.2 97.1 108.8 Thursday
2020-01-02 21:30:00 97.4 95.9 108.1 Thursday
2020-01-02 21:45:00 96.4 95.4 108.0 Thursday
2020-01-02 22:00:00 95.6 95.0 107.7 Thursday
2020-01-02 22:15:00 94.5 94.1 106.6 Thursday
2020-01-02 22:30:00 93.3 92.8 104.9 Thursday
2020-01-02 22:45:00 92.0 91.2 103.0 Thursday
2020-01-02 23:00:00 90.7 89.5 101.0 Thursday
2020-01-02 23:15:00 89.6 87.8 99.3 Thursday
2020-01-02 23:30:00 88.6 86.4 97.8 Thursday
2020-01-02 23:45:00 88.0 85.7 96.6 Thursday
2020-01-03 00:00:00 87.7 85.9 95.6 Thursday
2020-01-03 00:15:00 95.3 87.8 94.7 Friday
2020-01-03 00:30:00 95.5 88.3 94.1 Friday
2020-01-03 00:45:00 96.2 89.0 94.1 Friday
How can this be done??
Edit 1
import pandas as pd
df = pd.DataFrame({ 'dates': ['2020-01-01 22:15:00',
'2020-01-01 22:35:00',
'2020-01-01 22:45:00',
'2020-01-01 23:00:00',
'2020-01-01 23:15:00',
'2020-01-01 23:30:00',
'2020-01-01 23:45:00',
'2020-01-02 00:00:00',
'2020-01-02 22:15:00',
'2020-01-02 22:35:00',
'2020-01-02 22:45:00',
'2020-01-02 23:00:00',
'2020-01-02 23:15:00',
'2020-01-02 23:30:00',
'2020-01-02 23:45:00',
'2020-01-03 00:00:00'],
'expected_output':['Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Wednesday',
'Thursday',
'Thursday',
'Thursday',
'Thursday',
'Thursday','Thursday','Thursday','Thursday']})

Just check the minutes of Timestamp using apply.
# df = pd.DataFrame({'dates': ['2020-01-01 22:15:00', .....]}, )
# convert str date into Timestamp
df['dates'] = pd.to_datetime(df['dates'])
def calculate_day(x):
# get previous day
if x.hour == 0 and x.minute < 15:
return (x - pd.DateOffset(days=1)).day_name()
return x.day_name()
df['day'] = df['dates'].apply(calculate_day)
print(df)
# dates day
#0 2020-01-01 22:15:00 Wednesday
#...
JFYI: weekday_name deprecated. Use day_name().
Hope this helps.

Concatenate all dataframe columns into a single column

I have a dataframe that looks roughly like:
01/01/19 02/01/19 03/01/19 04/01/19
hour
1.0 27.08 47.73 54.24 10.0
2.0 26.06 49.53 46.09 22.0
...
24.0 12.0 34.0 22.0 40.0
I'd like to reduce its dimension to a single column with a proper date index concatenating all the columns. Is there a smart pandas way to do it?
Expected result... something like:
01/01/19 00:00:00 27.08
01/01/19 01:00:00 26.08
...
01/01/19 23:00:00 12.00
02/01/19 00:00:00 47.73
02/01/19 01:00:00 49.53
...
02/01/19 23:00:00 34.00
...

You can stack and then fix the index using pd.to_datetime and pd.to_timedelta:
u = df.stack()
u.index = (pd.to_datetime(u.index.get_level_values(1), dayfirst=True)
+ pd.to_timedelta(u.index.get_level_values(0) - 1, unit='h'))
u.sort_index()
2019-01-01 00:00:00 27.08
2019-01-01 01:00:00 26.06
2019-01-01 23:00:00 12.00
2019-01-02 00:00:00 47.73
2019-01-02 01:00:00 49.53
2019-01-02 23:00:00 34.00
2019-01-03 00:00:00 54.24
2019-01-03 01:00:00 46.09
2019-01-03 23:00:00 22.00
2019-01-04 00:00:00 10.00
2019-01-04 01:00:00 22.00
2019-01-04 23:00:00 40.00
dtype: float64

Pandas DataFrame Calculate time difference between 2 columns on specific time range

I want to calculate time difference between two columns on specific time range.
I try df.between_time but it only works on index.
Ex. Time range: between 18.00 - 8.00
Data :
start stop
0 2018-07-16 16:00:00 2018-07-16 20:00:00
1 2018-07-11 08:03:00 2018-07-11 12:03:00
2 2018-07-13 17:54:00 2018-07-13 21:54:00
3 2018-07-14 13:09:00 2018-07-14 17:09:00
4 2018-07-20 00:21:00 2018-07-20 04:21:00
5 2018-07-20 17:00:00 2018-07-21 09:00:00
Expect Result :
start stop time_diff
0 2018-07-16 16:00:00 2018-07-16 20:00:00 02:00:00
1 2018-07-11 08:03:00 2018-07-11 12:03:00 0
2 2018-07-13 17:54:00 2018-07-13 21:54:00 03:54:00
3 2018-07-14 13:09:00 2018-07-14 17:09:00 0
4 2018-07-20 00:21:00 2018-07-20 04:21:00 04:00:00
5 2018-07-20 17:00:00 2018-07-21 09:00:00 14:00:00
Note: If time_diff > 1 days, I already deal with that case.
Question: Should I build a function to do this or there are pandas build-in function to do this? Any help or guide would be appreciated.

I think this can be a solution
tmp = pd.DataFrame({'time1': pd.to_datetime(['2018-07-16 16:00:00', '2018-07-11 08:03:00',
'2018-07-13 17:54:00', '2018-07-14 13:09:00',
'2018-07-20 00:21:00', '2018-07-20 17:00:00']),
'time2': pd.to_datetime(['2018-07-16 20:00:00', '2018-07-11 12:03:00',
'2018-07-13 21:54:00', '2018-07-14 17:09:00',
'2018-07-20 04:21:00', '2018-07-21 09:00:00'])})
time1_date = tmp.time1.dt.date.astype(str)
tmp['rule18'], tmp['rule08'] = pd.to_datetime(time1_date + ' 18:00:00'), pd.to_datetime(time1_date + ' 08:00:00')
# if stop exceeds 18:00:00, compute time difference from this hour
tmp['time_diff_rule1'] = np.where(tmp.time2 > tmp.rule18, (tmp.time2 - tmp.rule18), (tmp.time2 - tmp.time1))
# rearrange the dataframe with your second rule
tmp['time_diff_rule2'] = np.where((tmp.time2 < tmp.rule18) & (tmp.time1 > tmp.rule08), 0, tmp['time_diff_rule1'])
time_diff_rule1 time_diff_rule2
0 02:00:00 02:00:00
1 04:00:00 00:00:00
2 03:54:00 03:54:00
3 04:00:00 00:00:00
4 04:00:00 04:00:00
5 15:00:00 15:00:00

Categories