We Keep Coding

use grouper to aggregate by week from the exact day

I would like to aggregate my data on week, using pandas grouper, where the week ends at the exact same day of my last date, and not the end of the week.
This is the code I wrote:
fp.groupby(pd.Grouper(key='date',freq='w')).collectionName.nunique().tail(10)
And these are the results:
date
2021-10-03 644
2021-10-10 698
2021-10-17 756
2021-10-24 839
2021-10-31 883
2021-11-07 905
2021-11-14 961
2021-11-21 1028
2021-11-28 990
2021-12-05 726
Freq: W-SUN, Name: collectionName, dtype: int64
The last date I have is 2021-12-02, so I would like that to be the last day of the week aggregate, and it goes back every 7 days, to the end (in this case beginning of the dataset).
I need help with this.

Use pd.DataFrame.resample with rule='1w', on='date' and origin='end_day'

This assumes you can find the last date prior to grouping. See references here: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#offset-aliases and here: https://www.programiz.com/python-programming/datetime/strftime
df = pd.DataFrame({'date': pd.date_range(start='2021-01-01 01:04:16', periods=250), 'val':range(1,251)})
date val
0 2021-01-01 01:04:16 1
1 2021-01-02 01:04:16 2
2 2021-01-03 01:04:16 3
3 2021-01-04 01:04:16 4
4 2021-01-05 01:04:16 5
.. ... ...
245 2021-09-03 01:04:16 246
246 2021-09-04 01:04:16 247
247 2021-09-05 01:04:16 248
248 2021-09-06 01:04:16 249
249 2021-09-07 01:04:16 250
[250 rows x 2 columns]
# locate last date and get day of week in correct format
anchor = df['date'].iat[-1].strftime("%a")
df.groupby(pd.Grouper(key='date',freq='w-' + anchor)).nunique().tail(5)
# week ends on the same day as the original dataset
val
date
2021-08-10 7
2021-08-17 7
2021-08-24 7
2021-08-31 7
2021-09-07 7

Python/Pandas: How to merge rows based on other column values

I have a table of work experience data. The problem I'm facing is some people's work experiences have overlapping dates (ie. See rows 240, 241 & 242, 243) where the start date occurs before end date of the previous job. This causes an overstatement of the total years of experience. For purposes of calculating total years of experience, how can I get total years of work experience without double counting overlapping jobs like example shown.
I initially summed the position tenure for each person to get the total years of experience but that doesn't account for double counting.

Try:
Input data:
>>> df
start_date end_date
237 2005-01-01 2007-12-01
238 2008-01-01 2012-09-01
239 2012-09-01 2013-07-01
240 2013-07-01 2016-05-01
241 2014-06-01 2016-05-01
242 2016-05-01 2019-10-01
243 2018-01-01 2019-10-01
244 2020-05-01 2021-08-03
First compute the diff between end_date and start_date:
df['diff1'] = df['end_date'] - df['start_date']
print(df)
start_date end_date diff1
237 2005-01-01 2007-12-01 1064 days
238 2008-01-01 2012-09-01 1705 days
239 2012-09-01 2013-07-01 303 days
240 2013-07-01 2016-05-01 1035 days
241 2014-06-01 2016-05-01 700 days
242 2016-05-01 2019-10-01 1248 days
243 2018-01-01 2019-10-01 638 days
244 2020-05-01 2021-08-03 459 days
Now, subtract start_date and end_date - 1 only if the start date occurs before end date of the previous row:
df['diff2'] = (df['start_date'] - df['end_date'].shift()) \
.mul(df['start_date'].le(df['end_date'].shift()))
print(df)
start_date end_date diff1 diff2
0 2005-01-01 2007-12-01 1064 days NaT
1 2008-01-01 2012-09-01 1705 days 0 days
2 2012-09-01 2013-07-01 303 days 0 days
3 2013-07-01 2016-05-01 1035 days 0 days
4 2014-06-01 2016-05-01 700 days -700 days
5 2016-05-01 2019-10-01 1248 days 0 days
6 2018-01-01 2019-10-01 638 days -638 days
7 2020-05-01 2021-08-03 459 days 0 days
Finally, add the two diffX columns:
df['real'] = df[['diff1', 'diff2']].sum(axis=1)
print(df)
start_date end_date diff1 diff2 real
0 2005-01-01 2007-12-01 1064 days NaT 1064 days
1 2008-01-01 2012-09-01 1705 days 0 days 1705 days
2 2012-09-01 2013-07-01 303 days 0 days 303 days
3 2013-07-01 2016-05-01 1035 days 0 days 1035 days
4 2014-06-01 2016-05-01 700 days -700 days 0 days
5 2016-05-01 2019-10-01 1248 days 0 days 1248 days
6 2018-01-01 2019-10-01 638 days -638 days 0 days
7 2020-05-01 2021-08-03 459 days 0 days 459 days
The real experience is df['real'].sum().days / 365, almost 16 years instead of 19.5 years. You can put this code into a function and call it when apply after groupby on person_id.
How would I create a function that can be used with the apply method?
def total_xp_years(df):
diff1 = df['end_date'] - df['start_date']
diff2 = df['start_date'] - df['end_date'].shift()
diff2 *= df['start_date'].le(df['end_date'].shift())
return (diff1.sum() + diff2.sum()).days / 365
dfxp = df.groupby('person_id').apply(total_xp_years)

Rearrange rows in dataframe pandas

I have a dataframe as follows
DIASEMANA DATAOCORRENCIA TOTAL
0 Domingo 2018-01-07 148
1 Domingo 2018-01-14 126
2 Domingo 2018-01-21 111
3 Domingo 2018-01-28 130
4 Domingo 2018-02-04 120
... ... ... ...
360 Terca 2018-11-27 228
361 Terca 2018-12-04 179
362 Terca 2018-12-11 121
363 Terca 2018-12-18 125
364 Terca 2018-12-25 60
And I want to rearrange by placing the lines in the following order of the DAYWEEK column: Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday.
I tried to use reindex but I have around 52 values ​​for each day of the week.
Does anyone have any ideas?
Thanks folks!

First you want to make sure your date is of datetime type, then you can extract day_of_week and sort:
df['DATAOCORRENCIA'] = pd.to_datetime(df['DATAOCORRENCIA'])
df['day_of_week'] = df.DATAOCORRENCIA.dt.dayofweek
df = df.sort_values('day_of_week')

Sum set of values from pandas dataframe within certain time frame

I have a fairly complicated question. I need to select rows from a data frame within a certain set of start and end dates, and then sum those values and put them in a new dataframe.
So I start off with with data frame, df:
import random
dates = pd.date_range('20150101 020000',periods=1000)
df = pd.DataFrame({'_id': random.choice(range(0, 1000)),
'time_stamp': dates,
'value': random.choice(range(2,60))
})
and define some start and end dates:
import pandas as pd
start_date = ["2-13-16", "2-23-16", "3-17-16", "3-24-16", "3-26-16", "5-17-16", "5-25-16", "10-10-16", "10-18-16", "10-23-16", "10-31-16", "11-7-16", "11-14-16", "11-22-16", "1-23-17", "1-29-17", "2-06-17", "3-11-17", "3-23-17", "6-21-17", "6-28-17"]
end_date = pd.DatetimeIndex(start_date) + pd.DateOffset(7)
Then what needs to happen is that I need to create a new data frame with weekly_sum which sums the value column of df which occur in between the the start_date and end_date.
So for example, the first row of the new data frame would return the sum of the values between 2-13-16 and 2-20-16. I imagine I'd use groupby.sum() or something similar.
It might look like this:
id start_date end_date weekly_sum
65 2016-02-13 2016-02-20 100
Any direction is greatly appreciated!
P.S. I know my use of random.choice is a little wonky so if you have a better way of generating random numbers, I'd love to see it!

You can use
def get_dates(x):
# Select the df values between start and ending datetime.
n = df[(df['time_stamp']>x['start'])&(df['time_stamp']<x['end'])]
# Return first id and sum of values
return n['id'].values[0],n['value'].sum()
dates = pd.date_range('20150101 020000',periods=1000)
df = pd.DataFrame({'id': np.random.randint(0,1000,size=(1000,)),
'time_stamp': dates,
'value': np.random.randint(2,60,size=(1000,))
})
ndf = pd.DataFrame({'start':pd.to_datetime(start_date),'end':end_date})
#Unpack and assign values to id and value column
ndf[['id','value']] = ndf.apply(lambda x : get_dates(x),1).apply(pd.Series)
print(df.head(5))
id time_stamp value
0 770 2015-01-01 02:00:00 59
1 781 2015-01-02 02:00:00 32
2 761 2015-01-03 02:00:00 40
3 317 2015-01-04 02:00:00 16
4 538 2015-01-05 02:00:00 20
print(ndf.head(5))
end start id value
0 2016-02-20 2016-02-13 569 221
1 2016-03-01 2016-02-23 28 216
2 2016-03-24 2016-03-17 152 258
3 2016-03-31 2016-03-24 892 265
4 2016-04-02 2016-03-26 606 244

You can calculate a weekly summary with the following code. The code below is based on Monday.
import pandas as pd
import random
dates = pd.date_range('20150101 020000',periods=1000)
df = pd.DataFrame({'_id': random.choice(range(0, 1000)),
'time_stamp': dates,
'value': random.choice(range(2,60))
})
df['day_of_week'] = df['time_stamp'].dt.weekday_name
df['start'] = np.where(df["day_of_week"]=="Monday", 1, 0)
df['week'] = df["start"].cumsum()
# It is based on Monday.
df.head(20)
# Out[109]:
# _id time_stamp value day_of_week start week
# 0 396 2015-01-01 02:00:00 59 Thursday 0 0
# 1 396 2015-01-02 02:00:00 59 Friday 0 0
# 2 396 2015-01-03 02:00:00 59 Saturday 0 0
# 3 396 2015-01-04 02:00:00 59 Sunday 0 0
# 4 396 2015-01-05 02:00:00 59 Monday 1 1
# 5 396 2015-01-06 02:00:00 59 Tuesday 0 1
# 6 396 2015-01-07 02:00:00 59 Wednesday 0 1
# 7 396 2015-01-08 02:00:00 59 Thursday 0 1
# 8 396 2015-01-09 02:00:00 59 Friday 0 1
# 9 396 2015-01-10 02:00:00 59 Saturday 0 1
# 10 396 2015-01-11 02:00:00 59 Sunday 0 1
# 11 396 2015-01-12 02:00:00 59 Monday 1 2
# 12 396 2015-01-13 02:00:00 59 Tuesday 0 2
# 13 396 2015-01-14 02:00:00 59 Wednesday 0 2
# 14 396 2015-01-15 02:00:00 59 Thursday 0 2
# 15 396 2015-01-16 02:00:00 59 Friday 0 2
# 16 396 2015-01-17 02:00:00 59 Saturday 0 2
# 17 396 2015-01-18 02:00:00 59 Sunday 0 2
# 18 396 2015-01-19 02:00:00 59 Monday 1 3
# 19 396 2015-01-20 02:00:00 59 Tuesday 0 3
aggfunc = {"time_stamp": [np.min, np.max], "value": [np.sum]}
df2 = df.groupby("week", as_index=False).agg(aggfunc)
df2.columns = ["week", "start_date", "end_date", "weekly_sum"]
df2.iloc[58:61]
# Out[110]:
# week start_date end_date weekly_sum
# 58 58 2016-02-08 02:00:00 2016-02-14 02:00:00 413
# 59 59 2016-02-15 02:00:00 2016-02-21 02:00:00 413
# 60 60 2016-02-22 02:00:00 2016-02-28 02:00:00 413

Grouping daily data by month in python/pandas while firstly grouping by user id

I have the table below in a Pandas dataframe:
date user_id whole_cost cost1
02/10/2012 00:00:00 1 1790 12
07/10/2012 00:00:00 1 364 15
30/01/2013 00:00:00 1 280 10
02/02/2013 00:00:00 1 259 24
05/03/2013 00:00:00 1 201 39
02/10/2012 00:00:00 3 623 1
07/12/2012 00:00:00 3 90 0
30/01/2013 00:00:00 3 312 90
02/02/2013 00:00:00 5 359 45
05/03/2013 00:00:00 5 301 34
02/02/2013 00:00:00 5 359 1
05/03/2013 00:00:00 5 801 12
..
The table was extracted from a csv file using the following query :
import pandas as pd
newnames = ['date','user_id', 'whole_cost', 'cost1']
df = pd.read_csv('expenses.csv', names = newnames, index_col = 'date')
I have to analyse the profile of my users and for this purpose:
I would like to group (for each user - they are thousands) queries by month summing the query whole_cost for the entire month e.g. if user_id=1 was has a whole cost of 1790 on 02/10/2012 with cost1 12 and on the 07/10/2012 with whole cost 364, then it should have an entry in the new table of 2154 (as the whole cost) on 31/10/2012 (end of the month end-point representing the month - all dates in the transformed table will be month ends representing the whole month to which they relate).

In 0.14 you'll be able to groupby monthly and another column at the same time:
In [11]: df
Out[11]:
user_id whole_cost cost1
2012-10-02 1 1790 12
2012-10-07 1 364 15
2013-01-30 1 280 10
2013-02-02 1 259 24
2013-03-05 1 201 39
2012-10-02 3 623 1
2012-12-07 3 90 0
2013-01-30 3 312 90
2013-02-02 5 359 45
2013-03-05 5 301 34
2013-02-02 5 359 1
2013-03-05 5 801 12
In [12]: df1 = df.sort_index() # requires sorted DatetimeIndex
In [13]: df1.groupby([pd.TimeGrouper(freq='M'), 'user_id'])['whole_cost'].sum()
Out[13]:
user_id
2012-10-31 1 2154
3 623
2012-12-31 3 90
2013-01-31 1 280
3 312
2013-02-28 1 259
5 718
2013-03-31 1 201
5 1102
Name: whole_cost, dtype: int64
until 0.14 I think you're stuck with doing two groupbys:
In [14]: g = df.groupby('user_id')['whole_cost']
In [15]: g.resample('M', how='sum').dropna()
Out[15]:
user_id
1 2012-10-31 2154
2013-01-31 280
2013-02-28 259
2013-03-31 201
3 2012-10-31 623
2012-12-31 90
2013-01-31 312
5 2013-02-28 718
2013-03-31 1102
dtype: float64

With timegrouper getting deprecated, you can replace it with Grouper to get the same results
df.groupby(['user_id', pd.Grouper(key='date', freq='M')]).agg({'whole_cost':sum})
df.groupby(['user_id', df['date'].dt.dayofweek]).agg({'whole_cost':sum})