I have the following dataset and would like to group by Name and Month and Year. I would like to calculate how many times the name appears in every month of the year:
Date Name
2019-11-10 18:59:31+00:00 A
2020-11-07 18:59:31+00:00 A
2021-05-10 18:59:31+00:00 B
2020-11-09 18:59:31+00:00 C
2021-05-01 18:59:31+00:00 B
2020-12-10 18:59:31+00:00 C
2019-12-10 18:59:31+00:00 B
I do not know how exactly the result would be, but I expect something similar to this so I can then make a graph:
2019-11 A 1
2020-11 A 1
2021-05 B 2
2020-11 C 1
2020-12 C 1
2019-12 B 1
I have tried the following method:
df.groupby(pd.Grouper(key='Date',freq='1M')).groupby('Name').count()
TypeError: Only valid with DatetimeIndex, TimedeltaIndex or PeriodIndex, but got an instance of 'Index'
Try:
df.groupby([df['Date'].dt.strftime('%Y-%m'),'Name'])['Name'].count().rename('count').reset_index()
Output:
Date Name count
0 2019-11 A 1
1 2019-12 B 1
2 2020-11 A 1
3 2020-11 C 1
4 2020-12 C 1
5 2021-05 B 2
Is Date the first column to the DataFrame, it should be
df = df.reset_index()
df.groupby(pd.Grouper(key='Date',freq='1M')).groupby('Name').count()
Try converting Date to year/month using to_period('M'):
df.groupby([df.Date.dt.to_period('M'),df.Name]).agg(count = ('Date','count')).reset_index()
Result is:
Date Name count
0 2019-11 A 1
1 2019-12 B 1
2 2020-11 A 1
3 2020-11 C 1
4 2020-12 C 1
5 2021-05 B 2
Related
I have the following dataframe:
A B start_date end_date id
0 1 2 2022-01-01 2022-01-10 1
1 2 2 2022-02-02 2022-02-05 2
2 1 2 2022-01-11 2022-01-15 3
3 2 2 2022-02-06 2022-02-10 4
4 2 2 2022-02-11 2022-02-15 5
5 2 3 2022-01-14 2022-01-17 6
6 2 3 2022-01-19 2022-01-22 7
There are several records that follow one after the other. For example, rows 1 and 3. Row 3 has the same values A and B and starts the next day when row 1 ends. I want to compress this dataframe into the following form:
A B start_date end_date id
0 1 2 2022-01-01 2022-01-15 1
1 2 2 2022-02-02 2022-02-15 2
2 2 3 2022-01-14 2022-01-17 3
3 2 3 2022-01-19 2022-01-22 4
That is, I save one record where the difference between the start_date of the next record and the end_date of the previous one is 1 day. In this case, end_date is changed to end_date for the last record inside such a sequence.
You can use a custom grouper to join the successive dates per group:
df[['start_date', 'end_date']] = df[['start_date', 'end_date']].apply(pd.to_datetime)
m = (df['start_date'].sub(df.groupby(['A', 'B'])
['end_date'].shift()
.add(pd.Timedelta('1d'))
).ne('0')
.groupby([df['A'], df['B']]).cumsum()
)
out = (df
.groupby(['A', 'B', m], as_index=False)
.agg({'start_date': 'first', 'end_date': 'last'})
.assign(id=lambda d: range(1, len(d)+1))
)
Output:
A B start_date end_date id
0 1 2 2022-01-01 2022-01-15 1
1 2 2 2022-02-02 2022-02-15 2
2 2 3 2022-01-14 2022-01-17 3
3 2 3 2022-01-19 2022-01-22 4
def function1(dd:pd.DataFrame):
col1=dd.start_date-dd.end_date.shift()
dd1=dd.assign(col1=col1.ne("1 days").cumsum())
return dd1.groupby("col1").agg(start_date=("start_date",min),end_date=("end_date",max))
df1.groupby(["A","B"]).apply(function1).reset_index().assign(id=lambda dd:dd.index+1)
out
A B col1 start_date end_date id
0 1 2 1 2022-01-01 2022-01-15 1
1 2 2 1 2022-02-02 2022-02-15 2
2 2 3 1 2022-01-14 2022-01-17 3
3 2 3 2 2022-01-19 2022-01-22 4
I have the following data-frame:
ID date X
0 A 2021-12-15 7
1 A 2022-01-30 6
2 A 2022-02-15 2
3 B 2022-01-30 2
4 B 2022-02-15 2
5 B 2022-02-18 7
6 C 2021-12-01 7
7 C 2021-12-15 4
8 C 2022-01-30 2
9 C 2022-02-15 7
10 D 2021-12-16 5
11 D 2022-01-30 4
12 D 2022-03-15 9
I want to keep the observations for those IDs who first showed up in week, say, 51 of the year (I would like to change this parameter in the future).
For example, IDs A and D showed up first in week 51 in the data, B didn't, C showed up in week 51, but not for the first time.
So I want to keep in this example only the data pertaining to A and D.
Filter if week match variable week and it is first time by ID in DataFrame by Series.duplicated, then get ID values:
week = 50
df['date'] = pd.to_datetime(df['date'])
s = df.loc[df['date'].dt.isocalendar().week.eq(week) & ~df['ID'].duplicated(), 'ID']
Or:
df1 = df.drop_duplicates(['ID'])
s = df1.loc[df1['date'].dt.isocalendar().week.eq(week) ,'ID']
print (s)
0 A
10 D
Name: ID, dtype: object
Last filter by ID with Series.isin and boolean indexing:
df = df[df['ID'].isin(s)]
print (df)
ID date X
0 A 2021-12-15 7
1 A 2022-01-30 6
2 A 2022-02-15 2
10 D 2021-12-16 5
11 D 2022-01-30 4
12 D 2022-03-15 9
By having the following table, how can I count the days by ID?
without use of for or any loop because it's large size data.
ID Date
a 01/01/2020
a 05/01/2020
a 08/01/2020
a 10/01/2020
b 05/05/2020
b 08/05/2020
b 12/05/2020
c 08/08/2020
c 22/08/2020
to have this result
ID Date Days Evolved Since Inicial date
a 01/01/2020 1
a 05/01/2020 4
a 08/01/2020 7
a 10/01/2020 9
b 05/05/2020 1
b 08/05/2020 3
b 12/05/2020 7
c 08/08/2020 1
c 22/08/2020 14
Use GroupBy.transform with GroupBy.first for first values to new column, so possible subtract. Then if not duplicated datetimes is possible replace 0:
df['new']=df['Date'].sub(df.groupby("ID")['Date'].transform('first')).dt.days.replace(0, 1)
print (df)
ID Date new
0 a 2020-01-01 1
1 a 2020-01-05 4
2 a 2020-01-08 7
3 a 2020-01-10 9
4 b 2020-05-05 1
5 b 2020-05-08 3
6 b 2020-05-12 7
7 c 2020-08-08 1
8 c 2020-08-22 14
Or set 1 for first value of group by Series.where and Series.duplicated:
df['new'] = (df['Date'].sub(df.groupby("ID")['Date'].transform('first'))
.dt.days.where(df['ID'].duplicated(), 1))
print (df)
ID Date new
0 a 2020-01-01 1
1 a 2020-01-05 4
2 a 2020-01-08 7
3 a 2020-01-10 9
4 b 2020-05-05 1
5 b 2020-05-08 3
6 b 2020-05-12 7
7 c 2020-08-08 1
8 c 2020-08-22 14
You could do something like (df your dataframe):
def days_evolved(sdf):
sdf["Days_evolved"] = sdf.Date - sdf.Date.iat[0]
sdf["Days_evolved"].iat[0] = pd.Timedelta(days=1)
return sdf
df = df.groupby("ID", as_index=False, sort=False).apply(days_evolved)
Result for the sample:
ID Date Days_evolved
0 a 2020-01-01 1 days
1 a 2020-01-05 4 days
2 a 2020-01-08 7 days
3 a 2020-01-10 9 days
4 b 2020-05-05 1 days
5 b 2020-05-08 3 days
6 b 2020-05-12 7 days
7 c 2020-08-08 1 days
8 c 2020-08-22 14 days
If you want int instead of pd.Timedelta then do
df["Days_evolved"] = df["Days_evolved"].dt.days
at the end.
I want to get the sum of values for next 7 days of a column
my dataframe :
date value
0 2021-04-29 1
1 2021-05-03 2
2 2021-05-06 1
3 2021-05-15 1
4 2021-05-17 2
5 2021-05-18 1
6 2021-05-21 2
7 2021-05-22 5
8 2021-05-24 4
i tried to make a new column that contains date 7 days from current date
df['temp'] = df['date'] + timedelta(days=7)
then calculate value between date range :
df['next_7days'] = df[(df.date > df.date) & (df.date <= df.temp)].value.sum()
But this gives me answer as all 0.
intended result:
date value next_7days
0 2021-04-29 1 3
1 2021-05-03 2 1
2 2021-05-06 1 0
3 2021-05-15 1 10
4 2021-05-17 2 12
5 2021-05-18 1 11
6 2021-05-21 2 9
7 2021-05-22 5 4
8 2021-05-24 4 0
The method iam using currently is quite tedious, are their any better methods to get the intended result.
With a list comprehension:
tomorrow_dates = df.date + pd.Timedelta("1 day")
next_week_dates = df.date + pd.Timedelta("7 days")
df["next_7days"] = [df.value[df.date.between(tomorrow, next_week)].sum()
for tomorrow, next_week in zip(tomorrow_dates, next_week_dates)]
where we first define tomorrow and next week's dates and store them. Then zip them together and use between of pd.Series to get a boolean series if the date is indeed between the desired range. Then using boolean indexing to get the actual values and sum them. Do this for each date pair.
to get
date value next_7days
0 2021-04-29 1 3
1 2021-05-03 2 1
2 2021-05-06 1 0
3 2021-05-15 1 10
4 2021-05-17 2 12
5 2021-05-18 1 11
6 2021-05-21 2 9
7 2021-05-22 5 4
8 2021-05-24 4 0
My pandas version is 0.23.4.
I tried to run this code:
df['date_time'] = pd.to_datetime(df[['year','month','day','hour_scheduled_departure','minute_scheduled_departure']])
and the following error appeared:
extra keys have been passed to the datetime assemblage: [hour_scheduled_departure, minute_scheduled_departure]
Any ideas of how to get the job done by pd.to_datetime?
#anky_91
In this image an extract of first 10 rows is presented. First column [int32]: year; Second column[int32]: month; Third column[int32]: day; Fourth column[object]: hour; Fifth column[object]: minute. The length of objects is 2.
Another solution:
>>pd.concat([df.A,pd.to_datetime(pd.Series(df[df.columns[1:]].fillna('').values.tolist(),name='Date').map(lambda x: '0'.join(map(str,x))))],axis=1)
A Date
0 a 2002-07-01 05:07:00
1 b 2002-08-03 03:08:00
2 c 2002-09-05 06:09:00
3 d 2002-04-07 09:04:00
4 e 2002-02-01 02:02:00
5 f 2002-03-05 04:03:00
For the example you have added as image (i have skipped the last 3 columns due to save time)
>>df.month=df.month.map("{:02}".format)
>>df.day = df.day.map("{:02}".format)
>>pd.concat([df.A,pd.to_datetime(pd.Series(df[df.columns[1:]].fillna('').values.tolist(),name='Date').map(lambda x: ''.join(map(str,x))))],axis=1)
A Date
0 a 2015-01-01 00:05:00
1 b 2015-01-01 00:01:00
2 c 2015-01-01 00:02:00
3 d 2015-01-01 00:02:00
4 e 2015-01-01 00:25:00
5 f 2015-01-01 00:25:00
You can use rename to columns, so possible use pandas.to_datetime with columns year, month, day, hour, minute:
df = pd.DataFrame({
'A':list('abcdef'),
'year':[2002,2002,2002,2002,2002,2002],
'month':[7,8,9,4,2,3],
'day':[1,3,5,7,1,5],
'hour_scheduled_departure':[5,3,6,9,2,4],
'minute_scheduled_departure':[7,8,9,4,2,3]
})
print (df)
A year month day hour_scheduled_departure minute_scheduled_departure
0 a 2002 7 1 5 7
1 b 2002 8 3 3 8
2 c 2002 9 5 6 9
3 d 2002 4 7 9 4
4 e 2002 2 1 2 2
5 f 2002 3 5 4 3
cols = ['year','month','day','hour_scheduled_departure','minute_scheduled_departure']
d = {'hour_scheduled_departure':'hour','minute_scheduled_departure':'minute'}
df['date_time'] = pd.to_datetime(df[cols].rename(columns=d))
#if necessary remove columns
df = df.drop(cols, axis=1)
print (df)
A date_time
0 a 2002-07-01 05:07:00
1 b 2002-08-03 03:08:00
2 c 2002-09-05 06:09:00
3 d 2002-04-07 09:04:00
4 e 2002-02-01 02:02:00
5 f 2002-03-05 04:03:00
Detail:
print (df[cols].rename(columns=d))
year month day hour minute
0 2002 7 1 5 7
1 2002 8 3 3 8
2 2002 9 5 6 9
3 2002 4 7 9 4
4 2002 2 1 2 2
5 2002 3 5 4 3