I'm not 100% if using apply + functools.reduce is the best approach for this problem, but I'm not sure exactly if multi-indices can be leveraged to accomplish this goal.
Background
The data is a list of activities performed by accounts.
user_id - the user's ID
activity - the string that represents the activity
performed_at - the timestamp when the activity was completed at
The Goal
To calculate the time spent between 2 statuses. An account can look like this:
user_id
activity
performed_at
1
activated
2020-01-01
1
deactivated
2020-01-02
1
activated
2020-01-03
In this example, the user was deactivated from January 1st to January 2nd, so the total "Time Deactivated" for this account would be 1 day.
Resulting Dataframe
Here's an example of the output I'm trying to achieve. The time_spent_deactivated column is just the addition of all deactivation periods on all accounts grouped by account.
user_id
time_spent_deactivated
1
24 hours
2
15 hours
3
72 hours
My Attempt
I'm trying to leverage .apply with the .groupby on the user_id, but I'm stuck at the point of calculating the total time spent in the deactivated state:
def calculate_deactivation_time(activities):
# reduce the given dataframe here
# this is totally ActiveRecord & JS inspired but it's the easiest way for me to describe how I expect to solve this
return activities.reduce(sum, current_row):
if current_row['activity'] == 'deactivated':
# find next "activated" activity and compute the delta
reactivated_row = activities.after(current_row).where(activity, '=', 'activated')
return sum + (reactivated_row['performed_at'] - current_row['performed_at'])
grouped = activities.groupby('user_id')
grouped.apply(calculate_deactivation_time)
Is there a better approach to doing this? I tried to use functools.reduce to compute the total time spent deactivated, but it doesn't support dataframes out of the box.
I have given it some more thought and I think this is what you are looking for.
import pandas as pd
import numpy as np
def myfunc(x):
df = x
# Shift columns activity_start and performed_at_start
df['performed_at_end'] = df['performed_at_start'].shift(-1)
df['activity_end'] = df['activity_start'].shift(-1)
# Make combinations of activity start-end
df['activity_start_end'] = df['activity_start']+'-'+df['activity_end']
# Take only those that start with deactivated, end with activated
df = df[df['activity_start_end']=='deactivated-activated']
# Drop all that don't have performed_at_end date (does not exist)
df = df[~pd.isna(df['performed_at_end'])]
# Compute time difference in days, then return sum of all delta's
df['delta'] = (df['performed_at_end']-df['performed_at_start'])/np.timedelta64(1,'D')
return df['delta'].sum()
# Example dataframe
df = pd.DataFrame({'UserId': [1]*4+[2]*2+[3]*1,
'activity_start': ['activated', 'deactivated', 'activated', 'deactivated', 'deactivated', 'activated', 'activated'],
'performed_at_start': [pd.Timestamp(2020,1,1), pd.Timestamp(2020,1,2), pd.Timestamp(2020,1,6), pd.Timestamp(2020,1,8),
pd.Timestamp(2020,1,1), pd.Timestamp(2020,1,3), pd.Timestamp(2020,1,1)]})
# Show dataframe
print(df)
UserId activity_start performed_at_start
0 1 activated 2020-01-01
1 1 deactivated 2020-01-02
2 1 activated 2020-01-06
3 1 deactivated 2020-01-08
4 2 deactivated 2020-01-01
5 2 activated 2020-01-03
6 3 activated 2020-01-01
# Compute result
res = (
df.groupby(by='UserId')
.apply(lambda x: myfunc(x)).reset_index(drop=False)
)
res.columns = ['UserId', 'time_spent_deactivated']
# Show result
print(res)
UserId time_spent_deactivated
0 1 4.0
1 2 2.0
2 3 0.0
Related
I have a simple definition which finds the third friday of the month. I use this function to populate the dataframe for the third fridays and that part works fine.
The trouble I'm having is finding the third friday for an expiration_date that doesn't fall on a third friday.
This is my code simplified:
import pandas as pd
def is_third_friday(d):
return d.weekday() == 4 and 15 <= d.day <= 21
x = ['09/23/2022','09/26/2022','09/28/2022','09/30/2022','10/3/2022','10/5/2022',
'10/7/2022','10/10/2022','10/12/2022','10/14/2022','10/17/2022','10/19/2022','10/21/2022',
'10/24/2022','10/26/2022','10/28/2022','11/4/2022','11/18/2022','12/16/2022','12/30/2022',
'01/20/2023','03/17/2023','03/31/2023','06/16/2023','06/30/2023','09/15/2023','12/15/2023',
'01/19/2024','06/21/2024','12/20/2024','01/17/2025']
df = pd.DataFrame(x)
df.rename( columns={0 :'expiration_date'}, inplace=True )
df['expiration_date']= pd.to_datetime(df['expiration_date'])
expiration_date = df['expiration_date']
df["is_third_friday"] = [is_third_friday(x) for x in expiration_date]
third_fridays = df.loc[df['is_third_friday'] == True]
df["current_monthly_exp"] = third_fridays['expiration_date'].min()
df["monthly_exp"] = third_fridays[['expiration_date']]
df.to_csv(path_or_buf = f'C:/Data/Date Dataframe.csv',index=False)
What I'm looking for is any expiration_date that is prior to the monthly expire, I want to populate the dataframe with that monthly expire. If it's past the monthly expire date I want to populate the dataframe with the following monthly expire.
I thought I'd be able to use a new dataframe with only the monthly expires as a lookup table and do a timedelta, but when you look at 4/21/2023 and 7/21/2023 these dates don't exist in that dataframe.
This is my current output:
This is the output I'm seeking:
I was thinking I could handle this problem with something like:
date_df["monthly_exp"][0][::-1].expanding().min()[::-1]
But, it wouldn't solve for the 4/21/2023 and 7/21/2023 problem. Additionally, Pandas won't let you do this in a datetime dataframe.
>>> df = pd.DataFrame([1, nan,2,nan,nan,nan,4])
>>> df
0
0 1.0
1 NaN
2 2.0
3 NaN
4 NaN
5 NaN
6 4.0
>>> df["b"] = df[0][::-1].expanding().min()[::-1]
>>> df
0 b
0 1.0 1.0
1 NaN 2.0
2 2.0 2.0
3 NaN 4.0
4 NaN 4.0
5 NaN 4.0
6 4.0 4.0
I've also tried something like the following in many different forms with little luck:
if df['is_third_friday'].any() == True:
df["monthly_exp"] = third_fridays[['expiration_date']]
else:
df["monthly_exp"] = third_fridays[['expiration_date']].shift(third_fridays)
Any suggestions to get me in the right direction would be appreciated. I've been stuck on this problem for sometime.
You could add these additional lines of code (to replace df["monthly_exp"] = third_fridays[['expiration_date']]:
# DataFrame of fridays from minimum expiration_date to 30 days after last
fri_3s = pd.DataFrame(pd.date_range(df["expiration_date"].min(),
df["expiration_date"].max()+pd.tseries.offsets.Day(30),
freq="W-FRI"),
columns=["monthly_exp"])
# only keep those that are between 15th and 21st (as your function did)
fri_3s = fri_3s[fri_3s.monthly_exp.dt.day.between(15, 21)]
# merge_asof to get next third friday
df = pd.merge_asof(df, fri_3s, left_on="expiration_date", right_on="monthly_exp", direction="forward")
This creates a second DataFrame with the 3rd Fridays, and then by merging with merge_asof returns the next of these from the expiration_date.
And to simplify your date_df["monthly_exp"][0][::-1].expanding().min()[::-1] and use it for datetime, you could instead write df["monthly_exp"].bfill() (which backward fills). As you mentioned, this will only include Fridays that exist in your DataFrame already, so creating a list of the possible Fridays might be the easiest way.
I have a large dataset of test results where I have a columns to represent the date a test was completed and number of hours it took to complete the test i.e.
df = pd.DataFrame({'Completed':['21/03/2020','22/03/2020','21/03/2020','24/03/2020','24/03/2020',], 'Hours_taken':[23,32,8,73,41]})
I have a months worth of test data and the tests can take anywhere from a couple of hours to a couple of days. I want to try and work out, for each day, what percentage of tests fall within the ranges of 24hrs/48hrs/72hrs ect. to complete, up to the percentage of tests that took longer than a week.
I've been able to work it out generally without taking the dates into account like so:
Lab_tests['one-day'] = Lab_tests['hours'].between(0,24)
Lab_tests['two-day'] = Lab_tests['hours'].between(24,48)
Lab_tests['GreaterThanWeek'] = Lab_tests['hours'] >168
one = Lab_tests['1-day'].value_counts().loc[True]
two = Lab_tests['two-day'].value_counts().loc[True]
eight = Lab_tests['GreaterThanWeek'].value_counts().loc[True]
print(one/10407 * 100)
print(two/10407 * 100)
print(eight/10407 * 100)
Ideally I'd like to represent the percentages in another dataset where the rows represent the dates and the columns represent the data ranges. But I can't work out how to take what I've done and modify it to get these percentages for each date. Is this possible to do in pandas?
This question, Counting qualitative values based on the date range in Pandas is quite similar but the fact that I'm counting the occurrences in specified ranges is throwing me off and I haven't been able to get a solution out of it.
Bonus Question
I'm sure you've noticed my current code is not the most elegant thing in the world, is the a cleaner way to do what I've done above, as I'm doing that for every data range that I want?
Edit:
So the Output for the sample data given would look like so:
df = pd.DataFrame({'1-day':[100,0,0,0], '2-day':[0,100,0,50],'3-day':[0,0,0,0],'4-day':[0,0,0,50]},index=['21/03/2020','22/03/2020','23/03/2020','24/03/2020'])
You're almost there. You just need to do a few final steps:
First, cast your bools to ints, so that you can sum them.
Lab_tests['one-day'] = Lab_tests['hours'].between(0,24).astype(int)
Lab_tests['two-day'] = Lab_tests['hours'].between(24,48).astype(int)
Lab_tests['GreaterThanWeek'] = (Lab_tests['hours'] > 168).astype(int)
Completed hours one-day two-day GreaterThanWeek
0 21/03/2020 23 1 0 0
1 22/03/2020 32 0 1 0
2 21/03/2020 8 1 0 0
3 24/03/2020 73 0 0 0
4 24/03/2020 41 0 1 0
Then, drop the hours column and roll the rest up to the level of Completed:
Lab_tests['one-day'] = Lab_tests['hours'].between(0,24).astype(int)
Lab_tests['two-day'] = Lab_tests['hours'].between(24,48).astype(int)
Lab_tests['GreaterThanWeek'] = (Lab_tests['hours'] > 168).astype(int)
Lab_tests.drop('hours', axis=1).groupby('Completed').sum()
one-day two-day GreaterThanWeek
Completed
21/03/2020 2 0 0
22/03/2020 0 1 0
24/03/2020 0 1 0
EDIT: To get to percent, you just need to divide each column by the sum of all three. You can sum columns by defining the axis of the sum:
...
daily_totals = Lab_tests.drop('hours', axis=1).groupby('Completed').sum()
daily_totals.sum(axis=1)
Completed
21/03/2020 2
22/03/2020 1
24/03/2020 1
dtype: int64
Then divide the daily totals dataframe by the column-wise sum of the daily totals (again, we use axis to define whether each value of the series will be the divisor for a row or a column.):
daily_totals.div(daily_totals.sum(axis=1), axis=0)
one-day two-day GreaterThanWeek
Completed
21/03/2020 1.0 0.0 0.0
22/03/2020 0.0 1.0 0.0
24/03/2020 0.0 1.0 0.0
Given a pandas dataframe in the following format:
toy = pd.DataFrame({
'id': [1,2,3,
1,2,3,
1,2,3],
'date': ['2015-05-13', '2015-05-13', '2015-05-13',
'2016-02-12', '2016-02-12', '2016-02-12',
'2018-07-23', '2018-07-23', '2018-07-23'],
'my_metric': [395, 634, 165,
144, 305, 293,
23, 395, 242]
})
# Make sure 'date' has datetime format
toy.date = pd.to_datetime(toy.date)
The my_metric column contains some (random) metric I wish to compute a time-dependent moving average of, conditional on the column id
and within some specified time interval that I specify myself. I will refer to this time interval as the "lookback time"; which could be 5 minutes
or 2 years. To determine which observations that are to be included in the lookback calculation, we use the date column (which could be the index if you prefer).
To my frustration, I have discovered that such a procedure is not easily performed using pandas builtins, since I need to perform the calculation conditionally
on id and at the same time the calculation should only be made on observations within the lookback time (checked using the date column). Hence, the output dataframe should consist of one row for each id-date combination, with the my_metric column now being the average of all observations that is contatined within the lookback time (e.g. 2 years, including today's date).
For clarity, I have included a figure with the desired output format (apologies for the oversized figure) when using a 2-year lookback time:
I have a solution but it does not make use of specific pandas built-in functions and is likely sub-optimal (combination of list comprehension and a single for-loop). The solution I am looking for will not make use of a for-loop, and is thus more scalable/efficient/fast.
Thank you!
Calculating lookback time: (Current_year - 2 years)
from dateutil.relativedelta import relativedelta
from dateutil import parser
import datetime
In [1691]: dt = '2018-01-01'
In [1695]: dt = parser.parse(dt)
In [1696]: lookback_time = dt - relativedelta(years=2)
Now, filter the dataframe on lookback time and calculate rolling average
In [1722]: toy['new_metric'] = ((toy.my_metric + toy[toy.date > lookback_time].groupby('id')['my_metric'].shift(1))/2).fillna(toy.my_metric)
In [1674]: toy.sort_values('id')
Out[1674]:
date id my_metric new_metric
0 2015-05-13 1 395 395.0
3 2016-02-12 1 144 144.0
6 2018-07-23 1 23 83.5
1 2015-05-13 2 634 634.0
4 2016-02-12 2 305 305.0
7 2018-07-23 2 395 350.0
2 2015-05-13 3 165 165.0
5 2016-02-12 3 293 293.0
8 2018-07-23 3 242 267.5
So, after some tinkering I found an answer that will generalize adequately. I used a slightly different 'toy' dataframe (slightly more relevant to my case). For completeness sake, here is the data:
Consider now the following code:
# Define a custom function which groups by time (using the index)
def rolling_average(x, dt):
xt = x.sort_index().groupby(lambda x: x.time()).rolling(window=dt).mean()
xt.index = xt.index.droplevel(0)
return xt
dt='730D' # rolling average window: 730 days = 2 years
# Group by the 'id' column
g = toy.groupby('id')
# Apply the custom function
df = g.apply(rolling_average, dt=dt)
# Massage the data to appropriate format
df.index = df.index.droplevel(0)
df = df.reset_index().drop_duplicates(keep='last', subset=['id', 'date'])
The result is as expected:
I have a program that ideally measures the temperature every second. However, in reality this does not happen. Sometimes, it skips a second or it breaks down for 400 seconds and then decides to start recording again. This leaves gaps in my 2-by-n dataframe, where ideally n = 86400 (the amount of seconds in a day). I want to apply some sort of moving/rolling average to it to get a nicer plot, but if I do that to the "raw" datafiles, the amount of data points becomes less. This is shown here, watch the x-axis. I know the "nice data" doesn't look nice yet; I'm just playing with some values.
So, I want to implement a data cleaning method, which adds data to the dataframe. I thought about it, but don't know how to implement it. I thought of it as follows:
If the index is not equal to the time, then we need to add a number, at time = index. If this gap is only 1 value, then the average of the previous number and the next number will do for me. But if it is bigger, say 100 seconds are missing, then a linear function needs to be made, which will increase or decrease the value steadily.
So I guess a training set could be like this:
index time temp
0 0 20.10
1 1 20.20
2 2 20.20
3 4 20.10
4 100 22.30
Here, I would like to get a value for index 3, time 3 and the values missing between time = 4 and time = 100. I'm sorry about my formatting skills, I hope it is clear.
How would I go about programming this?
Use merge with complete time column and then interpolate:
# Create your table
time = np.array([e for e in np.arange(20) if np.random.uniform() > 0.6])
temp = np.random.uniform(20, 25, size=len(time))
temps = pd.DataFrame([time, temp]).T
temps.columns = ['time', 'temperature']
>>> temps
time temperature
0 4.0 21.662352
1 10.0 20.904659
2 15.0 20.345858
3 18.0 24.787389
4 19.0 20.719487
The above is a random table generated with missing time data.
# modify it
filled = pd.Series(np.arange(temps.iloc[0,0], temps.iloc[-1, 0]+1))
filled = filled.to_frame()
filled.columns = ['time'] # Create a fully filled time column
merged = pd.merge(filled, temps, on='time', how='left') # merge it with original, time without temperature will be null
merged.temperature = merged.temperature.interpolate() # fill nulls linearly.
# Alternatively, use reindex, this does the same thing.
final = temps.set_index('time').reindex(np.arange(temps.time.min(),temps.time.max()+1)).reset_index()
final.temperature = final.temperature.interpolate()
>>> merged # or final
time temperature
0 4.0 21.662352
1 5.0 21.536070
2 6.0 21.409788
3 7.0 21.283505
4 8.0 21.157223
5 9.0 21.030941
6 10.0 20.904659
7 11.0 20.792898
8 12.0 20.681138
9 13.0 20.569378
10 14.0 20.457618
11 15.0 20.345858
12 16.0 21.826368
13 17.0 23.306879
14 18.0 24.787389
15 19.0 20.719487
First you can set the second values to actual time values as such:
df.index = pd.to_datetime(df['time'], unit='s')
After which you can use pandas' built-in time series operations to resample and fill in the missing values:
df = df.resample('s').interpolate('time')
Optionally, if you still want to do some smoothing you can use the following operation for that:
df.rolling(5, center=True, win_type='hann').mean()
Which will smooth with a 5 element wide Hanning window. Note: any window-based smoothing will cost you value points at the edges.
Now your dataframe will have datetimes (including date) as index. This is required for the resample method. If you want to lose the date, you can simply use:
df.index = df.index.time
I would like to calculate buy and sell signals for stocks based on simple moving average (SMA) crossing. A buy signal should be given as soon as the SMA_short is higher than the SMA_long (i.e., SMA_difference > 0). In order to avoid that the position is sold too quickly, I would like to have a sell signal only once the SMA_short has moved beyond the cross considerably (i.e., SMA_difference < -1), and, importantly, even if this would be for longer than one day.
I managed, by this help to implement it (see below):
Buy and sell signals are indicated by in and out.
Column Position takes first the buy_limit into account.
In Position_extended an in is then set for all the cases where the SMA_short just crossed through the SMA_long (SMA_short < SMA_long) but SMA_short > -1. For this it is taking the Position extended of i-1 into account in case the crossing was more than one day ago but SMA_short remained: 0 > SMA_short > -1.
Python code
import pandas as pd
import numpy as np
index = pd.date_range('20180101', periods=6)
df = pd.DataFrame(index=index)
df["SMA_short"] = [9,10,11,10,10,9]
df["SMA_long"] = 10
df["SMA_difference"] = df["SMA_short"] - df["SMA_long"]
buy_limit = 0
sell_limit = -1
df["Position"] = np.where((df["SMA_difference"] > buy_limit),"in","out")
df["Position_extended"] = df["Position"]
for i in range(1,len(df)):
df.loc[index[i],"Position_extended"] = \
np.where((df.loc[index[i], "SMA_difference"] > sell_limit) \
& (df.loc[index[i-1],"Position_extended"] == "in") \
,"in",df.loc[index[i],'Position'])
print df
The result is:
SMA_short SMA_long SMA_difference Position Position_extended
2018-01-01 9 10 -1 out out
2018-01-02 10 10 0 out out
2018-01-03 11 10 1 in in
2018-01-04 10 10 0 out in
2018-01-05 10 10 0 out in
2018-01-06 9 10 -1 out out
The code works, however, it makes use of a for loop, which slows down the script considerably and becomes inapplicable in the larger context of this analysis. As SMA crossing is such a highly used tool, I was wondering whether somebody could see a more elegant and faster solution for this.
Essentially you are trying to get rid of the ambivalent zero entries by propagating the last non-zero value. Similar to a zero-order hold. You can do so my first replacing the zero values by NaNs and then interpolating over the latter using ffill.
import pandas as pd
import numpy as np
index = pd.date_range('20180101', periods=6)
df = pd.DataFrame(index=index)
df["SMA_short"] = [9,10,11,10,10,9]
df["SMA_long"] = 10
df["SMA_difference"] = df["SMA_short"] - df["SMA_long"]
buy_limit = 0
sell_limit = -1
df["ZOH"] = df["SMA_difference"].replace(0,np.nan).ffill()
df["Position"] = np.where((df["ZOH"] > buy_limit),"in","out")
print df
results in:
SMA_short SMA_long SMA_difference ZOH Position
2018-01-01 9 10 -1 -1.0 out
2018-01-02 10 10 0 -1.0 out
2018-01-03 11 10 1 1.0 in
2018-01-04 10 10 0 1.0 in
2018-01-05 10 10 0 1.0 in
2018-01-06 9 10 -1 -1.0 out
If row T requires as input a value calculated in row T-1, then you'll probably want to do an iterative calculation. Typically backtesting is done by iterating through price data in sequence. You can calculate some signals just based on the state of the market, but you won't know the portfolio value, the pnl, or the portfolio positions unless you start at the beginning and work your way forward in time. That's why if you look at a site like Quantopian, the backtests always run from from start date to end date.