We Keep Coding

Python: How to calculate revenue which is generated by month between years?

The columns in the below dataset will have:
A: Date contract opened;
B: Date contract stops;
C: Unique account ID against which contract associated (can have multiple contracts live against one ID)
D: Monthly revenue for contract period - for simplicity, assume revenue generated from first month contract assumed up to month before the date the contract closes
Opp Start Date OPP contract end date Unique Account Field MRR(expected)
1/2/2013 1/2/2015 50e55 195.00
1/2/2013 1/2/2014 4ee75 50.00
1/2/2013 1/2/2014 4f031 75.00
1/2/2013 1/2/2016 4c3b2 133.00
1/2/2013 1/2/2016 49ec8 132.00
1/3/2013 1/3/2014 49fc8 59.00
1/4/2013 1/4/2015 49wc8 87.00
12/27/2013 12/27/2014 50bf7 190.00
12/27/2013 12/27/2014 49cc8 179.00
12/27/2013 12/27/2014 49ec8 147.00
etc....
I would like to calculate the following:
How much revenue was generated by month between Jan-2013 and Dec-2014?
How many active contracts (generated revenue in that month) were there by month between Jan-2013 and Dec-2014?
How many active accounts (generated revenue from at least one contract) were there by month between Jan-2013 and Dec-2014?
I tried the below code:
I was able to use sum() to get the revenues, but I'm not sure what to do beyond this.
from datetime import date
df['date'] = pd.to_datetime(df['Opp Start Date'])
df.groupby(df['Opp Start Date'].dt.strftime('%B'))['MRR(expected)'].sum().sort_values()
Result I got from the above code:
Opp Start Date
February 221744
January 241268
July 245811
August 247413
April 249702
March 251219
June 251494
May 259149
September 263395
October 293990
November 296590
December 311659
I need to calculate the above following.
How can I achieve this in python?

You could do it in either place, or a combination of the two. There are also lots of different ways you could approach this. Personally, I would create a query that pulled the information for one month, and then iterate through, either storing the results in an array or a temp table.
For a single month in the period, the query would look something like this:
Select count(unique_account_field), sum(MRR)
from contracts
where Opp_start_date <= #month_end
and Opp_contract_end_date > #month_start
That would take care of 1 and 2. #3 is a little different. That's a simple count(distinct unique_account_field) from the set over the whole period.

How to aggregate irregularly sampled data for Time Series Analysis

I am trying to forecast daily profit using time series analysis, but daily profit is not only recorded unevenly, but some of the data is missing.
Raw Data:
Date
Revenue
2020/1/19
10$
2020/1/20
7$
2020/1/25
14$
2020/1/29
18$
2020/2/1
12$
2020/2/2
17$
2020/2/9
28$
The above table is an example of what kind of data I have. Profit is not recorded daily, so date between 2020/1/20 and 2020/1/24 does not exist. Not only that, say the profit recorded during the period between 2020/2/3 and 2020/3/8 went missing in the database. I would like to recover this missing data and use time series analysis to predict the profit after 2020/2/9 ~.
My approach was to first aggregate the profit every 6 days since I have to recover the profit between 2020/2/3 and 2020/3/8. So my cleaned data will look something like this
Date
Revenue
2020/1/16 ~ 2020/1/21
17$
2020/1/22 ~ 2020/1/27
14$
2020/1/28 ~ 2020/2/2
47$
2020/2/3 ~ 2020/2/8
? (to predict)
After applying this to a time series model, I would like to further predict the profit after 2020/2/9 ~.
This is my general idea, but as a beginner at Python, using pandas library, I have trouble executing my ideas. Could you please help me how to aggregate the profit every 6 days and have the data look like the above table?

Easiest way is using pandas resample function.
Provided you have an index of type Datetime resampling to aggregate profits at every 6 days would be as simple as your_dataframe.resample('6D').sum()
You can do all sorts of resampling (end of month, end of quarter, begining of week, every hour, minute, second, ...). Check the full documentation if you're interested: https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.resample.html?highlight=resample#pandas.DataFrame.resample

I suggest using a combination of .rolling, pd.date_range, and .reindex
Say your DataFrame is df, with proper datetime indexing:
df = pd.DataFrame([['2020/1/19',10],
['2020/1/20',7],
['2020/1/25',14],
['2020/1/29',18],
['2020/2/1',12],
['2020/2/2',17],
['2020/2/9',28]],columns=['Date','Revenue'])
df['Date'] = pd.to_datetime(df['Date'])
df.set_index('Date',inplace=True)
The first step is to 'fill in' the missing days with dummy, zero revenue. We can use pd.date_range to get an index with evenly spaced dates from 2020/1/16 to 2020/2/8, and then .reindex to bring this into the main df DataFrame:
evenly_spaced_idx = pd.date_range(start='2020/1/16',end='2020/2/8',freq='1d')
df = df.reindex(evenly_spaced_idx, fill_value=0)
Now we can take a rolling sum for each 6 day period. We're not interested in every day's six day revenue total, only every 6th days, though:
summary_df = df.rolling('6d').sum().iloc[5::6, :]
The last thing with summary_df is just to format it the way you'd like so that it clearly states the date range which each row refers to.
summary_df['Start Date'] = summary_df.index-pd.Timedelta('6d')
summary_df['End Date'] = summary_df.index
summary_df.reset_index(drop=True,inplace=True)

You can use resample for this.
Make sure to have the "Date" column as datetime type.
>>> df = pd.DataFrame([["2020/1/19" ,10],
... ["2020/1/20" ,7],
... ["2020/1/25" ,14],
... ["2020/1/29" ,18],
... ["2020/2/1" ,12],
... ["2020/2/2" ,17],
... ["2020/2/9" ,28]], columns=['Date', 'Revenue'])
>>> df['Date'] = pd.to_datetime(df.Date)
For pandas < 1.1.0
>>> df.set_index('Date').resample('6D', base=3).sum()
Revenue
Date
2020-01-16 17
2020-01-22 14
2020-01-28 47
2020-02-03 0
2020-02-09 28
For pandas >= 1.1.0
>>> df.set_index('Date').resample('6D', origin='2020-01-16').sum()
Revenue
Date
2020-01-16 17
2020-01-22 14
2020-01-28 47
2020-02-03 0
2020-02-09 28

Grab last price of the day from other dataframe in pandas

Two dataframes:
Dataframe 'prices' contains minute pricing.
ts average
2017-12-13 15:55:00-05:00 339.389
2017-12-13 15:56:00-05:00 339.293
2017-12-13 15:57:00-05:00 339.172
2017-12-13 15:58:00-05:00 339.148
2017-12-13 15:59:00-05:00 339.144
Dataframe 'articles' contains articles:
ts title
2017-10-25 11:45:00-04:00 Your Evening Briefing
2017-11-24 14:15:00-05:00 Tesla's Grand Designs Distract From Model 3 Bo...
2017-10-26 11:09:00-04:00 UAW Files Claim That Tesla Fired Workers Who S...
2017-10-25 11:42:00-04:00 Forget the Grid of the Future, Puerto Ricans J...
2017-10-22 09:54:00-04:00 Tesla Reaches Deal for Shanghai Facility, WSJ ...
When 'article' happens, I want the current average stock price (easy), plus the stock price of the end of the day (problem).
My current approach:
articles['t-eod'] = prices.loc[articles.index.strftime('%Y-%m-%d')[0]].between_time('15:30','15:31')
However, it gives a warning:
/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
"""Entry point for launching an IPython kernel.
Reading the docs didn't make it a lot clearer to me.
So question: How can I, for every Article, get Prices' last average price of that day?
Thanks!
/Maurice

You could try using idxmax on ts to identify the index of the maximum timestamp of that date and extract the average value with loc
#Reset our index
prices_df.reset_index(inplace=True)
articles_df.reset_index(inplace=True)
#Ensure our ts field is datetime
prices_df['ts'] = pd.to_datetime(prices_df['ts'])
articles_df['ts'] = pd.to_datetime(articles_df['ts'])
#Get maximum average value from price_df by date
df_max = prices_df.loc[prices_df.groupby(prices_df.ts.dt.date, as_index=False).ts.idxmax()]
#We need to join df_max and articles on the date so we make a new index
df_max['date'] = df_max.ts.dt.date
articles_df['date'] = articles_df.ts.dt.date
df_max.set_index('date',inplace=True)
articles_df.set_index('date',inplace=True)
#Set our max field
articles_df['max'] = df_max['average']
articles_df.set_index('ts',inplace=True)

Remove non-business days rows from pandas dataframe

I have a dataframe with a timeseries data of wheat in df.
df = wt["WHEAT_USD"]
2016-05-02 02:00:00+02:00 4.780
2016-05-02 02:01:00+02:00 4.777
2016-05-02 02:02:00+02:00 4.780
2016-05-02 02:03:00+02:00 4.780
2016-05-02 02:04:00+02:00 4.780
Name: closeAsk, dtype: float64
When I plot the data it has these annoying horizontal lines because of weekends. Is there a simple way of removing the non-business days from the dataframe itself?
Something like
df = df.BDays()

One simple solution is to slice out the days not in Monday to Friday:
In [11]: s[s.index.dayofweek < 5]
Out[11]:
2016-05-02 00:00:00 4.780
2016-05-02 00:01:00 4.777
2016-05-02 00:02:00 4.780
2016-05-02 00:03:00 4.780
2016-05-02 00:04:00 4.780
Name: closeAsk, dtype: float64
Note: this doesn't take into account bank holidays etc.

Pandas BDay just ends up using .dayofweek<5 like the chosen answer, but can be extended to account for bank holidays, etc.
import pandas as pd
from pandas.tseries.offsets import BDay
isBusinessDay = BDay().onOffset
csv_path = 'C:\\Python27\\Lib\\site-packages\\bokeh\\sampledata\\daylight_warsaw_2013.csv'
dates_df = pd.read_csv(csv_path)
match_series = pd.to_datetime(dates_df['Date']).map(isBusinessDay)
dates_df[match_series]

I am building a backtester for stock/FX trading and I also have these issue with days that are nan because that they are holidays or other non trading days..
you can download a financial calendar for the days that there is no trading and then you need to think about timezone and weekends.. etc..
But the best solution is not to use date/time as the index for the candles or price.
So do not connect your price data to a date/time but just to a counter of candles or prices .. you can use a second index for this..
so for calculations of MA or other technical lines dont use date/time ..
if you look at Metatrader 4/5 it also doesnt use date/time but the index of the data is the candle number !!
I think that you need to let go of the date-time for the price if you work with stock or FX data , of cause you can put them in a column of the data-frame but dont use it as the index
This way you can avoid many problems

using workdays, you can count for holidays pretty easily
import workdays as wd
def drop_non_busdays(df, holidays=None):
if holidays is None:
holidays = []
start_date = df.index.to_list()[0].date()
end_date = df.index.to_list()[-1].date()
start_wd = wd.workday(wd.workday(start_date, -1, holidays), 1, holidays)
end_wd = wd.workday(wd.workday(end_date, 1, holidays), -1, holidays)
b_days = [start_wd]
while b_days[-1] < end_wd:
b_days.append(wd.workday(b_days[-1], 1, holidays))
valid = [i in b_days for i in df.index]
return df[valid]

Building on #Andy Hayden solution, you can also use query with a dataframe for better method chaining in a "modern pandas" fashion.
If the date is a column (e.g and is named my_date)
df.query("my_date.dt.dayofweek < 5")
If the date is the index and has a name (e.g. my_index_name or date)
df.query("my_index_name.dt.dayofweek < 5")
If the date is the index and it has no name
df.rename_axis("date").query("date.dt.dayofweek < 5")
( index.dt.dayofweek or index.dayofweek does not works for me ) .

simply, filtering can be done by day names. For example if you don't want saturdays and sundays you can use this:
df=df[(df['date'].dt.day_name()!='Saturday') & (df['date'].dt.day_name()!='Sunday')]
not for special holidays etc

converting daily stock data to weekly-based via pandas in Python

I've got a DataFrame storing daily-based data which is as below:
Date Open High Low Close Volume
2010-01-04 38.660000 39.299999 38.509998 39.279999 1293400
2010-01-05 39.389999 39.520000 39.029999 39.430000 1261400
2010-01-06 39.549999 40.700001 39.020000 40.250000 1879800
2010-01-07 40.090000 40.349998 39.910000 40.090000 836400
2010-01-08 40.139999 40.310001 39.720001 40.290001 654600
2010-01-11 40.209999 40.520000 40.040001 40.290001 963600
2010-01-12 40.160000 40.340000 39.279999 39.980000 1012800
2010-01-13 39.930000 40.669998 39.709999 40.560001 1773400
2010-01-14 40.490002 40.970001 40.189999 40.520000 1240600
2010-01-15 40.570000 40.939999 40.099998 40.450001 1244200
What I intend to do is to merge it into weekly-based data. After grouping:
the Date should be every Monday (at this point, holidays scenario should be considered when Monday is not a trading day, we should apply the first trading day in current week as the Date).
Open should be Monday's (or the first trading day of current week) Open.
Close should be Friday's (or the last trading day of current week) Close.
High should be the highest High of trading days in current week.
Low should be the lowest Low of trading days in current week.
Volumn should be the sum of all Volumes of trading days in current week.
which should look like this:
Date Open High Low Close Volume
2010-01-04 38.660000 40.700001 38.509998 40.290001 5925600
2010-01-11 40.209999 40.970001 39.279999 40.450001 6234600
Currently, my code snippet is as below, which function should I use to mapping daily-based data to the expected weekly-based data? Many thanks!
import pandas_datareader.data as web
start = datetime.datetime(2010, 1, 1)
end = datetime.datetime(2016, 12, 31)
f = web.DataReader("MNST", "yahoo", start, end, session=session)
print f

You can resample (to weekly), offset (shift), and apply aggregation rules as follows:
logic = {'Open' : 'first',
'High' : 'max',
'Low' : 'min',
'Close' : 'last',
'Volume': 'sum'}
offset = pd.offsets.timedelta(days=-6)
f = pd.read_clipboard(parse_dates=['Date'], index_col=['Date'])
f.resample('W', loffset=offset).apply(logic)
to get:
Open High Low Close Volume
Date
2010-01-04 38.660000 40.700001 38.509998 40.290001 5925600
2010-01-11 40.209999 40.970001 39.279999 40.450001 6234600

In general, assuming that you have the dataframe in the form you specified, you need to do the following steps:
put Date in the index
resample the index.
What you have is a case of applying different functions to different columns. See.
You can resample in various ways. for e.g. you can take the mean of the values or count or so on. check pandas resample.
You can also apply custom aggregators (check the same link).
With that in mind, the code snippet for your case can be given as:
f['Date'] = pd.to_datetime(f['Date'])
f.set_index('Date', inplace=True)
f.sort_index(inplace=True)
def take_first(array_like):
return array_like[0]
def take_last(array_like):
return array_like[-1]
output = f.resample('W', # Weekly resample
how={'Open': take_first,
'High': 'max',
'Low': 'min',
'Close': take_last,
'Volume': 'sum'},
loffset=pd.offsets.timedelta(days=-6)) # to put the labels to Monday
output = output[['Open', 'High', 'Low', 'Close', 'Volume']]
Here, W signifies a weekly resampling which by default spans from Monday to Sunday. To keep the labels as Monday, loffset is used.
There are several predefined day specifiers. Take a look at pandas offsets. You can even define custom offsets (see).
Coming back to the resampling method. Here for Open and Close you can specify custom methods to take the first value or so on and pass the function handle to the how argument.
This answer is based on the assumption that the data seems to be daily, i.e. for each day you have only 1 entry. Also, no data is present for the non-business days. i.e. Sat and Sun. So taking the last data point for the week as the one for Friday is ok. If you so want you can use business week instead of 'W'. Also, for more complex data you may want to use groupby to group the weekly data and then work on the time indices within them.
btw a gist for the solution can be found at:
https://gist.github.com/prithwi/339f87bf9c3c37bb3188

I had the exact same question and found a great solution here.
https://www.techtrekking.com/how-to-convert-daily-time-series-data-into-weekly-and-monthly-using-pandas-and-python/
The weekly code is posted below.
import pandas as pd
import numpy as np
print('*** Program Started ***')
df = pd.read_csv('15-06-2016-TO-14-06-2018HDFCBANKALLN.csv')
# ensuring only equity series is considered
df = df.loc[df['Series'] == 'EQ']
# Converting date to pandas datetime format
df['Date'] = pd.to_datetime(df['Date'])
# Getting week number
df['Week_Number'] = df['Date'].dt.week
# Getting year. Weeknum is common across years to we need to create unique index by using year and weeknum
df['Year'] = df['Date'].dt.year
# Grouping based on required values
df2 = df.groupby(['Year','Week_Number']).agg({'Open Price':'first', 'High Price':'max', 'Low Price':'min', 'Close Price':'last','Total Traded Quantity':'sum'})
# df3 = df.groupby(['Year','Week_Number']).agg({'Open Price':'first', 'High Price':'max', 'Low Price':'min', 'Close Price':'last','Total Traded Quantity':'sum','Average Price':'avg'})
df2.to_csv('Weekly_OHLC.csv')
print('*** Program ended ***')

Adding to #Stefan 's answer with recent pandas API as loffset was deprecated since version 1.1.0 and later removed.
df = pd.read_clipboard(parse_dates=['Date'], index_col=['Date'])
logic = {'Open' : 'first',
'High' : 'max',
'Low' : 'min',
'Close' : 'last',
'Volume': 'sum'}
dfw = df.resample('W').apply(logic)
# set the index to the beginning of the week
dfw.index = dfw.index - pd.tseries.frequencies.to_offset("6D")

At first I use df.resample() according to the answers forementioned, but it fills NaN when a week is missed, unhappy about that, after some research, I use groupby() instead of resample(). Thanks for your sharing.
My original data is:
c date h l o
260 6014.78 20220321 6053.90 5984.79 6030.43
261 6052.59 20220322 6099.53 5995.22 6012.17
262 6040.86 20220323 6070.85 6008.26 6059.11
263 6003.05 20220324 6031.73 5987.40 6020.00
264 5931.33 20220325 6033.04 5928.72 6033.04
265 5946.98 20220328 5946.98 5830.93 5871.35
266 5900.04 20220329 5958.71 5894.82 5950.89
267 6003.05 20220330 6003.05 5913.08 5913.08
268 6033.04 20220331 6059.11 5978.27 5993.92
269 6126.91 20220401 6134.74 5975.66 6006.96
270 6149.08 20220406 6177.77 6106.05 6126.91
271 6134.74 20220407 6171.25 6091.71 6130.83
272 6151.69 20220408 6160.82 6096.93 6147.78
273 6095.62 20220411 6166.03 6072.15 6164.73
274 6184.28 20220412 6228.62 6049.99 6094.32
275 6119.09 20220413 6180.37 6117.79 6173.85
276 6188.20 20220414 6201.24 6132.13 6150.38
277 6173.85 20220415 6199.93 6137.35 6137.35
278 6124.31 20220418 6173.85 6108.66 6173.85
279 6065.63 20220419 6147.78 6042.16 6124.31
I don't care the date is not Monday, so I didn't handle that, the code is:
data['Date'] = pd.to_datetime(data['date'], format="%Y%m%d")
# Refer to: https://www.techtrekking.com/how-to-convert-daily-time-series-data-into-weekly-and-monthly-using-pandas-and-python/
# and here: https://stackoverflow.com/a/60518425/5449346
# and this: https://github.com/pandas-dev/pandas/issues/11217#issuecomment-145253671
logic = {'o' : 'first',
'h' : 'max',
'l' : 'min',
'c' : 'last',
'Date': 'first',
}
data = data.groupby([data['Date'].dt.year, data['Date'].dt.week]).agg(logic)
data.set_index('Date', inplace=True)
And the result is, there's no NaN on 2022.01.31 which resample() will produce:
l o h c
Date
2021-11-29 6284.68 6355.09 6421.59 6382.47
2021-12-06 6365.52 6372.04 6700.62 6593.70
2021-12-13 6445.06 6593.70 6690.19 6450.28
2021-12-20 6415.07 6437.24 6531.12 6463.31
2021-12-27 6463.31 6473.75 6794.50 6649.77
2022-01-04 6625.00 6649.77 7089.18 7055.27
2022-01-10 6804.93 7055.27 7181.75 6808.84
2022-01-17 6769.73 6776.25 7098.30 6919.67
2022-01-24 6692.80 6906.63 7048.76 6754.08
2022-02-07 6737.13 6811.45 7056.58 7023.98
2022-02-14 6815.36 7073.53 7086.57 6911.85
2022-02-21 6634.12 6880.56 6904.03 6668.02
2022-02-28 6452.88 6669.33 6671.93 6493.30
2022-03-07 5953.50 6463.31 6468.53 6228.62
2022-03-14 5817.90 6154.30 6205.15 6027.82
2022-03-21 5928.72 6030.43 6099.53 5931.33
2022-03-28 5830.93 5871.35 6134.74 6126.91
2022-04-06 6091.71 6126.91 6177.77 6151.69
2022-04-11 6049.99 6164.73 6228.62 6173.85
2022-04-18 6042.16 6173.85 6173.85 6065.63

Updated solution for 2022
import pandas as pd
from pandas.tseries.frequencies import to_offset
df = pd.read_csv('your_ticker.csv')
logic = {'<Open>' : 'first',
'<High>' : 'max',
'<Low>' : 'min',
'<Close>' : 'last',
'<Volume>': 'sum'}
df['<DTYYYYMMDD>'] = pd.to_datetime(df['<DTYYYYMMDD>'])
df = df.set_index('<DTYYYYMMDD>')
df = df.sort_index()
df = df.resample('W').apply(logic)
df.index = df.index - pd.tseries.frequencies.to_offset("6D")

Not a direct answer, but suppose the columns are the dates (transpose of your table), without missing dates.
'''sum up daily results in df to weekly results in wdf'''
wdf = pd.DataFrame(index = df.index)
for i in range(len(df.columns)):
if (i!=0) & (i%7==0):
wdf['week'+str(i//7)]= df[df.columns[i-7:i]].sum(axis = 1)