Public companies in the US make quarterly filings (10-Q) and yearly filings (10-K). In most cases they will file three 10Qs per year and one 10K.
In most cases, the quarterly filings (10Qs) contain quarterly data. For example, "revenue for the three months ending March 31, 2005."
The yearly filings will often only have year end sums. For example: "revenue for the twelve months ending December 31, 2005."
In order to get the value for Q4 of 2005, I need to take the yearly data and subtract the values for each of the quarters (Q1-Q3).
In some cases, each of the quarterly data is expressed as year to date. For example, the first quarterly filing is "revenue for the three months ending March 31, 2005." The second is "revenue for the six months ending June 30, 2005." The third "revenue for the nine months ending September 30, 2005." The yearly is like above, "revenue for the twelve months ending December 31, 2005." This represents a generalization of the above issues in which the desire is to extract quarterly data which can be accomplished by repeated subtraction of the previous period data.
My question is what is the best way in pandas to accomplish this quarterly data extraction?
There a large number of fields (revenue, profit, exposes, etc) per period.
A related question that I asked in regards to how to express this period data in pandas: Creating Period for Multi Quarter Timespan in Pandas
Here is some example data of the first problem (three 10Qs and one 10K which only has year end data):
10Q:
http://www.sec.gov/Archives/edgar/data/1174922/000119312512225309/d326512d10q.htm#tx326512_4
http://www.sec.gov/Archives/edgar/data/1174922/000119312512347659/d360762d10q.htm#tx360762_3
http://www.sec.gov/Archives/edgar/data/1174922/000119312512463380/d411552d10q.htm#tx411552_3
10K:
http://www.sec.gov/Archives/edgar/data/1174922/000119312513087674/d459372d10k.htm#tx459372_29
Calcbench refers to this problem: http://www.calcbench.com/Home/userGuide: "Q4 calculation: Companies often do not report Q4 data, rather opting to report full year data instead. We’ll automatically calculate it for you. Data in blue is calculated.
There will be multiple years of data and for each year I want to calculate the missing fourth quarter:
2012Q2 2012Q3 2012Y 2013Q1 2013Q2 2013Q3 2013Y
Revenue 1 1 1 1 1 1 1
Expense 10 10 10 10 10 10 10
You could define a function to subtract the quarterly totals from the annual number, and then apply the function to each row, storing the result in a new column.
In [2]: df
Out[2]:
Annual Q1 Q2 Q3
Revenue 18 3 4 5
Expense 17 2 3 4
In [3]: def calc_Q4(row):
...: return row['Annual'] - row['Q1'] - row['Q2'] - row['Q3']
In [4]: df['Q4'] = df.apply(calc_Q4, axis = 1)
In [5]: df
Out[5]:
Annual Q1 Q2 Q3 Q4
Revenue 18 3 4 5 6
Expense 17 2 3 4 8
I work for Calcbench.
I wrote an API for Calcbench and have example of getting SEC data into Pandas dataframes, https://www.calcbench.com/home/api.
You would need to sign up for Calcbench to use it.
Related
The data have reported values for January 2006 through January 2019. I need to compute the total number of passengers Passenger_Count per month. The dataframe should have 121 entries (10 years * 12 months, plus 1 for january 2019). The range should go from 2009 to 2019.
I have been doing:
df.groupby(['ReportPeriod'])['Passenger_Count'].sum()
But it doesn't give me the right result, it gives
You can do
df['ReportPeriod'] = pd.to_datetime(df['ReportPeriod'])
out = df.groupby(df['ReportPeriod'].dt.strftime('%Y-%m-%d'))['Passenger_Count'].sum()
Try this:
df.index = pd.to_datetime(df["ReportPeriod"], format="%m/%d/%Y")
df = df.groupby(pd.Grouper(freq="M")).sum()
I have a data frame containing the daily CO2 data since 2015, and I would like to produce the monthly mean data for each year, then put this into a new data frame. A sample of the data frame I'm using is shown below.
month day cycle trend
year
2011 1 1 391.25 389.76
2011 1 2 391.29 389.77
2011 1 3 391.32 389.77
2011 1 4 391.36 389.78
2011 1 5 391.39 389.79
... ... ... ... ...
2021 3 13 416.15 414.37
2021 3 14 416.17 414.38
2021 3 15 416.18 414.39
2021 3 16 416.19 414.39
2021 3 17 416.21 414.40
I plan on using something like the code below to create the new monthly mean data frame, but the main problem I'm having is indicating the specific subset for each month of each year so that the mean can then be taken for this. If I could highlight all of the year "2015" for the month "1" and then average this etc. that might work?
Any suggestions would be hugely appreciated and if I need to make any edits please let me know, thanks so much!
dfs = list()
for l in L:
dfs.append(refined_data[index = 2015, "month" = 1. day <=31].iloc[l].mean(axis=0))
mean_matrix = pd.concat(dfs, axis=1).T
A similar question has been asked for cumsum and grouping but it didn't solve my case.
I have a financial balance sheet of a lot of years and need to sum all previous values by year.
This is my reproducible set:
df=pd.DataFrame(
{"Amount": [265.95,2250.00,-260.00,-2255.95,120],
"Year": [2018,2018,2018,2019,2019]})
The result I want is the following:
Year Amount
2017 0
2018 2255.95
2019 120.00
2020 120.00
So actually in a loop going from the lowest year in my whole set to the highest year in my set.
...
df[df.Year<=2017].Amount.sum()
df[df.Year<=2018].Amount.sum()
df[df.Year<=2019].Amount.sum()
df[df.Year<=2020].Amount.sum()
...
First step is aggregate sum, then use Series.cumsum and Series.reindex with forward filling missing values by all possible years, last replace first missing values to 0:
years = range(2017, 2021)
df1 = (df.groupby('Year')['Amount']
.sum()
.cumsum()
.reindex(years, method='ffill')
.fillna(0)
.reset_index())
print (df1)
Year Amount
0 2017 0.00
1 2018 2255.95
2 2019 120.00
3 2020 120.00
I am working with NLSY79 data and I am trying to construct a 'smoothed' income variable that averages over a period of 4 years. Between 1979 and 1994, the NLSY conducted surveys annually, while after 1996 the survey was conducted biennially. This means that my smoothed income variable will average four observations prior to 1994 and only two after 1996.
I would like my smoothed income variable to satisfy the following criteria:
1) It should be an average of 4 income observations from 1979 to 1994 and only 2 from 1996 onward
2) The window should START from a given observation rather than be centered at it. Therefore, my smoothed income variable should tell me the average income over the four years starting from that date
3) It should ignore NaNs
It should, therefore, look like the following (note that I only computed values for 'smoothed income' that could be computed with the data I have provided.)
id year income 'smoothed income'
1 1979 20,000 21,250
1 1980 22,000
1 1981 21,000
1 1982 22,000
...
1 2014 34,000 34,500
1 2016 35,000
2 1979 28,000 28,333
2 1980 NaN
2 1981 28,000
2 1982 29,000
I am relatively new to dataframe manipulation with pandas, so here is what I have tried:
smooth = DATA.groupby('id')['income'].rolling(window=4, min_periods=1).mean()
DATA['smoothIncome'] = smooth.reset_index(level=0, drop=True)
This code accounts for NaNs, but otherwise does not accomplish objectives 2) and 3).
Any help would be much appreciated
Ok, I've modified the code provided by ansev to make it work. filling in NaNs was causing the problems.
Here's the modified code:
df.set_index('year').groupby('id').income.apply(lambda x: x.reindex(range(x.index.min(),x.index.max()+1))
.rolling(4, min_periods = 1).mean().shift(-3)).reset_index()
The only problem I have now is that the mean is not calculated when there are fewer that 4 years remaining (e.g. from 2014 onward, because my data goes until 2016). Is there a way of shortening the window length after 2014?
In the dataframe below (small snippet show, actual dataframe spans from 2000 to 2014 in time), I want to compute the annual average but starting in September of one year and going till only May of next year.
Cnt Year JD Min_Temp
S 2000 1 277.139
S 2000 2 274.725
S 2001 1 270.945
S 2001 2 271.505
N 2000 1 257.709
N 2000 2 254.533
N 2000 3 258.472
N 2001 1 255.763
I can compute annual average (Jan - Dec) using this code:
df['Min_Temp'].groupby(df['YEAR']).mean()
How do I adapt this code to mean from Sept of first year to May of next year?
--EDIT: Based on comments below, you can assume that a MONTH column is also available, specifying the month for each row
Not sure which column refers to month or if it is missing, but in the past I've used a quick and dirty method to assign custom seasons (interested if anyone has found more elegant route).
I've used Yahoo Finance data to demonstrate approach, unless one of your columns is Month?
EDIT Requires dataframe to be sorted by date ascending
import pandas as pd
import pandas.io.data as web
import datetime
start = datetime.datetime(2010, 9, 1)
end = datetime.datetime(2015, 5, 31)
df = web.DataReader("F", 'yahoo', start, end)
#Ensure date sorted --required
df = df.sort_index()
#identify custom season and set months june-august to null
count = 0
season = 1
for i,row in df.iterrows():
if i.month in [9,10,11,12,1,2,3,4,5]:
if count == 1:
season += 1
df.set_value(i,'season', season)
count = 0
else:
count = 1
df.set_value(i,'season',None)
#new data frame excluding months june-august
df_data = df[~df['season'].isnull()]
df_data['Adj Close'].groupby(df_data.season).mean()