I am building a churn forecast model using features such as 1 year worth lags, holidays, moving averages, day/day ratios, seasonality factor extracted from statsmodels etc. It is clearly not an additive series, the magnitude of holiday churn each year is greater than that in previous years.
My XGB model predicts daily churn quite accurately, but it fails on holidays miserably (the trenches are slightly better predicted as compared to peaks):
in my opinion the model is unable to capture the exponential nature of the series. here is how it looks like at present. is there a way i can capture the exponential nature of the series, by using any additional features or something?
Related
I am working on the following timeseries multi-class classification problem:
42 possible classes that are dependent on each other, I want to know the probability of each class for up to 56 days ahead
1 year of daily data so 365 observations
the class probabilities have a strong weekly seasonality
I have exogenous regressors that are strongly correlated with the output classes
I realise that I am trying to predict a lot of output classes with little data, but I am looking for a model (preferably with Python implementation) that is most suited for this use case.
Any recommendations on what model could work for this problem?
So far I have tried:
a tree based model, but it struggles with the high amount of classes and does not capture the time series component well
a VAR model, but the number of parameters to estimate becomes too high compared to the series
predicting each class probability independently, but that assumes the series are independent, which is not the case
Struggling to build an arima model in python that is even close to useful for predicting household electricity usage. Would appreciate any thoughts and suggestions. (Might just be a silly error in my implementation!)
Some design thoughts:
Data is very messy in general but there is clearly daily seasonality (usage drops over night and while household at work/school) and a weekly seasonality (weekday usage differs from weekend)
Have tried statsmodels, sktime, fbprophet and pmdarima 'auto_arima' functions with no luck. Don't think these take seasonality into account particularly well
Currently trying to get a more manual approach to work: statsmodel's sarima with only daily seasonality incorporated (see code and results below), and maybe add fourier term as exogenous variable to handle weekly seasonality.
Will consider adding exogenous variables (like temperature) to account for annual seasonality but first just trying to get something reasonable on a smaller time scale (3-6 months).
Approach I am trying to get working: Use box-jenkins method to specify a sarima model for just daily seasonality (images below).
(1) Looking at Dickey-Fuller and KPSS for the time series, there appears to be minimal trend to correct for (expected), but ACF and PACF charts show significant seasonality (daily, weekly).
(2) Taking differences to account for week and day seasonality, then taking a further first-order difference, we can quickly get to a dataset that has minimal remaining seasonality and is stationary. This should be a really good sign and suggests there is a model we can build to predict this behaviour!
One more plot to show difference between original and differenced data when we zoom in for a typicaly week.
(3) Finally, I trained a sarima model in the following way with results. I configured D=d=0 since no identifiable trend (expected), p=2 to give model opportunity to learn from most recent behaviour, m=48 for seasonality (daily since data is in 30min time intervals), and P=Q=1 to capture those seasonality effects t-48.
model = SARIMAX(
train_data,
trend='n',
order=(2, 0, 0),
seasonal_order=(1, 0, 1, 48),
)
results = model.fit()
I am able to get an exponential smoothing model working, but I had expected double seasonal arima to blow it out of the water. Any thoughts and suggestions most welcome. Thank you in advance!
I am a bit confused about how to identify the seasonal component of a SARIMA model. I am currently looking at forecasting rates (ocean carrier rates to be specific). The first thing I did was to convert my original rates to the difference in rates, i.e. log(P2) - log(P1) as I wanted to forecast the change in rate itself. Then I checked for stationarity of the series and it was stationary. This is what the seasonal decomposed series look like
After that, I ran a basic ARIMA model and chose the p,d,q based on the ACF and PACF plots here but the predictions were pretty bad which was expected
I am now trying to run a SARIMA model instead. I obtained the seasonality component via seasonal_decompose and plotted the ACF and PACF over 52 lags (i have weekly data) and this is what I see. How do I choose the right pdq component for SARIMA?
Assume we have a time-series data that contains the daily orders count of last two years:
We can predict the future's orders using Python's statsmodels library:
fit = statsmodels.api.tsa.statespace.SARIMAX(
train.Count, order=(2, 1, 4),seasonal_order=(0,1,1,7)
).fit()
y_hat_avg['SARIMA'] = fit1.predict(
start="2018-06-16", end="2018-08-14", dynamic=True
)
Result (don't mind the numbers):
Now assume that our input data has some unusual increase or decrease, because of holidays or promotions in the company. So we added two columns that tell if each day was a "holiday" and a day that the company has had "promotion".
Is there a method (and a way of implementing it in Python) to use this new type of input data and help the model to understand the reason of outliers, and also predict the future's orders with providing "holiday" and "promotion_day" information?
fit1.predict('2018-08-29', holiday=True, is_promotion=False)
# or
fit1.predict(start="2018-08-20", end="2018-08-25", holiday=[0,0,0,1,1,0], is_promotion=[0,0,1,1,0,1])
SARIMAX, as a generalisation of the SARIMA model, is designed to handle exactly this. From the docs,
Parameters:
endog (array_like) – The observed time-series process y;
exog (array_like, optional) – Array of exogenous regressors, shaped (nobs, k).
You could pass the holiday and promotion_day as an array of size (nobs, 2) to exog, which will inform the model of the exogenous nature of some of these observations.
This problem have different names such as anomaly detection, rare event detection and extreme event detection.
There is some blog post at Uber engineering blog that may useful for understanding the problem and solution. Please look at here and here.
Although it's not from statsmodels, you can use facebook's prophet library for time series forecasting where you can pass dates with recurring events to your model.
See here.
Try this (it may or may not work based on your problem/data):
You can split your date into multiple features like day of week, day of month, month of year, year, is it last day in month?, is it first day in month? and many more if you think of it and then use some normal ML algorithm like Random Forests or Gradient Boosting Trees or Neural Networks (specially with embedding layers for your categorical features e.g. day of week) to train your model.
Data:
I have time series data for different countries and factors, e.g. birth rate for "Afghanistan" for years from 1972 up until 2007 (source).
Goal:
Predict e.g. birth rate for 2008 and 2012
Question:
I am familiar with linear regressions, but need some help on how to work with time series data and predict future values.
Can you point me to examples or share code snippets?
Take a look at the statsmodels Time Series Analysis module. Time series models are often based around autocorrelation, and the module has the standard univariate (for individual time series) AR(p) and MA(p) models, as well as the combined version ARIMA that allows for unit roots. You'll also find multivariate (for various interrelated time series) VAR models.
And here's a time series tutorial for statistical analysis and forecasting using pandas and statsmodels.
you can use ARIMA model and VAR Model in R.
ARIMA: Auto Regressive Integrated Moving Average model
VAR: Vector Auto Regressive model
For ARIMA model: click here
For VAR model: click here
For one time series data, use ARIMA model, however, if multiple time series data are related to each other, use VAR model.