We Keep Coding

How to predict future with tensotflow lib

I use the tensorflow library to solve the time series problem.
I get the dimensions or properties by subtracting the current value from the previous value (according to this article)
In this article, there is the data needed for forecasting. It chooses a value for training and a value for testing that there are no problems.
But my question is how can I predict the future? Suppose if I want to forecast 5 months later there will be no dimensions or attributes to send to the forecast function.
--If you have a better source, please introduce it ...Thanks in advance

If you have a lot of data it could be possible, it means that your model knows a lot of data and can generalize with new data and it can find a knowed pattern. If you have a poor model it will throws bad predictions because the new input is new and the model can't find a knowed pattern

Why do I need to call fit() before transform() when using PolynomialFeatures?

Hello to all you great minds,
I'm trying to understand more rigorously the way polynomial fitting works with scikit. More specifically, what I'm trying to do is break down the process, and to only show a dataframe with the new polynomial features generated based on a single value.
So I have data which with several entries, each is 1-dimensional. I want to generate a design matrix suitable for polynomial fitting. What I am currently doing is along these lines:
pd.DataFrame(PolynomialFeatures(k).fit_transform(X))
And this works as expected.
However, what I'm struggling with is the role of fit_transform(). As far as I am concerned, and I not trying to fit anything quiet yet, merely produce a dataframe with the newly constructed polynomial features. Naively I tried changing fit_transform() to transform(), but apparently I have to use fit before I am allowed to transform.
I would appreciate it if anyone could point me to my error. I am not yet trying to fit a model on the data, only to create a design matrix with the polynomial features, so why do I have to use fit() (or fit_transform(), to that matter)? In fact, I don't really understand what fit() actually does here, and the documentation didn't help me wrap my head around it.
Thank you!

I think the reason for this is to be consistent with their API. When doing preprocessing you still want to "fit" to some train data and apply the same preprocessing step to the train AND the test data.
An example where it becomes more clear is Standardscaling (which is a different preprocessing step). You calculate the mean and std from the train data and apply the same scaling (X - mean) / std to the train AND test data (with the mean and std taken from the train data.
Therefore the two methods fit and transform are separated.
In your case of polynomial features it probably makes no sense to "fit", because no information is extracted from the train data and the step can directly be applied to the test data without knowing the train data. But including the fit in PolynomialFeatures makes it consistent with their whole API. The consistency becomes necessary when you pipe multiple preprocessing steps.

How to perform multi-step out-of-time forecast which does not involve refitting the ARIMA model?

I have an already existing ARIMA (p,d,q) model fit to a time-series data (for ex, data[0:100]) using python. I would like to do forecasts (forecast[100:120]) with this model. However, given that I also have the future true data (eg: data[100:120]), how do I ensure that the multi-step forecast takes into account the future true data that I have instead of using the data it forecasted?
In essence, when forecasting I would like forecast[101] to be computed using data[100] instead of forecast[100].
I would like to avoid refitting the entire ARIMA model at every time step with the updated "history".
I fit the ARIMAX model as follows:
train, test = data[:100], data[100:]
ext_train, ext_test = external[:100], external[100:]
model = ARIMA(train, order=(p, d, q), exog=ext_train)
model_fit = model.fit(displ=False)
Now, the following code allows me to predict values for the entire dataset, including the test
forecast = model_fit.predict(end=len(data)-1, exog=external, dynamic=False)
However in this case after 100 steps, the ARIMAX predicted values quickly converge to the long-run mean (as expected, since after 100 time steps it is using the forecasted values only). I would like to know if there is a way to provide the "future" true values to give better online predictions. Something along the lines of:
forecast = model_fit.predict_fn(end = len(data)-1, exog=external, true=data, dynamic=False)
I know I can always keep refitting the ARIMAX model by doing
historical = train
historical_ext = ext_train
predictions = []
for t in range(len(test)):
model = ARIMA(historical, order=(p,d,q), exog=historical_ext)
model_fit = model.fit(disp=False)
output = model_fit.forecast(exog=ext_test[t])[0]
predictions.append(output)
observed = test[t]
historical.append(observed)
historical_ext.append(ext_test[t])
but this leads to me training the ARIMAX model again and again which doesn't make a lot of sense to me. It leads to using a lot of computational resources and is quite impractical. It further makes it difficult to evaluate the ARIMAX model cause the fitted params to keep on changing every iteration.
Is there something incorrect about my understanding/use of the ARIMAX model?

You are right, if you want to do online forecasting using new data you will need to estimate the parameters over and over again which is computationally inefficient.
One thing to note is that for the ARIMA model mainly the estimation of the parameters of the MA part of the model is computationally heavy, since these parameters are estimated using numerical optimization, not using ordinary least squares. Since after calculating the parameters once for the initial model you know what is expected for future models, since one observation won't change them much, you might be able to initialize the search for the parameters to improve computational efficiency.
Also, there may be a method to do the estimation more efficiently, since you have your old data and parameters for the model, the only thing you do is add one more datapoint. This means that you only need to calculate the theta and phi parameters for the combination of the new datapoint with all the others, while not computing the known combinations again, which would save quite some time. I very much like this book: Heij, Christiaan, et al. Econometric methods with applications in business and economics. Oxford University Press, 2004.
And this lecture might give you some idea of how this might be feasible: lecture on ARIMA parameter estimation
You would have to implement this yourself, I'm afraid. As far as I can tell, there is nothing readily available to do this.
Hope this gives you some new ideas!

As this very good blog suggests (3 facts about time series forecasting that surprise experienced machine learning practitioners):
"You need to retrain your model every time you want to generate a new prediction", it also gives the intuitive understanding of why this happens with examples. That basically highlights time-series forecasting challenge as a constant change, that needs refitting.

I was struggling with this problem. Luckily, I found a very useful discussion about it. As far as I know, the case is not supported by ARIMA in python, we need to use SARIMAX.
You can refer to the link of discussion: https://github.com/statsmodels/statsmodels/issues/2788

Prediction Model to predict when the future events will happen next month

I have to develop a Prediction Model using Python to predict if a site will crash next month or not depending on the occurances in the last 6 monthes. Input Parameters are: Environment(Dev,Prod,Test), Region(NA,APAC,EMEA) and the Date of the month.
I am using matplotlib, pandas and numpy. It will be a 2D Data Frame or a 3D Panel in Pandas. I am not sure as input parameters are 3 - Region, Env and Date.
I think below Machine Learning Algorithm should be used:
from sklearn.linear_model import LinearRegression
Please correct me if I am wrong.

Linear regression is fine, but calling it is just two line of work, i would suggest try multiple machine learning algorithms, tuning their hyperparameters and checking which gives the best performance, moreover you should look into feature engineering, maybe you could extract features from the already given data

How to train statsmodels.tsa.ARIMA model with multiple series

The usual way to fit an ARIMA model with the statsmodels python package is:
model = statsmodels.tsa.ARMA(series, order=(2,2))
result = model.fit(trend='nc', disp=1)
however, i have multiple time series data to train with, say, from the same underlying process, how could i do that?

When you say, multiple time series data, it is not clear if they are of the same type. There is no straightforward way to specify multiple series in ARMA model. However you could use the 'exog' optional variable to indicate the second series.
Please refer for the actual definition of ARMA model.
model = statsmodels.tsa.ARMA(endog = series1, exog=series2, order=(2,2))
Please refer for the explanation of the endog, exog variables.
Please see a working example of how this could be implemented