I use the tensorflow library to solve the time series problem.
I get the dimensions or properties by subtracting the current value from the previous value (according to this article)
In this article, there is the data needed for forecasting. It chooses a value for training and a value for testing that there are no problems.
But my question is how can I predict the future? Suppose if I want to forecast 5 months later there will be no dimensions or attributes to send to the forecast function.
--If you have a better source, please introduce it ...Thanks in advance
If you have a lot of data it could be possible, it means that your model knows a lot of data and can generalize with new data and it can find a knowed pattern. If you have a poor model it will throws bad predictions because the new input is new and the model can't find a knowed pattern
Related
My independent variable is a datetime object and my dependent variable is an float. Currently, I have a keras model that predicts accurately, but I found out that model.predict() only returns predictions for the values that are already known. Is there a method I can call to tell the program to use the model to predict unknown values? If there isn't please give me instructions about how to predict these unknown values.
Currently, I have a Keras model that predicts accurately, but I found out that model.predict() only returns predictions for the values that are already known
That is incorrect. A predict statement doesn't just 'search and return' results from training data. That's not how machine learning works at all. The whole reason that you build models and have a train and test dataset is to ensure you have a model that is generalizable (i.e. can be used to make predictions on unseen data, assuming the observation is coming from the same underlying distribution that the model is trained on)
In your specific case, you are using a DateTime variable an independent, which means you should refrain from using variable such as year, which are non-recurring since you can use it to make predictions about the future (model learns patterns in 2019 but 2020 may be out of its vocabulary and thus years after that are not feasible to use for predictions.)
Instead, you should engineer some features from your DateTime variable and use recurring variables which may show reveal some patterns in the dependent variable. These variables are like days of the week, months, seasons, hours of the day. Depending on what your dependent variable is, you can surely find some patterns in these.
All of this totally depends on what you are trying to model and what is the goal of the model.predict() w.r.t your problem statement. Please elaborate if possible so that people can give you more specific answers.
Your assumption is incorrect. model.predict is specifically intended to use a trained model to make predictions on a data set typically not used previously for example a test set and not a training or validation set. To use it you need to create a data set to feed to model.predict. See answer here. on how to provide input to model.predict
This is a question about a general approach rather than a specific coding problem. I'm trying to do time series forecasting with Tensorflow where features of the label timestep are known to the model. E.g. a human trying to predict a variable a week from now would know things that are going to happen in the next week that will affect that variable. So a window of 20 timesteps where the label is the 20th timestep would look something like this:
Timesteps 1-19 would each have a set of features plus the timeseries data
Timestep 20 would have a set of features which are known, plus the timeseries label which is unknown
Is there a model that could handle this sort of data? I've gone through the Tensorflow time series forecasting tutorial, done a Coursera course on Tensorflow time series forecasting and searched elsewhere but I can't find anything. I'm fairly new to this so apologies for any imprecise language.
I once tried to do this kind of TS problem by stacking a multivariate model and another machine learning model. My idea was that I use the normal TS model's output, add it as another feature in the other model that only takes the last time step's info as input. But it is complicated and might overfit a lot even if I carefully regularized the second model. The idea is that I use step 1 to window_size - 1 info to predict a rough output at step window_size, then use the info at step window_size to reduce the residual between my TS model output and the actual label; But I don't think this approach is theoretically correct and the result might be worse than using a TS model without feeding the target step's info.
I don't think tensorflow have any API for your problem because this type of problem is not a normal TS problem. Usually people would just treat this kind of problem as a regression or classification problem.
I am not an expert on this problem as well, but I just happened to attempt to solve the exact problem so this is just my personal experience...
SO I'm starting to test LSTM for time series prediction, and I've found a few different notebooks to use with my own data (here's one example)
What they all have in common is that they predict one timestep into the future, and do a really good job at matching the test data. I tried forcing an outlier in there, and the prediction almost perfectly matched it:
What's going on here? There's no way the model can learn this from the pattern of the data since it's a made up point, but supposedly by looking at the previous time steps this model will "know" an outlier is coming next? I must be missing something, because it predicts the data with an outlier just as well as the data without an outlier...
I have an already existing ARIMA (p,d,q) model fit to a time-series data (for ex, data[0:100]) using python. I would like to do forecasts (forecast[100:120]) with this model. However, given that I also have the future true data (eg: data[100:120]), how do I ensure that the multi-step forecast takes into account the future true data that I have instead of using the data it forecasted?
In essence, when forecasting I would like forecast[101] to be computed using data[100] instead of forecast[100].
I would like to avoid refitting the entire ARIMA model at every time step with the updated "history".
I fit the ARIMAX model as follows:
train, test = data[:100], data[100:]
ext_train, ext_test = external[:100], external[100:]
model = ARIMA(train, order=(p, d, q), exog=ext_train)
model_fit = model.fit(displ=False)
Now, the following code allows me to predict values for the entire dataset, including the test
forecast = model_fit.predict(end=len(data)-1, exog=external, dynamic=False)
However in this case after 100 steps, the ARIMAX predicted values quickly converge to the long-run mean (as expected, since after 100 time steps it is using the forecasted values only). I would like to know if there is a way to provide the "future" true values to give better online predictions. Something along the lines of:
forecast = model_fit.predict_fn(end = len(data)-1, exog=external, true=data, dynamic=False)
I know I can always keep refitting the ARIMAX model by doing
historical = train
historical_ext = ext_train
predictions = []
for t in range(len(test)):
model = ARIMA(historical, order=(p,d,q), exog=historical_ext)
model_fit = model.fit(disp=False)
output = model_fit.forecast(exog=ext_test[t])[0]
predictions.append(output)
observed = test[t]
historical.append(observed)
historical_ext.append(ext_test[t])
but this leads to me training the ARIMAX model again and again which doesn't make a lot of sense to me. It leads to using a lot of computational resources and is quite impractical. It further makes it difficult to evaluate the ARIMAX model cause the fitted params to keep on changing every iteration.
Is there something incorrect about my understanding/use of the ARIMAX model?
You are right, if you want to do online forecasting using new data you will need to estimate the parameters over and over again which is computationally inefficient.
One thing to note is that for the ARIMA model mainly the estimation of the parameters of the MA part of the model is computationally heavy, since these parameters are estimated using numerical optimization, not using ordinary least squares. Since after calculating the parameters once for the initial model you know what is expected for future models, since one observation won't change them much, you might be able to initialize the search for the parameters to improve computational efficiency.
Also, there may be a method to do the estimation more efficiently, since you have your old data and parameters for the model, the only thing you do is add one more datapoint. This means that you only need to calculate the theta and phi parameters for the combination of the new datapoint with all the others, while not computing the known combinations again, which would save quite some time. I very much like this book: Heij, Christiaan, et al. Econometric methods with applications in business and economics. Oxford University Press, 2004.
And this lecture might give you some idea of how this might be feasible: lecture on ARIMA parameter estimation
You would have to implement this yourself, I'm afraid. As far as I can tell, there is nothing readily available to do this.
Hope this gives you some new ideas!
As this very good blog suggests (3 facts about time series forecasting that surprise experienced machine learning practitioners):
"You need to retrain your model every time you want to generate a new prediction", it also gives the intuitive understanding of why this happens with examples. That basically highlights time-series forecasting challenge as a constant change, that needs refitting.
I was struggling with this problem. Luckily, I found a very useful discussion about it. As far as I know, the case is not supported by ARIMA in python, we need to use SARIMAX.
You can refer to the link of discussion: https://github.com/statsmodels/statsmodels/issues/2788
I have a time-series forecasting problem that I am using the statsmodels python package, I applied the ARIMA MODEL, In python sm.tsa.ARIMA(data, (p,1,q)) usually transform the data to the first different, for example if we have a raw data (y1,y2,y3,y4....), first thing ARIMA Find the first difference,(y1-y2,y2-y3,....), so it make the model from this new data (first difference data). my question when I found the model
arma_mod1=sm.tsa.ARIMA(firstdifference, (p,1,q))
I can predict the first difference data as follow
predict_oil =arma_mod11.predict('1980', '2026').
MY QUESTION: How can I predict the future raw data ( the main data not the first difference data) using Arima?
Thanks
The predict method takes an optional parameter named typ which lets you decide whether to have predictions in the original time series or in the differenced one.
You should use
predict_oil =arma_mod11.predict('1980', '2026', typ='levels')
I don't think this will be still helpful for you, but maybe it will be for others.