We Keep Coding

How can I predict on the trained SVR model and resolve error Value Error: X.shape[1] = 1 should be equal to 22

I have datasets that have more than 2000 rows and 23 columns including the age column. I have completed all of the processes for SVR. Now I want to predict the trained SVR model is where I need to input X_test to the model? Have faced an error that is
ValueError: X.shape[1] = 1 should be equal to 22, the number of features at training time
How may I resolve this problem? How may I write code for making predictions on the trained SVR model?
import pandas as pd
import numpy as np
# Make fake dataset
dataset = pd.DataFrame(data= np.random.rand(2000,22))
dataset['age'] = np.random.randint(2, size=2000)
# Separate the target from the other features
target = dataset['age']
data = dataset.drop('age', axis = 1)
X_train, y_train = data.loc[:1000], target.loc[:1000]
X_test, y_test = data.loc[1001], target.loc[1001]
X_test = np.array(X_test).reshape((len(X_test), 1))
print(X_test.shape)
SupportVectorRefModel = SVR()
SupportVectorRefModel.fit(X_train, y_train)
y_pred = SupportVectorRefModel.predict(X_test)
Output:
ValueError: X.shape[1] = 1 should be equal to 22, the number of features at training time

Your reshaping of X_test is not correct; it should be:
X_test = np.array(X_test).reshape(1, -1)
print(X_test.shape)
# (1, 22)
With that change, the rest of your code runs OK:
y_pred = SupportVectorRefModel.predict(X_test)
y_pred
# array([0.90156667])
UPDATE
In the case as you show it in your code, obviously X_test consists of one single sample, as defined here:
X_test, y_test = data.loc[1001], target.loc[1001]
But if (as I suspect) this is not what you actually want, but in fact you want the rest of your data as your test set, you should change the definition to:
X_test, y_test = data.loc[1001:], target.loc[1001:]
X_test.shape
# (999, 22)
and without any reshaping
y_pred = SupportVectorRefModel.predict(X_test)
y_pred.shape
# (999,)
i.e. a y_pred of 999 predictions.

LSTM/GRU TImeSeries multioutput strategy forecasts give dropped values

Currently, I'm playing with Stocks Predictions task which I try to solve using LSTM/GRU.
Problem: After training LSTM/GRU I get huge drop predicted values
Model training process
Train, test data is simply generated using pd.shift in series_to_supervised function.
df['Mid'] = df['Low'] + df['High'] / 2
n_lag = 1 # Lag columns back
n_seq = 1*50 # TimeSteps to predict
seq_col = 'Mid'
seq_col_t = f'{seq_col}(t)'
split_date = '2018-01-01'
def series_to_supervised(data: pd.DataFrame,
seq_col: str,
n_in: int = 1,
n_out: int = 1,
drop_seq_col: bool = True,
dropna: bool = True):
"""Convert time series into supervised learning problem
{input sequence, forecast sequence}
"""
# input sequence (t-n, ... t-1) -> pisitive shift
for i in range(n_in, 0, -1):
data[f'{seq_col}(t-{i})'] = data[seq_col].shift(i)
# no sequence -> no shift
data[f'{seq_col}(t)'] = data[seq_col]
for i in range(1, n_out+1):
# forecast sequence (t, t+1, ... t+n) -> negative shift
data[f'{seq_col}(t+{i})'] = data[seq_col].shift(-i)
if drop_seq_col:
data = data.drop(seq_col, axis=1)
if dropna:
data.dropna(inplace=True)
return data
df = series_to_supervised(df, seq_col=seq_col, n_in=n_lag, n_out=n_seq)
mask = df.index < split_date
train, test = df[mask], df[~mask]
X_cols = ['Mid(t-1)']
y_cols = train.filter(like='Mid(t+').columns
X_train, y_train, X_test, y_test = train[X_cols], train[y_cols], test[X_cols], test[y_cols]
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range=(-1, 1))
# also returns np.ndarray
X_train = scaler.fit_transform(X_train)
X_test = scaler.fit_transform(X_test)
y_train = y_train.values
y_test = y_test.values
from keras.models import Sequential
from keras.layers import Dense, Dropout, LSTM, GRU
from keras.optimizers import Adam, RMSprop, Adamax
from keras.callbacks import ModelCheckpoint
def get_model(X, y, n_batch):
num_classes=y.shape[1]
# design network
model = Sequential()
# For Stock Predictions has to be used LSTM stateful=True
model.add(GRU(10, batch_input_shape=(n_batch, X.shape[1], X.shape[2]), stateful=True))
model.add(Dropout(0.3))
model.add(Dense(num_classes))
opt = Adam(learning_rate=0.01)
# opt = RMSprop(learning_rate=0.001)
model.compile(loss='mean_squared_error', optimizer=opt)
return model
def reshape_batch(X_train, y_train, X_test, y_test, n_batch):
# reshape training into [samples, timesteps, features]
X_train = X_train.reshape(X_train.shape[0], 1, X_train.shape[1])
X_test = X_test.reshape(X_test.shape[0], 1, X_test.shape[1])
# cut to equally divided n_batches (without reminder).
# needed for LSTM stateful=True
train_cut = X_train.shape[0] % n_batch
test_cut = X_test.shape[0] % n_batch
if train_cut > 0:
X_train = X_train[:-train_cut]
y_train = y_train[:-train_cut]
if test_cut > 0:
X_test = X_test[:-test_cut]
y_test = y_test[:-test_cut]
return X_train, y_train, X_test, y_test
# fit an LSTM network to training data
def fit_lstm(X_train: np.ndarray,
y_train: np.ndarray,
n_lag: int,
n_seq: int,
n_batch: int,
nb_epoch: int,
X_test: np.ndarray=None,
y_test: np.ndarray=None):
model = get_model(X_train, y_train, n_batch)
# fit network
history = model.fit(X_train, y_train, validation_data=(X_test, y_test), callbacks=None,
epochs=nb_epoch, batch_size=n_batch, verbose=1, shuffle=False)
print('Predict:', model.predict(X_test, batch_size=n_batch))
model.reset_states()
return model, history
n_batch = 32
nb_epoch = 40
X_train, y_train, X_test, y_test = reshape_batch(X_train, y_train, X_test, y_test, n_batch)
model, history = fit_lstm(X_train, y_train, n_lag, n_seq, n_batch, nb_epoch, X_test=X_test, y_test=y_test)
What I Have tried
Different optimizers (kinda all available in keras)
DIfferent recurrent network structures (GRU/LSTM)
Different learning rates
Different epochs from 1 to 1500
Adding/Removing Drop layers with different params (0.1-0.7)
Different LSTM/GRU amount of neurons (1-100)
Number of LSTM/GRU layers, via return_sequences params with more Drop layers.
Different number of forecasts(t+1,t+2 ... t+n) features from 1-365
Different number of lag (t-1, t-2, t-n ...) features from 1-5
Different scale normalization borders (0,1) and (-1,1)
Different n_batch values: 1,8,16,32
What can affect LSTM/GRU give so strange behaviour? And What else should I try to make it work the normal way?

How to re-frame the data-frame with multiple inputs for LSTM in Keras?

I am trying to predict the temperature for the given area (its integer number from 1 to 142) for the given date and time.
The problem is that I have CSV with the following columns:
DateTime,AreaID,Temperature
How to reframe the data-frame for LSTM (Apologise as I am a new bee for the LSTM)?
For the information, I have data for two months with a measured by the period of every 5 minutes.
I have coded LSTM for Input DateTime. But I want to include AreaID too. to predict Temperature.
The dataset created for the Training and Testing sets are using the following code block:
dataset = dataset.temperature.values #numpy.ndarray
dataset = dataset.astype('float32')
dataset = np.reshape(dataset, (-1, 1))
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)
train_size = int(len(dataset) * 0.80)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]
def create_dataset(dataset, look_back=1):
X, Y = [], []
for i in range(len(dataset)-look_back-1):
a = dataset[i:(i+look_back), 0]
X.append(a)
Y.append(dataset[i + look_back, 0])
return np.array(X), np.array(Y)
look_back = 30
X_train, Y_train = create_dataset(train, look_back)
X_test, Y_test = create_dataset(test, look_back)
# reshape input to be [samples, time steps, features]
X_train = np.reshape(X_train, (X_train.shape[0], 1, X_train.shape[1]))
X_test = np.reshape(X_test, (X_test.shape[0], 1, X_test.shape[1]))
Before this, The sample code have sorted the data frame based on DateTime like:
dataset.sort_values('timestamp', inplace=True, ascending=True)
I want to change LSTM to take two inputs
1. DateTime
2. AreaID
& One Output :
1. Temperature
How to code LSTM for this requirements? (Please help me I am a new bee in the area of neural network)

Just for hint.
You prepare new dataset into x_train and y_train
Take an starting 60 days to train the model and predict 61th days thats my logic
X_train=[]
y_train=[]
count=0
for i in range(60,train.shape[0]):
count=count+1
X_train.append(df[i-60:i])
y_train.append(train['targetcol'][i])

LSTM keras multiple features: what I did wrong?

The code below predicts Close value (stock prices) with 3 inputs: Close, Open and Volume. Dataset:
Close Open Volume
Date
2019-09-20 5489.0 5389.0 1578781
2019-09-23 5420.0 5460.0 622325
2019-09-24 5337.5 5424.0 688395
2019-09-25 5343.5 5326.5 628849
2019-09-26 5387.5 5345.0 619344
... ... ... ...
2020-03-30 4459.0 4355.0 1725236
2020-03-31 4715.0 4550.0 2433310
2020-04-01 4674.5 4596.0 1919728
2020-04-02 5050.0 4865.0 3860103
2020-04-03 5204.5 5050.0 3133078
[134 rows x 3 columns]
Info:
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 134 entries, 2019-09-20 to 2020-04-03
Data columns (total 3 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Close 134 non-null float64
1 Open 134 non-null float64
2 Volume 134 non-null int64
dtypes: float64(2), int64(1)
The question is how to correct script to get right prediction with 3 features last 10 days, because I get this:
Epoch 1/1
64/64 [==============================] - 6s 88ms/step - loss: 37135470.9219
[[32.588608]
[32.587284]
[32.586754]
[32.587196]
[32.58649 ]
[32.58663 ]
[32.586098]
[32.58682 ]
[32.586452]
[32.588108]]
rmse: 4625.457010985681
The problem remains even if I remove scaling (fit_transform) at all. In other topics I was told there is no need to scale y_train.
Full script code:
from math import sqrt
from numpy import concatenate
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense, Dropout, Embedding
from keras.layers import LSTM
import numpy as np
from datetime import datetime, timedelta
import yfinance as yf
start = (datetime.now() - timedelta(days=200)).strftime("%Y-%m-%d")
end = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
df = yf.download(tickers="LKOH.ME", start=start, end=end, interval="1d")
dataset = df.loc[start:end].filter(['Close', 'Open', 'Volume']).values
scaler = MinMaxScaler(feature_range=(0,1))
training_data_len = len(dataset) - 10 # last 10 days to test
train_data = dataset[0:int(training_data_len), :]
x_train = []
y_train = []
for i in range(60, len(train_data)):
x_train.append(train_data[i-60:i, :]) # get all 3 features
y_train.append(train_data[i, 0]) # 0 means we predict Close
x_train, y_train = np.array(x_train), np.array(y_train)
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1]*x_train.shape[2])) # convert to 2d for fit_transform()
x_train = scaler.fit_transform(x_train)
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))
model = Sequential()
# Do I need to change it to input_shape=(x_train.shape[1], 3), because of 3 features?
model.add(LSTM(50, return_sequences=True, input_shape=(x_train.shape[1], 1)))
model.add(LSTM(50))
model.add(Dense(25))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, batch_size=1, epochs=1)
test_data = dataset[training_data_len - 60:, :]
x_test = []
y_test = dataset[training_data_len:, 0]
for i in range(60, len(test_data)):
x_test.append(test_data[i-60:i, :])
x_test = np.array(x_test)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1]*x_test.shape[2]))
x_test = scaler.fit_transform(x_test)
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))
predictions = model.predict(x_test)
print(predictions)
print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))

Even though the posted answer is technically correct and provides useful references, I found it a bit annoying that the results of fitting do not make much sense (you can notice that predictions are constant, even though the y_test isn't). Yes, scaling fixes the loss - with the values in the order of 1000 the L2 measure makes any gradient-based algorithm very unstable and Rishab's answer addresses that. Here is my code snippet. With the following changes in addition to the scaling:
Use more data. I randomly chose 10000 days, but if there is more, you probably will get better results. 200 points are not sufficient to get any convergence better than the straight line.
With more points use a larger batch as otherwise, it'll take a while to fit
With the larger batch, use more epochs (although in this case, more than 3 do not produce any better convergence)
Lastly, do not just look at the RMSE, plot your data. Small RMSE doesn't necessarily mean there is any meaningful fit.
With the snippet below I've got a somewhat good fit on the train data. And foreseeing the questions: yes, I totally know that I'm overfitting the data, but that is what the convergence should be doing at the very least here as fitting a straight line is much less meaningful for this of problem. This, at least, pretends to predict something.
from math import sqrt
from numpy import concatenate
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from keras.models import Sequential
from keras.layers import Dense, Dropout, Embedding
from keras.layers import LSTM
import numpy as np
from datetime import datetime, timedelta
import yfinance as yf
from matplotlib import pyplot as plt
start = (datetime.now() - timedelta(days=10000)).strftime("%Y-%m-%d")
end = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
df = yf.download(tickers="LKOH.ME", start=start, end=end, interval="1d")
scaler = MinMaxScaler(feature_range=(0,1))
dataset = scaler.fit_transform(df.loc[start:end].filter(['Close', 'Open', 'Volume']).values)
training_data_len = len(dataset) - 10 # last 10 days to test
train_data = dataset[0:int(training_data_len), :]
x_train = []
y_train = []
for i in range(60, len(train_data)):
x_train.append(train_data[i-60:i, :]) # get all 3 features
y_train.append(train_data[i, 0]) # 0 means we predict Close
x_train, y_train = np.array(x_train), np.array(y_train)
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1]*x_train.shape[2])) # convert to 2d for fit_transform()
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))
model = Sequential()
# Do I need to change it to input_shape=(x_train.shape[1], 3), because of 3 features?
model.add(LSTM(50, return_sequences=True, input_shape=(x_train.shape[1], 1)))
model.add(LSTM(50))
model.add(Dense(25))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(x_train, y_train, batch_size=100, epochs=3)
test_data = dataset[training_data_len - 60:, :]
x_test = []
y_test = dataset[training_data_len:, 0]
for i in range(60, len(test_data)):
x_test.append(test_data[i-60:i, :])
x_test = np.array(x_test)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1]*x_test.shape[2]))
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))
predictions = model.predict(x_test)
print(predictions)
print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))
Here is the output:
>>> print(predictions)
[[0.64643383]
[0.63276255]
[0.6288108 ]
[0.6320714 ]
[0.6572328 ]
[0.6998471 ]
[0.7333 ]
[0.7492812 ]
[0.7503019 ]
[0.75124526]]
>>> print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))
rmse: 0.0712241892828221
To plot use, for training data fit
plt.plot(model.predict(x_train))
plt.plot(y_train)
plt.show()
and for test predictions
plt.plot(model.predict(x_test))
plt.plot(y_test)
plt.show()

As #rvinas has already mentioned, we need to scale the values and then use
inverse_transform to get the desired predicted outcome. You can find the reference here.
After making some small changes in the code and I was able to come up with satisfactory results. We can play around with the data scaling methodologies and model architectures to improve the results.
After some enhancements
from math import sqrt
from numpy import concatenate
import pandas as pd
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Embedding
from tensorflow.keras.layers import LSTM
from tensorflow.keras.optimizers import SGD
import numpy as np
from datetime import datetime, timedelta
import yfinance as yf
start = (datetime.now() - timedelta(days=200)).strftime("%Y-%m-%d")
end = (datetime.now() - timedelta(days=1)).strftime("%Y-%m-%d")
df = yf.download(tickers="LKOH.ME", start=start, end=end, interval="1d")
dataset = df.loc[start:end].filter(['Close', 'Open', 'Volume']).values
scaler = MinMaxScaler(feature_range=(0,1))
dataset = scaler.fit_transform(dataset)
training_data_len = len(dataset) - 10 # last 10 days to test
train_data = dataset[0:int(training_data_len), :]
x_train = []
y_train = []
for i in range(60, len(train_data)):
x_train.append(train_data[i-60:i, :]) # get all 3 features
y_train.append(train_data[i, 0]) # 0 means we predict Close
x_train, y_train = np.array(x_train), np.array(y_train)
x_train = x_train.reshape((x_train.shape[0], x_train.shape[1]*x_train.shape[2])) # convert to 2d for fit_transform()
x_train_scale = scaler.fit(x_train)
x_train = np.reshape(x_train, (x_train.shape[0], x_train.shape[1], 1))
model = Sequential()
# Do I need to change it to input_shape=(x_train.shape[1], 3), because of 3 features?
# yes, i did that.
model.add(LSTM(units=50,return_sequences=True, kernel_initializer='random_uniform', input_shape=(x_train.shape[1], 1)))
model.add(Dropout(0.2))
model.add(LSTM(units=50,return_sequences=True, kernel_initializer='random_uniform'))
model.add(Dropout(0.2))
model.add(LSTM(units=50,return_sequences=True, kernel_initializer='random_uniform'))
model.add(Dropout(0.2))
model.add(LSTM(units=50, kernel_initializer='random_uniform'))
model.add(Dropout(0.2))
model.add(Dense(units=25, activation='relu'))
model.add(Dense(units=1))
# compile model
model.compile(optimizer='adam', loss='mean_squared_error')
model.summary()
model.fit(x_train, y_train, batch_size=5, epochs=2)
test_data = dataset[training_data_len - 60:, :]
x_test = []
y_test = dataset[training_data_len:, 0]
for i in range(60, len(test_data)):
x_test.append(test_data[i-60:i, :])
x_test = np.array(x_test)
x_test = x_test.reshape((x_test.shape[0], x_test.shape[1]*x_test.shape[2]))
x_test = np.reshape(x_test, (x_test.shape[0], x_test.shape[1], 1))
predictions = model.predict(x_test)
# predictions = y_train_scale.inverse_transform(predictions)
print(predictions)
print('rmse:', np.sqrt(np.mean(((predictions - y_test) ** 2))))
Predictions 1:
start = (datetime.now() - timedelta(days=200)).strftime("%Y-%m-%d")
opt = SGD(lr=0.01, momentum=0.9, clipnorm=1.0, clipvalue=0.5)
model.compile(loss='mean_squared_error', optimizer=opt)
[[0.6151125 ]
[0.6151124 ]
[0.6151121 ]
[0.6151119 ]
[0.61511195]
[0.61511236]
[0.61511326]
[0.615114 ]
[0.61511385]
[0.6151132 ]]
rmse: 0.24450220836260966
Prediction 2:
start = (datetime.now() - timedelta(days=1000)).strftime("%Y-%m-%d")
model.compile(optimizer='adam', loss='mean_squared_error')
[[0.647125 ]
[0.6458076 ]
[0.6405072 ]
[0.63450944]
[0.6315386 ]
[0.6384401 ]
[0.65666 ]
[0.68073314]
[0.703547 ]
[0.72095114]]
rmse: 0.1236932687978488
Stock market prices are highly unpredictable and volatile. This means that there are no consistent patterns in the data that allow you to model stock prices over time near-perfectly. So this needs a lot of R&D to come up with a good strategy.
Things that you can do:
Add more training data to your model, so that it is able to generalize it better.
Make the model deeper. Play around with the model Hyperparameters to squeeze out the performance of the model. You can find a good reference here about hyperparameter tuning.
You can find more information about various other data preprocessing techniques and model architecture from reference links below:
Stock Market Predictions with LSTM in Python
Machine Learning to Predict Stock Prices

How to add more parameters while predicting using LSTM

What changes should I make in this code to predict the output based on all the parameters listed on the dataset and predict the next day opening stock price?
When I tried to run it showed an out of shape error. This code works fine with only 1 parameter.
My code is as follows:
dataset_train = pd.read_csv('ongc_train.csv')
dataset_train = dataset_train.dropna()
training_set = dataset_train.iloc[:, 1:2].value
# Creating a dataset with 60 timesteps and 1 output
X_train = []
Y_train = []
for i in range(60,2493):
X_train.append(training_set_scaled[i-60:i, 0])
Y_train.append(training_set_scaled[i, 0])
X_train, Y_train = np.array(X_train), np.array(Y_train)
# Reshaping
X_train np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))
# Fitting the RNN to the training set
regressor.fit(X_train, Y_train, epochs=100, batch_size=32)
# Getting the predicted stock price of 2017
# Concatenating the original training and test set
# Vertical concatenating of open stock prices
dataset_total = pd.concat((dataset_train['Open'], dataset_test['Open']), axis=0)
inputs = dataset_total[len(dataset_total) - len(dataset_test) - 60:].values
inputs = inputs.reshape(-1, 1)
inputs = sc.transform(inputs)
X_test = []
for i in range(60, 61):
X_test.append(inputs[i-60:i, 0])
X_test=np.array(X_test)
X_test=np.reshape(X_test, (X_test.shape[0], X_test.shape[1], 1))
predicted_stock_price = regressor.predict(X_test)
predicted_stock_price = sc.inverse_transform(predicted_stock_price)
actual = dataset_test.iloc[:, 1:2].values
print("Predicted Stock Price:",predicted_stock_price)
Thank you.