My data is a matrix of 400k observations, where each observation is a single row of data, 200 elements in length. My output data is 400k 1-hot vectors corresponding to 11 output classes. I would like to format my data for a keras sequential model. (Keras 2.0.4, Python 3)
train_values.shape
>>> (400000, 200)
data_labels.shape
>>> (400000, 11)
Attempt 1:
model = Sequential()
model.add(LSTM(100, input_shape = (200,), return_sequences=True))
model.add(Dense(output_dim=11, input_dim=100, activation='softmax'))
model.compile(loss='categorical_crossentropy',optimizer='rmsprop',metrics=['accuracy'])
model.fit(train_values, train_labels, batch_size=200, epochs=10)
score = model.evaluate(test_values, test_labels, batch_size=200)
Error 1:
----> 3 model.add(LSTM(100, input_shape = (200,), return_sequences=True))
ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2
Attempt 2:
model = Sequential()
model.add(Embedding(400000,200))
model.add(LSTM(100, input_shape = (200,), return_sequences=True))
model.add(Dense(output_dim=11, input_dim=100, activation='softmax'))
model.compile(loss='categorical_crossentropy',optimizer='rmsprop',metrics=['accuracy'])
model.fit(train_values, train_labels, batch_size=200, epochs=10)
score = model.evaluate(test_values, test_labels, batch_size=200)
Error 2:
----> 7 model.fit(train_values, train_labels, batch_size=200, epochs=10)
ValueError: Error when checking target: expected dense_1 to have 3 dimensions, but got array with shape (400000, 11)
Conclusion: Is my error with the input formatting, the calling of my layers, or something else entirely that I'm unaware of?
Related
I'm building a model to classify text into one of 9 layers, and am having this error when running it. Activation 1 seems to refer to the Convolutional layer's input, but I'm unsure about what's wrong with the input.
num_classes=9
Y_train = keras.utils.to_categorical(Y_train, num_classes)
#Reshape data to add new dimension
X_train = X_train.reshape((100, 150, 1))
Y_train = Y_train.reshape((100, 9, 1))
model = Sequential()
model.add(Conv1d(1, kernel_size=3, activation='relu', input_shape=(None, 1)))
model.add(Dense(num_classes))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x=X_train,y=Y_train, epochs=200, batch_size=20)
Running this results in the following error:
"ValueError: Error when checking target: expected activation_1 to have shape (None, 9) but got array with shape (9,1)
There are several typos and bugs in your code.
Y_train = Y_train.reshape((100,9))
Since you reshape X_train to (100,150,1), I guess your input step is 150, and channel is 1. So for the Conv1D, (there is a typo in your code), input_shape=(150,1).
You need to flatten your output of conv1d before feeding into Dense layer.
import keras
from keras import Sequential
from keras.layers import Conv1D, Dense, Flatten
X_train = np.random.normal(size=(100,150))
Y_train = np.random.randint(0,9,size=100)
num_classes=9
Y_train = keras.utils.to_categorical(Y_train, num_classes)
#Reshape data to add new dimension
X_train = X_train.reshape((100, 150, 1))
Y_train = Y_train.reshape((100, 9))
model = Sequential()
model.add(Conv1D(2, kernel_size=3, activation='relu', input_shape=(150,1)))
model.add(Flatten())
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(x=X_train,y=Y_train, epochs=200, batch_size=20)
I would like to build a model (RNN >> LSTM) with an Embedding layer for a categorical feature (Item ID), My training set looks so:
train_x = [[[184563.1], [184324.1], [187853.1], [174963.1], [181663.1]], [[…],[…],[…],[…],[…]], …]
I predict the sixth item ID:
train_y = [0,1,2, …., 12691]
I have 12692 unique item IDs, length of timesteps = 5 and this is a classification task.
This is a brief summary for what I've done so far: (Please correct me if I'm wrong)
One-hot-encoding for the categorical feature:
train_x = [[[1 0 0 … 0 0 0], [0 1 0 … 0 0 0], [0 0 1 … 0 0 0], […], […]], [[…],[…],[…],[…],[…]], …]
Build model:
model = Sequential()
model.add(Embedding(input_dim=12692 , output_dim=250, input_length=5))
model.add(LSTM(128, return_sequences=True)
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(LSTM(128, return_sequences=True))
model.add(Dropout(0.1))
model.add(BatchNormalization())
model.add(LSTM(128))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(12692, activation='softmax'))
opt = tf.keras.optimizers.Adam(lr=0.001, decay=1e-6)
model.compile(
loss='sparse_categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])
print(model.summary())
history = model.fit(
train_x, train_y,
batch_size=64,
epochs=epochs,
validation_data=(validation_x, validation_y))
score = model.evaluate(validation_x, validation_y, verbose=0)
I get this model summary:
Train on 131204 samples, validate on 107904 samples
But after that, this error appears:
ValueError: Error when checking input: expected embedding_input to have 2 dimensions, but got array with shape (131204, 5, 12692)
Where is my mistake and what would be the solution?
The embedding layer turns positive integers (indexes) into dense vectors of fixed size (Docs). So your train_x is not one-hot-encoded but the integer representing its index in the vocab. It will be the integer corresponding to the categorical feature.
train_x.shape will be (No:of sample X 5) --> Each representing the index of of the categorical feature
train_y.shape will be (No:of sample) --> Each representing the index of the sixth item in your time series.
Working sample
import numpy as np
import keras
from keras.layers import Embedding, LSTM, Dense
n_samples = 100
train_x = np.random.randint(0,12692,size=(n_samples ,5))
train_y = np.random.randint(0,12692,size=(n_samples))
model = keras.models.Sequential()
model.add(Embedding(input_dim=12692+1, output_dim=250, input_length=5))
model.add(LSTM(128, return_sequences=True))
model.add(LSTM(32))
model.add(Dense(32, activation='relu'))
model.add(Dense(12692, activation='softmax'))
opt = keras.optimizers.Adam(lr=0.001, decay=1e-6)
model.compile(
loss='sparse_categorical_crossentropy',
optimizer=opt,
metrics=['accuracy'])
print(model.summary())
history = model.fit(
train_x, train_y,
batch_size=64,
epochs=32)
I am new to machine learning. I am having trouble getting my data into my network.
This is the error that I am receiving:
ValueError: Error when checking input: expected cu_dnnlstm_22_input to have 3 dimensions, but got array with shape (2101, 17)
I have tried adding model.add(Flatten()) before the dense layer. I would really appreciate your help!
BATCH_SIZE = 64
test_size_length = int(len(main_df)*TESTING_SIZE)
training_df = main_df[:test_size_length]
validation_df = main_df[test_size_length:]
train_x, train_y = training_df.drop('target',1).to_numpy(), training_df['target'].tolist()
validation_x, validation_y = validation_df.drop('target',1).to_numpy(), validation_df['target'].tolist()
#train_x.shape is (2101, 17)
model = Sequential()
# model.add(Flatten())
model.add(CuDNNLSTM(128, input_shape=(train_x.shape), return_sequences=True))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(CuDNNLSTM(128, return_sequences=True))
model.add(Dropout(0.1))
model.add(BatchNormalization())
model.add(CuDNNLSTM(128))
model.add(Dropout(0.2))
model.add(BatchNormalization())
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(2, activation='softmax'))
opt = tf.keras.optimizers.Adam(lr=0.001, decay=1e-6)
# Compile model
model.compile(
loss='sparse_categorical_crossentropy',
optimizer=opt,
metrics=['accuracy']
)
tensorboard = TensorBoard(log_dir="logs/{}".format(NAME))
filepath = "RNN_Final-{epoch:02d}-{val_acc:.3f}" # unique file name that will include the epoch and the validation acc for that epoch
checkpoint = ModelCheckpoint("models/{}.model".format(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')) # saves only the best ones
# Train model
history = model.fit(
train_x, train_y,
batch_size=BATCH_SIZE,
epochs=EPOCHS,
validation_data=(validation_x, validation_y),
callbacks=[tensorboard, checkpoint],
)
# Score model
score = model.evaluate(validation_x, validation_y, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
# Save model
model.save("models/{}".format(NAME))
The input to your LSTM layer (CuDNNLSTM) should have shape: (batch_size, timesteps, input_dim).
It looks like you are missing one of these dimensions.
Often we can oversee the last dimension in the case where the input dimension is 1. If this is the case with your model (if you are predicting from a sequence of single numbers), then you might consider expanding the dimensions before the CuDNNLSTM layer with something like this:
model.add(Lambda(lambda t: tf.expand_dims(t, axis=-1)))
model.add(CuDNNLSTM(128))
Without knowing the problem you are working on it's hard to know whether this is a valid way forward but certainly you should keep in mind the required shape of a LSTM layer and reshape/expand dims accordingly.
I am new in Deep Learning, and I'm trying to create this simple LSTM architecture in Keras using Google Colab:
Input layer of 12 input neurons
One Recurrent hidden layer of 1 hidden neuron for now
Output layer of 1 output neuron
The original error was:
ValueError: Error when checking input: expected lstm_2_input to have 3 dimensions, but got array with shape (4982, 12).
Then I tried:
input_shape=train_x.shape[1:]
But I got:
ValueError: Input 0 is incompatible with layer lstm_1: expected ndim=3, found ndim=2
Then I tried:
X_train = np.reshape(X_train, X_train.shape + (1,))
But I got another error again:
ValueError: Must pass 2-d input
Then I tried:
train_x = np.reshape(train_x, (train_x.shape[0], 1, train_x.shape[1]))
But it didn't work:
Must pass 2-d input
Here is my original code:
df_tea = pd.read_excel('cleaned2Apr2019pt2.xlsx')
df_tea.head()
train_x, valid_x = model_selection.train_test_split(df_tea,random_state=2, stratify=df_tea['offer_Offer'])
train_x.shape #(4982, 12)
valid_x.shape #(1661, 12)
model = Sequential()
model.add(LSTM(32, input_shape=train_x.shape, return_sequences=True))
model.add(LSTM(32, return_sequences=True))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['acc'])
history = model.fit(train_x, valid_x,
epochs=10,
batch_size=128,
validation_split=0.2)
I have looked through several stackoverflow and github suggestions for a similar problem, but none works.
Could someone help me please as I don't understand why all these methods failed.
According to your code, timesteps = 1 (in LSTM terminology), input_dim = 12. Hence you should let
input_shape = (1,12)
A general formula is input_shape = (None, timesteps, input_dim) or
input_shape = (timesteps, input_dim)
An example:
import numpy as np
from keras.layers import LSTM, Dense
from keras.models import Sequential
n_examples = 4982 #number of examples
n_ft = 12 #number of features
train_x= np.random.randn(n_examples, n_ft)
#valid_x.shape #(1661, 12)
model = Sequential()
model.add(LSTM(32, input_shape=(1, n_ft), return_sequences=True))
model.add(LSTM(32, return_sequences=True))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['acc'])
model.summary()
I'm trying to create a keras LSTM to predict time series. My x_train is shaped like 3000,15,10 (Examples, Timesteps, Features), y_train like 3000,15,1 and I'm trying to build a many to many model (10 input features per sequence make 1 output / sequence).
The code I'm using is this:
model = Sequential()
model.add(LSTM(
10,
input_shape=(15, 10),
return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(
100,
return_sequences=True))
model.add(Dropout(0.2))
model.add(Dense(1, activation='linear'))
model.compile(loss="mse", optimizer="rmsprop")
model.fit(
X_train, y_train,
batch_size=512, nb_epoch=1, validation_split=0.05)
However, I can't fit the model when using :
model.add(Dense(1, activation='linear'))
>> Error when checking model target: expected dense_1 to have 2 dimensions, but got array with shape (3000, 15, 1)
or when formatting it this way:
model.add(Dense(1))
model.add(Activation("linear"))
>> Error when checking model target: expected activation_1 to have 2 dimensions, but got array with shape (3000, 15, 1)
I already tried flattening the model ( model.add(Flatten()) ) before adding the dense layer but that just gives me ValueError: Input 0 is incompatible with layer flatten_1: expected ndim >= 3, found ndim=2. This confuses me because I think my data actually is 3 dimensional, isn't it?
The code originated from https://github.com/Vict0rSch/deep_learning/tree/master/keras/recurrent
In case of keras < 2.0: you need to use TimeDistributed wrapper in order to apply it element-wise to a sequence.
In case of keras >= 2.0: Dense layer is applied element-wise by default.
Since you updated your keras version and your error messages changed, here is what works on my machine (Keras 2.0.x)
This works:
model = Sequential()
model.add(LSTM(10,input_shape=(15, 10), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM( 100, return_sequences=True))
model.add(Dropout(0.2))
model.add(Dense(1, activation='linear'))
This also works:
model = Sequential()
model.add(LSTM(10,input_shape=(15, 10), return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM( 100, return_sequences=True))
model.add(Dropout(0.2))
model.add(LSTM(1,return_sequences=True, activation='linear'))
Testing with:
x = np.ones((3000,15,10))
y = np.ones((3000,15,1))
Compiling and training with:
model.compile(optimizer='adam',loss='mse')
model.fit(x,y,epochs=4,verbose=2)