I am doing a simple Conv1D using TensorFlow Keras to try out a time-series dataset.
Data:
train_df = dff[:177] #get train data
tdf = train_df.shape #get shape = (177,4)
test = tf.convert_to_tensor(train_df)
Model:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv1D(filters=32,
kernel_size=1,
strides=1,
padding="causal",
activation="relu",
input_shape=tdf),
tf.keras.layers.MaxPooling1D(pool_size=2, strides=1, padding="valid")
])
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(5e-4,
decay_steps=1000000,
decay_rate=0.98,
staircase=False)
model.compile(loss=tf.keras.losses.MeanSquaredError(),
optimizer=tf.keras.optimizers.SGD(learning_rate=lr_schedule, momentum=0.8),
metrics=['mae'])
model.summary()
Summary:
Model: "sequential_13"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv1d_16 (Conv1D) (None, 177, 32) 160
_________________________________________________________________
max_pooling1d_8 (MaxPooling1 (None, 176, 32) 0
=================================================================
Total params: 160
Trainable params: 160
Non-trainable params: 0
Fit:
trainedModel = model.fit(test,
epochs=100,
steps_per_epoch=1,
verbose=1)
Error raised # Fit:
ValueError: Input 0 of layer sequential_13 is incompatible with the layer: : expected min_ndim=3, found ndim=2. Full shape received: (2, 1)
From the various SO, it was said that this is due to the input data shape. So I tried a recommendation in a SO to reshape the my data and re-input it
Reshape:
X_train=np.reshape(test,(test.shape[0], test.shape[1],1))
Error raised # Fit after reshaping:
ValueError: Input 0 of layer sequential_14 is incompatible with the layer: expected axis -1 of input shape to have value 4 but received input with shape (177, 4, 1)
I am at a loss here. What is the way to tackle this?
Current Parametric values:
tdf = (177,4)
My assumption - "You have 4 features for 177 training samples".
The Reason of current Error - The model assumes that each sample is of shape (177,4) but when you try to pass it to the model the error comes which is
ValueError: Input 0 of layer sequential_13 is incompatible with the layer: :
expected min_ndim=3, found ndim=2. Full shape received: (2, 1)
This error says that the model expected the input to have a 3D dimension which in this case is that model wants a batch size. Say a batch of 16 images of height=177 and width=4. (Although you don't have images but model expects this cause of how you specified your input shape). That means the input should have a shape - (batch_size, 177, 4).
This could be solved by passing a parameter batch_size=1 in the model.fit. As below (Without reshaping the data)
trainedModel = model.fit(data,
epochs=100,
steps_per_epoch=1,
batch_size=16,
verbose=1)
But this will give another error which is as below
ValueError: Input 0 of layer sequential_1 is incompatible with the layer: :
expected min_ndim=3, found ndim=2. Full shape received: (None, 4)
Now this error mean that the input passed to the model had some batch_size represented by None and a feature vector of shape 4 but the model expects the input to have a shape of (batch_size, height, width). Here batch_size is expected by all the models but the rest 2 are specified by us while defining the input shape. We have defined this here:
tf.keras.layers.Conv1D(filters=32,
kernel_size=1,
strides=1,
padding="causal",
activation="relu",
input_shape=tdf), # Here We save input_shape = (177,4)
As you can see the input_shape has been defined as height=177, width=4. (I used height and width for easier explanation otherwise there is no height/width only the dimension number). BUT, we wanted the model to take input of 4 features. So, now we have to change this to below:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv1D(filters=32,
kernel_size=1,
strides=1,
padding="causal",
activation="relu",
input_shape=(4,)),
tf.keras.layers.MaxPooling1D(pool_size=2, strides=1, padding="valid")
])
But now when you try to run this you will get another error as below:
ValueError: Input 0 of layer conv1d_10 is incompatible with the layer: :
expected min_ndim=3, found ndim=2. Full shape received: (None, 4)
The thing to note is the error arise from the Conv1D and this is cause this layer expects the input in 3D including the batch_size but we never specify the batch_size while creating the model the parameter input_shape should have a value like this input_shape = (dim1, dim2) but in case we only have 4 features and hence only dim1 and not dim2. In this case we will reshape our input to have the 4 as (4,1). By this we will have the dim1 = 4 and the dim2 = 1. And we will update our model as below:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv1D(filters=32,
kernel_size=1,
strides=1,
padding="causal",
activation="relu",
input_shape=(4,1)),
tf.keras.layers.MaxPooling1D(pool_size=2, strides=1, padding="valid")
])
Now we also reshape our inputs to have shape (177,4, 1) as below:
train_df = dff[:177]
train_df = train_df.values.reshape(177, 4, 1)
test = tf.convert_to_tensor(train_df)
Now we can use it to pass into the model.
trainedModel = model.fit(test,
epochs=100,
steps_per_epoch=1,
verbose=1)
Sadly this will give yet another error like below.
ValueError: No gradients provided for any variable: ['conv1d_14/kernel:0',
'conv1d_14/bias:0'].
This is cause the model don't get any Y corresponding to your input X and hence it can't compute the gradients using the loss function and hence can't train. But it can still be used to get the outputs as below:
preds = model(test)
preds.shape # Result -> TensorShape([177, 3, 32])
Related
I am new to Tensorflow and trying to figure out how to build a simple text classification model. Taking a basic model from this tutorial, I am trying to adapt it to my own custom dataset.
I have tensors with shape=(32, 2, 500) grouped into training and validation datasets with shape=(None, 2, 500).
def get_model(max_features=20000, embedding_dim=128):
# A integer input for vocab indices.
inputs = tf.keras.Input(shape=(None,), dtype="int64")
# Next, we add a layer to map those vocab indices into a space of dimensionality
#'embedding_dim'.
x = layers.Embedding(max_features, embedding_dim)(inputs)
x = layers.Dropout(0.5)(x)
# Conv1D + global max pooling
x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.Conv1D(128, 7, padding="valid", activation="relu", strides=3)(x)
x = layers.GlobalMaxPooling1D()(x)
# We add a vanilla hidden layer:
x = layers.Dense(128, activation="relu")(x)
x = layers.Dropout(0.5)(x)
# We project onto a single unit output layer, and squash it with a sigmoid:
predictions = layers.Dense(1, activation="sigmoid", name="predictions")(x)
model = tf.keras.Model(inputs, predictions)
# Compile the model with binary crossentropy loss and an adam optimizer.
model.compile(loss="binary_crossentropy", optimizer="adam", metrics=["accuracy"])
return model
I get the following warning:
WARNING:tensorflow:Model was constructed with shape (None, None) for input KerasTensor(type_spec=TensorSpec(shape=(None, None), dtype=tf.int64, name='input_16'), name='input_16', description="created by layer 'input_16'"), but it was called on an input with incompatible shape (None, 2, 500).
And the following error message:
Input 0 of layer "global_max_pooling1d_6" is incompatible with the layer: expected ndim=3, found ndim=4. Full shape received: (None, 2, 53, 128)
Call arguments received by layer "model_7" " f"(type Functional):
• inputs=tf.Tensor(shape=(None, 2, 500), dtype=int64)
• training=True
• mask=None
What do I need to change to get rid of this error and get the model working?
I want to remove only the last dense layer from an already saved model in .h5 file and add a new dense layer.
Information about the saved model:
I used transfer learning on the EfficientNet B0 model and added a dropout with 2 dense layers. The last dense layer had 3 nodes equal to my number of classes, as shown below:
inputs = tf.keras.layers.Input(shape=(IMAGE_HEIGHT, IMAGE_WIDTH, 3))
x = img_augmentation(inputs)
model = tf.keras.applications.EfficientNetB0(include_top=False, input_tensor=x, weights="imagenet")
# Freeze the pretrained weights
model.trainable = False
# Rebuild top
x = tf.keras.layers.GlobalAveragePooling2D(name="avg_pool")(model.output)
x = tf.keras.layers.BatchNormalization()(x)
x = tf.keras.layers.Dropout(0.3)(x)
x = tf.keras.layers.Dense(5, activation=tf.nn.relu)(x)
outputs = tf.keras.layers.Dense(len(class_names), activation="softmax", name="pred")(x)
After training, I saved my model as my_h5_model.h5
Main Task: I want to use the saved model architecture with its weights and replace only the last dense layer with 4 nodes dense layer.
I tried many things as suggested by the StackOverflow community as:
Iterate over all the layers except the last layer and add them to a separate already defined sequential model
new_model = Sequential()
for layer in (model.layers[:-1]):
new_model.add(layer)
But it gives an error which state:
ValueError: Exception encountered when calling layer "block1a_se_excite" (type Multiply).
A merge layer should be called on a list of inputs. Received: inputs=Tensor("Placeholder:0", shape=(None, 1, 1, 32), dtype=float32) (not a list of tensors)
Call arguments received:
• inputs=tf.Tensor(shape=(None, 1, 1, 32), dtype=float32)
I also tried the functional approach as:
input_layer = model.input
for layer in (model.layers[:-1]):
x = layer(input_layer)
which throws an as mention below:
ValueError: Exception encountered when calling layer "stem_bn" (type BatchNormalization).
Dimensions must be equal, but are 3 and 32 for '{{node stem_bn/FusedBatchNormV3}} = FusedBatchNormV3[T=DT_FLOAT, U=DT_FLOAT, data_format="NHWC", epsilon=0.001, exponential_avg_factor=1, is_training=false](Placeholder, stem_bn/ReadVariableOp, stem_bn/ReadVariableOp_1, stem_bn/FusedBatchNormV3/ReadVariableOp, stem_bn/FusedBatchNormV3/ReadVariableOp_1)' with input shapes: [?,224,224,3], [32], [32], [32], [32].
Call arguments received:
• inputs=tf.Tensor(shape=(None, 224, 224, 3), dtype=float32)
• training=False
Lastly, I did something that came to my mind
inputs = tf.keras.layers.Input(shape=(IMAGE_HEIGHT, IMAGE_WIDTH, 3))
x = img_augmentation(inputs)
x = model.layers[:-1](x)
x = keras.layers.Dense(5, name="compress_1")(x)
which simply gave an error as:
'list' object is not callable
I did some more experiments and was able to remove the last layer and add the new dense layer
# imported a pretained saved model
from tensorflow import keras
import tensorflow as tf
model = keras.models.load_model('/content/my_h5_model.h5')
# selected all layers except last one
x= model.layers[-2].output
outputs = tf.keras.layers.Dense(4, activation="softmax", name="predictions")(x)
model = tf.keras.Model(inputs = model.input, outputs = outputs)
model.summary()
In the saved model, I had 3 nodes on dense layers, but in the current model, I added 4 layers. The last layer summary is shown below:
dropout_3 (Dropout) (None, 1280) 0 ['batch_normalization_4[0][0]']
dense_3 (Dense) (None, 5) 6405 ['dropout_3[0][0]']
predictions (Dense) (None, 4) 24 ['dense_3[0][0]']
==================================================================================================
Have you tried switching between import keras and tensorflow.keras in your import? This has worked in other issues.
I'm building a RNN and I use LSTM.
The X matrix has this dimension (1824, 7) instead Y has this dim (1824, 1).
This is my model:
num_units = 64
learning_rate = 0.0001
activation_function = 'sigmoid'
adam = Adam(lr=learning_rate)
loss_function = 'mse'
batch_size = 5
num_epochs = 50
# Initialize the RNN
model = Sequential()
model.add(LSTM(units = num_units, activation=activation_function, input_shape=(1824, 7, )))
model.add(LeakyReLU(alpha=0.5))
model.add(Dropout(0.1))
model.add(Dense(units = 1))
# Compiling the RNN
model.compile(optimizer=adam, loss=loss_function, metrics=['accuracy'])
history = model.fit(
X,
y,
validation_split=0.1,
batch_size=batch_size,
epochs=num_epochs,
shuffle=False
)
I know the error is in input_shape parameter. When I try to fit the model I get this error:
ValueError: Input 0 of layer sequential is incompatible with the layer: expected ndim=3, found ndim=2. Full shape received: [None, 7]
I have seen similar questions, And I tried to apply some of that changes, such as:
input_dim = X.shape
input_dim=(7,)
input_dim=(1824, 7, 1)
But in any case I got this kind of error. How can I fix it?
As commented by #Nicolas Gervais,
Tensorflow Keras LSTM expects inputs: A 3D tensor with shape [batch, timesteps, feature].
Working sample code
import tensorflow as tf
inputs = tf.random.normal([32, 10, 8])
print(inputs.shape)
lstm = tf.keras.layers.LSTM(4)
output = lstm(inputs)
print(output.shape)
Output
(32, 10, 8)
(32, 4)
I'm inputting a 2D array of (822222, 2) into this model:
regressor = Sequential()
regressor.add (LSTM (2, activation = 'sigmoid', dropout = .1))
regressor.add (Dense (2, activation = 'tanh'))
regressor.compile (optimizer = 'rmsprop', loss = 'mean_squared_error')
regressor.fit (x = document, y = document, batch_size = 32, epochs = 1000)
but this returns an error on the last line, only accepting an array with ndim = 3:
Input 0 is incompatible with layer lstm_34: expected ndim=3, found ndim=2
I tried reshaping, but most configurations I come up with make the data unusable. What would make the model accept my inputs?
I am training a classification network with training data which has X.shape = (1119, 7) and Y.shape = (1119, 6). Below is my simple Keras network with and output dim of 6 (size of labels). The error which is returned is below the code
hidden_size = 128
model = Sequential()
model.add(Embedding(7, hidden_size))
#model.add(LSTM(128, input_shape=(1,7)))
model.add(LSTM(hidden_size, return_sequences=True))
model.add(LSTM(hidden_size, return_sequences=True))
model.add(Dense(output_dim=6, activation='softmax'))
# Compile model
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=["categorical_accuracy"])
ValueError: Error when checking target: expected dense_13 to have shape (None, 6) but got array with shape (6, 1)
I would perfer not to do this in tensorflow because I am just prototyping yet it is my first run at Keras and am confused about why it cannot take this data. I attempted to reshape the data in a number of ways in which nothing worked. Any advice as to why this isn't work would be greatly appreciated.
You should probably remove the parameter return_sequences=True from your last LSTM layer. When using return_sequences=True, the output of the LSTM layer has shape (seq_len, hidden_size). Passing this on to a Dense layer gives you an output shape of (seq_len, 6), which is incompatible with your labels. If you instead omit return_sequences=True, then your LSTM layer returns shape (hidden_size,) (it only returns the last element of the sequence) and subsequently your final Dense layer will have output shape (6,) like your labels.