How can you add an LSTM Layer after (flattened) conv2d Layer in Tensorflow 2.0 / Keras? My Training input data has the following shape (size, sequence_length, height, width, channels). For a convolutional layer, I can only process one image a a time, for the LSTM Layer I need a sequence of features. Is there a way to reshape your data before the LSTM Layer, so you can combine both?
From an overview of shape you have provided which is (size, sequence_length, height, width, channels), it appears that you have sequences of images for each label. For this purpose, we usually make use of Conv3D. I am enclosing a sample code below:
import tensorflow as tf
SIZE = 64
SEQUENCE_LENGTH = 50
HEIGHT = 128
WIDTH = 128
CHANNELS = 3
data = tf.random.normal((SIZE, SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
input = tf.keras.layers.Input((SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
hidden = tf.keras.layers.Conv3D(32, (3, 3, 3))(input)
hidden = tf.keras.layers.Reshape((-1, 32))(hidden)
hidden = tf.keras.layers.LSTM(200)(hidden)
model = tf.keras.models.Model(inputs=input, outputs=hidden)
model.summary()
Output:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 50, 128, 128, 3)] 0
_________________________________________________________________
conv3d (Conv3D) (None, 48, 126, 126, 32) 2624
_________________________________________________________________
reshape (Reshape) (None, None, 32) 0
_________________________________________________________________
lstm (LSTM) (None, 200) 186400
=================================================================
Total params: 189,024
Trainable params: 189,024
Non-trainable params: 0
If you still wanted to make use of Conv2D which is not recommended in your case, you will have to do something like shown below. Basically, you are appending the sequence of images across the height dimension, which will make you to loose temporal dimensions.
import tensorflow as tf
SIZE = 64
SEQUENCE_LENGTH = 50
HEIGHT = 128
WIDTH = 128
CHANNELS = 3
data = tf.random.normal((SIZE, SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
input = tf.keras.layers.Input((SEQUENCE_LENGTH, HEIGHT, WIDTH, CHANNELS))
hidden = tf.keras.layers.Reshape((SEQUENCE_LENGTH * HEIGHT, WIDTH, CHANNELS))(input)
hidden = tf.keras.layers.Conv2D(32, (3, 3))(hidden)
hidden = tf.keras.layers.Reshape((-1, 32))(hidden)
hidden = tf.keras.layers.LSTM(200)(hidden)
model = tf.keras.models.Model(inputs=input, outputs=hidden)
model.summary()
Output:
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 50, 128, 128, 3)] 0
_________________________________________________________________
reshape (Reshape) (None, 6400, 128, 3) 0
_________________________________________________________________
conv2d (Conv2D) (None, 6398, 126, 32) 896
_________________________________________________________________
reshape_1 (Reshape) (None, None, 32) 0
_________________________________________________________________
lstm (LSTM) (None, 200) 186400
=================================================================
Total params: 187,296
Trainable params: 187,296
Non-trainable params: 0
_________________________________________________________________
Related
I'm trying to do an image recognition using Keras on Flask. While doing prediction, I encounter this error
ValueError: Input 0 of layer sequential_16 is incompatible with the layer: expected axis -1 of input shape to have value 24 but received input with shape [None, 150, 150, 3]
I kinda understand the problem but I'm not sure how to specify the shape. This is on Flask server & I don't do any training here. I use a model that I have trained before on Jupyter notebook.
This is the code
def predict(img):
# Preprocess input image
img_width, img_height = 150, 150
x = load_img(img, target_size=(img_width, img_height))
x = img_to_array(x)
x = np.expand_dims(x, axis=0)
# Load model
dependencies = {
'precision': Precision,
'recall': Recall
}
model = load_model('model.h5', custom_objects=dependencies)
# Predict
result = model.predict(x)[0]
label = np.argmax(result)
return label
The traceback says it happens on result = model.predict(x)[0]. Anyone knows how to approach this error? Tried googling but I don't find any similar error.
EDIT - Model Summary
Model: "sequential_16"
Layer (type) Output Shape Param #
=================================================================
dense_96 (Dense) (None, 32) 800
_________________________________________________________________
dense_97 (Dense) (None, 1024) 33792
_________________________________________________________________
dense_98 (Dense) (None, 512) 524800
_________________________________________________________________
dense_99 (Dense) (None, 256) 131328
_________________________________________________________________
dense_100 (Dense) (None, 128) 32896
_________________________________________________________________
dense_101 (Dense) (None, 3) 387
_________________________________________________________________
activation_16 (Activation) (None, 3) 0
=================================================================
Total params: 724,003
Trainable params: 724,003
Non-trainable params: 0
Ok, so your input shape is wrong. The input layer isn't shown in the model summary but it says that the first layer has 800 parameters.
This tells me that your input layer has dimensions [None, 24] because 24 * 32 (weights) + 32 (biases) = 800.
When you add the first dense layer it should be
model.add(Dense(32, input_shape=(150,150,3))
I am playing around with an NLP problem (sentence classification) and decided to use HuggingFace's TFBertModel along with Conv1D, Flatten, and Dense layers. I am using the functional API and my model compiles. However, during model.fit(), I get a shape error at the output Dense layer.
Model definition:
# Build model with a max length of 50 words in a sentence
max_len = 50
def build_model():
bert_encoder = TFBertModel.from_pretrained(model_name)
input_word_ids = tf.keras.Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
input_mask = tf.keras.Input(shape=(max_len,), dtype=tf.int32, name="input_mask")
input_type_ids = tf.keras.Input(shape=(max_len,), dtype=tf.int32, name="input_type_ids")
# Create a conv1d model. The model may not really be useful or make sense, but that's OK (for now).
embedding = bert_encoder([input_word_ids, input_mask, input_type_ids])[0]
conv_layer = tf.keras.layers.Conv1D(32, 3, activation='relu')(embedding)
dense_layer = tf.keras.layers.Dense(24, activation='relu')(conv_layer)
flatten_layer = tf.keras.layers.Flatten()(dense_layer)
output_layer = tf.keras.layers.Dense(3, activation='softmax')(flatten_layer)
model = tf.keras.Model(inputs=[input_word_ids, input_mask, input_type_ids], outputs=output_layer)
model.compile(tf.keras.optimizers.Adam(lr=1e-5), loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return model
# View model architecture
model = build_model()
model.summary()
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_word_ids (InputLayer) [(None, 50)] 0
__________________________________________________________________________________________________
input_mask (InputLayer) [(None, 50)] 0
__________________________________________________________________________________________________
input_type_ids (InputLayer) [(None, 50)] 0
__________________________________________________________________________________________________
tf_bert_model (TFBertModel) ((None, 50, 768), (N 177853440 input_word_ids[0][0]
input_mask[0][0]
input_type_ids[0][0]
__________________________________________________________________________________________________
conv1d (Conv1D) (None, 48, 32) 73760 tf_bert_model[0][0]
__________________________________________________________________________________________________
dense (Dense) (None, 48, 24) 792 conv1d[0][0]
__________________________________________________________________________________________________
flatten (Flatten) (None, 1152) 0 dense[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 3) 3459 flatten[0][0]
==================================================================================================
Total params: 177,931,451
Trainable params: 177,931,451
Non-trainable params: 0
__________________________________________________________________________________________________
# Fit model on input data
model.fit(train_input, train['label'].values, epochs = 3, verbose = 1, batch_size = 16,
validation_split = 0.2)
And this is the error message:
ValueError: Input 0 of layer dense_1 is incompatible with the layer: expected axis -1 of input shape to have value 1152 but received
input with shape [16, 6168]
I am unable to understand how the input shape to layer dense_1 (the output dense layer) can be 6168? As per the model summary, it should always be 1152.
The shape of your input is likely not as you expect. Check the shape of train_input.
I have following program taken from Internet
def my_model(input_shape):
# Define the input placeholder as a tensor with shape input_shape. Think of this as your input image!
X_input = Input(input_shape)
# Zero-Padding: pads the border of X_input with zeroes
X = ZeroPadding2D((3, 3))(X_input)
# CONV -> BN -> RELU Block applied to X
X = Conv2D(32, (7, 7), strides = (1, 1), name = 'conv0')(X)
X = BatchNormalization(axis = 3, name = 'bn0')(X)
X = Activation('relu')(X)
# MAXPOOL
X = MaxPooling2D((2, 2), name='max_pool')(X)
# FLATTEN X (means convert it to a vector) + FULLYCONNECTED
X = Flatten()(X)
X = Dense(1, activation='sigmoid', name='fc')(X)
# Create model. This creates your Keras model instance, you'll use this instance to train/test the model.
model = Model(inputs = X_input, outputs = X, name='myModel')
return model
mymodel = my_model((64,64,3))
mymodel.summary()
Here output of summary is shown as below
Layer (type) Output Shape Param #
=================================================================
input_3 (InputLayer) (None, 64, 64, 3) 0
_________________________________________________________________
zero_padding2d_3 (ZeroPaddin (None, 70, 70, 3) 0
_________________________________________________________________
conv0 (Conv2D) (None, 64, 64, 32) 4736
_________________________________________________________________
bn0 (BatchNormalization) (None, 64, 64, 32) 128
_________________________________________________________________
activation_2 (Activation) (None, 64, 64, 32) 0
_________________________________________________________________
max_pool (MaxPooling2D) (None, 32, 32, 32) 0
_________________________________________________________________
flatten_2 (Flatten) (None, 32768) 0
_________________________________________________________________
fc (Dense) (None, 1) 32769
=================================================================
Total params: 37,633
Trainable params: 37,569
Non-trainable params: 64
My question is from which layer this non-trainable params are taken i.e., 64.
Another question is how batch normalization has parameters 128?
Request your help how above numbers we got from model defined above. Thanks for the time and help.
BatchNormalization layer is composed of [gamma weights, beta weights, moving_mean(non-trainable), moving_variance(non-trainable)] and for each parameter there is one value for each element in the last axis (by default in keras, but you can change the axis if you want to).
In your code you have a size 32 in the last dimension before the BatchNormalization layer, so 32*4=128 parameters and since there are 2 non-trainable parameters there are 32*2=64 non-trainable parameters
This is a simple example that reproduces my issue in a network I am trying to deploy.
I have an image input layer (which I need to maintain), then a Dense layer, Conv2D layer and a dense layer.
The idea is that the inputs are 10x10 images and the labels are 10x10 images. Inspired by my code and this example.
import numpy as np
from keras.models import Model
from keras.layers import Input, Conv2D
#Building model
size=10
a = Input(shape=(size,size,1))
hidden = Dense(size)(a)
hidden = Conv2D(kernel_size = (3,3), filters = size*size, activation='relu', padding='same')(hidden)
outputs = Dense(size, activation='sigmoid')(hidden)
model = Model(inputs=a, outputs=outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
#Create random data and accounting for 1 channel of data
n_images=55
data = np.random.randint(0,2,(n_images,size,size,1))
labels = np.random.randint(0,2,(n_images,size,size,1))
#Fit model
model.fit(data, labels, verbose=1, batch_size=10, epochs=20)
print(model.summary())
I get the following error: ValueError: Error when checking target: expected dense_92 to have shape (10, 10, 10) but got array with shape (10, 10, 1)
I don't get an error if I change:
outputs = Dense(size, activation='sigmoid')(hidden)
with:
outputs = Dense(1, activation='sigmoid')(hidden)
No idea how Dense(1) is even valid and how it allows 10x10 output signal as model.summary() indicates:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_26 (InputLayer) (None, 10, 10, 1) 0
_________________________________________________________________
dense_93 (Dense) (None, 10, 10, 10) 20
_________________________________________________________________
conv2d_9 (Conv2D) (None, 10, 10, 100) 9100
_________________________________________________________________
dense_94 (Dense) (None, 10, 10, 1) 101
=================================================================
Total params: 9,221
Trainable params: 9,221
Non-trainable params: 0
_________________________________________________________________
None
Well, according to your comments:
what I am trying to do isn't standard. I have set of images and for
each image I want to find a binary image of the same size that if the
value of its pixel is 1 it means the feature exists in the input image
the insight wether a pixel has a feature should be taken both from
local information (extracted by a convolution layers) and global
information extracted by Dense layers.
I guess you are looking for creating a two branch model where one branch consists of convolution layers and another one is simply one or more dense layers on top of each other (although, I should mention that in my opinion one convolution network may achieve what you are looking for, because the combination of pooling and convolution layers and then maybe some up-sampling layers at the end somehow preserves both local and global information). To define such a model, you can use Keras functional API like this:
from keras import models
from keras import layers
input_image = layers.Input(shape=(10, 10, 1))
# branch one: dense layers
b1 = layers.Flatten()(input_image)
b1 = layers.Dense(64, activation='relu')(b1)
b1_out = layers.Dense(32, activation='relu')(b1)
# branch two: conv + pooling layers
b2 = layers.Conv2D(32, (3,3), activation='relu')(input_image)
b2 = layers.MaxPooling2D((2,2))(b2)
b2 = layers.Conv2D(64, (3,3), activation='relu')(b2)
b2_out = layers.MaxPooling2D((2,2))(b2)
# merge two branches
flattened_b2 = layers.Flatten()(b2_out)
merged = layers.concatenate([b1_out, flattened_b2])
# add a final dense layer
output = layers.Dense(10*10, activation='sigmoid')(merged)
output = layers.Reshape((10,10))(output)
# create the model
model = models.Model(input_image, output)
model.compile(optimizer='rmsprop', loss='binary_crossentropy')
model.summary()
Model summary:
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, 10, 10, 1) 0
__________________________________________________________________________________________________
conv2d_1 (Conv2D) (None, 8, 8, 32) 320 input_1[0][0]
__________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D) (None, 4, 4, 32) 0 conv2d_1[0][0]
__________________________________________________________________________________________________
flatten_1 (Flatten) (None, 100) 0 input_1[0][0]
__________________________________________________________________________________________________
conv2d_2 (Conv2D) (None, 2, 2, 64) 18496 max_pooling2d_1[0][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 64) 6464 flatten_1[0][0]
__________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D) (None, 1, 1, 64) 0 conv2d_2[0][0]
__________________________________________________________________________________________________
dense_2 (Dense) (None, 32) 2080 dense_1[0][0]
__________________________________________________________________________________________________
flatten_2 (Flatten) (None, 64) 0 max_pooling2d_2[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None, 96) 0 dense_2[0][0]
flatten_2[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None, 100) 9700 concatenate_1[0][0]
__________________________________________________________________________________________________
reshape_1 (Reshape) (None, 10, 10) 0 dense_3[0][0]
==================================================================================================
Total params: 37,060
Trainable params: 37,060
Non-trainable params: 0
__________________________________________________________________________________________________
Note that this is one way of achieving what you are looking for and it may or may not work for the specific problem and the data you are working on. You may modify this model (e.g. remove the pooling layers or add more dense layers) or completely use another architecture with different kind of layers (e.g. up-sampling, conv2dtrans) to reach a better accuracy. At the end, you must experiment to find the perfect solution.
Edit:
For completeness here is how to generate data and fitting the network:
n_images=10
data = np.random.randint(0,2,(n_images,size,size,1))
labels = np.random.randint(0,2,(n_images,size,size,1))
model.fit(data, labels, verbose=1, batch_size=32, epochs=20)
I'm stack and I need the wisdom of stackoverflow.
I have a two inputs neural network implemented in Keras using the Functional API, the input shapes are:
X.shape, X_size.shape, y.shape
((123, 9), (123, 2), (123, 9, 10))
So, my problem is I want to get output shape from LSTMs have 3-D shape, in order to use my y tensor. I know, I can reshape my y to 2-D shape, but I want to use it as a 3-D array.
from keras.models import Model
from keras import layers
from keras import Input
# first input
list_input = Input(shape=(None,), dtype='int32', name='li')
embedded_list = layers.Embedding(100,90)(list_input)
encoded_list = layers.LSTM(4, name = "lstm1")(embedded_list)
# second input
size_input = Input(shape=(None,), dtype='int32', name='si')
embedded_size = layers.Embedding(100,10)(size_input)
encoded_size = layers.LSTM(4, name = "lstm2")(embedded_size)
# concatenate
concatenated = layers.concatenate([encoded_size, encoded_list], axis=-1)
answer = layers.Dense(90, activation='sigmoid', name = 'outpuy_layer')(concatenated)
model = Model([list_input, size_input], answer)
model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=[f1])
Model summary:
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
si (InputLayer) (None, None) 0
____________________________________________________________________________________________________
li (InputLayer) (None, None) 0
____________________________________________________________________________________________________
embedding_16 (Embedding) (None, None, 10) 1000 si[0][0]
____________________________________________________________________________________________________
embedding_15 (Embedding) (None, None, 90) 9000 li[0][0]
____________________________________________________________________________________________________
lstm2 (LSTM) (None, 4) 240 embedding_16[0][0]
____________________________________________________________________________________________________
lstm1 (LSTM) (None, 4) 1520 embedding_15[0][0]
____________________________________________________________________________________________________
concatenate_8 (Concatenate) (None, 8) 0 lstm2[0][0]
lstm1[0][0]
____________________________________________________________________________________________________
outpuy_layer (Dense) (None, 90) 810 concatenate_8[0][0]
====================================================================================================
Total params: 12,570
Trainable params: 12,570
Non-trainable params: 0
One more time, the question is:
How to get output shape from LSTMs like (None, None, None/10)?
Keras ignores every timestep output except the last one by default, which creates a 2D array. To get a 3D array (meaning you get the output of every timestep), instantiate the layer with return_sequences set to True. In your case for example:
encoded_list = layers.LSTM(4, name = "lstm1", return_sequences=True)(embedded_list)