We Keep Coding

How to edit functional model in keras?

I am using the tf.keras.applications.efficientnet_v2.EfficientNetV2L model and I want to edit the last layers of the model to make the model a regression and classification layer. However, I am unsure of how to edit this model because it is not a linear sequential model, and thus I cannot do:
for layer in model.layers[:-2]:
model.add(layer)
as certain layers of the model have multiple inputs. Is there a way of preserving the model except the last layer so the model will diverge before the last layer?
efficentnet[:-2]
|
|
/ \
/ \
/ \
output1 output2

To enable a functional model to have a classification layer and a regression layer, you can change the model as follows. Note, there are various ways to achieve this, and this is one of them.
import tensorflow as tf
from tensorflow import keras
prev_model = keras.applications.EfficientNetV2B0(
input_tensor=keras.Input(shape=(224, 224, 3)),
include_top=False
)
Next, we will write our expected head layers, shown below.
neck_branch = keras.Sequential(
[
# we can add more layers i.e. batch norm, etc.
keras.layers.GlobalAveragePooling2D()
],
name='neck_head'
)
classification_head = keras.Sequential(
[
keras.layers.Dense(10, activation='softmax')
],
name='classification_head'
)
regression_head = keras.Sequential(
[
keras.layers.Dense(1, activation=None)
],
name='regression_head'
)
Now, we can build the desired model.
x = neck_branch(prev_model.output)
output_a = classification_head(x)
output_b = regression_head(x)
final_model = keras.Model(prev_model.inputs, [output_a, output_b])
Test
keras.utils.plot_model(final_model, expand_nested=True)
# OK
final_model(tf.ones(shape=(1, 224, 224, 3)))
# OK
Update
Based on your comments,
how you would tackle the problem if the previous model was imported from a h5 file since there I cannot declare the top layer not to be included?
If I understand your query, you have a saved model (in .h5 format) with top layers. In that case, you don't have include_top params to exclude the top branch. So, what you can do is remove the top branch of your saved model first. Here is how,
# a saved model with top layers
prev_model = keras.models.load_model('model.h5')
prev_model_with_top_remove = keras.Model(
prev_model.input ,
prev_model.layers[-4].output
)
prev_model_with_top_remove.summary()
This prev_model.layers[-4].output will remove the top branch. In the end, you will give similar output as we can get with include_top=True. Check the model summary to visually inspect.

Keras' functional API works by linking Keras tensors (hereby called KTensor) and not your everyday TF tensors.
Therefore, the first thing you need to do is feeding KTensors (created using tf.keras.Input) of proper shapes to the original model. This will trigger the forward chain, prompting the model's layers to produce their own output KTensors that are properly linked to the input KTensors. After the forward pass,
The layers will store their received/produced KTensors in their input and output attributes.
The model itself will also store the KTensors you fed to it and the corresponding final output KTensors in its inputs and outputs attributes (note the s).
Like so,
>>> from tensorflow.keras import Input
>>> from tensorflow.keras.layers import Dense
>>> from tensorflow.keras.models import Sequential, Model
>>> seq_model = Sequential()
>>> seq_model.add(Dense(1))
>>> seq_model.add(Dense(2))
>>> seq_model.add(Dense(3))
>>> hasattr(seq_model.layers[0], 'output')
False
>>> seq_model.inputs is None
True
>>> _ = seq_model(Input(shape=(10,))) # <--- Feed input KTensor to the model
>>> seq_model.layers[0].output
<KerasTensor: shape=(None, 1) dtype=float32 (created by layer 'dense')>
>>> seq_model.inputs
[<KerasTensor: shape=(None, 10) dtype=float32 (created by layer 'dense_input')>]
Once you've obtained these internal KTensors, everything becomes trivial. To extract the KTensor right before the last two layers and forward it to two different branches to form a new functional model, do
>>> intermediate_ktensor = seq_model.layers[-3].output
>>> branch_1_output = Dense(20)(intermediate_ktensor)
>>> branch_2_output = Dense(30)(intermediate_ktensor)
>>> branched_model = Model(inputs=seq_model.inputs, outputs=[branch_1_output, branch_2_output])
Note that the shapes of the KTensors you fed at the very first step must conform to the shape requirements of the layers that receive them. In my example, the input KTensor would be fed to Dense(1) layer. As Dense requires the input shape to be defined in the last dimension, the input KTensor could be of shapes, e.g., (10,) or (None,10) but not (None,) or (10, None).

Adding Dropout Layers to Segmentation_Models Resnet34 with Keras

I want to use the Segmentation_Models UNet (with ResNet34 Backbone) for uncertainty estimation, so i want to add some Dropout Layers into the upsampling part. The Model is not Sequential, so i think i have to reconnect some outputs to the new Dropout Layers and the following layer inputs to the output of Dropout.
I'm not sure, whats the right way to do this. I'm currently trying this:
# create model
model = sm.Unet('resnet34', classes=1, activation='sigmoid', encoder_weights='imagenet')
# define optimizer, loss and metrics
optim = tf.keras.optimizers.Adam(0.001)
total_loss = sm.losses.binary_focal_dice_loss # or sm.losses.categorical_focal_dice_loss
metrics = ['accuracy', sm.metrics.IOUScore(threshold=0.5), sm.metrics.FScore(threshold=0.5)]
# get input layer
updated_model_layers = model.layers[0]
# iterate over old model and add Dropout after given Convolutions
for layer in model.layers[1:]:
# take old layer and add to new Model
updated_model_layers = layer(updated_model_layers.output)
# after some convolutions, add Dropout
if layer.name in ['decoder_stage0b_conv', 'decoder_stage0a_conv', 'decoder_stage1a_conv', 'decoder_stage1b_conv', 'decoder_stage2a_conv',
'decoder_stage2b_conv', 'decoder_stage3a_conv', 'decoder_stage3b_conv', 'decoder_stage4a_conv']:
if (uncertain):
# activate dropout in predictions
next_layer = Dropout(0.1) (updated_model_layers, training=True)
else:
# add dropout layer
next_layer = Dropout(0.1) (updated_model_layers)
# add reconnected Droput Layer
updated_model_layers = next_layer
model = Model(model.layers[0], updated_model_layers)
This throws the following Error: AttributeError: 'KerasTensor' object has no attribute 'output'
But I think I'm doing something wrong. Does anybody have a Solution for this?

There is a problem with the Resnet model you are using. It is complex and has Add and Concatenate layers (residual layers, I guess), which take as input a list of tensors from several "subnetworks". In other words, the network is not linear, so you can't walk through the model with a simple loop.
Regarding your error, in the loop of your code: layer is a layer and updated_model_layers is a tensor (functional API). Therefore, updated_model_layers.output does not exist. You confuse the two a bit

I am trying Time series forecasting using CNN , LSTM and MLP. But when i use TimeDistributed with CNN it gives dimensionality error, during fitting

I have 9 features, one output variable i.e. to be predicted, window size is 5
code works very well without "TimeDistributed" command
MODEL INPUT SHAPE: feature_tensor.shape=(1649, 5, 9) MODEL OUTPUT
SHAPE: y_train.shape= (1649,)
Thats my Code:
#Build the network model
act_fn='relu'
modelq = Sequential()
modelq.add(TimeDistributed(Conv1D(filters=105, kernel_size=2, activation=act_fn, input_shape=(None, feature_tensor.shape[1],feature_tensor.shape[2]))))
modelq.add(TimeDistributed(AveragePooling1D(pool_size=1)))
modelq.add(TimeDistributed(Flatten()))
modelq.add(LSTM(50))
modelq.add(Dense(64, activation=act_fn))
modelq.add(Dense(1))
#Compile the model
modelq.compile(optimizer='adam', loss='mean_squared_error')
modelq.fit(feature_tensor, y_train ,batch_size=1, epochs=epoch_count)
THE ERROR STATEMENT IS :
ValueError: Input 0 of layer conv1d is incompatible with the layer: : expected min_ndim=3, found ndim=2. Full shape received: (5, 9)
I feel like there is some thing wrong with dimensionality of "feature_tensor" during "Model FITTING" i.e last command... But I don't know what's wrong with it :(

Your intuition is right, the problem is the dimensionality of tensor_feature. If you take a look in the documentation of TimeDistributed you see an example with images and Conv2d layers. There the the input has to have the following shape: batch_size, time steps, x_dim, y_dim, channels. Since you use time-series you need: batch_size, time steps, 1, features. E.g. you can reshape your data by numpy:
feature_tensor = np.reshape(feature_tensor, (-1, 5, 1, 9))
However, I am not sure if it useful to combine Conv1D with TimeDistributed, since in that case you apply the convolution only on the features and not on temporal contiguous values, where a 1d Convolution should be applied.

How do I set the initial state of a keras.layers.RNN instance?

I have created a stacked keras decoder model using the following loop:
# Create the encoder
# Define an input sequence.
encoder_inputs = keras.layers.Input(shape=(None, num_input_features))
# Create a list of RNN Cells, these are then concatenated into a single layer with the RNN layer.
encoder_cells = []
for hidden_neurons in hparams['encoder_hidden_layers']:
encoder_cells.append(keras.layers.GRUCell(hidden_neurons,
kernel_regularizer=regulariser,
recurrent_regularizer=regulariser,
bias_regularizer=regulariser))
encoder = keras.layers.RNN(encoder_cells, return_state=True)
encoder_outputs_and_states = encoder(encoder_inputs)
# Discard encoder outputs and only keep the states. The outputs are of no interest to us, the encoder's job is to create
# a state describing the input sequence.
encoder_states = encoder_outputs_and_states[1:]
print(encoder_states)
if hparams['encoder_hidden_layers'][-1] != hparams['decoder_hidden_layers'][0]:
encoder_states = Dense(hparams['decoder_hidden_layers'][0])(encoder_states[-1])
# Create the decoder, the decoder input will be set to zero
decoder_inputs = keras.layers.Input(shape=(None, 1))
decoder_cells = []
for hidden_neurons in hparams['decoder_hidden_layers']:
decoder_cells.append(keras.layers.GRUCell(hidden_neurons,
kernel_regularizer=regulariser,
recurrent_regularizer=regulariser,
bias_regularizer=regulariser))
decoder = keras.layers.RNN(decoder_cells, return_sequences=True, return_state=True)
# Set the initial state of the decoder to be the output state of the encoder. his is the fundamental part of the
# encoder-decoder.
decoder_outputs_and_states = decoder(decoder_inputs, initial_state=encoder_states)
# Only select the output of the decoder (not the states)
decoder_outputs = decoder_outputs_and_states[0]
# Apply a dense layer with linear activation to set output to correct dimension and scale (tanh is default activation for
# GRU in Keras
decoder_dense = keras.layers.Dense(num_output_features,
activation='linear',
kernel_regularizer=regulariser,
bias_regularizer=regulariser)
decoder_outputs = decoder_dense(decoder_outputs)
model = keras.models.Model(inputs=[encoder_inputs, decoder_inputs], outputs=decoder_outputs)
model.compile(optimizer=optimiser, loss=loss)
model.summary()
This setup works when I have a single layer encoder and a single layer decoder where the number of neurons is the same. However it does not work when the number of layers of the decoder is more than one.
I get the following error message:
ValueError: An `initial_state` was passed that is not compatible with `cell.state_size`. Received `state_spec`=[InputSpec(shape=(None, 48), ndim=2)]; however `cell.state_size` is (48, 58)
My decoder_layers list contains the entries [48, 58]. Therefore my RNN layer that the decoder is comprised of, is a stacked GRU where the first GRU contains 48 neurons and the second contains 58. I would like to set the initial state of the first GRU. I run the states through a Dense layer so that the shape is compatible with the first layer of the decoder. The error message indicates that I am trying to set the initial state of both the first layer and the second layer when I pass the initial state keyword to the decoder RNN layer. Is this correct behaviour? Normally I would set the initial state of the first decoder layer (not built using a cell structure like this) which then would just feed it's inputs into subsequent layers. Is there a way to achieve such behaviour in keras by default when creating a keras.layers.RNN from a list of GRUCell of LSTMCells?

In my own experiments, your intial_states should have batch_size as its first dimension. In other words, each element in one batch may have a different initial state. From your code, I think you missed this dimension.

Keras - Embedding Layer and GRU Layer Shape Error

# input_shape = (137861, 21, 1)
# output_sequence_length = 21
# english_vocab_size = 199
# french_vocab_size = 344
def embed_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):
'''
Build and train a RNN model using word embedding on x and y
:param input_shape: Tuple of input shape
:param output_sequence_length: Length of output sequence
:param english_vocab_size: Number of unique English words in the dataset
:param french_vocab_size: Number of unique French words in the dataset
:return: Keras model built, but not trained
'''
learning_rate = 1e-3
model = Sequential()
model.add(Embedding(english_vocab_size, 128, input_length=output_sequence_length, input_shape=input_shape[1:]))
model.add(GRU(units=128, return_sequences=True))
model.add(TimeDistributed(Dense(french_vocab_size)))
model.add(Activation('softmax'))
model.summary()
model.compile(loss=sparse_categorical_crossentropy,
optimizer=Adam(learning_rate),
metrics=['accuracy'])
return model
When invoking this method to train a model, it gets the error:
ValueError: Input 0 is incompatible with layer gru_1: expected ndim=3, found ndim=4
How to fix the shape error between Embedding Layer and GRU Layer?

The problem is that the Embedding layer takes a 2D array as the input. However, the shape of the input array is (137861, 21, 1) which makes it a 3D array. Simply remove the last axis using squeeze() method from numpy:
data = np.squeeze(data, axis=-1)
As a side, there is no need to use TimeDistributed layer here, since the Dense layer is applied on the last axis by defualt.