We Keep Coding

How to edit functional model in keras?

I am using the tf.keras.applications.efficientnet_v2.EfficientNetV2L model and I want to edit the last layers of the model to make the model a regression and classification layer. However, I am unsure of how to edit this model because it is not a linear sequential model, and thus I cannot do:
for layer in model.layers[:-2]:
model.add(layer)
as certain layers of the model have multiple inputs. Is there a way of preserving the model except the last layer so the model will diverge before the last layer?
efficentnet[:-2]
|
|
/ \
/ \
/ \
output1 output2

To enable a functional model to have a classification layer and a regression layer, you can change the model as follows. Note, there are various ways to achieve this, and this is one of them.
import tensorflow as tf
from tensorflow import keras
prev_model = keras.applications.EfficientNetV2B0(
input_tensor=keras.Input(shape=(224, 224, 3)),
include_top=False
)
Next, we will write our expected head layers, shown below.
neck_branch = keras.Sequential(
[
# we can add more layers i.e. batch norm, etc.
keras.layers.GlobalAveragePooling2D()
],
name='neck_head'
)
classification_head = keras.Sequential(
[
keras.layers.Dense(10, activation='softmax')
],
name='classification_head'
)
regression_head = keras.Sequential(
[
keras.layers.Dense(1, activation=None)
],
name='regression_head'
)
Now, we can build the desired model.
x = neck_branch(prev_model.output)
output_a = classification_head(x)
output_b = regression_head(x)
final_model = keras.Model(prev_model.inputs, [output_a, output_b])
Test
keras.utils.plot_model(final_model, expand_nested=True)
# OK
final_model(tf.ones(shape=(1, 224, 224, 3)))
# OK
Update
Based on your comments,
how you would tackle the problem if the previous model was imported from a h5 file since there I cannot declare the top layer not to be included?
If I understand your query, you have a saved model (in .h5 format) with top layers. In that case, you don't have include_top params to exclude the top branch. So, what you can do is remove the top branch of your saved model first. Here is how,
# a saved model with top layers
prev_model = keras.models.load_model('model.h5')
prev_model_with_top_remove = keras.Model(
prev_model.input ,
prev_model.layers[-4].output
)
prev_model_with_top_remove.summary()
This prev_model.layers[-4].output will remove the top branch. In the end, you will give similar output as we can get with include_top=True. Check the model summary to visually inspect.

Keras' functional API works by linking Keras tensors (hereby called KTensor) and not your everyday TF tensors.
Therefore, the first thing you need to do is feeding KTensors (created using tf.keras.Input) of proper shapes to the original model. This will trigger the forward chain, prompting the model's layers to produce their own output KTensors that are properly linked to the input KTensors. After the forward pass,
The layers will store their received/produced KTensors in their input and output attributes.
The model itself will also store the KTensors you fed to it and the corresponding final output KTensors in its inputs and outputs attributes (note the s).
Like so,
>>> from tensorflow.keras import Input
>>> from tensorflow.keras.layers import Dense
>>> from tensorflow.keras.models import Sequential, Model
>>> seq_model = Sequential()
>>> seq_model.add(Dense(1))
>>> seq_model.add(Dense(2))
>>> seq_model.add(Dense(3))
>>> hasattr(seq_model.layers[0], 'output')
False
>>> seq_model.inputs is None
True
>>> _ = seq_model(Input(shape=(10,))) # <--- Feed input KTensor to the model
>>> seq_model.layers[0].output
<KerasTensor: shape=(None, 1) dtype=float32 (created by layer 'dense')>
>>> seq_model.inputs
[<KerasTensor: shape=(None, 10) dtype=float32 (created by layer 'dense_input')>]
Once you've obtained these internal KTensors, everything becomes trivial. To extract the KTensor right before the last two layers and forward it to two different branches to form a new functional model, do
>>> intermediate_ktensor = seq_model.layers[-3].output
>>> branch_1_output = Dense(20)(intermediate_ktensor)
>>> branch_2_output = Dense(30)(intermediate_ktensor)
>>> branched_model = Model(inputs=seq_model.inputs, outputs=[branch_1_output, branch_2_output])
Note that the shapes of the KTensors you fed at the very first step must conform to the shape requirements of the layers that receive them. In my example, the input KTensor would be fed to Dense(1) layer. As Dense requires the input shape to be defined in the last dimension, the input KTensor could be of shapes, e.g., (10,) or (None,10) but not (None,) or (10, None).

failing to load weights into model XCeption-CNN

I make the same model, I load the weights. It had worked before but now it does not. Maybe I am stupid but I am thought I checked everything.
def make_model(trainable = False):
xception = Xception(include_top=False, weights='imagenet', input_shape=input_shape)
xception.trainable = trainable
inputs = Input(shape=input_shape, name='xception_input')
x = xception(inputs, training=False)
x = Flatten()(x)
x = Dense(256, activation='swish', name='xception_int', trainable=True)(x)
x = Dropout(0.6)(x)
outputs = Dense(17, activation='softmax', name='xception_output')(x)
model = Model(inputs, outputs)
return model
then I load the weights.
model.load_weights('models/xception2_weights.h5')
It has different things it says at the end of the error message. On Mac it says:
ValueError: Shapes (32,) and (3, 3, 32, 64) are incompatible
on Windows:
ValueError: axes don't match array
from what it says on the Mac, I guess it has something to do with the XCeption part. I had used load_weights in the same way successfully before I don't know why it is like this now.
If anyone can help, that would be great

I identified that when you create the model via tf.keras.applications.Xception as a functional model, has a different shape than including tf.keras.applications.Xception a Sequential model.
It also happens if you remove the top layer, adding a new classifier and then you return back to the original Xception shape with a Dense(1000) output.
If your initial model was generated with a different classifier, you added more layers and went back, it will also mismatch. Please check the axis of the original model and this new one and compare the shapes.

Keras: Modify/Rearrange elements of Input Layer to create a new Input Layer and feed the new input Layer to create a new Output

Suppose that I have a Keras Input Layer "input_layer_A" (in Python)
input_layer_A = tf.keras.layers.Input(shape=(10, ), name="input_A") # shape = (None, 10)
I also have a dense layer
dense_layer_1 = tf.keras.layers.Dense(units=1, activation='tanh', name="dense_layer_1")
Then I create an output as:
out1 = dense_layer_1(input_layer_A)
I want to modify/edit a copy of "input_layer_A" (as an example) as follows: (syntax on how to edit the Input Layer properly is a part of the question)
input_layer_B = input_layer_A #Copy the input_layer_A to create new input_layer_B
input_layer_B[0][5] = input_layer_B[0][4] #Replace 5th element with its 4th element
input_layer_B[0][4] = out1 #Replace the 4th element with previous output out1
Is it the correct way to edit/modify the input Layer? If not what is the proper way?
Then I feed “input_layer_B” to “dense_layer_1” to create a new output “out2” (kind of a recursion and it can be applied in a for loop as well)
out2 = dense_layer_1(input_layer_B)
Finally create the model
model = Model(inputs=[input_layer_A], outputs = [out1, out2])
When I run the model I get an error in the form of (related with out2: this is where the graph becomes disconnected, no problem for out1, which means there is something wrong about how I modify the "input_layer_B")
ValueError: Graph disconnected: cannot obtain value for tensor
What is the correct way to modify/edit an Input Layer so that I can still use it to feed successfully to "dense_layer_1" to end up with a connected and proper graph?
Any help, suggestion or an alternative method to accomplish what I want is appreciated.
Thank you
PS: My motivation for building such a recursive network is to model a NARX Neural Network with feedback to estimate multi-step ahead output of a dynamical system. So there can be n-many outputs in theory

Try this code:
import tensorflow as tf
input_layer_A = tf.keras.layers.Input(shape=(10, ), name="input_A") # shape = (None, 10)
dense_layer_1 = tf.keras.layers.Dense(units=1, activation='tanh', name="dense_layer_1")
out1 = dense_layer_1(input_layer_A)
out2 = tf.concat([input_layer_A[:, :4], out1, input_layer_A[:, 4][..., tf.newaxis], input_layer_A[:, 6:]], -1)
out2 = dense_layer_1(out2)
model = tf.keras.Model(inputs=[input_layer_A], outputs = [out1, out2])
inp = tf.random.uniform((1, 10))
pred = model(inp)
Note, that I've changed dimension of Dense layer (it should be 1 to get the following op possible):
input_layer_B[0][4] = out1

Keras give input to intermediate layer and get final output

My model is a simple fully connected network like this:
inp=Input(shape=(10,))
d=Dense(64, activation='relu')(inp)
d=Dense(128,activation='relu')(d)
d=Dense(256,activation='relu')(d) #want to give input here, layer3
d=Dense(512,activation='relu')(d)
d=Dense(1024,activation='relu')(d)
d=Dense(128,activation='linear')(d)
So, after saving the model I want to give input to layer 3. What I am doing right now is this:
model=load_model('blah.h5') #above described network
print(temp_input.shape) #(16,256), which is equal to what I want to give
index=3
intermediate_layer_model = Model(inputs=temp_input,
outputs=model.output)
End_output = intermediate_layer_model.predict(temp_input)
But it isn't working, i.e. I am getting errors like incompatible input, inputs should be tuple etc. The error message is:
raise TypeError('`inputs` should be a list or tuple.')
TypeError: `inputs` should be a list or tuple.
Is there any way I can pass my own inputs in middle of network and get the output instead of giving an input at the start and getting output from the end? Any help will be highly appreciated.

First you must learn that in Keras when you apply a layer on an input, a new node is created inside this layer which connects the input and output tensors. Each layer may have multiple nodes connecting different input tensors to their corresponding output tensors. To build a model, these nodes are traversed and a new graph of the model is created which consists all the nodes needed to reach output tensors from input tensors (i.e. which you specify when creating a model: model = Model(inputs=[...], outputs=[...]).
Now you would like to feed an intermediate layer of a model and get the output of the model. Since this is a new data-flow path, we need to create new nodes for each layer corresponding to this new computational graph. We can do it like this:
idx = 3 # index of desired layer
input_shape = model.layers[idx].get_input_shape_at(0) # get the input shape of desired layer
layer_input = Input(shape=input_shape) # a new input tensor to be able to feed the desired layer
# create the new nodes for each layer in the path
x = layer_input
for layer in model.layers[idx:]:
x = layer(x)
# create the model
new_model = Model(layer_input, x)
Fortunately, your model consists of one-branch and we could simply use a for loop to construct the new model. However, for more complex models it may not be easy to do so and you may need to write more codes to construct the new model.

Here is another method for achieving the same result. Initially create a new input layer and then connect it to the lower layers(with weights).
For this purpose, first re-initialize these layers(with same name) and reload the corresponding weights from the parent model using
new_model.load_weights("parent_model.hdf5", by_name=True)
This will load the required weights from the parent model.Just make sure you name your layers properly beforehand.
idx = 3
input_shape = model.layers[idx].get_input_shape_at(0) layer
new_input = Input(shape=input_shape)
d=Dense(256,activation='relu', name='layer_3')(new_input)
d=Dense(512,activation='relu', name='layer_4'))(d)
d=Dense(1024,activation='relu', name='layer_5'))(d)
d=Dense(128,activation='linear', name='layer_6'))(d)
new_model = Model(new_input, d)
new_model.load_weights("parent_model.hdf5", by_name=True)
This method will work for complex models with multiple inputs or branches.You just need to copy the same code for required layers, connect the new inputs and finally load the corresponding weights.

You can easily use keras.backend.function for this purpose:
import numpy as np
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
inp=Input(shape=(10,))
d=Dense(64, activation='relu')(inp)
d=Dense(128,activation='relu')(d)
d=Dense(256,activation='relu')(d) #want to give input here, layer3
d=Dense(512,activation='relu')(d)
d=Dense(1024,activation='relu')(d)
d=Dense(128,activation='linear')(d)
model = Model(inp, d)
foo1 = K.function(
[inp],
model.layers[2].output
)
foo2 = K.function(
[model.layers[2].output],
model.output
)
X = np.random.rand(1, 10)
X_intermediate = foo1([X])
print(np.allclose(foo2([X_intermediate]), model.predict(X)))
Sorry for ugly function naming - do it best)

I was having the same problem and the proposed solutions worked for me but I was looking for something more explicit, so here it is for future reference:
d1 = Dense(64, activation='relu')
d2 = Dense(128,activation='relu')
d3 = Dense(256,activation='relu')
d4 = Dense(512,activation='relu')
d5 = Dense(1024,activation='relu')
d6 = Dense(128,activation='linear')
inp = Input(shape=(10,))
x = d1(inp)
x = d2(x)
x = d3(x)
x = d4(x)
x = d5(x)
x = d6(x)
full_model = tf.keras.Model(inp, x)
full_model.summary()
intermediate_input = Input(shape=d3.get_input_shape_at(0)) # get shape at node 0
x = d3(intermediate_input)
x = d4(x)
x = d5(x)
x = d6(x)
partial_model = tf.keras.Model(intermediate_input, x)
partial_model.summary()
Reference:
https://keras.io/guides/functional_api/#shared-layers

Constructing a keras model

I don't understand what's happening in this code:
def construct_model(use_imagenet=True):
# line 1: how do we keep all layers of this model ?
model = keras.applications.InceptionV3(include_top=False, input_shape=(IMG_SIZE, IMG_SIZE, 3),
weights='imagenet' if use_imagenet else None) # line 1: how do we keep all layers of this model ?
new_output = keras.layers.GlobalAveragePooling2D()(model.output)
new_output = keras.layers.Dense(N_CLASSES, activation='softmax')(new_output)
model = keras.engine.training.Model(model.inputs, new_output)
return model
Specifically, my confusion is, when we call the last constructor
model = keras.engine.training.Model(model.inputs, new_output)
we specify input layer and output layer, but how does it know we want all the other layers to stay?
In other words, we append the new_output layer to the pre-trained model we load in line 1, that is the new_output layer, and then in the final constructor (final line), we just create and return a model with a specified input and output layers, but how does it know what other layers we want in between?
Side question 1): What is the difference between keras.engine.training.Model and keras.models.Model?
Side question 2): What exactly happens when we do new_layer = keras.layers.Dense(...)(prev_layer)? Does the () operation return new layer, what does it do exactly?

This model was created using the Functional API Model
Basically it works like this (perhaps if you go to the "side question 2" below before reading this it may get clearer):
You have an input tensor (you can see it as "input data" too)
You create (or reuse) a layer
You pass the input tensor to a layer (you "call" a layer with an input)
You get an output tensor
You keep working with these tensors until you have created the entire graph.
But this hasn't created a "model" yet. (One you can train and use other things).
All you have is a graph telling which tensors go where.
To create a model, you define it's start end end points.
In the example.
They take an existing model: model = keras.applications.InceptionV3(...)
They want to expand this model, so they get its output tensor: model.output
They pass this tensor as the input of a GlobalAveragePooling2D layer
They get this layer's output tensor as new_output
They pass this as input to yet another layer: Dense(N_CLASSES, ....)
And get its output as new_output (this var was replaced as they are not interested in keeping its old value...)
But, as it works with the functional API, we don't have a model yet, only a graph. In order to create a model, we use Model defining the input tensor and the output tensor:
new_model = Model(old_model.inputs, new_output)
Now you have your model.
If you use it in another var, as I did (new_model), the old model will still exist in model. And these models are sharing the same layers, in a way that whenever you train one of them, the other gets updated as well.
Question: how does it know what other layers we want in between?
When you do:
outputTensor = SomeLayer(...)(inputTensor)
you have a connection between the input and output. (Keras will use the inner tensorflow mechanism and add these tensors and nodes to the graph). The output tensor cannot exist without the input. The entire InceptionV3 model is connected from start to end. Its input tensor goes through all the layers to yield an ouptut tensor. There is only one possible way for the data to follow, and the graph is the way.
When you get the output of this model and use it to get further outputs, all your new outputs are connected to this, and thus to the first input of the model.
Probably the attribute _keras_history that is added to the tensors is closely related to how it tracks the graph.
So, doing Model(old_model.inputs, new_output) will naturally follow the only way possible: the graph.
If you try doing this with tensors that are not connected, you will get an error.
Side question 1
Prefer to import from "keras.models". Basically, this module will import from the other module:
https://github.com/keras-team/keras/blob/master/keras/models.py
Notice that the file keras/models.py imports Model from keras.engine.training. So, it's the same thing.
Side question 2
It's not new_layer = keras.layers.Dense(...)(prev_layer).
It is output_tensor = keras.layers.Dense(...)(input_tensor).
You're doing two things in the same line:
Creating a layer - with keras.layers.Dense(...)
Calling the layer with an input tensor to get an output tensor
If you wanted to use the same layer with different inputs:
denseLayer = keras.layers.Dense(...) #creating a layer
output1 = denseLayer(input1) #calling a layer with an input and getting an output
output2 = denseLayer(input2) #calling the same layer on another input
output3 = denseLayer(input3) #again
Bonus - Creating a functional model that is equal to a sequential model
If you create this sequential model:
model = Sequential()
model.add(Layer1(...., input_shape=some_shape))
model.add(Layer2(...))
model.add(Layer3(...))
You're doing exactly the same as:
inputTensor = Input(some_shape)
outputTensor = Layer1(...)(inputTensor)
outputTensor = Layer2(...)(outputTensor)
outputTensor = Layer3(...)(outputTensor)
model = Model(inputTensor,outputTensor)
What is the difference?
Well, functional API models are totally free to be build anyway you want. You can create branches:
out1 = Layer1(..)(inputTensor)
out2 = Layer2(..)(inputTensor)
You can join tensors:
joinedOut = Concatenate()([out1,out2])
With this, you can create anything you want with all kinds of fancy stuff, branches, gates, concatenations, additions, etc., which you can't do with a sequential model.
In fact, a Sequential model is also a Model, but created for a quick use in models without branches.

There's this way of building a model from a pretrained one that you may build upon.
See https://keras.io/applications/#fine-tune-inceptionv3-on-a-new-set-of-classes:
base_model = InceptionV3(weights='imagenet', include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(200, activation='softmax')(x)
model = Model(inputs=base_model.input, outputs=predictions)
for layer in base_model.layers:
layer.trainable = False
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
Each time a layer is added by an op like "x=Dense(...", information about the computational graph is updated. You can type this interactively to see what it contains:
x.graph.__dict__
You can see there's all kinds of attributes, including about previous and next layers. These are internal implementation details and possibly change over time.