We Keep Coding

How to get the latent vector as an output from a cnn model before training to the fully connected layer?

I am working on CNN model using Tensorflow frames in google collab. I am unable to extract the latent vectors from the convolutional layers. I want to extract the output of the convolutional layers, the layers before fully connected layer.
I have tried with the following code
a = dropout()(classifier_model.output)
print(a)
I am unable to understand the solution suggested on the link Stackoverflow solution to print the value of tensorflow object after applying a-conv-pool-layer
Anyone with any suggestion?

You can use get_layer method of the Model class to get a layer by its name, find bellow an example with a dummy 1D CNN and a binary classifier :
timesteps = 100
nfeatures = 2
# build the model using the functional API
# example of a 1D CNN inspired by the your stack overflow link, but using a model instead of successive *raw* layers
# the values of the Conv1D filters and kernels are different
input = Input((timesteps, nfeatures))
p = Conv1D(filters=16, kernel_size=10)(input)
p = ReLU()(p)
p = MaxPool1D(pool_size=2)(p)
p = Conv1D(filters=32, kernel_size=10)(p)
p = ReLU()(p)
p = MaxPool1D(pool_size=2)(p)
p = Conv1D(filters=64, kernel_size=10)(p)
p = ReLU()(p)
p = MaxPool1D(pool_size=2, name='conv1Dfeat')(p) # give a name to the CNN output
# fully connected part
p = Flatten()(p)
p = Dense(10)(p)
# could add a dropout layer to ease optimization
finaloutput = Dense(1, activation='sigmoid')(p)
# full model
model = Model(inputs=input, outputs=finaloutput)
# compile network, i.e. define optimizer, loss and metrics
model.compile(optimizer='Adam', loss='binary_crossentropy', metrics=['accuracy'])
model.summary()
You need to train the model using the fit method with some data. Then you can get the output of the layer which name is conv1Dfeat (the last layer of the convolutive part) by defining the model:
modelCNN = Model(inputs=input, outputs=model.get_layer('conv1Dfeat').output)
modelCNN.summary()
If you want to get the output of the convolutive part, let's say based on a single numpy input array of shape (timesteps, nfeatures), you can use the predict of the Model class on batched data:
data = np.random.normal(size=(timesteps, nfeatures)) # dummy data
data_tf = tf.expand_dims(data, axis=0) # convert to TF tensor and add batch dimension at the same time
cnn_out_np = modelCNN.predict(data_tf)
cnn_out_np = np.squeeze(cnn_out_np, axis=0) # remove batch dimension
print(cnn_out_np.shape)
(4, 64)

Keras Model with 2 inputs during training, but only 1 during inferencing

A similar question has been asked earlier by someone but the answer was not satisfactory.
Given a model made using Functional API in keras.
During training the model we have two inputs and one output. One input is image. Another input is an array of costs that is needed for custom loss function.
During inferencing however we will get only image as input and no costs are there. Hence only one input and one output.
How to adapt the same model which has been trained on two inputs for inferencing ?
The model during training is somewhat like this :
input1 = Input(shape=(64,64,3)) #RGB Image
input2 = Input(shape=(4,))#Costs associated with the image, input to the custom loss function
conv1 = Conv2D(16, 3 , padding = 'same', activation = 'relu')(input1)
#Other layers
output = Dense(6)(x) # last layer gives classification output
model = Model(inputs = [input1, input2] , outputs = output)
model.compile(loss = custom_loss_function(input2) , optimizer = 'adam')
This is the model during training.
What to do during inferencing when only one input for the image is needed and no cost inputs are present ?

Just use a "dummy" input if it doesn't affect the forward pass

You can wrap your inference model in a training model.
In pseudo-code:
def make_model():
input1 = Input(...)
conv1 = Conv2D(16, 3 , padding = 'same', activation = 'relu')(input1)
#Other layers
output = Dense(6)(x) # last layer gives classification output
return keras.Model(input1, output)
def make_train_model():
input1 = Input(...)
input2 = Input(...)
m_inner = make_model()
output = m_inner(input1)
model = keras.Model([input1, input2], output)
model.compile(...)
return model, m_inner
You can train using model and save the inner model for inference.

How do I get this model to predict the multi label classification value?

How do I get this model to predict the multi label classification value based on train input and test input? There are 3 classifications, which are good, bad, and ugly. train_input is a dictionary that holds the train dataset. test_input is a variable that holds the value of 241.43 which a value of 'good', 'bad', or 'ugly' should be predicted from, in this case probably the predicted value should be 'bad'.
from keras.layers import Input, Dense
from keras.models import Model
''' 3 multi label classification using deep learning '''
classification_labels_3 = ['good', 'bad', 'ugly']
train_input = {100.23:'good', 234.76:'bad', 500.90:'ugly'}
test_input = 241.43
'''
# Define the keras architecture of your model in 'build_model' and return it. Compilation must be
done in 'compile_model'.
# input_shapes - dictionary of shapes per input as defined train_input dictionary
# n_classes - For classification, number of target classes
'''
def build_model(input_shapes, n_classes=None):
'''
# This input will receive all the train_input features
# sent to 'main'
'''
input_main = Input(shape=input_shapes['main'], name='main')
x = Dense(64, activation='relu')(input_main)
x = Dense(64, activation='relu')(x)
predictions = Dense(n_classes, activation='softmax')(x)
'''
# The 'inputs' parameter of your model must contain the
# full list of inputs used in the architecture
'''
model = Model(inputs=[input_main], outputs=predictions)
return model
'''
# Compile your model and return it
# model - model defined in 'build_model'
'''
def compile_model(model):
'''
# The loss function depends on the type of problem you solve.
# 'categorical_crossentropy' is appropriate for a multiclass classification.
'''
model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
return model
'''
#print(build_model(input_shapes=,n_classes=))
#print(compile(source, filename, mode))
'''

For predicting multi label class you have to use softmax activation function. I can see that you are already using it, so you only have to define the number of classes of it's function:
predictions = Dense(n_classes, activation='softmax')(x)
As you can see, n_classes is defined as a variable of the function build_model, so just adjust this variable when you call the function and you will build a model with the outputs you specified.
But first, you have to obtain dataset in a dataframe format. Try do something like that:
val=[]
for e in train_input:
val.append(e)
clas=[]
for e in train_input:
clas.append(train_input[e])
df=pd.DataFrame(val, columns=["values"])
df["clas"]=clas
You will obtain a dataframe like this:
values clas
0 100.23 good
1 234.76 bad
2 500.90 ugl

Combine outputs of two Pre Trained models (trained on different dataset) and use some form of binary classifier to predict images

I have two Pre-Trained models.
Model_1 = Inception Model with Imagenet Dataset (1000 classes)
My_Model = Inception Model trained with a custom dataset (20 classes) via Transfer Learning and Fine-Tuning
I would like to combine the outputs of both models (Model_1 and My_Model) in a new layer.
The new layer should use some binary classifier to tell whether to use Model_1 or My_Model for prediction based on the input image.
For Example:
If I try to predict a "Dog" image, the binary classifier which combines both models should say that I need to use Model_1 to predict the Dog image (since My_Model dataset is not trained with Dog image) whereas Model_1 is trained with Dog images.
Can anyone tell me how to achieve this? Some example implementation or code snippet will be helpful.
Thanks

To do this you need to make a combined model and then train the combined model on another custom dataset here is an example of what the combined model can look like. To make the dataset, simply take each image and decide which model you'd like to use and then you can train the output of the combined model to give a positive value for one model and a negative value for the other model. hope it helps
import numpy as np
import pandas as pd
import keras
from keras.layers import Dense, Flatten, Concatenate
from tensorflow.python.client import device_lib
# check for my gpu
print(device_lib.list_local_devices())
# making some models like the ones you have
input_shape = (10000, 3)
m1_input = Input(shape = input_shape, name = "m1_input")
fc = Flatten()(m1_input)
m1_output = Dense(1000, activation='sigmoid',name = "m1_output")(fc)
Model_1 = Model(m1_input,m1_output)
m2_input = Input(shape = input_shape, name = "m2_input")
fc = Flatten()(m2_input)
m2_output = Dense(20, activation='sigmoid',name = "m2_output")(fc)
My_Model = Model(m2_input,m2_output)
# set the trained models to be untrainable
for layer in Model_1.layers:
layer.trainable = False
for layer in My_Model.layers:
layer.trainable = False
#build a combined model
combined_model_input = Input(shape = input_shape, name = "combined_model_input")
m1_predict = Model_1(combined_model_input)
m2_predict = My_Model(combined_model_input)
combined = Concatenate()([m1_predict, m2_predict])
fc = Dense(500, activation='sigmoid',name = "fc1")(combined)
fc = Dense(100, activation='sigmoid',name = "fc2")(fc)
output_layer = Dense(1, activation='tanh',name = "fc3")(fc)
model = Model(combined_model_input, output_layer)
#check the number of parameters that are trainable
print(model.summary())
#psudocode to show how to make a training set for the combined model:
combined_model_y= []
for im in images:
if class_of(im) in list_of_my_model_classes:
combined_model_y.append(1)
else:
combined_model_y.append(-1)
combined_model_y = np.array(combined_model_y)
# then train the combined model:
model.compile('adam', loss = 'binary_crossentropy')
model.fit(images, combined_model_y, ....)

Extract the output of cnn

I have trained a cnn model to classify images of dog and cat
it is giving 98% accuracy
But I want to visualize the output of cnn layer i.e the features from which my cnn is predicting whether it is a dog or a cat
If there any way to visualize the output of cnn?

You can divide your model into two models:
Previous Model:
input = Input(...)
# Your Layers
output = Dense(1)
old_model = Model(inputs=[input], output)
New Model:
input = Input(...)
#Add the first layers and the CNN here
cnn_layer = Conv2D(...)
feature_extraction_model = Model(inputs=[input], outputs=cnn_layer)
input_cnn = Input(...) # The shape of your CNN output
# Add the classification layer here
output = Dense(1)
classifier_model = Model(inputs=[input_cnn], outputs=output)
Now you define the new model as a combination of: feature_extraction_model and classifier_model
new_model = Model(inputs=[input], outputs=classifier_model(input_cnn))
# Train the model
new_model.fit(x, y)
Now you can have access to the CNNlayer post training:
cnn_output = feature_extraction_model.predict(x)