We Keep Coding

How to make all ResNet50 layers appear in model.summary?

I want to see all the ResNet50 layers and not the resnet50 block in model.summary. I saw a similar issue How can I use tf.keras.Model.summary to see the layers of a child model which in a father model?, but I'm having difficulties in adapting it to my model since it uses ResNet50 instead of MobileNet.
This is my model:
Resnet = ResNet101(include_top=False, weights='imagenet', input_shape=(224, 224, 3))
model = tf.keras.Sequential(Resnet)
model.add(tf.keras.layers.GlobalAveragePooling2D())
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(units=no_classes, activation="softmax"))
This is the code I have right now based on the previous issue.
class ResNet50(tf.keras.Sequential):
def __init__(self, input_shape=(224, 224, 3), classes=8):
super(ResNet50, self).__init__()
self.backbone_model = [layer for layer in
tf.keras.applications.ResNet50(input_shape, include_top=False, pooling='avg').layers]
self.classificator = tf.keras.layers.Dense(classes,activation='relu', name='classificator')
def call(self, inputs):
x = inputs
for layer in self.backbone_model:
x = layer(x)
x = self.classificator(x)
return x
model = ResNet50()
model.add(tf.keras.layers.GlobalAveragePooling2D())
model.add(tf.keras.layers.Dropout(0.5))
model.add(tf.keras.layers.Dense(units=no_classes, activation="softmax"))
How do I initialize it with imagenet weights? is it done automatically? Is the pooling average and the activation relu for Resnet50? Do I have to add more layers?

The model has a layers parameter so try to run this loop. I don't remember exactly but if model.layers is a list then.
summary = []
for layer, value in zip(model.layers, model.summary()):
try:
summary.append(layer.summary())
except:
summary.append(value)
you might need to modify it but it should be something similar.

CNN Visualization of output layers with pre-trained model

I trained my model and saved it:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.preprocessing.image import img_to_array
new_model=tf.keras.models.load_model('the_model.h5')
new_model.summary()
img = load_img('e.jpg',target_size=(227,227))
img=img_to_array(img)
img = np.expand_dims(img,axis=0)
img=img/255.
print(img.shape)
#prints out (1,227,227,3) the expected shapes
so the architecture of my model is the following one, i'm using pre-trained resnet50
backbone = ResNet50(input_shape=(227,227,3),weights='imagenet', include_top=False)
model = Sequential()
model.add(backbone)
model.add(GlobalAveragePooling2D())
model.add(Dropout(0.5))
model.add(Dense(64,activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1,activation='sigmoid'))
i tried visualize outputs of hidden layers, however with keras or keract i can't get the outputs
with keras :
layer_outputs=[]
for layer in new_model.layers:
if layer.name=='resnet50':
temp = [l.output for l in layer.layers]
layer_outputs=temp
else:
layer_outputs.append(layer.output)
activation_model = Model(inputs=new_model.input, outputs=layer_outputs)
the error caused by the last line :
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 227, 227, 3), dtype=float32) at layer "input_1". The following previous layers were accessed without issue: []
i feel like my model inputs are matching with layer_outputs so i don't really understand the error, indeed when i'm checking :
print(new_model.layers[0].input)
#prints out :Tensor("input_1:0", shape=(None, 227, 227, 3), dtype=float32)
print(layer_outputs[0])
#prints out : Tensor("input_1:0", shape=(None, 227, 227, 3), dtype=float32)
when using keract :
a = keract.get_activations(new_model, img) # with just one sample.
keract.display_activations(a, directory='f', save=True)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'input_1' with dtype float and shape [?,227,227,3]
Any idea of how i can fix it, or another viable solution to get outputs from hidden layers using pre-trained model?
Thanks,

Okey, i found a convenient solution to my problem.
Indeed i think this problem occurs because my sequential model is itself made up of another model (resnet).
Since i didn't add many layers on top of the pre-trained resnet model, i just decided to visualize the feature maps from the resnet model
img = load_img('e.jpg',target_size=(227,227))
img=img_to_array(img)
img = np.expand_dims(img,axis=0)
img=img/255.
print(img.shape)
loaded=tf.keras.models.load_model('age_gender_train.h5')
layer_outputs=[ layer.output for layer in loaded.layers[0].layers]
res = loaded.layers[0]
activation_model = Model(inputs=res.input, outputs=layer_outputs)
activations=activation_model.predict(img)
img = np.squeeze(img,axis=0)
Then you can easily display features maps using activations variable.
Note that since you have the outputs of the resnet model, it might be possible to get the feature maps from the layers on top by repeating the process. Using the output of resnet as input and removing the resnet model from the layer_outputs.(i did not try this, could not work)
Hope it could help someone

How do I force my training data to match the output shape of my neural network?

I am trying to use the transfer learning example on keras.applications using VGG19. I am trying to train on the cifar10 dataset, so 10 classes. My model is (conceptually) simple as it's just VGG 19 minus the top three layers and then some extra layers that are trainable.
import tensorflow as tf
from keras.utils import to_categorical
from keras.applications import VGG19
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D, Input
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
#%%
# Specify input and number of classes
input_tensor = Input(shape=(32, 32, 3))
num_classes=10
#Load the data (cifar100), if label mode is fine then 100 classes
(X_train,y_train),(X_test,y_test)=tf.keras.datasets.cifar10.load_data()
#One_Hot_encode y data
y_test=to_categorical(y_test,num_classes=num_classes,dtype='int32')
y_train=to_categorical(y_train,num_classes=num_classes,dtype='int32')
#%%
# create the base pre-trained model
base_model = VGG19(weights='imagenet', include_top=False,
input_tensor=input_tensor)
# Add a fully connected layer and then a logistic layer
x = base_model.output
# # let's add a fully-connected layer
x = Dense(1024, activation='relu',name='Fully_Connected')(x)
# and a logistic layer -- let's say we have 200 classes
predictions = Dense(num_classes, activation='softmax',name='Logistic')(x)
# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
layer.trainable = False
# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer='adam', loss='categorical_crossentropy',metrics=['accuracy'])
# train the model on the new data for a few epochs
model.fit(X_train,y_train,epochs=10)
#%%
model.evaluate(X_test,y_test)
Now when I try to train using X_train [dimensions of (50000, 32, 32, 3)] and y_test (dimensions of (50000,10),
I get an error:
ValueError: Error when checking target: expected Logistic to have 4
dimensions, but got array with shape (50000, 10)
So for some reason the model isn't realizing that its output shape should be a 1x10 vector with one-hot encoding for the 10 classes.
How can I make it so that the dimensions agree? I don't fully understand the dimensions of output that keras is expecting here. When I do model.summary() the Logistic layer yields that the output shape should be (None, 1, 1, 10), which when flattened should just give a

VGG19 without the top layers does not return a fully-connected layer and instead returns a 2D feature space (output of a Conv2D/max pooling2d I believe). You'll probably want to place a flatten after the VGG, that'd be the best practical choice, as it will make your output shape (None,10).
Otherwise, you could do
y_train = np.reshape(y_train, (50000,1,1,10))

Tensorflow: Weights of non-trainable model layers are updated

I have a trained model which is created using Keras. On this model I want to apply transfer learning by freezing all but the last convolutional layer. However, when I fit the model after freezing the layers I notice that some of the freezed layers have different weights. How can I avoid this?
I tried to freeze the entire model with model.trainable = False but this also didn't work out.
I am using python 3.5.0, tensorflow 1.0.1 and Keras 2.0.3
Example script
import os
import timeit
import datetime
import numpy as np
from keras.layers.core import Activation, Reshape, Permute
from keras.layers.convolutional import Convolution2D, MaxPooling2D, UpSampling2D, ZeroPadding2D
from keras.layers.normalization import BatchNormalization
from keras.optimizers import Adam
from keras import models
from keras import backend as K
K.set_image_dim_ordering('th')
def conv_model(input_shape, data_shape, kern_size, filt_size, pad_size,\
maxpool_size, n_classes, compile_model=True):
"""
Create a small conv neural network
input_shape: input shape of the images
data_shape: 1d shape of the data
kern_size: Kernel size used in all convolutional2d layers
filt_size: Filter size of the first and last convolutional2d layer
pad_size: size of padding
maxpool_size: Pool size of all maxpooling2d and upsampling2d layers
n_classes: number of output classes
compile_model: True if the model should be compiled
output: Keras deep learning model
"""
#keep track of compilation time
start_time = timeit.default_timer()
model = models.Sequential()
# Add a noise layer to get a denoising autoencoder. This helps avoid overfitting
model.add(ZeroPadding2D(padding=(pad_size, pad_size), input_shape=input_shape))
#Encoding layers
model.add(Convolution2D(filt_size, kern_size, kern_size, border_mode='valid'))
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(maxpool_size, maxpool_size)))
model.add(UpSampling2D(size=(maxpool_size, maxpool_size)))
model.add(ZeroPadding2D(padding=(pad_size, pad_size)))
model.add(Convolution2D(filt_size, kern_size, kern_size, border_mode='valid'))
model.add(BatchNormalization())
model.add(Convolution2D(n_classes, 1, 1, border_mode='valid'))
model.add(Reshape((n_classes, data_shape), input_shape=(n_classes,)+input_shape[1:]))
model.add(Permute((2, 1)))
model.add(Activation('softmax'))
if compile_model:
model.compile(loss="categorical_crossentropy", optimizer='adam', metrics=["accuracy"])
print('Model compiled in {0} seconds'.format(datetime.timedelta(seconds=round(\
timeit.default_timer() - start_time))))
return model
if __name__ == '__main__':
#Create some random training data
train_data = np.random.randint(0, 10, 3*512*512*20, dtype='uint8').reshape(-1, 3, 512, 512)
train_labels = np.random.randint(0, 1, 7*512*512*20, dtype='uint8').reshape(-1, 512*512, 7)
#Get dims of the data
data_dims = train_data.shape[2:]
data_shape = np.prod(data_dims)
#Create initial model
initial_model = conv_model((train_data.shape[1], train_data.shape[2], train_data.shape[3]),\
data_shape, 3, 4, 1, 2, train_labels.shape[-1])
#Train initial model on first part of the training data
initial_model.fit(train_data[0:10], train_labels[0:10], verbose=2)
#Store initial weights
initial_weights = initial_model.get_weights()
#Create transfer learning model
transf_model = conv_model((train_data.shape[1], train_data.shape[2], train_data.shape[3]),\
data_shape, 3, 4, 1, 2, train_labels.shape[-1], False)
#Set transfer model weights
transf_model.set_weights(initial_weights)
#Set all layers trainable to False (except final conv layer)
for layer in transf_model.layers:
layer.trainable = False
transf_model.layers[9].trainable = True
print(transf_model.layers[9])
#Compile model
transf_model.compile(loss="categorical_crossentropy", optimizer=Adam(lr=1e-4),\
metrics=["accuracy"])
#Train model on second part of the data
transf_model.fit(train_data[10:20], train_labels[10:20], verbose=2)
#Store transfer model weights
transf_weights = transf_model.get_weights()
#Check where the weights have changed
for i in range(len(initial_weights)):
update_w = np.sum(initial_weights[i] != transf_weights[i])
if update_w != 0:
print(str(update_w)+' updated weights for layer '+str(transf_model.layers[i]))

Once you compiled your model - you lost your previous weights, as they were resampled. You need to first transfer them, set weights to be not trainable and then compile it:
#Compile model
transf_model.set_weights(initial_weights)
#Set all layers trainable to False (except final conv layer)
for layer in transf_model.layers:
layer.trainable = False
transf_model.layers[9].trainable = True
transf_model.compile(loss="categorical_crossentropy", optimizer=Adam(lr=1e-4),\
metrics=["accuracy"])
Otherwise - weights would change as they are resampled.
EDIT:
The model should be compiled after changes - because during compilation keras is setting all trainable / not trainable weights in a list which is not further changed.

You should Upgrade Keras to Keras v2.1.3
This issue is just solved and this very last feature of freezing BatchNormalization layers is now available in the recent release:
trainable attribute in BatchNormalization now disables the updates of the batch statistics (i.e. if trainable == False the layer will now run 100% in inference mode).
The Reason of error:
In the previous versions, the variance and mean parameters of BatchNormalization layers couldn't set untrainable and it didn't work, although you sat layer.trainable = False.
Now, it works!

Fine tuning pretrained model in keras

I want to use a pretrained imagenet VGG16 model in keras and add my own small convnet on top. I am only interested in the features, not the predictions
from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
import numpy as np
import os
from keras.models import Model
from keras.models import Sequential
from keras.layers import Convolution2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
load images from directory (the dir contains 4 images)
IF = '/home/ubu/files/png/'
files = os.listdir(IF)
imgs = [img_to_array(load_img(IF + p, target_size=[224,224])) for p in files]
im = np.array(imgs)
load the base model, preprocess input and get the features
base_model = VGG16(weights='imagenet', include_top=False)
x = preprocess_input(aa)
features = base_model.predict(x)
this works, and I get the features for my images on the pretrained VGG.
I now want to finetune the model and add some convolutional layers.
I read https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html and https://keras.io/applications/ but cannot quite bring them together.
adding my model on top:
x = base_model.output
x = Convolution2D(32, 3, 3)(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Convolution2D(32, 3, 3)(x)
x = Activation('relu')(x)
feat = MaxPooling2D(pool_size=(2, 2))(x)
building the complete model
model_complete = Model(input=base_model.input, output=feat)
stop base layers from being learned
for layer in base_model.layers:
layer.trainable = False
new model
model_complete.compile(optimizer='rmsprop',
loss='binary_crossentropy')
now fit the new model, the model is 4 images and [1,0,1,0] are the class labels.
But this is obviously wrong:
model_complete.fit_generator((x, [1,0,1,0]), samples_per_epoch=100, nb_epoch=2)
ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None
How is this done?
How would I do it if I only wanted to replace the last convolutional block (conv block5 in VGG16) instead of adding something?
How would I only train the bottleneck features?
The features output features has shape (4, 512, 7, 7). There are four images, but what is in the other dimensions? How would I reduce that to a (1,x) array?

Fitting model
The problem with your generator code is that the fit_generator method expects a generator function to generate the data for fitting which you don't provide.
You can either define a generator as done in the tutorial that you have linked to or create the data and labels yourself and fit your model yourself:
model_complete.fit(images, labels, batch_size=100, nb_epoch=2)
where images are your generated training images and labels are the corresponding labels.
Removing last layer
Assuming you have a model variable and the "pop" method described below, you can do model = pop(model) to remove the last layer.
Training only specific layers
As you have done in your code, you can do:
for layer in base_model.layers:
layer.trainable = False
Then you can "unfreeze" and layer that you want by changing their trainable property to True.
Changing dimensions
To change the output to a 1D array you can use the Flatten layer
The pop method
def pop(model):
'''Removes a layer instance on top of the layer stack.
This code is thanks to #joelthchao https://github.com/fchollet/keras/issues/2371#issuecomment-211734276
'''
if not model.outputs:
raise Exception('Sequential model cannot be popped: model is empty.')
else:
model.layers.pop()
if not model.layers:
model.outputs = []
model.inbound_nodes = []
model.outbound_nodes = []
else:
model.layers[-1].outbound_nodes = []
model.outputs = [model.layers[-1].output]
model.built = False
return model

Use model.fit(X, y) to train on your dataset as explained here: https://keras.io/models/model/#fit
Additionally you should add a Flatten layer and a dense layer with an ouput shape of 1 to get the correct result shape.