We Keep Coding

High precision in training and validation but low precision in test

Hi to everyone!!
I have a problem with my model.
I am training a CNN with transfer learning using the MobileNet base model.
My dataset is made up of 3 classes "paper, scissors, rock" (8751 images, and all class are perfectly balanced) and I use it to create a hand gesture recognition model for the "paper, scissors, rock" game.
In the training phase with keras I get excellent results both with the training set and with the test set (accuracy, precision, AUC all more or less on 0.98%):
This is the last epochs.
When I go to use the validation set, these metrics have a very low result:
I think this could be due to overfitting and that I should do some tuning on my model, in fact through augmentation I increase the number of images in my dataset and then I try to modify the base model of the MobileNet by adding layers.
But things are not getting better ... Can you help me? I'm going crazy.
This is my model training code:
import matplotlib.pyplot as plt
import tensorflow
from tensorflow.keras.layers import Dense, Flatten, GlobalAveragePooling2D, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras import Sequential
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import keras
from keras.metrics import Precision, Recall
from collections import Counter
IMAGE_SIZE = (224, 224)
IMG_SHAPE = IMAGE_SIZE + (3,)
TRAIN_DATASET_DIR = "/content/PAPER_SCISSOR_ROCK/TRAIN"
TEST_DATASET_DIR = "/content/PAPER_SCISSOR_ROCK/TEST"
NUM_CLASSES = 3
BATCH_SIZE = 16
EPOCHS = 40
FC_LAYERS = [512, 512, 256, 256]
DROPOUT = 0.4
LEARNING_RATE = 0.0001
train_datagen = ImageDataGenerator(
vertical_flip=True,
validation_split=0.20,
rescale=1. / 255,
fill_mode = 'wrap',
rotation_range = 45,
brightness_range=[0.2,1.0]
#brightness_range=[1, 2],
#preprocessing_function = keras.applications.mobilenet.preprocess_input
)
# ONLY FOR TEST, SPLITT IN VALIDATION AND TEST IMAGES (TO CALCULATE PRECSION AND CONFUSION MATRIX AFTER)
test_datagen = ImageDataGenerator(
rescale=1./255,
validation_split=0.3
)
train_generator = train_datagen.flow_from_directory(
TRAIN_DATASET_DIR,
target_size=IMAGE_SIZE,
batch_size=BATCH_SIZE,
class_mode="categorical",
shuffle=True
)
val_generator = test_datagen.flow_from_directory(
TEST_DATASET_DIR,
target_size=IMAGE_SIZE,
batch_size=BATCH_SIZE,
class_mode="categorical",
subset='training',
shuffle=True
)
test_generator = test_datagen.flow_from_directory(
TEST_DATASET_DIR,
target_size=IMAGE_SIZE,
batch_size=BATCH_SIZE,
class_mode="categorical",
subset="validation",
shuffle=True
)
def build_finetune_model(base_model, dropout, fc_layers, num_classes):
# prevents weights from being updated in a given layer during training.
for layer in base_model.layers:
layer.trainable = False
# THE NEW PART SUGGESTED
for layer in base_model.layers[-30:]:
layer.trainable=True
for layer in base_model.layers:
if "BatchNormalization" in layer.__class__.__name__:
layer.trainable = False
x = base_model.output
x = Flatten()(x)
for fc in fc_layers:
print(fc)
x = Dense(fc, activation='relu')(x)
x = Dropout(dropout)(x)
preditions = Dense(num_classes, activation='softmax')(x)
finetune_model = Model(inputs = base_model.input, outputs = preditions)
return finetune_model
mobielNetV2 = tensorflow.keras.applications.MobileNetV2(input_shape=IMG_SHAPE, include_top=False, weights='imagenet')
finetune_model = build_finetune_model(mobielNetV2, dropout = DROPOUT, fc_layers = FC_LAYERS, num_classes = NUM_CLASSES)
finetune_model.compile(tensorflow.keras.optimizers.Adam(learning_rate=LEARNING_RATE), loss='categorical_crossentropy', metrics=['accuracy', 'AUC', Precision(), Recall()])
# Imposed EarlyStopping, in any era in which the model is seen to overfit, it stops.
es = keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0, patience=5, verbose=0, mode='auto')
r = finetune_model.fit_generator(train_generator, validation_data=val_generator, epochs=EPOCHS, steps_per_epoch=len(train_generator)/BATCH_SIZE,
validation_steps=len(test_generator), callbacks=[es])
print("\nSAVE THE MODEL")
finetune_model.save(f"/content/drive/My Drive/Computer_Vision/Models/MobileNet_ScissorPaperRock_{EPOCHS}_epochs.h5")
EDITED
This is the code about how I calculate the precision, recall and f1-scor of validation set:
import numpy as np
from sklearn.metrics import classification_report
# test_steps_per_epoch = np.math.ceil(val_generator.samples / val_generator.batch_size)
# print(test_steps_per_epoch)
predictions = finetune_model.predict(val_generator)
# Get most likely class
predicted_classes = np.argmax(predictions, axis=1)
print(val_generator, len(val_generator))
# Get ground-truth classes and class-labels
true_classes = val_generator.classes
#print(true_classes)
class_labels = list(val_generator.class_indices.keys())
#print(class_labels)
# Use scikit-learn to get statistics
report = classification_report(true_classes, predicted_classes, target_names=class_labels)
print(report)

Since you are fine-tuning a MobileNet V2 model, then it is a good idea to update the weights of the last few layers. MobileNet V2 is trained to classify 1000 differnet classes, but your domain contains only 3 classes of similar features. The first few layers are usually used for the general features, while the last few layers are less general, and those would affect your model the most since your domain is a lot smaller. I'd suggest that you allow the last 20%-30% layers of MobileNet V2 to update weights.

Simple tf.keras Resnet50 model not converging

I'm using the ResNet50v2 model from keras.applications for image classification but I have had persisting problems trying to get the model to converge on any meaningful accuracy. Previously, I have developed this same model with the same data in Matlab and reached around 75% accuracy but now the training just hovers around 30% accuracy and the loss does not drop. I'm thinking that there is a really simple mistake somewhere but I can't find it.
import tensorflow as tf
train_datagen = tf.keras.preprocessing.image.ImageDataGenerator(
rescale=1./224,
validation_split=0.2)
train_generator = train_datagen.flow_from_directory(main_dir,
class_mode='categorical',
batch_size=32,
target_size=(224,224),
shuffle=True,
subset='training')
validation_generator = train_datagen.flow_from_directory(main_dir,
target_size=(224, 224),
batch_size=32,
class_mode='categorical',
shuffle=True,
subset='validation')
IMG_SHAPE = (224, 224, 3)
base_model = tf.keras.applications.ResNet50V2(
input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
maxpool_layer = tf.keras.layers.GlobalMaxPooling2D()
prediction_layer = tf.keras.layers.Dense(4, activation='softmax')
model = tf.keras.Sequential([
base_model,
maxpool_layer,
prediction_layer
])
opt = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=opt,
loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
model.fit(
train_generator,
steps_per_epoch = train_generator.samples // 32,
validation_data = validation_generator,
validation_steps = validation_generator.samples // 32,
epochs = 20)

Since your last layer contains a softmax activation, your loss doesn't need from_logits=True. However, if you didn't have a softmax activation, you would need from_logits=True. This is because categorical_crossentropy handles probability outputs differently from logits.

Keras VGG16 with flow_from_directory val_acc not rising

I use keras and import VGG16 network with imagenet weights to classify male/female photos.
Strcture of directories is:
split_1/train/male/*.jpg
split_1/train/female/*.jpg
split_1/val/female/*.jpg
split_1/val/male/*.jpg
I tried most of the solutions I found over the internet but none of them worked:
changing batch_size
changing optimizers
changing class_mode/loss function
setting every layer to trainable
copying every layer from VGG to my sequential
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
from keras import applications
[...]
img_width, img_height = 224, 224
top_model_weights_path = "%s_retry2.h5" % split
train_data_dir = "%s/train" % split
validation_data_dir = "%s/val" % split
batch_size = 48
nb_train_samples = 4000
nb_validation_samples = ( 299 // batch_size ) * batch_size
epochs = 5
def train_top_model():
datagen = ImageDataGenerator(
horizontal_flip=True,
shear_range=0.2,
rescale=1. / 255)
vdatagen = ImageDataGenerator(rescale=1./255)
traingen = datagen.flow_from_directory(
train_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode='categorical',
follow_links=True,
shuffle=True)
valgen = vdatagen.flow_from_directory(
validation_data_dir,
target_size=(img_width, img_height),
batch_size=batch_size,
class_mode='categorical',
follow_links=True,
shuffle=True)
vgg_model = applications.VGG16(input_shape=(224,224,3), weights="imagenet", include_top=False)
model = Sequential()
model.add(vgg_model)
model.add(Flatten())
model.add(Dense(2, activation='softmax'))
model.compile(optimizer="rmsprop", loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit_generator(traingen,
epochs=epochs,
steps_per_epoch=nb_train_samples // batch_size,
validation_data=valgen,
validation_steps=nb_validation_samples // batch_size)
It reports actual amount of images so it finds the jpgs properly.
Accuracy in val keeps being "random" and the same (~50%) during entire training.

Try reducing the learning rate, it may be the case where your model is overshooting the minima every time and hence not able to converge.
If any kind of hyper parameter tuning doesn't work then you need to fix your data but i think male/female classification data shouldn't be that tough to learn for a CNN model with pre-trained weights.

How Many sample do you have per class???
It seems that you don't have enough data to fine tune these large scale parameters that VGG16 has. (138 million if you trainable all the layers)
I suggest :
1. for gender classification problem, Try to use an official dataset such as IMDB-WIKI
2. If you wanna use your own data first collect more label sample and after that augment all of them
3. finally, Use state of the art CNN architectures such as Xception (you can load imagenet pre-traind of xception in keras) freeze 20 first layers and fune tune others

Not able to load weights after fine tuning the model with VGG16

I have loaded the weights from VGG16 and added to my Sequential Model. I want to train the lower weights of VGG16 by freezing the top layers (Fine Tuning).
Everything was good: I was able to build the model and predict new images. But now I want to load the model, which I was unable to do.
This is what I have tried shown as following code:
model1 = applications.VGG16(weights='imagenet',
include_top=False,input_shape=(img_width,img_height,3))
train_datagen = ImageDataGenerator(rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
test_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(train_data_dir,
target_size=(img_width, img_height),
batch_size=size_batch,
class_mode='binary',
shuffle=False)
# repeat with the validation data
test_generator = test_datagen.flow_from_directory(validation_data_dir,
target_size=(img_width, img_height),
batch_size=size_batch,
class_mode='binary',
shuffle=False)
model = Sequential()
model.add(Flatten(input_shape=model1.output_shape[1:]))
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(1, activation='sigmoid'))
new_model=Sequential()
for l in model1.layers:
new_model.add(l)
new_model.add(model)
for layer in new_model.layers[:25]:
layer.trainable=False
new_model.compile(optimizer=optimizers.SGD(lr=1e-3,
momentum=0.9),loss='binary_crossentropy',
metrics=['accuracy'])
checkpoint = ModelCheckpoint(fine_tuned_model_path, monitor='val_acc',
verbose=1, save_best_only=True,
save_weights_only=False, mode='auto')
# fine-tune the model
fit=new_model.fit_generator(train_generator,
steps_per_epoch=33,
nb_epoch=1,
validation_data=test_generator,
verbose=1,callbacks=[checkpoint])
I then was trying to load the model:
load_model("C:/Users/hi/POC/Fine_Tune/model.h5")
This is the error I am receiving:
ValueError: You are trying to load a weight file containing 14 layers
into a model with 1 layers.

According to Keras issue 8898, this error can be avoided by editing the Keras code keras/applications/vgg16.py
so that the line(s) that used to read
model.load_weights(weights_path) now read model.load_weights(weights_path, by_name=True)
I have found this to work for imagenet weights with other Applications models as well, e.g. nasnet.

I don't see why you had to define a new model and load the previous layers of VGG16 into your new model.
The best work around that l would advice is freezing the layer of the VGG16
architecture you want and have the ones you want as trainable layers, as you did in the last for loop
This will ultimately result in you removing the two for loops you have
embedded inside.
# the way you loaded your images and did not include_top layer
model1 = applications.VGG16(weights='imagenet', include_top=False, input_shape = (img_width, img_height, 3))
#to see the structure of your architecture
model.summary()
#freezing the layers you do not want for training in your architecture
for layer in model1.layers[:25]:
layer.trainable = False
#the rest is the same from here on forth with the exclusion of the two for loops
#which you need to remove as they are no longer required.
train_datagen = ImageDataGenerator(rescale=1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
#etc...

Epoch does not start while training CNN with keras VGGFace Framework

I am trying to use VGG Face implementation with keras framework on my own dataset consisting of 12 classes of face images. I have applied augmentation on some classes with very less data in training set.
After finetuning with resnet50, when I try to train my model, it gets stuck in epoch i.e., it does not start to train but keep displaying Epoch 1/50.
Here's what it looks like:
Layer (type) Output Shape Param #
=================================================================
model_1 (Model) (None, 12) 23585740
=================================================================
Total params: 23,585,740
Trainable params: 23,532,620
Non-trainable params: 53,120
_________________________________________________________________
Found 1774 images belonging to 12 classes.
Found 313 images belonging to 12 classes.
Epoch 1/50
Here's my code:
train_data_path = 'dataset_cfps/train'
validation_data_path = 'dataset_cfps/validation'
#Parametres
img_width, img_height = 224, 224
vggface = VGGFace(model='resnet50', include_top=False, input_shape=(img_width, img_height, 3))
#vgg_model = VGGFace(include_top=False, input_shape=(224, 224, 3))
last_layer = vggface.get_layer('avg_pool').output
x = Flatten(name='flatten')(last_layer)
out = Dense(12, activation='sigmoid', name='classifier')(x)
custom_vgg_model = Model(vggface.input, out)
# Create the model
model = models.Sequential()
# Add the convolutional base model
model.add(custom_vgg_model)
# Add new layers
# model.add(layers.Flatten())
# model.add(layers.Dense(1024, activation='relu'))
# model.add(BatchNormalization())
# model.add(layers.Dropout(0.5))
# model.add(layers.Dense(12, activation='sigmoid'))
# Show a summary of the model. Check the number of trainable parameters
model.summary()
train_datagen = ImageDataGenerator(
rescale=1./255,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
validation_datagen = ImageDataGenerator(rescale=1./255)
train_batchsize = 16
val_batchsize = 16
train_generator = train_datagen.flow_from_directory(
train_data_path,
target_size=(img_width, img_height),
batch_size=train_batchsize,
class_mode='categorical')
validation_generator = validation_datagen.flow_from_directory(
validation_data_path,
target_size=(img_width, img_height),
batch_size=val_batchsize,
class_mode='categorical',
shuffle=True)
# Compile the model
model.compile(loss='categorical_crossentropy',
optimizer=optimizers.SGD(lr=1e-3),
metrics=['acc'])
# Train the model
history = model.fit_generator(
train_generator,
steps_per_epoch=train_generator.samples/train_generator.batch_size ,
epochs=50,
validation_data=validation_generator,
validation_steps=validation_generator.samples/validation_generator.batch_size,
verbose=1)
# Save the model
model.save('facenet_resnet.h5')
Does anyone know what could be the possible problem? And how can I make my model better(if there's something I could do). Feel free to suggest me improvements.

Waiting did not solve it, I solved it by restarting the whole program.

Just you wait few hours(based on your gpu). Finally it will tell the loss and val_loss per each epochs.