I am currently training a model using the Cars196 dataset from Stanford. However, with the dataset correctly imported and recognized by TensorFlow, my accuracy is still 0. I used a similar approach to train the model on other datasets and it works. Did I do anything wrong?
Here is my code
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import csv
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Flatten,Dense
car_dir = './src/'
test_dir = './src/cars_test/'
train_dir = './src/cars_train/'
train_labels_file = './src/labels-train.csv'
test_labels_file = './src/labels-test.csv'
IMG_SIZE = (150,150)
def read_labels(label_file:str):
pathAndClass = list()
with open(label_file) as csv_file:
reader = csv.reader(csv_file)
next(reader) # skip first row
for row in reader:
pathAndClass.append([row[5].lower(), row[4]])
return pd.DataFrame(pathAndClass,columns=['path', 'class'])
pathAndClass = read_labels(train_labels_file)
n_classes = np.size(np.unique(pathAndClass['class']))
pathAndClass['path'] = pathAndClass['path'].astype(str)
pathAndClass['class'] = pathAndClass['class'].astype(str)
data_gen = ImageDataGenerator(rescale = 1.0/255.0, validation_split=0.25)
BATCH_SIZE = 32
index_list = []
for i in range(0, n_classes):
index_list.append(str(i))
train_flow = data_gen.flow_from_dataframe(
dataframe=pathAndClass,
x_col='path',
y_col='class',
directory=train_dir,
subset="training",
seed=42,
target_size=IMG_SIZE,
batch_size=BATCH_SIZE,
shuffle=True,
classes=index_list,
class_mode='categorical')
valid_flow = data_gen.flow_from_dataframe(
dataframe=pathAndClass,
x_col='path',
y_col='class',
directory=train_dir,
subset="validation",
seed=42,
target_size=IMG_SIZE,
batch_size=BATCH_SIZE,
shuffle=True,
classes=index_list,
class_mode='categorical')
model_nn = Sequential()
model_nn.add(Flatten(input_shape=(150,150, 3)))
model_nn.add(Dense(300, activation="relu"))
model_nn.add(Dense(n_classes, activation="softmax"))
model_nn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model_nn.summary())
training = model_nn.fit(
train_flow,
steps_per_epoch=train_flow.n//train_flow.batch_size,
epochs=10,
validation_data=valid_flow,
validation_steps=valid_flow.n//valid_flow.batch_size)
print(model_nn.evaluate(train_flow))
plt.plot(training.history['accuracy'])
plt.plot(training.history['val_accuracy'])
plt.plot(training.history['loss'])
plt.plot(training.history['val_loss'])
plt.title('Model accuracy/loss')
plt.ylabel('accuracy/loss')
plt.xlabel('epoch')
plt.legend(['accuracy', 'val_accuracy', 'loss', 'val_loss'])
plt.show()
The output I got
Found 6078 validated image filenames belonging to 196 classes.
Found 2026 validated image filenames belonging to 196 classes.
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten_1 (Flatten) (None, 67500) 0
_________________________________________________________________
dense_2 (Dense) (None, 300) 20250300
_________________________________________________________________
dense_3 (Dense) (None, 196) 58996
=================================================================
Total params: 20,309,296
Trainable params: 20,309,296
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/10
189/189 [==============================] - 68s 361ms/step - loss: 9.6809 - accuracy: 0.0036 - val_loss: 5.2785 - val_accuracy: 0.0030
Epoch 2/10
189/189 [==============================] - 58s 307ms/step - loss: 5.2770 - accuracy: 0.0055 - val_loss: 5.2785 - val_accuracy: 0.0089
Epoch 3/10
189/189 [==============================] - 58s 307ms/step - loss: 5.2743 - accuracy: 0.0083 - val_loss: 5.2793 - val_accuracy: 0.0104
Epoch 4/10
189/189 [==============================] - 58s 306ms/step - loss: 5.2728 - accuracy: 0.0089 - val_loss: 5.2800 - val_accuracy: 0.0089
Epoch 5/10
189/189 [==============================] - 58s 307ms/step - loss: 5.2710 - accuracy: 0.0084 - val_loss: 5.2806 - val_accuracy: 0.0089
Epoch 6/10
189/189 [==============================] - 57s 305ms/step - loss: 5.2698 - accuracy: 0.0086 - val_loss: 5.2815 - val_accuracy: 0.0089
Epoch 7/10
189/189 [==============================] - 58s 307ms/step - loss: 5.2695 - accuracy: 0.0083 - val_loss: 5.2822 - val_accuracy: 0.0089
Epoch 8/10
189/189 [==============================] - 58s 310ms/step - loss: 5.2681 - accuracy: 0.0086 - val_loss: 5.2834 - val_accuracy: 0.0089
Epoch 9/10
189/189 [==============================] - 58s 306ms/step - loss: 5.2679 - accuracy: 0.0083 - val_loss: 5.2840 - val_accuracy: 0.0089
Epoch 10/10
189/189 [==============================] - 58s 308ms/step - loss: 5.2669 - accuracy: 0.0083 - val_loss: 5.2848 - val_accuracy: 0.0089
1578/Unknown - 339s 215ms/step - loss: 5.2657 - accuracy: 0.0085
Update 1
I have increased the training sample by decreasing the batch size to 8. I tried to train the model again. However, the accuracy is still nearly 0.
Epoch 1/10
759/759 [==============================] - 112s 147ms/step - loss: 7.6876 - accuracy: 0.0051 - val_loss: 5.2779 - val_accuracy: 0.0089
Epoch 2/10
759/759 [==============================] - 112s 148ms/step - loss: 5.2728 - accuracy: 0.0086 - val_loss: 5.2792 - val_accuracy: 0.0089
Epoch 3/10
759/759 [==============================] - 112s 148ms/step - loss: 5.2695 - accuracy: 0.0087 - val_loss: 5.2808 - val_accuracy: 0.0089
Epoch 4/10
759/759 [==============================] - 109s 143ms/step - loss: 5.2671 - accuracy: 0.0087 - val_loss: 5.2828 - val_accuracy: 0.0089
Epoch 5/10
759/759 [==============================] - 111s 146ms/step - loss: 5.2661 - accuracy: 0.0086 - val_loss: 5.2844 - val_accuracy: 0.0089
Epoch 6/10
759/759 [==============================] - 114s 151ms/step - loss: 5.2648 - accuracy: 0.0089 - val_loss: 5.2862 - val_accuracy: 0.0089
Epoch 7/10
759/759 [==============================] - 118s 156ms/step - loss: 5.2646 - accuracy: 0.0086 - val_loss: 5.2881 - val_accuracy: 0.0089
Epoch 8/10
759/759 [==============================] - 117s 155ms/step - loss: 5.2639 - accuracy: 0.0087 - val_loss: 5.2891 - val_accuracy: 0.0089
Epoch 9/10
759/759 [==============================] - 115s 151ms/step - loss: 5.2635 - accuracy: 0.0087 - val_loss: 5.2903 - val_accuracy: 0.0089
Epoch 10/10
759/759 [==============================] - 112s 147ms/step - loss: 5.2634 - accuracy: 0.0086 - val_loss: 5.2915 - val_accuracy: 0.0089
2390/Unknown - 141s 59ms/step - loss: 5.2611 - accuracy: 0.0088
Indeed the last dataset I used had less classes but more samples. Maybe there is another model that fits my dataset, any suggestions?
For computer vision problems, you want to look at Convolutional Neural Networks. If you're unfamiliar with them, they learn to identify features in images. Examples could be edges and textures in early layers, and then wheels, windows, doors, etc in later layers.
For this problem, I would suggest using an existing, pretrained network such as MobileNet V2 or InceptionNetV3 as a backbone, and then building your own classifier on top. This tutorial on the Tensorflow website will get you started https://www.tensorflow.org/tutorials/images/transfer_learning#create_the_base_model_from_the_pre-trained_convnets
Here's an excerpt from this tutorial:
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
Then adding your model code from above, you could try:
model = tf.keras.Sequential([
base_model,
Flatten(input_shape=(150,150, 3)),
Dense(300, activation="relu")),
Dense(n_classes, activation='softmax')
])
model_nn.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
This is the model I've used on similar datasets and got reasonable accuracy with it:
base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
include_top=False,
weights='imagenet')
model = tf.keras.Sequential([
base_model,
GlobalAveragePooling2D(),,
Dense(n_classes, activation='softmax')
])
In you current model, you are not trying to extract any features from the images. A single hidden layer with 300 neurons is nowhere near enough to be able to learn the features in images and give meaningful results.
You also need to check your input image size. MobileNet V2 works well with 224x224 colour images.
As per other comments, you will need to use the full dataset, you are not going to get any meaningful results with a few hundred images.
I would suggest that the ~6000 Training Samples for almost 200 classes is simply way to little for the model to work well.
The model did ~2000 wheight updates (200 in each epoch), which is way to few for it to learn to distinguish between ~200 classifications.
Maybe you had less classes and more training data in the other training sets?
Related
I am learning python deep learning tools on Tensorflow official websites.
Trying to build several Text-Classification network, do as tutorials. But LSTM does not work as except.
import numpy as np
import tensorflow_datasets as tfds
import tensorflow as tf
from tensorflow.keras import utils
from tensorflow.keras import losses
import matplotlib.pyplot as plt
seed = 42
BATCH_SIZE = 64
train_ds = utils.text_dataset_from_directory(
'stack_overflow_16k/train',
validation_split=0.2,
subset='training',
batch_size=BATCH_SIZE,
seed=seed)
val_ds = utils.text_dataset_from_directory(
'stack_overflow_16k/train',
validation_split=0.2,
subset='validation',
batch_size=BATCH_SIZE,
seed=seed)
test_ds = utils.text_dataset_from_directory(
'stack_overflow_16k/test',
batch_size=BATCH_SIZE)
class_names = train_ds.class_names
train_ds = train_ds.prefetch(buffer_size=tf.data.AUTOTUNE)
val_ds = val_ds.prefetch(buffer_size=tf.data.AUTOTUNE)
test_ds = test_ds.prefetch(buffer_size=tf.data.AUTOTUNE)
VOCAB_SIZE = 1000
MAX_SEQUENCE_LENGTH = 500
encoder = tf.keras.layers.TextVectorization(
max_tokens=VOCAB_SIZE,
output_sequence_length=MAX_SEQUENCE_LENGTH)
encoder.adapt(train_ds.map(lambda text, label: text))
model = tf.keras.Sequential([
encoder,
tf.keras.layers.Embedding(VOCAB_SIZE, 64, mask_zero=True),
tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
# tf.keras.layers.LSTM(128),
# tf.keras.layers.Dense(64, activation='relu'),
# tf.keras.layers.Conv1D(64, 5, padding="valid", activation="relu", strides=2),
# tf.keras.layers.GlobalMaxPooling1D(),
# tf.keras.layers.GRU(64),
# tf.keras.layers.SimpleRNN(64),
# tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(4)
])
model.summary()
model.compile(loss='sparse_categorical_crossentropy',
optimizer='sgd',
metrics=['accuracy'])
history = model.fit(train_ds, epochs=10,
validation_data=val_ds)
This is my complete code, the core part is same as tutorials.
But the training output as follow:
Epoch 1/10
100/100 [==============================] - 33s 273ms/step - loss: 9.6882 - accuracy: 0.2562 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 2/10
100/100 [==============================] - 25s 250ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 3/10
100/100 [==============================] - 25s 252ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 4/10
100/100 [==============================] - 25s 254ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 5/10
100/100 [==============================] - 25s 255ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 6/10
100/100 [==============================] - 26s 256ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 7/10
100/100 [==============================] - 26s 257ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 8/10
100/100 [==============================] - 26s 258ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 9/10
100/100 [==============================] - 26s 258ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
Epoch 10/10
100/100 [==============================] - 26s 256ms/step - loss: 12.1238 - accuracy: 0.2478 - val_loss: 11.9475 - val_accuracy: 0.2587
The accuracy does not increase and loss does not decrease at all, like not trained. And the accuracy just same as the reciprocal of the number of classes. (E.g. If it's a binary classification problem then the accuracy would keep aroud 0.5, four classification problem 0.25)
Later I compare with CNN, just change the LSTM layers to CNN layers as tutorials, it works as expect. (Same datasets, same params of model.compile() and model.fit())
I also tried GRU, same problem occurs.
I don't get it.
Am I missing some configuration with RNN-like model? Can somebody help me with this problem? Thanks!
P.S.
tensorflow-macos 2.9
tensorflow-metal 0.5.0
Chip: Apple M1
Datasets: https://storage.googleapis.com/download.tensorflow.org/data/stack_overflow_16k.tar.gz Same as tutorials.
I tried config the optimizer(sgd, adam) and learning rate, does not work. It is not like overfitting.
Methods I tried:
Keras accuracy does not change
Update 2023-01-30
I run the same code on my linux server it work expected. It maybe a bug of tensorflow-macos.
Update 2023-02-01
Tried the official version of tensorflow for macos m1, just
conda install tensorflow
it works. Suppposed to be the problem of tensorflow-macos GPU support. And I try again by using CPU only on tensorflow-macos, it works.
Conclusion:
The RNN-like model have problem on tensorflow-macos with GPU.
I built an algorithm in Python for data sets classification with Keras. It's a very simple LSTM network with 1 input layer, 1 hidden layer (LSTM) and 1 dense output layer.
My data consists of some analog measurements: 63 sets for training and 36 sets for testing, each set having 3 channels with 19200 samples each channel, so (following what I understood reading the documentation) the input shape I needed was x = (63,19200,3) and y = (36,19200,3). (If you want some additional information about the type of data, I can explain more.)
My code is as follows:
import numpy as np
import matplotlib.pyplot as plt
from sklearn.utils import shuffle
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Input
from keras.layers import Dropout
from keras.layers.embeddings import Embedding
from keras.preprocessing import sequence
from keras import initializers
from keras import optimizers
# Fix random seed for reproducibility.
np.random.seed(1)
# Loading data (shapes: X_test (36,19200,3), y_test (36,3), X_train (63,19200,3), y_train (63,3))
(X_test, y_test), (X_train, y_train) = np.load('path.npy',allow_pickle=True)
data = [(X_test, y_test), (X_train, y_train)]
# Manually separating the validation data.
x_val = X_train[-10:]
y_val = y_train[-10:]
X_train = X_train[:-10]
y_train = y_train[:-10]
# Creating model.
model = Sequential()
model.add(Input(shape=(19200,3)))
model.add(LSTM(50, name = 'LSTM', activation='tanh',recurrent_activation='tanh', kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=0.05, seed=1), bias_initializer=initializers.zeros()))
model.add(Dense(1, name = 'Saida', activation='sigmoid', kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=0.05, seed=1), bias_initializer=initializers.zeros()))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
print(model.summary())
history = model.fit(X_train, y_train, epochs=20, batch_size=12, shuffle=True, validation_data=(x_val, y_val))
# Final evaluation of the model.
scores = model.evaluate(X_test, y_test, verbose=1)
print("Accuracy: %.2f%%" % (scores[1]*100))
Very simple, but not that organized, still working on that.
And for this run, the results are:
Model: "sequential_8"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
LSTM (LSTM) (None, 50) 10800
_________________________________________________________________
Saida (Dense) (None, 1) 51
=================================================================
Total params: 10,851
Trainable params: 10,851
Non-trainable params: 0
_________________________________________________________________
None
Epoch 1/20
5/5 [==============================] - 17s 3s/step - loss: 0.6866 - accuracy: 0.6792 - val_loss: 0.6956 - val_accuracy: 0.0000e+00
Epoch 2/20
5/5 [==============================] - 20s 4s/step - loss: 0.6814 - accuracy: 0.8113 - val_loss: 0.6979 - val_accuracy: 0.0000e+00
Epoch 3/20
5/5 [==============================] - 21s 4s/step - loss: 0.6915 - accuracy: 0.7925 - val_loss: 0.7002 - val_accuracy: 0.0000e+00
Epoch 4/20
5/5 [==============================] - 24s 5s/step - loss: 0.6810 - accuracy: 0.7925 - val_loss: 0.7025 - val_accuracy: 0.0000e+00
Epoch 5/20
5/5 [==============================] - 25s 5s/step - loss: 0.6828 - accuracy: 0.7925 - val_loss: 0.7048 - val_accuracy: 0.0000e+00
Epoch 6/20
5/5 [==============================] - 24s 5s/step - loss: 0.6703 - accuracy: 0.8302 - val_loss: 0.7072 - val_accuracy: 0.0000e+00
Epoch 7/20
5/5 [==============================] - 24s 5s/step - loss: 0.6787 - accuracy: 0.7925 - val_loss: 0.7095 - val_accuracy: 0.0000e+00
Epoch 8/20
5/5 [==============================] - 26s 5s/step - loss: 0.6963 - accuracy: 0.7547 - val_loss: 0.7117 - val_accuracy: 0.0000e+00
Epoch 9/20
5/5 [==============================] - 25s 5s/step - loss: 0.6776 - accuracy: 0.7925 - val_loss: 0.7141 - val_accuracy: 0.0000e+00
Epoch 10/20
5/5 [==============================] - 25s 5s/step - loss: 0.6640 - accuracy: 0.8302 - val_loss: 0.7164 - val_accuracy: 0.0000e+00
Epoch 11/20
5/5 [==============================] - 24s 5s/step - loss: 0.6626 - accuracy: 0.8491 - val_loss: 0.7187 - val_accuracy: 0.0000e+00
Epoch 12/20
5/5 [==============================] - 24s 5s/step - loss: 0.6504 - accuracy: 0.8491 - val_loss: 0.7210 - val_accuracy: 0.0000e+00
Epoch 13/20
5/5 [==============================] - 24s 5s/step - loss: 0.6729 - accuracy: 0.7925 - val_loss: 0.7233 - val_accuracy: 0.0000e+00
Epoch 14/20
5/5 [==============================] - 24s 5s/step - loss: 0.6602 - accuracy: 0.8302 - val_loss: 0.7257 - val_accuracy: 0.0000e+00
Epoch 15/20
5/5 [==============================] - 25s 5s/step - loss: 0.6857 - accuracy: 0.7547 - val_loss: 0.7281 - val_accuracy: 0.0000e+00
Epoch 16/20
5/5 [==============================] - 23s 5s/step - loss: 0.6630 - accuracy: 0.8113 - val_loss: 0.7305 - val_accuracy: 0.0000e+00
Epoch 17/20
5/5 [==============================] - 25s 5s/step - loss: 0.6633 - accuracy: 0.7925 - val_loss: 0.7328 - val_accuracy: 0.0000e+00
Epoch 18/20
5/5 [==============================] - 24s 5s/step - loss: 0.6600 - accuracy: 0.8302 - val_loss: 0.7352 - val_accuracy: 0.0000e+00
Epoch 19/20
5/5 [==============================] - 25s 5s/step - loss: 0.6670 - accuracy: 0.8113 - val_loss: 0.7374 - val_accuracy: 0.0000e+00
Epoch 20/20
5/5 [==============================] - 24s 5s/step - loss: 0.6534 - accuracy: 0.8302 - val_loss: 0.7399 - val_accuracy: 0.0000e+00
2/2 [==============================] - 1s 314ms/step - loss: 0.7171 - accuracy: 0.4167
Accuracy: 41.67%
Summarizing: High loss, but decrease very slowly. Accuracy is varying, but in the end it stabilizes at the same value (usually 0,7925 or 0,8113). And my accuracy for the validation set don't even respond to any changes that occur with the other metrics.
My main concern is the validation data is not behaving as it should. I already tried changing the optimizers, activation functions of every layer, weight initializers, number of epochs (went till 100 several times but nothing changed), batch size, shuffling the data using Keras function and Python built-in method, and so on.
The only thing I did not tried was to change the input shapes, but, as I mentioned earlier, this was the only way I got the 3D array to be accepted in the Input Layer.
If you guys have any tips to what can be changed to achieve more consistent results, I would be very grateful.
Any additional commentary will be happily accepted.
This is my first question here and I am not a native english speaker, so sorry if any information was not very clear.
Cheers, Matheus Zimmermann.
I think you can apply to_categorical method or One hot encoding approach to the
y_train , y_val and y_test variables.
Hope than after applying it ,you will find your validation accuracy perfectly.
I faced this type of same problem before.
I tried to train a CNN to classify 9 class of image. Each class has 1000 image for training. I tried training on VGG16 and VGG19, both can achieve validation accuracy of 90%. But when I tried to train on InceptionResNetV2 model, the model seems to stuck around 20% and 30%. Below is my code for InceptionResNetV2 and the training. What can I do to improve the training?
base_model = tf.keras.applications.InceptionResNetV2(input_shape=(IMG_HEIGHT, IMG_WIDTH ,3),weights = 'imagenet',include_top=False)
base_model.trainable = False
model = tf.keras.Sequential([
base_model,
Flatten(),
Dense(1024, activation = 'relu', kernel_regularizer=regularizers.l2(0.001)),
LeakyReLU(alpha=0.4),
Dropout(0.5),
BatchNormalization(),
Dense(1024, activation = 'relu', kernel_regularizer=regularizers.l2(0.001)),
LeakyReLU(alpha=0.4),
Dense(9, activation = 'softmax')])
optimizer_model = tf.keras.optimizers.Adam(learning_rate=0.0001, name='Adam', decay=0.00001)
loss_model = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
model.compile(optimizer_model, loss="categorical_crossentropy", metrics=['accuracy'])
Epoch 1/10
899/899 [==============================] - 255s 283ms/step - loss: 4.3396 - acc: 0.3548 - val_loss: 4.2744 - val_acc: 0.3874
Epoch 2/10
899/899 [==============================] - 231s 257ms/step - loss: 3.5856 - acc: 0.4695 - val_loss: 3.9151 - val_acc: 0.3816
Epoch 3/10
899/899 [==============================] - 225s 250ms/step - loss: 3.1451 - acc: 0.4959 - val_loss: 4.8801 - val_acc: 0.2425
Epoch 4/10
899/899 [==============================] - 227s 252ms/step - loss: 2.7771 - acc: 0.5124 - val_loss: 3.7167 - val_acc: 0.3023
Epoch 5/10
899/899 [==============================] - 231s 257ms/step - loss: 2.4993 - acc: 0.5260 - val_loss: 3.7276 - val_acc: 0.3770
Epoch 6/10
899/899 [==============================] - 227s 252ms/step - loss: 2.3148 - acc: 0.5251 - val_loss: 3.7677 - val_acc: 0.3115
Epoch 7/10
899/899 [==============================] - 234s 260ms/step - loss: 2.1381 - acc: 0.5379 - val_loss: 3.4867 - val_acc: 0.2862
Epoch 8/10
899/899 [==============================] - 230s 256ms/step - loss: 2.0091 - acc: 0.5367 - val_loss: 4.1032 - val_acc: 0.3080
Epoch 9/10
899/899 [==============================] - 225s 251ms/step - loss: 1.9155 - acc: 0.5399 - val_loss: 4.1270 - val_acc: 0.2954
Epoch 10/10
899/899 [==============================] - 232s 258ms/step - loss: 1.8349 - acc: 0.5508 - val_loss: 4.3918 - val_acc: 0.2276
VGG-16/19 has a depth of 23/26 layers, whereas, InceptionResNetV2 has a depth of 572 layers. Now, there is minimal domain similarity between medical images and imagenet dataset. In VGG, due to low depth the features you're getting are not that complex and network is able to classify it on the basis of Dense layer features. However, in IRV2 network, as it's too much deep, the output of the fc layer is more complex (visualize it something object like but for imagenet dataset), and, then the features obtained from these layers are unable to connect to the Dense layer features, and, hence overfitting. I think you were able to get my point.
Check out my answer to very similar question of yours on this link: Link. It will help improve your accuracy.
I have written a CNN that takes in MFCC spectrograms and is meant to classify the images into five different classes. I trained the model for 30 epochs and after the first epoch, no metrics change. Could it be a problem with imbalanced classification, and if so, how would I bias the model for the dataset, if possible? Below is the data generator code, the model definition, and the outputs. The original model had two additional layers however, I started tweaking things when I was trying to troubleshoot the issue
Data Generator Definition:
path = 'path_to_dataset'
CLASS_NAMES = ['belly_pain', 'burping', 'discomfort', 'hungry', 'tired']
CLASS_NAMES = np.array(CLASS_NAMES)
BATCH_SIZE = 32
IMG_HEIGHT = 150
IMG_WIDTH = 150
# 457 is the number of images total
STEPS_PER_EPOCH = np.ceil(457/BATCH_SIZE)
img_generator = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255, validation_split=0.2, horizontal_flip=True, rotation_range=45, width_shift_range=.15, height_shift_range=.15)
train_data_gen = img_generator.flow_from_directory( directory=path, batch_size=BATCH_SIZE, shuffle=True, target_size=(IMG_HEIGHT, IMG_WIDTH), classes = list(CLASS_NAMES), subset='training', class_mode='categorical')
validation_data_gen = img_generator.flow_from_directory( directory=path, batch_size=BATCH_SIZE, shuffle=True, target_size=(IMG_HEIGHT, IMG_WIDTH), classes = list(CLASS_NAMES), subset='validation', class_mode='categorical')
Model Definition:
EPOCHS = 30
model = Sequential([
Conv2D(128, 3, activation='relu',
input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
MaxPooling2D(),
Flatten(),
Dense(512, activation='sigmoid'),
Dense(1)
])
opt = tf.keras.optimizers.Adamax(lr=0.001)
model.compile(optimizer=opt,
loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
metrics=['accuracy'])
First 5 Epochs:
Epoch 1/30
368/368 [==============================] - 371s 1s/step - loss: 0.6713 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 2/30
368/368 [==============================] - 235s 640ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 3/30
368/368 [==============================] - 233s 633ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 4/30
368/368 [==============================] - 236s 641ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 5/30
368/368 [==============================] - 234s 636ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Last Five Epochs:
Epoch 25/30
368/368 [==============================] - 231s 628ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 26/30
368/368 [==============================] - 227s 617ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 27/30
368/368 [==============================] - 228s 620ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 28/30
368/368 [==============================] - 234s 636ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 29/30
368/368 [==============================] - 235s 638ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
Epoch 30/30
368/368 [==============================] - 234s 636ms/step - loss: 0.5004 - accuracy: 0.8000 - val_loss: 0.5004 - val_accuracy: 0.8000
You are trying to achieve a classification task with 4 classes but your last layer only contain one neuron.
It should be a dense layer with 4 neurons and a softmax activation :
Dense(4, activation="softmax")
You need to also change the loss function accordingly to classification loss like for example categorical_crossentropy.
I am trying to train my model using transfer learning, for this I am using VGG16 model, stripped the top layers and froze first 2 layers for using imagenet initial weights. For fine tuning them I am using learning rate 0.0001, activation softmax, dropout 0.5, loss categorical crossentropy, optimizer SGD, classes 46.
I am just unable to understand the behavior while training. Train loss and acc both are fine (loss is decreasing, acc is increasing). Val loss is decreasing and acc is increasing as well, BUT they are always higher than the train loss and acc.
Assuming its overfitting I made the model less complex, increased the dropout rate, added more samples to val data, but nothing seemed to work. I am a newbie so any kind of help is appreciated.
26137/26137 [==============================] - 7446s 285ms/step - loss: 1.1200 - accuracy: 0.3810 - val_loss: 3.1219 - val_accuracy: 0.4467
Epoch 2/50
26137/26137 [==============================] - 7435s 284ms/step - loss: 0.9944 - accuracy: 0.4353 - val_loss: 2.9348 - val_accuracy: 0.4694
Epoch 3/50
26137/26137 [==============================] - 7532s 288ms/step - loss: 0.9561 - accuracy: 0.4530 - val_loss: 1.6025 - val_accuracy: 0.4780
Epoch 4/50
26137/26137 [==============================] - 7436s 284ms/step - loss: 0.9343 - accuracy: 0.4631 - val_loss: 1.3032 - val_accuracy: 0.4860
Epoch 5/50
26137/26137 [==============================] - 7358s 282ms/step - loss: 0.9185 - accuracy: 0.4703 - val_loss: 1.4461 - val_accuracy: 0.4847
Epoch 6/50
26137/26137 [==============================] - 7396s 283ms/step - loss: 0.9083 - accuracy: 0.4748 - val_loss: 1.4093 - val_accuracy: 0.4908
Epoch 7/50
26137/26137 [==============================] - 7424s 284ms/step - loss: 0.8993 - accuracy: 0.4789 - val_loss: 1.4617 - val_accuracy: 0.4939
Epoch 8/50
26137/26137 [==============================] - 7433s 284ms/step - loss: 0.8925 - accuracy: 0.4822 - val_loss: 1.4257 - val_accuracy: 0.4978
Epoch 9/50
26137/26137 [==============================] - 7445s 285ms/step - loss: 0.8868 - accuracy: 0.4851 - val_loss: 1.5568 - val_accuracy: 0.4953
Epoch 10/50
26137/26137 [==============================] - 7387s 283ms/step - loss: 0.8816 - accuracy: 0.4874 - val_loss: 1.4534 - val_accuracy: 0.4970
Epoch 11/50
26137/26137 [==============================] - 7374s 282ms/step - loss: 0.8779 - accuracy: 0.4894 - val_loss: 1.4605 - val_accuracy: 0.4912
Epoch 12/50
26137/26137 [==============================] - 7411s 284ms/step - loss: 0.8733 - accuracy: 0.4915 - val_loss: 1.4694 - val_accuracy: 0.5030
Yes, you are facing over-fitting issue. To mitigate, you can try to implement below steps
1.Shuffle the Data, by using shuffle=True in VGG16_model.fit. Code is shown below:
history = VGG16_model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1,
validation_data=(x_validation, y_validation), shuffle = True)
2.Use Early Stopping. Code is shown below
callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=15)
3.Use Regularization. Code for Regularization is shown below (You can try l1 Regularization or l1_l2 Regularization as well):
from tensorflow.keras.regularizers import l2
Regularizer = l2(0.001)
VGG16_model.add(Conv2D(96,11, 11, input_shape = (227,227,3),strides=(4,4), padding='valid', activation='relu', data_format='channels_last',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
VGG16_model.add(Dense(units = 2, activation = 'sigmoid',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
4.You can try using BatchNormalization.
5.Perform Image Data Augmentation using ImageDataGenerator. Refer this link for more info about that.
6.If the Pixels are not Normalized, Dividing the Pixel Values with 255 also helps