We Keep Coding

Keras - Is There an way to reduce value gap between categorical_accuracy and val_categorical_accuracy?

I'm trying to build and train LSTM Neural Network.
Here is my code (summary version):
X_train, X_test, y_train, y_test = train_test_split(np.array(sequences), to_categorical(labels).astype(int), test_size=0.2)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2)
log_dir = os.path.join('Logs')
tb_callback = TensorBoard(log_dir=log_dir)
model = Sequential()
model.add(LSTM(64, return_sequences=True, activation='tanh', input_shape=(60,1662)))
model.add(LSTM(128, return_sequences=True, activation='tanh', dropout=0.31))
model.add(LSTM(64, return_sequences=False, activation='tanh'))
model.add(Dense(32, activation='relu'))
model.add(Dense(len(actions), activation='softmax'))
model.compile(optimizer='Adam', loss='categorical_crossentropy', metrics=['categorical_accuracy'])
val_dataset = tf.data.Dataset.from_tensor_slices((X_val, y_val)) # default slice percentage check
val_dataset = val_dataset.batch(256)
model.fit(X_train, y_train, batch_size=256, epochs=250, callbacks=[tb_callback], validation_data=val_dataset)
And model fit result:
Epoch 248/250
8/8 [==============================] - 2s 252ms/step - loss: 0.4563 - categorical_accuracy: 0.8641 - val_loss: 2.1406 - val_categorical_accuracy: 0.6104
Epoch 249/250
8/8 [==============================] - 2s 255ms/step - loss: 0.4542 - categorical_accuracy: 0.8672 - val_loss: 2.2365 - val_categorical_accuracy: 0.5667
Epoch 250/250
8/8 [==============================] - 2s 234ms/step - loss: 0.4865 - categorical_accuracy: 0.8562 - val_loss: 2.1668 - val_categorical_accuracy: 0.5875
I wanna reduce value gap between categorical_accuracy and val_categorical_accuracy.
Can I know how to do it?
Thank you for reading my article.

When there is so much difference between your train and validation data this mean your model is overfitting.
So look for how prevent from overfitting. Usually what you have to do is add more data to your dataset.
It won’t work every time, but training with more data can help algorithms detect the signal better.
Try to stop before overfit
another aspect is to try stop the model and reduce the learning rate

TensorFlow image classification loss doesn't decrease

import tensorflow as tf
import matplotlib.pyplot as plt
import cv2
import os
import numpy as np
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.preprocessing import image
from tensorflow.keras.optimizers import Nadam
from tensorflow.keras.models import load_model
train = ImageDataGenerator(rescale=1 / 255)
validation = ImageDataGenerator(rescale=1 / 255)
train_dataset = train.flow_from_directory('raw-img/training', target_size=(200, 200), batch_size=3,
class_mode='categorical')
validation_dataset = train.flow_from_directory('raw-img/validation', target_size=(200, 200), batch_size=3,
class_mode='categorical')
model = tf.keras.models.Sequential([tf.keras.layers.Conv2D(16, (3, 3), activation='relu', input_shape=(200, 200, 3),padding='same'),
tf.keras.layers.MaxPool2D(2, 2,padding='same'),
#
tf.keras.layers.Conv2D(32, (3, 3), activation='relu',padding='same'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.Dropout(rate=0.6),
#
tf.keras.layers.Conv2D(64, (3, 3), activation='relu',padding='same'),
tf.keras.layers.MaxPool2D(2, 2),
tf.keras.layers.Dropout(rate=0.6),
#
tf.keras.layers.Conv2D(128, (3, 3), activation='relu',padding='same'),
tf.keras.layers.MaxPool2D(2, 2),
#
tf.keras.layers.Conv2D(128, (3, 3), activation='relu',padding='same'),
tf.keras.layers.MaxPool2D(2, 2),
#
tf.keras.layers.Conv2D(256, (3, 3), activation='relu',padding='same'),
tf.keras.layers.MaxPool2D(2, 2,),
#
tf.keras.layers.Flatten(),
#
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dense(10, activation='sigmoid'),
])
print(model.summary())
model.compile(loss='binary_crossentropy', optimizer=Nadam(learning_rate=0.003), metrics['accuracy'])
model_fit = model.fit(train_dataset, epochs=70, batch_size=3, validation_data=validation_dataset,steps_per_epoch=len(train_dataset),validation_steps=len(validation_dataset))
loss, accuracy = model.evaluate(train_dataset)
print("Loss: ", loss)
print("Accuracy: ", accuracy)
Found 26179 images belonging to 10 classes.
Found 8196 images belonging to 10 classes.
Epoch 1/70
2909/2909 [==============================] - 1005s 345ms/step - loss: 0.3292 - accuracy: 0.1805 - val_loss: 0.3533 - val_accuracy: 0.0000e+00
Epoch 2/70
2909/2909 [==============================] - 645s 222ms/step - loss: 0.3167 - accuracy: 0.1758 - val_loss: 0.3654 - val_accuracy: 0.0000e+00
...
Epoch 8/70
2909/2909 [==============================] - 445s 153ms/step - loss: 0.3160 - accuracy: 0.1835 - val_loss: 0.3666 - val_accuracy: 0.0000e+00
Epoch 9/70
2909/2909 [==============================] - ETA: 0s - loss: 0.3146 - accuracy: 0.1867
What the problem with this code? Accuracy stuck at 0.1800 and 0.1900 and loss in doesn't decrease.

A couple of issues
since your images are 200 X 200 you could use a larger batch size like say 32
in the train = ImageDataGenerator(rescale=1 / 255) you might want to consider
adding some image augmentation like horizontal_flip=True, etc. If you do add
augmentation do not use the train in your code for the validation_data set, use
validation
in your model you have the code tf.keras.layers.Dense(10, activation='sigmoid')
unless you are doing multi-label classification change the activation to "softmax'
your model will likely over-fit. I would add at least one dropout layer to
your model like tf.keras.layers.Dropout(.3)
First do the above changes and look at the performance. If it starts to over-fit
increase the dropout rate or reduce the number of nodes in the hidden layers
If you are not satisfied with the accuracy you might get an improvement by
using and adjustable learning rate. You can do this with the Keras callback
ReduceLROnPlateau. Documentation is here. My suggest code is
rlronp=tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=.5,patience=1,verbose=1)
I also recommend you use the Keras callback EarlyStopping. It will monitor
the validation loss if it fails to reduce after 'patience' number of epochs.
Documentation is here. set restore_best_weights=True so that is this
callback activates it will load your model with the weights from the epoch with
the lowest validation loss. My suggested code is
es=tf.keras.callbacks.EarlyStopping(monitor="val_loss",patience=4,verbose=1,
restore_best_weights=True)
if you use the callbacks then in model.fit set callbacks=[rlronp,es]

keras classifier wrong evaluation while learning is great

have small dataset
Found 1836 images belonging to 2 classes.
Found 986 images belonging to 2 classes.
standard architecture of model
image_generator = ImageDataGenerator(
rescale=1./255,
validation_split=0.35
)
train_data_gen = image_generator.flow_from_directory(
directory=directory,
target_size=(IMG_SHAPE, IMG_SHAPE),
subset='training',
)
val_data_gen = image_generator.flow_from_directory(
directory=directory,
target_size=(IMG_SHAPE, IMG_SHAPE),
subset='validation',
)
---
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_SHAPE, IMG_SHAPE, 3)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(2, activation='softmax'),
])
model.compile(
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy']
)
BATCH_SIZE = 128
EPOCHS = 7
total_train, total_val = train_data_gen.samples, val_data_gen.samples
steps_per_epoch = int(np.ceil(total_train / float(BATCH_SIZE)))
validation_freq = int(np.ceil(total_val / float(BATCH_SIZE)))
history = model.fit(
train_data_gen,
epochs=EPOCHS,
steps_per_epoch=steps_per_epoch,
validation_data=val_data_gen,
validation_freq=validation_freq
)
getting perfect metrics
Epoch 1/7
15/15 [==============================] - 66s 4s/step - loss: 1.0809 - accuracy: 0.4917
Epoch 2/7
15/15 [==============================] - 56s 4s/step - loss: 0.3475 - accuracy: 0.8729
Epoch 3/7
15/15 [==============================] - 60s 4s/step - loss: 0.1113 - accuracy: 0.9583
Epoch 4/7
15/15 [==============================] - 58s 4s/step - loss: 0.1987 - accuracy: 0.9109
Epoch 5/7
15/15 [==============================] - 59s 4s/step - loss: 0.1127 - accuracy: 0.9438
Epoch 6/7
15/15 [==============================] - 60s 4s/step - loss: 0.0429 - accuracy: 0.9854
Epoch 7/7
15/15 [==============================] - 49s 3s/step - loss: 0.0542 - accuracy: 0.9812
but after i evaluate it, i get completely biased to first class results
it works only when i run it for 1 epoch, but with a lack of accuracy
eval code
def make_pred(model, labled_dataset, IMG_SHAPE, img_path) -> LabelName:
def make_image(img_path):
# img = img_path.resize((IMG_SHAPE, IMG_SHAPE), Image.ANTIALIAS)
img = image.load_img(img_path, target_size=(IMG_SHAPE, IMG_SHAPE))
img = image.img_to_array(img)
return np.expand_dims(img, axis=0)
pred_id: List[List] = np.argmax(model.predict(make_image(img_path)), axis=1)
all_labels = list(labled_dataset.class_indices.keys())
return all_labels[int(pred_id)]
what wrong with it?
should i downsize source image before eval it?

I believe you need to do two things. One resize the images you wish to predict, then rescale the images as you did for the training images. I also recommend that you set the validation_freq=1 so that you can set how the validation loss and accuracy are trending. This allows you to see how your model is performing relative to over fitting etc. You can detect if your model is over fitting if the training loss continues to declined but in later epochs your validation loss begins to increase. If you see over fitting add a Dropout layer after your dense 512 node dense layer. Documentation is here. Prediction accuracy should be close to the validation accuracy for the last epoch. I also recommend you consider using the keras callback ModelCheckpoint. Documentation is here. Set it up to monitor validation loss and save the model with the lowest validation loss. Then load the saved model to do predictions. Finally I find it effective to use an adjustable learning rate. The keras callback ReduceLROnPlateau makes this easy to do. Documentation is here. Set it up to monitor validation loss. The callback will automatically reduce the learning rate by a factor (parameter factor) if after (parameter patience) patience number of epochs the validation loss fails to decrease. I use factor=.5 and patience=1. This allows you to use a larger learning rate initially and have it decrease as needed so convergence will be faster. One more thing in your val_data_gen set shuffle=False so the validation images are processed in the same order each time.

problem was in validation_freq which is should be
validation_steps after that we finally getting val_accuracy, so
training starts validating in proper way
on top of it IMG_SHAPE wasn't the same for ImageDataGenerator and
inside the model input_shape=(IMG_SHAPE, IMG_SHAPE, 3)),
using PIL for prediction may helped too, it gives slightly different
from keras.preprocessing.image results
def make_pred_PIL(model, labled_dataset, IMG_SHAPE, img_path) -> LabelName:
img = cv2.imread(img_path)
img = cv2.resize(img, (IMG_SHAPE, IMG_SHAPE))
img = np.array(img, dtype=np.float32)
img = np.reshape(img, (-1, IMG_SHAPE, IMG_SHAPE, 3))
pred_id: List[List] = np.argmax(model.predict(img), axis=1)
all_labels = list(labled_dataset.class_indices.keys())
return all_labels[int(pred_id)]

CNN Acc and Loss constant with custom image data

I am trying to train a simple CNN model for a binary classification task in Keras with a dataset of images I mined. The problem is that I am getting constant accuracy, val_accuracy and loss after a couple of epochs. Am I processing the data the wrong way? Or is it something in the model settings?
At the beginning I was using softmax as the final activation function and categorical cossentropy, I was also using the to_categorical function on the labels.
After reading up on what usually causes this to happen I decided to use sigmoid and binary_crossentropy instead and not use to_categorical. Still the problem persists and I am starting to wonder whether it's my data the problem (the two classes are too similar) or the way I am feeding the image arrays.
conkeras1 = []
pics = os.listdir("/Matrices/")
# I do this for the images of both classes, just keeping it short.
for x in range(len(pics)):
img = image.load_img("Matrices/"+pics[x])
conkeras1.append(img)
conkeras = conkeras1+conkeras2
conkeras = np.array([image.img_to_array(x) for x in conkeras]).astype("float32")
conkeras = conkeras / 255 # I also tried normalizing with a z-score with no success
yecs1 = [1]*len(conkeras1)
yecs2 = [0]*len(conkeras2)
y_train = yecs1+yecs2
y_train = np.array(y_train).astype("float32")
model = Sequential([
Conv2D(64, (3, 3), input_shape=conkeras.shape[1:], padding="same", activation="relu"),
Conv2D(32, (3, 3), activation="relu", padding="same"),
Flatten(),
Dense(500, activation="relu"),
#Dense(4096, activation="relu"),
Dense(1, activation="sigmoid")
])
model.compile(loss=keras.losses.binary_crossentropy,
optimizer=keras.optimizers.Adam(lr=0.001),
metrics=['accuracy'])
history = model.fit(conkeras, y_train,
batch_size=32,
epochs=32, shuffle=True,
verbose=1,
callbacks=[tensorboard])
The output I get is this:
975/975 [==============================] - 107s 110ms/step - loss: 8.0022 - acc: 0.4800
Epoch 2/32
975/975 [==============================] - 99s 101ms/step - loss: 8.1756 - acc: 0.4872
Epoch 3/32
975/975 [==============================] - 97s 100ms/step - loss: 8.1756 - acc: 0.4872
Epoch 4/32
975/975 [==============================] - 97s 99ms/step - loss: 8.1756 - acc: 0.4872
and these are the shapes of the traning set and labels
>>> conkeras.shape
(975, 100, 100, 3)
>>> y_train.shape
(975,)

Negative Loss function when using cross_val_score in sklearn/keras. Works when not using k fold

I am trying to implement a model which takes in an array of 167 categorical variables (0 or 1), and outputs an estimated value between 0 and 1. Over 300 datapoints are available.
The model works when using a basic model, below:
classifier = Sequential()
classifier.add(Dense(units = 80, kernel_initializer = 'uniform', activation = 'relu', input_dim = 167))
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
classifier.fit(X_train, y_train, batch_size = 10, epochs = 200)
y_pred = classifier.predict(X_test)
Output is similar to:
Epoch 105/200
253/253 [==============================] - 0s - loss: 0.5582 - acc: 0.0079
Epoch 106/200
253/253 [==============================] - 0s - loss: 0.5583 - acc: 0.0079
Unfortunately, when I try to use cross validation, model stops working, and loss function becomes large and negative. Code is below:
def build_classifier():
classifier = Sequential()
classifier.add(Dense(units = 80, kernel_initializer = 'uniform', activation = 'relu', input_dim = 167))
classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))
classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])
return classifier
classifier = KerasClassifier(build_fn = build_classifier, batch_size = 10, epochs = 100)
accuracies = cross_val_score(estimator = classifier, X=X_train, y=y_train, cv=3,n_jobs=1)
Output looks like:
Epoch 59/100
168/168 [==============================] - 0s - loss: -1106.9519 - acc: 0.0060
Epoch 60/100
168/168 [==============================] - 0s - loss: -1106.9519 - acc: 0.0060
I have toyed with different parameters, but I cannot seem to find what is causing the issue. Still learning, so any help is very appreciated.

It can happen if the data is sparse. A lot NaNs and Infs can cause this problem. If you are doing a 3 fold validation, it may be possible that in one of the folds, the data selected doesn't contain enough info. Possible solutions can be:
Change the random seed.
Increase the data set.