We Keep Coding

Transfer learning model running on inference does not learn

I'm doing a species classification task from kaggle (https://www.kaggle.com/competitions/yum-or-yuck-butterfly-mimics-2022/overview). I decided to use transfer learning to tackle this problem since there aren't that many images. The model is as follows:
inputs = tf.keras.layers.Input(shape=(224, 224, 3))
base_model = tf.keras.applications.resnet50.ResNet50(
input_shape=(224,224,3),
include_top=False,
weights="imagenet")
for layer in base_model.layers:
layer.trainable = False
x = base_model(inputs, training=False)
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.3)(x)
x = tf.keras.layers.Dense(1024, activation="relu")(x)
x = tf.keras.layers.Dropout(0.3)(x)
x = tf.keras.layers.Dense(512, activation="relu")(x)
x = tf.keras.layers.Dropout(0.3)(x)
x = tf.keras.layers.Dense(64, activation="relu")(x)
output = tf.keras.layers.Dense(6, activation="softmax")(x)
model = tf.keras.Model(inputs=inputs, outputs=output)
As per the guidelines when doing transfer learning:https://keras.io/guides/transfer_learning/, I'm freezing the resnet layers and training the model on inference only (training=False). However, the results show that the model is not learning properly. Convergence doesn't seem like it will be possible even after nearly 200 epochs:
model.compile(
optimizer=tf.keras.optimizers.Adam(),
loss="categorical_crossentropy",
metrics="accuracy",
)
stop_early = tf.keras.callbacks.EarlyStopping(
monitor='val_loss',
min_delta=0.0001,
patience=20,
restore_best_weights=True
)
history = model.fit(train_generator,
validation_data = val_generator,
epochs = 200,
callbacks=[stop_early])
22/22 [==============================] - 19s 442ms/step - loss: 1.9317 - accuracy: 0.1794 - val_loss: 1.8272 - val_accuracy: 0.1618
Epoch 2/200
22/22 [==============================] - 9s 398ms/step - loss: 1.8250 - accuracy: 0.1882 - val_loss: 1.7681 - val_accuracy: 0.2197
Epoch 3/200
22/22 [==============================] - 9s 402ms/step - loss: 1.7927 - accuracy: 0.2294 - val_loss: 1.7612 - val_accuracy: 0.2139
Epoch 4/200
22/22 [==============================] - 9s 424ms/step - loss: 1.7930 - accuracy: 0.2000 - val_loss: 1.7640 - val_accuracy: 0.2139
Epoch 5/200
22/22 [==============================] - 9s 391ms/step - loss: 1.7872 - accuracy: 0.2132 - val_loss: 1.7489 - val_accuracy: 0.3121
Epoch 6/200
22/22 [==============================] - 9s 389ms/step - loss: 1.7700 - accuracy: 0.2574 - val_loss: 1.7378 - val_accuracy: 0.2543
Epoch 7/200
22/22 [==============================] - 9s 396ms/step - loss: 1.7676 - accuracy: 0.2353 - val_loss: 1.7229 - val_accuracy: 0.3064
Epoch 8/200
22/22 [==============================] - 9s 427ms/step - loss: 1.7721 - accuracy: 0.2353 - val_loss: 1.7225 - val_accuracy: 0.2948
Epoch 9/200
22/22 [==============================] - 9s 399ms/step - loss: 1.7522 - accuracy: 0.2588 - val_loss: 1.7267 - val_accuracy: 0.2948
Epoch 10/200
22/22 [==============================] - 9s 395ms/step - loss: 1.7434 - accuracy: 0.2735 - val_loss: 1.7151 - val_accuracy: 0.2948
Epoch 11/200
22/22 [==============================] - 9s 391ms/step - loss: 1.7500 - accuracy: 0.2632 - val_loss: 1.7083 - val_accuracy: 0.3064
Epoch 12/200
22/22 [==============================] - 9s 425ms/step - loss: 1.7307 - accuracy: 0.2721 - val_loss: 1.6899 - val_accuracy: 0.3179
Epoch 13/200
22/22 [==============================] - 9s 407ms/step - loss: 1.7439 - accuracy: 0.2794 - val_loss: 1.7045 - val_accuracy: 0.2948
Epoch 14/200
22/22 [==============================] - 9s 404ms/step - loss: 1.7376 - accuracy: 0.2706 - val_loss: 1.7118 - val_accuracy: 0.2659
Epoch 15/200
22/22 [==============================] - 9s 419ms/step - loss: 1.7588 - accuracy: 0.2647 - val_loss: 1.6684 - val_accuracy: 0.3237
Epoch 16/200
22/22 [==============================] - 9s 394ms/step - loss: 1.7289 - accuracy: 0.2824 - val_loss: 1.6733 - val_accuracy: 0.3064
Epoch 17/200
22/22 [==============================] - 9s 387ms/step - loss: 1.7184 - accuracy: 0.2809 - val_loss: 1.7185 - val_accuracy: 0.2659
Epoch 18/200
22/22 [==============================] - 9s 408ms/step - loss: 1.7242 - accuracy: 0.2765 - val_loss: 1.6961 - val_accuracy: 0.2717
Epoch 19/200
22/22 [==============================] - 9s 424ms/step - loss: 1.7218 - accuracy: 0.2853 - val_loss: 1.6757 - val_accuracy: 0.3006
Epoch 20/200
22/22 [==============================] - 9s 396ms/step - loss: 1.7248 - accuracy: 0.2882 - val_loss: 1.6716 - val_accuracy: 0.3064
Epoch 21/200
22/22 [==============================] - 9s 401ms/step - loss: 1.7134 - accuracy: 0.2838 - val_loss: 1.6666 - val_accuracy: 0.2948
Epoch 22/200
22/22 [==============================] - 9s 393ms/step - loss: 1.7140 - accuracy: 0.2941 - val_loss: 1.6427 - val_accuracy: 0.3064
I need to unfreeze the layers and turn off inference in order for the model to learn. I tested the same scenario with EfficientNet and the same thing happened. Finally, I also used Xception, and freezing the layers and running with inference was fine. So it seems they behave differently, even though they all have batchnorm layers.
I'm not understanding what is going on here. Why would I need to turn inference off? Could anyone have a clue about this?
EDIT:
results from Resnet50:
results from Xception:

Why does accuracy not increase in training but loss and val_loss decrease?

I state that I am not at all familiar with neural networks and this is the first time that I have tried to develop one.
The problem lies in predicting a week's pollution forecast, based on the previous month.
Unstructured data with 15 features are:
Start data
The data to be predicted is 'gas', for a total of 168 hours in the next week, is the hours in a week.
MinMaxScaler(feature_range (0,1)) is applied to the data. And then the data is split into train and test data. Since only one year of hourly measurements is available, the data is resampled in series of 672 hourly samples that each starts from every day of the year at midnight. Therefore, from about 8000 starting hourly surveys, about 600 series of 672 samples are obtained.
The 'date' is removed from the initial data and the form of train_x and train_y is:
Shape of train_x and train_y
In train_x[0] there are 672 hourly readings for the first 4 weeks of the data set and consist of all features including 'gas'.
In train_y [0], on the other hand, there are 168 hourly readings for the following week which begins when the month ends in train_x [0].
Train_X[0] where column 0 is 'gas' and Train_y[0] with only gas column for the next week after train_x[0]
TRAIN X SHAPE = (631, 672, 14)
TRAIN Y SHAPE = (631, 168, 1)
After organizing the data in this way (if it's wrong please let me know), I built the neural network as the following:
train_x, train_y = to_supervised(train, n_input)
train_x = train_x.astype(float)
train_y = train_y.astype(float)
# define parameters
verbose, epochs, batch_size = 1, 200, 50
n_timesteps, n_features, n_outputs = train_x.shape[1], train_x.shape[2], train_y.shape[1]
# define model
model = Sequential()
opt = optimizers.RMSprop(learning_rate=1e-3)
model.add(layers.GRU(14, activation='relu', input_shape=(n_timesteps, n_features),return_sequences=False, stateful=False))
model.add(layers.Dense(1, activation='relu'))
#model.add(layers.Dense(14, activation='linear'))
model.add(layers.Dense(n_outputs, activation='sigmoid'))
model.summary()
model.compile(loss='mse', optimizer=opt, metrics=['accuracy'])
train_y = np.concatenate(train_y).reshape(len(train_y), 168)
callback_early_stopping = EarlyStopping(monitor='val_loss',
patience=5, verbose=1)
callback_tensorboard = TensorBoard(log_dir='./23_logs/',
histogram_freq=0,
write_graph=False)
callback_reduce_lr = ReduceLROnPlateau(monitor='val_loss',
factor=0.1,
min_lr=1e-4,
patience=0,
verbose=1)
callbacks = [callback_early_stopping,
callback_tensorboard,
callback_reduce_lr]
history = model.fit(train_x, train_y, epochs=epochs, batch_size=batch_size, verbose=verbose, shuffle=False
, validation_split=0.2, callbacks=callbacks)
When i fit the network i get:
11/11 [==============================] - 5s 305ms/step - loss: 0.1625 - accuracy: 0.0207 - val_loss: 0.1905 - val_accuracy: 0.0157
Epoch 2/200
11/11 [==============================] - 2s 179ms/step - loss: 0.1594 - accuracy: 0.0037 - val_loss: 0.1879 - val_accuracy: 0.0157
Epoch 3/200
11/11 [==============================] - 2s 169ms/step - loss: 0.1571 - accuracy: 0.0040 - val_loss: 0.1855 - val_accuracy: 0.0079
Epoch 4/200
11/11 [==============================] - 2s 165ms/step - loss: 0.1550 - accuracy: 0.0092 - val_loss: 0.1832 - val_accuracy: 0.0079
Epoch 5/200
11/11 [==============================] - 2s 162ms/step - loss: 0.1529 - accuracy: 0.0102 - val_loss: 0.1809 - val_accuracy: 0.0079
Epoch 6/200
11/11 [==============================] - 2s 160ms/step - loss: 0.1508 - accuracy: 0.0085 - val_loss: 0.1786 - val_accuracy: 0.0079
Epoch 7/200
11/11 [==============================] - 2s 160ms/step - loss: 0.1487 - accuracy: 0.0023 - val_loss: 0.1763 - val_accuracy: 0.0079
Epoch 8/200
11/11 [==============================] - 2s 158ms/step - loss: 0.1467 - accuracy: 0.0023 - val_loss: 0.1740 - val_accuracy: 0.0079
Epoch 9/200
11/11 [==============================] - 2s 159ms/step - loss: 0.1446 - accuracy: 0.0034 - val_loss: 0.1718 - val_accuracy: 0.0000e+00
Epoch 10/200
11/11 [==============================] - 2s 160ms/step - loss: 0.1426 - accuracy: 0.0034 - val_loss: 0.1695 - val_accuracy: 0.0000e+00
Epoch 11/200
11/11 [==============================] - 2s 162ms/step - loss: 0.1406 - accuracy: 0.0034 - val_loss: 0.1673 - val_accuracy: 0.0000e+00
Epoch 12/200
11/11 [==============================] - 2s 159ms/step - loss: 0.1387 - accuracy: 0.0034 - val_loss: 0.1651 - val_accuracy: 0.0000e+00
Epoch 13/200
11/11 [==============================] - 2s 159ms/step - loss: 0.1367 - accuracy: 0.0052 - val_loss: 0.1629 - val_accuracy: 0.0000e+00
Epoch 14/200
11/11 [==============================] - 2s 159ms/step - loss: 0.1348 - accuracy: 0.0052 - val_loss: 0.1608 - val_accuracy: 0.0000e+00
Epoch 15/200
11/11 [==============================] - 2s 161ms/step - loss: 0.1328 - accuracy: 0.0052 - val_loss: 0.1586 - val_accuracy: 0.0000e+00
Epoch 16/200
11/11 [==============================] - 2s 162ms/step - loss: 0.1309 - accuracy: 0.0052 - val_loss: 0.1565 - val_accuracy: 0.0000e+00
Epoch 17/200
11/11 [==============================] - 2s 171ms/step - loss: 0.1290 - accuracy: 0.0052 - val_loss: 0.1544 - val_accuracy: 0.0000e+00
Epoch 18/200
11/11 [==============================] - 2s 174ms/step - loss: 0.1271 - accuracy: 0.0052 - val_loss: 0.1523 - val_accuracy: 0.0000e+00
Epoch 19/200
11/11 [==============================] - 2s 161ms/step - loss: 0.1253 - accuracy: 0.0052 - val_loss: 0.1502 - val_accuracy: 0.0000e+00
Epoch 20/200
11/11 [==============================] - 2s 161ms/step - loss: 0.1234 - accuracy: 0.0052 - val_loss: 0.1482 - val_accuracy: 0.0000e+00
Epoch 21/200
11/11 [==============================] - 2s 159ms/step - loss: 0.1216 - accuracy: 0.0052 - val_loss: 0.1461 - val_accuracy: 0.0000e+00
Epoch 22/200
11/11 [==============================] - 2s 164ms/step - loss: 0.1198 - accuracy: 0.0052 - val_loss: 0.1441 - val_accuracy: 0.0000e+00
Epoch 23/200
11/11 [==============================] - 2s 164ms/step - loss: 0.1180 - accuracy: 0.0052 - val_loss: 0.1421 - val_accuracy: 0.0000e+00
Epoch 24/200
11/11 [==============================] - 2s 163ms/step - loss: 0.1162 - accuracy: 0.0052 - val_loss: 0.1401 - val_accuracy: 0.0000e+00
Epoch 25/200
11/11 [==============================] - 2s 167ms/step - loss: 0.1145 - accuracy: 0.0052 - val_loss: 0.1381 - val_accuracy: 0.0000e+00
Epoch 26/200
11/11 [==============================] - 2s 188ms/step - loss: 0.1127 - accuracy: 0.0052 - val_loss: 0.1361 - val_accuracy: 0.0000e+00
Epoch 27/200
11/11 [==============================] - 2s 169ms/step - loss: 0.1110 - accuracy: 0.0052 - val_loss: 0.1342 - val_accuracy: 0.0000e+00
Epoch 28/200
11/11 [==============================] - 2s 189ms/step - loss: 0.1093 - accuracy: 0.0052 - val_loss: 0.1323 - val_accuracy: 0.0000e+00
Epoch 29/200
11/11 [==============================] - 2s 183ms/step - loss: 0.1076 - accuracy: 0.0079 - val_loss: 0.1304 - val_accuracy: 0.0000e+00
Epoch 30/200
11/11 [==============================] - 2s 172ms/step - loss: 0.1059 - accuracy: 0.0079 - val_loss: 0.1285 - val_accuracy: 0.0000e+00
Epoch 31/200
11/11 [==============================] - 2s 164ms/step - loss: 0.1042 - accuracy: 0.0079 - val_loss: 0.1266 - val_accuracy: 0.0000e+00
Epoch 32/200
Accuracy always remains very low and sometimes (like this case) val_accuracy becomes 0 and never changes. While loss and val_loss do not converge well but decrease. I realize that I am certainly doing many things wrong and I cannot understand how I can fix it. I have obviously tried with other hyperparameters and also with other networks like LSTM, but I didn't get satisfactory results.
How can I improve the model so that the accuracy is at least decent? Any advice is welcome, thank you very much!

My CNN Keras doesn't predict properly and I don't know what to do

I have to do a CNN to diagnosis Diabetic retinopathy in 4th stage (binary classification - or 0 (non4thStage - nonPdr), or 1 (4thStage - pdr)) I'm using vgg16 and a gaussianBlur to better classification. I have 1400 test images (700 each class) and this is my train.py:
#import tensorflow as tf
import cv2
import os
import numpy as np
from keras.layers.core import Flatten, Dense, Dropout, Reshape
from keras.models import Model
from keras.layers import Input, ZeroPadding2D, Dropout
from keras import optimizers
from keras.optimizers import SGD
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import EarlyStopping
from keras.applications.vgg16 import VGG16
TRAIN_DIR = 'train/'
TEST_DIR = 'test/'
v = 'v/'
BATCH_SIZE = 32
NUM_EPOCHS = 5
def ReadImages(Path):
LabelList = list()
ImageCV = list()
classes = ["nonPdr", "pdr"]
# Get all subdirectories
FolderList = [f for f in os.listdir(Path) if not f.startswith('.')]
# Loop over each directory
for File in FolderList:
for index, Image in enumerate(os.listdir(os.path.join(Path, File))):
# Convert the path into a file
ImageCV.append(cv2.resize(cv2.imread(os.path.join(Path, File) + os.path.sep + Image), (224,224)))
LabelList.append(classes.index(os.path.splitext(File)[0]))
ImageCV[index] = cv2.addWeighted (ImageCV[index],4,cv2.GaussianBlur(ImageCV[index] , (0,0) , 10) ,-4 ,128)
return ImageCV, LabelList
data, labels = ReadImages(TRAIN_DIR)
valid, vlabels = ReadImages(TEST_DIR)
vgg16_model = VGG16(weights="imagenet", include_top=True)
# (2) remove the top layer
base_model = Model(input=vgg16_model.input,
output=vgg16_model.get_layer("block5_pool").output)
# (3) attach a new top layer
base_out = base_model.output
base_out = Reshape((25088,))(base_out)
top_fc1 = Dense(64, activation="relu")(base_out)
top_fc1 = Dropout(0.50)(base_out)
# output layer: (None, 5)
top_preds = Dense(1, activation="sigmoid")(top_fc1)
# (4) freeze weights until the last but one convolution layer (block4_pool)
for layer in base_model.layers[0:14]:
layer.trainable = False
# (5) create new hybrid model
model = Model(input=base_model.input, output=top_preds)
# (6) compile and train the model
sgd = SGD(lr=0.000001, momentum=0.9)
model.compile(optimizer=sgd, loss="binary_crossentropy", metrics=["accuracy"])
data = np.asarray(data)
valid = np.asarray(valid)
data = data.astype('float32')
valid = valid.astype('float32')
data /= 255
valid /= 255
labels = np.array(labels)
datagen = ImageDataGenerator(
featurewise_center=True,
featurewise_std_normalization=True,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True)
# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(data)
mean = datagen.mean
std = datagen.std
print(mean, "mean")
print(std, "std")
es = EarlyStopping(monitor='val_loss', verbose=1)
# fits the model on batches with real-time data augmentation:
model.fit_generator(datagen.flow(data, np.array(labels), batch_size=32),
steps_per_epoch=len(data) / 32, epochs=50,
validation_data=(valid, np.array(vlabels)),
nb_val_samples=72, callbacks=[es])
model.save('model.h5')
and this returns the follow:
Epoch 1/50
44/43 [==============================] - 475s 11s/step - loss: 0.9671 - acc: 0.4789 - val_loss: 0.6808 - val_acc: 0.6389
Epoch 2/50
44/43 [==============================] - 467s 11s/step - loss: 0.8427 - acc: 0.5007 - val_loss: 0.6364 - val_acc: 0.6389
Epoch 3/50
44/43 [==============================] - 468s 11s/step - loss: 0.7703 - acc: 0.5204 - val_loss: 0.6136 - val_acc: 0.6806
Epoch 4/50
44/43 [==============================] - 466s 11s/step - loss: 0.7324 - acc: 0.5512 - val_loss: 0.5941 - val_acc: 0.7500
Epoch 5/50
44/43 [==============================] - 466s 11s/step - loss: 0.7074 - acc: 0.5679 - val_loss: 0.5758 - val_acc: 0.7639
Epoch 6/50
44/43 [==============================] - 461s 10s/step - loss: 0.6640 - acc: 0.6146 - val_loss: 0.5584 - val_acc: 0.8194
Epoch 7/50
44/43 [==============================] - 455s 10s/step - loss: 0.6562 - acc: 0.6077 - val_loss: 0.5418 - val_acc: 0.8333
Epoch 8/50
44/43 [==============================] - 458s 10s/step - loss: 0.6076 - acc: 0.6700 - val_loss: 0.5263 - val_acc: 0.8889
Epoch 9/50
44/43 [==============================] - 456s 10s/step - loss: 0.5743 - acc: 0.7005 - val_loss: 0.5119 - val_acc: 0.9167
Epoch 10/50
44/43 [==============================] - 457s 10s/step - loss: 0.5649 - acc: 0.7041 - val_loss: 0.4981 - val_acc: 0.9306
Epoch 11/50
44/43 [==============================] - 452s 10s/step - loss: 0.5654 - acc: 0.7088 - val_loss: 0.4855 - val_acc: 0.9444
Epoch 12/50
44/43 [==============================] - 458s 10s/step - loss: 0.5046 - acc: 0.7616 - val_loss: 0.4740 - val_acc: 0.9444
Epoch 13/50
44/43 [==============================] - 465s 11s/step - loss: 0.5002 - acc: 0.7808 - val_loss: 0.4633 - val_acc: 0.9444
Epoch 14/50
44/43 [==============================] - 459s 10s/step - loss: 0.4694 - acc: 0.7924 - val_loss: 0.4514 - val_acc: 0.9583
Epoch 15/50
44/43 [==============================] - 463s 11s/step - loss: 0.4482 - acc: 0.8184 - val_loss: 0.4432 - val_acc: 0.9444
Epoch 16/50
44/43 [==============================] - 456s 10s/step - loss: 0.4326 - acc: 0.8343 - val_loss: 0.4330 - val_acc: 0.9583
Epoch 17/50
44/43 [==============================] - 454s 10s/step - loss: 0.4291 - acc: 0.8303 - val_loss: 0.4233 - val_acc: 0.9583
Epoch 18/50
44/43 [==============================] - 457s 10s/step - loss: 0.4060 - acc: 0.8376 - val_loss: 0.4145 - val_acc: 0.9583
Epoch 19/50
44/43 [==============================] - 457s 10s/step - loss: 0.3933 - acc: 0.8686 - val_loss: 0.4069 - val_acc: 0.9583
Epoch 20/50
44/43 [==============================] - 455s 10s/step - loss: 0.3786 - acc: 0.8684 - val_loss: 0.3985 - val_acc: 0.9583
Epoch 21/50
44/43 [==============================] - 456s 10s/step - loss: 0.3661 - acc: 0.8774 - val_loss: 0.3902 - val_acc: 0.9583
Epoch 22/50
44/43 [==============================] - 454s 10s/step - loss: 0.3493 - acc: 0.8956 - val_loss: 0.3833 - val_acc: 0.9583
Epoch 23/50
44/43 [==============================] - 456s 10s/step - loss: 0.3355 - acc: 0.9065 - val_loss: 0.3765 - val_acc: 0.9444
Epoch 24/50
44/43 [==============================] - 456s 10s/step - loss: 0.3332 - acc: 0.9053 - val_loss: 0.3680 - val_acc: 0.9583
Epoch 25/50
44/43 [==============================] - 457s 10s/step - loss: 0.3236 - acc: 0.9160 - val_loss: 0.3625 - val_acc: 0.9444
Epoch 26/50
44/43 [==============================] - 458s 10s/step - loss: 0.3097 - acc: 0.9181 - val_loss: 0.3559 - val_acc: 0.9583
Epoch 27/50
44/43 [==============================] - 469s 11s/step - loss: 0.2915 - acc: 0.9242 - val_loss: 0.3517 - val_acc: 0.9444
Epoch 28/50
44/43 [==============================] - 473s 11s/step - loss: 0.2832 - acc: 0.9368 - val_loss: 0.3454 - val_acc: 0.9583
Epoch 29/50
44/43 [==============================] - 468s 11s/step - loss: 0.2747 - acc: 0.9418 - val_loss: 0.3416 - val_acc: 0.9583
Epoch 30/50
44/43 [==============================] - 470s 11s/step - loss: 0.2627 - acc: 0.9508 - val_loss: 0.3350 - val_acc: 0.9722
Epoch 31/50
44/43 [==============================] - 469s 11s/step - loss: 0.2517 - acc: 0.9638 - val_loss: 0.3311 - val_acc: 0.9722
Epoch 32/50
44/43 [==============================] - 470s 11s/step - loss: 0.2517 - acc: 0.9484 - val_loss: 0.3266 - val_acc: 0.9722
Epoch 33/50
44/43 [==============================] - 490s 11s/step - loss: 0.2348 - acc: 0.9560 - val_loss: 0.3211 - val_acc: 0.9722
Epoch 34/50
44/43 [==============================] - 461s 10s/step - loss: 0.2427 - acc: 0.9517 - val_loss: 0.3158 - val_acc: 0.9722
Epoch 35/50
44/43 [==============================] - 467s 11s/step - loss: 0.2260 - acc: 0.9616 - val_loss: 0.3109 - val_acc: 0.9722
Epoch 36/50
44/43 [==============================] - 459s 10s/step - loss: 0.2243 - acc: 0.9706 - val_loss: 0.3064 - val_acc: 0.9722
Epoch 37/50
44/43 [==============================] - 456s 10s/step - loss: 0.2099 - acc: 0.9687 - val_loss: 0.3029 - val_acc: 0.9722
Epoch 38/50
44/43 [==============================] - 457s 10s/step - loss: 0.2094 - acc: 0.9733 - val_loss: 0.2994 - val_acc: 0.9722
Epoch 39/50
44/43 [==============================] - 465s 11s/step - loss: 0.2014 - acc: 0.9744 - val_loss: 0.2941 - val_acc: 0.9722
Epoch 40/50
44/43 [==============================] - 465s 11s/step - loss: 0.1924 - acc: 0.9709 - val_loss: 0.2915 - val_acc: 0.9722
Epoch 41/50
44/43 [==============================] - 457s 10s/step - loss: 0.1908 - acc: 0.9735 - val_loss: 0.2897 - val_acc: 0.9722
Epoch 42/50
44/43 [==============================] - 463s 11s/step - loss: 0.1864 - acc: 0.9709 - val_loss: 0.2861 - val_acc: 0.9722
Epoch 43/50
44/43 [==============================] - 464s 11s/step - loss: 0.1787 - acc: 0.9773 - val_loss: 0.2822 - val_acc: 0.9722
Epoch 44/50
44/43 [==============================] - 468s 11s/step - loss: 0.1820 - acc: 0.9744 - val_loss: 0.2794 - val_acc: 0.9722
Epoch 45/50
44/43 [==============================] - 469s 11s/step - loss: 0.1646 - acc: 0.9818 - val_loss: 0.2763 - val_acc: 0.9722
Epoch 46/50
44/43 [==============================] - 469s 11s/step - loss: 0.1689 - acc: 0.9820 - val_loss: 0.2730 - val_acc: 0.9722
Epoch 47/50
44/43 [==============================] - 471s 11s/step - loss: 0.1495 - acc: 0.9879 - val_loss: 0.2711 - val_acc: 0.9722
Epoch 48/50
44/43 [==============================] - 469s 11s/step - loss: 0.1578 - acc: 0.9858 - val_loss: 0.2676 - val_acc: 0.9722
Epoch 49/50
44/43 [==============================] - 462s 10s/step - loss: 0.1557 - acc: 0.9858 - val_loss: 0.2643 - val_acc: 0.9722
Epoch 50/50
44/43 [==============================] - 454s 10s/step - loss: 0.1501 - acc: 0.9794 - val_loss: 0.2612 - val_acc: 0.9722
This is my predict.py:
from keras.models import load_model
import cv2
import os
import numpy as np
from keras.preprocessing import image
TEST_DIR = 'v/'
pdr = 0
nonPdr = 0
model = load_model('model.h5')
def normalize(x, mean, std):
x[..., 0] -= mean[0]
x[..., 1] -= mean[1]
x[..., 2] -= mean[2]
x[..., 0] /= std[0]
x[..., 1] /= std[1]
x[..., 2] /= std[2]
return x
for filename in os.listdir(r'v/'):
if filename.endswith(".jpg") or filename.endswith(".ppm") or filename.endswith(".jpeg") or filename.endswith(".png"):
ImageCV = cv2.resize(cv2.imread(os.path.join(TEST_DIR) + filename), (224,224))
ImageCV = cv2.addWeighted (ImageCV,4,cv2.GaussianBlur(ImageCV , (0,0) , 10) ,-4 ,128)
ImageCV = np.asarray(ImageCV)
ImageCV = ImageCV.astype('float32')
ImageCV /= 255
x = ImageCV
x = np.expand_dims(x, axis=0)
x = normalize(x, [0.23883381, 0.23883381, 0.23883381], [0.24483591, 0.24579705, 0.2510857])
prob = model.predict(x)
if prob <= 0.75: #.75 = 80% | .70=79% >>>> .70 = 82% | .75 = 79%
print("nonPDR >>>", filename)
nonPdr += 1
else:
print("PDR >>>", filename)
pdr += 1
print(prob)
print("Number of retinas with PDR: ",pdr)
print("Number of retinas without PDR: ",nonPdr)
The problem is: after the train returns about 97% of accuracy, all my predictions are wrong... for example, this 3 images must be PDR(class1):
nonPDR >>> 16_left.jpeg
[[0.07062916]]
nonPDR >>> 16_right.jpeg
[[0.09434311]]
nonPDR >>> 217_left.jpeg
[[0.14126943]]
If I put to test the same images I put in train base, the model doesn't predict properly too...
I'd already tried to train without gaussianBlur but the accuracy was very poor.
What I'm doing wrong? Please, I appreciate your help!!

A couple of things to try: I would suggest not using data augmentation until you have some confidence that your training process is working, even if the performance is not good initially. As a double check, you might want to directly do prediction right after the model.fit, using the training data, just to verify that the resulting accuracy is the same as what you got in training. You might have some small differences in the processing of the test data that is causing the network to behave poorly, so it would be a good first step to convince yourself that the training part is okay then you can focus on the test part. I hope this helps.

Keras model does not generalise

I am trying to build a deep learning model on Keras for a test and I am not very good at this. I have a scaled dataset with 128 features and these correspond to 6 different classes.
I have already tried adding/deleting layers or using regularisation like dropout/l1/l2, My model learns and accuracy goes up so high. But accuracy on test set is around 15%.
from tensorflow.keras.layers import Dense, Dropout
model = Sequential()
model.add(Dense(128, activation='tanh', input_shape=(128,)))
model.add(Dropout(0.5))
model.add(Dense(60, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(20, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(6, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='Nadam', metrics=['accuracy'])
model.fit(train_X, train_y, epochs=20, batch_size=32, verbose=1)
6955/6955 [==============================] - 1s 109us/sample - loss: 1.5805 - acc: 0.3865
Epoch 2/20
6955/6955 [==============================] - 0s 71us/sample - loss: 1.1512 - acc: 0.6505
Epoch 3/20
6955/6955 [==============================] - 0s 71us/sample - loss: 0.9191 - acc: 0.7307
Epoch 4/20
6955/6955 [==============================] - 0s 67us/sample - loss: 0.7819 - acc: 0.7639
Epoch 5/20
6955/6955 [==============================] - 0s 66us/sample - loss: 0.6939 - acc: 0.7882
Epoch 6/20
6955/6955 [==============================] - 0s 69us/sample - loss: 0.6284 - acc: 0.8099
Epoch 7/20
6955/6955 [==============================] - 0s 70us/sample - loss: 0.5822 - acc: 0.8240
Epoch 8/20
6955/6955 [==============================] - 1s 73us/sample - loss: 0.5305 - acc: 0.8367
Epoch 9/20
6955/6955 [==============================] - 1s 75us/sample - loss: 0.5130 - acc: 0.8441
Epoch 10/20
6955/6955 [==============================] - 1s 75us/sample - loss: 0.4703 - acc: 0.8591
Epoch 11/20
6955/6955 [==============================] - 1s 73us/sample - loss: 0.4679 - acc: 0.8650
Epoch 12/20
6955/6955 [==============================] - 1s 77us/sample - loss: 0.4399 - acc: 0.8705
Epoch 13/20
6955/6955 [==============================] - 1s 80us/sample - loss: 0.4055 - acc: 0.8904
Epoch 14/20
6955/6955 [==============================] - 1s 77us/sample - loss: 0.3965 - acc: 0.8874
Epoch 15/20
6955/6955 [==============================] - 1s 77us/sample - loss: 0.3964 - acc: 0.8877
Epoch 16/20
6955/6955 [==============================] - 1s 77us/sample - loss: 0.3564 - acc: 0.9048
Epoch 17/20
6955/6955 [==============================] - 1s 80us/sample - loss: 0.3517 - acc: 0.9087
Epoch 18/20
6955/6955 [==============================] - 1s 78us/sample - loss: 0.3254 - acc: 0.9133
Epoch 19/20
6955/6955 [==============================] - 1s 78us/sample - loss: 0.3367 - acc: 0.9116
Epoch 20/20
6955/6955 [==============================] - 1s 76us/sample - loss: 0.3165 - acc: 0.9192
The result I am recieving 39% With other models like GBM or XGB I can reach upto 85%
What am I doing wrong? Any suggestions?

What is wrong with metrics?

I'm trying to do Simple classification of road picture ( 1way/2way) with CNN, my dataset is composed of 4k images of class 1 and ~4K of class 2, so normaly the classes are equilibrate, each class is stored in different folder.
But the metrics do some kind of "jump" ? i tryed different size of the input_shape, different optimizer ( 'adam','rmsprop'), batch size ( 10,16,20) and i get the same result... any one know what causes this beavior ?
code:
model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=(300, 300,3)))
model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(64, (3, 3)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(128, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
batch_size = 10
train_datagen = ImageDataGenerator(
# rescale=1./255,
# shear_range=0.2,
# zoom_range=0.2,
# horizontal_flip=True
featurewise_std_normalization=True,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest')
test_datagen = ImageDataGenerator(featurewise_std_normalization=True,
rescale=1./255)
train_generator = train_datagen.flow_from_directory(
'data/train',
target_size=(300, 300),
batch_size=batch_size,
class_mode='binary')
validation_generator = test_datagen.flow_from_directory(
'data/validation',
target_size=(300, 300),
batch_size=batch_size,
class_mode='binary')
model.fit_generator(
train_generator,
steps_per_epoch=2000 // batch_size,
epochs=50,
validation_data=validation_generator,
validation_steps=800 // batch_size)
When i run this code i obtain this result:
Epoch 1/50
125/125 [==============================] - 253s 2s/step - loss: 0.8142 - acc: 0.5450 - val_loss: 0.4937 - val_acc: 0.8662
Epoch 2/50
125/125 [==============================] - 254s 2s/step - loss: 0.6748 - acc: 0.5980 - val_loss: 0.5782 - val_acc: 0.7859
Epoch 3/50
125/125 [==============================] - 255s 2s/step - loss: 0.6679 - acc: 0.6580 - val_loss: 0.5068 - val_acc: 0.8562
Epoch 4/50
125/125 [==============================] - 255s 2s/step - loss: 0.6438 - acc: 0.6780 - val_loss: 0.5018 - val_acc: 0.8766
Epoch 5/50
125/125 [==============================] - 257s 2s/step - loss: 0.6427 - acc: 0.7245 - val_loss: 0.3760 - val_acc: 0.9213
Epoch 6/50
125/125 [==============================] - 256s 2s/step - loss: 0.5635 - acc: 0.7435 - val_loss: 0.6140 - val_acc: 0.6398
Epoch 7/50
125/125 [==============================] - 254s 2s/step - loss: 0.6226 - acc: 0.7320 - val_loss: 0.1852 - val_acc: 0.9433
Epoch 8/50
125/125 [==============================] - 252s 2s/step - loss: 0.4858 - acc: 0.7765 - val_loss: 0.1617 - val_acc: 0.9437
Epoch 9/50
125/125 [==============================] - 253s 2s/step - loss: 0.4433 - acc: 0.8120 - val_loss: 0.5577 - val_acc: 0.6788
Epoch 10/50
125/125 [==============================] - 252s 2s/step - loss: 0.4621 - acc: 0.7935 - val_loss: 0.1000 - val_acc: 0.9762
Epoch 11/50
125/125 [==============================] - 254s 2s/step - loss: 0.4572 - acc: 0.8035 - val_loss: 0.3797 - val_acc: 0.8161
Epoch 12/50
125/125 [==============================] - 257s 2s/step - loss: 0.4707 - acc: 0.8105 - val_loss: 0.0903 - val_acc: 0.9761
Epoch 13/50
125/125 [==============================] - 254s 2s/step - loss: 0.4134 - acc: 0.8390 - val_loss: 0.1587 - val_acc: 0.9437
Epoch 14/50
125/125 [==============================] - 252s 2s/step - loss: 0.4023 - acc: 0.8355 - val_loss: 0.1149 - val_acc: 0.9584
Epoch 15/50
125/125 [==============================] - 253s 2s/step - loss: 0.4286 - acc: 0.8255 - val_loss: 0.0897 - val_acc: 0.9700
Epoch 16/50
125/125 [==============================] - 253s 2s/step - loss: 0.4665 - acc: 0.8140 - val_loss: 0.6411 - val_acc: 0.8136
Epoch 17/50
125/125 [==============================] - 252s 2s/step - loss: 0.4010 - acc: 0.8315 - val_loss: 0.1205 - val_acc: 0.9736
Epoch 18/50
125/125 [==============================] - 253s 2s/step - loss: 0.3790 - acc: 0.8550 - val_loss: 0.0993 - val_acc: 0.9613
Epoch 19/50
125/125 [==============================] - 251s 2s/step - loss: 0.3717 - acc: 0.8620 - val_loss: 0.1154 - val_acc: 0.9748
Epoch 20/50
125/125 [==============================] - 250s 2s/step - loss: 0.4434 - acc: 0.8405 - val_loss: 0.1251 - val_acc: 0.9537
Epoch 21/50
125/125 [==============================] - 253s 2s/step - loss: 0.4535 - acc: 0.7545 - val_loss: 0.6766 - val_acc: 0.3640
Epoch 22/50
125/125 [==============================] - 252s 2s/step - loss: 0.7482 - acc: 0.7140 - val_loss: 0.4803 - val_acc: 0.7950
Epoch 23/50
125/125 [==============================] - 252s 2s/step - loss: 0.3712 - acc: 0.8585 - val_loss: 0.1056 - val_acc: 0.9685
Epoch 24/50
125/125 [==============================] - 251s 2s/step - loss: 0.3836 - acc: 0.8545 - val_loss: 0.1267 - val_acc: 0.9673
Epoch 25/50
125/125 [==============================] - 250s 2s/step - loss: 0.3879 - acc: 0.8805 - val_loss: 0.8669 - val_acc: 0.8100
Epoch 26/50
125/125 [==============================] - 250s 2s/step - loss: 0.3735 - acc: 0.8825 - val_loss: 0.1472 - val_acc: 0.9685
Epoch 27/50
125/125 [==============================] - 250s 2s/step - loss: 0.4577 - acc: 0.8620 - val_loss: 0.3285 - val_acc: 0.8925
Epoch 28/50
125/125 [==============================] - 252s 2s/step - loss: 0.3805 - acc: 0.8875 - val_loss: 0.3930 - val_acc: 0.7821
Epoch 29/50
125/125 [==============================] - 250s 2s/step - loss: 0.3565 - acc: 0.8930 - val_loss: 0.1087 - val_acc: 0.9647
Epoch 30/50
125/125 [==============================] - 250s 2s/step - loss: 0.4680 - acc: 0.8845 - val_loss: 0.1012 - val_acc: 0.9688
Epoch 31/50
125/125 [==============================] - 250s 2s/step - loss: 0.3293 - acc: 0.9080 - val_loss: 0.0700 - val_acc: 0.9811
Epoch 32/50
125/125 [==============================] - 250s 2s/step - loss: 0.4197 - acc: 0.9060 - val_loss: 0.1464 - val_acc: 0.9700
Epoch 33/50
125/125 [==============================] - 251s 2s/step - loss: 0.3656 - acc: 0.9005 - val_loss: 8.8236 - val_acc: 0.4307
Epoch 34/50
125/125 [==============================] - 249s 2s/step - loss: 0.4593 - acc: 0.9015 - val_loss: 4.3916 - val_acc: 0.6826
Epoch 35/50
125/125 [==============================] - 250s 2s/step - loss: 0.4824 - acc: 0.8605 - val_loss: 0.0748 - val_acc: 0.9850
Epoch 36/50
125/125 [==============================] - 250s 2s/step - loss: 0.4629 - acc: 0.8875 - val_loss: 0.2257 - val_acc: 0.8728
Epoch 37/50
125/125 [==============================] - 250s 2s/step - loss: 0.3708 - acc: 0.9075 - val_loss: 0.1196 - val_acc: 0.9537
Epoch 38/50
125/125 [==============================] - 250s 2s/step - loss: 0.9151 - acc: 0.8605 - val_loss: 0.1266 - val_acc: 0.9559
Epoch 39/50
125/125 [==============================] - 250s 2s/step - loss: 0.3700 - acc: 0.9035 - val_loss: 0.1038 - val_acc: 0.9812
Epoch 40/50
125/125 [==============================] - 249s 2s/step - loss: 0.5900 - acc: 0.8625 - val_loss: 0.0838 - val_acc: 0.9887
Epoch 41/50
125/125 [==============================] - 250s 2s/step - loss: 0.4409 - acc: 0.9065 - val_loss: 0.0828 - val_acc: 0.9773
Epoch 42/50
125/125 [==============================] - 250s 2s/step - loss: 0.3415 - acc: 0.9115 - val_loss: 0.8084 - val_acc: 0.8788
Epoch 43/50
125/125 [==============================] - 250s 2s/step - loss: 0.5181 - acc: 0.8440 - val_loss: 0.0998 - val_acc: 0.9786
Epoch 44/50
125/125 [==============================] - 249s 2s/step - loss: 0.3270 - acc: 0.8970 - val_loss: 0.1155 - val_acc: 0.9625
Epoch 45/50
125/125 [==============================] - 250s 2s/step - loss: 0.3810 - acc: 0.9125 - val_loss: 0.2881 - val_acc: 0.9484
Epoch 46/50
125/125 [==============================] - 249s 2s/step - loss: 0.3499 - acc: 0.9220 - val_loss: 0.3109 - val_acc: 0.8564
Epoch 47/50
125/125 [==============================] - 250s 2s/step - loss: 0.3505 - acc: 0.9160 - val_loss: 0.0861 - val_acc: 0.9788
Epoch 48/50
125/125 [==============================] - 250s 2s/step - loss: 0.3073 - acc: 0.9225 - val_loss: 0.0999 - val_acc: 0.9874
Epoch 49/50
125/125 [==============================] - 250s 2s/step - loss: 0.4418 - acc: 0.9000 - val_loss: 0.0301 - val_acc: 0.9925
Epoch 50/50
125/125 [==============================] - 250s 2s/step - loss: 0.3501 - acc: 0.9190 - val_loss: 0.0351 - val_acc: 0.9861
it's overfit ? or just the random set parameters position of my loss function ? i will try to found other picture to build new validation dataset ...

"each class is stored in different folder"
So do you mean 1 class is inside 'train' folder,
and another class is inside 'validate' folder?
try setting batch size to 32
and size of training vs validate sets to the ratio of 0.8 vs 0.2
EDIT
I found a link that you may refer to:
https://stats.stackexchange.com/questions/187335/validation-error-less-than-training-error
EDIT
Try getting more samples.
If there's difficulty getting more samples,
try creating/modifying from the existing samples.