I am training a neural network and get the following output. The loss and val_loss are both decreasing, which makes me happy. However, the val_acc keeps constant. What reasons can that have? My data is quite imbalanced, but I am weighing it via the sklearn compute_class_weight function.
Train on 109056 samples, validate on 27136 samples
Epoch 1/200
- 1174s - loss: 1.0353 - acc: 0.5843 - val_loss: 1.0749 - val_acc: 0.7871
Epoch 00001: val_acc improved from -inf to 0.78711, saving model to
nn_best_weights.h5
Epoch 2/200
- 1174s - loss: 1.0122 - acc: 0.6001 - val_loss: 1.0642 - val_acc: 0.9084
Epoch 00002: val_acc improved from 0.78711 to 0.90842, saving model to
nn_best_weights.h5
Epoch 3/200
- 1176s - loss: 0.9974 - acc: 0.5885 - val_loss: 1.0445 - val_acc: 0.9257
Epoch 00003: val_acc improved from 0.90842 to 0.92571, saving model to
nn_best_weights.h5
Epoch 4/200
- 1177s - loss: 0.9834 - acc: 0.5760 - val_loss: 1.0071 - val_acc: 0.9260
Epoch 00004: val_acc improved from 0.92571 to 0.92597, saving model to
nn_best_weights.h5
Epoch 5/200
- 1182s - loss: 0.9688 - acc: 0.5639 - val_loss: 1.0175 - val_acc: 0.9260
Epoch 00005: val_acc did not improve from 0.92597
Epoch 6/200
- 1177s - loss: 0.9449 - acc: 0.5602 - val_loss: 0.9976 - val_acc: 0.9246
Epoch 00006: val_acc did not improve from 0.92597
Epoch 7/200
- 1186s - loss: 0.9070 - acc: 0.5598 - val_loss: 0.9667 - val_acc: 0.9258
Epoch 00007: val_acc did not improve from 0.92597
Epoch 8/200
- 1178s - loss: 0.8541 - acc: 0.5663 - val_loss: 0.9254 - val_acc: 0.9221
Epoch 00008: val_acc did not improve from 0.92597
Epoch 9/200
- 1171s - loss: 0.7859 - acc: 0.5853 - val_loss: 0.8686 - val_acc: 0.9237
Epoch 00009: val_acc did not improve from 0.92597
Epoch 10/200
- 1172s - loss: 0.7161 - acc: 0.6139 - val_loss: 0.8119 - val_acc: 0.9260
Epoch 00010: val_acc did not improve from 0.92597
Epoch 11/200
- 1168s - loss: 0.6500 - acc: 0.6416 - val_loss: 0.7531 - val_acc: 0.9259
Epoch 00011: val_acc did not improve from 0.92597
Epoch 12/200
- 1164s - loss: 0.5967 - acc: 0.6676 - val_loss: 0.7904 - val_acc: 0.9260
Epoch 00012: val_acc did not improve from 0.92597
Epoch 13/200
- 1175s - loss: 0.5608 - acc: 0.6848 - val_loss: 0.7589 - val_acc: 0.9259
Epoch 00013: val_acc did not improve from 0.92597
Epoch 14/200
- 1221s - loss: 0.5377 - acc: 0.6980 - val_loss: 0.7811 - val_acc: 0.9260
Epoch 00014: val_acc did not improve from 0.92597
My model is the following. I know the kernel size is quite large, but that is on purpose because the data is structured in a certain way.
cnn = Sequential()
cnn.add(Conv2D(16, kernel_size=(2, 100), padding='same', data_format="channels_first", input_shape=(1,10, 100)))
cnn.add(LeakyReLU(alpha=0.01))
cnn.add(BatchNormalization())
cnn.add(Conv2D(16, (1, 1)))
cnn.add(LeakyReLU(alpha=0.01))
cnn.add(Conv2D(16, (1, 8)))
cnn.add(Flatten())
rnn = Sequential()
rnn = LSTM(100, return_sequences=False, dropout=0.2)
dense = Sequential()
dense.add(Dense(3, activation='softmax'))
main_input = Input(batch_shape=(512, 1, 1, 10, 100))
model = TimeDistributed(cnn)(main_input)
model = rnn(model)
model = dense(model)
replica = Model(inputs=main_input, outputs=model)
replica.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
It is hard to answer you question not knowing your model.
The possible answers are:
There is nothing wrong with your model. This may be the highest
accuracy you can get.
Your data can be imbalanced or not shuffled. Higher val_acc then acc indicates that the can be something wrong with train, evaluation
and test split. Train accuracy tends to be higher then val_acc on the begining. Then val_acc catches up, or not ;) I can also indicate that there is not much of variance in your dataset, then you could have this kind of behaviour.
Your learn rate may be to big. Try to decrease it.
And I guess the actual metric for the model to minimize is the loss, so during the optimization process you should follow the loss and monitor its improvements.
Check this link for more information how to check your model.
It seems to be the case where learning rate is too high, missing the local minimum and preventing the Neural Network to improve learning:
It would be good if you can customize your optimizer, like this:
learning_rate = 0.008
decay_rate = 5e-6
momentum = 0.65
sgd = SGD(lr=learning_rate,momentum=momentum, decay=decay_rate, nesterov=False)
model.compile(loss="categorical_crossentropy", optimizer=sgd,metrics=['accuracy'])
Also, increase the number of convolutions. Weights may be saturated.
Related
I tried to train a CNN to classify 9 class of image. Each class has 1000 image for training. I tried training on VGG16 and VGG19, both can achieve validation accuracy of 90%. But when I tried to train on InceptionResNetV2 model, the model seems to stuck around 20% and 30%. Below is my code for InceptionResNetV2 and the training. What can I do to improve the training?
base_model = tf.keras.applications.InceptionResNetV2(input_shape=(IMG_HEIGHT, IMG_WIDTH ,3),weights = 'imagenet',include_top=False)
base_model.trainable = False
model = tf.keras.Sequential([
base_model,
Flatten(),
Dense(1024, activation = 'relu', kernel_regularizer=regularizers.l2(0.001)),
LeakyReLU(alpha=0.4),
Dropout(0.5),
BatchNormalization(),
Dense(1024, activation = 'relu', kernel_regularizer=regularizers.l2(0.001)),
LeakyReLU(alpha=0.4),
Dense(9, activation = 'softmax')])
optimizer_model = tf.keras.optimizers.Adam(learning_rate=0.0001, name='Adam', decay=0.00001)
loss_model = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
model.compile(optimizer_model, loss="categorical_crossentropy", metrics=['accuracy'])
Epoch 1/10
899/899 [==============================] - 255s 283ms/step - loss: 4.3396 - acc: 0.3548 - val_loss: 4.2744 - val_acc: 0.3874
Epoch 2/10
899/899 [==============================] - 231s 257ms/step - loss: 3.5856 - acc: 0.4695 - val_loss: 3.9151 - val_acc: 0.3816
Epoch 3/10
899/899 [==============================] - 225s 250ms/step - loss: 3.1451 - acc: 0.4959 - val_loss: 4.8801 - val_acc: 0.2425
Epoch 4/10
899/899 [==============================] - 227s 252ms/step - loss: 2.7771 - acc: 0.5124 - val_loss: 3.7167 - val_acc: 0.3023
Epoch 5/10
899/899 [==============================] - 231s 257ms/step - loss: 2.4993 - acc: 0.5260 - val_loss: 3.7276 - val_acc: 0.3770
Epoch 6/10
899/899 [==============================] - 227s 252ms/step - loss: 2.3148 - acc: 0.5251 - val_loss: 3.7677 - val_acc: 0.3115
Epoch 7/10
899/899 [==============================] - 234s 260ms/step - loss: 2.1381 - acc: 0.5379 - val_loss: 3.4867 - val_acc: 0.2862
Epoch 8/10
899/899 [==============================] - 230s 256ms/step - loss: 2.0091 - acc: 0.5367 - val_loss: 4.1032 - val_acc: 0.3080
Epoch 9/10
899/899 [==============================] - 225s 251ms/step - loss: 1.9155 - acc: 0.5399 - val_loss: 4.1270 - val_acc: 0.2954
Epoch 10/10
899/899 [==============================] - 232s 258ms/step - loss: 1.8349 - acc: 0.5508 - val_loss: 4.3918 - val_acc: 0.2276
VGG-16/19 has a depth of 23/26 layers, whereas, InceptionResNetV2 has a depth of 572 layers. Now, there is minimal domain similarity between medical images and imagenet dataset. In VGG, due to low depth the features you're getting are not that complex and network is able to classify it on the basis of Dense layer features. However, in IRV2 network, as it's too much deep, the output of the fc layer is more complex (visualize it something object like but for imagenet dataset), and, then the features obtained from these layers are unable to connect to the Dense layer features, and, hence overfitting. I think you were able to get my point.
Check out my answer to very similar question of yours on this link: Link. It will help improve your accuracy.
I am trying to train my model using transfer learning, for this I am using VGG16 model, stripped the top layers and froze first 2 layers for using imagenet initial weights. For fine tuning them I am using learning rate 0.0001, activation softmax, dropout 0.5, loss categorical crossentropy, optimizer SGD, classes 46.
I am just unable to understand the behavior while training. Train loss and acc both are fine (loss is decreasing, acc is increasing). Val loss is decreasing and acc is increasing as well, BUT they are always higher than the train loss and acc.
Assuming its overfitting I made the model less complex, increased the dropout rate, added more samples to val data, but nothing seemed to work. I am a newbie so any kind of help is appreciated.
26137/26137 [==============================] - 7446s 285ms/step - loss: 1.1200 - accuracy: 0.3810 - val_loss: 3.1219 - val_accuracy: 0.4467
Epoch 2/50
26137/26137 [==============================] - 7435s 284ms/step - loss: 0.9944 - accuracy: 0.4353 - val_loss: 2.9348 - val_accuracy: 0.4694
Epoch 3/50
26137/26137 [==============================] - 7532s 288ms/step - loss: 0.9561 - accuracy: 0.4530 - val_loss: 1.6025 - val_accuracy: 0.4780
Epoch 4/50
26137/26137 [==============================] - 7436s 284ms/step - loss: 0.9343 - accuracy: 0.4631 - val_loss: 1.3032 - val_accuracy: 0.4860
Epoch 5/50
26137/26137 [==============================] - 7358s 282ms/step - loss: 0.9185 - accuracy: 0.4703 - val_loss: 1.4461 - val_accuracy: 0.4847
Epoch 6/50
26137/26137 [==============================] - 7396s 283ms/step - loss: 0.9083 - accuracy: 0.4748 - val_loss: 1.4093 - val_accuracy: 0.4908
Epoch 7/50
26137/26137 [==============================] - 7424s 284ms/step - loss: 0.8993 - accuracy: 0.4789 - val_loss: 1.4617 - val_accuracy: 0.4939
Epoch 8/50
26137/26137 [==============================] - 7433s 284ms/step - loss: 0.8925 - accuracy: 0.4822 - val_loss: 1.4257 - val_accuracy: 0.4978
Epoch 9/50
26137/26137 [==============================] - 7445s 285ms/step - loss: 0.8868 - accuracy: 0.4851 - val_loss: 1.5568 - val_accuracy: 0.4953
Epoch 10/50
26137/26137 [==============================] - 7387s 283ms/step - loss: 0.8816 - accuracy: 0.4874 - val_loss: 1.4534 - val_accuracy: 0.4970
Epoch 11/50
26137/26137 [==============================] - 7374s 282ms/step - loss: 0.8779 - accuracy: 0.4894 - val_loss: 1.4605 - val_accuracy: 0.4912
Epoch 12/50
26137/26137 [==============================] - 7411s 284ms/step - loss: 0.8733 - accuracy: 0.4915 - val_loss: 1.4694 - val_accuracy: 0.5030
Yes, you are facing over-fitting issue. To mitigate, you can try to implement below steps
1.Shuffle the Data, by using shuffle=True in VGG16_model.fit. Code is shown below:
history = VGG16_model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1,
validation_data=(x_validation, y_validation), shuffle = True)
2.Use Early Stopping. Code is shown below
callback = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=15)
3.Use Regularization. Code for Regularization is shown below (You can try l1 Regularization or l1_l2 Regularization as well):
from tensorflow.keras.regularizers import l2
Regularizer = l2(0.001)
VGG16_model.add(Conv2D(96,11, 11, input_shape = (227,227,3),strides=(4,4), padding='valid', activation='relu', data_format='channels_last',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
VGG16_model.add(Dense(units = 2, activation = 'sigmoid',
activity_regularizer=Regularizer, kernel_regularizer=Regularizer))
4.You can try using BatchNormalization.
5.Perform Image Data Augmentation using ImageDataGenerator. Refer this link for more info about that.
6.If the Pixels are not Normalized, Dividing the Pixel Values with 255 also helps
I want to save my model weights at certain intervals.
I have:
checkpoint = ModelCheckpoint('models/' + self._model_name + '.h5', period=10,
monitor='loss', verbose=1, save_best_only=True, save_weights_only=True, mode='auto')
return self._model.fit(X, Y, epochs=50, verbose=0, callbacks=[checkpoint])
I call this function a few different times. It's a class, so the self._model remains across the different times I call it.
I one run, I get the output:
Epoch 00010: loss improved from inf to 9.95919, saving model to models/2019-04-07-23-02-16.h5
Epoch 00020: loss improved from 9.95919 to 7.46431, saving model to models/2019-04-07-23-02-16.h5
Epoch 00030: loss improved from 7.46431 to 5.46186, saving model to models/2019-04-07-23-02-16.h5
Epoch 00040: loss improved from 5.46186 to 4.57174, saving model to models/2019-04-07-23-02-16.h5
Epoch 00050: loss improved from 4.57174 to 3.75795, saving model to models/2019-04-07-23-02-16.h5
But then after that, I get:
Epoch 00010: loss improved from inf to 20.38285, saving model to models/2019-04-07-23-02-16.h5
Epoch 00020: loss improved from 20.38285 to 11.98181, saving model to models/2019-04-07-23-02-16.h5
Epoch 00030: loss did not improve from 11.98181
Epoch 00040: loss improved from 11.98181 to 10.54640, saving model to models/2019-04-07-23-02-16.h5
Epoch 00050: loss improved from 10.54640 to 6.20022, saving model to models/2019-04-07-23-02-16.h5
So why did it go back to inf? Shouldn't it keep 3.75795 as the lowest loss and therefore continue to use that as the checkpoint?
What am I doing wrong?
You initiate checkpoint in each method call so its a new checkpoint and it starts from inf.
I know it seems a simple solution but I did it with a for loop. I needed to evaluate my model with some developed metrics, so I save weight, then evaluate it based on weights which are generated in each loop.
checkpointer = ModelCheckpoint(filepath="w1.h5", monitor='val_loss', verbose=1, save_best_only=True, mode='min')
for i in range(0,3):
if(i >0):
model.load_weights('weight'+str(i-1)+'.h5')
model.fit(inputX,outputY , validation_data=(inputTestX,outputTest), batch_size=None, epochs=3, steps_per_epoch=200,validation_steps=200, callbacks=[checkpointer])
model.save_weights('model'+str(i)+'.h5')
evaluate(i)
It works and produce such logs. As you can see it does not go back ti inf and it continues training.
Epoch 1/3
98/98 [==============================] - 14s 145ms/step - loss: 14.2190 - acc: 0.0110 - val_loss: 13.9000 - val_acc: 0.0000e+003s - loss: 14.5255 - acc: 0.0126
Epoch 00001: val_loss improved from inf to 13.89997, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 2/3
98/98 [==============================] - 5s 46ms/step - loss: 13.8863 - acc: 0.0128 - val_loss: 13.5243 - val_acc: 0.0000e+00
Epoch 00002: val_loss improved from 13.89997 to 13.52433, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 3/3
98/98 [==============================] - 4s 39ms/step - loss: 13.5929 - acc: 0.0135 - val_loss: 13.2898 - val_acc: 0.0000e+00
Epoch 00003: val_loss improved from 13.52433 to 13.28980, saving model to oldData/main/result/GCN-fullgraph-w1.h5
0.6165177671418206
0.6264390563241374
Epoch 1/3
98/98 [==============================] - 6s 58ms/step - loss: 13.2707 - acc: 0.0156 - val_loss: 12.9703 - val_acc: 0.0027
Epoch 00001: val_loss improved from 13.28980 to 12.97031, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 2/3
98/98 [==============================] - 7s 72ms/step - loss: 12.8552 - acc: 0.0175 - val_loss: 12.6153 - val_acc: 0.0035
Epoch 00002: val_loss improved from 12.97031 to 12.61535, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 3/3
98/98 [==============================] - 5s 55ms/step - loss: 12.5612 - acc: 0.0194 - val_loss: 12.2473 - val_acc: 0.0049
Epoch 00003: val_loss improved from 12.61535 to 12.24730, saving model to oldData/main/result/GCN-fullgraph-w1.h5
0.638404356344817
0.6429751200231312
If I put checkpoint inside for loop, I get this result, which starts from inf:
Epoch 1/3
98/98 [==============================] - 14s 145ms/step - loss: 14.2190 - acc: 0.0110 - val_loss: 13.9000 - val_acc: 0.0000e+003s - loss: 14.5255 - acc: 0.0126
Epoch 00001: val_loss improved from inf to 13.54957, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 2/3
98/98 [==============================] - 5s 46ms/step - loss: 13.8863 - acc: 0.0128 - val_loss: 13.5243 - val_acc: 0.0000e+00
Epoch 00002: val_loss improved from 13.54957 to 13.22187, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 3/3
98/98 [==============================] - 4s 39ms/step - loss: 13.5929 - acc: 0.0135 - val_loss: 13.2898 - val_acc: 0.0000e+00
Epoch 00003: val_loss improved from 13.22187 to 13.105615, saving model to oldData/main/result/GCN-fullgraph-w1.h5
0.6165177671418206
0.6264390563241374
Epoch 1/3
98/98 [==============================] - 6s 58ms/step - loss: 13.2707 - acc: 0.0156 - val_loss: 12.9703 - val_acc: 0.0027
Epoch 00001: val_loss improved from inf to 13.97031, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 2/3
98/98 [==============================] - 7s 72ms/step - loss: 12.8552 - acc: 0.0175 - val_loss: 12.6153 - val_acc: 0.0035
Epoch 00002: val_loss improved from 13.97031 to 12.86802, saving model to oldData/main/result/GCN-fullgraph-w1.h5
Epoch 3/3
98/98 [==============================] - 5s 55ms/step - loss: 12.5612 - acc: 0.0194 - val_loss: 12.2473 - val_acc: 0.0049
Epoch 00003: val_loss improved from 12.86802 to 12.23080, saving model to oldData/main/result/GCN-fullgraph-w1.h5
0.638404356344817
0.6429751200231312
Training convolutional neural network from scratch on my own dataset with Keras and Tensorflow.
learning rate = 0.0001,
5 classes to sort,
no Dropout used,
dataset checked twice, no wrong labels found
Model:
model = models.Sequential()
model.add(layers.Conv2D(16,(2,2),activation='relu',input_shape=(75,75,3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(16,(2,2),activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(32,(2,2),activation='relu'))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dense(128,activation='relu'))
model.add(layers.Dense(5,activation='sigmoid'))
model.compile(optimizer=optimizers.adam(lr=0.0001),
loss='categorical_crossentropy',
metrics=['acc'])
history = model.fit_generator(train_generator,
steps_per_epoch=100,
epochs=50,
validation_data=val_generator,
validation_steps=25)
Everytime when model achieves 25-35 epochs (80-90% accuracy) this happens:
Epoch 31/50
100/100 [==============================] - 3s 34ms/step - loss: 0.3524 - acc: 0.8558 - val_loss: 0.4151 - val_acc: 0.7992
Epoch 32/50
100/100 [==============================] - 3s 34ms/step - loss: 0.3393 - acc: 0.8700 - val_loss: 0.4384 - val_acc: 0.7951
Epoch 33/50
100/100 [==============================] - 3s 34ms/step - loss: 0.3321 - acc: 0.8702 - val_loss: 0.4993 - val_acc: 0.7620
Epoch 34/50
100/100 [==============================] - 3s 33ms/step - loss: 1.5444 - acc: 0.3302 - val_loss: 1.6062 - val_acc: 0.1704
Epoch 35/50
100/100 [==============================] - 3s 34ms/step - loss: 1.6094 - acc: 0.2935 - val_loss: 1.6062 - val_acc: 0.1724
There is some similar problems with answers, but mostly they recommend to lower learning rate, but it doesnt help at all.
UPD: almost all weights and biases in network became nan. Network somehow died inside
Solution in this case:
I changed sigmoid function in last layer to softmax function and drops are gone
Why this worked out?
sigmoid activation function is used for binary (two-class) classifications.
In multiclassification problems we should use softmax function - special extension of sigmoid function for multiclassification problems.
More information: Sigmoid vs Softmax
Special thanks to #desertnaut and #Shubham Panchal for error indication
I want to retrain Google's inception v3 for my own problem with Keras and imagenet weights. The problem is that, when you load inception v3 imagenet weights, you must specify that the number of classes are 1000, as follows:
base_network=inception_v3.InceptionV3(include_top=False, weights='imagenet',classes=1000)
If I the custom number of classes my dataset have (are not 1000), it raises an error saying that if use imagenet weights, you must, mandatory set classes to 1000.
In order to customize the top layer of inception, I've read that you can use a bottleneck. This is nothing more that not use the standard inception top layer and customize it, so, I can use the include_top=False parameter and program my own top layer.
If I do so as follows:
x = base_network.output
x = GlobalAveragePooling2D()(x)
x = Dense(128, activation='relu')(x)
predictions = Dense(globals_params.num_classes, activation='softmax')(x)
network = Model(inputs=base_network.input, outputs=predictions)
It works (trains) but validation loss and accuracy never change, as you can see (obviously, inception layers are set trainable=False in order to keep imagenet weights).
...
Epoch 00071: acc did not improve
Epoch 72/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0471 - acc: 0.9810 - val_loss: 8.5221 - val_acc: 0.4643
Epoch 00072: acc did not improve
Epoch 73/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0354 - acc: 0.9872 - val_loss: 8.4629 - val_acc: 0.4718
Epoch 00073: acc did not improve
Epoch 74/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0277 - acc: 0.9891 - val_loss: 8.2515 - val_acc: 0.4881
Epoch 00074: acc did not improve
Epoch 75/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0330 - acc: 0.9880 - val_loss: 8.5953 - val_acc: 0.4618
Epoch 00075: acc did not improve
Epoch 76/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0402 - acc: 0.9854 - val_loss: 8.3820 - val_acc: 0.4793
Epoch 00076: acc did not improve
Epoch 77/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0337 - acc: 0.9880 - val_loss: 8.1831 - val_acc: 0.4906
Epoch 00077: acc did not improve
Epoch 78/300
2741/2741 [==============================] - 12s 4ms/step - loss: 0.0381 - acc: 0.9858 - val_loss: 8.4118 - val_acc: 0.4756
...
The question is: how can I program a top layer for inception that allows me train on my own dataset and changing my validation accuracy? I've looked for every site on the internet and I didn't find anything.
Thanks in advance!