We Keep Coding

How can I call a test set at the end of each epoch during the training? I am using tensorflow

I am using Tensorflow-Keras to develop a CNN model in which I have split the data set into train, validation, and test sets. I need to call the test set at the end of each epoch as well as train and validation sets to evaluate the model performance. Below is my code to track train and validation sets.
result_dic = {"epochs": []}
json_logging_callback = LambdaCallback(
on_epoch_begin=lambda epoch, logs: [learning_rate],
on_epoch_end=lambda epoch, logs:
result_dic["epochs"].append({
'epoch': epoch + 1,
'acc': str(logs['acc']),
'val_acc': str(logs['val_acc'])
}))
model.fit(x_train, y_train,
validation_data=(x_val, y_val),
batch_size=batch_size,
epochs=epochs,
callbacks=[json_logging_callback])
output:
Epoch 1/5
1/1 [==============================] - 4s 4s/step - acc: 0.8611 - val_acc: 0.8333
However, I'm not sure how to add the test set to my callback to produce the following output.
Expected output:
Epoch 1/5
1/1 [==============================] - 4s 4s/step - acc: 0.8611 - val_acc: 0.8333 - test_acc: xxx

To display your test accuracy after each epoch, you could customize your fit function to display this metric. Check out this documentation or you could, as shown here, define a simple callback for your test dataset and pass it into your fit function:
model.fit(x_train, y_train,
validation_data=(x_val, y_val),
batch_size=batch_size,
epochs=epochs,
callbacks=[json_logging_callback,
your_test_callback((X_test, Y_test))])
If you want complete flexibility, you could try using a training loop.
Update: Since you want to have a single JSON for all metrics, you should do the following:
Define your TestCallBack and add your test accuracy (and loss if you want) to your logs dictionary:
import tensorflow as tf
class TestCallback(tf.keras.callbacks.Callback):
def __init__(self, test_data):
self.test_data = test_data
def on_epoch_end(self, epoch, logs):
x, y = self.test_data
loss, acc = self.model.evaluate(x, y, verbose=0)
logs['test_accuracy'] = acc
Then add the test accuracy to your results dictionary:
result_dic = {"epochs": []}
json_logging_callback = tf.keras.callbacks.LambdaCallback(
on_epoch_begin=lambda epoch, logs: [learning_rate],
on_epoch_end=lambda epoch, logs:
result_dic["epochs"].append({
'epoch': epoch + 1,
'acc': str(logs['accuracy']),
'val_acc': str(logs['val_accuracy']),
'test_acc': str(logs['test_accuracy'])
}))
And then just use both callbacks in your fit function but note the order of the callbacks:
model.fit(x_train, y_train,
validation_data=(x_val, y_val),
batch_size=batch_size,
epochs=epochs,
callbacks=callbacks=[TestCallback((x_test, y_test)), json_logging_callback])

Unknown metric function: val_accuracy tensorflow keras

I am trying to train DenseNet121 (among other models) in tensorflow/keras and I need to keep track of accuracy and val_accuracy. However, running this does not log the val_accuracy in the model's history:
clf_model = tf.keras.models.Sequential()
clf_model.add(pre_trained_model)
clf_model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))
clf_model.compile(loss='sparse_categorical_crossentropy', metrics=['accuracy'])
history = clf_model.fit(train_processed, epochs=10, validation_data=validation_processed)
output (no val_accuracy, I need val_accuracy):
Epoch 1/10
1192/1192 [==============================] - 75s 45ms/step - loss: 2.3908 - accuracy: 0.4374
Epoch 2/10
451/1192 [==========>...................] - ETA: 22s - loss: 1.3556 - accuracy: 0.6217
When I tried to pass val_accuracy to the metrics as follows:
clf_model = tf.keras.models.Sequential()
clf_model.add(pre_trained_model)
clf_model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))
clf_model.compile(loss='sparse_categorical_crossentropy', metrics=['accuracy','val_accuracy'])
history = clf_model.fit(train_processed, epochs=10, validation_data=validation_processed)
I get the following error:
ValueError: Unknown metric function: val_accuracy. Please ensure this object is passed to the `custom_objects` argument.
See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.
Any idea what am I doing wrong?
Update
It turned out the test dataset was empty.

training model for different batch sizes in keras

I want to train my model for different batch sizes i.e: [64, 128]
I am doing it with for loop like below
epoch=2
batch_sizes = [128,256]
for i in range(len(batch_sizes)):
history = model.fit(x_train, y_train, batch_sizes[i], epochs=epochs,
callbacks=[early_stopping, chk], validation_data=(x_test, y_test))
for above code my model produce following results:
Epoch 1/2
311/311 [==============================] - 157s 494ms/step - loss: 0.2318 -
f1: 0.0723
Epoch 2/2
311/311 [==============================] - 152s 488ms/step - loss: 0.1402 -
f1: 0.4360
Epoch 1/2
156/156 [==============================] - 137s 877ms/step - loss: 0.1197 -
f1: **0.5450**
Epoch 2/2
156/156 [==============================] - 136s 871ms/step - loss: 0.1132 -
f1: 0.5756
it looks like the model continues training after completing training for batch size 64, i.e I want to get my model trained for the next batch from scratch, how can I do it kindly guide me.
p.s: what i have tried:
epoch=2
batch_sizes = [128,256]
for i in range(len(batch_sizes)):
history = model.fit(x_train, y_train, batch_sizes[i], epochs=epochs,
callbacks=[early_stopping, chk], validation_data=(x_test, y_test))
keras.backend.clear_session()
it also did not worked

You can write a function to define a model, and you would need to call that before the subsequent fit calls. If your model is contained within model, the weights are updated during training, and they stay that way after the fit call. That is why you need to redefine the model. This can help you
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import numpy as np
X = np.random.rand(1000,5)
Y = np.random.rand(1000,1)
def build_model():
model = Sequential()
model.add(Dense(64,input_shape=(X.shape[1],)))
model.add(Dense(Y.shape[1]))
model.compile(loss='mse',optimizer='Adam')
return model
epoch=2
batch_sizes = [128,256]
for i in range(len(batch_sizes)):
model = build_model()
history = model.fit(X, Y, batch_sizes[i], epochs=epoch, verbose=2)
model.save('Model_' + str(batch_sizes[i]) + '.h5')
Then, the output looks like:
Epoch 1/2
8/8 - 0s - loss: 0.3164
Epoch 2/2
8/8 - 0s - loss: 0.1367
Epoch 1/2
4/4 - 0s - loss: 0.7221
Epoch 2/2
4/4 - 0s - loss: 0.4787

Tensorflow model.evaluate gives different result from that obtained from training

I am using tensorflow to do a multi-class classification
I load the training dataset and validation dataset in the following way
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="training",
shuffle=True,
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
val_ds = tf.keras.preprocessing.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
shuffle=True,
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
Then when I train the model using model.fit()
history = model.fit(
train_ds,
validation_data=val_ds,
epochs=epochs,
shuffle=True
)
I get validation accuracy around 95%.
But when I load the same validation set and use model.evaluate()
model.evaluate(val_ds)
I get very low accuracy (around 10%).
Why am I getting such different results? Am I using the model.evaluate function incorrectly?
Note : In the model.compile() I am specifying the following,
Optimizer - Adam,
Loss - SparseCategoricalCrossentropy,
Metric - Accuracy
Model.evaluate() output
41/41 [==============================] - 5s 118ms/step - loss: 0.3037 - accuracy: 0.1032
Test Loss - 0.3036555051803589
Test Acc - 0.10315627604722977
Model.fit() output for last three epochs
Epoch 8/10
41/41 [==============================] - 3s 80ms/step - loss: 0.6094 - accuracy: 0.8861 - val_loss: 0.4489 - val_accuracy: 0.9483
Epoch 9/10
41/41 [==============================] - 3s 80ms/step - loss: 0.5377 - accuracy: 0.8953 - val_loss: 0.3868 - val_accuracy: 0.9554
Epoch 10/10
41/41 [==============================] - 3s 80ms/step - loss: 0.4663 - accuracy: 0.9092 - val_loss: 0.3404 - val_accuracy: 0.9590

Try saving your model as .h5 instead of .tf - this worked for me in Keras 2.4.0 and tensorflow 2.4.0

Why am I getting such different results? Am I using the model.evaluate
function incorrectly?
I suppose that it is the over fitting that cause this issue. You can check them out in this way!
Extract the history of model
history_dict = history.history
history_dict.keys()
Visualize the history
import matplotlib.pyplot as plt
acc=history_dict['accuracy']
val_acc=history_dict['val_accuracy']
loss=history_dict['loss']
val_loss=history_dict['val_loss']
epochs=range(1,len(acc)+1)
plt.figure(figsize=(10,10))
ax1=plt.subplot(221)
ax1.plot(epochs,loss,'bo',label='Training loss')
ax1.plot(epochs,acc,'ro',label='Training acc')
ax1.set_title('loss and acc of Training')
ax1.set_xlabel('Epochs')
ax1.set_ylabel('Loss')
ax1.legend()
ax2=plt.subplot(222)
ax2.plot(epochs,val_acc,'r',label='Validation acc')
ax2.plot(epochs,val_loss,'b',label='Validation loss')
ax2.set_title('loss and acc of Training')
ax2.set_xlabel('Epochs')
ax2.set_ylabel('Acc')
ax2.legend()
Maybe, the results you get are like these:
In training process, acc and loss changed with epochs
But in validation, acc and loss seem to reached a peak after 20 epochs
Solution
It turns out that, when overfitting occurs, fewer epochs can be set to avoid this problem!

Accuracy metric fails with trivial custom Keras loss function

I am playing around with custom loss functions on Keras models. My "custom" loss seems to fail (in terms of accuracy score), even though I am only using a wrapper that returns an original keras loss.
As a toy example, I am using the "Basic classification" Tensorflow/Keras tutorial that uses a simple NN on the fashion-MNIST data set and I am following the related Keras documentation and this SO post.
This is the model:
model = keras.Sequential([
keras.layers.Flatten(input_shape=(28, 28)),
keras.layers.Dense(128, activation='relu'),
keras.layers.Dense(10, activation='softmax')
])
Now, if I leave the sparse_categorical_crossentropy as a string argument in compile() function, the training results to a ~ 87% accuracy which is fine:
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(train_images, train_labels, epochs=10)
test_loss, test_acc = model.evaluate(test_images, test_labels, verbose=2)
print('\nTest accuracy:', test_acc)
But when I just create a trivial wrapper function to call keras' cross-entropy I get a ~ 10% accuracy both on training and test sets:
from tensorflow.keras import losses
def my_loss(y_true, y_pred):
return losses.sparse_categorical_crossentropy(y_true, y_pred)
model.compile(optimizer='adam',
loss=my_loss,
metrics=['accuracy'])
Epoch 1/10 60000/60000 [==============================] - 3s 51us/sample - loss: 0.5030 - accuracy: 0.1032
Epoch 2/10 60000/60000 [==============================] - 3s 45us/sample - loss: 0.3766 - accuracy: 0.1035
...
Test accuracy: 0.1013
By plotting a few images and checking their classified labels, it doesn't look like the results differ in each case, but accuracies printed are very different. So, is it the case that the default metrics do not play nicely with custom losses? Can it be the case that what I see is the error rather than the accuracy? Am I missing something from the documentation?
Edit: The values of the loss functions in both cases end up roughly the same, so training indeed takes place. The accuracy is the point of failure.

Here's the reason:
When you use inbuilt loss and use loss='sparse_categorical_crossentropy' at that time accuracy metric used is sparse_categorical_accuracy But when you use custom loss function at that time accuracy metric used is categorical_accuracy.
Example:
model.compile(optimizer='adam',
loss=losses.sparse_categorical_crossentropy,
metrics=['categorical_accuracy', 'sparse_categorical_accuracy'])
model.fit(train_images, train_labels, epochs=1)
'''
Train on 60000 samples
60000/60000 [==============================] - 5s 86us/sample - loss: 0.4955 - categorical_accuracy: 0.1045 - sparse_categorical_accuracy: 0.8255
'''
model.compile(optimizer='adam',
loss=my_loss,
metrics=['accuracy', 'sparse_categorical_accuracy'])
model.fit(train_images, train_labels, epochs=1)
'''
Train on 60000 samples
60000/60000 [==============================] - 5s 87us/sample - loss: 0.4956 - acc: 0.1043 - sparse_categorical_accuracy: 0.8256
'''