We Keep Coding

Higher starting loss compared to previous epoch

I am currently using code from https://keras.io/examples/vision/handwriting_recognition/ which is a tutorial on text recognition. I am using a local dataset to test the model. And during my experiments I have encountered something which made me question.
1.) Is it normal for a loss value to start at a higher value than the previous loss? If not what could be the cause of this and how can I prevent this?
2.) Is a val_loss of 1 good enough for bi-LSTM networks? If not how can I lessen the loss?
Here is the snippet of two consecutive epochs.
1520/1520 [==============================] - 735s 484ms/step - loss: 2.5462 - val_loss: 2.7302
Epoch 12/100
443/1520 [=======>......................] - ETA: 8:18 - loss: 3.9221
Below is the summary of the current model
Layer (type) Output Shape Param # Connected to
==================================================================================================
image (InputLayer) [(None, 128, 32, 1) 0 []
]
Conv1 (Conv2D) (None, 128, 32, 32) 320 ['image[0][0]']
batchnorm1 (BatchNormalization (None, 128, 32, 32) 128 ['Conv1[0][0]']
)
pool1 (MaxPooling2D) (None, 64, 16, 32) 0 ['batchnorm1[0][0]']
Conv2 (Conv2D) (None, 64, 16, 64) 18496 ['pool1[0][0]']
Conv3 (Conv2D) (None, 64, 16, 64) 36928 ['Conv2[0][0]']
batchnorm2 (BatchNormalization (None, 64, 16, 64) 256 ['Conv3[0][0]']
)
pool2 (MaxPooling2D) (None, 32, 8, 64) 0 ['batchnorm2[0][0]']
reshape (Reshape) (None, 32, 512) 0 ['pool2[0][0]']
dense1 (Dense) (None, 32, 64) 32832 ['reshape[0][0]']
dropout_3 (Dropout) (None, 32, 64) 0 ['dense1[0][0]']
bidirectional_9 (Bidirectional (None, 32, 256) 197632 ['dropout_3[0][0]']
)
bidirectional_10 (Bidirectiona (None, 32, 256) 394240 ['bidirectional_9[0][0]']
l)
bidirectional_11 (Bidirectiona (None, 32, 128) 164352 ['bidirectional_10[0][0]']
l)
label (InputLayer) [(None, None)] 0 []
dense2 (Dense) (None, 32, 85) 10965 ['bidirectional_11[0][0]']
ctc_loss (CTCLayer) (None, 32, 85) 0 ['label[0][0]',
'dense2[0][0]']
==================================================================================================
Total params: 856,149
Trainable params: 855,957
Non-trainable params: 192
__________________________________________________________________________________________________
optimizer = Adam
batch_size = 64
total_dataset = 100,000+
activation = relu

To answer the first query:
Yes, it is common for a loss value to start higher than it was in the previous epoch. During each epoch, your model is trained on different batches of data, and the loss is accumulated or averaged (depends on your loss function) over these batches. At the end of the epoch, you observe the loss over the entire dataset. At the start of the next epoch, you observe the loss over the first batch of the dataset that your model is training on.
Your dataset (ideally) follows a general pattern that you want your model to learn. A batch of your dataset will likely contain a sub-pattern out of the general pattern. At the end of an epoch, given that your model has been exposed to the entire dataset before, it will be better optimized to predict the general pattern of your data than a sub-pattern. Therefore, the loss on the batch/ data containing the sub-pattern will be higher.
For the second question:
It's hard to say if a certain numerical value of loss will be good or bad for a network, since your validation loss will depend on many factors. These include what loss function you are using, how many data points were used to compute the loss, and so on. The numerical value of your loss should not matter as long as your model meets the performance criteria you define in your evaluation metric.

Keras different model.evaluate() and model.predict() accuracy on the same data

So I need to be able to use predict on my model to test a benchmark but it doesn't work. When I use predict on the same validation data as my model uses whilst training, I only get an accuracy of .529. When using model.evaluate I get .85, it doesn't make sense and other threads talk about np.argmax or forgetting normalization but I've tried it all.
PS: I use transfer learning so it gets trained twice and some layers get frozen but that shouldn't influence this.
Model: "model_7"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_17 (InputLayer) [(None, 150, 150, 3)] 0
_________________________________________________________________
sequential_8 (Sequential) (None, 150, 150, 3) 0
_________________________________________________________________
normalization_7 (Normalizati (None, 150, 150, 3) 7
_________________________________________________________________
xception (Functional) (None, 5, 5, 2048) 20861480
_________________________________________________________________
global_average_pooling2d_7 ( (None, 2048) 0
_________________________________________________________________
dropout_7 (Dropout) (None, 2048) 0
_________________________________________________________________
dense_7 (Dense) (None, 1) 2049
=================================================================
Total params: 20,863,536
Trainable params: 20,809,001
Non-trainable params: 54,535
model.compile(
optimizer=keras.optimizers.Adam(1e-5), # Low learning rate
loss=keras.losses.BinaryCrossentropy(from_logits=True),
metrics=['accuracy', keras.metrics.BinaryAccuracy(),f1_m,precision_m, recall_m],
)
print(model.evaluate(x_test,y_test))
OUTPUT: 38/38 [==============================] - 2s 45ms/step - loss: 0.3570 - binary_accuracy: 0.8550 - f1_m: 0.8509 - precision_m: 0.8786 - recall_m: 0.8326
[0.3569563925266266, 0.8550000190734863, 0.8509402871131897, 0.8786242604255676, 0.8326380848884583]
better = norm_layer(x_test)
# print(better[0])
y_pred = model.predict(better)
print(y_pred)
y_pred[y_pred <= 0.5] = 0
y_pred[y_pred > 0.5] = 1
print(y_pred)
print(sum(1 for x,y in zip(y_pred,y_test) if x == y) / len(y_pred))
OUTPUT: [[0.42335328]
[0.3409149 ]
[0.45328587]
...
[0.38108858]
[0.44630498]
[0.76832736]]
[[0.]
[0.]
[0.]
...
[0.]
[0.]
[1.]]
0.5291666666666667
Code normalization
# Pre-trained Xception weights requires that input be normalized
# from (0, 255) to a range (-1., +1.), the normalization layer
# does the following, outputs = (inputs - mean) / sqrt(var)
norm_layer = keras.layers.experimental.preprocessing.Normalization()
mean = np.array([127.5] * 3)
var = mean ** 2
# Scale inputs to [-1, +1]
x = norm_layer(x)
norm_layer.set_weights([mean, var])
For full code the following link (the file Xception detection):
https://console.paperspace.com/tees0czt0/notebook/rj824evpaot143v?file=Xception%20detection.ipynb

How to prevent overfitting in Keras sequential model?

I am already adding dropout regularization. I am trying to build a multiclass text classification multilayer perceptron model.
My model:
model = Sequential([
Dropout(rate=0.2, input_shape=features),
Dense(units=64, activation='relu'),
Dropout(rate=0.2),
Dense(units=64, activation='relu'),
Dropout(rate=0.2),
Dense(units=16, activation='softmax')])
My model.summary():
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
dropout_1 (Dropout) (None, 20000) 0
_________________________________________________________________
dense_1 (Dense) (None, 64) 1280064
_________________________________________________________________
dropout_2 (Dropout) (None, 64) 0
_________________________________________________________________
dense_2 (Dense) (None, 64) 4160
_________________________________________________________________
dropout_3 (Dropout) (None, 64) 0
_________________________________________________________________
dense_3 (Dense) (None, 16) 1040
=================================================================
Total params: 1,285,264
Trainable params: 1,285,264
Non-trainable params: 0
_________________________________________________________________
None
Train on 6940 samples, validate on 1735 samples
I am getting:
Epoch 16/1000
- 4s - loss: 0.4926 - acc: 0.8719 - val_loss: 1.2640 - val_acc: 0.6640
Validation accuracy: 0.6639769498140736, loss: 1.2639631692545559
The validation accuracy is ~20% less than the accuracy, and the validation loss is way higher than the training loss.
I am already using dropout regularization, and using epochs = 1000, batch size = 512 and early stopping on val_loss.
Any suggestions?

Why does this neural network have zero accuracy and very low loss?

I have network :
Tensor("input_1:0", shape=(?, 5, 1), dtype=float32)
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) (None, 5, 1) 0
_________________________________________________________________
bidirectional_1 (Bidirection (None, 5, 64) 2176
_________________________________________________________________
activation_1 (Activation) (None, 5, 64) 0
_________________________________________________________________
bidirectional_2 (Bidirection (None, 5, 128) 16512
_________________________________________________________________
activation_2 (Activation) (None, 5, 128) 0
_________________________________________________________________
bidirectional_3 (Bidirection (None, 1024) 656384
_________________________________________________________________
activation_3 (Activation) (None, 1024) 0
_________________________________________________________________
dense_1 (Dense) (None, 1) 1025
_________________________________________________________________
p_re_lu_1 (PReLU) (None, 1) 1
=================================================================
Total params: 676,098
Trainable params: 676,098
Non-trainable params: 0
_________________________________________________________________
None
Train on 27496 samples, validate on 6875 samples
I fit and compile it by:
model.compile(loss='mse',optimizer=Adamx,metrics=['accuracy'])
model.fit(x_train,y_train,batch_size=100,epochs=10,validation_data=(x_test,y_test),verbose=2)
When I run it and also evaluate it on unseen data,it returns 0.0 Accuracy with very low loss. I can't figure out what's the problem.
Epoch 10/10
- 29s - loss: 1.6972e-04 - acc: 0.0000e+00 - val_loss: 1.7280e-04 - val_acc: 0.0000e+00

What you are getting is expected. Your model is working correctly, it is your metrics of measure that is incorrect. The aim of the optimization function is to minimize loss, not to increase accuracy.
Since you are using PRelu as the activation function of your last layer, you always get float output from the network. Comparing these float output with actual label for measure of accuracy doesn't seem the right option. Since the outputs and labels are continuous random variable the joint probability for specific value will be zero. Therefore, even if the model predicts values very close to the true label value the model accuracy still will be zero unless the model predicts exactly the same value as true label - which is improbable.
e.g if y_true is 1.0 and the model predicts 0.99999 still this value does not add value to accuracy of the model since 1.0 != 0.99999
Update
The choice of metrics function depends on the type of problem. Keras also provides functionality for implementing custom metrics.
Assuming the problem on question is linear regression and two values are equal if difference between the two values is less than 0.01, the custom loss metrics can be defined as:-
import keras.backend as K
import tensorflow as tf
accepted_diff = 0.01
def linear_regression_equality(y_true, y_pred):
diff = K.abs(y_true-y_pred)
return K.mean(K.cast(diff < accepted_diff, tf.float32))
Now you can use this metrics for your model
model.compile(loss='mse',optimizer=Adamx,metrics=[linear_regression_equality])

Adding visualization for every layer activation layer of Convolution Network - Keras

I have a Convolution Network (CNN) as followed. I would like to add visualization for every layer activation layer as in the
There are several layer of the CNN that are doing the required task. I only want to probe the output of each layer.
def get_model():
input_shape = (IMG_MODE, img_rows, img_cols)
model = Sequential()
model.add(ZeroPadding2D(padding=(1,1), input_shape=input_shape))
model.add(Conv2D(32, (3, 3), padding = 'valid'))
model.add(LeakyReLU(alpha=0.01))
model.add(MaxPooling2D(pool_size=pool_size2))
....
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
print(model.summary())
return model
The code output:
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
zero_padding2d_1 (ZeroPaddin (None, 1, 114, 94) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 32, 112, 92) 320
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU) (None, 32, 112, 92) 0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 32, 56, 46) 0
_________________________________________________________________
....
_________________________________________________________________
dense_1 (Dense) (None, 1024) 8258560
_________________________________________________________________
leaky_re_lu_4 (LeakyReLU) (None, 1024) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 1024) 0
_________________________________________________________________
dense_2 (Dense) (None, 40) 41000
_________________________________________________________________
activation_1 (Activation) (None, 40) 0
=================================================================
Total params: 8,392,232
Trainable params: 8,392,232
Non-trainable params: 0
_________________________________________________________________
None
Train on 320 samples, validate on 80 samples
Epoch 1/20
- 18s - loss: 3.7036 - acc: 0.0187 - val_loss: 3.6824 - val_acc: 0.0250
Epoch 2/20
- 17s - loss: 3.6903 - acc: 0.0250 - val_loss: 3.6786 - val_acc: 0.0250
...
Epoch 20/20
- 17s - loss: 0.2067 - acc: 0.9312 - val_loss: 0.9892 - val_acc: 0.7625
Test score: 0.9891735315322876
Test accuracy: 0.7625
I tried to use the following code to do my task:
import matplotlib.pyplot as plt
from keras import models
layer_outputs = [layer.output for layer in model.layers[:8]]
activation_model = models.Model(inputs=model.input, outputs=layer_outputs)
activations = activation_model.predict(img_tensor)
import matplotlib.pyplot as plt
plt.matshow(first_layer_activation[0, :, :, 7], cmap='viridis')
layer_names = []
for layer in model.layers[:8]:
layer_names.append(layer.name)
images_per_row = 16
for layer_name, layer_activation in zip(layer_names, activations):
n_features = layer_activation.shape[-1]
size = layer_activation.shape[1]
n_cols = n_features // images_per_row
display_grid = np.zeros((size * n_cols, images_per_row * size))
for col in range(n_cols):
for row in range(images_per_row):
channel_image = layer_activation[0,
:, :,
col * images_per_row + row]
channel_image -= channel_image.mean()
channel_image /= channel_image.std()
channel_image *= 64
channel_image += 128
channel_image = np.clip(channel_image, 0, 255).astype('uint8')
display_grid[col * size : (col + 1) * size,
row * size : (row + 1) * size] = channel_image
scale = 1. / size
plt.figure(figsize=(scale * display_grid.shape[1],
scale * display_grid.shape[0]))
plt.title(layer_name)
plt.grid(False)
plt.imshow(display_grid, aspect='auto', cmap='viridis')

I describe a general approach how to get outputs of convolutional layers of any model and how to visualize them. I will use Keras of TensorFlow, the code for other implementation of Keras may differ slightly.
First a function is required to get outputs of a convolutional layer of a model
def getConvOutput(model,index=-1):
# index=-1 means last convolutional layer
layers = model.layers
return [layer.output for layer in layers if type(layer) is tf.keras.layers.Conv2D][index]
This will be a four dimensional tensor (batch_size,height,width,number_of_channels)
Next we we need a function which constructs a grid for single data element (single image ) for output of convolutional layer. It will construct nearly square image which will be a grid of activation maps.
def mapsToGrid(output):
numMaps = int(output.shape[-1])
#calculate the number of rows and columns which we want to have
numColumns = math.ceil(numMaps**0.5)
numRows = math.ceil(numMaps/numColumns)
# the end of the grid may be filled with zeros
zerosNum = numRows*numColumns-numMaps
zerosShape = [int(i) for i in output.shape]
zerosShape[-1] = zerosNum
zeros = tf.zeros(
zerosShape,
dtype=tf.float32,
name=None)
# extend the activation maps with zeros
concated = tf.concat([output,zeros],-1)
len,width,depth= [s for s in concated.shape]
# unstack the activation maps and construct the grid
mapStack =tf.unstack(concated,axis=2)
rowStacks = [tf.concat(mapStack[i:i+numColumns],axis=1) for i in range(0,numColumns*numRows,numColumns)]
result = tf.concat(rowStacks,axis=0)
return result
Once you have these functions you can get the grid as follows
activation_map_grid_tensor = mapsToGrid(getConvOutput(model)[0])
Index 0 is required as mapsToGrid works with activation maps for single image, so we pick first element of the batch.
Now you can evalutate the tensor, and show it with e.g. cv2.imshow()
This approach is taken from https://github.com/cyberneuron/RT-CNN-Vis which is a platform for CNN visualization. One may also find it easier to get the code from there directly.