We Keep Coding

Save tensorflow model with StringLookup layer with encoded vocabulary

I'm having some issues saving a trained TensorFlow model, where I have a StringLookup layer and I'm required to use TFRecods as input for training. A minimal example to reproduce the issue:
First I define the training data
vocabulary = [str(i) for i in range(100, 200)]
X_train = np.random.choice(vocabulary, size=(100,))
y_train = np.random.choice([0,1], size=(100,))
I save it in a file as tfrecords
def _int64_feature(value):
return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
def _string_feature(value):
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[str(value).encode('utf-8')]))
with tf.io.TFRecordWriter('train.tfrecords') as writer:
for i in range(len(X_train)):
example = tf.train.Example(features=tf.train.Features(feature={
'user_id': _string_feature(X_train[i]),
'label': _int64_feature(y_train[i])
}))
writer.write(example.SerializeToString())
Then I use the tf.data API to be able to stream the data into training (the original data doesn't fit into memory)
data = tf.data.TFRecordDataset(['train.tfrecords'])
features = {
'user_id': tf.io.FixedLenFeature([], tf.string),
'label': tf.io.FixedLenFeature([], tf.int64)
}
def parse(record):
parsed = tf.io.parse_single_example(record, features)
return (parsed['user_id'], parsed['label'])
data = data.map(parse)
The data looks like this:
print(list(data.take(5).as_numpy_iterator()))
[(b'166', 1), (b'144', 0), (b'148', 1), (b'180', 0), (b'192', 0)]
The strings of the original dataset were converted to bytes in the process. I have to pass this new vocabulary to the StringLookup contructor, as passing strings and training with bytes will throw an error
new_vocab = [w.encode('utf-8') for w in vocabulary]
inp = tf.keras.Input(shape=(1,), dtype=tf.string)
x = tf.keras.layers.StringLookup(vocabulary=new_vocab)(inp)
x = tf.keras.layers.Embedding(len(new_vocab)+1, 32)(x)
out = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs=[inp], outputs=[out])
model.compile(optimizer='adam', loss='BinaryCrossentropy')
model.fit(data.batch(10), epochs=5)
But when I try to save the model, I get an error because the vocabulary input to the StringLookup layer is encoded as bytes and can't be dumped into json
model.save('model/')
TypeError: ('Not JSON Serializable:', b'100')
I really don't know what to do, I read that TensorFlow recommends using encoded strings instead of normal strings but that doesn't allow to save the model. I also tried to preprocess the data decoding the strings before thay are fed to the model, but I wasn't able to do it without loading all the data into memory (using just tf.data operations)

Using your data and original vocabulary:
import tensorflow as tf
import numpy as np
vocabulary = [str(i) for i in range(100, 200)]
X_train = np.random.choice(vocabulary, size=(100,))
y_train = np.random.choice([0,1], size=(100,))
...
...
data = data.map(parse)
I ran your code (with an additional Flatten layer) and was able to save your model:
inp = tf.keras.Input(shape=(1,), dtype=tf.string)
x = tf.keras.layers.StringLookup(vocabulary=vocabulary)(inp)
x = tf.keras.layers.Embedding(len(vocabulary)+1, 32)(x)
x = tf.keras.layers.Flatten()(x)
out = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs=[inp], outputs=[out])
model.compile(optimizer='adam', loss='BinaryCrossentropy')
model.fit(data.batch(10), epochs=5)
model.save('model/')
Epoch 1/5
10/10 [==============================] - 1s 8ms/step - loss: 0.6949
Epoch 2/5
10/10 [==============================] - 0s 4ms/step - loss: 0.6864
Epoch 3/5
10/10 [==============================] - 0s 5ms/step - loss: 0.6787
Epoch 4/5
10/10 [==============================] - 0s 5ms/step - loss: 0.6707
Epoch 5/5
10/10 [==============================] - 0s 5ms/step - loss: 0.6620
INFO:tensorflow:Assets written to: model/assets
I do not see why you need new_vocab = [w.encode('utf-8') for w in vocabulary].
If you really need to use new_vocab, you can try setting it during training and afterwards setting vocabulary for saving your model, since the only difference is the encoding:
new_vocab = [w.encode('utf-8') for w in vocabulary]
lookup_layer = tf.keras.layers.StringLookup()
lookup_layer.adapt(new_vocab)
inp = tf.keras.Input(shape=(1,), dtype=tf.string)
x = lookup_layer(inp)
x = tf.keras.layers.Embedding(len(new_vocab)+1, 32)(x)
x = tf.keras.layers.Flatten()(x)
out = tf.keras.layers.Dense(1, activation='sigmoid')(x)
model = tf.keras.Model(inputs=[inp], outputs=[out])
model.compile(optimizer='adam', loss='BinaryCrossentropy')
model.fit(data.batch(10), epochs=5)
model.layers[1].adapt(vocabulary)
model.save('/model')
Admittingly, this is quite hacky.

training model for different batch sizes in keras

I want to train my model for different batch sizes i.e: [64, 128]
I am doing it with for loop like below
epoch=2
batch_sizes = [128,256]
for i in range(len(batch_sizes)):
history = model.fit(x_train, y_train, batch_sizes[i], epochs=epochs,
callbacks=[early_stopping, chk], validation_data=(x_test, y_test))
for above code my model produce following results:
Epoch 1/2
311/311 [==============================] - 157s 494ms/step - loss: 0.2318 -
f1: 0.0723
Epoch 2/2
311/311 [==============================] - 152s 488ms/step - loss: 0.1402 -
f1: 0.4360
Epoch 1/2
156/156 [==============================] - 137s 877ms/step - loss: 0.1197 -
f1: **0.5450**
Epoch 2/2
156/156 [==============================] - 136s 871ms/step - loss: 0.1132 -
f1: 0.5756
it looks like the model continues training after completing training for batch size 64, i.e I want to get my model trained for the next batch from scratch, how can I do it kindly guide me.
p.s: what i have tried:
epoch=2
batch_sizes = [128,256]
for i in range(len(batch_sizes)):
history = model.fit(x_train, y_train, batch_sizes[i], epochs=epochs,
callbacks=[early_stopping, chk], validation_data=(x_test, y_test))
keras.backend.clear_session()
it also did not worked

You can write a function to define a model, and you would need to call that before the subsequent fit calls. If your model is contained within model, the weights are updated during training, and they stay that way after the fit call. That is why you need to redefine the model. This can help you
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import numpy as np
X = np.random.rand(1000,5)
Y = np.random.rand(1000,1)
def build_model():
model = Sequential()
model.add(Dense(64,input_shape=(X.shape[1],)))
model.add(Dense(Y.shape[1]))
model.compile(loss='mse',optimizer='Adam')
return model
epoch=2
batch_sizes = [128,256]
for i in range(len(batch_sizes)):
model = build_model()
history = model.fit(X, Y, batch_sizes[i], epochs=epoch, verbose=2)
model.save('Model_' + str(batch_sizes[i]) + '.h5')
Then, the output looks like:
Epoch 1/2
8/8 - 0s - loss: 0.3164
Epoch 2/2
8/8 - 0s - loss: 0.1367
Epoch 1/2
4/4 - 0s - loss: 0.7221
Epoch 2/2
4/4 - 0s - loss: 0.4787

fitting model several times in keras

I am using model.fit() several times, each time is responsible for training a block of layers where other layers are freezed
CODE
# create the base pre-trained model
base_model = efn.EfficientNetB0(input_tensor=input_tensor,weights='imagenet', include_top=False)
# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# add a fully-connected layer
x = Dense(x.shape[1], activation='relu',name='first_dense')(x)
x=Dropout(0.5)(x)
x = Dense(x.shape[1], activation='relu',name='output')(x)
x=Dropout(0.5)(x)
no_classes=10
predictions = Dense(no_classes, activation='softmax')(x)
# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional layers
for layer in base_model.layers:
layer.trainable = False
#FIRST COMPILE
model.compile(optimizer='Adam', loss=loss_function,
metrics=['accuracy'])
#FIRST FIT
model.fit(features[train], labels[train],
batch_size=batch_size,
epochs=top_epoch,
verbose=verbosity,
validation_split=validation_split)
# Generate generalization metrics
scores = model.evaluate(features[test], labels[test], verbose=1)
print(scores)
#Let all layers be trainable
for layer in model.layers:
layer.trainable = True
from tensorflow.keras.optimizers import SGD
#FIRST COMPILE
model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss=loss_function,
metrics=['accuracy'])
#SECOND FIT
model.fit(features[train], labels[train],
batch_size=batch_size,
epochs=no_epochs,
verbose=verbosity,
validation_split=validation_split)
What is weird is that in the second fit, accuracy resulted from first epoch is much lower that the accuracy of the last epoch of the first fit.
RESULT
Epoch 40/40
6286/6286 [==============================] - 14s 2ms/sample - loss: 0.2370 - accuracy: 0.9211 - val_loss: 1.3579 - val_accuracy: 0.6762
874/874 [==============================] - 2s 2ms/sample - loss: 0.4122 - accuracy: 0.8764
Train on 6286 samples, validate on 1572 samples
Epoch 1/40
6286/6286 [==============================] - 60s 9ms/sample - loss: 5.9343 - accuracy: 0.5655 - val_loss: 2.4981 - val_accuracy: 0.5115
I think the weights of the second fit are not taken from the first fit
Thanks in advance!!!

I think this is the result of using a different optimizer. You used Adam in the first fit and SGD in the second fit. Try using Adam in the second fit and see if it works correctly

I solved this by removing the second compiler.

Why a Tensorflow fit call with a validation generator uses the same samples?

I am trying to fit a NN on some data that I stream to my model with generators. I use generators both for the train and for the validation samples. I expected that the fit method would iterate over both the train and the validation datasets, however I saw that the model uses the same validation samples over and over again. That is, the validation generator resets every epoch.
Here is a reproducible example:
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Dense
def gen(use):
while True:
for i in range(10):
print(use + f' using mat of {i}')
X = (np.ones(40)*i).reshape(10,4)
y = np.ones(10).reshape(-1,1)
yield (X,y)
dataset_train = tf.data.Dataset.from_generator(generator = lambda:gen('train'),
output_types = (tf.float32, tf.float32),
output_shapes = ((10,4), (10,1)))
dataset_val = tf.data.Dataset.from_generator(generator = lambda:gen('validation'),
output_types = (tf.float32, tf.float32),
output_shapes = ((10,4), (10,1)))
dataset_train.batch(2)
dataset_val.batch(2)
model = tf.keras.models.Sequential()
model.add(Dense(units = 10))
model.compile(loss = 'mse')
history = model.fit(dataset_train,
steps_per_epoch = 1,
epochs = 5,
shuffle = False,
verbose = 2,
validation_data = dataset_val,
validation_steps = 1)
Note that I use tensorflow 2.2.0-rc3 (the google colab default version).
In this code, my generator produces a 10 by 4 martix of some number n that changes in each iteration. This matrix represent my input features. The labels here are just a vector of ones (10 by 1). What I would have wanted to get in this toy example is that for each epoch i, the features matrix would be a matrix of is. The output I recieve:
Epoch 1/5
train using mat of 0
validation using mat of 0
1/1 - 0s - loss: 1.0000 - val_loss: 0.9937
Epoch 2/5
train using mat of 1
validation using mat of 0
1/1 - 0s - loss: 1.5841 - val_loss: 0.9909
Epoch 3/5
train using mat of 2
validation using mat of 0
1/1 - 0s - loss: 3.8616 - val_loss: 0.9902
Epoch 4/5
train using mat of 3
validation using mat of 0
1/1 - 0s - loss: 7.7457 - val_loss: 0.9906
Epoch 5/5
train using mat of 4
validation using mat of 0
1/1 - 0s - loss: 13.1401 - val_loss: 0.9915
So the training generator works as I expected, but the validation is stuck on 0.
Is there a way to iterate also over the validation dataset?

Internally, Tensorflow extracts the validation data at once and uses it for all the epochs. Refer here. Hence, that is the reason for your output.
I think, even in the documentation it is inline with this, i.e. there is no mention about the generators in validation_data.
From Documentation:
validation_data: Data on which to evaluate the loss and any model metrics at the end of each epoch. The model will not be trained on this data. validation_data will override validation_split. validation_data could be:
tuple (x_val, y_val) of Numpy arrays or tensors
tuple (x_val, y_val, val_sample_weights) of Numpy arrays
dataset
For the first two cases, batch_size must be provided. For the last case, validation_steps could be provided.

Training accuracy while fitting the model not reflected in confusion matrix

I am training an image classification model to classify certain images containing mountain, waterfall and people classes. Currently, I am using Vgg-net (transfer learning) to train this data. While training, I am getting almost 93% accuracy on training data and 90% accuracy on validation data. However, when I want to check the classes being wrongly classified during training by using confusion matrix on the training data, the accuracy seems to be much less. Only about 30% of the data is classified correctly.
I have tried checking the confusion matrix on other similar image classification problems but there it seems to be showing the accuracy that we see while training.
Code to create ImageDataGenerator objects for training and validation
from keras.applications.inception_v3 import preprocess_input, decode_predictions
#Train DataSet Generator with Augmentation
print("\nTraining Data Set")
train_generator = ImageDataGenerator(preprocessing_function=preprocess_input)
train_flow = train_generator.flow(
train_images, train_labels,
batch_size = BATCH_SIZE
)
#Validation DataSet Generator with Augmentation
print("\nValidation Data Set")
val_generator = ImageDataGenerator(preprocessing_function=preprocess_input)
val_flow = val_generator.flow(
validation_images,validation_labels,
batch_size = BATCH_SIZE
)
Code to build the model and compile it
# Initialize InceptionV3 with transfer learning
base_model = applications.vgg16.VGG16(weights='imagenet',
include_top=False,
input_shape=(WIDTH, HEIGHT,3))
# add a global spatial average pooling layer
x = base_model.output
x = GlobalAveragePooling2D()(x)
# and a dense layer
x = Dense(1024, activation='relu')(x)
predictions = Dense(len(categories), activation='softmax')(x)
# first: train only the top layers (which were randomly initialized)
# i.e. freeze all convolutional InceptionV3 layers
for layer in base_model.layers:
layer.trainable = False
# this is the model we will train
model = Model(inputs=base_model.input, outputs=predictions)
# compile the model (should be done *after* setting layers to non-trainable)
model.compile(optimizer=optimizers.RMSprop(lr=1e-4), metrics=['accuracy'], loss='categorical_crossentropy')
model.summary()
Code to fit the model to the training dataset and validate using validation dataset
import math
top_layers_file_path=r"/content/drive/My Drive/intel-image-classification/intel-image-classification/top_layers.iv3.hdf5"
checkpoint = ModelCheckpoint(top_layers_file_path, monitor='loss', verbose=1, save_best_only=True, mode='min')
tb = TensorBoard(log_dir=r'/content/drive/My Drive/intel-image-classification/intel-image-classification/logs', batch_size=BATCH_SIZE, write_graph=True, update_freq='batch')
early = EarlyStopping(monitor="loss", mode="min", patience=5)
csv_logger = CSVLogger(r'/content/drive/My Drive/intel-image-classification/intel-image-classification/logs/iv3-log.csv', append=True)
history = model.fit_generator(train_flow,
epochs=30,
verbose=1,
validation_data=val_flow,
validation_steps=math.ceil(validation_images.shape[0]/BATCH_SIZE),
steps_per_epoch=math.ceil(train_images.shape[0]/BATCH_SIZE),
callbacks=[checkpoint, early, tb, csv_logger])
training steps show the following accuracy:
Epoch 1/30
91/91 [==============================] - 44s 488ms/step - loss: 0.6757 - acc: 0.7709 - val_loss: 0.4982 - val_acc: 0.8513
Epoch 2/30
91/91 [==============================] - 32s 349ms/step - loss: 0.4454 - acc: 0.8395 - val_loss: 0.3980 - val_acc: 0.8557
.
.
Epoch 20/30
91/91 [==============================] - 32s 349ms/step - loss: 0.2026 - acc: 0.9238 - val_loss: 0.2684 - val_acc: 0.8940
.
.
Epoch 30/30
91/91 [==============================] - 32s 349ms/step - loss: 0.1739 - acc: 0.9364 - val_loss: 0.2616 - val_acc: 0.8984
Ran predictions on the training dataset itself:
import math
import numpy as np
predictions = model.predict_generator(
train_flow,
verbose=1,
steps=math.ceil(train_images.shape[0]/BATCH_SIZE))
predicted_classes = [x[0] for x in enc.inverse_transform(predictions)]
true_classes = [x[0] for x in enc.inverse_transform(train_labels)]
enc - OneHotEncoder
However, the confusion matrix looks like the following:
import sklearn
sklearn.metrics.confusion_matrix(
true_classes,
predicted_classes,
labels=['mountain','people','waterfall'])
Confusion Matrix (Could not upload a better looking picture)
([[315, 314, 283],
[334, 309, 273],
[337, 280, 263]])
The complete code has been uploaded on https://nbviewer.jupyter.org/github/paridhichoudhary/scene-image-classification/blob/master/Classification_v7.ipynb

I believe it's because of the train_generator.flow has shuffle=True by default. This result in predicted_classes doesn't match the train_labels.
Maybe set shuffle=False on train_generator.flow should help this, or instead use something like this might easier to understand.
predicted_classes = []
true_classes = []
for i in range(len(train_flow)): # you can just use `len(train_flow)` instead of `math.ceil(train_images.shape[0]/BATCH_SIZE))`
x, y = train_flow[i] # you can use `train_flow[i]` like this
z = model.predict(x)
for j in range(x.shape[0]):
predicted_classes.append(z[j])
true_classes.append(y[j])
I didn't try this yet but it should work though.