I trained a model on initial data, got some good scores, and now after receiving more data I want to load the pre-trained model and continue to train.
Here some snippet of what I did:
(1) I follow this post which says to save the model in 'tf' format
# saving initial model
model.save(path2initial, save_format='tf')
# load pre-trained model
clf = tf.keras.models.load_model(path2initial)
# create new data generators
train_gen = generators.create(generator_config, 'train')
val_gen = generators.create(generator_config, 'val')
# create metrics, loss, optimizer and callbacks
loss = losses.create(loss_config)
callback_list = callbacks.create(callback_config)
optimizer = optimizers.create(optimizer_confing)
metrics = metrics.create(metrics_config)
# compile model
clf.compile(optimizer=optimizer, loss=loss, metrics=metrics)
# train
clf.fit(x=train_gen,
epochs=NB_EPCOHS,
validation_data=val_gen,
steps_per_epochs=math.ceil(len(train_steps)/ BATCH_SIZE),
validation_steps=math.ceil(len(val_steps)/ BATCH_SIZE),
callbacks=callback_list,
use_multiprocessing=True,
workers=16,
max_queue_size=8,
verbose=1
)
I should note that two of my callbacks are
EarlyStopping(monitor='val_loss', restore_best_weights=True,
min_delta=0.001, patience=10, mode='min', verbose=1)
ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=True,
mode='min', save_freq='epoch')
And that train_gen is consisted with both initial_data and new_data.
This method trained only for 4 epochs, and hadn't changed for the
rest of the 10 'patient' epochs. Moreover the results were way worse than the initial model's results.
(2) The second method I tried was to save the model in the default format (that's the only change):
model.save(path2initial)
.
.
.
This model had trained for 71/200 epochs, but it seems that it ignored my EarlyStopping() callback. In some epochs the val-loss had changed by 1e-4 or even less, and still it continue with the training (weirddd), And it stopped (by EarlyStopping()) in epoch 71 even that the val-loss had change! Moreover, the results had barely changed.
For comparison I trained a model from scratch on all the data (both initial and new data) and got way better results:
Initial data Method (1) Method (2) New model on all data
mean F1 score: 0.735 0.422 0.74 0.803
Is there a proven way to how to continue training a keras model?
When loading the model does the optimizer status reset?
When loading the model, do I need to define all the callbacks, loss, opt, metrics all over again? Do I need to compile it again?
Related
I would like to know if it's possible to know the loss/accuracy with model.predict() in Keras?
I tried to figured out by myself, but I failed. Thank you very much for any inputs.
Keras model.predict() only gets the input data (X) and produces the output from the trained model. It does not know anything about the actual expected value (y).
You can do what you are asking for using model.evaluate(), that actually requires the X and y values in your data set and will produce the loss value and metrics values for the model in test mode.
Use model.predict() in production, to just get the model output. During your testing and validation phase, you may want to also use model.evaluate().
So my code is like this :
# Fetch dataset
train_data, test_data = tfds.load(name="imdb_reviews", split=["train", "test"],
batch_size=-1, as_supervised=True)
train_examples, train_labels = tfds.as_numpy(train_data)
test_examples, test_labels = tfds.as_numpy(test_data)
np.save("train_examples", train_examples)
np.save("train_labels", train_labels)
np.save("test_examples", test_examples)
np.save("test_labels", test_labels)
# BUILD MODEL
model = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1"
hub_layer = hub.KerasLayer(model, output_shape=[], input_shape=[],
dtype=tf.string, trainable=True, name='gnews_embedding')
model = build_model(hub_layer)
model.summary()
# SAVE AS CHECKPOINT (THE BEST ONLY)
es = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=4, verbose=1, mode='min')
checkpoint = tf.keras.callbacks.ModelCheckpoint('model.h5', monitor='val_loss', save_best_only=True)
# TRAIN MODEL
history = model.fit(
train_examples,
train_labels,
epochs=20,
batch_size=BATCH_SIZE,
validation_split = .2,
shuffle = True,
callbacks = [checkpoint, es],
verbose=1)
# CHECK ACCURACY AND LOSS VALUE
model.load_weights('/app/model.h5')
results = model.evaluate(test_examples, test_labels)
text = "The gold rush apple from natora- this is the most expensive apple."
dataset = tf.data.Dataset.from_tensor_slices([text])
np.save("test_txt", test_txt)
resultsTest = model.evaluate(dataset, test_labels)
print("RESULT ACCURACY = ", resultsTest)
so I try to know the accuracy/loss of this prediction. Can i evaluate a simple sentence like that or just re-train the model to get the loss/accuracy ?
I'm trying to train multiple machine learning models using tensorflow keras, I was just wondering is there a way to obtain the best score achieved while training after training is complete. I found online that the .fit function returns a history object which can be accessed to get the best score, though from code i've tried it says "AttributeError: 'History' object has no attribute 'best_score'", I cannot find an attribute list online so this is why I am asking here.
Thanks in advance.
History = model.fit(Xtrain, ytrain, epochs=1, validation_data=(Xtest, ytest), verbose=1)
print("Best: %f using %s" % (History.best_score, History.best_params_))
PS, I know training for 1 epoch will achieve nothing, I'm just trying to test the code
I'm assuming you just want the best score from the history object.
hist = model.fit(...)
print(hist.history) # this will print a dictionary object, now you need to grab the metrics / score you're looking for
# if your score == 'acc', if not replace 'acc' with your metric
best_score = max(hist.history['acc'])
print(best_score)
If your metric is 'accuracy', use that instead.
If you want the best model, you can just use ModelCheckPoint https://keras.io/callbacks/
The fit() method on keras return a history object. The history.history attribute is a dictionary recording training loss values and metrics values at successive epochs, as well as validation loss values and validation metrics values (if applicable). You can call "history.history['loss']" or "history.history['val_loss']" to access it. Here is an example of it.
model.compile(loss = 'mean_squared_error', optimizer =optimizer,metrics=['accuracy'])
History = model.fit(Xtrain, ytrain, epochs=1, validation_data=(Xtest, ytest), verbose=1)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
For the best params, I think using callbacks.ModelCheckpoint might be helpful. This method has an argument 'save_best_only' to the latest best model.
keras.callbacks.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)
Kindly refer to the Training history visualization and callback for more information.
I’m trying to solve a problem in my CNN model. I'm using a dataset that is structured as follow:
My data is organized as follow:
train
parasitized
uninfected
test
parasitized
uninfected
validation
parasitized
uninfected
My dataset is too large and i'm using ImageDataGenerator to preprocess the images and also upload them in batches (reduce computational cost). In a first i configured the ImageDataGenerator as follows:
from keras.preprocessing.image import ImageDataGenerator
#Define a ImageDataGenerator for each dataset.
#This augmentation process is only to rescale each imagem to 1/255
datagen_train = ImageDataGenerator(rescale=1./255) #rescale=1./255
datagen_test = ImageDataGenerator(rescale=1./255)
datagen_valid = ImageDataGenerator(rescale=1./255)
#Define a batch_size parameter
batch_size=32
# Here .flow_from_directory is used to transform
train_generator = datagen_train.flow_from_directory(
'content/cell_images/train', #Train folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=batch_size,
class_mode='categorical') # We use categorical_crossentropy loss,
# we need categorical labels
test_generator = datagen_test.flow_from_directory(
'content/cell_images/test', #Test folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=batch_size,
class_mode='categorical')
valid_generator = datagen_valid.flow_from_directory(
'content/cell_images/valid',
target_size=(150,150),
batch_size=32,
class_mode='categorical')
To fit the model it was used fit_generator and a checkpointer to save the best weights based on the validation_accuracy:
from keras.callbacks import ModelCheckpoint
# Define epochs number
epochs = 10
# Create a checkpointer to save only the best params
checkpointer = ModelCheckpoint(filepath='cnn_model.weights.best.hdf5',
verbose=1, save_best_only=True)
model.fit_generator(train_generator,
steps_per_epoch=train_generator.samples//batch_size,
epochs=epochs,
callbacks=[checkpointer],
validation_data=valid_generator,
validation_steps=valid_generator.samples//batch_size)
And finally, the best weights was loaded to the model. The model was evaluated using test_set:
# load the weights that yielded the best validation accuracy
model.load_weights('cnn_model.weights.best.hdf5')
#evaluate and print test accuracy
score = model.evaluate_generator(test_generator,
test_generator.samples//batch_size)
print('\n', 'Test accuracy:', score[1])
But, here is my problem: each time i run only the model.evaluate_generator without train the model again (i.e. keeping the same weights), it returns different accuracy scores.
I've been looking for a solution, reading a lot of posts to get some insights and recently i got some advance.
Recently, i discovered based on this post that if i set Shuffle=True and batch_size=1 in test_generator:
test_generator = datagen_test.flow_from_directory(
'content/cell_images/test', #Test folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=1,
class_mode='categorical',
shuffle=False)`
and steps = test_generator.samples in test_generator:
score = model.evaluate_generator(test_generator, test_generator.samples)
the values doesn't change any more.
I was investigating the effect of rescale 1./255 based on this post. For this, i used callbacks with a checkpointer to save the weights only for the best validation socore. After, i loaded the best weights to the model and evaluated using model.evaluate_generator, as mentioned above. To check the score consistency i also use the validation score to check if the values returned by the callbacks for the best weights is the same returned by the evaluate_generator. Before run the evaluate_generator with validation_set i used the same params of the test set:
valid_generator = datagen_valid.flow_from_directory(
'content/cell_images/valid',
target_size=(150,150),
batch_size=1,
class_mode='categorical',
shuffle=False)
#evaluate and print test accuracy
score = model.evaluate_generator(valid_generator,
valid_generator.samples)
print('\n', 'Valid accuracy:', score[1])
#evaluate and print test accuracy
score = model.evaluate_generator(test_generator,
test_generator.samples)
print('\n', 'Test accuracy:', score[1])
Curiously i noticed that:
When i don't use the rescale (1./255):
datagen_train = ImageDataGenerator()
datagen_test = ImageDataGenerator()
datagen_valid = ImageDataGenerator()
the validation_score displayed by the callbacks (0.5) it's exacly the same obtained from model.evaluate_generator (0.5); Also, the test set returns an accuracy score = 0.5.
When i use the rescale (1./255):
datagen_train = ImageDataGenerator(rescale=1./255)
datagen_test = ImageDataGenerator(rescale=1./255)
datagen_valid = ImageDataGenerator(rescale=1./255)
the difference between validation_score displayed by the callbacks (0.9515):
Epoch 7/10
688/688 [==============================] - 67s 97ms/step - loss:
0.2017 - acc: 0.9496 - val_loss: 0.1767 - val_acc: 0.9515
Epoch 00007: val_loss improved from 0.19304 to 0.17671, saving model
to cnn_model.weights.best.hdf5
and the score obtained from model.evaluate_generator (Valid accuracy: 0.9466618287373004) it's very tiny; Using test set -
Test accuracy: 0.9078374455732946
Based on this small difference between validation scores, could i infer that the evaluate_generator is working corrreclty? And, could i infer that the accuracy score on the test_set is also correctly ? Or there is another approach to solve this problem?
I'm frustating with this problem.
Sorry for the long post, i'm trying to be the more didactic i can.
Thanks!
I'm training a time series forecasting LSTM-model in Keras.
When I save the trained model and reload it, to train it on different data (so the weights get better and can predict several types of data), is it alright to:
model.fit(x_train, y_train, epochs=60, batch_size=1, verbose=2)
# make predictions
y_pred_train = np.squeeze(model.predict(x_train), axis=(1,))
y_pred_val = np.squeeze(model.predict(x_val), axis=(1,))
# save whole model
model.save('lstm_model.h5')
# load and fit model
loaded_model = load_model('lstm_model.h5')
loaded_model.fit(x_train, y_train, epochs=60, batch_size=1, verbose=2)
I wonder if the training starts all over again and the trained status is gone after fitting in a second time.
Thank you for helping! :)
I'm using Keras to predict a time series. As standard I'm using 20 epochs.
I want to check if my model is learning well, by predicting for each one of the 20 epochs.
By using model.predict() I'm getting only one prediction among all epochs (not sure how Keras selects it). I want all predictions, or at least the 10 best.
Would anyone know how to help me?
I think there is a bit of a confusion here.
An epoch is only used while training the neural network, so when training stops (in this case, after the 20th epoch), then the weights correspond to the ones computed on the last epoch.
Keras prints current loss values on the validation set during training after each epoch. If the weights after each epoch are not saved, then they are lost. You can save weights for each epoch with the ModelCheckpoint callback, and then load them back with load_weights on your model.
You can compute your predictions after each training epoch by implementing an appropriate callback by subclassing Callback and calling predict on the model inside the on_epoch_end function.
Then to use it, you instantiate your callback, make a list and use it as keyword argument callbacks to model.fit.
The following code will do the desired job:
import tensorflow as tf
import keras
# define your custom callback for prediction
class PredictionCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs={}):
y_pred = self.model.predict(self.validation_data[0])
print('prediction: {} at epoch: {}'.format(y_pred, epoch))
# ...
# register the callback before training starts
model.fit(X_train, y_train, batch_size=32, epochs=25,
validation_data=(X_valid, y_valid),
callbacks=[PredictionCallback()])
In case you want to make predictions on the test data, after every epoch while the training is going-on you can try this
class CustomCallback(keras.callbacks.Callback):
def __init__(self, model, x_test, y_test):
self.model = model
self.x_test = x_test
self.y_test = y_test
def on_epoch_end(self, epoch, logs={}):
y_pred = self.model.predict(self.x_test, self.y_test)
print('y predicted: ', y_pred)
You need mention the callback during model.fit
model.sequence()
# your model architecture
model.fit(x_train, y_train, epochs=10,
callbacks=[CustomCallback(model, x_test, y_test)])
Similar to on_epoch_end there are many other methods provided by keras
on_train_begin, on_train_end, on_epoch_begin, on_epoch_end, on_test_begin,
on_test_end, on_predict_begin, on_predict_end, on_train_batch_begin, on_train_batch_end,
on_test_batch_begin, on_test_batch_end, on_predict_batch_begin,on_predict_batch_end