I'm using Keras to predict a time series. As standard I'm using 20 epochs.
I want to check if my model is learning well, by predicting for each one of the 20 epochs.
By using model.predict() I'm getting only one prediction among all epochs (not sure how Keras selects it). I want all predictions, or at least the 10 best.
Would anyone know how to help me?
I think there is a bit of a confusion here.
An epoch is only used while training the neural network, so when training stops (in this case, after the 20th epoch), then the weights correspond to the ones computed on the last epoch.
Keras prints current loss values on the validation set during training after each epoch. If the weights after each epoch are not saved, then they are lost. You can save weights for each epoch with the ModelCheckpoint callback, and then load them back with load_weights on your model.
You can compute your predictions after each training epoch by implementing an appropriate callback by subclassing Callback and calling predict on the model inside the on_epoch_end function.
Then to use it, you instantiate your callback, make a list and use it as keyword argument callbacks to model.fit.
The following code will do the desired job:
import tensorflow as tf
import keras
# define your custom callback for prediction
class PredictionCallback(tf.keras.callbacks.Callback):
def on_epoch_end(self, epoch, logs={}):
y_pred = self.model.predict(self.validation_data[0])
print('prediction: {} at epoch: {}'.format(y_pred, epoch))
# ...
# register the callback before training starts
model.fit(X_train, y_train, batch_size=32, epochs=25,
validation_data=(X_valid, y_valid),
callbacks=[PredictionCallback()])
In case you want to make predictions on the test data, after every epoch while the training is going-on you can try this
class CustomCallback(keras.callbacks.Callback):
def __init__(self, model, x_test, y_test):
self.model = model
self.x_test = x_test
self.y_test = y_test
def on_epoch_end(self, epoch, logs={}):
y_pred = self.model.predict(self.x_test, self.y_test)
print('y predicted: ', y_pred)
You need mention the callback during model.fit
model.sequence()
# your model architecture
model.fit(x_train, y_train, epochs=10,
callbacks=[CustomCallback(model, x_test, y_test)])
Similar to on_epoch_end there are many other methods provided by keras
on_train_begin, on_train_end, on_epoch_begin, on_epoch_end, on_test_begin,
on_test_end, on_predict_begin, on_predict_end, on_train_batch_begin, on_train_batch_end,
on_test_batch_begin, on_test_batch_end, on_predict_batch_begin,on_predict_batch_end
Related
I am building a MLP using TensorFlow 2.0. I am plotting the learning curve and also using keras.evaluate on both training and test data to see how well it performed. The code I'm using:
history = model.fit(X_train, y_train, batch_size=32,
epochs=200, validation_split=0.2, verbose=0)
# evaluate the model
eval_result_tr = model.evaluate(X_train, y_train)
eval_result_te = model.evaluate(X_test, y_test)
print("[training loss, training accuracy]:", eval_result_tr)
print("[test loss, test accuracy]:", eval_result_te)
#[training loss, training accuracy]: [0.5734676122665405, 0.9770742654800415]
#[test loss, test accuracy]: [0.7273344397544861, 0.9563318490982056]
#plot the learning rate curve
import matplotlib.pyplot as plt
plt.plot(history.history["loss"], label='eğitim')
plt.plot(history.history['val_loss'], label='doğrulama')
plt.xlabel("Öğrenme ivmesi")
plt.ylabel("Hata payı")
plt.title("Temel modelin öğrenme eğrisi")
plt.legend()
The output is:
My question is: How keras.evaluate() calculates the training loss to be 0.5734676122665405? I take the average of history.history["loss"] bu it returns different (0.7975356701016426) value.
Or, am I mistaken to begin with by trying to evaluate the model performance on training data by eval_result_tr = model.evaluate(X_train, y_train)?
For community benefit, adding #Dr. Snoopy's answer here in the answer section.
This has been asked many times before, the loss you see is with
changing weights during training, evaluating outside of training will
use fixed weights, so you will always see a different loss value.
I trained a model on initial data, got some good scores, and now after receiving more data I want to load the pre-trained model and continue to train.
Here some snippet of what I did:
(1) I follow this post which says to save the model in 'tf' format
# saving initial model
model.save(path2initial, save_format='tf')
# load pre-trained model
clf = tf.keras.models.load_model(path2initial)
# create new data generators
train_gen = generators.create(generator_config, 'train')
val_gen = generators.create(generator_config, 'val')
# create metrics, loss, optimizer and callbacks
loss = losses.create(loss_config)
callback_list = callbacks.create(callback_config)
optimizer = optimizers.create(optimizer_confing)
metrics = metrics.create(metrics_config)
# compile model
clf.compile(optimizer=optimizer, loss=loss, metrics=metrics)
# train
clf.fit(x=train_gen,
epochs=NB_EPCOHS,
validation_data=val_gen,
steps_per_epochs=math.ceil(len(train_steps)/ BATCH_SIZE),
validation_steps=math.ceil(len(val_steps)/ BATCH_SIZE),
callbacks=callback_list,
use_multiprocessing=True,
workers=16,
max_queue_size=8,
verbose=1
)
I should note that two of my callbacks are
EarlyStopping(monitor='val_loss', restore_best_weights=True,
min_delta=0.001, patience=10, mode='min', verbose=1)
ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=True,
mode='min', save_freq='epoch')
And that train_gen is consisted with both initial_data and new_data.
This method trained only for 4 epochs, and hadn't changed for the
rest of the 10 'patient' epochs. Moreover the results were way worse than the initial model's results.
(2) The second method I tried was to save the model in the default format (that's the only change):
model.save(path2initial)
.
.
.
This model had trained for 71/200 epochs, but it seems that it ignored my EarlyStopping() callback. In some epochs the val-loss had changed by 1e-4 or even less, and still it continue with the training (weirddd), And it stopped (by EarlyStopping()) in epoch 71 even that the val-loss had change! Moreover, the results had barely changed.
For comparison I trained a model from scratch on all the data (both initial and new data) and got way better results:
Initial data Method (1) Method (2) New model on all data
mean F1 score: 0.735 0.422 0.74 0.803
Is there a proven way to how to continue training a keras model?
When loading the model does the optimizer status reset?
When loading the model, do I need to define all the callbacks, loss, opt, metrics all over again? Do I need to compile it again?
I'm trying to train multiple machine learning models using tensorflow keras, I was just wondering is there a way to obtain the best score achieved while training after training is complete. I found online that the .fit function returns a history object which can be accessed to get the best score, though from code i've tried it says "AttributeError: 'History' object has no attribute 'best_score'", I cannot find an attribute list online so this is why I am asking here.
Thanks in advance.
History = model.fit(Xtrain, ytrain, epochs=1, validation_data=(Xtest, ytest), verbose=1)
print("Best: %f using %s" % (History.best_score, History.best_params_))
PS, I know training for 1 epoch will achieve nothing, I'm just trying to test the code
I'm assuming you just want the best score from the history object.
hist = model.fit(...)
print(hist.history) # this will print a dictionary object, now you need to grab the metrics / score you're looking for
# if your score == 'acc', if not replace 'acc' with your metric
best_score = max(hist.history['acc'])
print(best_score)
If your metric is 'accuracy', use that instead.
If you want the best model, you can just use ModelCheckPoint https://keras.io/callbacks/
The fit() method on keras return a history object. The history.history attribute is a dictionary recording training loss values and metrics values at successive epochs, as well as validation loss values and validation metrics values (if applicable). You can call "history.history['loss']" or "history.history['val_loss']" to access it. Here is an example of it.
model.compile(loss = 'mean_squared_error', optimizer =optimizer,metrics=['accuracy'])
History = model.fit(Xtrain, ytrain, epochs=1, validation_data=(Xtest, ytest), verbose=1)
plt.plot(history.history['acc'])
plt.plot(history.history['val_acc'])
For the best params, I think using callbacks.ModelCheckpoint might be helpful. This method has an argument 'save_best_only' to the latest best model.
keras.callbacks.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)
Kindly refer to the Training history visualization and callback for more information.
I am new to keras and I want to train the model with F1-score as my metrics.
I came across two things, one is that I can add callbacks and other is using the in built metrics function
Here, it says that the metrics function will not be used for training the model. So, does that mean I can anything in metrics argument while compiling the model?
Specfically,
model.compile(optimizer='rmsprop',
loss='binary_crossentropy',
metrics=['accuracy'])
In the above case even though accuracy is passed as metrics, it will not be used for training the model.
Second thing is to use callbacks as defined here,
import numpy as np
from keras.callbacks import Callback
from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score
class Metrics(Callback):
def on_train_begin(self, logs={}):
self.val_f1s = []
self.val_recalls = []
self.val_precisions = []
def on_epoch_end(self, epoch, logs={}):
val_predict = (np.asarray(self.model.predict(self.model.validation_data[0]))).round()
val_targ = self.model.validation_data[1]
_val_f1 = f1_score(val_targ, val_predict)
_val_recall = recall_score(val_targ, val_predict)
_val_precision = precision_score(val_targ, val_predict)
self.val_f1s.append(_val_f1)
self.val_recalls.append(_val_recall)
self.val_precisions.append(_val_precision)
print “ — val_f1: %f — val_precision: %f — val_recall %f” %(_val_f1, _val_precision, _val_recall)
return
metrics = Metrics()
Then fit the model,
model.fit(training_data, training_target,
validation_data=(validation_data, validation_target),
nb_epoch=10,
batch_size=64,
callbacks=[metrics])
I am not sure if this will train the model on f1 score.
You can't train a neural network with f1-scores. For back propagating the error during training you need some sort of function which tells you, how far away your prediction is from the expected value. Such a function is as example the MSE loss.
F1 score on the other hand is just the harmonic mean between precision and recall from your samples. It does not tell you, in which direction you have to update the weights in order to get a better model. It also does not tell you, how far away you prediction is from the expected value.
What you could do is to print the F1 score after every epoch. An example on how to do this can be found in this blogpost
I'm training a time series forecasting LSTM-model in Keras.
When I save the trained model and reload it, to train it on different data (so the weights get better and can predict several types of data), is it alright to:
model.fit(x_train, y_train, epochs=60, batch_size=1, verbose=2)
# make predictions
y_pred_train = np.squeeze(model.predict(x_train), axis=(1,))
y_pred_val = np.squeeze(model.predict(x_val), axis=(1,))
# save whole model
model.save('lstm_model.h5')
# load and fit model
loaded_model = load_model('lstm_model.h5')
loaded_model.fit(x_train, y_train, epochs=60, batch_size=1, verbose=2)
I wonder if the training starts all over again and the trained status is gone after fitting in a second time.
Thank you for helping! :)