I set the following:
np.random.seed(7)
# split data to train, validate, test (60%, 20%, 20%)
train, validate, test = np.split(data, [int(.6*len(data)), int(.8*len(data))])
history = model.fit(train, train, epochs=1, batch_size=32, verbose=1, shuffle=True,
validation_data=(validate, validate), callbacks=[cb])
score = model.evaluate(test, test, verbose=1)
shuffle=True shouldn't matter here since I'm only training for one epoch.
Now from what I've read this should ensure that my model's accuracy is always the same after training from scratch, but the accuracy results for various runs are 48%, 48%, 56%, 48%, 56%, 47.5% and so on. So I'm wondering if there is something else I have to do to ensure that the resulting accuracy stays the same?
The parameters of the model are initialized differently everytime you fit the model, even if it is for the same data. So it will result in different accuracies. If you are insistent on getting the same accuracy, run the model once, save the parameters in a file and then load them again when you are running the code. Refer Keras Documentation for more details.
Related
I came across a notebook that had its fit function set-up like this:
history = autoencoder.fit(blurry_frames,
clean_frames,
validation_data=(blurry_frames, clean_frames),
epochs=100,
batch_size=batch_size,
callbacks=callbacks)
They're using the same set for both training and validation. This looks very wrong to me but I wonder if its unique to Autoencoders. This person is getting 82% in val accuracy.
I'm a beginner in Machine Learning. At first, my model gave me an accuracy of 85.82%, which was good. But now I would like to test the model again total new data but I can't figure out what to add to the code as I can only get the test accuracy when the model is tested with validation data.
The following is my code:
Create a new data set in EXACTLY the same manner by which you created the original test set. The term exactly is important, If you pre=processed your training and validation data a you must do the same pre-processing on this new data.
One other approach is to split the training data into 3 groups, train, validation and test. You can do that with train test split as follows.
X_train, X_temp, Y_train, Y_temp = train_test_split(X, Y,train_size=.8 random_state = 1)
X_valid, X_test, Y_valid, Y_test = train_test_split(X_temp, Y_temp, train_size=.5, shuffle=False)
This will take your input set and use 80% of it for training, 10 % for validation and 10% for test. In your code what you called the test set is actually the validation set so when you did model.evaluate you got the same accuracy as you did for validation accuracy. So now in model.fit make validation_data=(X_valid, Y_valid). Now your test set is independent of the validation set so when you run model.evaluate you should get a somewhat different accuracy than that of the validation accuracy.
'''model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, verbose=2)'''
in the above line of code model is a sequential keras model having layers and is compiled.
what is the use of the parameter validaiton_data. the model is going to train on X_train and y_train data. so based on y_train the parameters would be adjusted and back propagation is done.
what is the use of validation_data and why is a different data in this case testing data provided.
During training, (x_train, y_train) data is used to adjust trainable parameters of the model. However, we don't know whether model is overfit or under fit, whether model is going to do well when new data is provided. So, that is the reason why we have validation data (x_test, y_test) to test the accuracy of model on any unseen data.
Depending on training and validation accuracy, we can decide.
- Whether the model is overfit/underfit,
- whether to collect more data,
- do we need to implement regularization technique,
- do we need to use data augmentation techniques,.
- whether we need to do tune hyper parameters etc.
I have a few questions about interpreting the performance of certain optimizers on MNIST using a Lenet5 network and what does the validation loss/accuracy vs training loss/accuracy graphs tell us exactly.
So everything is done in Keras using a standard LeNet5 network and it is ran for 15 epochs with a batch size of 128.
There are two graphs, train acc vs val acc and train loss vs val loss. I made 4 graphs because I ran it twice, once with validation_split = 0.1 and once with validation_data = (x_test, y_test) in model.fit parameters. Specifically the difference is shown here:
train = model.fit(x_train, y_train, epochs=15, batch_size=128, validation_data=(x_test,y_test), verbose=1)
train = model.fit(x_train, y_train, epochs=15, batch_size=128, validation_split=0.1, verbose=1)
These are the graphs I produced:
using validation_data=(x_test, y_test):
using validation_split=0.1:
So my two questions are:
1.) How do I interpret both the train acc vs val acc and train loss vs val acc graphs? Like what does it tell me exactly and why do different optimizers have different performances (i.e the graphs are different as well).
2.) Why do the graphs change when I use validation_split instead? Which one would be a better choice to use?
I will attempt to provide an answer
You can see that towards the end training accuracy is slightly higher than validation accuracy and training loss is slightly lower than validation loss. This hints at overfitting and if you train for more epochs the gap should widen.
Even if you use the same model with same optimizer you will notice slight difference between runs because weights are initialized randomly and randomness associated with GPU implementation. You can look here for how to address this issue.
Different optimizers will usually produce different graph because they update model parameters differently. For example, vanilla SGD will do update at constant rate for all parameters and at all training steps. But if you add momentum the rate will depend on previous updates and usually will result in faster convergence. Which means you can achieve same accuracy as vanilla SGD in lower number of iteration.
Graphs will change because training data will be changed if you split randomly. But for MNIST you should use standard test split provided with the dataset.
I didn't find any answers to the following question:
Is there a way to print the trained model accuracy, total model loss and model evaluation accuracy after loading the saved trained Keras model?
from keras.models import load_model
m = load_model.load("lstm_model_01.hd5")
I checked all the callable methods of m but didn't find what I was looking for.
Model is really a graph with weights and that's all that gets saved. You have to evaluate the restored model on data to get predictions and from that you'll obtain an accuracy.
Pleas save your model fit results as follows:
history = model.fit(input_train, y_train,
epochs=10,
batch_size=128,
validation_split=0.2)
And use pickle to dump it (i.e. history) out. There you will see the training loss or validation accuracy from the model. You can load it back anytime.