This is rather a popular error, but I couldn't find a proper answer given my setup.
I found this tutorial code, but when running, I get this error:
val_acc = history.history['val_acc']
KeyError: 'val_acc'
The fit_generator() function unlike fit(), doesn't allow a validation split. So how to fix it?
Here is the code:
def plot_training(history):
print (history.history.keys())
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(len(acc))
plt.plot(epochs, acc, 'r.')
plt.plot(epochs, val_acc, 'r')
plt.title('Training and validation accuracy')
# plt.figure()
# plt.plot(epochs, loss, 'r.')
# plt.plot(epochs, val_loss, 'r-')
# plt.title('Training and validation loss')
plt.show()
plt.savefig('acc_vs_epochs.png')
#....
finetune_model = build_finetune_model(base_model, dropout=dropout, fc_layers=FC_LAYERS, num_classes=len(class_list))
adam = Adam(lr=0.00001)
finetune_model.compile(adam, loss='categorical_crossentropy', metrics=['accuracy'])
filepath="./checkpoints/" + "ResNet50" + "_model_weights.h5"
checkpoint = ModelCheckpoint(filepath, monitor=["acc"], verbose=1, mode='max')
callbacks_list = [checkpoint]
history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8,
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
plot_training(history)
Hi writting my suggestions here because I'm not able to comment yet,
You are right the fuction fit_generator() dosen't have the validation split attribute.
Therefore you need to make your own validation dataset and feed it to the fit generator through validation_data=(val_X, val_y) as eg.:
history = finetune_model.fit_generator(train_generator, epochs=NUM_EPOCHS, workers=8, validation_data=(val_X, val_y),
steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
Hope this helps.
EDIT
To get a validation dataset from your data you can use the methode train_test_split() from sklearn. For example a split with 77% train and 33% validation data:
X_train, val_X, y_train, val_y= train_test_split(
X, y, test_size=0.33, random_state=42)
Look here for more information.
Alternatively you could write your own split methode :)
Edit 2
If you don't have the possibility to use train_test split and with the assumption you have a pandas dataframe called train_data with the features and labels together:
val_data=train_data.sample(frac=0.33,random_state=1)
This should creates a validation dataset with 33% of the data and a train dataset with 77% of the data.
Edit3
It turns out you are using ImageDataGenerator() to create your data. This is quite handy because you can set your validation percentage via validation_split= while you initialize the ImageDataGenerator() as seen in the documentation (here). This should look something like this:
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
validation_split=0.33)
After this you need two "generated" datasets. One to train and one to make your validation. This should look as following:
train_generator = train_datagen.flow_from_directory(TRAIN_DIR,
target_size=(HEIGHT, WIDTH),
batch_size=BATCH_SIZE,subset="training")
validation_generator = train_datagen.flow_from_directory(TRAIN_DIR,
target_size=(HEIGHT, WIDTH),
batch_size=BATCH_SIZE,subset="validation")
Finally you can use both sets in your fit_generator as following:
history = finetune_model.fit_generator(train_generator,epochs=NUM_EPOCHS, workers=8,
validation_data=validation_generator, validation_steps = validation_generator.samples,steps_per_epoch=steps_per_epoch,
shuffle=True, callbacks=callbacks_list)
Let me know if this solves your problem :)
Related
i am trying to train model with kfold cross validation , now i want to keep history for plotting and saving the history. how can i do that?
it seems that some questions post answers of this question but i want to save and plot all history once, not with parted files
num_folds = 10
kfold = KFold(n_splits=num_folds, shuffle=True)
# K-fold Cross Validation model evaluation
fold_no = 1
a = []
for train, test in kfold.split(X, label):
print("---"*20)
history = siamese.fit(
[tf.gather(X[:,0], train),tf.gather(X[:,1], train)],
tf.gather(label, train),
validation_data=([tf.gather(X[:,0], test),tf.gather(X[:,1], test)], tf.gather(label, test)),
batch_size=batch_size,
epochs=epochs,
)
a.append(history)
by adding a dictionary to my code i can use all histories at once
num_folds = 10
kfold = KFold(n_splits=num_folds, shuffle=True)
# K-fold Cross Validation model evaluation
fold_no = 1
histories = {'accuracy':[], 'loss':[], 'val_accuracy':[], 'val_loss':[]}
for train, test in kfold.split(X, label):
print("---"*20)
history = siamese.fit(
[tf.gather(X[:,0], train),tf.gather(X[:,1], train)],
tf.gather(label, train),
validation_data=([tf.gather(X[:,0], test),tf.gather(X[:,1], test)], tf.gather(label, test)),
batch_size=batch_size,
epochs=epochs,
)
histories['accuracy'].append(history.history['accuracy'])
histories['loss'].append(history.history['loss'])
histories['val_accuracy'].append(history.history['val_accuracy'])
histories['val_loss'].append(history.history['val_loss'])
with open('./trainHistoryDict', 'wb') as file_pi:
pickle.dump(histories, file_pi)
I would like to know if it's possible to know the loss/accuracy with model.predict() in Keras?
I tried to figured out by myself, but I failed. Thank you very much for any inputs.
Keras model.predict() only gets the input data (X) and produces the output from the trained model. It does not know anything about the actual expected value (y).
You can do what you are asking for using model.evaluate(), that actually requires the X and y values in your data set and will produce the loss value and metrics values for the model in test mode.
Use model.predict() in production, to just get the model output. During your testing and validation phase, you may want to also use model.evaluate().
So my code is like this :
# Fetch dataset
train_data, test_data = tfds.load(name="imdb_reviews", split=["train", "test"],
batch_size=-1, as_supervised=True)
train_examples, train_labels = tfds.as_numpy(train_data)
test_examples, test_labels = tfds.as_numpy(test_data)
np.save("train_examples", train_examples)
np.save("train_labels", train_labels)
np.save("test_examples", test_examples)
np.save("test_labels", test_labels)
# BUILD MODEL
model = "https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1"
hub_layer = hub.KerasLayer(model, output_shape=[], input_shape=[],
dtype=tf.string, trainable=True, name='gnews_embedding')
model = build_model(hub_layer)
model.summary()
# SAVE AS CHECKPOINT (THE BEST ONLY)
es = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=4, verbose=1, mode='min')
checkpoint = tf.keras.callbacks.ModelCheckpoint('model.h5', monitor='val_loss', save_best_only=True)
# TRAIN MODEL
history = model.fit(
train_examples,
train_labels,
epochs=20,
batch_size=BATCH_SIZE,
validation_split = .2,
shuffle = True,
callbacks = [checkpoint, es],
verbose=1)
# CHECK ACCURACY AND LOSS VALUE
model.load_weights('/app/model.h5')
results = model.evaluate(test_examples, test_labels)
text = "The gold rush apple from natora- this is the most expensive apple."
dataset = tf.data.Dataset.from_tensor_slices([text])
np.save("test_txt", test_txt)
resultsTest = model.evaluate(dataset, test_labels)
print("RESULT ACCURACY = ", resultsTest)
so I try to know the accuracy/loss of this prediction. Can i evaluate a simple sentence like that or just re-train the model to get the loss/accuracy ?
I am trying to build a model using the functional api of Keras.
Here is the entire model that I have made. I am not sure if it is correct, and I would be very happy if someone could take a look at it for a moment.
I have first splittet the data into train and test data set.
from sklearn.model_selection import train_test_split
X1_train, X1_test, X2_train, X2_test, y_train, y_test = train_test_split(X1_scaled, X2_scaled, end_y, test_size=0.2)
[i.shape for i in (X1_train, X1_test, X2_train, X2_test, y_train, y_test)]
Here is the part, where I start to build the model
from tensorflow.keras import layers, Model, utils
# Build the model
input1 = layers.Input((10, 6))
input2 = layers.Input((10, 2, 5))
x1 = layers.Flatten()(input1)
x2 = layers.Flatten()(input2)
concat = layers.concatenate([x1, x2])
# Add hidden and dropout layers
hidden1 = layers.Dense(64, activation='relu')(concat)
hid1_out = layers.Dropout(0.5)(hidden1)
hidden2 = layers.Dense(32, activation='relu')(hid1_out)
hid2_out = layers.Dropout(0.5)(hidden2)
output = layers.Dense(1, activation='sigmoid')(hid2_out)
model = Model(inputs=[input1, input2], outputs=output)
# summarize layers
print(model.summary())
# compile the model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
# fit the keras model on the dataset
history = model.fit([X1_train, X2_train], y_train, epochs=200, batch_size=5, verbose=0, validation_data=([X1_test, X2_test], y_test))
# evaluate the keras model
_, train_accuracy = model.evaluate([X1_train, X2_train], y_train, verbose=0)
_, test_accuracy = model.evaluate([X1_test, X2_test], y_test, verbose=0)
print('Accuracy NN: %.2f' % (train_accuracy*100))
print('Accuracy NN: %.2f' % (test_accuracy*100))
A problem occurs here. No plot is showing.
# Plots
from matplotlib import pyplot
pyplot.subplot(211)
pyplot.title('Loss')
pyplot.plot(history.history['loss'], label='train')
pyplot.plot(history.history['val_loss'], label='test')
pyplot.legend()
# plot accuracy
pyplot.subplot(212)
pyplot.title('Accuracy')
pyplot.plot(history.history['accuracy'], label='train')
pyplot.plot(history.history['val_accuracy'], label='test')
pyplot.legend()
pyplot.show(`
Could someone give me any hints on how to manage it ?
Thank you for giving me some of your time
below is the code for a function that will produce two plots side by side. The first plot
shows the training loss and validation loss versus epochs. The second plot shows training accuracy and validation accuracy versus epochs. It also places a dot in the first plot for the epoch with the lowest validation loss and a dot on the second plot for the epoch with the highest validation accuracy.
def tr_plot(history):
#Plot the training and validation data
tacc=history.history['accuracy']
tloss=history.history['loss']
vacc=history.history['val_accuracy']
vloss=history.history['val_loss']
Epoch_count=len(tacc)
Epochs=[]
for i in range (Epoch_count):
Epochs.append(i+1)
index_loss=np.argmin(vloss)# this is the epoch with the lowest validation loss
val_lowest=vloss[index_loss] # lowest validation loss value
index_acc=np.argmax(vacc) # this is the epoch with the highest training accuracy
acc_highest=vacc[index_acc] # this is the highest accuracy value
plt.style.use('fivethirtyeight')
sc_label='best epoch= '+ str(index_loss+1 )
vc_label='best epoch= '+ str(index_acc + 1)
fig,axes=plt.subplots(nrows=1, ncols=2, figsize=(20,8))
axes[0].plot(Epochs,tloss, 'r', label='Training loss')
axes[0].plot(Epochs,vloss,'g',label='Validation loss' )
axes[0].scatter(index_loss+1 ,val_lowest, s=150, c= 'blue', label=sc_label)
axes[0].set_title('Training and Validation Loss')
axes[0].set_xlabel('Epochs')
axes[0].set_ylabel('Loss')
axes[0].legend()
axes[1].plot (Epochs,tacc,'r',label= 'Training Accuracy')
axes[1].plot (Epochs,vacc,'g',label= 'Validation Accuracy')
axes[1].scatter(index_acc+1 ,acc_highest, s=150, c= 'blue', label=vc_label)
axes[1].set_title('Training and Validation Accuracy')
axes[1].set_xlabel('Epochs')
axes[1].set_ylabel('Accuracy')
axes[1].legend()
plt.tight_layout
plt.show()
The resulting plot looks like this
I’m trying to solve a problem in my CNN model. I'm using a dataset that is structured as follow:
My data is organized as follow:
train
parasitized
uninfected
test
parasitized
uninfected
validation
parasitized
uninfected
My dataset is too large and i'm using ImageDataGenerator to preprocess the images and also upload them in batches (reduce computational cost). In a first i configured the ImageDataGenerator as follows:
from keras.preprocessing.image import ImageDataGenerator
#Define a ImageDataGenerator for each dataset.
#This augmentation process is only to rescale each imagem to 1/255
datagen_train = ImageDataGenerator(rescale=1./255) #rescale=1./255
datagen_test = ImageDataGenerator(rescale=1./255)
datagen_valid = ImageDataGenerator(rescale=1./255)
#Define a batch_size parameter
batch_size=32
# Here .flow_from_directory is used to transform
train_generator = datagen_train.flow_from_directory(
'content/cell_images/train', #Train folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=batch_size,
class_mode='categorical') # We use categorical_crossentropy loss,
# we need categorical labels
test_generator = datagen_test.flow_from_directory(
'content/cell_images/test', #Test folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=batch_size,
class_mode='categorical')
valid_generator = datagen_valid.flow_from_directory(
'content/cell_images/valid',
target_size=(150,150),
batch_size=32,
class_mode='categorical')
To fit the model it was used fit_generator and a checkpointer to save the best weights based on the validation_accuracy:
from keras.callbacks import ModelCheckpoint
# Define epochs number
epochs = 10
# Create a checkpointer to save only the best params
checkpointer = ModelCheckpoint(filepath='cnn_model.weights.best.hdf5',
verbose=1, save_best_only=True)
model.fit_generator(train_generator,
steps_per_epoch=train_generator.samples//batch_size,
epochs=epochs,
callbacks=[checkpointer],
validation_data=valid_generator,
validation_steps=valid_generator.samples//batch_size)
And finally, the best weights was loaded to the model. The model was evaluated using test_set:
# load the weights that yielded the best validation accuracy
model.load_weights('cnn_model.weights.best.hdf5')
#evaluate and print test accuracy
score = model.evaluate_generator(test_generator,
test_generator.samples//batch_size)
print('\n', 'Test accuracy:', score[1])
But, here is my problem: each time i run only the model.evaluate_generator without train the model again (i.e. keeping the same weights), it returns different accuracy scores.
I've been looking for a solution, reading a lot of posts to get some insights and recently i got some advance.
Recently, i discovered based on this post that if i set Shuffle=True and batch_size=1 in test_generator:
test_generator = datagen_test.flow_from_directory(
'content/cell_images/test', #Test folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=1,
class_mode='categorical',
shuffle=False)`
and steps = test_generator.samples in test_generator:
score = model.evaluate_generator(test_generator, test_generator.samples)
the values doesn't change any more.
I was investigating the effect of rescale 1./255 based on this post. For this, i used callbacks with a checkpointer to save the weights only for the best validation socore. After, i loaded the best weights to the model and evaluated using model.evaluate_generator, as mentioned above. To check the score consistency i also use the validation score to check if the values returned by the callbacks for the best weights is the same returned by the evaluate_generator. Before run the evaluate_generator with validation_set i used the same params of the test set:
valid_generator = datagen_valid.flow_from_directory(
'content/cell_images/valid',
target_size=(150,150),
batch_size=1,
class_mode='categorical',
shuffle=False)
#evaluate and print test accuracy
score = model.evaluate_generator(valid_generator,
valid_generator.samples)
print('\n', 'Valid accuracy:', score[1])
#evaluate and print test accuracy
score = model.evaluate_generator(test_generator,
test_generator.samples)
print('\n', 'Test accuracy:', score[1])
Curiously i noticed that:
When i don't use the rescale (1./255):
datagen_train = ImageDataGenerator()
datagen_test = ImageDataGenerator()
datagen_valid = ImageDataGenerator()
the validation_score displayed by the callbacks (0.5) it's exacly the same obtained from model.evaluate_generator (0.5); Also, the test set returns an accuracy score = 0.5.
When i use the rescale (1./255):
datagen_train = ImageDataGenerator(rescale=1./255)
datagen_test = ImageDataGenerator(rescale=1./255)
datagen_valid = ImageDataGenerator(rescale=1./255)
the difference between validation_score displayed by the callbacks (0.9515):
Epoch 7/10
688/688 [==============================] - 67s 97ms/step - loss:
0.2017 - acc: 0.9496 - val_loss: 0.1767 - val_acc: 0.9515
Epoch 00007: val_loss improved from 0.19304 to 0.17671, saving model
to cnn_model.weights.best.hdf5
and the score obtained from model.evaluate_generator (Valid accuracy: 0.9466618287373004) it's very tiny; Using test set -
Test accuracy: 0.9078374455732946
Based on this small difference between validation scores, could i infer that the evaluate_generator is working corrreclty? And, could i infer that the accuracy score on the test_set is also correctly ? Or there is another approach to solve this problem?
I'm frustating with this problem.
Sorry for the long post, i'm trying to be the more didactic i can.
Thanks!
I am trying to plot the training data. I am using
for i in range(epoch_size):
history = model.fit(trainingX[:10], trainingY[:10], epochs=1, batch_size=batch_size,callbacks = [early_stop], verbose=2)
in loop of 50
and then
plt.plot(history.history['acc'])
plt.title('model accuracy')
plt.ylabel('acc')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
yet the plot is always empty, why is that?
img
You are overwriting the history with each iteration of the for loop and for only one epoch there is no change that is possible to plot. So you either need more than one epoch or need to store the history values with a seperate array.
Try to print the following print(history.history) and check the contents.
And we generally run its once.
history = model.fit(X_train, Y_train,
batch_size=128, epochs=10,
verbose=2, validation_data=(X_test, Y_test))