I trained my first categorical image classifier, but due I´m new to deep learning, I don't notice what is happening with my validation set. I´m doing a 5 Fold CV, and in each fold, I split the data set into train/validation/test.
After the split, I apply data augmentation in x_train:
datagen = ImageDataGenerator(
rotation_range=90,
horizontal_flip=True)
Then fit my model with the generator:
history = model.fit_generator(datagen.flow(x_train, y_train, batch_size=25),
validation_data=(x_validation,y_validation),
steps_per_epoch=math.ceil(len(x_train) / batch),
epochs=epochs)
Params: len(x_train)=31 and batch=1 cause I want sgd batch.
The data set has 76 images spplited into test=15, Train=31 and Validation=30.
After running this for 5 fold it gives me this graphic:
What is happening?
Thanks in advance!
Related
I am building a MLP using TensorFlow 2.0. I am plotting the learning curve and also using keras.evaluate on both training and test data to see how well it performed. The code I'm using:
history = model.fit(X_train, y_train, batch_size=32,
epochs=200, validation_split=0.2, verbose=0)
# evaluate the model
eval_result_tr = model.evaluate(X_train, y_train)
eval_result_te = model.evaluate(X_test, y_test)
print("[training loss, training accuracy]:", eval_result_tr)
print("[test loss, test accuracy]:", eval_result_te)
#[training loss, training accuracy]: [0.5734676122665405, 0.9770742654800415]
#[test loss, test accuracy]: [0.7273344397544861, 0.9563318490982056]
#plot the learning rate curve
import matplotlib.pyplot as plt
plt.plot(history.history["loss"], label='eğitim')
plt.plot(history.history['val_loss'], label='doğrulama')
plt.xlabel("Öğrenme ivmesi")
plt.ylabel("Hata payı")
plt.title("Temel modelin öğrenme eğrisi")
plt.legend()
The output is:
My question is: How keras.evaluate() calculates the training loss to be 0.5734676122665405? I take the average of history.history["loss"] bu it returns different (0.7975356701016426) value.
Or, am I mistaken to begin with by trying to evaluate the model performance on training data by eval_result_tr = model.evaluate(X_train, y_train)?
For community benefit, adding #Dr. Snoopy's answer here in the answer section.
This has been asked many times before, the loss you see is with
changing weights during training, evaluating outside of training will
use fixed weights, so you will always see a different loss value.
I am using tensorflow and keras for a binary classification problem.
I have only a training set of 81 samples (Testsize 21), but ~1900 features. I know its too less samples and too many features, but its a biological problem (gene-expression data), so i have to deal with it.
My model looks like this: (using different neurons per layer, different number of hidden layers, regularization and dropout to deal with the high dimensional data)
model = Sequential()
model.add(Input((input_shape,)))
for i in range(num_hidden):
model.add(Dense(n_neurons, activation="relu",kernel_regularizer=keras.regularizers.l1_l2(l1_reg, l2_reg)))
model.add(Dropout(dropout_rate))
model.add(Dense(1, activation="sigmoid"))
ann_optimizer= keras.optimizers.Adam()
model.compile(loss="binary_crossentropy",
optimizer=ann_optimizer, metrics=['accuracy'])
I am using a 10 fold nested cross validation and grid search in the inner fold like this:
# fit and evaluate the model
# configure the inner cross-validation procedure (5 fold, 80 inner training dataset, 20 inner test dataset)
cv_inner = ShuffleSplit(n_splits=5, test_size=0.2, random_state=1)
# define the mode
ann = KerasRegressor(build_fn=regressionModel_sequential, input_shape=X_train.shape[1],
batch_size=batch_size)
# use pipeline as prevent from Leaky Preprocessing (StandardScaler on 80% inner-training dataset))
pipe = Pipeline(steps=[('scaler', StandardScaler()), ('ann', ann)])
# define the grid search of with inner cv to get good parameters
grid_search_result = GridSearchCV(
pipe, param_grid, n_jobs=-1, cv=cv_inner, refit=True, verbose=0)
#refit = True a final model with the entire inner-training dataset
# execute search
grid_search_result.fit(X_train, y_train, ann__verbose=0)
logger.info('>>>>> est=%.3f, params=%s' % (grid_search_result.best_score_, grid_search_result.best_params_))
# to get loss curve
ann_val = regressionModel_sequential(input_shape=X_train.shape[1],
n_neurons=grid_search_result.best_params_['ann__n_neurons'],
l1_reg=grid_search_result.best_params_['ann__l1_reg'],
l2_reg=grid_search_result.best_params_['ann__l2_reg'],
num_hidden=grid_search_result.best_params_['ann__num_hidden'],
dropout_rate=grid_search_result.best_params_['ann__dropout_rate'])
# Validation with outer 20 %
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
history = ann_val.fit(X_train, y_train, batch_size=batch_size, verbose=0,
validation_split=0.25, shuffle=True, epochs=grid_search_result.best_params_['ann__epochs'])
plot_history(history, directory, i)
# use best grid search reult for predicting on outer test dataset
y_predicted = ann_val.predict(X_test)
# print predicted
logger.info(y_predicted[:5])
logger.info(y_test[:5])
rmse = (np.sqrt(metrics.mean_squared_error(y_test, y_predicted)))
mae = (metrics.mean_squared_error(y_test, y_predicted))
r_squared = metrics.r2_score(y_test, y_predicted)
My loss seems good: loss
But accuracy is very bad. accuracy (example from one outer fold)
Does anyone have suggestions on what i could do to improve my results?
I also know that the biological question behind is very hard/maybe not possible to solve.
I’m trying to solve a problem in my CNN model. I'm using a dataset that is structured as follow:
My data is organized as follow:
train
parasitized
uninfected
test
parasitized
uninfected
validation
parasitized
uninfected
My dataset is too large and i'm using ImageDataGenerator to preprocess the images and also upload them in batches (reduce computational cost). In a first i configured the ImageDataGenerator as follows:
from keras.preprocessing.image import ImageDataGenerator
#Define a ImageDataGenerator for each dataset.
#This augmentation process is only to rescale each imagem to 1/255
datagen_train = ImageDataGenerator(rescale=1./255) #rescale=1./255
datagen_test = ImageDataGenerator(rescale=1./255)
datagen_valid = ImageDataGenerator(rescale=1./255)
#Define a batch_size parameter
batch_size=32
# Here .flow_from_directory is used to transform
train_generator = datagen_train.flow_from_directory(
'content/cell_images/train', #Train folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=batch_size,
class_mode='categorical') # We use categorical_crossentropy loss,
# we need categorical labels
test_generator = datagen_test.flow_from_directory(
'content/cell_images/test', #Test folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=batch_size,
class_mode='categorical')
valid_generator = datagen_valid.flow_from_directory(
'content/cell_images/valid',
target_size=(150,150),
batch_size=32,
class_mode='categorical')
To fit the model it was used fit_generator and a checkpointer to save the best weights based on the validation_accuracy:
from keras.callbacks import ModelCheckpoint
# Define epochs number
epochs = 10
# Create a checkpointer to save only the best params
checkpointer = ModelCheckpoint(filepath='cnn_model.weights.best.hdf5',
verbose=1, save_best_only=True)
model.fit_generator(train_generator,
steps_per_epoch=train_generator.samples//batch_size,
epochs=epochs,
callbacks=[checkpointer],
validation_data=valid_generator,
validation_steps=valid_generator.samples//batch_size)
And finally, the best weights was loaded to the model. The model was evaluated using test_set:
# load the weights that yielded the best validation accuracy
model.load_weights('cnn_model.weights.best.hdf5')
#evaluate and print test accuracy
score = model.evaluate_generator(test_generator,
test_generator.samples//batch_size)
print('\n', 'Test accuracy:', score[1])
But, here is my problem: each time i run only the model.evaluate_generator without train the model again (i.e. keeping the same weights), it returns different accuracy scores.
I've been looking for a solution, reading a lot of posts to get some insights and recently i got some advance.
Recently, i discovered based on this post that if i set Shuffle=True and batch_size=1 in test_generator:
test_generator = datagen_test.flow_from_directory(
'content/cell_images/test', #Test folder path
target_size=(150,150), #all images will be resized to 150x150
batch_size=1,
class_mode='categorical',
shuffle=False)`
and steps = test_generator.samples in test_generator:
score = model.evaluate_generator(test_generator, test_generator.samples)
the values doesn't change any more.
I was investigating the effect of rescale 1./255 based on this post. For this, i used callbacks with a checkpointer to save the weights only for the best validation socore. After, i loaded the best weights to the model and evaluated using model.evaluate_generator, as mentioned above. To check the score consistency i also use the validation score to check if the values returned by the callbacks for the best weights is the same returned by the evaluate_generator. Before run the evaluate_generator with validation_set i used the same params of the test set:
valid_generator = datagen_valid.flow_from_directory(
'content/cell_images/valid',
target_size=(150,150),
batch_size=1,
class_mode='categorical',
shuffle=False)
#evaluate and print test accuracy
score = model.evaluate_generator(valid_generator,
valid_generator.samples)
print('\n', 'Valid accuracy:', score[1])
#evaluate and print test accuracy
score = model.evaluate_generator(test_generator,
test_generator.samples)
print('\n', 'Test accuracy:', score[1])
Curiously i noticed that:
When i don't use the rescale (1./255):
datagen_train = ImageDataGenerator()
datagen_test = ImageDataGenerator()
datagen_valid = ImageDataGenerator()
the validation_score displayed by the callbacks (0.5) it's exacly the same obtained from model.evaluate_generator (0.5); Also, the test set returns an accuracy score = 0.5.
When i use the rescale (1./255):
datagen_train = ImageDataGenerator(rescale=1./255)
datagen_test = ImageDataGenerator(rescale=1./255)
datagen_valid = ImageDataGenerator(rescale=1./255)
the difference between validation_score displayed by the callbacks (0.9515):
Epoch 7/10
688/688 [==============================] - 67s 97ms/step - loss:
0.2017 - acc: 0.9496 - val_loss: 0.1767 - val_acc: 0.9515
Epoch 00007: val_loss improved from 0.19304 to 0.17671, saving model
to cnn_model.weights.best.hdf5
and the score obtained from model.evaluate_generator (Valid accuracy: 0.9466618287373004) it's very tiny; Using test set -
Test accuracy: 0.9078374455732946
Based on this small difference between validation scores, could i infer that the evaluate_generator is working corrreclty? And, could i infer that the accuracy score on the test_set is also correctly ? Or there is another approach to solve this problem?
I'm frustating with this problem.
Sorry for the long post, i'm trying to be the more didactic i can.
Thanks!
I'm training a time series forecasting LSTM-model in Keras.
When I save the trained model and reload it, to train it on different data (so the weights get better and can predict several types of data), is it alright to:
model.fit(x_train, y_train, epochs=60, batch_size=1, verbose=2)
# make predictions
y_pred_train = np.squeeze(model.predict(x_train), axis=(1,))
y_pred_val = np.squeeze(model.predict(x_val), axis=(1,))
# save whole model
model.save('lstm_model.h5')
# load and fit model
loaded_model = load_model('lstm_model.h5')
loaded_model.fit(x_train, y_train, epochs=60, batch_size=1, verbose=2)
I wonder if the training starts all over again and the trained status is gone after fitting in a second time.
Thank you for helping! :)
I am having a large training model with 59536 column and 200 lines that I want to train using tensorflow's cnn but I am facing the OutOfMemory error. How can I split my model (split it in smaller column) and train it in a small batches?
Here is my code
# Create the Estimator
mnist_classifier = tf.estimator.Estimator(
model_fn=cnn_model_fn, model_dir="path/to/model")
# Load the data
train_input_fn = tf.estimator.inputs.numpy_input_fn(
x={"x": np.array(training_set.data)},
y=np.array(training_set.target),
num_epochs=None,
batch_size=5, # I added this option but it seems to split my model by lines, I need to split it by column
shuffle=True)
# Train the model
mnist_classifier.train(
input_fn=train_input_fn,
steps=100,
hooks=[logging_hook])