I am training a model batch by batch. The training goal is to minimize the batch log loss. When I test the model, batches are used as well. For accuracy and AUC, I can use tf.metrics.auc and tf.matrics.accuracy to get running accuracy and AUC. However, how can I get running logloss for the test data?
You can get a running metric from any metric M by calling tf.metrics.mean(M).
For example,
my_loss = tf.losses.log_loss(labels, predictions)
my_running_loss, update_op = tf.metrics.mean(my_loss)
Related
I am trying to feed a CNN model(Human body pose estimation)with a dataset contains 1000 numbers,
first, how can I make sure that the number of my datasets is already enough?
second, how should i split my data to train and test size? (when I put train size = 0.6 and test_size = 0.4 the network doesnt work well and show me NAN for weights and bias and loss value!)
There is no fixed way to determine when you have a sufficient size data set. It depends on many factors. Best thing to do is run with what you have and see how it performs. I usually split my data into 3 sets, training, validation and test. I usually try 75% for training, 15% for validation and 10% for final test.The validation set is what I use to tweek the hyper parameters. Initially I monitor the training accuracy and loss. If I can get that up to over 95% then I monitor the validation accuracy and loss. I use the model_checkpoint keras callback to save the model with the lowest validation loss. If the validation accuracy and loss is not satisfactory I tweek the hyper parameters to try to improve it. I have found using an adjustable learning rate to be useful for this purpose. Finally when I am satisfied with the training accuracy and validation accuracy I use the saved model to make predictions on the test set. This is the final measure of how the model performs.
I want to ask a question about how to monitor validation loss in the training process of estimators in TensorFlow. I have checked a similar question (validation during training of Estimator) asked before, but it did not help much.
If I use estimators to build a model, I will give an input function to the Estimator.train() function. But there is no way to add another validation_x, and validation_y data in the training process. Therefore, when the training started, I can only see the training loss. The training loss is expected to decrease when the training process running longer. However, this information is not helpful to prevent overfitting. The more valuable information is validation loss. Usually, the validation loss is the U-shape with the number of epochs. To prevent overfitting, we want to find the number of epochs that the validation loss is minimum.
So this is my problem. How can I get validation loss for each epoch in the training process of using estimators?
You need to create a validation input_fn and either use estimator.train() and estimator.evaluate() alternatively or simpy use tf.estimator.train_and_evaluate()
x = ...
y = ...
...
# For example, if x and y are numpy arrays < 2 GB
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
val_dataset = tf.data.Dataset.from_tensor_slices((x_val_, y_val))
...
estimator = ...
for epoch in n_epochs:
estimator.train(input_fn = train_dataset)
estimator.evaluate(input_fn = val_dataset)
estimator.evaluate() will compute the loss and any other metrics that are defined in your model_fn and will save the events in a new "eval" directory inside your job_dir.
I have a Keras sequential model and I'm using:
model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
I can see the training accuracy printed when I use Keras fit() function to train the model.
I need to use Estimator API to train the model and I'm using model_to_estimator to convert the model to estimator. Then I use train_and_evaluate() to train the model.
However I don't see the accuracy graph in Tensorboard. There's only one accuracy value (from evaluation), so the graph is just a dot.
What I need is the accuracy graph from training like the one shown here:
https://www.tensorflow.org/guide/custom_estimators#tensorboard
I checked the examples and all I could find were ones where they use Estimator API to build the model and use following code to define a summary scalar.
# Compute evaluation metrics.
accuracy = tf.metrics.accuracy(labels=labels,
predictions=predicted_classes,
name='acc_op')
metrics = {'accuracy': accuracy}
tf.summary.scalar('accuracy', accuracy[1])
Does anyone know how to use this with models converted from Keras?
I'm using Tensorflow version r1.10.
It's well known that when fitting a keras model, the metrics that are reported include all the layers including dropouts. As such, reported metrics (precision, recall, f1, even accuracy) will differ from what was reported during the fit, and when using the predicted results using model.predict().
I'm curious whether it is possible to have keras perform checkpoints on a user-defined routine that uses the current model.predict() rather than the internal metric.
Example:
model.fit(X,y,validation_data=(Xv,yv),\
callbacks=[K.ModelCheckpoint('/tmp/dump.h5',\
monitor='val_acc')])
will perform checkpoints on the validation accuracy that's been measured during the fit (and reported using verbose=1). But suppose I want to perform the checkpoint with F1-score using the actual predictions that model would make on the validation set. Is it possible to get ModelCheckpoint to use something like sklearn.metrics.f1_score(yv,model.predict(Xv))?
I am training a CNN and I am getting results of 85% accuracy in the training set, and 65% accuracy in the test set.
Is it okey to assume that, with a proper setting of the regularization of the network (dropout and L2 in my case), my test accuracy should get very close to my training accuracy (which will at the same time decrease as the regularization increases) ?
So let's say for instance, a 75%-74% accuracy ?
With a proper setting of the regularization of all parameters of the network and with a well representative data batch, you should have a small difference between your test accuracy and your training accuracy. But of course you need to
optimize your model with parameter optimization and feature selection.
Maybe you can check this link to find some more informations.
Hope it helps !