I want to be able to know the rounded accuracy of my neural network when the prediction is above or below a certain threshold. For example, I want it to only calculate accuracy when the prediction is above 0.55 or below 0.45 in order to filter out near 50/50 cases.
I tried using the soft_acc function on stackoverflow and adding an if else to the beginning to filter out the near 50/50s.
def soft_acc(y_true, y_pred):
if y_pred > 0.55 or y_pred < 0.45:
return K.mean(K.equal(K.round(y_true), K.round(y_pred)))
I received the following error message.
TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.
Use tf.boolean_mask to filter out values at indices that don't meet the required threshold.
# remove values from `X` in interval (lo, hi)
mask = tf.math.logical_or(tf.lesser(X, lo), tf.greater(X, hi))
X = tf.boolean_mask(X, mask)
In your case, you would define soft_acc as
def soft_acc(y_true, y_pred):
mask = tf.math.logical_or(tf.greater(y_pred, 0.55), tf.lesser(y_pred, 0.45))
y_true2 = tf.boolean_mask(y_true, mask)
y_pred2 = tf.boolean_mask(y_pred, mask)
return K.mean(K.equal(K.round(y_true2), K.round(y_pred2)))
Related
I want to build a custom accuracy metric with tolerance. Instead of counting elements exactly equal in y_true and y_pred, this accuracy regards the two elements are consistent if their difference within a given tolerance value. For example, if the differences between predicted degrees and true degrees are smaller than 5 degree, we can think the results are correct and calculate the accuracy based on this rule. I want to use this metric in model.compile so it should be a callable function.
I wrote a function as follows.
def accuracy_with_tolerence(y_true,y_pred):
"""
y_true/y_pred: batch of samples; (BatchSize, 1)
"""
threshold = 5
differnece = tf.abs(tf.subtract(y_true,y_pred)) - threshold
boolean_results = [True if i < 0 else False for i in differnece]
return K.mean(math_ops.cast(boolean_results, K.floatx()))
It can return the correct accuracy value.
x = tf.constant([1, 2, 3], dtype=tf.float32)
y = tf.constant([5, 8, 10], dtype=tf.float32)
acc = accuracy_with_tolerence(x,y)
print(acc)
tf.Tensor(0.33333334, shape=(), dtype=float32)
But when I want to use it in compile, there is an error:
# Initialize ResNet50
model = resnet50()
model.compile(optimizer='adam',loss='mse',metrics=[accuracy_with_tolerence])
model.load_weights(checkpoint_filepath_0)
model.evaluate(x_test,y_test)
OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.
It seems I cannot iterate the Tensor. So how can I get element-wise boolean comparison results in the metric function? How can I realize this accuracy function?
Thank you in advance.
You can't make a list comprehension with a tensor. The operation you're looking for is tf.where and you can use it as follows:
def accuracy_with_tolerence(y_true, y_pred):
threshold = 5
differnece = tf.abs(tf.subtract(y_true, y_pred)) - threshold
boolean_results = tf.where(differnece>0, True, False)
return K.mean(math_ops.cast(boolean_results, K.floatx()))
Note that you can simplify the code further:
...
boolean_results = tf.where(tf.abs(tf.subtract(y_true, y_pred)) - threshold>0, 1., 0.)
return K.mean(boolean_results)
I am currently working with an LSTM sequence to sequence model for time domain signal predictions. Because of domain knowledge, I know that the first part of the prediction (about 20%) can never be predicted correctly, since the information required is not available in the given input sequence. The remaining 80% of the predicted sequence are usually predicted quite well. In order to exclude the first 20% from the training optimization, it would be nice to define a loss function that would basically operate on a given index range like the numpy code below:
start = int(0.2*sequence_length)
stop = sequence_length
def mse(pred, target):
""" Mean squared error between two time series np.arrays """
return 1/target.shape[0]*np.sum((pred-target)**2)
def range_mse_loss(y_pred, y):
return mse(y_pred[start:stop],y[start:stop])
How do I have to write this loss function in order to have it work with my preexisting keras code, where loss is simply given by model.compile(loss='mse') ?
You can slice your tensor to just last 80% of the data.
size = int(y_true.shape[0] * 0.8) # for 2D vector, e.g., (100, 1)
loss_fn = tf.keras.losses.MeanSquaredError(name='mse')
loss_fn(y_pred[:-size], y_true[:-size])
You can also use the sample_weights at the tf.keras.losses.MeanSquaredError(), passing an array of weights and the first 20% of weights is zero
size = int(y_true.shape[0] * 0.8) # for 2D vector, e.g., (100, 1)
zeros = tf.zeros((y_true.shape[0] - size), dtype=tf.int32)
ones = tf.ones((size), dtype=tf.int32)
weights = tf.concat([zeros, ones], 0)
loss_fn = tf.keras.losses.MeanSquaredError(name='mse')
loss_fn(y_pred, y_true, sample_weights=weights)
There is a warming of the second solution, the final loss will be lower than the first solution, because you are putting zero in the first predictions values, but you aren't removing them in the formula MSE = 1 /n * sum((y-y_hat)^2).
One workaround would be to mark the observations as None/nan and then overwrite the train_step method. Following tensorflow's tutorial about customizing train_step, you would do something like this
#tf.function
def train_step(keras_model, data):
print('custom train_step')
# Unpack the data. Its structure depends on your model and
# on what you pass to `fit()`.
x, y = data
with tf.GradientTape() as tape:
y_pred = keras_model(x, training=True) # Forward pass
# masking nan values in observations, also assuming that targets are >0.0
mask = tf.greater(y, 0.0)
true_y = tf.boolean_mask(y, mask)
pred_y = tf.boolean_mask(y_pred, mask)
# Compute the loss value
# (the loss function is configured in `compile()`)
loss = keras_model.compiled_loss(true_y, pred_y, regularization_losses=keras_model.losses)
# Compute gradients
trainable_vars = keras_model.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
# Update weights
keras_model.optimizer.apply_gradients(zip(gradients, trainable_vars))
# Update metrics (includes the metric that tracks the loss)
keras_model.compiled_metrics.update_state(true_y, pred_y)
# Return a dict mapping metric names to current value
return {m.name: m.result() for m in keras_model.metrics}
This will work for all the performance metrics you are tracking. Alternative way would be to mask the nans inside the loss function but that would be limited to only one loss function and not any other loss function/performance metrics.
I'm trying to segment data where the label can be quite sparse. Therefore I want to only calculate gradients in columns that have at least one nonzero value.
I've tried some methods where I apply an extra input which is the mask of these nonzero columns, but given that all the necessary information already is contained in y_true, a method which only looks at y_true to find the mask would definitely be preferable.
If I would implement it with numpy, it would probably look something like this:
def loss(y_true, y_pred):
indices = np.where(np.sum(y_true, axis=1) > 0)
return binary_crossentropy(y_true[indices], y_pred[indices])
y_true and y_pred are in this example vectorized 2D images.
How could this be "translated" to a differentiable Keras loss function?
Use tf-compatible operations, via tf and keras.backend:
import tensorflow as tf
import keras.backend as K
from keras.losses import binary_crossentropy
def custom_loss(y_true, y_pred):
indices = K.squeeze(tf.where(K.sum(y_true, axis=1) > 0))
y_true_sparse = K.cast(K.gather(y_true, indices), dtype='float32')
y_pred_sparse = K.cast(K.gather(y_pred, indices), dtype='float32')
return binary_crossentropy(y_true_sparse, y_pred_sparse) # returns a tensor
I'm unsure about the exact dimensionality specs of your problem, but loss must evaluate to a single value - which above doesn't, since you're passing multi-dimensional predictions and labels. To reduce dims, wrap the return above with e.g. K.mean. Example:
y_true = np.random.randint(0,2,(10,2))
y_pred = np.abs(np.random.randn(10,2))
y_pred /= np.max(y_pred) # scale between 0 and 1
print(K.get_value(custom_loss(y_true, y_pred))) # get_value evaluates returned tensor
print(K.get_value(K.mean(custom_loss(y_true, y_pred))
>> [1.1489482 1.2705883 0.76229745 5.101402 3.1309896] # sparse; 5 / 10 results
>> 2.28284 # single value, as required
(Lastly, note that this sparsity will bias the loss by excluding all-zero columns from the total label/pred count; if undesired, you can average via K.sum and K.shape or K.size)
I'm trying to use exact match / subset accuracy as a metric for my Keras model. I understand basically how it's supposed to work, but I'm having a hard time with the tensor manipulation.
I'm working on a multilabel classification task with 55 possible labels. I'm considering an output > 0.5 to be a positive for that label. I want a metric that describes how often the output exactly matches the true labels.
My approach is to convert y_true to tf.bool, and y_pred > 0.5 to tf.bool, and then return a tensor containing True if they match exactly, and False otherwise. It appears to be working when I do basic tests, but when I train the model, it stays at 0.0000 without ever changing.
def subset_accuracy(y_true, y_pred):
y_pred_bin = tf.cast(y_pred > 0.5, tf.bool)
equality = tf.equal(tf.cast(y_true, tf.bool), y_pred_bin)
return tf.equal(
tf.cast(tf.math.count_nonzero(equality), tf.int32),
tf.size(y_true)
)
I am expecting to see the metric slowly climb, even if it only goes up to 50% or something. But it's staying at 0.0.
Here is another option, tested with tensorflow 2.3:
def subset_accuracy(y_true, y_pred):
threshold = tf.constant(.8, tf.float32)
gtt_pred = tf.math.greater(y_pred, threshold)
gtt_true = tf.math.greater(y_true, threshold)
accuracy = tf.reduce_mean(tf.cast(tf.equal(gtt_pred, gtt_true), tf.float32), axis=-1)
return accuracy
I would imagine that tf.cast(y_true, tf.bool) could be a problem, as it casts float to bool, so depending on how tf deals with it internally, it may first get cast to int, so anything < 1.0 will be zero, and then to bool. That's why nothing will match and you only get zero accuracy.
The above suggestion avoids that problem.
Suggestion: test your metric independently from the model. Use a model (untrained works as well) and model.evaluate a single batch. Compute the metric manually by using the output of model.predict.
Make sure that your computation and the metric the model outputs come to the same result and that the result makes sense for the values in this batch.
Once you are sure that your loss is really mathematically correct; you can then try to debug your model.
It is not clear from your code snippet what you consider to be subset accuracy.
For instance Keras defines categorical_acuracy as:
def categorical_accuracy(y_true, y_pred):
return K.cast(K.equal(K.argmax(y_true, axis=-1),
K.argmax(y_pred, axis=-1)),
K.floatx())
How do you intend your accuracy metric to be different. Just ensure that the value is greater than 0.5 ? Perhaps you may consider modifying the Keras metric.
I have an LSTM predicting time series values in tensorflow.
The model is working using an MSE as a loss function.
However, I'd like to be able to create a custom loss function where one of the error values is multiplied by two (therefore producing a higher error value).
In my batch of size 10, I want the 3rd value of the first input to be multiplied by 2, but because this is time series, this corresponds to the second value in the second input and the first value in the third input.
The error I get is:
ValueError: No gradients provided for any variable, check your graph for ops that do not support gradients
How do I make the gradients?
def loss_function(y_true, y_pred, peak_value=3, weight=2):
# peak value is where the multiplication happens on the first line
# weight is the how much the error is multiplied by
all_dif = tf.squared_difference(y_true, y_pred) # should be shape=[10,10]
peak = [peak_value] * 10
listy = range(0, 10)
c = [(i - j) % 10 for i, j in zip(peak, listy)]
for i in range(0, 10):
indices = [[i, c[i]]]
values = [1.0]
shape = [10,10]
delta = tf.SparseTensor(indices, values, shape)
all_dif = all_dif + tf.sparse_tensor_to_dense(delta)
return tf.reduce_sum(all_dif)
I believe the psuedo code would look something like this:
#tf.custom_gradient
def loss_function(y_true, y_pred, peak_value=3, weight=2)
## your code
def grad(dy):
return dy * partial_derivative
return loss, grad
Where partial_derivative is the analytically evaluated partial derivative with respect to your loss function. If your loss function is a function of more than one variable, it will require a partial derivative respect to each variable, I believe.
If you need more information, the documentation is good: https://www.tensorflow.org/api_docs/python/tf/custom_gradient
And I've yet to find an example of this functionality embedded in a model that's not a toy.