I want to build a custom accuracy metric with tolerance. Instead of counting elements exactly equal in y_true and y_pred, this accuracy regards the two elements are consistent if their difference within a given tolerance value. For example, if the differences between predicted degrees and true degrees are smaller than 5 degree, we can think the results are correct and calculate the accuracy based on this rule. I want to use this metric in model.compile so it should be a callable function.
I wrote a function as follows.
def accuracy_with_tolerence(y_true,y_pred):
"""
y_true/y_pred: batch of samples; (BatchSize, 1)
"""
threshold = 5
differnece = tf.abs(tf.subtract(y_true,y_pred)) - threshold
boolean_results = [True if i < 0 else False for i in differnece]
return K.mean(math_ops.cast(boolean_results, K.floatx()))
It can return the correct accuracy value.
x = tf.constant([1, 2, 3], dtype=tf.float32)
y = tf.constant([5, 8, 10], dtype=tf.float32)
acc = accuracy_with_tolerence(x,y)
print(acc)
tf.Tensor(0.33333334, shape=(), dtype=float32)
But when I want to use it in compile, there is an error:
# Initialize ResNet50
model = resnet50()
model.compile(optimizer='adam',loss='mse',metrics=[accuracy_with_tolerence])
model.load_weights(checkpoint_filepath_0)
model.evaluate(x_test,y_test)
OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.
It seems I cannot iterate the Tensor. So how can I get element-wise boolean comparison results in the metric function? How can I realize this accuracy function?
Thank you in advance.
You can't make a list comprehension with a tensor. The operation you're looking for is tf.where and you can use it as follows:
def accuracy_with_tolerence(y_true, y_pred):
threshold = 5
differnece = tf.abs(tf.subtract(y_true, y_pred)) - threshold
boolean_results = tf.where(differnece>0, True, False)
return K.mean(math_ops.cast(boolean_results, K.floatx()))
Note that you can simplify the code further:
...
boolean_results = tf.where(tf.abs(tf.subtract(y_true, y_pred)) - threshold>0, 1., 0.)
return K.mean(boolean_results)
Related
I'm trying to segment data where the label can be quite sparse. Therefore I want to only calculate gradients in columns that have at least one nonzero value.
I've tried some methods where I apply an extra input which is the mask of these nonzero columns, but given that all the necessary information already is contained in y_true, a method which only looks at y_true to find the mask would definitely be preferable.
If I would implement it with numpy, it would probably look something like this:
def loss(y_true, y_pred):
indices = np.where(np.sum(y_true, axis=1) > 0)
return binary_crossentropy(y_true[indices], y_pred[indices])
y_true and y_pred are in this example vectorized 2D images.
How could this be "translated" to a differentiable Keras loss function?
Use tf-compatible operations, via tf and keras.backend:
import tensorflow as tf
import keras.backend as K
from keras.losses import binary_crossentropy
def custom_loss(y_true, y_pred):
indices = K.squeeze(tf.where(K.sum(y_true, axis=1) > 0))
y_true_sparse = K.cast(K.gather(y_true, indices), dtype='float32')
y_pred_sparse = K.cast(K.gather(y_pred, indices), dtype='float32')
return binary_crossentropy(y_true_sparse, y_pred_sparse) # returns a tensor
I'm unsure about the exact dimensionality specs of your problem, but loss must evaluate to a single value - which above doesn't, since you're passing multi-dimensional predictions and labels. To reduce dims, wrap the return above with e.g. K.mean. Example:
y_true = np.random.randint(0,2,(10,2))
y_pred = np.abs(np.random.randn(10,2))
y_pred /= np.max(y_pred) # scale between 0 and 1
print(K.get_value(custom_loss(y_true, y_pred))) # get_value evaluates returned tensor
print(K.get_value(K.mean(custom_loss(y_true, y_pred))
>> [1.1489482 1.2705883 0.76229745 5.101402 3.1309896] # sparse; 5 / 10 results
>> 2.28284 # single value, as required
(Lastly, note that this sparsity will bias the loss by excluding all-zero columns from the total label/pred count; if undesired, you can average via K.sum and K.shape or K.size)
I want to create a loss function where the MSE is only calculated on a subset of the outputs. The subset depends on the input data. I used the answer to this question to figure out how to create a custom function based on the input data:
Custom loss function in Keras based on the input data
However, I'm having trouble implementing the custom function to work.
Here is what I've put together.
def custom_loss(input_tensor):
def loss(y_true, y_pred):
board = input_tensor[:81]
answer_vector = board == .5
#assert np.sum(answer_vector) > 0
return K.mean(K.square(y_pred * answer_vector - y_true), axis=-1)
return loss
def build_model(input_size, output_size):
learning_rate = .001
a = Input(shape=(input_size,))
b = Dense(60, activation='relu')(a)
b = Dense(60, activation='relu')(b)
b = Dense(60, activation='relu')(b)
b = Dense(output_size, activation='linear')(b)
model = Model(inputs=a, outputs=b)
model.compile(loss=custom_loss(a), optimizer=Adam(lr=learning_rate))
return model
model = build_model(83, 81)
I want the MSE to treat the output as 0 wherever the board is not equal to 0.5. (The true value is one hot encoded with the one being within the subset). For some reason my output my output is treated as always zero. In other words, the custom loss function doesn't seem to be finding any places where the board is equal to 0.5.
I can't tell if I'm misinterpretting the dimensions or if the comparisons are failing due to the tensors, or even if there is just a generally much easier approach to do what I'm trying.
The problem is that answer_vector = board == .5 is not what you think it is. It is not a tensor, but the boolean value False, since board is a tensor and 0.5 is a number:
a = tf.constant([0.5, 0.5])
print(a == 0.5) # False
Now, a * False is a vector fo zeros:
with tf.Session() as sess:
print(sess.run(a * False)) # [0.0, 0.0]
You need to use tf.equal instead of ==. Another possible pitfall is that comparing floats with equality is dangerous, see e.g. What's wrong with using == to compare floats in Java?
I want to be able to know the rounded accuracy of my neural network when the prediction is above or below a certain threshold. For example, I want it to only calculate accuracy when the prediction is above 0.55 or below 0.45 in order to filter out near 50/50 cases.
I tried using the soft_acc function on stackoverflow and adding an if else to the beginning to filter out the near 50/50s.
def soft_acc(y_true, y_pred):
if y_pred > 0.55 or y_pred < 0.45:
return K.mean(K.equal(K.round(y_true), K.round(y_pred)))
I received the following error message.
TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.
Use tf.boolean_mask to filter out values at indices that don't meet the required threshold.
# remove values from `X` in interval (lo, hi)
mask = tf.math.logical_or(tf.lesser(X, lo), tf.greater(X, hi))
X = tf.boolean_mask(X, mask)
In your case, you would define soft_acc as
def soft_acc(y_true, y_pred):
mask = tf.math.logical_or(tf.greater(y_pred, 0.55), tf.lesser(y_pred, 0.45))
y_true2 = tf.boolean_mask(y_true, mask)
y_pred2 = tf.boolean_mask(y_pred, mask)
return K.mean(K.equal(K.round(y_true2), K.round(y_pred2)))
I want to do evaluation of a classification Tensorflow model.
To compute the accuracy, I have the following code :
predictions = tf.argmax(logits, axis=-1, output_type=tf.int32)
accuracy = tf.metrics.accuracy(labels=label_ids, predictions=logits)
It work well in single label classification, but now I want to do multilabel classification, where my labels are Array of Integers instead of Integers.
Here is an example of label [0, 1, 1, 0, 1, 0] that are stored in label_ids, and an example of predictions [0.1, 0.8, 0.9, 0.1, 0.6, 0.2] from the Tensor logits
What function should I use instead of argmax to do so ? (My labels are arrays of 6 Integers with value of either 0 or 1)
If needed, we can suppose that there is a threshold of 0.5.
It is probably better to do this type of post-processing evaluation outside of tensorflow, where it is more natural to try several different thresholds.
If you want to do it in tensorflow, you can consider:
predictions = tf.math.greater(logits, tf.constant(0.5))
This will return a tensor of the original logits shape with True for all entries greater than 0.5. You can then calculate accuracy as before. This is suitable for cases where many labels can be simultaneously true for a given sample.
Use below code to caclutae accuracy in multiclass classification:
tf.argmax will return the axis where y value is max for both y_pred and y_true(actual y).
Further tf.equal is used to find total number of matches (It returns True, False).
Convert the boolean into float(i.e. 0 or 1) and use tf.reduce_mean to calculate the accuracy.
correct_mask = tf.equal(tf.argmax(y_pred,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_mask, tf.float32))
Edit
Example with data:
import numpy as np
y_pred = np.array([[0.1,0.5,0.4], [0.2,0.6,0.2], [0.9,0.05,0.05]])
y_true = np.array([[0,1,0],[0,0,1],[1,0,0]])
correct_mask = tf.equal(tf.argmax(y_pred,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_mask, tf.float32))
with tf.Session() as sess:
# print(sess.run([correct_mask]))
print(sess.run([accuracy]))
Output:
[0.6666667]
I want to create a custom metric for pearson correlation as defined here
I'm not sure how exactly to apply it to batches of y_pred and y_true
What I did:
def pearson_correlation_f(y_true, y_pred):
y_true,_ = tf.split(y_true[:,1:],2,axis=1)
y_pred, _ = tf.split(y_pred[:,1:], 2, axis=1)
fsp = y_pred - K.mean(y_pred,axis=-1,keepdims=True)
fst = y_true - K.mean(y_true,axis=-1, keepdims=True)
corr = K.mean((K.sum((fsp)*(fst),axis=-1))) / K.mean((
K.sqrt(K.sum(K.square(y_pred -
K.mean(y_pred,axis=-1,keepdims=True)),axis=-1) *
K.sum(K.square(y_true - K.mean(y_true,axis=-1,keepdims=True)),axis=-1))))
return corr
Is it necessary for me to use keepdims and handle the batch dimension manually and the take the mean over it? Or does Keras somehow do this automatically?
When you use K.mean without an axis, Keras automatically calculates the mean for the entire batch.
And the backend already has standard deviation functions, so it might be cleaner (and perhaps faster) to use them.
If your true data is shaped like (BatchSize,1), I'd say keep_dims is unnecessary. Otherwise I'm not sure and it would be good to test the results.
(I don't understand why you use split, but it seems also unnecessary).
So, I'd try something like this:
fsp = y_pred - K.mean(y_pred) #being K.mean a scalar here, it will be automatically subtracted from all elements in y_pred
fst = y_true - K.mean(y_true)
devP = K.std(y_pred)
devT = K.std(y_true)
return K.mean(fsp*fst)/(devP*devT)
If it's relevant to have the loss for each feature instead of putting them all in the same group:
#original shapes: (batch, 10)
fsp = y_pred - K.mean(y_pred,axis=0) #you take the mean over the batch, keeping the features separate.
fst = y_true - K.mean(y_true,axis=0)
#mean shape: (1,10)
#fst shape keeps (batch,10)
devP = K.std(y_pred,axis=0)
devt = K.std(y_true,axis=0)
#dev shape: (1,10)
return K.sum(K.mean(fsp*fst,axis=0)/(devP*devT))
#mean shape: (1,10), making all tensors in the expression be (1,10).
#sum is only necessary because we need a single loss value
Summing the result of the ten features or taking a mean of them is the same, being one 10 times the other (That is not very relevant to keras models, affecting only the learning rate, but many optimizers quickly find their way around this).