I have a custom loss function using keras:
import keras.backend as K
def IoU(y_true, y_pred, eps=1e-6):
if K.max(y_true) == 0.0:
return IoU(1 - y_true, 1 - y_pred) ## empty image; calc IoU of zeros
intersection = K.sum(y_true * y_pred, axis=[1,2,3])
union = K.sum(y_true, axis=[1,2,3]) + K.sum(y_pred, axis=[1,2,3]) - intersection
return -K.mean((intersection + eps) / (union + eps), axis=0)
This results in an endless recursion, since if K.max(y_true) == 0.0: always evaluates to true. Why is this the case? Do I need to extract a single value out of the output of K.max? I tried converting y_true to a numpy array and using np.max instead but this was not easily possible.
Or does 1 - y_true not work the way numpy arrays would work?
Edit: y_true and y_pred are both tensors with shape:
Tensor("IoU/Shape:0", shape=(4,), dtype=int32). y_true is mostly filled with zeros, but some non zero values are present.
Related
In my Tensorflow model y_pred contains probabilities from 0 to 1 and y_true contains labels of 0 and 1.
In my custom loss function I'd like to use the information of 4 (or n) consecutive pairs of y_true and y_pred.
In numpy I could do something like this
y_true=np.array([1,1,1,1,0,0,0,])
y_pred=np.array([0.5,0.5,0.5,0.5,0.2,0.2,0.2,0.2])
def custom_loss(y_true, y_pred):
n=len(t)
res= 0
for i in range(0,n,4):
res += np.sum(y_true[i:i+4])-np.sum(y_pred[i:i+4])
return res
Is there a way to achieve this in Tensorflow with tensors?
I am using Tensorflow version 2.2.0 and python 3.8
Taking care of when the len(y_true) % 4 != 0:
#tf.function
def custom_loss_tf(y_true, y_pred):
length = tf.shape(y_true)[0]
end_i = length % 4
start_y_true, end_y_true = y_true[:length-end_i], y_true[length-end_i:]
start_y_pred, end_y_pred = y_pred[:length-end_i], y_pred[length-end_i:]
sum_start_y_true = tf.reduce_sum(tf.reshape(start_y_true, (-1,4)), axis=0)
sum_start_y_pred = tf.reduce_sum(tf.reshape(start_y_pred, (-1,4)), axis=0)
res = tf.reduce_sum(tf.cast(sum_start_y_true, tf.float32)) - tf.reduce_sum(tf.cast(sum_start_y_pred, tf.float32))
res_ending = tf.reduce_sum(tf.cast(end_y_true, tf.float32) - tf.cast(end_y_pred, tf.float32))
return res_ending + res
Your function doesn't make a lot of sense though, you are calculating sums of sums. Can't you just sum everything?
What I am trying to achieve now is to create a custom loss function in Keras that takes in two tensors (y_true, y_pred) with shapes (None, None, None) and (None, None, 3), respectively. However, the None's are so, that the two shapes are always equal for every (y_true, y_pred). From these tensors I want to produce two distance matrices that contain the squared distances between every possible point pair (the third, length 3 dimension contains x, y, and z spatial values) inside them and then return the difference between these distance matrices. The first code I tried was this:
def distanceMatrixLoss1(y_true, y_pred):
distMatrix1 = [[K.sum(K.square(y_true[i] - y_true[j])) for j in range(i + 1, y_true.shape[1])] for j in range(y_true.shape[1])]
distMatrix2 = [[K.sum(K.square(y_pred[i] - y_pred[j])) for j in range(i + 1, y_pred.shape[1])] for j in range(y_pred.shape[1])]
return K.mean(K.square(K.flatten(distMatrix1) - K.flatten(distMatrix2)))
(K is the TensorFlow backend.) Needless to say, I got the following error:
'NoneType' object cannot be interpreted as an integer
This is understandable, since range(None) does not make a lot of sense and y_true.shape[0] or y_pred.shape[0] is None. I searched whether others got somehow the same problem or not and I found that I could use the scan function of TensorFlow:
def distanceMatrixLoss2(y_true, y_pred):
subtractYfromXi = lambda x, y: tf.scan(lambda xi: K.sum(K.square(xi - y)), x)
distMatrix = lambda x, y: K.flatten(tf.scan(lambda yi: subtractYfromXi(x, yi), y))
distMatrix1 = distMatrix(y_true, y_true)
distMatrix2 = distMatrix(y_pred, y_pred)
return K.mean(K.square(distMatrix1-distMatrix2))
What I got from this is a different error, that I do not fully understand.
TypeError: <lambda>() takes 1 positional argument but 2 were given
So this went into the trash too. My last try was using the backend's map_fn function:
def distanceMatrixLoss3(y_true, y_pred):
subtractYfromXi = lambda x, y: K.map_fn(lambda xi: K.sum(K.square(xi - y)), x)
distMatrix = lambda x, y: K.flatten(K.map_fn(lambda yi: subtractYfromXi(x, yi), y))
distMatrix1 = distMatrix(y_true, y_true)
distMatrix2 = distMatrix(y_pred, y_pred)
return K.mean(K.square(distMatrix1-distMatrix2))
This did not throw an error, but when the training started the loss was constant 0 and stayed that way. So now I am out of ideas and I kindly ask you to help me untangle this problem. I have already tried to do the same in Mathematica and also failed (here is the link to the corresponding question, if it helps).
Assuming that dimension 0 is the batch size as usual and you don't want to mix samples.
Assuming that dimension 1 is the one you want to make pairs
Assuming that the last dimension is 3 for all cases although your model returns None.
Iterating tensors is a bad idea. It might be better just to make a 2D matrix from the original 1D, though having repeated values.
def distanceMatrix(true, pred): #shapes (None1, None2, 3)
#------ creating the distance matrices 1D to 2D -- all vs all
true1 = K.expand_dims(true, axis=1) #shapes (None1, 1, None2, 3)
pred1 = K.expand_dims(pred, axis=1)
true2 = K.expand_dims(true, axis=2) #shapes (None1, None2, 1, 3)
pred2 = K.expand_dims(pred, axis=2)
trueMatrix = true1 - true2 #shapes (None1, None2, None2, 3)
predMatrix = pred1 - pred2
#--------- euclidean x, y, z distance
#maybe needs a sqrt?
trueMatrix = K.sum(K.square(trueMatrix), axis=-1) #shapes (None1, None2, None2)
predMatrix = K.sum(K.square(predMatrix), axis=-1)
#-------- loss for each pair
loss = K.square(trueMatrix - predMatrix) #shape (None1, None2, None2)
#----------compensate the duplicated non-diagonals
diagonal = K.eye(K.shape(true)[1]) #shape (None2, None2)
#if Keras complains because the input is a tensor, use `tf.eye`
diagonal = K.expand_dims(diagonal, axis=0) #shape (1, None2, None2)
diagonal = 0.5 + (diagonal / 2.)
loss = loss * diagonal
#--------------
return K.mean(loss, axis =[1,2]) #or just K.mean(loss)
I have two arrays (y_true and y_pred), both consisting of 0's and 1's of the same length.
I want a more efficient/faster way of counting how many times y_pred == y_true, AND when y_pred == 1. I'm not interested in counting the matching 0's.
Right now, my function looks like this using a for loop:
from sklearn.metrics.scorer import make_scorer
# Make a custom metric function
def my_custom_accuracy(y_true, y_pred): # Bring in the arrays
good_matches = 0 # Set counter to 0
for num, i in enumerate(y_pred): # for each y_pred in array...
if i == y_true[num] & i == 1: # if y_pred == y_true AND y_pred == 1...
good_matches += 1 # count it as a good match
return float(good_matches / sum(y_true)) # return good matches as a % of all the 1's in y_true
....it works, but the for loop is slow and not very efficient. I was hoping to utilize something like this:
# Make a custom metric function
def my_custom_accuracy(y_true, y_pred):
return float(sum(y_pred == y_true)) / sum(y_true)
...simple, but I don't know how to add in the "& y_pred == 1" part. Any ideas? Thanks!
If the arrays aren't already boolean, make them boolean. This can be done cheaply with a view, or more simply with astype:
y_pred = y_pred.astype(bool)
y_true = y_true.astype(bool)
This step can be omitted if the arrays are already boolean, or if they really will never contain anything but zeros and ones.
Now good_matches is just
good_matches = np.sum(y_pred & y_true)
To see why that's so, note that in addition to obviously containing y_pred == y_true, the expression can only be true when y_pred is true, so it automatically implies y_pref == 1 and y_true == 1, by the definition of the & operator.
Your final result is therefore
np.sum(y_pred & y_true) / np.sum(y_true)
This can be alternatively written as
np.count_nonzero(y_pred & y_true) / np.count_nonzero(y_true)
You can use a list comprehension to check the lists against each other while filtering out y_pred == 0, then get your accuracy by dividing the matches by the length of the compare list.
compare = [p == t for p, t in zip(y_pred, y_true) if p == 1]
accuracy = compare.count(True) / len(compare)
Or for something utilizing numpy:
mask = np.where(y_true == y_pred)
matches = y_pred[mask]
accuracy = np.sum(matches) / len(matches)
I am trying to use Poisson unscaled deviance as a loss function for my neural network, but there's a major flow with this : y_true can take (and will take very often) the value 0.
Unscaled deviance works like this for Poisson case :
If y_true = 0, then loss = 2 * d * y_pred
If y_true > 0, then loss = 2 * d *y_pred * (y_true * log(y_true)-y_true * log(y_pred)-y_true+y_pred
Note that as soon as log(0) is computed, the loss becomes -inf so my goal is to prevent this to happen.
I tried using the switch function to solve this but here's the trick:
If I have the value log(0), I don't want to replace it by 0 (with K.zeros()) because it would be considering that y_true = 1 since log(1) = 0.
Therefore I want to try using a large negative value in this case (-10000 for example) but I don't know how to do this since K.variable(-10000) gives the error:
ValueError: Rank of `condition` should be less than or equal to rank of `then_expression` and `else_expression`. ndim(condition)=1, ndim(then_expression)=0
Using K.zeros_like(y_true) instead of K.variable(-10000) will work for keras but it is mathematically incorrect and the optimisation doesn't work properly because of this.
I'd like to know how to replace the log by a large negative value in the switch function. Here's my attempt:
def custom_loss3(data, y_pred):
y_true = data[:, 0]
d = data[:, 1]
# condition
loss_value = KB.switch(KB.less_equal(y_true, 0),
2 * d * y_pred, 2 * d * (y_true * KB.switch(KB.less_equal(y_true, 0),
KB.variable(-10000), KB.log(y_true)) - y_true * KB.switch(KB.less_equal(y_pred, 0.), KB.variable(-10000), KB.log(y_pred)) - y_true + y_pred))
return loss_value
I am training to train a Neural Network using Keras and I am using my own metric function as the loss function. The reason for this is that the actual values in the test set have a lot of NaN values. Let me give an example of the actual values in the test set:
12
NaN
NaN
NaN
8
NaN
NaN
3
In the preprocessing of my data, I replaced all the NaN values with zeros, so the above example contains zeros on each NaN row.
The Neural Network produces an output like this:
14
12
9
9
8
7
6
3
I only want to calculate the root mean squared error between the non-zero values. So for the example above, it should only calculate the RMSE for rows 1, 5 and 8. To do this, I created the following function:
from sklearn.metrics import mean_squared_error
from math import sqrt
[...]
def evaluation_metric(y_true, y_pred):
y_true = y_true[np.nonzero(y_true)]
y_pred = y_pred[np.nonzero(y_true)]
error = sqrt(mean_squared_error(y_true, y_pred))
return error
When you test the function by hand, by feeding the actual values from the test set and an output from the neural network that is initialized with random weights, it works well an produces an error value. I am able to optimize the weights using an Evolutionary approach, and I am able to optimize this error measure by adjusting the weights of the network.
Now, I want to train the network with evaluation_metric as the loss function using the model.compile function from Keras. When I run:
model.compile(loss=evaluation_metric, optimizer='rmsprop', metrics=[evaluation_metric])
I get the following error:
TypeError: Using a tf.Tensor as a Python bool is not allowed. Use if t is not None: instead of if t: to test if a tensor is defined, and use TensorFlow ops such as tf.cond to execute subgraphs conditioned on the value of a tensor.
I think this has to do with the usage of np.nonzero. Since I am working with Keras, I should probably use a function of the Keras Backend, or using something like tf.cond to check for the non zero values of y_true.
Can someone help me with this?
EDIT
The code works after applying the following fix:
def evaluation_metric(y_true, y_pred):
y_true = y_true * (y_true != 0)
y_pred = y_pred * (y_true != 0)
error = root_mean_squared_error(y_true, y_pred)
return error
Along with the following function for calculating the RMSE of a tf object:
def root_mean_squared_error(y_true, y_pred):
return K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))
Yes, indeed the problem lies in using numpy function. Here is a quick fix:
def evaluation_metric(y_true, y_pred):
y_true = y_true * (y_true != 0)
y_pred = y_pred * (y_true != 0)
error = sqrt(mean_squared_error(y_true, y_pred))
return error
I would write the metric in tensorflow on my own like:
import tensorflow as tf
import numpy as np
data = np.array([0, 1, 2, 0, 0, 3, 7, 0]).astype(np.float32)
pred = np.random.randn(8).astype(np.float32)
gt = np.random.randn(8).astype(np.float32)
data_op = tf.convert_to_tensor(data)
pred_op = tf.convert_to_tensor(pred)
gt_op = tf.convert_to_tensor(gt)
expected = np.sqrt(((gt[data != 0] - pred[data != 0]) ** 2).mean())
def nonzero_mean(gt_op, pred_op, data_op):
mask_op = 1 - tf.cast(tf.equal(data_op, 0), tf.float32)
actual_op = ((gt_op - pred_op) * mask_op)**2
actual_op = tf.reduce_sum(actual_op) / tf.cast(tf.count_nonzero(mask_op), tf.float32)
actual_op = tf.sqrt(actual_op)
return actual_op
with tf.Session() as sess:
actual = sess.run(nonzero_mean(gt_op, pred_op, data_op))
print actual, expected
The y_true != 0 is not possible in plain Tensorflow. Not sure, if keras does some magic here.