I have two arrays (y_true and y_pred), both consisting of 0's and 1's of the same length.
I want a more efficient/faster way of counting how many times y_pred == y_true, AND when y_pred == 1. I'm not interested in counting the matching 0's.
Right now, my function looks like this using a for loop:
from sklearn.metrics.scorer import make_scorer
# Make a custom metric function
def my_custom_accuracy(y_true, y_pred): # Bring in the arrays
good_matches = 0 # Set counter to 0
for num, i in enumerate(y_pred): # for each y_pred in array...
if i == y_true[num] & i == 1: # if y_pred == y_true AND y_pred == 1...
good_matches += 1 # count it as a good match
return float(good_matches / sum(y_true)) # return good matches as a % of all the 1's in y_true
....it works, but the for loop is slow and not very efficient. I was hoping to utilize something like this:
# Make a custom metric function
def my_custom_accuracy(y_true, y_pred):
return float(sum(y_pred == y_true)) / sum(y_true)
...simple, but I don't know how to add in the "& y_pred == 1" part. Any ideas? Thanks!
If the arrays aren't already boolean, make them boolean. This can be done cheaply with a view, or more simply with astype:
y_pred = y_pred.astype(bool)
y_true = y_true.astype(bool)
This step can be omitted if the arrays are already boolean, or if they really will never contain anything but zeros and ones.
Now good_matches is just
good_matches = np.sum(y_pred & y_true)
To see why that's so, note that in addition to obviously containing y_pred == y_true, the expression can only be true when y_pred is true, so it automatically implies y_pref == 1 and y_true == 1, by the definition of the & operator.
Your final result is therefore
np.sum(y_pred & y_true) / np.sum(y_true)
This can be alternatively written as
np.count_nonzero(y_pred & y_true) / np.count_nonzero(y_true)
You can use a list comprehension to check the lists against each other while filtering out y_pred == 0, then get your accuracy by dividing the matches by the length of the compare list.
compare = [p == t for p, t in zip(y_pred, y_true) if p == 1]
accuracy = compare.count(True) / len(compare)
Or for something utilizing numpy:
mask = np.where(y_true == y_pred)
matches = y_pred[mask]
accuracy = np.sum(matches) / len(matches)
Related
I have a custom loss function using keras:
import keras.backend as K
def IoU(y_true, y_pred, eps=1e-6):
if K.max(y_true) == 0.0:
return IoU(1 - y_true, 1 - y_pred) ## empty image; calc IoU of zeros
intersection = K.sum(y_true * y_pred, axis=[1,2,3])
union = K.sum(y_true, axis=[1,2,3]) + K.sum(y_pred, axis=[1,2,3]) - intersection
return -K.mean((intersection + eps) / (union + eps), axis=0)
This results in an endless recursion, since if K.max(y_true) == 0.0: always evaluates to true. Why is this the case? Do I need to extract a single value out of the output of K.max? I tried converting y_true to a numpy array and using np.max instead but this was not easily possible.
Or does 1 - y_true not work the way numpy arrays would work?
Edit: y_true and y_pred are both tensors with shape:
Tensor("IoU/Shape:0", shape=(4,), dtype=int32). y_true is mostly filled with zeros, but some non zero values are present.
In my Tensorflow model y_pred contains probabilities from 0 to 1 and y_true contains labels of 0 and 1.
In my custom loss function I'd like to use the information of 4 (or n) consecutive pairs of y_true and y_pred.
In numpy I could do something like this
y_true=np.array([1,1,1,1,0,0,0,])
y_pred=np.array([0.5,0.5,0.5,0.5,0.2,0.2,0.2,0.2])
def custom_loss(y_true, y_pred):
n=len(t)
res= 0
for i in range(0,n,4):
res += np.sum(y_true[i:i+4])-np.sum(y_pred[i:i+4])
return res
Is there a way to achieve this in Tensorflow with tensors?
I am using Tensorflow version 2.2.0 and python 3.8
Taking care of when the len(y_true) % 4 != 0:
#tf.function
def custom_loss_tf(y_true, y_pred):
length = tf.shape(y_true)[0]
end_i = length % 4
start_y_true, end_y_true = y_true[:length-end_i], y_true[length-end_i:]
start_y_pred, end_y_pred = y_pred[:length-end_i], y_pred[length-end_i:]
sum_start_y_true = tf.reduce_sum(tf.reshape(start_y_true, (-1,4)), axis=0)
sum_start_y_pred = tf.reduce_sum(tf.reshape(start_y_pred, (-1,4)), axis=0)
res = tf.reduce_sum(tf.cast(sum_start_y_true, tf.float32)) - tf.reduce_sum(tf.cast(sum_start_y_pred, tf.float32))
res_ending = tf.reduce_sum(tf.cast(end_y_true, tf.float32) - tf.cast(end_y_pred, tf.float32))
return res_ending + res
Your function doesn't make a lot of sense though, you are calculating sums of sums. Can't you just sum everything?
I want my model to increase the loss for a false positive prediction when training by creating a custom loss function.
The class_weight parameter in model.fit() does not work for this issue. The class_weight is already set to { 0: 1, 1:23 } as I have skewed training data where there are 23 times as many non-true labels as there are true labels.
I am not too experienced when working with the keras backend. I have mostly worked with the functional model.
What I want to create is:
def weighted_binary_crossentropy(y_true, y_pred):
#where y_true == 0 and y_pred == 1:
# weight this loss and make it 50 times larger
#return loss
I can do simple stuff with the tensors such as getting the mean squared error but I have no idea how to do logical stuff.
I have tried to do some hacky solution which doesnt work and feels totally wrong:
def weighted_binary_crossentropy(y_true, y_pred):
false_positive_weight = 50
thresh = 0.5
y_pred_true = K.greater_equal(thresh,y_pred)
y_not_true = K.less_equal(thresh,y_true)
false_positive_tensor = K.equal(y_pred_true,y_not_true)
loss_weights = K.ones_like(y_pred) + false_positive_weight*false_positive_tensor
return K.binary_crossentropy(y_true, y_pred)*loss_weights
I am using python 3 with keras 2 and tensorflow as backend.
Thanks in advance!
I think you're almost there...
from keras.losses import binary_crossentropy
def weighted_binary_crossentropy(y_true, y_pred):
false_positive_weight = 50
thresh = 0.5
y_pred_true = K.greater_equal(thresh,y_pred)
y_not_true = K.less_equal(thresh,y_true)
false_positive_tensor = K.equal(y_pred_true,y_not_true)
#changing from here
#first let's transform the bool tensor in numbers - maybe you need float64 depending on your configuration
false_positive_tensor = K.cast(false_positive_tensor,'float32')
#and let's create it's complement (the non false positives)
complement = 1 - false_positive_tensor
#now we're going to separate two groups
falsePosGroupTrue = y_true * false_positive_tensor
falsePosGroupPred = y_pred * false_positive_tensor
nonFalseGroupTrue = y_true * complement
nonFalseGroupPred = y_pred * complement
#let's calculate one crossentropy loss for each group
#(directly from the keras loss functions imported above)
falsePosLoss = binary_crossentropy(falsePosGroupTrue,falsePosGroupPred)
nonFalseLoss = binary_crossentropy(nonFalseGroupTrue,nonFalseGroupPred)
#return them weighted:
return (false_positive_weight*falsePosLoss) + nonFalseLoss
I'm trying to implement a new loss function of my own.
When I tried to debug it (or print in it) I've noticed it is called only once at the model creating section of the code.
How can I know what y_pred and y_true contains (shapes, data etc..) if I cannot run my code into this function while fitting the model?
I wrote this loss function:
def my_loss(y_true, y_pred):
# run over the sequence, jump by 3
# calc the label
# if the label incorrect punish
y_pred = K.reshape(y_pred, (1, 88, 3))
y_pred = K.argmax(y_pred, axis=1)
zero_count = K.sum(K.clip(y_pred, 0, 0))
one_count = K.sum(K.clip(y_pred, 1, 1))
two_count = K.sum(K.clip(y_pred, 2, 2))
zero_punish = 1 - zero_count / K.count_params(y_true)
one_punish = 1- one_count/ K.count_params(y_true)
two_punish = 1- two_count/ K.count_params(y_true)
false_arr = K.not_equal(y_true, y_pred)
mask0 = K.equal(y_true, K.zeros_like(y_pred))
mask0_miss = K.dot(false_arr, mask0) * zero_punish
mask1 = K.equal(y_true, K.ones_like(y_pred))
mask1_miss = K.dot(false_arr, mask1) * one_punish
mask2 = K.equal(y_true, K.zeros_like(y_pred)+2)
mask2_miss = K.dot(false_arr, mask2) * two_punish
return K.sum(mask0_miss) + K.sum(mask1_miss) + K.sum(mask2_miss)
It fails on:
theano.gof.fg.MissingInputError: A variable that is an input to the graph was
neither provided as an input to the function nor given a value. A chain of
variables leading from this input to an output is [/dense_1_target, Shape.0].
This chain may not be unique
Backtrace when the variable is created:
How can I fix it?
You have to understand that Theano is a symbolic language. For example, when we define the following loss function in Keras:
def myLossFn(y_true, y_pred):
return K.mean(K.abs(y_pred - y_true), axis=-1)
Theano is just making a symbolic rule in a computational graph, which would be executed when it gets values i.e. when you train the model with some mini-batches.
As far as your question on how to debug your model goes, you can use theano.function for that. Now, you want to know if your loss calculation is correct. You do the following.
You can implement the python/numpy version of your loss function. Pass two random vectors to your numpy-loss-function and get a number. To verify if theano gives nearly identical result, define something as follows.
import theano
from theano import tensor as T
from keras import backend as K
Y_true = T.frow('Y_true')
Y_pred = T.fcol('Y_pred')
out = K.mean(K.abs(Y_pred - Y_true), axis=-1)
f = theano.function([Y_true, Y_pred], out)
# creating some values
y_true = np.random.random((10,))
y_pred = np.random.random((10,))
numpy_loss_result = np.mean(np.abs(y_true-y_pred))
theano_loss_result = f(y_true, y_pred)
# check if both are close enough
print numpy_loss_result-theano_loss_result # should be less than 1e-5
Basically, theano.function is a way to put values and evaluate those symbolic expressions. I hope this helps.
My classifier produces soft classifications and I wish to select an optimal threshold (that is, one that maximizes accuracy) from the results of the method on the training cases, and use this threshold to produce the hard classification. While in general the problem is relatively easy, I find it hard to optimise the code so that the computation does not last forever. Below you'll find the code that essentially recreates the optimisation procedure on some dummy data. Could you please point me into any direction which could possibly improve performance?
y_pred = np.random.rand(400000)
y_true = np.random.randint(2, size=400000)
accs = [(accuracy_score(y_true, y_pred > t), t) for t in np.unique(y_pred)]
train_acc, train_thresh = max(accs, key=lambda pair: pair[0])
I realize that I could sort both y_pred and y_true prior to the loop, and use that to my advantage when binarizing y_pred but that didn't bring much improvement (unless I did something wrong).
Any help would be much appreciated.
Sort y_pred descendantly and use Kadane's Algorithm to calculate an index i such that the subarray of y_true from 0 to i has maximum sum. Your optimal threshold b is then b = (y_pred[i] + y_pred[i+i]) / 2. This will be the same output that SVM would give you, that is, the hyperplane (or for your 1-dimensional case, a threshold) that maximizes the margin between classes.
I wrote a helper function in python:
def opt_threshold_acc(y_true, y_pred):
A = list(zip(y_true, y_pred))
A = sorted(A, key=lambda x: x[1])
total = len(A)
tp = len([1 for x in A if x[0]==1])
tn = 0
th_acc = []
for x in A:
th = x[1]
if x[0] == 1:
tp -= 1
else:
tn += 1
acc = (tp + tn) / total
th_acc.append((th, acc))
return max(th_acc, key=lambda x: x[1])