I want to create a custom loss which gets the output of the net and multiple arguments from a data generator.
I found this article, which describes how to calculate one loss from multiple layers with one label. But I want to calculate the loss from a single layer with multiple labels using the fit_generator. My problem is that Keras expects the output and the label to be of the same shape.
example:
Regular custom loss:
def custom_loss(y_pred, y_label):
return K.mean(y_pred - y_label)
An example for the type of custom loss I want to use:
def custom_loss(y_pred, y_label, y_weights):
loss = K.mean(y_pred - y_label)
return tf.compat.v1.losses.compute_weighted_loss(loss, y_weights)
This is just an example my original code is a little more complicated. I just want to be able to give the loss function two parameters (y_label and y_weights) instead of only one (y_label).
Does anyone know how to solve this problem?
I am not sure what exactly you are asking but maybe you can use this. You can try something like a custom function that returns a loss function.
def custom_loss(y_weights):
# Create a loss function that calculates what you want
def example_loss(y_true,y_pred):
loss = K.mean(y_pred - y_label)
return tf.compat.v1.losses.compute_weighted_loss(loss, y_weights)
# Return a function
return example_loss
# Compile the model
model.compile(optimizer='adam',
loss=custom_loss(y_weights), # Call the loss function with the preferred weights
metrics=['accuracy'])
You can also take a look at this question
Related
I have a model with multiple outputs like:
model_in = Input((None, None, 3))
out_1 = Dense(10)(model_in)
out_2 = Dense(20)(model_in)
model = keras.Model(model_in, [out_1, out_2])
model.compile('adam', loss = [loss_fn_1, loss_fn_2])
And for each training sample, I have a mask which should be used in my custom loss function (actually my data generator produces this mask based on input sample). So for each sample I have one input, two outputs and a mask that should be used in loss function. My question is how should I pass the mask to appropriate loss function (I use different loss function for each output)?
Your loss function has three arguments, so it can not be used with keras compile(), fit() (https://keras.io/api/losses/).
You have two options:
create a custom loop or
create a custom layer and use add_loss().
So I implemented a neural network with this code:
self.model = keras.Sequential()
self.model.add(keras.Input(shape=(self.wejscia,), name="Input"))
self.model.add(layers.Dense(64, activation="relu", name="dense_1"))
self.model.add(layers.Dense(64, activation="relu", name="dense_2"))
self.model.add(layers.Dense(8, activation="softmax", name="predictions"))
But I wanted to make it possible to perform gradient descent on only one, chosen position of the output vector. The way i did it was like this:
First I created a class like that:
class CustomMSE(keras.losses.Loss):
def __init__(self, my_output, name="custom_mse"):
super().__init__(name=name)
self.my_output = my_output
def call(self, y_true, y_pred):
mse = tf.math.reduce_mean(tf.square(y_true[0,self.my_output] - y_pred[0,self.my_output]))
return mse
and then I just applied compile method like that:
self.model.compile(optimizer=keras.optimizers.Adam(), loss=CustomMSE(i))
I am not sure of two things.
First: will the .fit method modify the wages between the second hidden layer and the j-th output for j !=i (I hope it won't)
Second: will the instruction self.model.compile(optimizer=keras.optimizers.Adam(), loss=CustomMSE(i)) applied many times for different values of i affect the current wages of the model, or will it just change the further behavior of the network after aplying the .fit method?
With the code you have, it will not work as expected, as you are using tf. functions rather than keras.backend functions to create loss functions. Here is an example of how you can create a custom loss function:
import tensorflow.keras.backend as kb
def custom_loss(y_actual,y_pred):
custom_loss=kb.square(y_actual-y_pred)
return custom_loss
You can use this loss function like this:
model.compile(loss=custom_loss,optimizer=optimizer)
Of course, this is not the same loss function you implemented, but it shows the methodology.
I want to develop a neural network with three inputs pos,anc,neg and three outputs pos_out,anc_out,neg_out. While calculating loss in my customized loss function in keras, I want to access pos_out, anc_out, neg_out in y_pred. I can access y_pred as a whole. But how to access individual part pos_out, anc_out and neg_out
I have applied max function to y_pred. It calculates max value correctly. If I am passing only one output in Model as Model(input=[pos,anc,neg], output=pos_out) then also it calculates max value correctly. But when it comes to accessing max values form pos_out, anc_out and neg_out separately in customized function, it does not work.
def testmodel(input_shape):
pos = Input(shape=(14,300))
anc = Input(shape=(14,300))
neg = Input(shape=(14,300))
model = Sequential()
model.add(Flatten(batch_input_shape=(1,14,300)))
pos_out = model(pos)
anc_out = model(anc)
neg_out = model(neg)
model = Model(input=[pos,anc,neg], output=[pos_out,anc_out,neg_out])
return model
def customloss(y_true,y_pred):
print((K.int_shape(y_pred)[1]))
#loss = K.max((y_pred))
loss = K.max[pos_out]
return loss
You can create a loss function that contains a closure that lets you access the model and thus the targets and the model layer outputs.
class ExampleCustomLoss(object):
""" The loss function can access model.inputs, model.targets and the outputs
of specific layers. These are all tensors and will have the expected results
for the batch.
"""
def __init__(self, model):
self.model = model
def loss(self, y_true, y_pred, **kwargs):
...
return loss
model = Model(..., ...)
loss_calculator = ExampleCustomLoss(model)
model.compile('adam', loss_calculator.loss)
However, it may be simpler to do the inverse. i.e. have a single model output
out = Concatenate(axis=1)([pos_out, anc_out, neg_out])
And then in the loss function slice y_true and y_pred.
From the names of variables, it looks as if you are trying to use a triplet loss. You may find this other question useful:
How to deal with triplet loss when at time of input i have only two files i.e. at time of testing
Your loss function gets 2 arguments, model output and true label, your model output will have the shape that you define when you define the net. Your loss function needs to output a single difference value, between your model's output and the true value of the label while training.
Also please add some trainable layers to your model, because your custom loss function will be useless otherwise.
I have a "How can I do that" question with keras :
Assuming that I have a first neural network, say NNa which has 4 inputs (x,y,z,t) which is already trained.
If I have a second neural network, say NNb, and that its loss function depends on the first neural network.
The custom loss function of NNb customLossNNb calls the prediction of NNa with a fixed grid (x,y,z) and just modify the last variable t.
Here in pseudo-python-code what I would like to do to traine the second NN : NNb:
grid=np.mgrid[0:10:1,0:10:1,0:10:1].reshape(3,-1).T
Y[:,0]=time
Y[:,1]=something
def customLossNNb(NNa,grid):
def diff(y_true,y_pred):
for ii in range(y_true.shape[0]):
currentInput=concatenation of grid and y_true[ii,0]
toto[ii,:]=NNa.predict(currentInput)
#some stuff with toto
return #...
return diff
Then
NNb.compile(loss=customLossNNb(NNa,K.variable(grid)),optimizer='Adam')
NNb.fit(input,Y)
In fact the line that cause me troubles is currentInput=concatenation of grid and y_true[ii,0]
I tried to send to customLossNNb the grid as a tensor with K.variable(grid). But I can't defined a new tensor inside the loss function, something like CurrentY which has a shape (grid.shape[0],1) fill with y[ii,0](i.e. the current t) and then concatenate grid and currentY to build currentInput
Any ideas?
Thanks
You can include your custom loss function into the graph using functional API of keras. The model in this case can be used as a function, something like this:
for l in NNa.layers:
l.trainable=False
x=Input(size)
y=NNb(x)
z=NNa(y)
Predict method will not work, since loss function should be part of the graph, and predict method returns np.array
First, make NNa untrainable. Notice that you should do this recursively if your model has inner models.
def makeUntrainable(layer):
layer.trainable = False
if hasattr(layer, 'layers'):
for l in layer.layers:
makeUntrainable(l)
makeUntrainable(NNa)
Then you have two options:
Attach NNa to the end of your model (notice that both y_true and y_pred will be changed)
Then change your targets (predict with NNa) for correct results since your model is now expecting the output of NNa, not NNb.
Create a custom loss function that uses NNa inside it, without changing your targets
Option 1 - Attaching models
inputs = NNb.inputs
outputs = NNa(NNb.outputs) #make sure NNb is outputing 4 tensors to match NNa inputs
fullModel = Model(inputs,outputs)
#changing the targets:
newY_train = NNa.predict(oldY_train)
Option 2 - Creating a custom loss
Warning: please test whether NNa's weights are really frozen while training this configuration
from keras.losses import binary_crossentropy
def customLoss(true,pred):
true = NNa(true)
pred = NNa(pred)
#use some of the usual losses or create your own
binary_crossentropy(true,pred)
NNb.compile(optimizer=anything, loss = customLoss)
I have a VAE model that I've broken down into the encoder and decoder parts, and implemented a custom loss. A simplified example is as below
input = Input(shape=(self.image_height, self.image_width, self.image_channel))
encoded = build_encoder(input)
decoded = build_decoder(encoded)
model = Model(input, decoded)
The loss (simplified) is
loss = K.mean(decoded[0] + decoded[1] + encoded[0]**2)
model.add_loss(loss)
model.compile(optimizer=self.optimizer)
My main problem is that I want to use Keras' modelcheckpoint function, which would then require me to set custom metrics. However, everything I have seen online is similar to https://keras.io/metrics/#custom_metrics. This only takes in y_true and y_pred, and modify the validation loss from there. How would I implement it in my example model, where the loss is calculated from multiple inputs, not only the final output of "decoded"?
Well apparently you can still use the variables (keras layers) without passing those into the custom loss function.
So for my example, the loss can be calculated as
def custom_loss(y_true, y_pred):
return K.mean(decoded[0] + decoded[1] + encoded[0]**2)
model.compile(optimizer=self.optimizer, loss=custom_loss)
y_true and y_pred is never used, but the actual required inputs can still be called (as long as they are in the same scope as the custom loss function of course).