I want to minimize two loss functions the mean squared error and the KL Divergence.
It is possible to implement this on Keras
something like
loss =tf.keras.losses.KLDivergence() + tf.keras.losses.MeanSquaredError()
model.compile(optimizer="Adam",
loss=loss
)
This code gives me error, as I can't sum those functions
You could define a custom loss like this -
import tensorflow.keras as K
def custom_loss(y_true,y_pred):
l = K.backend.sum(K.losses.KLDivergence(y_true, y_pred), K.losses.MeanSquaredError(y_true, y_pred))
return l
model.compile(optimizer='adam',
loss=custom_loss,
metrics=['accuracy'])
The reason you can sum those 2 is that you are trying to sum up the objects of the 2 classes. Instead, you need to call them and sum up their results.
Related
I would like to train two Autoencoders jointly and connect their activation layer in the deepest layer.
How can I add all the terms in one loss function?
Assume:
diffLR = Lambda(lambda x: abs(x[0] - x[1]))([model1_act7, model2_act5])
model = Model(inputs=[in1, in2], outputs=[diffLR, model1_conv15, model2_conv10])
model.compile(loss=['MAE', 'mean_squared_error','mean_squared_error'],
optimizer='SGD',
metrics=['mae', rmse])
model.fit([x_train_n, y_train_n], [yM1, x_train_n, y_train_n], batch_size=10, epochs=350, validation_split=0.2, shuffle=True) #, callbacks=[es])
Two networks are convolutional Autoencoders mapping x->x and y->y. Lambda layer connects the latent space of two networks. Target for diffLR is to train the network to the point that two feature spaces represent same distribution. (yM1 is a zero matrix of the same size as latent feature space.)
Now each are optimized separately (or I think they are optimized separately...), I would like to join them in a single loss function like this:
def my_loss(z, x, y, z_pred, x_pred, y_pred):
loss = backend.sqrt(backend.mean(backend.square(x_pred-x))) + backend.sqrt(backend.mean(backend.square(y_pred-y))) + backend.sqrt(backend.mean(backend.square(z_pred-z)))
return loss
model.compile(loss=[my_loss],
optimizer='SGD',
metrics=['mae', rmse])
I get this error:
ValueError: When passing a list as loss, it should have one entry per model outputs. The model has 3 outputs, but you passed loss=[<function my_loss at 0x7fa3d17f2158>]
or
model.compile(loss=my_loss,
optimizer='SGD',
metrics=['mae', rmse])
TypeError: my_loss() missing 4 required positional arguments: 'y', 'z_pred', 'x_pred', and 'y_pred'
Is this possible to do? How can I do this?
So, what you are doing is performing RootMeanSquareError on each of your n=3 output followed by a weighted sum (same weight in your case).
As the Error message says clearly:
ValueError: When passing a list as loss, it should have one entry per
model outputs. The model has 3 outputs, but you passed....
By passing a list of 3 loss function (might be same or different) while compiling your model You can do the same thing what you are doing in your custom loss function. Additionally, you can also define the weight for each individual loss by passing loss_weights argument value. moreover, you can do something following:
def my_loss(y_true, y_pred):
return backend.sqrt(K.mean(K.square(y_pred - y_true)))
model.compile(loss=[my_loss, my_loss, my_loss], # you can pass 3 different (custom) loss function as well
loss_weight=[1.0, 1.0, 1.0], # Default value is 1
optimizer='SGD',
metrics=['mae', rmse])
I have a fully connected feed-forward network implemented with Keras. Initially, I used binary cross-entropy as the loss and the metric, and Adam optimizer as follows
adam = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['binary_crossentropy'])
This model trains well and gives good results. In order to get better results I want to use a different loss function and metric as below,
import keras.backend as K
def soft_bit_error_loss(yTrue, yPred):
loss = K.pow(1 - yPred, yTrue) * K.pow(yPred, 1-yTrue)
return K.mean(loss)
def ber(yTrue, yPred):
x_hat_train = K.cast(K.greater(yPred, 0.5), 'uint8')
train_errors = K.cast(K.not_equal(K.cast(yTrue, 'uint8'), x_hat_train), 'float32')
train_ber = K.mean(train_errors)
return train_ber
I use it to compile my model as below
model.compile(optimizer=adam, loss=soft_bit_error_loss, metrics=[ber])
However, when I do that, the loss and the metric diverge after some training, everytime as in the following pictures.
What can be the cause of this?
Your loss function is very unstable, look at it:
Where I replaced y_pred (variable) with x and y_true (constant) with c for simplicity.
As your predictions approach zero, at least one operation will tend to 1/0, which is infinite. Although by the limits theory you can know the result is ok, Keras doesn't know the "whole" function as one, it calculates derivatives based on the basic operations used.
So, one easy solution is the one pointed by #today:
loss = K.switch(yTrue == 1, 1 - yPred, yPred)
It's exactly the same function (difference only when c is not zero or 1).
Also, even easier, for c=0 or c=1, it's just a plain loss='mae'.
My understanding is that keras requires loss functions to have the signature:
def custom_loss(y_true, y_pred):
I am trying to use sklearn.metrics.cohen_kappa_score, which takes
(y1, y2, labels=None, weights=None, sample_weight=None)`
If I use it as is:
model.compile(loss=metrics.cohen_kappa_score,
optimizer='adam', metrics=['accuracy'])
Then the weights won't be set. I want to set that to quadtratic. Is there some what to pass this through?
There are two steps in implementing a parameterized custom loss function (cohen_kappa_score) in Keras. Since there are implemented function for your needs, there is no need for you to implement it yourself. However, according to TensorFlow Documentation, sklearn.metrics.cohen_kappa_score does not support weighted matrix.
Therefore, I suggest TensorFlow's implementation of cohen_kappa. However, using TensorFlow in Keras is not that easy...
According to this Question, they used control_dependencies to use a TensorFlow metric in Keras. Here is a example:
import keras.backend as K
def _cohen_kappa(y_true, y_pred, num_classes, weights=None, metrics_collections=None, updates_collections=None, name=None):
kappa, update_op = tf.contrib.metrics.cohen_kappa(y_true, y_pred, num_classes, weights, metrics_collections, updates_collections, name)
K.get_session().run(tf.local_variables_initializer())
with tf.control_dependencies([update_op]):
kappa = tf.identity(kappa)
return kappa
Since Keras loss functions take (y_true, y_pred) as parameters, you need a wrapper function that returns another function. Here is some code:
def cohen_kappa_loss(num_classes, weights=None, metrics_collections=None, updates_collections=None, name=None):
def cohen_kappa(y_true, y_pred):
return -_cohen_kappa(y_true, y_pred, num_classes, weights, metrics_collections, updates_collections, name)
return cohen_kappa
Finally, you can use it as follows in Keras:
# get the loss function and set parameters
model_cohen_kappa = cohen_kappa_loss(num_classes=3,weights=weights)
# compile model
model.compile(loss=model_cohen_kappa,
optimizer='adam', metrics=['accuracy'])
Regarding using the Cohen-Kappa metric as a loss function. In general it is possible to use weighted kappa as a loss function. Here is a paper using weighted kappa as a loss function for multi-class classification.
You can define it as a custom loss and yes you are right that keras accepts only two arguments in the loss function. Here is how you can define your loss:
def get_cohen_kappa(weights=None):
def cohen_kappa_score(y_true, y_pred):
"""
Define your code here. You can now use `weights` directly
in this function
"""
return score
return cohen_kappa_score
Now you can pass this function to your model as:
model.compile(loss=get_cohen_kappa_score(weights=weights),
optimizer='adam')
model.fit(...)
I have a training data:
And, I have a model in Keras with more than one dimension of output. I want to predict A, B and C:
model = Sequential()
model.add(GRU(32, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(3))
model.compile(loss='mean_squared_error', optimizer='adam')
But I want the minimum mean_squared_error in A, i.e. only want to consider A for the loss function.
What I can do?
You can define a custom loss function and only compute the mean_squared_error() loss based on the value of A:
from keras import losses
def loss_A(y_true, y_pred):
return losses.mean_squared_error(y_true[:,0], y_pred[:,0])
#...
model.compile(loss=loss_A, optimizer='adam')
What you need to look into is a custom loss function:
def only_A_mean_squared(y_true, y_pred):
return keras.losses.mean_squared_error(y_true[:,0], y_pred[:,0])
And in order to use it:
model.compile(loss=only_A_mean_squared, optimizer='adam')
What i am doing in the above is creating a custom loss function, which only takes the first dimension (the 'A') and feed it to the normal keras mean squared error loss function.
I am trying to use huber loss in a keras model (writing DQN), but I am getting bad result, I think I am something doing wrong. My is code is below.
model = Sequential()
model.add(Dense(output_dim=64, activation='relu', input_dim=state_dim))
model.add(Dense(output_dim=number_of_actions, activation='linear'))
loss = tf.losses.huber_loss(delta=1.0)
model.compile(loss=loss, opt='sgd')
return model
I came here with the exact same question. The accepted answer uses logcosh which may have similar properties, but it isn't exactly Huber Loss. Here's how I implemented Huber Loss for Keras (note that I'm using Keras from Tensorflow 1.5).
import numpy as np
import tensorflow as tf
'''
' Huber loss.
' https://jaromiru.com/2017/05/27/on-using-huber-loss-in-deep-q-learning/
' https://en.wikipedia.org/wiki/Huber_loss
'''
def huber_loss(y_true, y_pred, clip_delta=1.0):
error = y_true - y_pred
cond = tf.keras.backend.abs(error) < clip_delta
squared_loss = 0.5 * tf.keras.backend.square(error)
linear_loss = clip_delta * (tf.keras.backend.abs(error) - 0.5 * clip_delta)
return tf.where(cond, squared_loss, linear_loss)
'''
' Same as above but returns the mean loss.
'''
def huber_loss_mean(y_true, y_pred, clip_delta=1.0):
return tf.keras.backend.mean(huber_loss(y_true, y_pred, clip_delta))
Depending if you want to reduce the loss or the mean of the loss, use the corresponding function above.
You can wrap Tensorflow's tf.losses.huber_loss in a custom Keras loss function and then pass it to your model.
The reason for the wrapper is that Keras will only pass y_true, y_pred to the loss function, and you likely want to also use some of the many parameters to tf.losses.huber_loss. So, you'll need some kind of closure like:
def get_huber_loss_fn(**huber_loss_kwargs):
def custom_huber_loss(y_true, y_pred):
return tf.losses.huber_loss(y_true, y_pred, **huber_loss_kwargs)
return custom_huber_loss
# Later...
model.compile(
loss=get_huber_loss_fn(delta=0.1)
...
)
I was looking through the losses of keras. Apparently logcosh has same properties as huber loss. More details of their similarity can be seen here.
How about:
loss=tf.keras.losses.Huber(delta=100.0)