I have a training data:
And, I have a model in Keras with more than one dimension of output. I want to predict A, B and C:
model = Sequential()
model.add(GRU(32, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(3))
model.compile(loss='mean_squared_error', optimizer='adam')
But I want the minimum mean_squared_error in A, i.e. only want to consider A for the loss function.
What I can do?
You can define a custom loss function and only compute the mean_squared_error() loss based on the value of A:
from keras import losses
def loss_A(y_true, y_pred):
return losses.mean_squared_error(y_true[:,0], y_pred[:,0])
#...
model.compile(loss=loss_A, optimizer='adam')
What you need to look into is a custom loss function:
def only_A_mean_squared(y_true, y_pred):
return keras.losses.mean_squared_error(y_true[:,0], y_pred[:,0])
And in order to use it:
model.compile(loss=only_A_mean_squared, optimizer='adam')
What i am doing in the above is creating a custom loss function, which only takes the first dimension (the 'A') and feed it to the normal keras mean squared error loss function.
Related
I am trying to understand the calculation of binary_crossentropy when used as the loss for a network that outputs a 2 probabilities rather than just 1. Basically I wanted to reproduce the calculation that keras/tf is doing in this case rather than in the common case where the network outputs a single value (the logit of the probability of positive classification). Here's some minimal reproducer code:
from tensorflow import keras
import numpy as np
loss_func = keras.losses.BinaryCrossentropy()
nn = keras.Sequential([
keras.layers.Dense(2**8, input_shape=(1,), activation='relu'),
keras.layers.Dense(2, activation='softmax')
])
nn.compile(loss=loss_func,optimizer='adam')
train_x = np.array([0.4,0.7,0.3,0.2])
train_y = np.array([[0,1],[1,0],[0,1],[0,1]])
print("Evaluted loss = ",nn.evaluate(train_x,train_y))
print("Function loss = ",loss_func(train_y,nn.predict(train_x)).numpy())
print("Manual loss = ",np.average( -train_y*np.log(nn.predict(train_x)) -(1-train_y)*np.log(1. - nn.predict(train_x)) ))
This outputs:
Evaluted loss = 0.6944893002510071
Function loss = 0.6959093
Manual loss = 0.6959095224738121
So there's a difference between the loss calculated by the evaluate method vs using the loss as a function or even calculating the loss by hand. I note that if I swap to using keras.losses.CategoricalCrossentropy() then all three calculations agree. I also note that if I use network with a single logit output then everything also agrees, i.e. if I do the following:
loss_func = keras.losses.BinaryCrossentropy(from_logits=True)
nn = keras.Sequential([
keras.layers.Dense(2**8, input_shape=(1,), activation='relu'),
keras.layers.Dense(1)
])
nn.compile(loss=loss_func,optimizer='adam')
train_x = np.array([0.4,0.7,0.3,0.2])
train_y = np.array([[1.],[0.],[1.],[1.]])
print("Evaluted loss = ",nn.evaluate(train_x,train_y))
print("Function loss = ",loss_func(train_y,nn.predict(train_x)).numpy())
print("Manual loss = ",np.average( -train_y*np.log(1./(1+np.exp(-nn.predict(train_x)))) -(1-train_y)*np.log(1. - 1./(1+np.exp(-nn.predict(train_x)))) ))
gives:
Evaluted loss = 0.6919926404953003
Function loss = 0.69199264
Manual loss = 0.6919926702976227
So my question is: What is the calculation that evaluate is doing on the first network with the 2 probabilities being output and why is it different to the value calculated using the loss function as a standalone function or doing the calculation by hand?
Thanks!
I want to minimize two loss functions the mean squared error and the KL Divergence.
It is possible to implement this on Keras
something like
loss =tf.keras.losses.KLDivergence() + tf.keras.losses.MeanSquaredError()
model.compile(optimizer="Adam",
loss=loss
)
This code gives me error, as I can't sum those functions
You could define a custom loss like this -
import tensorflow.keras as K
def custom_loss(y_true,y_pred):
l = K.backend.sum(K.losses.KLDivergence(y_true, y_pred), K.losses.MeanSquaredError(y_true, y_pred))
return l
model.compile(optimizer='adam',
loss=custom_loss,
metrics=['accuracy'])
The reason you can sum those 2 is that you are trying to sum up the objects of the 2 classes. Instead, you need to call them and sum up their results.
I would like to train two Autoencoders jointly and connect their activation layer in the deepest layer.
How can I add all the terms in one loss function?
Assume:
diffLR = Lambda(lambda x: abs(x[0] - x[1]))([model1_act7, model2_act5])
model = Model(inputs=[in1, in2], outputs=[diffLR, model1_conv15, model2_conv10])
model.compile(loss=['MAE', 'mean_squared_error','mean_squared_error'],
optimizer='SGD',
metrics=['mae', rmse])
model.fit([x_train_n, y_train_n], [yM1, x_train_n, y_train_n], batch_size=10, epochs=350, validation_split=0.2, shuffle=True) #, callbacks=[es])
Two networks are convolutional Autoencoders mapping x->x and y->y. Lambda layer connects the latent space of two networks. Target for diffLR is to train the network to the point that two feature spaces represent same distribution. (yM1 is a zero matrix of the same size as latent feature space.)
Now each are optimized separately (or I think they are optimized separately...), I would like to join them in a single loss function like this:
def my_loss(z, x, y, z_pred, x_pred, y_pred):
loss = backend.sqrt(backend.mean(backend.square(x_pred-x))) + backend.sqrt(backend.mean(backend.square(y_pred-y))) + backend.sqrt(backend.mean(backend.square(z_pred-z)))
return loss
model.compile(loss=[my_loss],
optimizer='SGD',
metrics=['mae', rmse])
I get this error:
ValueError: When passing a list as loss, it should have one entry per model outputs. The model has 3 outputs, but you passed loss=[<function my_loss at 0x7fa3d17f2158>]
or
model.compile(loss=my_loss,
optimizer='SGD',
metrics=['mae', rmse])
TypeError: my_loss() missing 4 required positional arguments: 'y', 'z_pred', 'x_pred', and 'y_pred'
Is this possible to do? How can I do this?
So, what you are doing is performing RootMeanSquareError on each of your n=3 output followed by a weighted sum (same weight in your case).
As the Error message says clearly:
ValueError: When passing a list as loss, it should have one entry per
model outputs. The model has 3 outputs, but you passed....
By passing a list of 3 loss function (might be same or different) while compiling your model You can do the same thing what you are doing in your custom loss function. Additionally, you can also define the weight for each individual loss by passing loss_weights argument value. moreover, you can do something following:
def my_loss(y_true, y_pred):
return backend.sqrt(K.mean(K.square(y_pred - y_true)))
model.compile(loss=[my_loss, my_loss, my_loss], # you can pass 3 different (custom) loss function as well
loss_weight=[1.0, 1.0, 1.0], # Default value is 1
optimizer='SGD',
metrics=['mae', rmse])
My understanding is that keras requires loss functions to have the signature:
def custom_loss(y_true, y_pred):
I am trying to use sklearn.metrics.cohen_kappa_score, which takes
(y1, y2, labels=None, weights=None, sample_weight=None)`
If I use it as is:
model.compile(loss=metrics.cohen_kappa_score,
optimizer='adam', metrics=['accuracy'])
Then the weights won't be set. I want to set that to quadtratic. Is there some what to pass this through?
There are two steps in implementing a parameterized custom loss function (cohen_kappa_score) in Keras. Since there are implemented function for your needs, there is no need for you to implement it yourself. However, according to TensorFlow Documentation, sklearn.metrics.cohen_kappa_score does not support weighted matrix.
Therefore, I suggest TensorFlow's implementation of cohen_kappa. However, using TensorFlow in Keras is not that easy...
According to this Question, they used control_dependencies to use a TensorFlow metric in Keras. Here is a example:
import keras.backend as K
def _cohen_kappa(y_true, y_pred, num_classes, weights=None, metrics_collections=None, updates_collections=None, name=None):
kappa, update_op = tf.contrib.metrics.cohen_kappa(y_true, y_pred, num_classes, weights, metrics_collections, updates_collections, name)
K.get_session().run(tf.local_variables_initializer())
with tf.control_dependencies([update_op]):
kappa = tf.identity(kappa)
return kappa
Since Keras loss functions take (y_true, y_pred) as parameters, you need a wrapper function that returns another function. Here is some code:
def cohen_kappa_loss(num_classes, weights=None, metrics_collections=None, updates_collections=None, name=None):
def cohen_kappa(y_true, y_pred):
return -_cohen_kappa(y_true, y_pred, num_classes, weights, metrics_collections, updates_collections, name)
return cohen_kappa
Finally, you can use it as follows in Keras:
# get the loss function and set parameters
model_cohen_kappa = cohen_kappa_loss(num_classes=3,weights=weights)
# compile model
model.compile(loss=model_cohen_kappa,
optimizer='adam', metrics=['accuracy'])
Regarding using the Cohen-Kappa metric as a loss function. In general it is possible to use weighted kappa as a loss function. Here is a paper using weighted kappa as a loss function for multi-class classification.
You can define it as a custom loss and yes you are right that keras accepts only two arguments in the loss function. Here is how you can define your loss:
def get_cohen_kappa(weights=None):
def cohen_kappa_score(y_true, y_pred):
"""
Define your code here. You can now use `weights` directly
in this function
"""
return score
return cohen_kappa_score
Now you can pass this function to your model as:
model.compile(loss=get_cohen_kappa_score(weights=weights),
optimizer='adam')
model.fit(...)
I am trying to use huber loss in a keras model (writing DQN), but I am getting bad result, I think I am something doing wrong. My is code is below.
model = Sequential()
model.add(Dense(output_dim=64, activation='relu', input_dim=state_dim))
model.add(Dense(output_dim=number_of_actions, activation='linear'))
loss = tf.losses.huber_loss(delta=1.0)
model.compile(loss=loss, opt='sgd')
return model
I came here with the exact same question. The accepted answer uses logcosh which may have similar properties, but it isn't exactly Huber Loss. Here's how I implemented Huber Loss for Keras (note that I'm using Keras from Tensorflow 1.5).
import numpy as np
import tensorflow as tf
'''
' Huber loss.
' https://jaromiru.com/2017/05/27/on-using-huber-loss-in-deep-q-learning/
' https://en.wikipedia.org/wiki/Huber_loss
'''
def huber_loss(y_true, y_pred, clip_delta=1.0):
error = y_true - y_pred
cond = tf.keras.backend.abs(error) < clip_delta
squared_loss = 0.5 * tf.keras.backend.square(error)
linear_loss = clip_delta * (tf.keras.backend.abs(error) - 0.5 * clip_delta)
return tf.where(cond, squared_loss, linear_loss)
'''
' Same as above but returns the mean loss.
'''
def huber_loss_mean(y_true, y_pred, clip_delta=1.0):
return tf.keras.backend.mean(huber_loss(y_true, y_pred, clip_delta))
Depending if you want to reduce the loss or the mean of the loss, use the corresponding function above.
You can wrap Tensorflow's tf.losses.huber_loss in a custom Keras loss function and then pass it to your model.
The reason for the wrapper is that Keras will only pass y_true, y_pred to the loss function, and you likely want to also use some of the many parameters to tf.losses.huber_loss. So, you'll need some kind of closure like:
def get_huber_loss_fn(**huber_loss_kwargs):
def custom_huber_loss(y_true, y_pred):
return tf.losses.huber_loss(y_true, y_pred, **huber_loss_kwargs)
return custom_huber_loss
# Later...
model.compile(
loss=get_huber_loss_fn(delta=0.1)
...
)
I was looking through the losses of keras. Apparently logcosh has same properties as huber loss. More details of their similarity can be seen here.
How about:
loss=tf.keras.losses.Huber(delta=100.0)