My Model:
I built a siamese network that take two input and has three outputs. So My loss functions is :
total loss = alpha( loss1) + alpah( loss2) + (1_alpah) ( loss3)
loss1 and loss2 is categorical cross entropy loss function, to classify the class identity from total of 8 classes.
loss3 is similarity loss function ( euclidean distance loss), to verify if the both input from same class or different classes.
My questions are as follow:
If I have different losses, and I want to weight them by using the variable alpha which its value depend on the epoch number. So I have to set the value pf alpha through callback. My question is it possible to pass this alpha variable that its value changed with epoch number (i.e not scalar) through the loss_weights in model.complie. The documentation said:
loss_weights: Optional list or dictionary specifying scalar
coefficients (Python floats) to weight the loss contributions of
different model outputs. The loss value that will be minimized by the
model will then be the weighted sum of all individual losses, weighted
by the loss_weights coefficients. If a list, it is expected to have a
1:1 mapping to the model's outputs. If a tensor, it is expected to map
output names (strings) to scalar coefficients.
Example
alpha = K.variable(0., dtype=tf.float32)
def changeAlpha(epoch,logs):
new_alpha = some_function(epoch)
K.set_value(alpha, new_alpha)
return
alphaChanger = LambdaCallback(on_epoch_end=changeAlpha)
model.compile(loss= [loss1, loss2, loss3], loss_weights = [alpha, alpha, (1-alpha)])
My dataset is imbalanced, so I want to use class_wights in model.fit(). So for the same model of three losses, I want to apply the class weights only for categorical cross entropy losses ( loss 1 and loss2) , So will it work on both losses and except the third loss if I pass it to model.fit? Knowing that the third loss is custom loss function.
If I want to classify the classes for siamese network, would my metric be model.compile(metrics= ['out1':'accuracy', 'out2':accuracy']])? But the final accuracy need to be the average of both,I can solve it by building my own custom metric. But is there anyway to weighted summing of both metrics.
Related
Is there a way to write up a custom MSE loss function in keras?
My training sample is cross-sectional data of k x n inputs and my outputs are a k x 1 at time t, but t ranges from t-1 to t-120 (monthly time stamps of cross-sectional data).
I want to write up a custom MSE loss function that essentially puts a lower weight on training samples t-120 and a higher weight on training samples t-1.
Is there a way to do this?
Here is some simple code to write up a custom loss function in keras.
def my_loss_fn(y_true, y_pred):
squared_difference = tf.square(y_true - y_pred)
return tf.reduce_mean(squared_difference, axis=-1) # Note the `axis=-1`
model.compile(optimizer='adam', loss=my_loss_fn)
You can use class weights in order to specify a custom weight for each of your output units and then specify it on model.fit() 1
I am currently working with an LSTM sequence to sequence model for time domain signal predictions. Because of domain knowledge, I know that the first part of the prediction (about 20%) can never be predicted correctly, since the information required is not available in the given input sequence. The remaining 80% of the predicted sequence are usually predicted quite well. In order to exclude the first 20% from the training optimization, it would be nice to define a loss function that would basically operate on a given index range like the numpy code below:
start = int(0.2*sequence_length)
stop = sequence_length
def mse(pred, target):
""" Mean squared error between two time series np.arrays """
return 1/target.shape[0]*np.sum((pred-target)**2)
def range_mse_loss(y_pred, y):
return mse(y_pred[start:stop],y[start:stop])
How do I have to write this loss function in order to have it work with my preexisting keras code, where loss is simply given by model.compile(loss='mse') ?
You can slice your tensor to just last 80% of the data.
size = int(y_true.shape[0] * 0.8) # for 2D vector, e.g., (100, 1)
loss_fn = tf.keras.losses.MeanSquaredError(name='mse')
loss_fn(y_pred[:-size], y_true[:-size])
You can also use the sample_weights at the tf.keras.losses.MeanSquaredError(), passing an array of weights and the first 20% of weights is zero
size = int(y_true.shape[0] * 0.8) # for 2D vector, e.g., (100, 1)
zeros = tf.zeros((y_true.shape[0] - size), dtype=tf.int32)
ones = tf.ones((size), dtype=tf.int32)
weights = tf.concat([zeros, ones], 0)
loss_fn = tf.keras.losses.MeanSquaredError(name='mse')
loss_fn(y_pred, y_true, sample_weights=weights)
There is a warming of the second solution, the final loss will be lower than the first solution, because you are putting zero in the first predictions values, but you aren't removing them in the formula MSE = 1 /n * sum((y-y_hat)^2).
One workaround would be to mark the observations as None/nan and then overwrite the train_step method. Following tensorflow's tutorial about customizing train_step, you would do something like this
#tf.function
def train_step(keras_model, data):
print('custom train_step')
# Unpack the data. Its structure depends on your model and
# on what you pass to `fit()`.
x, y = data
with tf.GradientTape() as tape:
y_pred = keras_model(x, training=True) # Forward pass
# masking nan values in observations, also assuming that targets are >0.0
mask = tf.greater(y, 0.0)
true_y = tf.boolean_mask(y, mask)
pred_y = tf.boolean_mask(y_pred, mask)
# Compute the loss value
# (the loss function is configured in `compile()`)
loss = keras_model.compiled_loss(true_y, pred_y, regularization_losses=keras_model.losses)
# Compute gradients
trainable_vars = keras_model.trainable_variables
gradients = tape.gradient(loss, trainable_vars)
# Update weights
keras_model.optimizer.apply_gradients(zip(gradients, trainable_vars))
# Update metrics (includes the metric that tracks the loss)
keras_model.compiled_metrics.update_state(true_y, pred_y)
# Return a dict mapping metric names to current value
return {m.name: m.result() for m in keras_model.metrics}
This will work for all the performance metrics you are tracking. Alternative way would be to mask the nans inside the loss function but that would be limited to only one loss function and not any other loss function/performance metrics.
I am developing a Keras model. My dataset is badly unbalanced, so I want to set a threshold for training and testing. If I'm not mistaken, when doing a backward propagation, neural network checks the predicted values with the original ones and calculate the error and based on the error, set new weights for neurons.
As I know, Keras uses 0.5 for the threshold. I know there are ways to apply custom metrics (as recall and precision) with custom threshold, but that threshold is only used for calculating the recall, and it is not applied in the loss function. To be more clear, If I want to set 0.85 as my threshold, the neural network would use 0.5 as threshold to calculate loss and 0.85 for recall.
Is there any ways to set this threshold for training as well?
There is no such a thing as a threshold for loss.
A loss function must be "differentiable", thus it must be a "continuous" function.
The best you can do is to set "class weights", such as these examples: Higher loss penalty for true non-zero predictions
In addition to class weights...
you can use metric function with the threshold parameter:
model.compile(..., metrics=[tf.keras.metrics.BinaryAccuracy(threshold=0.5)])
you can use sigmoid activation in the last layer and select after that threshold manually:
pred_labels = np.where(y_pred>0.5, 1, 0)
score = sklearn.metrics.accuracy_score(pred_labels, labels)
I don't understand: since that the model is evaluated on a group of images, not a single image, so I think the score should return the average value of loss or metrics over the group of images.
score = model.evaluate_generator(evaluateGene,test_images, verbose=1)
This is a neural network model based on Keras, to evaluate the performance of the model, we need to calculate the average loss and metrics and their standard deviation.
score = model.evaluate_generator(evaluateGene,test_images, verbose=1)
print('%.3f' %score[0], '%.3f' %score[1],'%.3f' %score[2])
Since I want to calculate the mean loss, metrics and standard deviation. But this function seems can't do that. Are there some good solutions to return a mean value and std? Thanks a lot!
I defined a custom loss function in Keras (tensorflow backend) that is comprised of reconstruction MSE and the kullback leibler divergence between the learned probability distribution and a standard normal distribution. (It is for a variational autoencoder.)
I want to be able to slowly increase how much the cost is affected by the KL divergence term during training, with a weight called "reg", starting at reg=0.0 and increasing until it gets to 1.0. I would like the rate of increase to be tuned as a hyperparameter.(As of now, I just have the "reg" parameter set constant at 0.5.)
Is there functionality in Keras to do this?
def vae_loss(y_true,y_pred):
reg = 0.5
# Average cosine distance for all words in a sequence
reconstruction_loss = tf.reduce_mean(mean_squared_error(y_true, y_pred),1)
# Second part of the loss ensures the z probability distribution doesn't stray too far from normal
KL_divergence_loss = tf.reduce_mean(tf.log(z_sigma) + tf.div((1 + tf.square(z_mu)),2*tf.square(z_sigma)) - 0.5,1)
loss = reconstruction_loss + tf.multiply(reg,KL_divergence_loss)
return loss