I would like to train two Autoencoders jointly and connect their activation layer in the deepest layer.
How can I add all the terms in one loss function?
Assume:
diffLR = Lambda(lambda x: abs(x[0] - x[1]))([model1_act7, model2_act5])
model = Model(inputs=[in1, in2], outputs=[diffLR, model1_conv15, model2_conv10])
model.compile(loss=['MAE', 'mean_squared_error','mean_squared_error'],
optimizer='SGD',
metrics=['mae', rmse])
model.fit([x_train_n, y_train_n], [yM1, x_train_n, y_train_n], batch_size=10, epochs=350, validation_split=0.2, shuffle=True) #, callbacks=[es])
Two networks are convolutional Autoencoders mapping x->x and y->y. Lambda layer connects the latent space of two networks. Target for diffLR is to train the network to the point that two feature spaces represent same distribution. (yM1 is a zero matrix of the same size as latent feature space.)
Now each are optimized separately (or I think they are optimized separately...), I would like to join them in a single loss function like this:
def my_loss(z, x, y, z_pred, x_pred, y_pred):
loss = backend.sqrt(backend.mean(backend.square(x_pred-x))) + backend.sqrt(backend.mean(backend.square(y_pred-y))) + backend.sqrt(backend.mean(backend.square(z_pred-z)))
return loss
model.compile(loss=[my_loss],
optimizer='SGD',
metrics=['mae', rmse])
I get this error:
ValueError: When passing a list as loss, it should have one entry per model outputs. The model has 3 outputs, but you passed loss=[<function my_loss at 0x7fa3d17f2158>]
or
model.compile(loss=my_loss,
optimizer='SGD',
metrics=['mae', rmse])
TypeError: my_loss() missing 4 required positional arguments: 'y', 'z_pred', 'x_pred', and 'y_pred'
Is this possible to do? How can I do this?
So, what you are doing is performing RootMeanSquareError on each of your n=3 output followed by a weighted sum (same weight in your case).
As the Error message says clearly:
ValueError: When passing a list as loss, it should have one entry per
model outputs. The model has 3 outputs, but you passed....
By passing a list of 3 loss function (might be same or different) while compiling your model You can do the same thing what you are doing in your custom loss function. Additionally, you can also define the weight for each individual loss by passing loss_weights argument value. moreover, you can do something following:
def my_loss(y_true, y_pred):
return backend.sqrt(K.mean(K.square(y_pred - y_true)))
model.compile(loss=[my_loss, my_loss, my_loss], # you can pass 3 different (custom) loss function as well
loss_weight=[1.0, 1.0, 1.0], # Default value is 1
optimizer='SGD',
metrics=['mae', rmse])
Related
Say I have a function F that takes in a parameter vector P (say, a 5-element vector), and produces a (numerical) time series Y[t] of length T (eg T=100, so t=1,...,100). The function could be complicated (eg enzyme reaction models)
I want to make a neural network that predicts the output (Y[t]) that would result from feeding a new parameter set (P') into the function. How can this be done?
A simple feed-forward network can work, but it requires a very large number of output nodes, and doesn't take into account the temporal correlation / relationships between points. Is it possible/better to use a RNN or Transformer instead?
Using RNN might work for you. Here is some example code in Keras to get you started:
param_length = 5
time_length = 100
hidden_size = 20
model = tf.keras.Sequential([
# Encode input parameters.
tf.keras.layers.Dense(hidden_size, input_shape=[param_length]),
# Generate a sequence.
tf.keras.layers.RepeatVector(time_length),
tf.keras.layers.LSTM(32, return_sequences=True),
tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(1))
])
model.compile(loss="mse", optimizer="nadam")
model.fit(train_x, train_y, validation_data=(val_x, val_y), epochs=10)
The first Dense layer converts input parameters to a hidden state. Then LSTM RNN units generate time sequences. You will need to experiment with hyperparameters like the number of dense and LTSM layers, the size of hidden layers etc.
One more thing you can try is to use different loss function like:
early_stopping_cb = tf.keras.callbacks.EarlyStopping(
monitor="val_mae", patience=50, restore_best_weights=True)
model.compile(loss=tf.keras.losses.Huber(), optimizer="nadam", metrics=["mae"])
history = model.fit(train_x, train_y, validation_data=(val_x, val_y), epochs=500,
callbacks=[early_stopping_cb])
I have a fully connected feed-forward network implemented with Keras. Initially, I used binary cross-entropy as the loss and the metric, and Adam optimizer as follows
adam = keras.optimizers.Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
model.compile(optimizer=adam, loss='binary_crossentropy', metrics=['binary_crossentropy'])
This model trains well and gives good results. In order to get better results I want to use a different loss function and metric as below,
import keras.backend as K
def soft_bit_error_loss(yTrue, yPred):
loss = K.pow(1 - yPred, yTrue) * K.pow(yPred, 1-yTrue)
return K.mean(loss)
def ber(yTrue, yPred):
x_hat_train = K.cast(K.greater(yPred, 0.5), 'uint8')
train_errors = K.cast(K.not_equal(K.cast(yTrue, 'uint8'), x_hat_train), 'float32')
train_ber = K.mean(train_errors)
return train_ber
I use it to compile my model as below
model.compile(optimizer=adam, loss=soft_bit_error_loss, metrics=[ber])
However, when I do that, the loss and the metric diverge after some training, everytime as in the following pictures.
What can be the cause of this?
Your loss function is very unstable, look at it:
Where I replaced y_pred (variable) with x and y_true (constant) with c for simplicity.
As your predictions approach zero, at least one operation will tend to 1/0, which is infinite. Although by the limits theory you can know the result is ok, Keras doesn't know the "whole" function as one, it calculates derivatives based on the basic operations used.
So, one easy solution is the one pointed by #today:
loss = K.switch(yTrue == 1, 1 - yPred, yPred)
It's exactly the same function (difference only when c is not zero or 1).
Also, even easier, for c=0 or c=1, it's just a plain loss='mae'.
With Reference to the concept of Capsule Network, I am trying to extract just the classification module from the Intel's Implementation of Capsules in Keras, As I don't need the decoder or reconstruction part that is provided in the mentioned link.
My Try:
So I just commented out the decoder part of the network
#dec = Dense(512, activation='relu')(mask)
#dec = Dense(1024, activation='relu')(dec)
#dec = Dense(784, activation='sigmoid')(dec)
#dec = Reshape(input_shape)(dec)
and the decoder argument in the following line
#model = Model([x, mask_input], [output_capsule, dec])
model = Model([x, mask_input], [output_capsule])
model.compile(optimizer='adam', loss=[ margin_loss, 'mae' ], metrics=[ margin_loss, 'mae', 'accuracy'])
model.fit([X, Y], [Y, X], batch_size=128, epochs=3, validation_split=0.2)
Error
I am getting the following error.
ValueError: When passing a list as loss, it should have one entry per model outputs. The model has 1 outputs, but you passed loss=[<function margin_loss at 0x0000020C3E7A30D0>, 'mae']
Help Required:
Can somebody guide me how can I use that only the classification part of the Module because I have images that have dimension 90 x 90 and I want to use the classification part to check the accuracy and later on I would analyze each capsules?
To answer why you are getting that error:
The original model has a two outputs. Each output has its own loss function. In this case, the outputs are [output_capsule, dec] and the corresponding loss functions are [margin_loss, 'mae']. Because you removed the dec output, you need to remove its loss function from the compilation of the model.
In addition, you will need to make sure that you are passing the correct inputs and outputs everywhere else. In the code that you have here, that matters for model.fit. The first argument is the inputs. You still have two inputs to your model (not sure if you want two inputs, but that's a different problem), so passing in a list of two inputs to fit is good. However, the second argument is the desired outputs. You're currently passing in two output arrays, but you only have one output for your model. Since you removed your model's second output, you should remove the second output that you pass in to fit, i.e. X.
Modified code:
model = Model([x, mask_input], [output_capsule])
model.compile(optimizer='adam', loss=[margin_loss], metrics=[margin_loss, 'mae', 'accuracy'])
model.fit([X, Y], [Y], batch_size=128, epochs=3, validation_split=0.2)
I have a training data:
And, I have a model in Keras with more than one dimension of output. I want to predict A, B and C:
model = Sequential()
model.add(GRU(32, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(3))
model.compile(loss='mean_squared_error', optimizer='adam')
But I want the minimum mean_squared_error in A, i.e. only want to consider A for the loss function.
What I can do?
You can define a custom loss function and only compute the mean_squared_error() loss based on the value of A:
from keras import losses
def loss_A(y_true, y_pred):
return losses.mean_squared_error(y_true[:,0], y_pred[:,0])
#...
model.compile(loss=loss_A, optimizer='adam')
What you need to look into is a custom loss function:
def only_A_mean_squared(y_true, y_pred):
return keras.losses.mean_squared_error(y_true[:,0], y_pred[:,0])
And in order to use it:
model.compile(loss=only_A_mean_squared, optimizer='adam')
What i am doing in the above is creating a custom loss function, which only takes the first dimension (the 'A') and feed it to the normal keras mean squared error loss function.
The MLP I created when tested on test sets shows a test score more than 100 multiple times. Could there be any mistake in coding or the data entered?
My code:
model = Sequential()
model.add(Dense(3, input_dim = 6))
model.add(Dense(3, activation='tanh'))
model.add(Dense(1))
opt = optimizers.Adam(learning_rate=0.01)
model.compile(optimizer=opt , loss='mean_squared_error')
model.fit(x, y, epochs=ep, batch_size = 50 ,verbose=0)
test_score = model.evaluate(test_x, test_y, verbose = 0)
test_score = sqrt(test_score)
test_score = get_unscaled (test_sf, np.array([test_score]))
model.evaluate can return two types of values:
Scalar: If you have not explicitly passed a value to the metric attribute to model.compile, it returns only the loss (which if it is mean squared error can be any non-negative real).
List of scalars: If you have passed metrics to model.compile, model.evaluate returns a list of scalars, the first element is the loss and all others are values of the metrics you passed.
To solve your question simply, pass your desired metric (say accuracy) to model.compile like model.compile(optimizer=opt, loss='mean_squared_error', metrics=['accuracy']). Running model.evaluate will then return [loss, accuracy]. Refer this.
You need to understand what you're doing before you start coding. It seems you are unclear on the meaning of mean squared error. Please do some reading up, both the theory and the Keras documentation, first.