I have a tf model that has two outputs, as indicated by this model.compile():
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=7e-4),
loss={"BV": tf.keras.losses.MeanAbsoluteError(), "Rsp": tf.keras.losses.MeanAbsoluteError()},
metrics={"BV": [tf.keras.metrics.RootMeanSquaredError(name="RMSE"), tfa.metrics.r_square.RSquare(name="R2")],
"Rsp": [tf.keras.metrics.RootMeanSquaredError(name="RMSE"), tfa.metrics.r_square.RSquare(name="R2")]})
I would like to use the ModelCheckpoint callback, which should monitor a sum of val_BV_R2 and val_Rsp_R2. I am able to run the callback like this:
save_best_model = tf.keras.callbacks.ModelCheckpoint("xyz.hdf5", monitor="val_Rsp_R2")
However, I don't know how to make it to save the model with the highest sum of two metrics.
According to the tf.keras.callbacks.ModelCheckpoint documentation, the metric to monitor che be only one at a time.
One way to achieve what you want, could be to define an additional custom metric, that performs the sum of the two metrics. Then you could monitor your custom metric and save the checkpoints as you are already doing. However this is a bit complicated, due to having multiple outputs.
Alternatively you could define a custom callback that does the same combining. Below a simple example of this second option. It should work (sorry I can't test it right now):
class CombineCallback(tf.keras.callbacks.Callback):
def __init__(self, **kargs):
super(CombineCallback, self).__init__(**kargs)
def on_epoch_end(self, epoch, logs={}):
logs['combine_metric'] = 0.5*logs['val_BV_R2'] + 0.5*logs['val_Rsp_R2']
Inside the callback you should be able to access your metrics directly with logs['name_of_my_metric'] or through the get function logs.get("name_of_my_metric").
Also I multiplied by 0.5 to leave the combined metric approximately in the same range, but see if this works for your case.
To use it just do:
save_best_model = CombineCallback("xyz.hdf5")
model.fit(..., callbacks=[save_best_model])
More information can be found at the Examples of Keras callback applications.
Related
I'm trying to use Vetiver to deploy an isolation forest model (for anomaly detection) to an API endpoint.
All is going well by adapting the example here.
However, when deployed, the endpoint uses the model.predict() method by default (which returns 1 for normal or -1 for anomaly).
I want the model to return a score between 0 and 1 as given by the model.score_samples() method.
Does anybody know how I can configure the Vetiver endpoint such that it uses .score_samples() rather than .predict() for scoring?
Thanks
vetiver.predict() primarily acts as a router to an API endpoint, so it does not have to have the same behavior as model.predict(). You can overload what function vetiver.predict() uses on your model by defining a custom handler.
In your case, an implementation might look something like below.
from vetiver.handlers.base import VetiverHandler
class ScoreSamplesHandler(VetiverHandler):
def __init__(self, model, ptype_data):
super().__init__(model, ptype_data)
def handler_predict(self, input_data, check_ptype):
"""
Define how to make predictions from your model
"""
prediction = self.model.score_samples(input_data)
return prediction
Initialize your custom handler and pass it into VetiverModel().
custom_model = ScoreSamplesHandler(model, ptype_data)
VetiverModel(custom_model, "model_name")
Then, when you use the vetiver.predict() function, it will return the values for model.score_samples(), rather than model.predict().
Transformers also provide their own schedulers for learning rates like get_constant_schedule, get_constant_schedule_with_warmup, etc. They are again returning torch.optim.lr_scheduler.LambdaLR (torch scheduler). Is the warmup_steps the only difference between the two?
How can we create a custom transformer-based scheduler similar to other torch schedulers like lr_scheduler.MultiplicativeLR, lr_scheduler.StepLR, lr_scheduler.ExponentialLR?
You can create a custom scheduler by just creating a function in a class that takes in an optimizer and its state dicts and edits the values in its param_groups.
To understand how to structure this in a class, just take a look at how Pytorch creates its schedulers and use the same functions just change the functionality to your liking.
The Permalink I found that will be a good reference is over here
EDIT After comments:
This is like a template you can use
from torch.optim import lr_scheduler
class MyScheduler(lr_scheduler._LRScheduler # Optional inheritance):
def __init__(self, # optimizer, epoch, step size, whatever you need as input to lr scheduler, you can even use vars from LRShceduler Class that you can inherit from etc.):
super(MyScheduler, self).__init__(optimizer, last_epoch, verbose)
# Put variables that you will need for updating scheduler like gamma, optimizer, or step size etc.
self.optimizer = optimizer
def get_lr(self):
# How will you use the above variables to update the optimizer
for group in self.optimizer.param_groups:
group["lr"] = # Fill this out with updated optimizer
return self.optimizer
You can add more functions for increased functionality. Or you can just use a function to update your learning rate. This will take in the optimizer and change it optimizer.param_groups[0]["lr"] and return the new optimizer.
I have added lots of metrics for tracking the performance of my multi-class segmentation model in Keras and Tensorflow. These metrics include class-wise and aggregated metric functions. Now Tensorboard contains everything I want, but my command line output looks overloaded now. I would like to remove the class-wise metrics from the command line output while keeping them in Tensorboard. Is that possible?
model.compile(loss=dice_loss,
metrics=[f1score, f1score_class0, f1score_class1, f1score_class2])
Is it possible when implementing train_step and test_step on my own? Would I need to implement the training loop from scratch?
This can be easily done using callbacks. Namely:
write a callback printing only the metrics you are interested in. On how to write callbacks, see this tutorial.
in model.fit set verbose=0 and add your callback to callbacks.
run the training.
As far as I know, tensorboard uses all logs. Hence it will get all metrics from all epoches.
If you do not need any information to be printed during training at all, you may just set verbose=0.
Based on the answer of Ivan K. I came up with another idea:
The keras.callbacks.ProgbarLogger is responsible for logging to the command line.
It is possible to subclass the ProgbarLogger of keras and add an ProgbarLogger to the callbacks. It will prevent Keras to add the default ProgbarLogger and therefore replace it. Just implement all methods receiving the logs as argument, filter the logs and pass the filtered logs to the corresponding parent's class method.
This example removes logs which contain substrings in opt_out list.
import tensorflow as tf
class CustomProgbarLogger(tf.keras.callbacks.ProgbarLogger):
def __init__(self, count_mode="samples", stateful_metrics=None, opt_out=[]):
super().__init__(count_mode=count_mode, stateful_metrics=stateful_metrics)
self.opt_out = opt_out
def _filter(self, logname):
return all(word not in logname for word in self.opt_out)
def _filter_logs(self, logs):
return logs and {key: value for key, value in logs.items() if self._filter(key)}
def on_train_batch_end(self, batch, logs=None):
super().on_train_batch_end(batch, self._filter_logs(logs))
def on_test_batch_end(self, batch, logs=None):
super().on_test_batch_end(batch, self._filter_logs(logs))
def on_epoch_end(self, epoch, logs=None):
super().on_epoch_end(epoch, self._filter_logs(logs))
def on_test_end(self, logs=None):
super().on_test_end(self._filter_logs(logs))
def on_predict_end(self, logs=None):
super().on_predict_end(self._filter_logs(logs))
this example will remove every log containing one of the string class, precision, recall in the key.
progbar_callback = CustomProgbarLogger(opt_out=["class", "precision", "recall"])
model.fit(dataset, callbacks=[progbar_callback, tensorboard_callback])
I am using tensorflow.keras and want to know if it is possible to create reusable blocks of inbuilt Keras layers. For example, I would like to repeatedly use the same set of layers (that able to learn the different weights) at a different position in a model. I would like to use the following block at different times in my model.
keep_prob_=0.5
input_features=Input(shape=(29, 1664))
Imortant_features= SelfAttention(activation='tanh',
kernel_regularizer=tf.keras.regularizers.l2(0.), kernel_initializer='glorot_uniform'
(input_features)
drop3=tf.keras.layers.Dropout(keep_prob_)(Imortant_features)
Layer_norm_feat=tf.keras.layers.Add()([input_features, drop3])
Layer_norm=tf.keras.layers.LayerNormalization(axis=-1)(Layer_norm_feat)
ff_out=tf.keras.layers.Dense(Layer_norm.shape[2], activation='relu')(Layer_norm)
ff_out=tf.keras.layers.Dense(Layer_norm.shape[2])(ff_out)
drop4=tf.keras.layers.Dropout(keep_prob_)(ff_out)
Layer_norm_input=tf.keras.layers.Add()([Layer_norm, drop4])
Attention_block_out=tf.keras.layers.LayerNormalization(axis=-1)(Layer_norm_input)
intraEpoch_att_block=tf.keras.Model(inputs=input_features, outputs=Attention_block_out)
I have read about creating custom layers in Keras but I did not find the documentation to be clear enough. I want to reuse the sub-model which able to learn the different set of weight in a single functional API model in tensorflow.keras.
Use this code (I removed SelfAttention, so add it back):
import tensorflow as tf
class my_model(tf.keras.layers.Layer):
def __init__(self):
super(my_model, self).__init__()
keep_prob_=0.5
input_features=tf.keras.layers.Input(shape=(29, 1664))
drop3=tf.keras.layers.Dropout(keep_prob_)(input_features)
Layer_norm_feat=tf.keras.layers.Add()([input_features, drop3])
Layer_norm=tf.keras.layers.LayerNormalization(axis=-1)(Layer_norm_feat)
ff_out=tf.keras.layers.Dense(Layer_norm.shape[2], activation='relu')(Layer_norm)
ff_out=tf.keras.layers.Dense(Layer_norm.shape[2])(ff_out)
drop4=tf.keras.layers.Dropout(keep_prob_)(ff_out)
Layer_norm_input=tf.keras.layers.Add()([Layer_norm, drop4])
Attention_block_out=tf.keras.layers.LayerNormalization(axis=-1)(Layer_norm_input)
self.intraEpoch_att_block=tf.keras.Model(inputs=input_features, outputs=Attention_block_out)
def call(self, inp, training=False):
x = self.intraEpoch_att_block(inp)
return x
model1 = my_model()
model2 = my_model()
What is the best way to store a trainer and all necessary components?
1. Storing:
Store checkpoint of the trainer: Use its trainer.save_checkpoint(filename, external_state={}) function
Additionally store the model separately: Use the z.save(filename) method, every cntk operation has. You can also get z = trainer.model.
2. Reloading:
Restore the model: Use C.load_model(...). (Don't get confused by the deprecated persist namespace from the Cntk 1.)
Get the inputs from the restored model.
Restore the trainer itself: Use trainer.restore_from_checkpoint as eg. shown here. The problem is, this function already needs a trainer object which probably has to be initialized in the same way as the trainer used to create the check point!?
How do I now restore the label-inputs which are going into the error function used by the trainer? In the following code I marked the variables which I think I have to restore after I once stored them.
z = C.layers.Dense(.... )
loss = error = C.squared_error(z, **l**)
**trainer** = C.Trainer(**z**, (loss, error), [mylearner], my_tensorboard_writer)
You can restore your trainer, but I actually prefer to just load my model m. The simple reason is that it is much easier to create a whole new trainer, beacuse then you can change all the other parameters of the trainer more easily.
Then you can get the input variable from the loaded model (if your network has only one input):
input_var = m.arguments[0]
then you need the output of your model:
output = m(input_var)
and define the loss function using your target output target_output:
C.squared_error(output, target_output)
using your model and the loss function you can recreate your trainer from there, setting the learning rate etc. as you like