We Keep Coding

Tensorflow Keras Model subclassing -- call function

I am experimenting with self supervised learning using tensorflow. The example code I'm running can be found in the Keras examples website. This is the link to the NNCLR example. The Github link to download the code can be found here. While I have no issues running the examples, I am running into issues when I try to save the pretrained or the finetuned model using model.save().
The error I'm getting is this:
f"Model {model} cannot be saved either because the input shape is not "
ValueError: Model <__main__.NNCLR object at 0x7f6bc0f39550> cannot be saved either
because the input shape is not available or because the forward pass of the model is
not defined. To define a forward pass, please override `Model.call()`.
To specify an input shape, either call `build(input_shape)` directly, or call the model on actual data using `Model()`, `Model.fit()`, or `Model.predict()`.
If you have a custom training step, please make sure to invoke the forward pass in train step through
`Model.__call__`, i.e. `model(inputs)`, as opposed to `model.call()`.
I am unsure how to override the Model.call() method. Appreciate some help.

One way to achieve model saving in such cases is to override the save (or save_weights) method in the keras.Model class. In your case, first initialize the finetune model in the NNCLR class. And next, override the save method for it. FYI, in this way, you may also able to use ModelCheckpoint API.
As said, define the finetune model in the NNCLR model class and override the save method for it.
class NNCLR(keras.Model):
def __init__(...):
super().__init__()
...
self.finetuning_model = keras.Sequential(
[
layers.Input(shape=input_shape),
self.classification_augmenter,
self.encoder,
layers.Dense(10),
],
name="finetuning_model",
)
...
def save(
self, filepath, overwrite=True, include_optimizer=True,
save_format=None, signatures=None, options=None
):
self.finetuning_model.save(
filepath=filepath,
overwrite=overwrite,
save_format=save_format,
options=options,
include_optimizer=include_optimizer,
signatures=signatures
)
model = NNCLR(...)
model.compile
model.fit
Next, you can do
model.save('finetune_model') # SavedModel format
finetune_model = tf.keras.models.load_model('finetune_model', compile=False)
'''
NNCLR code example: Evaluate sections
"A popular way to evaluate a SSL method in computer vision or
for that fact any other pre-training method as such is to learn
a linear classifier on the frozen features of the trained backbone
model and evaluate the classifier on unseen images."
'''
for layer in finetune_model.layers:
if not isinstance(layer, layers.Dense):
layer.trainable = False
finetune_model.summary() # OK
finetune_model.compile(
optimizer=keras.optimizers.Adam(),
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[keras.metrics.SparseCategoricalAccuracy(name="acc")],
)
finetune_model.fit

LightGBM produces same probabilities on any input (C++)

I have trained a LGBM model (gbdt) with python on a dataset with 5 classes (classification problem) and I'm able to make a correct inference on a test set, loading that model in a python script.
Now I need to use this model in a C++ program. To do this I have exported this model and I have lodaded it in C++ to make inference. The problem is that in C++ the output probabilities are always the same so I can't choose a winner class (each class results always 0.2).
To save the model I've tried this 2 ways.
First I've tried to save a model like a string:
s = lgb_model.model_to_string(num_iteration=114)
f = open('model_out.txt','w')
f.write(s)
f.close()
Second directly with save model method:
lgb_model.save_model('model_out.txt')
To load the model in C++ I've used this with no error:
int ret = LGBM_BoosterLoadModelFromString(model_string, &num_iter, &booster_handle);
To make inference I have prepared an input buffer and I have passed it on this function:
int res = LGBM_BoosterPredictForMat(booster_handle, input_data, C_API_DTYPE_FLOAT64,
n_row, n_cols, 1, C_API_PREDICT_NORMAL, 0, -1,"", &out_len, out_result);
I obtained a matrix with 5 rows, and a column for each sample like this:
0.2
0.2
0.2
0.2
0.2
I have tried to make inference with a lot of changes but the results are always the same (random inputs, different parameters, etc.). Moreover I have checked the loaded model trying to re-dump it with this function and the result seemed corrent:
LGBM_BoosterDumpModel(booster_handle, 0, -1, C_API_FEATURE_IMPORTANCE_SPLIT, 1, &out_len, out_string);
Where am I wrong?

I had a similar issue and in my case I found that the problem was the is_linear property in the model.
I compared the model that I generated from the binary_classification example with the model I was using and I noticed that the model in the example has the is_linear=0 property for each tree. On my model it was missing.
Then I checked the c++ code and found that if this property is missing, the variable describing this is true. I set it to false as default and that works for me.
I can't give more details as I just recently began working with LGBM models and c++.

How do I train a chain of two models in Keras, but with a function in between them?

I'm using the Keras functional API and attempting to stack and train two models with a non-linear step in between them.
Say I want to train a chain of two models, Model A and Model B, where the output of Model A is used as the input of Model B, as one model, Model C. My understanding of how to do this is:
input_A = Input(input_shape_A)
output_A = ModelA(inputA)
output_B = ModelB(outputA)
model_C = Model(inputA, outputB)
Source
The problem is that, in my case, I want to slice up the output of Model A before it goes into Model B, call another function on the slices which returns only one of them, and then use that slice as the input to Model B. So, Model B is only being trained on a fixed-size subset of Model A's output, which could be at arbitrary indices.
Something more like:
input_A = Input(input_shape_A)
output_A = ModelA(inputA)
input_B = custom_function(output_A)
output_B = ModelB(input_B)
model_C = Model(inputA, outputB)
I have not found any code examples so far that resemble this, and I am still trying to figure out if I can. The loss of Model B has to be integrated into Model A's training, but I need a function in between them. I was considering keeping them separate and trying to write out a custom loss function for Model A, but custom loss functions in Keras are seem to be very restrictive, and I haven't seen any examples for that approach so far, either.
Is this possible?

Saving/Restoring weights under different variable scopes in Tensorflow

I've been trying to research model/weight saving for a while, but I still can't fully grasp it. I feel what I'd like to do should be simple enough, but I've not found a solution.
The final goal is to do transfer laerning with a collection of pretrained networks. I write my models/layers as classes, so class method(s) for saving the weights and restoring would be ideal.
Example:
If I have a graph, features > A > B > labels, where A and B are sub-networks, I'd like to save and/or restore weights for these sections. Say I already have the weights for A trained, but the variable scope is now different, how would I restore the weights I've trained for A from a different training session? At the end of training this new graph i'd like 1 directory for my new A weights, 1 directory for my new B weights, and 1 directory for the full graph (I can handle the full graph bit).
It's very possible I keep overlooking the solution, but model saving is so poorly documented.
Hope I've explained the scenario well.

You can do this with tf.train.init_from_checkpoint
Define your model
def model_fn():
with tf.variable_scope('One'):
layer = any_tf_layer
with tf.variable_scope('Two'):
layer = any_tf_layer
Output variable names in checkpoint file
vars = [i[0] for i in tf.train.list_variables(ckpt_file)]
Then you can create assignment map to load only variables, defined in your model.
You can also assign new names to restored variables
map = {variable.op.name: variable for variable in tf.global_variables() if variable.op.name in vars}
This line is placed before session or outside model function for Estimator API
tf.train.init_from_checkpoint(ckpt_file, map)
https://www.tensorflow.org/api_docs/python/tf/train/init_from_checkpoint
You also can do it with tf.train.Saver
First you need to know the names of variables
vars_dict = {}
for var_current in tf.global_variables():
print(var_current)
print(var_current.op.name) # this gets only name
for var_ckpt in tf.train.list_variables(ckpt):
print(var_ckpt[0]) this gets only name
When you know exact names of all variables you can assign whatever value you need, provided variables have same shape and dtype. So to get a dictionary
vars_dict[var_ckpt[0]) = tf.get_variable(var_current.op.name, shape) # remember to specify shape, you can always get it from var_current
saver = tf.train.Saver(vars_dict)
Take a look at my other answer to similar question
How to restore pretrained checkpoint for current model in Tensorflow?

Store/Reload CNTK Trainer, Model, Inputs, Outputs

What is the best way to store a trainer and all necessary components?
1. Storing:
Store checkpoint of the trainer: Use its trainer.save_checkpoint(filename, external_state={}) function
Additionally store the model separately: Use the z.save(filename) method, every cntk operation has. You can also get z = trainer.model.
2. Reloading:
Restore the model: Use C.load_model(...). (Don't get confused by the deprecated persist namespace from the Cntk 1.)
Get the inputs from the restored model.
Restore the trainer itself: Use trainer.restore_from_checkpoint as eg. shown here. The problem is, this function already needs a trainer object which probably has to be initialized in the same way as the trainer used to create the check point!?
How do I now restore the label-inputs which are going into the error function used by the trainer? In the following code I marked the variables which I think I have to restore after I once stored them.
z = C.layers.Dense(.... )
loss = error = C.squared_error(z, **l**)
**trainer** = C.Trainer(**z**, (loss, error), [mylearner], my_tensorboard_writer)

You can restore your trainer, but I actually prefer to just load my model m. The simple reason is that it is much easier to create a whole new trainer, beacuse then you can change all the other parameters of the trainer more easily.
Then you can get the input variable from the loaded model (if your network has only one input):
input_var = m.arguments[0]
then you need the output of your model:
output = m(input_var)
and define the loss function using your target output target_output:
C.squared_error(output, target_output)
using your model and the loss function you can recreate your trainer from there, setting the learning rate etc. as you like