I have been working on a complicated Keras model with a custom metric, and I recently converted it to tensorflow lite. The models are not exactly the same, and the outputs are different, however it is difficult to evaluate because the output is a tensor of size 128. Is there any way I can run my custom metric on this model? I have been using Tf 1.14. Below is some relevant code.
# compiler and train the model
model.save('model.h5')
# save the model in TFLite
converter = tf.lite.TFLiteConverter.from_keras_model_file('model.h5', custom_objects={'custom_metric': custom_metric})
tflite_model = converter.convert()
open('model.tflite', 'wb').write(tflite_model)
# run the model
interpreter = tf.lite.Interpreter(model_path='model.tflite')
interpreter.allocate_tensors()
input_dets = interpreter.get_input_details()
output_dets = interpreter.get_output_details()
input_shape = input_dets[0]['shape']
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
interpreter.set_tensor(input_dets[0]['index'], input_data)
interpreter.invoke()
The models are supposed to be different because the converter does graph transformations (such as fuse activation and fold batch norm) and the resulting graph is targeted in inference only scenarios.
To run metrics: the interpreter provides an API to get the output value (as an array):
output = interpreter.tensor(interpreter.get_output_details()[0]["index"])
Then you apply your metric on the output.
Related
BTHOG.
I defined and fine tuned mobile multi lingual bert model constructed using following keras code:
bert_preprocess = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_multi_cased_preprocess/3")
bert_encoder = hub.KerasLayer("https://tfhub.dev/tensorflow/mobilebert_multi_cased_L-24_H-128_B-512_A-4_F-4_OPT/1", trainable=True)
i = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
x = bert_preprocess(i)
x = bert_encoder(x)
x = tf.keras.layers.Dropout(0.2, name="dropout")(x['pooled_output'])
x = tf.keras.layers.Dense(num_classes, activation='softmax', name="output")(x)
model = tf.keras.Model(i, x)
Eventually I saved the model as tf savedmodel,
And then converted it to tflite version with the following supported ops:
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
The problem started when I tried to load the tflite converted model using the code from tensorflow github:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/examples/minimal
I get the following errors:
ERROR: Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-lite-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/guide/ops_select
ERROR: Node number 0 (FlexHashTableV2) failed to prepare.
Error at /home/nativ/dev/tflite_inference/minimal.cc:60
What can I do to fix those errors?
Your model has some TF ops, and you need to link the Select ops (Also known as flex delegate) which allows you to run TF kernels for the operation(s) that are not available in TFLite as native operations.
Mainly you will need to build the delegate and link it to the binary
bazel build -c opt --config=monolithic tensorflow/lite/delegates/flex:tensorflowlite_flex
See more details here
Model.summary() gives me a this output
Now how can i check sequential_1 layers and sequential_3 layer?
I want whole model summary but it gives two sequential so that means two model are combined so how can i get summary of both model?
I only have model.h5 file nothing else
Models saved in .h5 format includes everything about the model.
To inspect the layers summary inside the Model in a Model, like in your case.
You could extract the layers, then call the summary method from each of them.
ie.
layer_summary = [layer.summary() for layer in loaded_model.layers]
Here is the complete code I used in reproducing your scenario.
import tensorflow as tf
print('Running Tensorflow version {}'.format(tf.__version__)) # Tensorflow 2.1.0
model_path = '/content/keras_model.h5'
loaded_model = tf.keras.models.load_model(model_path)
loaded_model.summary()
inp = loaded_model.input
layer_summary = [layer.summary() for layer in loaded_model.layers]
I've also used the model.h5 file you uploaded.
In my tensorflow model, output of one network is a tensor. This value I need to feed as input to another pretrained network. I'm loading the pretrained network as follows
input_b_ph = tf.placeholder(shape=(), dtype=float.32, name='input_b_ph')
sess1 = tf.Session()
saver = tf.train.import_meta_graph(model_path.as_posix() + '.meta', input_map={'input/Identity:0': input_b_ph})
graph = tf.get_default_graph()
saver.restore(sess1, model_path.as_posix())
output_b = graph.get_tensor_by_name('output/Identity:0')
I need to feed a tensor to feature_input. How can I achieve this?
Edit 1: Adding end-to-end details:
I have a network A defined in tensorflow which takes input input_a and produces output output_a. This I need to feed to ResNet50 pretrained model. For this I used ResNet50 from tf.keras
from tensorflow.keras.applications.resnet50 import ResNet50
from tensorflow.keras.applications.resnet50 import preprocess_input
resnet_model = ResNet50(include_top=False, pooling='avg')
preprocessed_input = preprocess_input(tf.cast(output_a, tf.float32))
output_resnet = resnet_model([preprocessed_input])
Output of ResNet is output_resnet. This I need to feed to another pretrained network, say network B. B is actually written in Keras. I modified it to use tf.keras. Then I save the trained model as below:
import tensorflow as tf
from tensorflow import keras
curr_sess = keras.backend.get_session()
with tf.name_scope('input'):
_ = tf.identity(quality_net.model.input)
with tf.name_scope('output'):
__ = tf.identity(quality_net.model.output)
saver = tf.train.Saver()
saver.save(curr_sess, output_filepath.as_posix())
I have access to this network B and tried to save the model in h5 format but it gave error that model is thread lock. On searching in internet, I got to know this error comes when there are Lambda layers in the network. So, I resorted to saving the model in tensorflow format - 3 files: meta, weights and index. (Any solution using h5 format is also acceptible).
There is a caveat here. That the structure of network B can keep changing and it is from a different project. So I can't hardcode the architecture of B. I've to load it from saved model. My problem is how can I restore this pretrained model and pass output_resnet as input to network B. The output of network B i.e. output_b is the loss function to train my original network A. Currently I'm able to restore network B as follows:
input_b_ph = tf.placeholder(shape=(), dtype=float.32, name='input_b_ph')
sess1 = tf.Session()
saver = tf.train.import_meta_graph(model_path.as_posix() + '.meta', input_map={'input/Identity:0': input_b_ph})
graph = tf.get_default_graph()
saver.restore(sess1, model_path.as_posix())
output_b = graph.get_tensor_by_name('output/Identity:0')
I have output from resnet as output_resnet which is a tensor. I need a way to set this to input_b_ph. How can I achieve that? Any alternate solutions are also acceptible
Mentioning the Answer in this (Answer) Section (although it is present in Comments Section), for the benefit of the Community.
Placeholder is not required in this case. Just passing output_resnet to input_map should resolve the issue.
Replacing the code,
saver = tf.train.import_meta_graph(model_path.as_posix() + '.meta',
input_map={'input/Identity:0': input_b_ph})
with
saver = tf.train.import_meta_graph(model_path.as_posix() + '.meta',
input_map={'input/Identity:0': output_resnet})
has resolved the issue.
I am trying to use a frozen, pretrained, DeepLabv3 model in a larger tf.keras training pipeline, but have been having trouble figuring out how to use it as a tf.keras Model. I am trying to use tf.keras as I feel there would be a slowdown using a feed_dict (the only way I know of to use a frozen graph) in the middle of multiple forward passes. The deeplab model referenced in the code below is built in regular keras (as opposed to tf.contrib.keras)
from keras import backend as K
# Create, compile and train model...
frozen_graph = freeze_session(K.get_session(),
output_names=[out.op.name for out in deeplab.outputs])
tf.train.write_graph(frozen_graph, "./", "my_model.pb", as_text=False)
graph = load_graph("my_model.pb")
# We can verify that we can access the list of operations in the graph
for op in graph.get_operations():
print(op.name)
# prefix/Placeholder/inputs_placeholder
# ...
# prefix/Accuracy/predictions
# We access the input and output nodes
x = graph.get_tensor_by_name("prefix/input_1:0")
y = graph.get_tensor_by_name("prefix/bilinear_upsampling_2/ResizeBilinear:0")
# We launch a Session
with tf.Session(graph=graph) as sess:
print(graph)
model2 = models.Model(inputs=x,outputs=y)
model2.summary()
and i get an error
ValueError: Input tensors to a Model must come from `tf.layers.Input`. Received: Tensor("prefix/input_1:0", shape=(?, 512, 512, 3), dtype=float32) (missing previous layer metadata).
I feel like I've seen others replace the input tensor with an Input Layer to trick tf.keras into building the graph, but after a few hours I am feeling stuck. Any help would be appreciated!
You can recreate the model object from its config. See the from_config method here https://keras.io/models/about-keras-models/.
The config is stored and loaded back by the save_model/load_model functions. I am not familiar with freeze_session.
tflite_convert is a python script used to invoke TOCO (TensorFlow Lite Optimizing Converter) to convert files from Tensorflow's formats to tflite-compatible files.
I am trying to generate a quantized TFlite model starting from a network I trained with an Estimator. The training code is pretty straightforward and I added the required modifications for fine-tuning the model as required in the Fixed Point Quantization guide:
def input_fn(mode, num_classes, batch_size=1):
#[...]
return {'images': images}, labels
def model_fn(features, labels, num_classes, mode):
images = features['images']
with tf.contrib.slim.arg_scope(net_arg_scope()):
logits, end_points = build_net(...)
if FLAGS.with_quantization:
tf.logging.info("Applying quantization to the graph.")
if mode == tf.estimator.ModeKeys.EVAL:
tf.contrib.quantize.create_eval_graph()
tf.losses.softmax_cross_entropy(onehot_labels=labels, logits=logits)
total_loss = tf.losses.get_total_loss() #obtain the regularization losses as well
if FLAGS.with_quantization:
tf.logging.info("Applying quantization to the graph.")
if mode == tf.estimator.ModeKeys.TRAIN:
tf.contrib.quantize.create_training_graph()
# Configure the training op, etc [...]
return tf.estimator.EstimatorSpec(...)
def main(unused_argv):
regex = FINETUNE_LAYER_RE if not FLAGS.with_quantization else '^((?!_quant).)*$'
ws_settings = tf.estimator.WarmStartSettings(FLAGS.pretrained_checkpoint, regex)
# Create the Estimator
estimator = tf.estimator.Estimator(
model_fn=lambda features, labels, mode: model_fn(features, labels, NUM_CLASSES, mode),
model_dir=FLAGS.model_dir,
#config=run_config,
warm_start_from=ws_settings)
# Set up input functions for training and evaluation
train_input_fn = lambda : input_fn(tf.estimator.ModeKeys.TRAIN, NUM_CLASSES, FLAGS.batch_size)
eval_input_fn = lambda : input_fn(tf.estimator.ModeKeys.EVAL, NUM_CLASSES, FLAGS.batch_size)
#[...]
train_spec = tf.estimator.TrainSpec(...)
eval_spec = tf.estimator.EvalSpec(...)
tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)
The first problem I ran into is that it's impossible to simply continue the training using the latest checkpoint after adding the quantization operations. This is because quantization adds extra variables that won't be found in the checkpoint. I solved that writing a warm start spec that filters out all the new variables by name and using as warm start checkpoint the latest checkpoint from the training.
Now, I want to generate an evaluation graph to save (with the related variables) in order to then feed it to TOCO via the tflite_convert script.
I have tried converting one of the SavedModels exported after every evaluation, but that raises the following error:
Array conv0_bn/FusedBatchNorm, which is an input to the Relu operator
producing the output array cell_stem_0/Relu, is lacking min/max data,
which is necessary for quantization. Either target a non-quantized
output format, or change the input graph to contain min/max
information, or pass --default_ranges_min= and --default_ranges_max=
if you do not care about the accuracy of results.
Aborted (core dumped)
I don't know how to get the right SavedModel or a pair GraphDef + checkpoint (although the SavedModel is preferrable)
Has anyone tried to quantized an estimator model? How do you generate the quantized evaluation graph?