I have a model saved after training as pb file, I want to use tensorflow mobile and it's important to work with TFLITE file.
The problem is most of the examples I found after googling for converters are command on terminal or cmd.
Can you please share with me an example of converting to tflite files using python code?
You can convert to tflite directly in python directly. You have to freeze the graph and use toco_convert. It needs the input and output names and shapes to be determined ahead of calling the API just like in the commandline case.
An example code snippet
Copied from documentation, where a "frozen" (no variables) graph is defined as part of your code:
import tensorflow as tf
img = tf.placeholder(name="img", dtype=tf.float32, shape=(1, 64, 64, 3))
val = img + tf.constant([1., 2., 3.]) + tf.constant([1., 4., 4.])
out = tf.identity(val, name="out")
with tf.Session() as sess:
tflite_model = tf.contrib.lite.toco_convert(sess.graph_def, [img], [out])
open("test.tflite", "wb").write(tflite_model)
In the example above, there is no freeze graph step since there are no variables. If you have variables and run toco without freezing graph, i.e. converting those variables to constants first, then toco will complain!
If you have frozen graphdef and know the inputs and outputs
Then you don't need the session. You can directly call toco API:
path_to_frozen_graphdef_pb = '...'
input_tensors = [...]
output_tensors = [...]
frozen_graph_def = tf.GraphDef()
with open(path_to_frozen_graphdef_pb, 'rb') as f:
frozen_graph_def.ParseFromString(f.read())
tflite_model = tf.contrib.lite.toco_convert(frozen_graph_def, input_tensors, output_tensors)
If you have non-frozen graphdef and know the inputs and outputs
Then you have to load the session and freeze the graph first before calling toco:
path_to_graphdef_pb = '...'
g = tf.GraphDef()
with open(path_to_graphdef_pb, 'rb') as f:
g.ParseFromString(f.read())
output_node_names = ["..."]
input_tensors = [..]
output_tensors = [...]
with tf.Session(graph=g) as sess:
frozen_graph_def = tf.graph_util.convert_variables_to_constants(
sess, sess.graph_def, output_node_names)
# Note here we are passing frozen_graph_def obtained in the previous step to toco.
tflite_model = tf.contrib.lite.toco_convert(frozen_graph_def, input_tensors, output_tensors)
If you don't know inputs / outputs of the graph
This can happen if you did not define the graph, ex. you downloaded the graph from somewhere or used a high level API like the tf.estimators that hide the graph from you. In this case, you need to load the graph and poke around to figure out the inputs and outputs before calling toco. See my answer to this SO question.
Following this TF example you can pass "--Saved_model_dir" parameter to export the saved_model.pb and variables folder to some directory (none existing dir) before running retrain.py script:
python retrain.py ...... --saved_model_dir /home/..../export
In order to convert your model to tflite you need to use below line:
convert_saved_model.convert(saved_model_dir='/home/.../export',output_arrays="final_result",output_tflite='/home/.../export/graph.tflite')
Note: you need to import convert_saved_model:
from tensorflow.contrib.lite.python import convert_saved_model
Remember you can convert to tflite in 2 ways:
But the easiest way is to export saved_model.pb with variables in case you want to avoid using builds tools like Bazel.
This is what worked for me: (SSD_InceptionV2 model)
After finishing the training. i used model_main.py from object_detection folder. TFv1.11
ExportGraph as TFLITE:
python /tensorflow/models/research/object_detection/export_tflite_ssd_graph.py
--pipeline_config_path annotations/ssd_inception_v2_coco.config
--trained_checkpoint_prefix trained-inference-graphs/inference_graph_v7.pb/model.ckpt
--output_directory trained-inference-graphs/inference_graph_v7.pb/tflite
--max_detections 3
This generates a .pb file so you can generate the tflite file from it like this:
tflite_convert
--output_file=test.tflite
--graph_def_file=tflite_graph.pb
--input_arrays=normalized_input_image_tensor
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3'
--input_shape=1,300,300,3
--allow_custom_ops
Now the inputs/outputs i am not 100 sure how to get this but this code helps me before:
import tensorflow as tf
frozen='/tensorflow/mobilenets/mobilenet_v1_1.0_224.pb'
gf = tf.GraphDef()
gf.ParseFromString(open(frozen,'rb').read())
[n.name + '=>' + n.op for n in gf.node if n.op in ( 'Softmax','Placeholder')]
[n.name + '=>' + n.op for n in gf.node if n.op in ( 'Softmax','Mul')]
converter = tf.contrib.lite.TFLiteConverter.from_frozen_graph(
frozen_model_filename, INPUT_NODE, OUTPUT_NODE)
tflite_model = converter.convert()
open(TFLITE_OUTPUT_FILE, "wb").write(tflite_model)
INPUT_NODE and OUTPUT_NODE are the lists of names of input and output respectively.
Related
I trained a model on darknet using the YOLOv3-SPP model. I need to be able to use this model in my iPhone app so I need to convert it to CoreML. I started by converting the .weights file to a .pb file. Now I am trying to convert it from TensorFlow to CoreML with tfcoreml. However I cannot seem to determine my input and output tensor names. I tried to use tensorboard to visualize the model and determine the inputs and outputs but since I am quite new to TensorFlow I can't figure out what to use. I am using this script to convert the model from TensorFlow to CoreML:
import tfcoreml
import os
import tensorflow as tf
frozen_model_file = os.path.abspath('frozen_darknet_yolov3_model.pb')
input_tensor_shapes = {"input/placeholder:0": [1, 32, 32, 9]}
# Output CoreML model path
coreml_model_file = './model.mlmodel'
output_tensor_names = ['output/prediction:0']
def convert():
# Read the pb model
with tf.gfile.GFile(frozen_model_file, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
# Then, we import the graph_def into a new Graph
tf.import_graph_def(graph_def, name="")
# Convert
tfcoreml.convert(
tf_model_path=frozen_model_file,
mlmodel_path=coreml_model_file,
input_name_shape_dict=input_tensor_shapes,
output_feature_names=output_tensor_names)
convert()
This what my tensorboard looks like:
What should I set the input_tensor_shapes and output_tensor_names too so that I don't get an error saying that my TensorFlow graph does not contain a tensor with that name.
I suggest using Netron to view the TensorFlow file. It makes the graph much easier to understand.
I am trying to freeze a pre-trained model to then convert it in TF Lite and deploy it into an Android device.
By inspecting the resulting .pb file with xxd I see that it only contains the placeholder output variable. The size of the .pb is a few Bytes.
Why all the graph and variables are not included in the model?
I used the code below derived from https://github.com/sankit1/cv-tricks.com/tree/master/Tensorflow-tutorials/freeze_model_and_deploy. It works fine with other models but not with mine.
import tensorflow as tf
from tensorflow.python.framework import graph_util
import os,sys
path = './model-gaze/'
output_node_names = "pos"
model_name = 'model-23'
saver = tf.train.import_meta_graph(path+model_name+'.meta', clear_devices=True)
graph = tf.get_default_graph()
input_graph_def = graph.as_graph_def()
sess = tf.Session()
saver.restore(sess, path+model_name)
output_graph_def = graph_util.convert_variables_to_constants(
sess, # The session is used to retrieve the weights
input_graph_def, # The graph_def is used to retrieve the nodes
output_node_names.split(",") # The output node names are used to select the usefull nodes
)
output_graph=path+model_name+".pb"
with tf.gfile.GFile(output_graph, "wb") as f:
f.write(output_graph_def.SerializeToString())
sess.close()
I would expect that all the weights and graph data are included inside the .pb but cannot manage to get them there.
The link which you are following is the right procedure to freeze a tensorflow model.
Freezing a model reduces the size of the model as it only saves the ""output_node_names"" which you give it to store.
Please refer to the below link on the entire process.
https://cv-tricks.com/how-to/freeze-tensorflow-models/
Here, can you please elaborate on what ""pos"" is ?
Also, here if you pass the prediction op as that is the required final op for predictions, it should work fine.
And if this does not help, please share your model and the code from where you have saved the model, to further debug the issue.
I'm trying to convert this CPM-TF model to TFLite, but to use the TocoConverter, I need to specify input and output tensors.
https://github.com/timctho/convolutional-pose-machines-tensorflow
I ran the included run_freeze_model.py and got the cpm_hand_frozen.pb (GraphDef?) file.
From this post I copied the code snippet for converting the ProtoBuf file with known inputs and outputs. But looking through the model definition code, I have some trouble finding the correct answers for the in- and outputs.
Tensorflow Convert pb file to TFLITE using python
import tensorflow as tf
import numpy as np
from config import FLAGS
path_to_frozen_graphdef_pb = 'frozen_models/cpm_hand_frozen.pb'
def main(argv):
input_tensors = [1, FLAGS.input_size, FLAGS.input_size, 3]
output_tensors = np.zeros(FLAGS.num_of_joints)
frozen_graph_def = tf.GraphDef()
with open(path_to_frozen_graphdef_pb, 'rb') as f:
frozen_graph_def.ParseFromString(f.read())
tflite_model = tf.contrib.lite.toco_convert(frozen_graph_def, input_tensors, output_tensors)
if __name__ == '__main__':
tf.app.run()
I'm quite new to Tensorflow, but I think the input should be defined as
[1, FLAGS.input_size, FLAGS.input_size, 3]
Found that here: https://github.com/timctho/convolutional-pose-machines-tensorflow/blob/master/models/nets/cpm_hand.py#L23
Not sure what 1 represents, but None does not work and I guess the other parameters are the image size and color channels.
However, with that input, it returns an error:
AttributeError: 'int' object has no attribute 'dtype'
I got no clue on what the output should be, other than it should be an array.
UPDATE 1
Looking through the TF docs, it appears that I need to define the input as a tensor (obvious!).
https://www.tensorflow.org/lite/convert/python_api
input_tensors = tf.placeholder(name="img", dtype=tf.float32, shape=(1,FLAGS.input_size, FLAGS.input_size, 3))
This does not return an error, but I still need to figure out if the input are correct and what the output should be like.
UPDATE 2
Alright, so I finally got it to spit out the tflite model with this code snippet
def tflite_converter():
graph_def_file = os.path.join('frozen_models', '{}_frozen.pb'.format('cpm_hand'))
input_arrays = ['input_placeholer']
output_arrays = [FLAGS.output_node_names]
converter = tf.lite.TFLiteConverter.from_frozen_graph(graph_def_file, input_arrays, output_arrays)
tflite_model = converter.convert()
open('{}.tflite'.format('cpm_hand'), 'wb').write(tflite_model)
I hope that I did it correctly. I will try to do inference on the model on Android.
I do also think there is a misspelling in the input tensor input_placeholder. It appears to be corrected in the code itself, but from printing out all node names from the pretrained model, the spelling input_placeholer is present.
Node names can be seen here: https://github.com/timctho/convolutional-pose-machines-tensorflow/issues/59
My setup is:
Ubuntu 18.04
CUDA 9.1 & cuDNN 7.0
Python 3.6.5
Tensorflow GPU 1.6
Inference works like a charm, so there should be no issue on the setup itself.
I trained my model with estimator of TensorFlow. It seems that export_savedmodel should be used to make .pb file, but I don't really know how to construct the serving_input_receiver_fn. Anybody any ideas?
Example code is welcomed.
Extra questions:
Is .pb the only file I need when I want to reload the model? Variable unnecessary?
How much will .pb reduced the model file size compared with .ckpt with adam optimizer?
You can use freeze_graph.py to produce a .pb from .ckpt + .pbtxt
if you're using tf.estimator.Estimator, then you'll find these two files in the model_dir
python freeze_graph.py \
--input_graph=graph.pbtxt \
--input_checkpoint=model.ckpt-308 \
--output_graph=output_graph.pb
--output_node_names=<output_node>
Is .pb the only file I need when I want to reload the model? Variable unnecessary?
Yes, You'll have to know you're model's input nodes and output node names too. Then use import_graph_def to load the .pb file and get the input and output operations using get_operation_by_name
How much will .pb reduced the model file size compared with .ckpt with adam optimizer?
A .pb file is not a compressed .ckpt file, so there is no "compression rate".
However, there is a way to optimize your .pb file for inference, and this optimization may reduce the file size as it removes parts of the graph that are training only operations (see the complete description here).
[comment] how can I get the input and output node names?
You can set the input and output node names using the op name parameter.
To list the node names in your .pbtxt file, use the following script.
import tensorflow as tf
from google.protobuf import text_format
with open('graph.pbtxt') as f:
graph_def = text_format.Parse(f.read(), tf.GraphDef())
print [n.name for n in graph_def.node]
[comment] I found that there is a tf.estimator.Estimator.export_savedmodel(), is that the function to store model in .pb directly? And I'm struggling in it's parameter serving_input_receiver_fn. Any ideas?
export_savedmodel() generates a SavedModel which is a universal serialization format for TensorFlow models. It should contain everything's needed to fit with TensorFlow Serving APIs
serving_input_receiver_fn() is a part of those needed things you have to provide in order to generate a SavedModel, it determines the input signature of your model by adding placeholders to the graph.
From the doc
This function has the following purposes:
To add placeholders to the graph that the serving system will feed
with inference requests.
To add any additional ops needed to convert
data from the input format into the feature Tensors expected by the
model.
If you're receiving your inference requests in the form of serialized tf.Examples (which is a typical pattern) then you can use the example provided in the doc.
feature_spec = {'foo': tf.FixedLenFeature(...),
'bar': tf.VarLenFeature(...)}
def serving_input_receiver_fn():
"""An input receiver that expects a serialized tf.Example."""
serialized_tf_example = tf.placeholder(dtype=tf.string,
shape=[default_batch_size],
name='input_example_tensor')
receiver_tensors = {'examples': serialized_tf_example}
features = tf.parse_example(serialized_tf_example, feature_spec)
return tf.estimator.export.ServingInputReceiver(features, receiver_tensors)
[comment] Any idea to list the node names in '.pb'?
It depends on how it was generated.
if it's a SavedModel the use:
import tensorflow as tf
with tf.Session() as sess:
meta_graph_def = tf.saved_model.loader.load(
sess,
[tf.saved_model.tag_constants.SERVING],
'./saved_models/1519232535')
print [n.name for n in meta_graph_def.graph_def.node]
if it's a MetaGraph then use:
import tensorflow as tf
from tensorflow.python.platform import gfile
with tf.Session() as sess:
with gfile.FastGFile('model.pb', 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
sess.graph.as_default()
tf.import_graph_def(graph_def, name='')
print [n.name for n in graph_def.node]
New checkpoint format generates three files: model.ckpt-1000.data-00000-of-00001,model.ckpt-1000.meta,model.ckpt-1000.index. Old checkpoint format only generates two files: model.ckpt-1000 and model.ckpt-1000.meta.
When I restore model wrote with new checkpoint format using code saver.restore(sess, './model.ckpt-1000'), it can restore model successfully. Now current directory has three files model.ckpt-1000.data-00000-of-00001, model.ckpt-1000.meta, model.ckpt-1000.index. If I delete model.ckpt-1000.index, it cannot restore model. Error information:DataLossError (see above for traceback): Unable to open table file ./model.ckpt-1000: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?.
But I notice tensorflow pretrained model that only has one file: *.ckpt. When I use pretrained model provided by tensorflow, it can restore sucessfully. It does not need *.index file.
My tensorflow version is 1.1.0.
How did it save model? Did it use old checkpoint format?
I believe it did use the old checkpoint format. Here's a simple example I used to verify:
import tensorflow as tf
slim = tf.contrib.slim
x = tf.placeholder(tf.float32, [None, 16])
y = slim.fully_connected(x, 4)
saver_v1 = tf.train.Saver(write_version=tf.train.SaverDef.V1)
saver_v2 = tf.train.Saver()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
saver_v1.save(sess, 'model_v1.ckpt',
write_meta_graph=False,
write_state=False)
saver_v2.save(sess, 'model_v2.ckpt',
write_meta_graph=False,
write_state=False)
The results of ls model* after running the script:
model_v1.ckpt model_v2.ckpt.data-00000-of-00001 model_v2.ckpt.index