We Keep Coding

Tensorflow frozen inference graph from .meta .info .data and combining frozen inference graphs

I am new to tensorflow, and currently struggling with some issues :
How to get frozen inference graph from .meta .data .info without pipeline config
I wanted to check pre trained models of traffic sign detection in real time. Model contains 3 files - .meta .data .info, but i cant find information, how to convert them into frozen inference graph without pipeline config. Everything i find is either outdated or needs pipeline config.
Also, i tried to train model myself, but i think that problem is .ppa files (GTSDB dataset), because with .png or .jpg everything worked just fine.
How to combine two or more frozen inference graphs
I have successfully trained model on my own dataset (detect some specific object), but i want that model to work with some pre trained models like faster rcnn inception or ssd mobilenet. I understand that i have to load both models, but i have no idea how to make them work at the same time and is it even possible?
UPDATE
I'm halfway there on first problem - now i have frozen_model.pb, problem was in output node names, i got confused and didn't know what to put there, so after hours of "investigating", got working code:
import os, argparse
import tensorflow as tf
# The original freeze_graph function
# from tensorflow.python.tools.freeze_graph import freeze_graph
dir = os.path.dirname(os.path.realpath(__file__))
def freeze_graph(model_dir):
"""Extract the sub graph defined by the output nodes and convert
all its variables into constant
Args:
model_dir: the root folder containing the checkpoint state file
output_node_names: a string, containing all the output node's names,
comma separated
"""
if not tf.gfile.Exists(model_dir):
raise AssertionError(
"Export directory doesn't exists. Please specify an export "
"directory: %s" % model_dir)
# if not output_node_names:
# print("You need to supply the name of a node to --output_node_names.")
# return -1
# We retrieve our checkpoint fullpath
checkpoint = tf.train.get_checkpoint_state(model_dir)
input_checkpoint = checkpoint.model_checkpoint_path
# We precise the file fullname of our freezed graph
absolute_model_dir = "/".join(input_checkpoint.split('/')[:-1])
output_graph = absolute_model_dir + "/frozen_model.pb"
# We clear devices to allow TensorFlow to control on which device it will load operations
clear_devices = True
# We start a session using a temporary fresh Graph
with tf.Session(graph=tf.Graph()) as sess:
# We import the meta graph in the current default Graph
saver = tf.train.import_meta_graph(input_checkpoint + '.meta', clear_devices=clear_devices)
# We restore the weights
saver.restore(sess, input_checkpoint)
# We use a built-in TF helper to export variables to constants
output_graph_def = tf.graph_util.convert_variables_to_constants(
sess, # The session is used to retrieve the weights
tf.get_default_graph().as_graph_def(), # The graph_def is used to retrieve the nodes
[n.name for n in tf.get_default_graph().as_graph_def().node] # The output node names are used to select the usefull nodes
)
# Finally we serialize and dump the output graph to the filesystem
with tf.gfile.GFile(output_graph, "wb") as f:
f.write(output_graph_def.SerializeToString())
print("%d ops in the final graph." % len(output_graph_def.node))
return output_graph_def
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument("--model_dir", type=str, default="", help="Model folder to export")
# parser.add_argument("--output_node_names", type=str, default="", help="The name of the output nodes, comma separated.")
args = parser.parse_args()
freeze_graph(args.model_dir)
I had to change few lines - remove --output_node_names and change output_node_names in output_graph_def to [n.name for n in tf.get_default_graph().as_graph_def().node]
Now i got new problems - I can't convert .pb to .pbtxt, and error is :
ValueError: Input 0 of node prefix/Variable/Assign was passed float from prefix/Variable:0 incompatible with expected float_ref.
And once again, information on this problem is outdated - everything i found is at least year old. I'm starting to think that fix for frozen_graph is not correct, and that is the reason why i'm having new error.
I would really appreciate some advice on this matter.

if you write
[n.name for n in tf.get_default_graph().as_graph_def().node]
in your convert_variables_to_constants function, you define every node the graph has as an output node, which of course will not work. (This is probably the reason for your ValueError)
You need to find the name of the real output node, the best way for this is often to look at the trained model in tensorboard and analyze the graph there, or you print out every node of your graph. Often the last node that is printed out is your output node (ignore everything that has 'gradients' in the name or 'Adam' if you have used that as an optimizer)
An easy way to do this (insert it after you restore the session):
gd = sess.graph.as_graph_def()
for node in gd.node:
print(node.name)

How to store model in `.pb` file with Estimator in TensorFlow?

I trained my model with estimator of TensorFlow. It seems that export_savedmodel should be used to make .pb file, but I don't really know how to construct the serving_input_receiver_fn. Anybody any ideas?
Example code is welcomed.
Extra questions:
Is .pb the only file I need when I want to reload the model? Variable unnecessary?
How much will .pb reduced the model file size compared with .ckpt with adam optimizer?

You can use freeze_graph.py to produce a .pb from .ckpt + .pbtxt
if you're using tf.estimator.Estimator, then you'll find these two files in the model_dir
python freeze_graph.py \
--input_graph=graph.pbtxt \
--input_checkpoint=model.ckpt-308 \
--output_graph=output_graph.pb
--output_node_names=<output_node>
Is .pb the only file I need when I want to reload the model? Variable unnecessary?
Yes, You'll have to know you're model's input nodes and output node names too. Then use import_graph_def to load the .pb file and get the input and output operations using get_operation_by_name
How much will .pb reduced the model file size compared with .ckpt with adam optimizer?
A .pb file is not a compressed .ckpt file, so there is no "compression rate".
However, there is a way to optimize your .pb file for inference, and this optimization may reduce the file size as it removes parts of the graph that are training only operations (see the complete description here).
[comment] how can I get the input and output node names?
You can set the input and output node names using the op name parameter.
To list the node names in your .pbtxt file, use the following script.
import tensorflow as tf
from google.protobuf import text_format
with open('graph.pbtxt') as f:
graph_def = text_format.Parse(f.read(), tf.GraphDef())
print [n.name for n in graph_def.node]
[comment] I found that there is a tf.estimator.Estimator.export_savedmodel(), is that the function to store model in .pb directly? And I'm struggling in it's parameter serving_input_receiver_fn. Any ideas?
export_savedmodel() generates a SavedModel which is a universal serialization format for TensorFlow models. It should contain everything's needed to fit with TensorFlow Serving APIs
serving_input_receiver_fn() is a part of those needed things you have to provide in order to generate a SavedModel, it determines the input signature of your model by adding placeholders to the graph.
From the doc
This function has the following purposes:
To add placeholders to the graph that the serving system will feed
with inference requests.
To add any additional ops needed to convert
data from the input format into the feature Tensors expected by the
model.
If you're receiving your inference requests in the form of serialized tf.Examples (which is a typical pattern) then you can use the example provided in the doc.
feature_spec = {'foo': tf.FixedLenFeature(...),
'bar': tf.VarLenFeature(...)}
def serving_input_receiver_fn():
"""An input receiver that expects a serialized tf.Example."""
serialized_tf_example = tf.placeholder(dtype=tf.string,
shape=[default_batch_size],
name='input_example_tensor')
receiver_tensors = {'examples': serialized_tf_example}
features = tf.parse_example(serialized_tf_example, feature_spec)
return tf.estimator.export.ServingInputReceiver(features, receiver_tensors)
[comment] Any idea to list the node names in '.pb'?
It depends on how it was generated.
if it's a SavedModel the use:
import tensorflow as tf
with tf.Session() as sess:
meta_graph_def = tf.saved_model.loader.load(
sess,
[tf.saved_model.tag_constants.SERVING],
'./saved_models/1519232535')
print [n.name for n in meta_graph_def.graph_def.node]
if it's a MetaGraph then use:
import tensorflow as tf
from tensorflow.python.platform import gfile
with tf.Session() as sess:
with gfile.FastGFile('model.pb', 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
sess.graph.as_default()
tf.import_graph_def(graph_def, name='')
print [n.name for n in graph_def.node]

Export Tensorflow Estimator

I'm trying to build a CNN with Tensorflow (r1.4) based on the API tf.estimator. It's a canned model. The idea is to train and evaluate the network with estimator in python and use the prediction in C++ without estimator by loading a pb file generated after the training.
My first question is, is it possible?
If yes, the training part works and the prediction part works too (with pb file generated without estimator) but it doesn't work when I load a pb file from estimator.
I got this error : "Data loss: Can't parse saved_model.pb as binary proto"
My pyhon code to export my model :
feature_spec = {'input_image': parsing_ops.FixedLenFeature(dtype=dtypes.float32, shape=[1, 48 * 48])}
export_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec)
input_fn = tf.estimator.inputs.numpy_input_fn(self.eval_features,
self.eval_label,
shuffle=False,
num_epochs=1)
eval_result = self.model.evaluate(input_fn=input_fn, name='eval')
exporter = tf.estimator.FinalExporter('save_model', export_input_fn)
exporter.export(estimator=self.model, export_path=MODEL_DIR,
checkpoint_path=self.model.latest_checkpoint(),
eval_result=eval_result,
is_the_final_export=True)
It doesn't work neither with tf.estimator.Estimator.export_savedmodel()
If one of you knows an explicit tutorial on estimator with canned model and how to export it, I'm interested

Please look at this issue on github, it looks like you have the same problem. Apparently (at least when using estimator.export_savedmodel) you should load the graph with LoadSavedModel instead of ReadBinaryProto, because it's not saved as a graphdef file.
You'll find here a bit more instructions about how to use it:
const string export_dir = ...
SavedModelBundle bundle;
...
LoadSavedModel(session_options, run_options, export_dir, {kSavedModelTagTrain},
&bundle);
I can't seem to find the SavedModelBundle documentation for c++ to use it afterwards, but it's likely close to the same class in Java, in which case it basically contains the session and the graph you'll be using.

Load / restore models into tensorflow at specific iteration or checkpoint

I have a model , which I am saving at every 10 iterations . So , i am having following files in my saved directory .
checkpoint model-50.data-00000-of-00001 model-50.index model-50.meta
model-60.data-00000-of-00001 model-60.index model-60.meta
and so on up to 100 . I have to load only the model-50. Because I have got
NaN values after 70 iterations. By deafault, when i am restoring the saver will look for the final checkpoint. So, how could I specifically load the model-50. please help, otherwise, i have to run the model gain from scratch, which is time consuming.

Since you are using tf.train.Saver's function restore(), you can make use of the last_checkpoints functions to get a list of all available checkpoints. You will see both model-50 and model-60 in this list.
Pick the correct model, and pass it directly to restore() like this,
saver.restore(sess, ckpt_path)

I'm not sure if things were different in the past, but at least as of now, you can use tf.train.get_checkpoint_state() to get CheckpointState proto which contains all_model_checkpoint_paths.

When you execute the command shown in most of the tutorials about saving/restoring a model saver.restore(sess, tf.train.latest_checkpoint(_dir_models)) the second parameter which you are passing is just a string to model path. This is defined in a saver.restore documentation.
save_path: Path where parameters were previously saved.
So you can path any string there and latest_checkpoint is just a convenient function to extract this path from a checkpoint file. Open this file in a notebook and you will see all the model paths available and what is the latest.
You can substitute that path with any path you want. You can get it from that file (either opening it manually or using get_checkpoin_state which will programmatically do it for you.

how make correct predictions of jpeg image in cloud-ml

I want to predict a jpeg image in cloud-ml.
My training model is the inception model, and I would like to send the input to the first layer of the graph: 'DecodeJpeg/contents:0' (where I have to send a jpeg image). I have set this layer as possible input by adding in retrain.py:
inputs = {'image_bytes': 'DecodeJpeg/contents:0'}
tf.add_to_collection('inputs', json.dumps(inputs))
Then I save the results of the training in two files (export and export.meta) with:
saver.save(sess, os.path.join(output_directory,'export'))
and I create a model in cloud-ml using these files.
As suggested in some posts (here, here, and here from Google cloud official blog) I'm trying to make the prediction with
gcloud beta ml predict --json-instances=request.json --model=MODEL
where the instance is the jpeg image decoded in base64 format with:
python -c 'import base64, sys, json; img = base64.b64encode(open(sys.argv[1], "rb").read()); print json.dumps({"key":"0", "image_bytes": {"b64": img}})' image.jpg &> request.json
However the request return me:
error: 'Prediction failed: '
What is the problem of my procedure? Do you have any suggestion?
I particular from this post I assume that cloud-ml automatically convert the base64 image in jpeg format when it reads a request with image_bytes. Is it correct? Otherwise how can I do?

CloudML requires you to feed the graph with a batch of images.
I'm pretty sure this is the issue with re-using retrain.py. See that code's sess.run line; it is feeding a single image at a time. Compare with the batched jpeg placeholder in the flowers sample.

Note that three slightly different TF graphs need to be constructed: Training, Evaluation, and Prediction. See this recent blog post for details. The training and Prediction graphs directly consume embedding from preprocessing so they do not contain an Inception graph. For prediction, we need to take image bytes as input and use Inception to extract embeddings.
For online prediction, you need to export the prediction graph.You should also specify the outputs and a key for inputs.
To build the prediction graph (the code):
def build_prediction_graph(self):
"""Builds prediction graph and registers appropriate endpoints."""
tensors = self.build_graph(None, 1, GraphMod.PREDICT)
keys_placeholder = tf.placeholder(tf.string, shape=[None])
inputs = {
'key': keys_placeholder.name,
'image_bytes': tensors.input_jpeg.name
}
tf.add_to_collection('inputs', json.dumps(inputs))
# To extract the id, we need to add the identity function.
keys = tf.identity(keys_placeholder)
outputs = {
'key': keys.name,
'prediction': tensors.predictions[0].name,
'scores': tensors.predictions[1].name
}
tf.add_to_collection('outputs', json.dumps(outputs))
To export the preciction graph:
def export(self, last_checkpoint, output_dir):
# Build and save prediction meta graph and trained variable values.
with tf.Session(graph=tf.Graph()) as sess:
self.build_prediction_graph()
init_op = tf.global_variables_initializer()
sess.run(init_op)
self.restore_from_checkpoint(sess, self.inception_checkpoint_file,
last_checkpoint)
saver = tf.train.Saver()
saver.export_meta_graph(filename=os.path.join(output_dir, 'export.meta'))
saver.save(sess, os.path.join(output_dir, 'export'), write_meta_graph=False)
last_checkpoint must point to the latest checkpoint file from training:
self.model.export(tf.train.latest_checkpoint(self.train_path), self.model_path)

In your post, you indicated that your inputs collection has only "image_bytes" tensor alias. However, in the code where you are framing the request, you are including 2 inputs: One is "key" and the other is "image_bytes". So, my suggestion would be to remove "key" from the request or add "key" to the inputs collection.
Second issue is that the shape of DecodeJpeg/contents:0', is (). For Cloud ML, you need to have a shape like (None, ) so that you can feed that in.
There are some suggestions in other answers to your question here, on how you might be able to follow the public posts to modify your graph, but at hand I can tell these two issues.
Let us know if you encounter any further issues.