I am trying to download embeddings from a trained GCMLE prediction model locally so that I can play with my own custom embedding visualizations that aren't available in tensorboard. I'd like to extract these embeddings into a largy numpy matrix, but I'm having trouble with a few steps. I can successfully download all of the files (saved_model.pb + assets/* + variables/*, and I appear to be able to restore the model with the following code:
with tf.Session(graph=tf.Graph()) as sess:
tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], _EXPORT_DIR)
which successfully returns:
INFO:tensorflow:Restoring parameters from Servo/variables/variables
I then tried to extract the weights like this:
constant_values = {}
with tf.Session(graph=tf.Graph()) as sess:
tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], _EXPORT_DIR)
constant_ops = [op for op in sess.graph.get_operations() if op.type == "Const"]
for constant_op in constant_ops:
constant_values[constant_op.name] = sess.run(constant_op.outputs[0])
which did succesfully output quite a lot, but the only parts relevant to the embedding were:
u'embedding_layer/embeddings/Initializer/random_uniform/max': 0.012765553,
u'embedding_layer/embeddings/Initializer/random_uniform/min': -0.012765553,
u'embedding_layer/embeddings/Initializer/random_uniform/shape': array([vocab_size, word_embedding_size], dtype=int32)
and no sign of the actual embedding weights. How can I modify my approach above to get the actual embedding weight matrix?
It will depend a bit on how you exported model, but in most cases, the embeddings are Variables and not constants. So you want something like:
with tf.Session(graph=tf.Graph()) as sess:
tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], _EXPORT_DIR)
trainable_coll = sess.graph.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)
vars = {v.name:sess.run(v.value()) for v in trainable_coll}
Related
Problem:
I am very new to Tensorflow. My specific question is what particular arguments should I put inside sess.run(fetches, feed_dict) function. For instance, how could find out what the values of the arguments?
Steps:
Here is my understanding of the steps after looking at other posts.
Save tranied tensorflow model, it should consists of 4 files, below are my outputs:
checkpoint
Inception_resnet_v2.ckpt.data-00000-of-00001
Inception_resnet_v2.ckpt.index
Inception_resnet_v2.ckpt.meta
Resize the input image to whatever format required by the neural network.
Start tensorflow session.
Retrive the Graph and associated parameters, tensors...
Predict the input image.
Code：
Traning code:
https://github.com/taki0112/SENet-Tensorflow/blob/master/SE_Inception_resnet_v2.py
[Solved] Test code:
import tensorflow as tf
import numpy as np
import cv2
labels = ["airplane","automobile","bird","cat","deer","dog","frog","horse","ship","truck"]
# Load graph and parameters, etc.
sess=tf.Session()
saver = tf.train.import_meta_graph('./model/Inception_resnet_v2.ckpt.meta')
saver.restore(sess, tf.train.latest_checkpoint("./model/"))
graph = tf.get_default_graph()
# Get tensor names
x = graph.get_tensor_by_name("Placeholder:0")
training_flag = graph.get_tensor_by_name("Placeholder_2:0")
op_to_restore = graph.get_tensor_by_name("final_fully_connected/dense/BiasAdd:0")
# Preprocess imgae imput
src = cv2.imread("./input/car3.jpg")
dst = cv2.resize(src, (32, 32), interpolation=cv2.INTER_CUBIC)
b,g,r = cv2.split(dst)
b = (b - np.mean(b)) / np.std(b) * .1
g = (g - np.mean(g)) / np.std(g) * .1
r = (r - np.mean(r)) / np.std(r) * .1
src = cv2.merge((b,g,r))
picture = dst.reshape(1, 32, 32, 3)
feed_dict ={x: picture, training_flag:False}
result_index = sess.run(op_to_restore,feed_dict)
print(result_index)
print (labels[np.argmax(result_index)])
the arguments actually depend on what you're doing, but mostly the first argument is the weights and placeholders. Whenever you are working with Tensorflow, you define a graph which is fed examples(training data) and some hyperparameters like learning rate, global step etc. It’s a standard practice to feed all the training data and hyperparameters using placeholders. when you build a network using placeholders and save it the network is saved, however, values of the placeholders are not saved.
Let's see a toy example:
import tensorflow as tf
#Prepare to feed input, i.e. feed_dict and placeholders
w1 = tf.placeholder("float", name="w1")
w2 = tf.placeholder("float", name="w2")
b1= tf.Variable(2.0,name="bias")
feed_dict ={w1:4,w2:8}
#Define a test operation that we will restore
w3 = tf.add(w1,w2)
w4 = tf.multiply(w3,b1,name="op_to_restore")
sess = tf.Session()
sess.run(tf.global_variables_initializer())
#Create a saver object which will save all the variables
saver = tf.train.Saver()
#Run the operation by feeding input
print sess.run(w4,feed_dict)
#Prints 24 which is sum of (w1+w2)*b1
#Now, save the graph
saver.save(sess, 'my_test_model',global_step=1000)
Now, when we want to restore it, we not only have to restore the graph and weights, but also prepare a new feed_dict that will feed the new training data to the network. We can get reference to these saved operations and placeholder variables via graph.get_tensor_by_name() method. So if you want to train the same model with further new data, then you would have to utilize those weigtages, if however you just want to get the prediction from the model you trained, you could utilize the op_to_restore and the feed_dict as new data. Something like this, if you follow the above example:
import tensorflow as tf
sess=tf.Session()
#First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('my_test_model-1000.meta')
saver.restore(sess,tf.train.latest_checkpoint('./'))
# Now, let's access and create placeholders variables and
# create feed-dict to feed new data
graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict ={w1:13.0,w2:17.0}
#Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")
print sess.run(op_to_restore,feed_dict)
#This will print 60 which is calculated
#using new values of w1 and w2 and saved value of b1.
So, this is how it works, in your case, since you're trying to load the Inception model, your op_to_restore should depend on what you're trying to restore if you could tell us what you're trying to do, then only it's possible to suggest something. However in the other parameter feed_dict , it's just the numpy array of image pixel, of you, you're trying to classify/predict or whatever you're doing.
I took the code from the following article. This will help you as well. http://cv-tricks.com/tensorflow-tutorial/save-restore-tensorflow-models-quick-complete-tutorial/
Update: For your particular case, you may like to try the following code to predict the classes in the new images.
import tensorflow as tf
slim = tf.contrib.slim
from inception_resnet_v2 import *
#Well, since you're using resnet_v2, this may be equivalent to you.
checkpoint_file = 'inception_resnet_v2_2016_08_30.ckpt'
sample_images = ['dog.jpg', 'panda.jpg']
#Load the model
sess = tf.Session()
arg_scope = inception_resnet_v2_arg_scope()
with slim.arg_scope(arg_scope):
logits, end_points = inception_resnet_v2(input_tensor, is_training=False)
#With this, you could consider the op_variable with the following
predict_values, logit_values = sess.run([end_points['Predictions'], logits], feed_dict={input_tensor: im})
#Here im is the normalized numpy array of the image pixels.
Furthermore, the following resources may help you even more:
Using pre-trained inception_resnet_v2 with Tensorflow
https://github.com/tensorflow/tensorflow/issues/7172
I fine-tuned InceptionNet v3 model given in the slim-tensorflow library. I trained the model on my own dataset. Now, I have .ckpt and .meta file for the model.
Now, as far as from I understand, I have two ways to restore the model and the weights. First, from the .meta file like this
checkpoint = './fine_tuned_model/model.ckpt-233700.meta'
with tf.Session() as sess:
new_saver = tf.train.import_meta_graph(checkpoint)
print(new_saver)
new_saver.restore(sess, tf.train.latest_checkpoint('./fine_tuned_model/'))
The second way is to call the model and restore the checkpoint. Like this
with slim.arg_scope(slim.nets.inception.inception_v3_arg_scope()):
logits, inceptionv3 = nets.inception.inception_v3(inputs=img, num_classes=5980, is_training=True,
dropout_keep_prob=.6)
# Restore convolutional layers:
variables_to_restore = slim.get_variables_to_restore(exclude=['InceptionV3/Logits', 'InceptionV3/AuxLogits'])
init_fn = slim.assign_from_checkpoint_fn(model_path, variables_to_restore)
Now, I think for my purpose second way is easier.
Now, my issue is that after restoring the model, how can I extract features from the last to last layer given an Image? I have included a screenshot of the model here
One solution I found was like this and here as far as I know I have to extract features from PreLogits layer but I am not sure how to set the values here
with tf.Session() as sess:
feature_tensor = sess.graph.get_tensor_by_name('mul:0')
features_last_layer = sess.run(feature_tensor,{inputs: img})
print features_last_layer
Here, I am unable to find out what should I pass to get_tenor_by_name(??) and also, how pass sess.run here?
Thank you. Please let me know if there are other ways to restore the model and get the features.
I have figured out the solution for this.
# Placeholder for the image,
image_tensor = tf.placeholder(tf.float32, shape=(None, 128, 128, 3))
# load the model
with slim.arg_scope(slim.nets.inception.inception_v3_arg_scope()):
logits, inceptionv3 = nets.inception.inception_v3(image_tensor,is_training=False)
# Restore convolutional layers:
variables_to_restore = slim.get_variables_to_restore(exclude=['InceptionV3/Logits', 'InceptionV3/AuxLogits'])
# get the latest checkpoint you want to use after training
init_fn = slim.assign_from_checkpoint_fn(model_path, variables_to_restore)
checkpoint = './fine_tuned_model/model.ckpt-233700.data-00000-of-00001'
saver = tf.train.Saver(variables_to_restore)
saver.restore(sess, tf.train.latest_checkpoint('./fine_tuned_model/'))
# the required image
img_car = cv2.imread('car.jpeg')
img_car = cv2.resize(img_car,(128, 128))
imgnumpy = np.ndarray((1,128,128,3))
imgnumpy[0] = img_car
# get output from any layer you want.
output = sess.run(inceptionv3["PreLogits"], feed_dict={image_tensor: imgnumpy})
I'm using the retrain script provided in the image_retraining folder from the tensorflow repository.
One of the parser arguments/flags let you store intermediate graphs every X steps
parser.add_argument(
'--intermediate_output_graphs_dir',
type=str,
default='tf_files2/tmp/intermediate_graph/',
help='Where to save the intermediate graphs.'
However, this seems to store the graph as a frozen graph with .pb extension.
There is very little information on how to properly load a .pb file to continue training.
Most of the info I found uses .meta graphs and .ckpts.
Is .pb going to be deprecated?
If so, should I just retrain the model from start and use a tf.Saver to get
.meta and ckpt graphs as intermediate checkpoints?
Yesterday, I was training a model and for some reason training froze, so I want to load up the intermediate graph, and continue training.
I'm using the inception model for retraining:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/image_retraining/retrain.py
If anyone could point me or show me how to properly load up a .pb intermediate graph (step by step) and continue from where I left off -- I would really appreciate it.
Thank you.
EDIT:
#Mingxing
So I'm assuming I should just let retrain.py first create the default graph based on default inception model (This function below) and then just overwrite it with the loaded graph?
def create_model_graph(model_info):
""""Creates a graph from saved GraphDef file and returns a Graph object.
Args:
model_info: Dictionary containing information about the model architecture.
Returns:
Graph holding the trained Inception network, and various tensors we'll be
manipulating.
"""
with tf.Graph().as_default() as graph:
model_path = os.path.join(FLAGS.model_dir, model_info['model_file_name'])
with gfile.FastGFile(model_path, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
bottleneck_tensor, resized_input_tensor = (tf.import_graph_def(
graph_def,
name='',
return_elements=[
model_info['bottleneck_tensor_name'],
model_info['resized_input_tensor_name'],
]))
return graph, bottleneck_tensor, resized_input_tensor
EDIT_2:
An error I'm getting is:
ValueError: Tensor("second_to_final_fC_layer_ops/weights/final_weights_1:0", shape=(2048, 102
4), dtype=float32_ref) must be from the same graph as Tensor("BottleneckInputPlaceholder:0",
shape=(?, 2048), dtype=float32).
I had added an extra FC layer after the first FC layer.
So 2048 -> 1024 -> num of classes prior to tranining.
When training the model I had no problems, but now loading the graph I seem to be running into this error above.
This is how the added layer looks:
layer_name = 'second_to_final_fC_layer_ops'
with tf.name_scope(layer_name):
with tf.name_scope('weights'):
initial_value = tf.truncated_normal(
[bottleneck_tensor_size, 1024], stddev=0.001)
layer_weights = tf.Variable(initial_value, name='weights')
variable_summaries(layer_weights)
with tf.name_scope('biases'):
layer_biases = tf.Variable(tf.zeros([1024]), name='biases')
variable_summaries(layer_biases)
with tf.name_scope('Wx_plus_b'):
logits = tf.matmul(bottleneck_input, layer_weights) + layer_biases
tf.summary.histogram('pre_activations', logits)
with tf.name_scope('Relu_activation'):
relu_activated =tf.nn.relu(logits, name= 'Relu')
tf.summary.histogram('final_relu_activation', relu_activated)
And then the final layer (which was the original final layer, but now the inputs are the outputs from last layer instead of the bottleneck tensor):
layer_name = 'final_training_ops'
with tf.name_scope(layer_name):
with tf.name_scope('weights'):
initial_value = tf.truncated_normal(
[1024, class_count], stddev=0.001)
layer_weights = tf.Variable(initial_value, name='final_weights')
variable_summaries(layer_weights)
with tf.name_scope('biases'):
layer_biases = tf.Variable(tf.zeros([class_count]), name='final_biases')
variable_summaries(layer_biases)
with tf.name_scope('Wx_plus_b'):
logits = tf.matmul(relu_activated, layer_weights) + layer_biases
tf.summary.histogram('pre_activations', logits)
final_tensor = tf.nn.softmax(logits, name=final_tensor_name)
tf.summary.histogram('activations', final_tensor)
EDIT: Still don't know how to load the weights-- Loading the graph structure seems to be easy, but I don't know how to load the weights and inputs of Inception that has been used trained once again using transfer learning.
A clear example using the weights and variables from image_retraining/retrain.py would be really helpful. Thank you.
You can use tf.import_graph_def to import your frozen .pb file:
# Read the .pb file into graph_def.
with tf.gfile.GFile(FLAGS.graph, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
# Restore the graph.
with tf.Graph().as_default() as graph:
tf.import_graph_def(graph_def, name="")
# After this, graph is the what you need.
Although there is nothing wrong to directly use frozen .pb file, I still want to point out that the recommended way is to follow the standard save/restore (official doc).
I have trained a Tensorflow Model, and now I want to export the "function" to use it in my python program. Is that possible, and if yes, how? Any help would be nice, could not find much in the documentation. (I dont want to save a session!)
I have now stored the session as you suggested. I am loading it now like this:
f = open('batches/batch_9.pkl', 'rb')
input = pickle.load(f)
f.close()
sess = tf.Session()
saver = tf.train.Saver()
saver.restore(sess, 'trained_network.ckpt')
y_pred = []
sess.run(y_pred, feed_dict={x: input})
print(y_pred)
However, I get the error "no Variables to save" when I try to initialize the saver.
What I want to do is this: I am writing a bot for a board game, and the input is the situation on the board formatted into a tensor. Now I want to return a tensor which gives me the best position to play next, i.e. a tensor with 0 everywhere and a 1 at one position.
I don't know if there is any other way to do it, but you can use your model in another Python program by saving your session:
Your training code:
# build your model
sess = tf.Session()
# train your model
saver = tf.train.Saver()
saver.save(sess, 'model/model.ckpt')
In your application:
# build your model (same as training)
sess = tf.Session()
saver = tf.train.Saver()
saver.restore(sess, 'model/model.ckpt')
You can then evaluate any tensor in your model using a feed_dict. This obviously depends on your model. For example:
#evaluate tensor
sess.run(y_pred, feed_dict={x: input_data})
I've recently reviewed an interesting implementation for convolutional text classification. However all TensorFlow code I've reviewed uses a random (not pre-trained) embedding vectors like the following:
with tf.device('/cpu:0'), tf.name_scope("embedding"):
W = tf.Variable(
tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0),
name="W")
self.embedded_chars = tf.nn.embedding_lookup(W, self.input_x)
self.embedded_chars_expanded = tf.expand_dims(self.embedded_chars, -1)
Does anybody know how to use the results of Word2vec or a GloVe pre-trained word embedding instead of a random one?
There are a few ways that you can use a pre-trained embedding in TensorFlow. Let's say that you have the embedding in a NumPy array called embedding, with vocab_size rows and embedding_dim columns and you want to create a tensor W that can be used in a call to tf.nn.embedding_lookup().
Simply create W as a tf.constant() that takes embedding as its value:
W = tf.constant(embedding, name="W")
This is the easiest approach, but it is not memory efficient because the value of a tf.constant() is stored multiple times in memory. Since embedding can be very large, you should only use this approach for toy examples.
Create W as a tf.Variable and initialize it from the NumPy array via a tf.placeholder():
W = tf.Variable(tf.constant(0.0, shape=[vocab_size, embedding_dim]),
trainable=False, name="W")
embedding_placeholder = tf.placeholder(tf.float32, [vocab_size, embedding_dim])
embedding_init = W.assign(embedding_placeholder)
# ...
sess = tf.Session()
sess.run(embedding_init, feed_dict={embedding_placeholder: embedding})
This avoid storing a copy of embedding in the graph, but it does require enough memory to keep two copies of the matrix in memory at once (one for the NumPy array, and one for the tf.Variable). Note that I've assumed that you want to hold the embedding matrix constant during training, so W is created with trainable=False.
If the embedding was trained as part of another TensorFlow model, you can use a tf.train.Saver to load the value from the other model's checkpoint file. This means that the embedding matrix can bypass Python altogether. Create W as in option 2, then do the following:
W = tf.Variable(...)
embedding_saver = tf.train.Saver({"name_of_variable_in_other_model": W})
# ...
sess = tf.Session()
embedding_saver.restore(sess, "checkpoint_filename.ckpt")
I use this method to load and share embedding.
W = tf.get_variable(name="W", shape=embedding.shape, initializer=tf.constant_initializer(embedding), trainable=False)
2.0 Compatible Answer: There are many Pre-Trained Embeddings, which are developed by Google and which have been Open Sourced.
Some of them are Universal Sentence Encoder (USE), ELMO, BERT, etc.. and it is very easy to reuse them in your code.
Code to reuse the Pre-Trained Embedding, Universal Sentence Encoder is shown below:
!pip install "tensorflow_hub>=0.6.0"
!pip install "tensorflow>=2.0.0"
import tensorflow as tf
import tensorflow_hub as hub
module_url = "https://tfhub.dev/google/universal-sentence-encoder/4"
embed = hub.KerasLayer(module_url)
embeddings = embed(["A long sentence.", "single-word",
"http://example.com"])
print(embeddings.shape) #(3,128)
For more information the Pre-Trained Embeddings developed and open-sourced by Google, refer TF Hub Link.
The answer of #mrry is not right because it provoques the overwriting of the embeddings weights each the network is run, so if you are following a minibatch approach to train your network, you are overwriting the weights of the embeddings. So, on my point of view the right way to pre-trained embeddings is:
embeddings = tf.get_variable("embeddings", shape=[dim1, dim2], initializer=tf.constant_initializer(np.array(embeddings_matrix))
With tensorflow version 2 its quite easy if you use the Embedding layer
X=tf.keras.layers.Embedding(input_dim=vocab_size,
output_dim=300,
input_length=Length_of_input_sequences,
embeddings_initializer=matrix_of_pretrained_weights
)(ur_inp)
I was also facing embedding issue, So i wrote detailed tutorial with dataset.
Here I would like to add what I tried You can also try this method,
import tensorflow as tf
tf.reset_default_graph()
input_x=tf.placeholder(tf.int32,shape=[None,None])
#you have to edit shape according to your embedding size
Word_embedding = tf.get_variable(name="W", shape=[400000,100], initializer=tf.constant_initializer(np.array(word_embedding)), trainable=False)
embedding_loopup= tf.nn.embedding_lookup(Word_embedding,input_x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for ii in final_:
print(sess.run(embedding_loopup,feed_dict={input_x:[ii]}))
Here is working detailed Tutorial Ipython example if you want to understand from scratch , take a look .