We Keep Coding

Tensorflow weights not restored even after restoring from ckpt file

I am using the following code and the weights that are restored are random initializations and not the actual weights stored in ckpt file. Kindly help me in understanding where I went wrong.
best_val_model = 'val_E1_A86.ckpt'
model_dir = './models/'
with tf.Session(config = config) as sess:
sess.run(tf.global_variables_initializer())
print('Testing the model on 10000 Images!')
ckpt_file = os.path.join(model_dir, best_val_model)
saver = tf.train.import_meta_graph(ckpt_file)
saver.restore(sess, ckpt_file)
weights = {}
for v in tf.trainable_variables():
weights[v.name] = v.eval()
Actual model weights saved using tf.train.Saver(sess,filename) during training. While restoring, random weights are being restored.

I just had to remove two lines sess.run(tf.golbal_variables_initializer()) , saver = tf.train.import_meta_graph(ckpt_file) and it worked fine. Final Code used:
saver = tf.train.Saver(tf.trainable_variables())
with tf.Session(config = config) as sess:
sess.run(tf.local_variables_initializer())
print('Testing the model on 10000 Images!')
ckpt_file = os.path.join(model_dir, best_val_model)
saver.restore(sess, ckpt_file)

Restore a saved neural network in Tensorflow

Before marking my question as duplicate, I want you to understand that I have went through a lot of questions, but none of the solutions there were able to clear my doubts and solve my problem. I have a trained neural network which I want to save, and later use this model to test this model against test dataset.
I tried saving and restoring it, but I am not getting the expected results. Restoring doesn't seem to work, maybe I am using it wrongly, it is just using the values given by the global variable initializer.
This is the code I am using for saving the model.
sess.run(tf.initializers.global_variables())
#num_epochs = 7
for epoch in range(num_epochs):
start_time = time.time()
train_accuracy = 0
train_loss = 0
val_loss = 0
val_accuracy = 0
for bid in range(int(train_data_size/batch_size)):
X_train_batch = X_train[bid*batch_size:(bid+1)*batch_size]
y_train_batch = y_train[bid*batch_size:(bid+1)*batch_size]
sess.run(optimizer, feed_dict = {x:X_train_batch, y:y_train_batch,prob:0.50})
train_accuracy = train_accuracy + sess.run(model_accuracy, feed_dict={x : X_train_batch,y:y_train_batch,prob:0.50})
train_loss = train_loss + sess.run(loss_value, feed_dict={x : X_train_batch,y:y_train_batch,prob:0.50})
for bid in range(int(val_data_size/batch_size)):
X_val_batch = X_val[bid*batch_size:(bid+1)*batch_size]
y_val_batch = y_val[bid*batch_size:(bid+1)*batch_size]
val_accuracy = val_accuracy + sess.run(model_accuracy,feed_dict = {x:X_val_batch, y:y_val_batch,prob:0.75})
val_loss = val_loss + sess.run(loss_value, feed_dict = {x:X_val_batch, y:y_val_batch,prob:0.75})
train_accuracy = train_accuracy/int(train_data_size/batch_size)
val_accuracy = val_accuracy/int(val_data_size/batch_size)
train_loss = train_loss/int(train_data_size/batch_size)
val_loss = val_loss/int(val_data_size/batch_size)
end_time = time.time()
saver.save(sess,'./blood_model_x_v2',global_step = epoch)
After saving the model, the files are written in my working directory something like this.
blood_model_x_v2-2.data-0000-of-0001
blood_model_x_v2-2.index
blood_model_x_v2-2.meta
Similarly, v2-3, so on to v2-6, and then a 'checkpoint' file. I then tried restoring it using this code snippet (after initializing),but getting different results from the expected one. What am I doing wrong ?
saver = tf.train.import_meta_graph('blood_model_x_v2-5.meta')
saver.restore(test_session,tf.train.latest_checkpoint('./'))

According to tensorflow docs:
Restore
Restores previously saved variables.
This method runs the ops added by the constructor for restoring
variables. It requires a session in which the graph was launched. The
variables to restore do not have to have been initialized, as
restoring is itself a way to initialize variables.
Let's see an example:
We save the model similar to this:
import tensorflow as tf
# Prepare to feed input, i.e. feed_dict and placeholders
w1 = tf.placeholder("float", name="w1")
w2 = tf.placeholder("float", name="w2")
b1 = tf.Variable(2.0, name="bias")
feed_dict = {w1: 4, w2: 8}
# Define a test operation that we will restore
w3 = tf.add(w1, w2)
w4 = tf.multiply(w3, b1, name="op_to_restore")
sess = tf.Session()
sess.run(tf.global_variables_initializer())
# Create a saver object which will save all the variables
saver = tf.train.Saver()
# Run the operation by feeding input
print (sess.run(w4, feed_dict))
# Prints 24 which is sum of (w1+w2)*b1
# Now, save the graph
saver.save(sess, './ckpnt/my_test_model', global_step=1000)
And then load the trained model with:
import tensorflow as tf
sess = tf.Session()
# First let's load meta graph and restore weights
saver = tf.train.import_meta_graph('./ckpnt/my_test_model-1000.meta')
saver.restore(sess, tf.train.latest_checkpoint('./ckpnt'))
# Now, let's access and create placeholders variables and
# create feed-dict to feed new data
graph = tf.get_default_graph()
w1 = graph.get_tensor_by_name("w1:0")
w2 = graph.get_tensor_by_name("w2:0")
feed_dict = {w1: 13.0, w2: 17.0}
# Now, access the op that you want to run.
op_to_restore = graph.get_tensor_by_name("op_to_restore:0")
print (sess.run(op_to_restore, feed_dict))
# This will print 60 which is calculated
# using new values of w1 and w2 and saved value of b1.
As you can see we do not initialize our session in the restoring part. There is better way to save and restore model with Checkpoint which allows you to check whether the model is restored correctly or not.

How to restore pretrained model to initialize parameters

I have downloaded an network with its pretrained model. I added several layers and parameters to the network, I want to use this pretrained model to initialize the original parameters,and random initialize new added parameters by myself.I use this code:
saver = tf.train.Saver()
with tf.Session() as sess:
saver.restore(sess, "output/saver-test")
sess.run(tf.global_variables_initializer())
sess.run(tf.local_variables_initializer())
but I met the error:"Key global_step not found in checkpoint",this error because I have some new parameters that didn't exist in pretrained model.But how can I solve this problem? What's more,I want to use this code "sess.run(tf.global_variables_initializer())" to initialize the new added parameters,but the extracted parameters from pretrained model will be covered by it?

It happens because of your network is not perfectly match to the loaded one.
You can use selective checkpoint loader something like that:
reader = tf.train.NewCheckpointReader(os.path.join(checkpoint_dir, ckpt_name))
restore_dict = dict()
for v in tf.trainable_variables():
tensor_name = v.name.split(':')[0]
if reader.has_tensor(tensor_name):
print('has tensor ', tensor_name)
restore_dict[tensor_name] = v
restore_dict['my_new_var_scope/my_new_var'] = self.get_my_new_var_variable()
Where get_my_new_var_variable() is something like that:
def get_my_new_var_variable(self):
with tf.variable_scope("my_new_var_scope",reuse=tf.AUTO_REUSE):
my_new_var = tf.get_variable("my_new_var", dtype=tf.int32,initializer=tf.constant([23, 42]))
return my_new_var
Loading the weights:
self.saver = tf.train.Saver(restore_dict)
self.saver.restore(self.sess, os.path.join(checkpoint_dir, ckpt_name))
Edited:
Note that in order to avoid override the loaded variables you can use this method:
def initialize_uninitialized(sess):
global_vars = tf.global_variables()
is_not_initialized = sess.run([tf.is_variable_initialized(var) for var in global_vars])
not_initialized_vars = [v for (v, f) in zip(global_vars, is_not_initialized) if not f]
if len(not_initialized_vars):
sess.run(tf.variables_initializer(not_initialized_vars))
Or simply calling tf.global_variables_initializer() before loading the variables should work here.

TensorFlow: Restoring a model

I'm trying to save my model at the end of triaining and restore it every time the training begins. I just followed what this link did.
saver = tf.train.Saver()
with tf.Session(graph=graph) as session:
# Initializate the weights and biases
tf.global_variables_initializer().run()
new_saver = tf.train.import_meta_graph('model.meta')
new_saver.restore(sess,tf.train.latest_checkpoint('./'))
W1 = session.run(W)
print(W1)
for curr_epoch in range(num_epochs):
train_cost = train_ler = 0
start = time.time()
for batch in range(num_batches_per_epoch):
...Some training...
W2 = session.run(W)
print(W2)
save_path = saver.save(session, "models/model")
But it gives error below:
---> new_saver.restore(session, tf.train.latest_checkpoint('./'))
SystemError: <built-in function TF_Run> returned a result with an error set
Can anyone help me please? Many thanks!

If you're gonna load with ./ you have to make sure, that your console (that you use to start the python program) is actually set on that directory (models/).
But in that case, it will save your new data in a new directory. So load with ./models/ instead
(Also you don't need to initiate variables, the restore does that for you.)

restoring weights of an already saved Tensorflow .pb model

I have seen many posts about restoring an already saved TF models here, but none could answer my question. Using TF 1.0.0
Specifically, I am interested in seeing the weights for inceptionv3 model which is publicly available in .pb file here. I managed to restore it back using a small chunk of Python code and can access the graphs high-level view in tensorboard:
from tensorflow.python.platform import gfile
INCEPTION_LOG_DIR = '/tmp/inception_v3_log'
if not os.path.exists(INCEPTION_LOG_DIR):
os.makedirs(INCEPTION_LOG_DIR)
with tf.Session() as sess:
model_filename = './model/tensorflow_inception_v3_stripped_optimized_quantized.pb'
with gfile.FastGFile(model_filename, 'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
_= tf.import_graph_def(graph_def,name='')
writer = tf.train.SummaryWriter(INCEPTION_LOG_DIR, graph_def)
writer=tf.summary.FileWriter(INCEPTION_LOG_DIR, graph_def)
writer.close()
However, I failed to access any layers' weights.
tensors= tf.import_graph_def(graph_def,name='')
returns empty, even if I add the arbitrary return_elements=. Does it have any weights at all? If yes, what is the appropriate procedure here? Thanks.

use this code to print your tensor's value :
with tf.Session() as sess:
print sess.run('your_tensor_name')
you can use this code to retrieve tensor names:
op = sess.graph.get_operations()
for m in op :
print(m.values())

There is a difference between restoring weights and printing them. The former one denotes that one would like to import the weight values from already saved ckpt files for retraining or inference while the latter may be for inspection. Also .pb file encodes model parameters as tf.constant() ops. As a result, the model parameters would not appear in tf.trainable_variables(), hence you can't use .pb directly to restore the weights. From your question I take that you just want to 'see' the weights for inspection.
Let us first load the graph from .pb file.
import tensorflow as tf
from tensorflow.python.platform import gfile
GRAPH_PB_PATH = './model/tensorflow_inception_v3_stripped_optimized_quantized.pb' #path to your .pb file
with tf.Session(config=config) as sess:
print("load graph")
with gfile.FastGFile(GRAPH_PB_PATH,'rb') as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
sess.graph.as_default()
tf.import_graph_def(graph_def, name='')
graph_nodes=[n for n in graph_def.node]
Now when you freeze a graph to .pb file your variables are converted to Const type and the weights which were trainabe variables would also be stored as Const in .pb file. graph_nodes contains all the nodes in graph. But we are interested in all the Const type nodes.
wts = [n for n in graph_nodes if n.op=='Const']
Each element of wts is of NodeDef type. It has several atributes such as name, op etc. The values can be extracted as follows -
from tensorflow.python.framework import tensor_util
for n in wts:
print "Name of the node - %s" % n.name
print "Value - "
print tensor_util.MakeNdarray(n.attr['value'].tensor)
Hope this solves your concern.

You can use this code to get the names of tensor.
[tensor.name for tensor in tf.get_default_graph().as_graph_def().node]

Just small utils to print .pb model weights:
import argparse
import tensorflow as tf
from tensorflow.python.framework import tensor_util
def print_pb_weights(pb_filepath):
graph_def = tf.GraphDef()
with tf.gfile.GFile(pb_filepath, "rb") as f:
graph_def.ParseFromString(f.read())
tf.import_graph_def(graph_def, name='')
for node in graph_def.node:
if node.op == 'Const':
print('-' * 60)
print('op:', node.op)
print('name:', node.name)
arr = tensor_util.MakeNdarray(node.attr['value'].tensor)
print('shape:', list(arr.shape))
print(arr)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('pb_filepath')
args = parser.parse_args()
print_pb_weights(args.pb_filepath)