I have trained a Tensorflow Model, and now I want to export the "function" to use it in my python program. Is that possible, and if yes, how? Any help would be nice, could not find much in the documentation. (I dont want to save a session!)
I have now stored the session as you suggested. I am loading it now like this:
f = open('batches/batch_9.pkl', 'rb')
input = pickle.load(f)
f.close()
sess = tf.Session()
saver = tf.train.Saver()
saver.restore(sess, 'trained_network.ckpt')
y_pred = []
sess.run(y_pred, feed_dict={x: input})
print(y_pred)
However, I get the error "no Variables to save" when I try to initialize the saver.
What I want to do is this: I am writing a bot for a board game, and the input is the situation on the board formatted into a tensor. Now I want to return a tensor which gives me the best position to play next, i.e. a tensor with 0 everywhere and a 1 at one position.
I don't know if there is any other way to do it, but you can use your model in another Python program by saving your session:
Your training code:
# build your model
sess = tf.Session()
# train your model
saver = tf.train.Saver()
saver.save(sess, 'model/model.ckpt')
In your application:
# build your model (same as training)
sess = tf.Session()
saver = tf.train.Saver()
saver.restore(sess, 'model/model.ckpt')
You can then evaluate any tensor in your model using a feed_dict. This obviously depends on your model. For example:
#evaluate tensor
sess.run(y_pred, feed_dict={x: input_data})
Related
I have a TensorFlow model that uses tf.data.Dataset feedable iterators to switch between training and validation. Both dataset share the same structure, that is they have a features matrix and the corresponding labels vector. In order to use the same model and iterator for inference (no labels vector only featurex matrix) I need to ideally supply a zero labels vector. Is there a more efficient and elegant way to use the dataset API for both training (validation) and inference?
In code:
training_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train))
validation_dataset = tf.data.Dataset.from_tensor_slices((X_validation, y_validation))
handle = tf.placeholder(tf.string, shape=[])
iterator = tf.data.Iterator.from_string_handle(handle, training_dataset.output_types, training_dataset.output_shapes)
features, labels = iterator.get_next()
Features and lables are used inside the model as input placeholders.
In order to switch between dataset I need to create one iterator for each dataset:
training_iterator = training_dataset.make_initializable_iterator()
validation_iterator = validation_dataset.make_initializable_iterator()
then create the handle
training_handle = sess.run(training_iterator.string_handle())
validation_handle = sess.run(validation_iterator.string_handle())
And use the handle to select which dataset to use, for example:
sess.run(next_element, feed_dict={handle: training_handle})
Now, what happens if I have inference data with no labels?
inference_dataset = tf.data.Dataset.from_tensor_slices(X_inference) # NO y values
inferece_iterator = inference_dataset.make_initializable_iterator()
If I add this iterator it will throw and exception because "Number of components does not match: expected 2 types but got 1."
Any suggestions?
This post How to use tf.Dataset design in both training and inferring? is related to this question, but tf.data.Dataset does not have an unzip method.
What are the best practices for this problem?
If your graph code I assume you are trying to extract a value for labels y from the dataset right? At inference time that was probably baked into the tensorflow dependency graph.
You have a few choices here. Probably the easiest solution is to recreate the graph from code (run your build_graph() function, then load the weights using something like saver.restore(sess, "/tmp/model.ckpt")). If you do it this way you can re-create the graph without the labels y. I assume there are no other dependencies on y (sometimes tensorboard summaries add dependencies you need to check too). Your problem should now be solved.
However, now that I've written the above comment (which I'll leave as-is because it's still useful information), I realize you might not even need that. At inference time you should not be using the labels anywhere (again, double check tensorboard summaries). If you don't need y then tensorflow should not run any of the operations that use y. This should include not trying to extract them from the dataset. Double check that you are not asking tensorflow to use your labels anywhere at inference time.
I think that the first solution proposed by David Parks looks like this, and I think is better than messing with tf.cond in the code.
import tensorflow as tf
import numpy as np
def build_model(features, labels=None, train=False):
linear_model = tf.layers.Dense(units=1)
y_pred = linear_model(features)
if train:
loss = tf.losses.mean_squared_error(labels=labels, predictions=y_pred)
optimizer = tf.train.GradientDescentOptimizer(1e-4)
train = optimizer.minimize(loss)
return train, loss
else:
return y_pred
X_train = np.random.random(100).reshape(-1, 1)
y_train = np.random.random(100).reshape(-1, 1)
training_dataset = tf.data.Dataset.from_tensor_slices((X_train, y_train))
training_dataset = training_dataset.batch(10)
training_dataset = training_dataset.shuffle(20)
handle = tf.placeholder(tf.string, shape=[])
iterator = tf.data.Iterator.from_string_handle(handle, training_dataset.output_types, training_dataset.output_shapes)
features, labels = iterator.get_next()
training_iterator = training_dataset.make_one_shot_iterator()
train, loss = build_model(features, labels, train=True)
saver = tf.train.Saver()
init = tf.global_variables_initializer()
sess = tf.Session()
training_handle = sess.run(training_iterator.string_handle())
sess.run(init)
for i in range(10):
_, loss_value = sess.run((train, loss), feed_dict={handle: training_handle})
print(loss_value)
saver.save(sess, "tmp/model.ckpt")
sess.close()
tf.reset_default_graph()
X_test = np.random.random(10).reshape(-1, 1)
inference_dataset = tf.data.Dataset.from_tensor_slices(X_test)
inference_dataset = inference_dataset.batch(5)
handle = tf.placeholder(tf.string, shape=[])
iterator_inference = tf.data.Iterator.from_string_handle(handle, inference_dataset.output_types, inference_dataset.output_shapes)
inference_iterator = inference_dataset.make_one_shot_iterator()
features_inference = iterator_inference.get_next()
y_pred = build_model(features_inference)
saver = tf.train.Saver()
sess = tf.Session()
inference_handle = sess.run(inference_iterator.string_handle())
saver.restore(sess, "tmp/model.ckpt") # Restore variables from disk.
print(sess.run(y_pred, feed_dict={handle: inference_handle}))
sess.close()
I'm trying to build an object detector with CNN using tensorflow with python framework. I would like to train my model to do just object recognition (classification) at first and then using several convolutional layers of the pretarined model train it to predict bounding boxes. I will need to replace fully connected layers and probably some last convolutional layers. So, for this reason, I would like to know if it is possible to import only weights from tensorflow graph that was used to train object classifier to a newly defined graph that I will train to do object detection. So basically I would like to do something like this:
# here I initialize the new graph
conv_1=tf.nn.conv2d(in, weights_from_old_graph)
conv_2=tf.nn.conv2d(conv_1, weights_from_old_graph)
...
conv_n=tf.nn.nnconv2d(conv_n-1,randomly_initialized_weights)
fc_1=tf.matmul(conv_n, randomly_initalized_weights)
Use saver with no arguments to save the entire model.
tf.reset_default_graph()
v1 = tf.get_variable("v1", [3], initializer = tf.initializers.random_normal)
v2 = tf.get_variable("v2", [5], initializer = tf.initializers.random_normal)
saver = tf.train.Saver()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
saver.save(sess, save_path='./test-case.ckpt')
print(v1.eval())
print(v2.eval())
saver = None
v1 = [ 2.1882825 1.159807 -0.26564872]
v2 = [0.11437789 0.5742971 ]
Then in the model you want to restore to certain values, pass a list of variable names you want to restore or a dictionary of {"variable name": variable} to the Saver.
tf.reset_default_graph()
b1 = tf.get_variable("b1", [3], initializer= tf.initializers.random_normal)
b2 = tf.get_variable("b2", [3], initializer= tf.initializers.random_normal)
saver = tf.train.Saver(var_list={'v1': b1})
with tf.Session() as sess:
saver.restore(sess, "./test-case.ckpt")
print(b1.eval())
print(b2.eval())
INFO:tensorflow:Restoring parameters from ./test-case.ckpt
b1 = [ 2.1882825 1.159807 -0.26564872]
b2 = FailedPreconditionError: Attempting to use uninitialized value b2
Although I agree with Aechlys to restore variables. The problem is harder when we want to fix these variables. For example, we trained these variables and we want to use them in another model, but this time without training them (training new variables like in transfer-learning). You can see the answer I posted here.
Quick example:
with tf.session() as sess:
new_saver = tf.train.import_meta_graph(pathToMeta)
new_saver.restore(sess, pathToNonMeta)
weight1 = sess.run(sess.graph.get_tensor_by_name("w1:0"))
tf.reset_default_graph() #this will eliminate the variables we restored
with tf.session() as sess:
weights =
{
'1': tf.Variable(weight1 , name='w1-bis', trainable=False)
}
...
We are now sure the restored variables are not a part of the graph.
I fine-tuned InceptionNet v3 model given in the slim-tensorflow library. I trained the model on my own dataset. Now, I have .ckpt and .meta file for the model.
Now, as far as from I understand, I have two ways to restore the model and the weights. First, from the .meta file like this
checkpoint = './fine_tuned_model/model.ckpt-233700.meta'
with tf.Session() as sess:
new_saver = tf.train.import_meta_graph(checkpoint)
print(new_saver)
new_saver.restore(sess, tf.train.latest_checkpoint('./fine_tuned_model/'))
The second way is to call the model and restore the checkpoint. Like this
with slim.arg_scope(slim.nets.inception.inception_v3_arg_scope()):
logits, inceptionv3 = nets.inception.inception_v3(inputs=img, num_classes=5980, is_training=True,
dropout_keep_prob=.6)
# Restore convolutional layers:
variables_to_restore = slim.get_variables_to_restore(exclude=['InceptionV3/Logits', 'InceptionV3/AuxLogits'])
init_fn = slim.assign_from_checkpoint_fn(model_path, variables_to_restore)
Now, I think for my purpose second way is easier.
Now, my issue is that after restoring the model, how can I extract features from the last to last layer given an Image? I have included a screenshot of the model here
One solution I found was like this and here as far as I know I have to extract features from PreLogits layer but I am not sure how to set the values here
with tf.Session() as sess:
feature_tensor = sess.graph.get_tensor_by_name('mul:0')
features_last_layer = sess.run(feature_tensor,{inputs: img})
print features_last_layer
Here, I am unable to find out what should I pass to get_tenor_by_name(??) and also, how pass sess.run here?
Thank you. Please let me know if there are other ways to restore the model and get the features.
I have figured out the solution for this.
# Placeholder for the image,
image_tensor = tf.placeholder(tf.float32, shape=(None, 128, 128, 3))
# load the model
with slim.arg_scope(slim.nets.inception.inception_v3_arg_scope()):
logits, inceptionv3 = nets.inception.inception_v3(image_tensor,is_training=False)
# Restore convolutional layers:
variables_to_restore = slim.get_variables_to_restore(exclude=['InceptionV3/Logits', 'InceptionV3/AuxLogits'])
# get the latest checkpoint you want to use after training
init_fn = slim.assign_from_checkpoint_fn(model_path, variables_to_restore)
checkpoint = './fine_tuned_model/model.ckpt-233700.data-00000-of-00001'
saver = tf.train.Saver(variables_to_restore)
saver.restore(sess, tf.train.latest_checkpoint('./fine_tuned_model/'))
# the required image
img_car = cv2.imread('car.jpeg')
img_car = cv2.resize(img_car,(128, 128))
imgnumpy = np.ndarray((1,128,128,3))
imgnumpy[0] = img_car
# get output from any layer you want.
output = sess.run(inceptionv3["PreLogits"], feed_dict={image_tensor: imgnumpy})
I'm a bit new to all this so could you please help me? I tried finding the answer to this question but found nothing.
I'm trying to load Tensorflow model in python in a separate function so I can use the model in a loop without having to load it in every iteration of the for loop.
This is my code now:
def load_network():
prediction = neural_network_model(x)
return (prediction)
def use_neural_network(data, prediction):
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
saver = tf.train.import_meta_graph(model_name+'.meta')
saver.restore(sess,model_name)
pred = sess.run(prediction, feed_dict={x: data})
pred = np.asarray(pred)
return pred
if __name__ == '__main__':
result=[]
Load= start_network()
for i in data:
result.append(use_neural_network(i,Load))
And I would like to get something like this:
def load_network():
prediction = neural_network_model(x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
saver = tf.train.import_meta_graph(model_name+'.meta')
saver.restore(sess,model_name)
return (prediction)
def use_neural_network(data, prediction):
with tf.Session() as sess:
pred = sess.run(prediction, feed_dict={x: data})
pred = np.asarray(pred)
return pred
if __name__ == '__main__':
result=[]
Load= start_network()
for i in data:
result.append(use_neural_network(i,Load))
Generally what you're trying to achieve is easily doable and you're on the right track. In the main block you have start_network() instead of load_network() as in your first line. I'd also recommend against using Load as a variable name but that should not be a problem. Also the TensorFlow Session (sess in your code) should either be a global variable, or you should initialize it either in the main block or in the load_network() function and then pass it on to the use_neural_network() function. The way it's currently written the two sess variables in the two functions are local and therefore refer to different sessions.
If you want to avoid having to use the neural_network_model( x ) function, that is building the model at the beginning, you might want to freeze the model and load it that way, with the architecture embedded as well. Easiest to follow a guide on that, like this one.
I have loaded a pretrained model (Model 1) using the following code:
def load_seq2seq_model(sess):
with open(os.path.join(seq2seq_config_dir_path, 'config.pkl'), 'rb') as f:
saved_args = pickle.load(f)
# Initialize the model with saved args
model = Model1(saved_args)
#Inititalize Tensorflow saver
saver = tf.train.Saver()
# Checkpoint
ckpt = tf.train.get_checkpoint_state(seq2seq_config_dir_path)
print('Loading model: ', ckpt.model_checkpoint_path)
# Restore the model at the checkpoint
saver.restore(sess, ckpt.model_checkpoint_path)
return model
Now, I want to train another model (Model 2) from scratch which will take the output of the Model 1. But for that I need to define a session and load the pre-trained model and initialize the model tf.initialize_all_variables(). So, the pre-trained model will also get initialized.
Can anyone please tell me how to train the Model 2 after getting the output from the pre-trained model Model 1 properly?
What I am trying is given below -
with tf.Session() as sess:
# Initialize all the variables of the graph
seq2seq_model = load_seq2seq_model(sess)
sess.run(tf.initialize_all_variables())
.... Rest of the training code goes here....
All variables that are restored using a saver don't need to be initialized. Therefore, instead of using tf.initialize_all_variables() you can use tf.variables_initializer(var_list) to only initialize the weights of the second network.
To get a list of all the weights of the second network you can create the Model 2 network in a variable scope:
with tf.variable_scope("model2"):
model2 = Model2(...)
Then use
model_2_variables_list = tf.get_collection(
tf.GraphKeys.GLOBAL_VARIABLES,
scope="model2"
)
to get the variable list of the Model 2 network. Finally you can create the initialisier for the second network:
init2 = tf.variables_initializer(model_2_variables_list)
with tf.Session() as sess:
# Initialize all the variables of the graph
seq2seq_model = load_seq2seq_model(sess)
sess.run(init2)
.... Rest of the training code goes here....