How can I edit the sessions.run function so that it runs on Tensorflow 2.0?
with tf.compat.v1.Session(graph=graph) as sess:
start = time.time()
results = sess.run(output_operation.outputs[0],
{input_operation.outputs[0]: t})
I read the documentation over here and learned that you have to change a function like this:
normalized = tf.divide(tf.subtract(resized, [input_mean]), [input_std])
sess = tf.compat.v1.Session()
result = sess.run(normalized)
return result
to this:
def myFunctionToReplaceSessionRun(resized,input_mean,input_std):
return tf.divide(tf.subtract(resized, [input_mean]), [input_std])
normalized = myFunctionToReplaceSessionRun(resized,input_mean,input_std)
but I'm unable to figure out how to change the first one.
Here's a bit of context, I was trying out this code lab, and in this found that the sess.run, that was giving me trouble.
This is the command line output when running the label_images file.
And this is the function that gave errors.
With TensorFlow 1.x, we used to create tf.placeholder tensors by which the data could enter the graph. We used a feed_dict= along with the tf.Session() object.
In TensorFlow 2.0, we can directly feed the data to the graph as eager execution is enabled by default. With the #tf.function annotation, we can include the function directly in our graph. The official docs say,
At the centre of this merger is tf.function, which allows you to
transform a subset of Python syntax into portable, high-performance
TensorFlow graphs.
Here's a simple example from the docs,
#tf.function
def simple_nn_layer(x, y):
return tf.nn.relu(tf.matmul(x, y))
x = tf.random.uniform((3, 3))
y = tf.random.uniform((3, 3))
simple_nn_layer(x, y)
Now, looking into your problem, you can convert your function like,
#tf.function
def get_output_operation( input_op ):
# The function goes here
# from here return `results`
results = get_output_operation( some_input_op )
In simple and less precise words, the placeholder tensors are transformed to function arguments, the tensor in sess.run( tensor ) is returned by the function. All this happens in a #tf.function annotated function.
Related
I'm new to tensorflow and try to understand how to use outside of a machine learning context. I would like to optimize a python function with the ADAM implemenation of tensorflow.
Let's assume I have the following function:
def fun_test(x):
"""
:param x: List of parameters, e.g. [1,2,3]
:return: real value
"""
res=do_something(x)
return res
When using scipy, I would call 'scipy.minimize(fun_test,x0,method="Nelder-Mead")'. How could I do this with tensorflow?
Best,
Michael
You need to rewrite the function do_something to take tensors as inputs and returns a scalar tensor (i.e. creating a computation graph). Then the following code is a sketch of how to perform optimization on the function. (BTW, in your code fun_test and do_something has no real difference so I picked the latter).
x = tf.get_variable("x", dtype=..., initializer=...)
target = do_something(x)
opt = tf.train.AdamOptimizer(...).minimize(target) # Defines one optimization step
with tf.Session() as sess:
sess.run(tf.global_variables_initializer()) # Initialize x
NUM_STEPS = 1000
for _ in range(NUM_STEPS):
sess.run(opt) # Run optimization for NUM_STEPS steps
print(sess.run(x)) # Show values of x
print(sess.run(target)) # Show target value
I want to use Tensorflow to calculate the gradients of a function. However, if I use the tf.gradients function, it returns a single list of gradients. How to return a list for each point of the batch?
# in a tensorflow graph I have the following code
tf_x = tf.placeholder(dtype=tf.float32, shape=(None,N_in), name='x')
tf_net #... conveniently defined neural network
tf_y = tf.placeholder(dtype=tf.float32, shape=(None,1), name='y')
tf_cost = (tf_net(tf_x) - tf_y)**2 # this should have length N_samples because I did not apply a tf.reduce_mean
tf_cost_gradients = tf.gradients(tf_cost,tf_net.trainable_weights)
If we run it in a tensorflow session,
# suppose myx = np.random.randn(N_samples,N_in) and myy conveniently chosen
feed = {tf_x:myx, tx_y:myy}
sess.run(tf_cost_gradients,feed)
I get only one list, and not a list for each sample as I would like. I can use
for i in len(myx):
feed = {tf_x:myx[i], tx_y:myy[i]}
sess.run(tf_cost_gradients,feed)
but this is extremely slow! What can I do? Thank you
Although, there is an 'aggregation_method' parameter in tf.gradients, it is not easy to get the individual gradients.
aggregation_method: Specifies the method used to combine gradient terms.
Please see these threads:
https://github.com/tensorflow/tensorflow/issues/15760
https://github.com/tensorflow/tensorflow/issues/4897
In one of the threads(#4897), Ian Goodfellow makes the following suggestion to speed up individual gradient computation:
This is only pseudocode, but basic idea is:
examples = tf.split(batch)
weight_copies = [tf.identity(weights) for x in examples]
output = tf.stack(f(x, w) in zip(examples, weight_copies))
cost = cost_function(output)
per_example_gradients = tf.gradients(cost, weight_copies)
I have seen variations of this question asked, but I haven't quite found a satisfactory answer yet. Basically, I would like to do the equivalent from keras model.to_json(), model.get_weights(), model.from_json(), model.set_weights() to tensorflow. I think I am getting close to there, but I am at a point where I am stuck. I'd prefer if I could get the weights and graph in the same string, but I understand if that isn't possible.
Currently, what I have is:
g = optimizer.minimize(loss_op,
global_step=tf.train.get_global_step())
de = g.graph.as_graph_def()
json_string = json_format.MessageToJson(de)
gd = tf.GraphDef()
gd = json_format.Parse(json_string, gd)
That seems to create the graph fine, but obviously the meta graph is not included for variable, weights, etc. There is also the meta graph, but the only thing I see is export_meta_graph, which doesn't seem to serialize in the same manner. I saw that MetaGraph has a proto function, but I don't know how to serialize those variables.
So in short, how would you take a tensorflow model (model as in weights, graph, etc), serialize it to a string (preferably json), then deserialize it and continue training or serve predictions.
Here are things that get me close to there and I have tried, but mostly has limitations in needing to write to disk, which I can't do in this case:
Gist on GitHub
This is the closest one I found, but the link to serializing a metagraph doesn't exist.
Note that the solution from #Maxim will create new operations in the graph each time it runs.
If you run the function very frequently this will cause your code to get slower and slower.
Two solutions to work around this problem:
Create the assign operations at the same time as the rest of the graph and reuse them:
assign_ops = []
for var_name in tf.trainable_variables():
assign_placeholder = tf.placeholder(var.dtype, shape=value.shape)
assign_op = var.assign(assign_placeholder)
assign_ops.append(assign_op)
Use the load function on the variables, I prefer this one as it removes the need for the code above:
self.params = tf.trainable_variables()
def get_weights(self):
values = tf.get_default_session().run(self.params)
return values
def set_weights(self, weights):
for i, value in enumerate(weights):
value = np.asarray(value)
self.params[i].load(value, self.sess)
(I can't comment so I put this as an answer instead)
If you want the equivalent of keras Model.get_weights() and Model.set_weights(), these methods aren't strongly tied to keras internals and can be easily extracted.
Original code
Here's how they look like in keras source code:
def get_weights(self):
weights = []
for layer in self.layers:
weights += layer.weights
return K.batch_get_value(weights) # this is just `get_session().run(weights)`
def set_weights(self, weights):
tuples = []
for layer in self.layers:
num_param = len(layer.weights)
layer_weights = weights[:num_param]
for sw, w in zip(layer.weights, layer_weights):
tuples.append((sw, w))
weights = weights[num_param:]
K.batch_set_value(tuples) # another wrapper over `get_session().run(...)`
Keras's weights is the list of numpy arrays (not json). As you can see, it uses the fact that model architecture is known (self.layers) which allows it to reconstruct the correct mapping from variables to values. Some seemingly non-trivial work is done in K.batch_set_value, but in fact it simply prepares assign ops and runs them in session.
Getting and setting weights in pure tensorflow
def tensorflow_get_weights():
vars = tf.trainable_variables()
values = tf.get_default_session().run(vars)
return zip([var.name for var in vars], values)
def tensorflow_set_weights(weights):
assign_ops = []
feed_dict = {}
for var_name, value in weights:
var = tf.get_default_session().graph.get_tensor_by_name(var_name)
value = np.asarray(value)
assign_placeholder = tf.placeholder(var.dtype, shape=value.shape)
assign_op = tf.assign(var, assign_placeholder)
assign_ops.append(assign_op)
feed_dict[assign_placeholder] = value
tf.get_default_session().run(assign_ops, feed_dict=feed_dict)
Here I assume that you want to serialize / deserialize the whole model (i.e., all trainable variables) and in the default session. If this is not the case, functions above are easily customizable.
Testing
x = tf.placeholder(shape=[None, 5], dtype=tf.float32, name='x')
W = tf.Variable(np.zeros([5, 5]), dtype=tf.float32, name='W')
b = tf.Variable(np.zeros([5]), dtype=tf.float32, name='b')
y = tf.add(tf.matmul(x, W), b)
with tf.Session() as session:
session.run(tf.global_variables_initializer())
# Save the weights
w = tensorflow_get_weights()
print(W.eval(), b.eval())
# Update the model
session.run([tf.assign(W, np.ones([5, 5])), tf.assign(b, np.ones([5]) * 2)])
print(W.eval(), b.eval())
# Restore the weights
tensorflow_set_weights(w)
print(W.eval(), b.eval())
If you run this test, you should see the model was freezed at zeros, then got updated and then restored back to zeros.
You can use freeze_graph
This script is included in Tensorflow and allows you to take a GraphDef proto, a SaverDef proto, and a set of variable values stored in a checkpoint file.
In this way you can output a GraphDef with all of the variable ops converted into const ops containing the values of the variables.
To restore a frozen model you have to reinitialize graphs and remap inputs from the frozen model, see this example
Thanks to Maxim for getting me to the solution. I wanted to post an answer with both the graph and weights being converted to json for people that stumble across this problem. To just serialize the graph and not the weights, I created a gist that encapsulates what Maxim wrote here: Tensorflow graph with non json serialized weights
Now to serialize/deserialize both the graph and weights, I created a separate gist here: Tensorflow graph with json serialized weights and graph.
To run through the explanation, I first slightly tweaked the weight functions by not returning the variables in get weights, and in set weights, grabbing the current variables there. The is an important caveat, especially if the graph is slightly different than the current trainable variables:
import tensorflow as tf
import numpy as np
from google.protobuf import json_format
import json
def tensorflow_get_weights():
vs = tf.trainable_variables()
values = tf.get_default_session().run(vs)
return values
def tensorflow_set_weights(weights):
assign_ops = []
feed_dict = {}
vs = tf.trainable_variables()
zipped_values = zip(vs, weights)
for var, value in zipped_values:
value = np.asarray(value)
assign_placeholder = tf.placeholder(var.dtype, shape=value.shape)
assign_op = var.assign(assign_placeholder)
assign_ops.append(assign_op)
feed_dict[assign_placeholder] = value
tf.get_default_session().run(assign_ops, feed_dict=feed_dict)
Next, I created two utility functions that would convert weights to and from json:
def convert_weights_to_json(weights):
weights = [w.tolist() for w in weights]
weights_list = json.dumps(weights)
return weights_list
def convert_json_to_weights(json_weights):
loaded_weights = json.loads(json_weights)
loaded_weights = [np.asarray(x) for x in loaded_weights]
return loaded_weights
Than I had a method that initially ran to kick off training. This method would initialize variables, run the optimization, get the weights and graph, and convert them into json. It looks like:
def run_initial_with_json_weights(opti, feed_dict):
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(0, 250):
sess.run(opti, feed_dict=feed_dict)
first_weights = tensorflow_get_weights()
g = tf.get_default_graph().as_graph_def()
json_string = json_format.MessageToJson(g)
return json_string, convert_weights_to_json(first_weights)
Now that we have the serialized weights and graph, if we want to continue training and or make predictions, we can do the following. This method deserializes the graphdef and weights, runs the optimization, then makes predictions.
def run_serialized(json_graph, json_weights, feed_dict):
gd = tf.GraphDef()
gd = json_format.Parse(json_graph, gd)
weights = convert_json_to_weights(json_weights)
with tf.Session() as sess:
tf.import_graph_def(gd)
sess.run(tf.global_variables_initializer())
nu_out = tf.get_default_graph().get_tensor_by_name('outer/Sigmoid:0')
mini = tf.get_default_graph().get_tensor_by_name('mini:0')
tensorflow_set_weights(weights)
for i in range(0, 50):
sess.run(mini, feed_dict=feed_dict)
predicted = sess.run(nu_out, feed_dict=feed_dict)
return predicted
A full xor example is in the gist above.
As far as I understand it, I should be able to add a print operator to my graph by doing something like this:
a = nn_ops.softmax(s)
a = tf.Print(a, [tf.shape(a)], message="This is shape a: ")
and when the graph is executed this should print the shape of a. However, this statement produces no output for me (I am running the seq2seq tensorflow tutorial and this softmax belongs to the attention function, so it's definitely executed).
I do get output if instead I do something like this:
ph = tf.placeholder(tf.float32, [3,4,5,6])
ts = tf.shape(ph)
tp = tf.Print(ts, [ts], message="PRINT=")
sess = tf.Session()
sess.run(tp)
However, in my real example, sess.run() is called in seq2seq_model.py, and if I try to do sess.run(a) in the attention function, tensorflow complains:
You must feed a value for placeholder tensor 'encoder0' with dtype int32
but I don't have access to the input feed at this point in the code. How can I fix this?
In case you just want to know the tensor shape, often it can be inferred without running the graph. Then you do not need tf.Print.
For example in the second code fragment you can just use:
ph = tf.placeholder(tf.float32, [3,4,5,6])
print(ph.get_shape())
If you want to see the shape which depends on the input size (using tf.shape) or you want to see a value which also depends on the input, it is not possible to do without providing the input data.
For example if you train the model where x and y are your samples and labels respectively, you cannot compute cost without providing them.
If you have the following code:
predictions = ...
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y_output, y_train))
cost = tf.Print(cost, [tf.shape(cost)], message='cost:')
Trying to evaluate it without providing placeholder values won't work:
sess.run(cost)
# error, no placeholder provided
However this will work as expected:
sess.run(cost, {x: x_train, y: y_train})
Regarding your first code fragment. In order to work, tf.Print node needs to be executed in order to print a message. I suspect in your case the print node is not used during further computations.
For example the following code won't produce output:
import tensorflow as tf
a = tf.Variable([1., 2., 3])
b = tf.Variable([1., 2., 3])
c = tf.add(a, b)
# we create a print node, but it is never used
a = tf.Print(a, [tf.shape(a)], message='a.shape: ')
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print(sess.run(c))
However if you reverse lines such that the print node is used during computations, you will see the output:
a = tf.Print(a, [tf.shape(a)], message='a.shape: ')
# now c depends on the tf.Print node
c = tf.add(a, b)
I created model in tensorflow of neural network.
I saved the model and restore it in another python file.
The code is below:
def restoreModel():
prediction = neuralNetworkModel(x)
tf_p = tensorFlow.nn.softmax(prediction)
temp = np.array([2,1,541,161124,3,3])
temp = np.vstack(temp)
with tensorFlow.Session() as sess:
new_saver = tensorFlow.train.import_meta_graph('model.ckpt.meta')
new_saver.restore(sess, tensorFlow.train.latest_checkpoint('./'))
all_vars = tensorFlow.trainable_variables()
tensorFlow.initialize_all_variables().run()
sess.run(tensorFlow.initialize_all_variables())
predict = sess.run([tf_p], feed_dict={
tensorFlow.transpose(x): temp,
y : ***
})
when "temp" variable in what I want to predict!
X is the vector shape, and I "transposed" it to match the shapes.
I dont understand what I need to write in feed_dict variable.
I am answering late but maybe it can still be useful. feed_dict is used to give tensorflow the values you want your placeholders to take. fetches (the first argument of run) is the list of results you want. The keys of feed_dict and the elements of fetches must be either the names of the tensors (I didn't try it though) or variables you can get by
graph = tf.get_default_graph()
var = graph.get_operation_by_name('name_of_operation').outputs[0]
Maybe graph.get_tensor_by_name('name_of_operation:0') works too, I didn't try.
By default, the name of placeholders are simply 'Placeholder', 'Placeholder_1' etc, following the order of creation in the graph definition.