I'm trying to get acquainted with TensorFlow, and I'm not sure about placeholders, variables and such. To make things easy, I tried to create a very simple calculation - a placeholder and a variable that is just the placeholder times two.
I've put everything in a function, like so:
import tensorflow as tf
def try_variable(value):
x = tf.placeholder(tf.float64, name='x')
v = tf.Variable(x * 2, name='v', validate_shape=False)
with tf.Session() as session:
init = tf.global_variables_initializer()
session.run(init, feed_dict={x: value})
return session.run(v)
I then call the function:
print(try_variable(80))
And indeed the output is 160.
But when I call it again:
print(try_variable(80))
I get an error:
InvalidArgumentError: You must feed a value for placeholder tensor 'x' with dtype double
What am I missing?
Right now you're creating a new variable and placeholder each time you call the function, so on the second time you call the try_variable function you actually have 2 placeholders and 2 TensorFlow variables! x, x_1, v, v_1.
So, on the second time you run the init operation, you provide the initial value only for placeholder x_1 which is now binded to python variable x.
If you want to print the name of all the Tensors in the current graph, you can call
print [n.name for n in tf.get_default_graph().as_graph_def().node]
If you still want to create 2 new tensors each time you call the function, one option is to reset the default graph with the command tf.reset_default_graph()
each time the function is called - it is highly unrecommended.
Related
I have one question about random variables in TensorFlow. Let's suppose I need a random variable inside my loss function.
In TensorFlow tutorials I find random functions used for initialize variables, like weights that in a second time are modified by training process.
In my case I need a random vector of floats (let's say 128 values), that follows a particular distribution (uniform or Gaussian) but that can change in each loss calculation.
Defining this variable in my loss function, is this the simple thing that I need to do, since at each epoch I get new values (that anyway follow the selected distribution) or do I get that the values that are always the same in all the iterations?
A random node in TensorFlow always takes a different value each time it is called, as you can verify by calling it several times
import tensorflow as tf
x = tf.random_uniform(shape=())
sess = tf.Session()
sess.run(x)
# 0.79877698
sess.run(x)
# 0.76016617
It is not a Variable in the tensorflow terminology, as you can check from the code above, which runs without calling variable initialization.
If you assign the values randomly generated to a Variable then this value will remain fixed until you update this variable.
If you, instead, put in the loss function directly the "generation" (tf.random_*) of the numbers, then they'll be different at each call.
Just try this out:
import tensorflow as tf
# generator
x = tf.random_uniform((3,1), minval=0, maxval=10)
# variable
a = tf.get_variable("a", shape=(3,1), dtype=tf.float32)
# assignment
b = tf.assign(a, x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(5):
# 5 different values
print(sess.run(x))
# assign the value
sess.run(b)
for i in range(5):
# 5 equal values
print(sess.run(a))
I want to initialize a w_gate tensor with a custom np.array as in the code below:
w_init = np.ones(shape=(dim, self.config.nmodels)) / self.config.nmodels
w_gate = tf.Variable(
name="W",
initial_value=w_init,
dtype=tf.float32)
Every a certain number of train iterations, I want w_gate to be re-initialized again to the w_init array. For this, and based on Re-initialize variables in Tensorflow, I tried
sess.run(tf.variables_initializer([w_gate]))
inside my training loop. This line is executed every certain number of iterations. Although, w_gate doesn't seem to be re-initialized. What am I missing here?
Could you try this and check ?
w_gate_assign = tf.assign(w_gate, w_init)
sess.run(w_gate_assign)
As far as I understand it, I should be able to add a print operator to my graph by doing something like this:
a = nn_ops.softmax(s)
a = tf.Print(a, [tf.shape(a)], message="This is shape a: ")
and when the graph is executed this should print the shape of a. However, this statement produces no output for me (I am running the seq2seq tensorflow tutorial and this softmax belongs to the attention function, so it's definitely executed).
I do get output if instead I do something like this:
ph = tf.placeholder(tf.float32, [3,4,5,6])
ts = tf.shape(ph)
tp = tf.Print(ts, [ts], message="PRINT=")
sess = tf.Session()
sess.run(tp)
However, in my real example, sess.run() is called in seq2seq_model.py, and if I try to do sess.run(a) in the attention function, tensorflow complains:
You must feed a value for placeholder tensor 'encoder0' with dtype int32
but I don't have access to the input feed at this point in the code. How can I fix this?
In case you just want to know the tensor shape, often it can be inferred without running the graph. Then you do not need tf.Print.
For example in the second code fragment you can just use:
ph = tf.placeholder(tf.float32, [3,4,5,6])
print(ph.get_shape())
If you want to see the shape which depends on the input size (using tf.shape) or you want to see a value which also depends on the input, it is not possible to do without providing the input data.
For example if you train the model where x and y are your samples and labels respectively, you cannot compute cost without providing them.
If you have the following code:
predictions = ...
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(y_output, y_train))
cost = tf.Print(cost, [tf.shape(cost)], message='cost:')
Trying to evaluate it without providing placeholder values won't work:
sess.run(cost)
# error, no placeholder provided
However this will work as expected:
sess.run(cost, {x: x_train, y: y_train})
Regarding your first code fragment. In order to work, tf.Print node needs to be executed in order to print a message. I suspect in your case the print node is not used during further computations.
For example the following code won't produce output:
import tensorflow as tf
a = tf.Variable([1., 2., 3])
b = tf.Variable([1., 2., 3])
c = tf.add(a, b)
# we create a print node, but it is never used
a = tf.Print(a, [tf.shape(a)], message='a.shape: ')
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
print(sess.run(c))
However if you reverse lines such that the print node is used during computations, you will see the output:
a = tf.Print(a, [tf.shape(a)], message='a.shape: ')
# now c depends on the tf.Print node
c = tf.add(a, b)
I created model in tensorflow of neural network.
I saved the model and restore it in another python file.
The code is below:
def restoreModel():
prediction = neuralNetworkModel(x)
tf_p = tensorFlow.nn.softmax(prediction)
temp = np.array([2,1,541,161124,3,3])
temp = np.vstack(temp)
with tensorFlow.Session() as sess:
new_saver = tensorFlow.train.import_meta_graph('model.ckpt.meta')
new_saver.restore(sess, tensorFlow.train.latest_checkpoint('./'))
all_vars = tensorFlow.trainable_variables()
tensorFlow.initialize_all_variables().run()
sess.run(tensorFlow.initialize_all_variables())
predict = sess.run([tf_p], feed_dict={
tensorFlow.transpose(x): temp,
y : ***
})
when "temp" variable in what I want to predict!
X is the vector shape, and I "transposed" it to match the shapes.
I dont understand what I need to write in feed_dict variable.
I am answering late but maybe it can still be useful. feed_dict is used to give tensorflow the values you want your placeholders to take. fetches (the first argument of run) is the list of results you want. The keys of feed_dict and the elements of fetches must be either the names of the tensors (I didn't try it though) or variables you can get by
graph = tf.get_default_graph()
var = graph.get_operation_by_name('name_of_operation').outputs[0]
Maybe graph.get_tensor_by_name('name_of_operation:0') works too, I didn't try.
By default, the name of placeholders are simply 'Placeholder', 'Placeholder_1' etc, following the order of creation in the graph definition.
In the PTB language model tutorial at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/ptb/ptb_word_lm.py.
I don't understand the need for line 248 (and the passing of state into session.run on line 254)
state = m.initial_state.eval()
Isn't the tensor of the initial state:
self._initial_state = cell.zero_state(batch_size, tf.float32)
evaluated when the graph is loaded into the session?
For example, this code prints 11 as one would expect,
x = constant(6)
y = tf.placeholder(tf.int32)
z = x + y
with tf.Session() as sess:
print sess.run(z,{y:5})
without the need to replace the last line with
print sess.run(z,{y:5,x:x.eval()})
So is that eval needed? And if so, why?
Okay, I figured it out. The RNN is called multiple times, and each time it is called you want it to start with a clean initial state. If you were to just call it once, you wouldn't need to pass in a clean initial state to sess.run().