I have one question about random variables in TensorFlow. Let's suppose I need a random variable inside my loss function.
In TensorFlow tutorials I find random functions used for initialize variables, like weights that in a second time are modified by training process.
In my case I need a random vector of floats (let's say 128 values), that follows a particular distribution (uniform or Gaussian) but that can change in each loss calculation.
Defining this variable in my loss function, is this the simple thing that I need to do, since at each epoch I get new values (that anyway follow the selected distribution) or do I get that the values that are always the same in all the iterations?
A random node in TensorFlow always takes a different value each time it is called, as you can verify by calling it several times
import tensorflow as tf
x = tf.random_uniform(shape=())
sess = tf.Session()
sess.run(x)
# 0.79877698
sess.run(x)
# 0.76016617
It is not a Variable in the tensorflow terminology, as you can check from the code above, which runs without calling variable initialization.
If you assign the values randomly generated to a Variable then this value will remain fixed until you update this variable.
If you, instead, put in the loss function directly the "generation" (tf.random_*) of the numbers, then they'll be different at each call.
Just try this out:
import tensorflow as tf
# generator
x = tf.random_uniform((3,1), minval=0, maxval=10)
# variable
a = tf.get_variable("a", shape=(3,1), dtype=tf.float32)
# assignment
b = tf.assign(a, x)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(5):
# 5 different values
print(sess.run(x))
# assign the value
sess.run(b)
for i in range(5):
# 5 equal values
print(sess.run(a))
Related
I'm new to tensorflow and try to understand how to use outside of a machine learning context. I would like to optimize a python function with the ADAM implemenation of tensorflow.
Let's assume I have the following function:
def fun_test(x):
"""
:param x: List of parameters, e.g. [1,2,3]
:return: real value
"""
res=do_something(x)
return res
When using scipy, I would call 'scipy.minimize(fun_test,x0,method="Nelder-Mead")'. How could I do this with tensorflow?
Best,
Michael
You need to rewrite the function do_something to take tensors as inputs and returns a scalar tensor (i.e. creating a computation graph). Then the following code is a sketch of how to perform optimization on the function. (BTW, in your code fun_test and do_something has no real difference so I picked the latter).
x = tf.get_variable("x", dtype=..., initializer=...)
target = do_something(x)
opt = tf.train.AdamOptimizer(...).minimize(target) # Defines one optimization step
with tf.Session() as sess:
sess.run(tf.global_variables_initializer()) # Initialize x
NUM_STEPS = 1000
for _ in range(NUM_STEPS):
sess.run(opt) # Run optimization for NUM_STEPS steps
print(sess.run(x)) # Show values of x
print(sess.run(target)) # Show target value
I am trying to compute the local variance map of an image by taking data from all possible window of fixed-size (eg 5x5), inside a training loop. To vectorize this operation I am thinking about expanding the original image with an operation similar to this using scatter_update/scatter_nd_update inside the training loop. What this operation essentially does is to map each element in the original tensor to potentially many locations in the new tensor, and the locations are computed inside the training loop.
However, scatter_update does not allow gradient propagation, and my attempt at creating a simple custom gradient for the scatter_update did not work.
#tf.RegisterGradient("CustomGrad")
def _clip_grad(unused_op, grad):
return tf.constant(5., dtype=tf.float32, shape=(1)) # tf.clip_by_value(grad, -0.1, 0.1)
x = tf.Variable([3.0], dtype=tf.float32)
y = tf.get_variable('y', shape=(1), dtype=tf.float32)
g = tf.get_default_graph()
with g.gradient_override_map({"ScatterNdUpdate1": "CustomGrad"}):
output = tf.scatter_nd_update(y, [[0]], x, name="ScatterNdUpdate1")
grad_custom = tf.gradients(output, y)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
print(grad_custom)
Running the code above shows that grad_custom contains None. Does any one have any idea of how to properly implement a local variance map that can be used in the training loop? Solving the gradient problem would also help me with another problem I am having.
I'm trying to get acquainted with TensorFlow, and I'm not sure about placeholders, variables and such. To make things easy, I tried to create a very simple calculation - a placeholder and a variable that is just the placeholder times two.
I've put everything in a function, like so:
import tensorflow as tf
def try_variable(value):
x = tf.placeholder(tf.float64, name='x')
v = tf.Variable(x * 2, name='v', validate_shape=False)
with tf.Session() as session:
init = tf.global_variables_initializer()
session.run(init, feed_dict={x: value})
return session.run(v)
I then call the function:
print(try_variable(80))
And indeed the output is 160.
But when I call it again:
print(try_variable(80))
I get an error:
InvalidArgumentError: You must feed a value for placeholder tensor 'x' with dtype double
What am I missing?
Right now you're creating a new variable and placeholder each time you call the function, so on the second time you call the try_variable function you actually have 2 placeholders and 2 TensorFlow variables! x, x_1, v, v_1.
So, on the second time you run the init operation, you provide the initial value only for placeholder x_1 which is now binded to python variable x.
If you want to print the name of all the Tensors in the current graph, you can call
print [n.name for n in tf.get_default_graph().as_graph_def().node]
If you still want to create 2 new tensors each time you call the function, one option is to reset the default graph with the command tf.reset_default_graph()
each time the function is called - it is highly unrecommended.
I want to initialize a w_gate tensor with a custom np.array as in the code below:
w_init = np.ones(shape=(dim, self.config.nmodels)) / self.config.nmodels
w_gate = tf.Variable(
name="W",
initial_value=w_init,
dtype=tf.float32)
Every a certain number of train iterations, I want w_gate to be re-initialized again to the w_init array. For this, and based on Re-initialize variables in Tensorflow, I tried
sess.run(tf.variables_initializer([w_gate]))
inside my training loop. This line is executed every certain number of iterations. Although, w_gate doesn't seem to be re-initialized. What am I missing here?
Could you try this and check ?
w_gate_assign = tf.assign(w_gate, w_init)
sess.run(w_gate_assign)
In the PTB language model tutorial at https://github.com/tensorflow/tensorflow/blob/master/tensorflow/models/rnn/ptb/ptb_word_lm.py.
I don't understand the need for line 248 (and the passing of state into session.run on line 254)
state = m.initial_state.eval()
Isn't the tensor of the initial state:
self._initial_state = cell.zero_state(batch_size, tf.float32)
evaluated when the graph is loaded into the session?
For example, this code prints 11 as one would expect,
x = constant(6)
y = tf.placeholder(tf.int32)
z = x + y
with tf.Session() as sess:
print sess.run(z,{y:5})
without the need to replace the last line with
print sess.run(z,{y:5,x:x.eval()})
So is that eval needed? And if so, why?
Okay, I figured it out. The RNN is called multiple times, and each time it is called you want it to start with a clean initial state. If you were to just call it once, you wouldn't need to pass in a clean initial state to sess.run().