In ipython I imported tensorflow as tf and numpy as np and created an TensorFlow InteractiveSession.
When I am running or initializing some normal distribution with numpy input, everything runs fine:
some_test = tf.constant(np.random.normal(loc=0.0, scale=1.0, size=(2, 2)))
session.run(some_test)
Returns:
array([[-0.04152317, 0.19786302],
[-0.68232622, -0.23439092]])
Just as expected.
...but when I use the Tensorflow normal distribution function:
some_test = tf.constant(tf.random_normal([2, 2], mean=0.0, stddev=1.0, dtype=tf.float32))
session.run(some_test)
...it raises a Type error saying:
(...)
TypeError: List of Tensors when single Tensor expected
What am I missing here?
The output of:
sess.run(tf.random_normal([2, 2], mean=0.0, stddev=1.0, dtype=tf.float32))
alone returns the exact same thing which np.random.normal generates -> a matrix of shape (2, 2) with values taken from a normal distribution.
The tf.constant() op takes a numpy array (or something implicitly convertible to a numpy array), and returns a tf.Tensor whose value is the same as that array. It does not accept a tf.Tensor as its argument.
On the other hand, the tf.random_normal() op returns a tf.Tensor whose value is generated randomly according to the given distribution each time it runs. Since it returns a tf.Tensor, it cannot be used as the argument to tf.constant(). This explains the TypeError (which is unrelated to the use of tf.InteractiveSession, since it occurs when you build the graph).
I'm assuming you want your graph to include a tensor that (i) is randomly generated on its first use, and (ii) constant thereafter. There are two ways to do this:
Use NumPy to generate the random value and put it in a tf.constant(), as you did in your question:
some_test = tf.constant(
np.random.normal(loc=0.0, scale=1.0, size=(2, 2)).astype(np.float32))
(Potentially faster, as it can use the GPU to generate the random numbers) Use TensorFlow to generate the random value and put it in a tf.Variable:
some_test = tf.Variable(
tf.random_normal([2, 2], mean=0.0, stddev=1.0, dtype=tf.float32)
sess.run(some_test.initializer) # Must run this before using `some_test`
Related
I have a n-D array. I need to create a 1-D range tensor based on dimensions.
for an example:
x = tf.placeholder(tf.float32, shape=[None,4])
r = tf.range(start=0, limit=, delta=x.shape[0],dtype=tf.int32, name='range')
sess = tf.Session()
result = sess.run(r, feed_dict={x: raw_lidar})
print(r)
The problem is, x.shape[0] is none at the time of building computational graph. So I can not build the tensor using range. It gives an error.
ValueError: Cannot convert an unknown Dimension to a Tensor: ?
Any suggestion or help for the problem.
Thanks in advance
x.shape[0] might not exist yet when running this code is graph mode. If you want a value, you need to use tf.shape(x)[0].
More information about that behaviour in the documentation for tf.Tensor.get_shape. An excerpt (emphasis is mine):
tf.Tensor.get_shape() is equivalent to tf.Tensor.shape.
When executing in a tf.function or building a model using tf.keras.Input, Tensor.shape may return a partial shape (including None for unknown dimensions). See tf.TensorShape for more details.
>>> inputs = tf.keras.Input(shape = [10])
>>> # Unknown batch size
>>> print(inputs.shape)
(None, 10)
The shape is computed using shape inference functions that are registered for each tf.Operation.
The returned tf.TensorShape is determined at build time, without executing the underlying kernel. It is not a tf.Tensor. If you need a shape tensor, either convert the tf.TensorShape to a tf.constant, or use the tf.shape(tensor) function, which returns the tensor's shape at execution time.
As said in the title, I am trying to create a mixture of multivariate normal distributions using tensorflow probability package.
In my original project, am feeding the weights of the categorical, the loc and the variance from the output of a neural network. However when creating the graph, I get the following error:
components[0] batch shape must be compatible with cat shape and other component batch shapes
I recreated the same problem using placeholders:
import tensorflow as tf
import tensorflow_probability as tfp # dist= tfp.distributions
tf.compat.v1.disable_eager_execution()
sess = tf.compat.v1.InteractiveSession()
l1 = tf.compat.v1.placeholder(dtype=tf.float32, shape=[None, 2], name='observations_1')
l2 = tf.compat.v1.placeholder(dtype=tf.float32, shape=[None, 2], name='observations_2')
log_std = tf.compat.v1.get_variable('log_std', [1, 2], dtype=tf.float32,
initializer=tf.constant_initializer(1.0),
trainable=True)
mix = tf.compat.v1.placeholder(dtype=tf.float32, shape=[None,1], name='weights')
cat = tfp.distributions.Categorical(probs=[mix, 1.-mix])
components = [
tfp.distributions.MultivariateNormalDiag(loc=l1, scale_diag=tf.exp(log_std)),
tfp.distributions.MultivariateNormalDiag(loc=l2, scale_diag=tf.exp(log_std)),
]
bimix_gauss = tfp.distributions.Mixture(
cat=cat,
components=components)
So, my question is, what am I doing wrong? I looked into the error and it seems tensorshape_util.is_compatible_with is what raises the error but I don't see why.
Thanks!
When the components are the same type, MixtureSameFamily should be more performant.
There you only pass a single Categorical instance (with .batch_shape [b1,b2,...,bn]) and a single MVNDiag instance (with .batch_shape [b1,b2,...,bn,numcats]).
For only two classes, I wonder if Bernoulli would work?
It seems you provided a mis-shaped input to tfp.distributions.Categorical. It's probs parameter should be of shape [batch_size, cat_size] while the one you provide is rather [cat_size, batch_size, 1]. So maybe try to parametrize probs with tf.concat([mix, 1-mix], 1).
There may also be a problem with yourlog_std which doesn't have the same shape as l1and l2. In case MultivariateNormalDiag doesn't properly broadcast it, try to specify it's shape as (None, 2) or to tile it so that it's first dimension corresponds to that of your location parameters.
If I understand correctly (no), there are two ways to pass a numpy array to a tensorflow computational graph:
use tf.placeholder and pass this numpy array in the feed_dict
use tf.convert_to_tensor, convert this numpy array as a tensor, then use this tensor for whatever calculation.
Are there any differences between these two methods?
tf_convert_to_tensor is highly unpractical because it does not scale. See the example below:
X = np.random.rand(3,3)
Y = np.random.rand(3,3)
X_tensor = tf.convert_to_tensor(X)
X_squared = tf.square(X_tensor)
Y_tensor = tf.convert_to_tensor(Y)
Y_squared = tf.square(Y)
with tf.Session() as sess:
x = sess.run(X_squared)
y = sess.run(Y_squared)
As you can see, for every Numpy array that we want to provide as input to the graph, we have to create a new tf.convert_to_tensor op. One example where this is fine is if you have a static input that won't change, then you have a single tf.convert_to_tensor op that you use. On the other hand, let's have a look at the same example using tf.placeholder:
X = np.random.rand(3,3)
Y = np.random.rand(3,3)
graph_input = tf.placeholder(shape=[None, None], dtype=tf.float64)
squared = tf.square(graph_input)
with tf.Session() as sess:
x = sess.run(squared, feed_dict={graph_input: X})
y = sess.run(squared, feed_dict={graph_input: Y})
As you can see, we use the tf.placeholder to dynamically provide input data to the graph. Consider it as a funnel, that you use to pour data in the graph. The tf.placeholder is fine for data that might/will change throughout the training such as learning rate, dropout probability, etc.
Placeholder tensors:
As far as I know, tf.placeholder creates a placeholder tensor into the default tf.Graph object. A placeholder tensor basically creates a pathway/entrance for values to enter the graph. They act as inputs to a graph which may correspond to a model too.
The values ( not belonging to the graph ) are fed to the placeholders using the feed_dict mentioned by you.
tf.convert_to_tensor() :
As mentioned in the official docs,
This function converts Python objects of various types to Tensor
objects. It accepts Tensor objects, numpy arrays, Python lists, and
Python scalars.
It converts an existing NumPy array or Python object to a corresponding tensor ( with same dtype and shape ). This tensor will not be a placeholder tensor. It's just a utility to convert NumPy arrays and create tf.Tensor objects out of them.
Hence, placeholder tensors allow you feed NumPy arrays or Python objects directly to the graph using feed_dict. If your data is static then you can include it in the graph by creating a tensor out of it using tf.convert_to_tensor().
I would like to slice a tensor and store it in a variable. Slicing with fixed numbers works fine eg: t[0:2]. But slicing a variable with another tensor doesnt work. eg t[t1:t2] Also storing the slice in a tensor works fine but when I try to store it in a tf.Variable i get errors.
import tensorflow as tf
import numpy
i=tf.zeros([2,1],tf.int32)
i2=tf.get_variable('i2_variable',initializer=i) #putting a multidimensional tensor in a variable
i4=tf.ones([10,1],tf.int32)
sess=tf.Session()
sess.run(tf.global_variables_initializer()) #initializing variables
itr=tf.constant(0,tf.int32)
def w_c(i2,itr):
return tf.less(itr,2)
def w_b(i2,itr):
i2=i4[(itr*0):((itr*0)+2)] #doesnt work
#i2=i4[0:2] #works
#i=i4[(itr*0):((itr*0)+2)] #works with tensor i
itr=tf.add(itr,1)
return[i2,itr]
OP=tf.while_loop(w_c,w_b,[i2,itr])
print(sess.run(OP))
I get following error:
ValueError: Input tensor 'i2_variable/read:0' enters the
loop with shape (2, 1), but has shape (?, 1) after one iteration.
To allow the shape to vary across iterations,
use the `shape_invariants` argument of tf.while_loop to specify a less-specific shape.
The code does not throw the error if you specify shape_invariants.
OP=tf.while_loop(w_c,w_b,[i2,itr],shape_invariants=
[ tf.TensorShape([None, None]),
itr.get_shape()])
It returns this.
[array([[1],
[1]]), 2]
I'm playing around with tensorflow and ran into a problem with the following code:
def _init_parameters(self, input_data, labels):
# the input shape is (batch_size, input_size)
input_size = tf.shape(input_data)[1]
# labels in one-hot format have shape (batch_size, num_classes)
num_classes = tf.shape(labels)[1]
stddev = 1.0 / tf.cast(input_size, tf.float32)
w_shape = tf.pack([input_size, num_classes], 'w-shape')
normal_dist = tf.truncated_normal(w_shape, stddev=stddev, name='normaldist')
self.w = tf.Variable(normal_dist, name='weights')
(I'm using tf.pack as suggested in this question, since I was getting the same error)
When I run it (from a larger script that invokes this one), I get this error:
ValueError: initial_value must have a shape specified: Tensor("normaldist:0", shape=TensorShape([Dimension(None), Dimension(None)]), dtype=float32)
I tried to replicate the process in the interactive shell. Indeed, the dimensions of normal_dist are unspecified, although the supplied values do exist:
In [70]: input_size.eval()
Out[70]: 4
In [71]: num_classes.eval()
Out[71]: 3
In [72]: w_shape.eval()
Out[72]: array([4, 3], dtype=int32)
In [73]: normal_dist.eval()
Out[73]:
array([[-0.27035281, -0.223277 , 0.14694688],
[-0.16527176, 0.02180306, 0.00807841],
[ 0.22624688, 0.36425814, -0.03099642],
[ 0.25575709, -0.02765726, -0.26169327]], dtype=float32)
In [78]: normal_dist.get_shape()
Out[78]: TensorShape([Dimension(None), Dimension(None)])
This is weird. Tensorflow generates the vector but can't say its shape. Am I doing something wrong?
As Ishamael says, all tensors have a static shape, which is known at graph construction time and accessible using Tensor.get_shape(); and a dynamic shape, which is only known at runtime and is accessible by fetching the value of the tensor, or passing it to an operator like tf.shape. In many cases, the static and dynamic shapes are the same, but they can be different - the static shape can be partially defined - in order allow the dynamic shape to vary from one step to the next.
In your code normal_dist has a partially-defined static shape, because w_shape is a computed value. (TensorFlow sometimes attempts to evaluate
these computed values at graph construction time, but it gets stuck at tf.pack.) It infers the shape TensorShape([Dimension(None), Dimension(None)]), which means "a matrix with an unknown number of rows and columns," because it knowns that w_shape is a vector of length 2, so the resulting normal_dist must be 2-dimensional.
You have two options to deal with this. You can set the static shape as Ishamael suggests, but this requires you to know the shape at graph construction time. For example, the following may work:
normal_dist.set_shape([input_data.get_shape()[1], labels.get_shape()[1]])
Alternatively, you can pass validate_shape=False to the tf.Variable constructor. This allows you to create a variable with a partially-defined shape, but it limits the amount of static shape information that can be inferred later on in the graph.
Similar question is nicely explained in TF FAQ:
In TensorFlow, a tensor has both a static (inferred) shape and a
dynamic (true) shape. The static shape can be read using the
tf.Tensor.get_shape method: this shape is inferred from the operations
that were used to create the tensor, and may be partially complete. If
the static shape is not fully defined, the dynamic shape of a Tensor t
can be determined by evaluating tf.shape(t).
So tf.shape() returns you a tensor, will always have a size of shape=(N,), and can be calculated in a session:
a = tf.Variable(tf.zeros(shape=(2, 3, 4)))
with tf.Session() as sess:
print sess.run(tf.shape(a))
On the other hand you can extract the static shape by using x.get_shape().as_list() and this can be calculated anywhere.
The variable can have a dynamic shape. get_shape() returns the static shape.
In your case you have a tensor that has a dynamic shape, and currently happens to hold value that is 4x3 (but at some other time it can hold a value with a different shape -- because the shape is dynamic). To set the static shape, use set_shape(w_shape). After that the shape you set will be enforced, and the tensor will be a valid initial_value.