We Keep Coding

Logits have the wrong shape

When I attempt to use the softmax cross entropy function, I get a ValueError saying
ValueError: Rank mismatch: Rank of labels (received 2) should equal rank of logits minus 1 (received 2).
The thing is that my layers are built in such a way that my logits should output only 1 value.
The shape of my logits is (5, 1) but I have no idea why there is a 5. The X for each instance is a 5x7 matrix
X = tf.placeholder(shape=(1, 5, 7), name='inputs', dtype=tf.float32)
y = tf.placeholder(shape=(1, 1), name='outputs', dtype=tf.int32)
hidden1 = tf.layers.dense(X, 150)
hidden2 = tf.layers.dense(hidden1, 50)
logits = tf.layers.dense(hidden2, 1)
with tf.name_scope("loss"):
xentropy= tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y,
logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")

Edit
Check the comment, and try this code.
X = tf.placeholder(shape=(1, 5, 7), name='inputs', dtype=tf.float32)
y = tf.placeholder(shape=(1), name='outputs', dtype=tf.int32)
flattened = tf.layers.flatten(X) # shape (1,35)
hidden1 = tf.layers.dense(flattened, 150) # shape (1,150)
hidden2 = tf.layers.dense(hidden1, 50) # shape (1,50)
logits = tf.layers.dense(hidden2, 1) # shape (1,1)
with tf.name_scope("loss"):
# expects logits of shape (1,1) against labels of shape (1)
xentropy= tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y,
logits=logits)
loss = tf.reduce_mean(xentropy, name="loss")
Original
Let's think through what's going on here.
You create an X placeholder with the shape (5,7) (presumably (batch_size, data_size)).
You feed it into a hidden layer, which transforms the shape from (batch_size, data_size) to (batch_size, units) (units here is 150)
Likewise for the next two layers with hidden2 and logits, resulting in logits having shape (batch_size, 1), which is (5, 1) in this case
You're computing cross entropy between the labels and logits. The requirement for shapes here is for logits to have shape (batch_size, num_classes), where each value is the weight for a particular class, and for labels to have shape (batch_size), where each value is the class number for that particular sample. So this is where things go wrong for you. Your y has shape (1,1), and TF is expecting just a tensor or shape (5).
From what I'm guessing, I think you're trying to directly feed forward X as the data of a single sample (so like a (5,7) shaped matrix). If this is the case, you should have X take the shape (1,5,7) to signify to Tensorflow that X only represents one piece of data.

The thing is that my layers are built in such a way that my logits should output only 1 value.
That's not true. When X is an a X b tensor and you do tf.layers.dense(X, c), you are multiplying X by a b X c matrix (and bias also added of size c). So output size is a X c.
In your case since the first dimension of X is 5, it continues to be 5 even for logits. And your logits should be of size 5. So you are definitely doing something wrong. It is difficult to say what is without more information.

Is it possible to remove batch dimension from frozen graph?

Checking frozen tensorflow model:
wget https://storage.googleapis.com/download.tensorflow.org/models/inception_v3_2016_08_28_frozen.pb.tar.gz
I see that input size is Tensor 'input:0', which has shape '(1, 299, 299, 3)', I wonder is it possible to make input (None, 299, 299, 3) to make availible batch prediction with batch_size > 1?

In the general case it may not be possible to do this, as there could be operations that rely on the first dimension being 1 (e.g. suppose tf.squeeze is used on input:0). However, you can try to replace the input with a placeholder of the desired shape. You can do this with tf.graph_util.import_graph_def. If the operations allow it, then TensorFlow should import the graph adjusting the node shapes accordingly. See the following example:
import tensorflow as tf
# First graph
with tf.Graph().as_default():
x = tf.placeholder(tf.float32, [1, 10, 20], name='Input')
y = tf.square(x, name='Output')
print(y)
# Tensor("Output:0", shape=(1, 10, 20), dtype=float32)
gd = tf.get_default_graph().as_graph_def()
# Second graph
with tf.Graph().as_default():
x = tf.placeholder(tf.float32, [None, 10, 20], name='Input')
y, = tf.graph_util.import_graph_def(gd, input_map={'Input:0': x},
return_elements=['Output:0'], name='')
print(y)
# Tensor("Output:0", shape=(?, 10, 20), dtype=float32)
In the first graph, the Output:0 node has a shape (1, 10, 20), which is inferred from the shape of the Input:0 tensor. However, when I take the graph definition from the first graph and load in the second graph, replacing the Input:0 tensor with a placeholder with undefined first dimension, the shape of Output:0 is updated to (?, 10, 20). If I run the operations in the second graph giving an input value with a first dimension greater than one, it will work as expected, because the graph is correct.

How to determine weight dimension for input tensor of Rank-3?

I'm trying to design an Autoencoder for activity classification for 3-channel input (Tri-axial accelerometer data).
The input tensor is of shape [None,200,3] ([Batch size, window size, number of channels]) and in the first layer, I want to simply reduce the dimension of input layer to [None,150,3]. Here is the code for creating placeholders and the first layer:
import tensorflow as tf
def denseLayer(inputVal,weight,bias):
return tf.nn.relu((tf.matmul(inputVal,weight)+bias))
x = tf.placeholder(dtype=tf.float32,shape=[None,200,3]) #Input tensor
wIn = tf.get_variable(name='wIn',initializer=tf.truncated_normal(stddev=0.1,dtype=tf.float32,shape=[200,150]))
bIn = tf.get_variable(name='bIn',initializer=tf.constant(value = 0.1,shape=[150,3],dtype=tf.float32))
firstLayer = denseLayer(x,weight=wIn,bias=bIn)
This code will, of course, result in an error (due to the difference in rank between x and wIn) and i am unable to determine the shape of wIn variable to get the desired shape of firstLayer that is [None,150,3].
Here is how the final network should look (simplified version with lesser layers):

I think this does what you want:
import tensorflow as tf
def denseLayer(inputVal, weight, bias):
# Each input "channel" uses the corresponding set of weights
value = tf.einsum('nic,ijc->njc', inputVal, weight) + bias
return tf.nn.relu(value)
#Input tensor
x = tf.placeholder(dtype=tf.float32, shape=[None, 200, 3])
# Weights and biases have three "channels" each
wIn = tf.get_variable(name='wIn',
shape=[200, 150, 3],
initializer=tf.truncated_normal_initializer(stddev=0.1))
bIn = tf.get_variable(name='bIn',
shape=[150, 3],
initializer=tf.constant_initializer(value=0.1))
firstLayer = denseLayer(x, weight=wIn, bias=bIn)
print(firstLayer)
# Tensor("Relu:0", shape=(?, 150, 3), dtype=float32)
Here wIn can be seen as three sets of [200, 150] parameters that are applied to each input channel. I think tf.einsum is the easiest way to implement that in this case.

Rank mismatch: Rank of labels (received 2) should equal rank of logits minus 1 (received 2)

I'm building DNN to predict if the object is present in the image or not. My network has two hidden layers and the last layer looks like this:
# Output layer
W_fc2 = weight_variable([2048, 1])
b_fc2 = bias_variable([1])
y = tf.matmul(h_fc1, W_fc2) + b_fc2
Then I have placeholder for labels:
y_ = tf.placeholder(tf.float32, [None, 1], 'Output')
I run training in batches (therefore first argument in Output layer shape is None).
I use the following loss function:
cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(
y[:, :1], y_[:, :1], name='xentropy')
loss = tf.reduce_mean(cross_entropy, name='xentropy_mean')
predict_hand = tf.greater(y, 0.5)
correct_prediction = tf.equal(tf.to_float(predict_hand), y_)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
But in runtime I got the following error:
Rank mismatch: Rank of labels (received 2) should equal rank of logits
minus 1 (received 2).
I guess I should reshape labels layer, but not sure what it expects. I looked up in documentation and it says:
logits: Unscaled log probabilities of rank r and shape [d_0, d_1, ...,
d_{r-2}, num_classes] and dtype float32 or float64. labels: Tensor of
shape [d_0, d_1, ..., d_{r-2}] and dtype int32 or int64. Each entry in
labels must be an index in [0, num_classes).
If I have just single class, what my labels should look like (now it is just 0 or 1)? Any help appreciated

From the documentation* for tf.nn.sparse_softmax_cross_entropy_with_logits:
"A common use case is to have logits of shape [batch_size,
num_classes] and labels of shape [batch_size]. But higher dimensions
are supported."
So I suppose your labels tensor should be of shape [None]. Note that a given tensor with shape [None, 1] or shape [None] will contain the same number of elements.
Example input with concrete dummy values:
>>> logits = np.array([[11, 22], [33, 44], [55, 66]])
>>> labels = np.array([1, 0, 1])
Where there's 3 examples in the mini-batch, the logits for the first example are 11 and 22 and there's 2 classes: 0 and 1.
*https://www.tensorflow.org/versions/r0.11/api_docs/python/nn.html#sparse_softmax_cross_entropy_with_logits

The problem may be the activation function in your network. Use tf.nn.softmax_cross_entropy_with_logits instead of sparse_softmax. This will solve the issue.

In short, here is implements of it
cost = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
logits=hypothesis,labels=tf.argmax(Y,1)))
sparse_softmax_cross_entropy_with_logits
Computes sparse softmax cross entropy between logits and labels.
Measures the probability error in discrete classification tasks in
which the classes are mutually exclusive (each entry is in exactly
one class).
For example, each CIFAR-10 image is labeled with one and only one
label: an image can be a dog or a truck, but not both.
NOTE: For this operation, the probability of a given label is
considered exclusive. That is, soft classes are not allowed,
and the labels vector must provide a single specific index for the
true class for each row of logits (each minibatch entry).
For soft softmax classification with a probability distribution
for each entry, see softmax_cross_entropy_with_logits.
WARNING: This op expects unscaled logits, since it performs a softmax
on logits internally for efficiency. Do not call this op with the
output of softmax, as it will produce incorrect results.
A common use case is to have logits of shape [batch_size, num_classes]
and labels of shape [batch_size]. But higher dimensions are supported.
Note that to avoid confusion, it is required to pass only named
arguments to this function.
softmax_cross_entropy_with_logits_v2 and softmax_cross_entropy_with_logits
Computes softmax cross entropy between logits and labels. (deprecated)
THIS FUNCTION IS DEPRECATED. It will be removed in a future version.
Instructions for updating:
Future major versions of TensorFlow will allow gradients to flow into
the labels input on backprop by default. Backpropagation will happen
only into logits. To calculate a cross entropy loss that allows
backpropagation into both logits and labels, see softmax_cross_entropy_with_logits_v2
Measures the probability error in discrete classification tasks in
which the classes are mutually exclusive (each entry is in exactly one
class).
For example, each CIFAR-10 image is labeled with one and only one
label: an image can be a dog or a truck, but not both.
here is the same implements of softmax_cross_entropy_with_logits_v2
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(
logits=hypothesis,labels=Y))

How to reshape data to None in tensorflow?

I'm using tensorflow run queues to feed my data during training time:
X, Y = tf.train.batch(
[image, label],
batch_size=64
)
However X, Y have forced shape of [64, 32, 32,3], and [64, 10]. During evaluation time I'd like to run loss operation on whole test set, which has dimensions: [10000, 32, 32, 3] and [10000, 10]. I would use feed_dict property in session.run() to overwrite X,Y with my values, however they have incompatible shapes.
Can I somehow instruct tensorflow to forget about first dimension, that is reshape [64, 32, 32, 3] -> [None, 32, 32, 3]? Or is there any other option for me to replace X,Y with another value.
Whole dataset is small enough to fit in memory, therefore I'm using similar approach as in https://github.com/tensorflow/tensorflow/blob/r0.9/tensorflow/examples/how_tos/reading_data/fully_connected_preloaded.py

This is a bit subtle: in TensorFlow terminology, you don't actually want to reshape the tensor (i.e. change the number of elements in each dimension), but instead you want TensorFlow to "forget" a specific dimension, in order to feed values with a range of sizes.
The tf.placeholder_with_default() op is designed to support this case. It takes a default input, which in your case would be the next training batch (of shape [64, ...]); and a shape, which in your case would be the same shape as the input, with the first dimension set to None. You can then feed this placeholder with values of any batch size.
Here's an example of how you'd use it:
X_batch, Y_batch = tf.train.batch([image, label], batch_size=64)
# Alternatively, `X_shape = [None, 32, 32, 3]`
X_shape = tf.TensorShape([None]).concatenate(X_batch.get_shape()[1:])
# Alternatively, `Y_shape = [None, 10]`
Y_shape = tf.TensorShape([None]).concatenate(Y_batch.get_shape()[1:])
# Create tensors that can be fed with a less specific shape
# than `X_batch`, `Y_batch`.
X = tf.placeholder_with_default(X_batch, shape=X_shape)
Y = tf.placeholder_with_default(Y_batch, shape=Y_shape)