I'm using tensorflow run queues to feed my data during training time:
X, Y = tf.train.batch(
[image, label],
batch_size=64
)
However X, Y have forced shape of [64, 32, 32,3], and [64, 10]. During evaluation time I'd like to run loss operation on whole test set, which has dimensions: [10000, 32, 32, 3] and [10000, 10]. I would use feed_dict property in session.run() to overwrite X,Y with my values, however they have incompatible shapes.
Can I somehow instruct tensorflow to forget about first dimension, that is reshape [64, 32, 32, 3] -> [None, 32, 32, 3]? Or is there any other option for me to replace X,Y with another value.
Whole dataset is small enough to fit in memory, therefore I'm using similar approach as in https://github.com/tensorflow/tensorflow/blob/r0.9/tensorflow/examples/how_tos/reading_data/fully_connected_preloaded.py
This is a bit subtle: in TensorFlow terminology, you don't actually want to reshape the tensor (i.e. change the number of elements in each dimension), but instead you want TensorFlow to "forget" a specific dimension, in order to feed values with a range of sizes.
The tf.placeholder_with_default() op is designed to support this case. It takes a default input, which in your case would be the next training batch (of shape [64, ...]); and a shape, which in your case would be the same shape as the input, with the first dimension set to None. You can then feed this placeholder with values of any batch size.
Here's an example of how you'd use it:
X_batch, Y_batch = tf.train.batch([image, label], batch_size=64)
# Alternatively, `X_shape = [None, 32, 32, 3]`
X_shape = tf.TensorShape([None]).concatenate(X_batch.get_shape()[1:])
# Alternatively, `Y_shape = [None, 10]`
Y_shape = tf.TensorShape([None]).concatenate(Y_batch.get_shape()[1:])
# Create tensors that can be fed with a less specific shape
# than `X_batch`, `Y_batch`.
X = tf.placeholder_with_default(X_batch, shape=X_shape)
Y = tf.placeholder_with_default(Y_batch, shape=Y_shape)
Related
Checking frozen tensorflow model:
wget https://storage.googleapis.com/download.tensorflow.org/models/inception_v3_2016_08_28_frozen.pb.tar.gz
I see that input size is Tensor 'input:0', which has shape '(1, 299, 299, 3)', I wonder is it possible to make input (None, 299, 299, 3) to make availible batch prediction with batch_size > 1?
In the general case it may not be possible to do this, as there could be operations that rely on the first dimension being 1 (e.g. suppose tf.squeeze is used on input:0). However, you can try to replace the input with a placeholder of the desired shape. You can do this with tf.graph_util.import_graph_def. If the operations allow it, then TensorFlow should import the graph adjusting the node shapes accordingly. See the following example:
import tensorflow as tf
# First graph
with tf.Graph().as_default():
x = tf.placeholder(tf.float32, [1, 10, 20], name='Input')
y = tf.square(x, name='Output')
print(y)
# Tensor("Output:0", shape=(1, 10, 20), dtype=float32)
gd = tf.get_default_graph().as_graph_def()
# Second graph
with tf.Graph().as_default():
x = tf.placeholder(tf.float32, [None, 10, 20], name='Input')
y, = tf.graph_util.import_graph_def(gd, input_map={'Input:0': x},
return_elements=['Output:0'], name='')
print(y)
# Tensor("Output:0", shape=(?, 10, 20), dtype=float32)
In the first graph, the Output:0 node has a shape (1, 10, 20), which is inferred from the shape of the Input:0 tensor. However, when I take the graph definition from the first graph and load in the second graph, replacing the Input:0 tensor with a placeholder with undefined first dimension, the shape of Output:0 is updated to (?, 10, 20). If I run the operations in the second graph giving an input value with a first dimension greater than one, it will work as expected, because the graph is correct.
I want to compute the reconstruction accuracy of my autoencoder using CrossEntropyLoss:
ae_criterion = nn.CrossEntropyLoss()
ae_loss = ae_criterion(X, Y)
where X is the autoencoder's reconstruction and Y is the target (since it is an autoencoder, Y is the same as the original input X).
Both X and Y have shape [42, 32, 130] = [batch_size, timesteps, number_of_classes]. When I run the code above I get the following error:
ValueError: Expected target size (42, 130), got torch.Size([42, 32,
130])
After looking the docs, I'm still unsure on how should I call nn.CrossEntropyLoss() in the appropriate way. It seems that I should change Y to be of shape [42, 32, 1], with each element being a scalar in the interval [0, 129] (or [1, 130]), am I right?
Is there a way to avoid this? Since X and Y are between 0 and 1, could I just use binary cross-entropy loss element-wise in an equivalent way?
For CrossEntropyLoss, shape of the Y must be (42, 32), each element must be a Long scalar in the interval [0, 129].
You may want to use BCELoss or BCEWithLogitsLoss for your problem.
I am training a model using 3D point cloud data in TensorFlow. My batch size is 64, so TensorFlow expects to receive batch of 64 of 3D points like: (64,1024,3). When I run the training code:
feed_dict = {ops['points_pl']: augmented_data,
ops['labels_pl']: current_label[start_idx:end_idx],
ops['w_pl']: gmm.weights_,
ops['mu_pl']: gmm.means_,
ops['sigma_pl']: np.sqrt(gmm.covariances_),
ops['is_training_pl']: is_training, }
summary, step, _, loss_val, pred_val = sess.run([ops['merged'], ops['step'],
ops['train_op'], ops['loss'], ops['pred']],
feed_dict=feed_dict)
In the last batch because the remaining data is less than 64, I get this error:
ValueError: Cannot feed value of shape (36, 1024, 3) for Tensor 'Placeholder_4:0', which has shape '(64, 1024, 3)'
I tried to manually add data at end of a batch when it is smaller than 64 but it significantly reduced the performance. When I set batch size to 1,2,4 it works okay but it ran very slowly. How can I get rid of this problem in an efficient way? Is there a way that TF to recognize such a situation and continue training without throwing an error?
You don't need to define the size of the batch dimension precisely. Instead you put None as the size of that dimension. You can define your placeholders e.g.:
n1 = 1024
n2 = 3
ops['points_pl'] = tf.placeholder(tf.float32, [None, n1, n2])
ops['labels_pl'] = tf.placeholder(tf.float32, [None])
Tensorflow will then allow you to feed those placeholders arrays without any restriction on the first dimension. This solves the problem of the final batch, and is also useful during inference (when you may want to apply the model to a different number of inputs than your batch size).
I have built a deep CNN based on a research paper and now I am attempting to train it. After all the convolutions and deconvolutions I have performed, I have a result called final.
final = tf.add(add1,add2)
print(final)
Tensor("Add_35:0", shape=(1, 32, 32, 7, 1), dtype=float32)
In my model, I have an image of the size 32x32x7 where each pixel has a corresponding density. The output of the model will be a label for each pixel. Therefore, I have declared two placeholders where "x" represents the input and "y_" represents the output.
x = tf.placeholder(tf.float32, shape=[None, 7168])
y_ = tf.placeholder(tf.float32, shape=[None, 7168])
Now that I am attempting to train the model, I have this line
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=final))
When the model is being trained I receive the error : logits and labels must be same size: logits_size=[7168,1] labels_size=[1,7168]
It makes sense that labels would be this size since this is how I declared it. However, I do not understand why logits is of size [7168,1] when printing out "final" has the shape (1, 32, 32, 7, 1).
Just tf.reshape your final:
final = tf.reshape(final, [None, 7168])
Although I am not sure why it is automatically flattening when you call softmax_cross_entropy_with_logits...
I need to create a random variable inside my model_fn(), having shape [batch_size, 20].
I do not want to pass batch_size as an argument, because then I cannot use a different batch size for prediction.
Removing the parts which do not concern this question, my model_fn() is:
def model(inp, out):
eps = tf.random_normal([batch_size, 20], 0, 1, name="eps"))) # batch_size is the
# value I do not want to hardcode
# dummy example
predictions = tf.add(inp, eps)
return predictions, 1
if I replace [batch_size, 20] by inp.get_shape(), I get
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 20)
when running myclf.setup_training().
If I try
def model(inp, out):
batch_size = tf.placeholder("float", [])
eps = tf.random_normal([batch_size.eval(), 20], 0, 1, name="eps")))
# dummy example
predictions = tf.add(inp, eps)
return predictions, 1
I get ValueError: Cannot evaluate tensor using eval(): No default session is registered. Usewith sess.as_default()or pass an explicit session to eval(session=sess) (understandably, because I have not provided a feed_dict)
How can I access the value of batch_size inside model_fn(), while remaining able to change it during prediction?
I wasn't aware of the difference between Tensor.get_shape() and tf.shape(Tensor). The latter works:
eps = tf.random_normal(tf.shape(inp), 0, 1, name="eps")))
As mentionned in Tensorflow 0.8 FAQ:
How do I build a graph that works with variable batch sizes?
It is often useful to build a graph that works with variable batch
sizes, for example so that the same code can be used for (mini-)batch
training, and single-instance inference. The resulting graph can be
saved as a protocol buffer and imported into another program.
When building a variable-size graph, the most important thing to
remember is not to encode the batch size as a Python constant, but
instead to use a symbolic Tensor to represent it. The following tips
may be useful:
Use batch_size = tf.shape(input)[0] to extract the batch dimension
from a Tensor called input, and store it in a Tensor called
batch_size.
Use tf.reduce_mean() instead of tf.reduce_sum(...) / batch_size.
If you use placeholders for feeding input, you can specify a variable
batch dimension by creating the placeholder with tf.placeholder(...,
shape=[None, ...]). The None element of the shape corresponds to a
variable-sized dimension.