I am training a model using 3D point cloud data in TensorFlow. My batch size is 64, so TensorFlow expects to receive batch of 64 of 3D points like: (64,1024,3). When I run the training code:
feed_dict = {ops['points_pl']: augmented_data,
ops['labels_pl']: current_label[start_idx:end_idx],
ops['w_pl']: gmm.weights_,
ops['mu_pl']: gmm.means_,
ops['sigma_pl']: np.sqrt(gmm.covariances_),
ops['is_training_pl']: is_training, }
summary, step, _, loss_val, pred_val = sess.run([ops['merged'], ops['step'],
ops['train_op'], ops['loss'], ops['pred']],
feed_dict=feed_dict)
In the last batch because the remaining data is less than 64, I get this error:
ValueError: Cannot feed value of shape (36, 1024, 3) for Tensor 'Placeholder_4:0', which has shape '(64, 1024, 3)'
I tried to manually add data at end of a batch when it is smaller than 64 but it significantly reduced the performance. When I set batch size to 1,2,4 it works okay but it ran very slowly. How can I get rid of this problem in an efficient way? Is there a way that TF to recognize such a situation and continue training without throwing an error?
You don't need to define the size of the batch dimension precisely. Instead you put None as the size of that dimension. You can define your placeholders e.g.:
n1 = 1024
n2 = 3
ops['points_pl'] = tf.placeholder(tf.float32, [None, n1, n2])
ops['labels_pl'] = tf.placeholder(tf.float32, [None])
Tensorflow will then allow you to feed those placeholders arrays without any restriction on the first dimension. This solves the problem of the final batch, and is also useful during inference (when you may want to apply the model to a different number of inputs than your batch size).
Related
I am building a CNN structure which is able to take multiple images as input. The number of inputs is varying —- for example it can be 3 or 4 or any other number ideally.
Here is what I want:
When input 3 images, there will be 3 streams of vgg16 which share the same weights.
When input 4 images, there will be 4 streams of vgg16 which share the same weights.
In conclusion, the network structure can only be define after I feed data during training/testing. How can I do it with tensorflow?
Consider an example of a TensorFlow placeholder :
x_train_placeholder = tf.placeholder(tf.float32, [None, 256, 256, 3])
In the above placeholder None is used for the size of the 1st dimension which allows us to pass a variable number of input tensors (images) of size (256, 256, 3) as input to the CNN model without explicitly specifying/hard-coding these values.
Consider, we input a batch of images x_train_batch which is of dimensions [4, 256, 256, 3] in which the first dimension is 4 indicating we're passing a batch of 4 images to the model (this could be modified to 3 according to your problem description). Now, we could pass x_train_batch to the placeholder x_train_placeholder above using the below code of code:
_, loss_train = sess.run([optimizer, loss], feed_dict={x_train_placeholder: x_train_batch})
where loss is the definition of the cost function we're using to optimize our network. Notice that since we're not hard-coding the value of the first dimension of x_train_batch anywhere, this allows us to pass a variable number of images to the tensorflow placeholder during training or testing.
The output layer of my CNN should use the RBF function, described as "each neuron outputs the square of the Euclidean distance between its input vector and its weight vector". I've implemented this as
dense2 = tf.square(tf.norm(dense1 - tf.transpose(dense2_W)))
where dense1 is a tensor of shape (?, 84). I've tried declaring dense2_W, the weights, as a variable of shape (84, 10) since it's doing number classification and should have 10 outputs. Running the code with a batch of 100 I get this error: InvalidArgumentError: Incompatible shapes: [100,84] vs. [10,84]. I believe it is due to the subtraction.
I train the network by iterating this code:
x_batch, y_batch = mnist.train.next_batch(100)
x_batch = tf.pad(x_batch, [[0,0],[2,2],[2,2],[0,0]]).eval() # Pad 28x28 -> 32x32
sess.run(train_step, {X: x_batch, Y: y_batch})
and then test it using the entire test set, thus the batch size in the network must be dynamic.
How can I work around this? The batch size must be dynamic, as in dense1's case, but I don't understand how to make a variable with dynamic size and transposing it (dense2_W).
You need the shapes of the two tensors to match. Assuming you want to share the weights across the batch and also having separate set of weights for each output class, you could reshape both of the tensors in order to be correctly broadcasted, e.g:
# broadcasting will copy the input to every output class neuron
input_dense = tf.expand_dims(dense1, axis=2)
# broadcasting here will copy the weights across the batch
weights = tf.expand_dims(tf.transpose(dense2_W), axis=0)
dense2 = tf.square(tf.norm(input_dense - weights, axis=1))
The resulting tensor dense2 should have shape of [batch_size, num_classes], which is [100, 10] in your case (so it will hold logits for every data instance over the number of output classes)
EDIT: added axis argument to the tf.norm call so that the distance is computed in the hidden dimension (not over the whole matrices).
I am working on OCR software optimized for phone camera images.
Currently, each 300 x 1000 x 3 (RGB) image is reformatted as a 900 x 1000 numpy array. I have plans for a more complex model architecture, but for now I just want to get a baseline working. I want to get started by training a static RNN on the data that I've generated.
Formally, I am feeding in n_t at each timestep t for T timesteps, where n_t is a 900-vector and T = 1000 (similar to reading the whole image left to right). Here is the Tensorflow code in which I create batches for training:
sequence_dataset = tf.data.Dataset.from_generator(example_generator, (tf.int32,
tf.int32))
sequence_dataset = sequence_dataset.batch(experiment_params['batch_size'])
iterator = sequence_dataset.make_initializable_iterator()
x_batch, y_batch = iterator.get_next()
The tf.nn.static_bidirectional_rnn documentation claims that the input must be a "length T list of inputs, each a tensor of shape [batch_size, input_size], or a nested tuple of such elements." So, I go through the following steps in order to get the data into the correct format.
# Dimensions go from [batch, n , t] -> [t, batch, n]
x_batch = tf.transpose(x_batch, [2, 0, 1])
# Unpack such that x_batch is a length T list with element dims [batch_size, n]
x_batch = tf.unstack(x_batch, experiment_params['example_t'], 0)
Without altering the batch any further, I make the following call:
output, _, _ = tf.nn.static_rnn(lstm_fw_cell, x_batch, dtype=tf.int32)
Note that I do not explicitly tell Tensorflow the dimensions of the matrices (this could be the problem). They all have the same dimensionality, yet I am getting the following bug:
ValueError: Input size (dimension 0 of inputs) must be accessible via shape
inference, but saw value None.
At which point in my stack should I be declaring the dimensions of my input? Because I am using a Dataset and hoping to get its batches directly to the RNN, I am not sure that the "placeholder -> feed_dict" route makes sense. If that in fact is the method that makes the most sense, let me know what that looks like (I definitely do not know). Otherwise, let me know if you have any other insights to the problem. Thanks!
The reason for the absence of static shape information is that TensorFlow doesn't understand enough about the example_generator function to determine the shapes of the arrays it yields, and so it assumes the shapes can be completely different from one element to the next. The best way to constrain this is to specify the optional output_shapes argument to tf.data.Dataset.from_generator(), which accepts a nested structure of shapes matching the structure of the yielded elements (and the output_types argument).
In this case you'd pass a tuple of two shapes, which can be partially specified. For example, if the x elements are 900 x 1000 arrays and the y elements are scalars:
sequence_dataset = tf.data.Dataset.from_generator(
example_generator, (tf.int32, tf.int32),
output_shapes=([900, 1000], []))
I need to create a random variable inside my model_fn(), having shape [batch_size, 20].
I do not want to pass batch_size as an argument, because then I cannot use a different batch size for prediction.
Removing the parts which do not concern this question, my model_fn() is:
def model(inp, out):
eps = tf.random_normal([batch_size, 20], 0, 1, name="eps"))) # batch_size is the
# value I do not want to hardcode
# dummy example
predictions = tf.add(inp, eps)
return predictions, 1
if I replace [batch_size, 20] by inp.get_shape(), I get
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 20)
when running myclf.setup_training().
If I try
def model(inp, out):
batch_size = tf.placeholder("float", [])
eps = tf.random_normal([batch_size.eval(), 20], 0, 1, name="eps")))
# dummy example
predictions = tf.add(inp, eps)
return predictions, 1
I get ValueError: Cannot evaluate tensor using eval(): No default session is registered. Usewith sess.as_default()or pass an explicit session to eval(session=sess) (understandably, because I have not provided a feed_dict)
How can I access the value of batch_size inside model_fn(), while remaining able to change it during prediction?
I wasn't aware of the difference between Tensor.get_shape() and tf.shape(Tensor). The latter works:
eps = tf.random_normal(tf.shape(inp), 0, 1, name="eps")))
As mentionned in Tensorflow 0.8 FAQ:
How do I build a graph that works with variable batch sizes?
It is often useful to build a graph that works with variable batch
sizes, for example so that the same code can be used for (mini-)batch
training, and single-instance inference. The resulting graph can be
saved as a protocol buffer and imported into another program.
When building a variable-size graph, the most important thing to
remember is not to encode the batch size as a Python constant, but
instead to use a symbolic Tensor to represent it. The following tips
may be useful:
Use batch_size = tf.shape(input)[0] to extract the batch dimension
from a Tensor called input, and store it in a Tensor called
batch_size.
Use tf.reduce_mean() instead of tf.reduce_sum(...) / batch_size.
If you use placeholders for feeding input, you can specify a variable
batch dimension by creating the placeholder with tf.placeholder(...,
shape=[None, ...]). The None element of the shape corresponds to a
variable-sized dimension.
My code used to work on tensorflow 0.6, but it no longer works on the lastest tensorflow.
I would like to perform inference every few training iterations. My training data is pulled from a queue, my inference data is from feed_dict. The training batch size is 128 while the inference batch size is 1. What Should I do to make the network accept the two different batch sizes?
batch_size = 128
x_batch = tf.placeholder("float", [None, 100])
q = tf.FIFOQueue(10, [tf.float32], shapes=[[batch_size, 100]])
enqueue_op = q.enqueue([x_batch])
# during training
x = q.dequeue() # dequeue operation
# network definition, takes x as input, and output y
......
# during inference
x_array_of_batch_size_1 = .. # a 1x100 numpy array
sess.run([y], feed_dict={x: x_array_of_batch_size_1))
I got the following error:
ValueError: Cannot feed value of shape (1, 100) for Tensor u'fifo_queue_Dequeue:0', which has shape '(128, 100)'
We added this check recently to prevent errors (and add a few optimization opportunities). You can make your program work again by changing the declaration of x to use the new tf.placeholder_with_default() op:
x = tf.placeholder_with_default(q.dequeue(), shape=[None, 100])