I need to extract the the sub tensors along axis=1 based on the value from another tensor.
state = tf.placeholder(shape=[None, None, 10], dtype=tf.float32)
length = tf.placeholder(shape=[None], dtype=tf.int32)
# this won't work, just be put here to demonstrate what I need
next_init_state = state[:, length - 1, :]
if state and length have deterministic shape, then next_init_state can be derived through gather_nd
state = tf.placeholder(shape=[10, 10, 10], dtype=tf.float32)
length = tf.placeholder(shape=[10], dtype=tf.int32)
index = tf.stack([tf.range(0, 10), length])
next_init_state = tf.gather_nd(state, index)
However since state and length all have nondeterministic shape None in the problem I have, the gather_nd approach won't work. At least I cannot think of a way to make it work. Is there any way to address it ?
I realized this is actually solved by Tensorflow's higher order function.
tf.map_fn(lambda x: x[0][x[1], :], (state, length), dtype=tf.float32)
Related
I've been following the tensorflow walkthrough here to create my own categorical OHE layer. The layer suggested is below and I've followed the preceding steps to the guide very closely:
def get_category_encoding_layer(name, dataset, dtype, max_tokens=None):
# Create a StringLookup layer which will turn strings into integer indices
if dtype == 'string':
index = preprocessing.StringLookup(max_tokens=max_tokens)
else:
index = preprocessing.IntegerLookup(max_tokens=max_tokens)
# Prepare a Dataset that only yields our feature
feature_ds = dataset.map(lambda x, y: x[name])
# Learn the set of possible values and assign them a fixed integer index.
index.adapt(feature_ds)
# Create a Discretization for our integer indices.
encoder = preprocessing.CategoryEncoding(num_tokens=index.vocabulary_size())
# Apply one-hot encoding to our indices. The lambda function captures the
# layer so we can use them, or include them in the functional model later.
return lambda feature: encoder(index(feature))
However the output isn't aligned with the guide. When my input to the layer is a list of n strings, instead of the output being shape (n, vocabulary size), I am receiving an output of shape (1, vocabulary size), with multiple categories incorrectly marked '1'.
e.g. using n=2 and vocabulary size=3
Instead of getting an OHE of [[1, 0, 0], [0, 1, 0]], I am getting [1, 1, 0].
My code is exactly the same as the guide, but it looks like the layer is "merging" the encoding of each element of my input. Is there something wrong with the layer they provided or could someone give pointer on what I could test?
By default, CategoryEncoding uses output_mode="multi_hot". That's why you're getting output of size (1, vocab_size). To get OHE of size (n, vocab_size), make this change in your code
encoder = preprocessing.CategoryEncoding(num_tokens=index.vocabulary_size(), output_mode='one_hot')
I am relatively new to tensorflow and am running into problems trying to index tensors properly. Towards the bottom of the shown, I am trying to use x (which itself is a tensor containing an array form such as [[0,1], [2,3]]) in order to index the y_rt tensor (one can think it as slicing the y_rt tensor). However, I am having troubles converting the tensor into an array or list. I am aware that there is the .eval() function, however I cannot use it here since the shown code happens before the .run() call. Any help would be much appreciated.
with tf.name_scope('placeholders'):
x_true = tf.placeholder(tf.float32, shape=[None, size, size, 1], name="x_true")
y_rt = tf.placeholder(tf.float32, shape=[None, operator.range.shape[0], operator.range.shape[1], 1], name="y_rt")
is_training = tf.placeholder(tf.bool, shape=(), name='is_training')
angle = tf.placeholder(tf.float32, shape=[n_batches, number], name="projection_order")
ordersel = tf.placeholder(tf.int32, shape=[n_batches, number], name='order_selection')
selection = tf.placeholder(tf.float32, shape=[number], name='iteration_selection')
dual = tf.placeholder(tf.float32, shape=[None, number, 183, 1], name='dual')
y_par = tf.placeholder(tf.float32, shape=[None, number, 183, 1], name='y_partial')
for i in range(n_batches): #iterations, the amount of projection batches we have
with tf.variable_scope('my_iterate{}'.format(i)):
value = layer_sub(primal[..., 1:2], epoch_angle[i])
x = (selection[i])
y_partial = y_rt[:, selection, :, :] #y_rt is of form (?,total, 183, 1)
As of version 1.9 TensorFlow doesn't support indexing by arrays using slice notation. From the tf.Tensor documentation for __getitem__:
This operation extracts the specified region from the tensor. The notation is similar to NumPy with the restriction that currently only support basic indexing. That means that using a non-scalar tensor as input is not currently allowed.
If you want more advanced indexing than simple scalars, tf.boolean_mask can help you select tensor elements using a boolean array and tf.gather_nd can help you select elements using an integer array.
Note in your example, the index specified by x would be a scalar with your 1-d selection tensor, and would work for slice notation if you used it:
x = selection[i]
y_partial = y_rt[:, x, :, :]
but indexing into selection for each of your training batches probably isn't what you want here.
I've got a params tensor with shape (?,368,5), as well as a query tensor with shape (?,368). The query tensor stores indices for sorting the first tensor.
The required output has shape: (?,368,5). Since I need it for a loss function in a neural network, the used operations should stay differentiable. Also, at runtime the size of the first axis ? corresponds to the batchsize.
So far I experimented with tf.gather and tf.gather_nd, however
tf.gather(params,query) results in a tensor with shape (?,368,368,5).
The query tensor is achieved by performing:
query = tf.nn.top_k(params[:, :, 0], k=params.shape[1], sorted=True).indices
Overall, I try to sort the params tensor by the first element on the third axis (for kind of a chamfer distance). At last to mention is, that I work with the Keras framework.
You need to add the indices of the first dimension to query in order to use it with tf.gather_nd. Here is a way to do it:
import tensorflow as tf
import numpy as np
np.random.seed(100)
with tf.Graph().as_default(), tf.Session() as sess:
params = tf.placeholder(tf.float32, [None, 368, 5])
query = tf.nn.top_k(params[:, :, 0], k=params.shape[1], sorted=True).indices
n = tf.shape(params)[0]
# Make tensor of indices for the first dimension
ii = tf.tile(tf.range(n)[:, tf.newaxis], (1, params.shape[1]))
# Stack indices
idx = tf.stack([ii, query], axis=-1)
# Gather reordered tensor
result = tf.gather_nd(params, idx)
# Test
out = sess.run(result, feed_dict={params: np.random.rand(10, 368, 5)})
# Check the order is correct
print(np.all(np.diff(out[:, :, 0], axis=1) <= 0))
# True
I'm trying to build a seq2seq model in tensorflow (1.4) using the tf.contrib.rnn.ConvLSTMCell API together with the tf.nn.dynamic_rnn API, but I got an error with the dimension of the inputs.
My code is:
# features is an image sequence with shape [600, 400, 10],
# so features is a tensor with shape [batch_size, 600, 400, 10]
features = tf.transpose(features, [0,3,1,2])
features = tf.reshape(features, [params['batch_size'],10,600,400])
encoder_cell = tf.contrib.rnn.ConvLSTMCell(conv_ndims=2,
input_shape=[600, 400,1],
output_channels=5,
kernel_shape=[7,7],
skip_connection=False)
_, encoder_state = tf.nn.dynamic_rnn(cell=encoder_cell,
inputs=features,
sequence_length=[10]*params['batch_size'],
dtype=tf.float32)
I get the following error
ValueError: Conv Linear expects all args to be of same Dimension: [[2, 600, 400], [2, 600, 400, 5]]
Looking at the tf implementation, it seems that the inputs to dynamic_rnn is only 3-dimensional in contrary to the hidden state, which is 4-dimensional. I tried to pass the input as a nested tuple, but it didn't work.
The problem is similar to TensorFlow dynamic_rnn regressor: ValueError dimension mismatch, it's slightly different though, as they're using a plain LSTMCell (which worked for me).
Can anyone give me a minimal example how to use these 2 APIs together?
Thanks!
As I understand from here https://github.com/iwyoo/ConvLSTMCell-tensorflow/issues/2
Currently, tf.nn.dynamic_rnn doesn't support ConvLSTMCell.
Therefore, as described here, https://github.com/iwyoo/ConvLSTMCell-tensorflow/issues/1 you have to manually create the RNN.
An example is provided in the documentation, https://github.com/iwyoo/ConvLSTMCell-tensorflow/blob/master/README.md
Below I have modified your code according to the above example with the comments where necessary.
height = 400
width = 400
time_steps = 25
channel = 10
batch_size = 2
p_input = tf.placeholder(tf.float32, [None, height, width, time_steps, channel])
p_label = tf.placeholder(tf.float32, [None, height, width, 3])
p_input_list = tf.split(p_input, step_size, 3) # creates a list of leghth time_steps and one elemnt has the shape of (?, 400, 400, 1, 10)
p_input_list = [tf.squeeze(p_input_, [3]) for p_input_ in p_input_list] #remove the third dimention now one list elemnt has the shape of (?, 400, 400, 10)
cell = tf.contrib.rnn.ConvLSTMCell(conv_ndims=2, # ConvLSTMCell definition
input_shape=[height, width, channel],
output_channels=5,
kernel_shape=[7, 7],
skip_connection=False)
state = cell.zero_state(batch_size, dtype=tf.float32) #initial state is zero
with tf.variable_scope("ConvLSTM") as scope: # as BasicLSTMCell # create the RNN with a loop
for i, p_input_ in enumerate(p_input_list):
if i > 0:
scope.reuse_variables()
# ConvCell takes Tensor with size [batch_size, height, width, channel].
t_output, state = cell(p_input_, state)
Notice that you have to input an image that has the same height and width. If your height and width doesn't match, then you may have to do padding.
hope this helps.
In the meantime I figured out how to use the 2 APIs together. The trick is to pass a 5D-Tensor as input to tf.nn.dynamic_rnn(), where the last dimension is the size of the "vector on the spatial grid" (which comes from the transformation of the input from 2D to 3D, inspired by the paper on which the implementation is based: https://arxiv.org/pdf/1506.04214.pdf). In my case the vector size is 1, I have to expand the dimension anyway though.
While fixing this error another issue emerged: In the paper mentioned above in section 3.1 they state the equations for the convLSTM. They use the Hadamard-product for weights connected to the cell outputs C. Printing the weights of my ConvLSTMCell in Tensorflow, it seems like they don't use the weights Wci, Wcf and Wco at all. So, can anybody tell me the exact implementation of the TF ConvLSTMCell?
Btw. the output of the tensorflow ConvSTMCell is C or H (in the notation of the paper)?
I have sequences of different lengths that I want to classify using LSTMs in Tensorflow. For the classification I just need the LSTM output of the last timestep of each sequence.
max_length = 10
n_dims = 2
layer_units = 5
input = tf.placeholder(tf.float32, [None, max_length, n_dims])
lengths = tf.placeholder(tf.int32, [None])
cell = tf.nn.rnn_cell.LSTMCell(num_units=layer_units, state_is_tuple=True)
sequence_outputs, last_states = tf.nn.dynamic_rnn(cell, sequence_length=lengths, inputs=input)
I would like to get, in NumPy notation: output = sequence_outputs[:,lengths]
Is there any way or workaround to get this behaviour in Tensorflow?
---UPDATE---
Following this post How to select rows from a 3-D Tensor in TensorFlow? it seems that is possible to solve the problem in an efficient manner with tf.gather and manipulating the indices. The only requirement is that the batch size must be known in advance. Here is the adaptation of the referred post to this concrete problem:
max_length = 10
n_dims = 2
layer_units = 5
batch_size = 2
input = tf.placeholder(tf.float32, [batch_size, max_length, n_dims])
lengths = tf.placeholder(tf.int32, [batch_size])
cell = tf.nn.rnn_cell.LSTMCell(num_units=layer_units, state_is_tuple=True)
sequence_outputs, last_states = tf.nn.dynamic_rnn(cell,
sequence_length=lengths, inputs=input)
#Code adapted from #mrry response in StackOverflow:
#https://stackoverflow.com/questions/36088277/how-to-select-rows-from-a-3-d-tensor-in-tensorflow
rows_per_batch = tf.shape(input)[1]
indices_per_batch = 1
# Offset to add to each row in indices. We use `tf.expand_dims()` to make
# this broadcast appropriately.
offset = tf.range(0, batch_size) * rows_per_batch
# Convert indices and logits into appropriate form for `tf.gather()`.
flattened_indices = lengths - 1 + offset
flattened_sequence_outputs = tf.reshape(self.sequence_outputs, tf.concat(0, [[-1],
tf.shape(sequence_outputs)[2:]]))
selected_rows = tf.gather(flattened_sequence_outputs, flattened_indices)
last_output = tf.reshape(selected_rows,
tf.concat(0, [tf.pack([batch_size, indices_per_batch]),
tf.shape(self.sequence_outputs)[2:]]))
#petrux option (Get the last output of a dynamic_rnn in TensorFlow) seems also to work but the need of building a list within a for loop may be less optimized, although I did not perform any benchmark to support this statement.
This could be an answer. I don't think there is anything similar to the NumPy notation you pointed out, but the effect is the same.
Here's a solution, using gather_nd, where batch size does not need to be known ahead of time.
def extract_axis_1(data, ind):
"""
Get specified elements along the first axis of tensor.
:param data: Tensorflow tensor that will be subsetted.
:param ind: Indices to take (one for each element along axis 0 of data).
:return: Subsetted tensor.
"""
batch_range = tf.range(tf.shape(data)[0])
indices = tf.stack([batch_range, ind], axis=1)
res = tf.gather_nd(data, indices)
return res
output = extract_axis_1(sequence_outputs, lengths - 1)
Now output is a tensor of dimension [batch_size, num_cells].