I am trying to process a tensor of variable size, in a python way that would be something like:
# X is of shape [m, n]
for x in X:
process(x)
I have tried to use tf.scan, the thing is that I want to process every sub-tensor, so I have tried to use a nested scan, but I was enable to do it, because tf.scan work with the accumulator, if not found it will take the first entry of the elems as initializer, which I don't want to do.
As an example, suppose I want to add one to every element of my tensor (this is just an example), and I want to process it element by element. If I run the code bellow, I will only have one added to a sub-tensor, because scan consider the first tensor as initializer, along with the first element of every sub-tensor.
import numpy as np
import tensorflow as tf
batch_x = np.random.randint(0, 10, size=(5, 10))
x = tf.placeholder(tf.float32, shape=[None, 10])
def inner_loop(x_in):
return tf.scan(lambda _, x_: x_ + 1, x_in)
outer_loop = tf.scan(lambda _, input_: inner_loop(input_), x, back_prop=True)
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
rs = sess.run(outer_loop, feed_dict={x: batch_x})
Any suggestions ?
To loop over a tensor you could try tf.unstack
Unpacks the given dimension of a rank-R tensor into rank-(R-1) tensors.
So adding 1 to each tensor would look something like:
import tensorflow as tf
x = tf.placeholder(tf.float32, shape=(None, 10))
x_unpacked = tf.unstack(x) # defaults to axis 0, returns a list of tensors
processed = [] # this will be the list of processed tensors
for t in x_unpacked:
# do whatever
result_tensor = t + 1
processed.append(result_tensor)
output = tf.concat(processed, 0)
with tf.Session() as sess:
print(sess.run([output], feed_dict={x: np.zeros((5, 10))}))
Obviously you can further unpack each tensor from the list to process it, down to single elements. To avoid lots of nested unpacking though, you could maybe try flattening x with tf.reshape(x, [-1]) first, and then loop over it like
flattened_unpacked = tf.unstack(tf.reshape(x, [-1])
for elem in flattened_unpacked:
process(elem)
In this case elem is a scalar.
Most of tensorflow built-in functions could be applied elementwise. So you could just pass a tensor into a function. Like:
outer_loop = inner_loop(x)
However, if you have some function that could not be applied this way (it's really tempting to see that function), you could use map_fn.
Say, your function simply adds 1 to every element of a tensor (or whatever):
inputs = tf.placeholder...
def my_elementwise_func(x):
return x + 1
def recursive_map(inputs):
if tf.shape(inputs).ndims > 0:
return tf.map_fn(recursive_map, inputs)
else:
return my_elementwise_func(inputs)
result = recursive_map(inputs)
Related
I'm using tensorflow 1.14.0. I would like to know how I can type cast list into tensor. I get this error when trying to use tf.convert_to_tensor(). Appreciate any help
Failed to convert object of type to Tensor. Contents: [None]. Consider casting elements to a supported type.
Here is my code
def testtf4():
x = tf.placeholder(tf.float32, shape=[None])
y = tf.placeholder(tf.float32, shape=[None])
op = tf.placeholder(tf.float32, shape=[None,3])
print("\nshape of x,y", x.shape, y.shape)
arr = np.genfromtxt("C:\\Data\\Training_and_codes\\ML\\TF Samples\\Data.csv", delimiter=",");
gradmulx_op = tf.gradients(op[:,0],x)
gradmuly_op = tf.gradients(op[:,0],y)
tgradmulx_op = tf.convert_to_tensor(gradmulx_op)
tgradmuly_op = tf.convert_to_tensor(gradmuly_op)
print("\nshape of gradmul tensors", tgradmulx_op.shape, tgradmuly_op.shape)
with tf.Session() as sess:
print("started session......\n")
input_feed={}
input_feed[x]=arr[:,0]
input_feed[y]=arr[:,1]
input_feed[op]=arr[:,2:4]
[gradx, grady] = sess.run([tgradmulx_op, tgradmuly_op],input_feed)
print("x gradient",gradx)
print("y gradient",grady)
Your problem does not have to do with tf.convert_to_tensor, but with the fact that your are trying to compute some gradients that do not exist. You have these two placeholders:
x = tf.placeholder(tf.float32, shape=[None])
op = tf.placeholder(tf.float32, shape=[None, 3])
And then you try to get the following gradients:
gradmulx_op = tf.gradients(op[:, 0], x)
gradmuly_op = tf.gradients(op[:, 0], y)
For these gradients to exist (that is, not be None), the value of op[:, 0] would have to be the result of one or more differentiable operations using x and y. For example, if op were defined as:
op = tf.stack([2 * x + 3 * y, x - 1, 2 * y + 2], axis=1)
Then it would work, because op[:, 0] would be computed from x and y (and possibly other values), so there is a gradient between the tensors. Or, put it a different way, changing x or y changes the value of op[:, 0]. TensorFlow keeps track of the operations used to compute each value and uses that information to automatically compute the gradients.
But op is not calculated from x and y, in fact it is not calculated from anything, since it is a placeholder, it is just a given value. A change in x or y does not entail a change in op. So there is no gradients between those tensors. I am not sure what you are trying to achieve with your code, but you probably need to rethink what exactly is the result that you want to compute.
I've searched across many tutorials/blogs/guides and official Tensorflow documentation to understand this. For example, see below lines:
lstm = tf.nn.rnn_cell.LSTMCell(512)
output, state_tuple = lstm(current_input, last_state_tuple)
Now if I unpack state,
last_cell_memory, last_hidden_state = state_tuple
Both output and last_hidden_state have exact same dimensions of [batch_size, 512]. Can both be used interchangeably? I mean, can I do this? :
last_state_tuple= last_cell_memory, output
and then feed last_state_tuple in lstm?
Jacques's answer is correct, but it doesn't mention an important point: the state of LSTM layer almost always equals to the output. The difference becomes important when the chain of LSTM cells is long and not all input sequences have equal length (and hence are padded). That's when you should distinguish the state and output.
See the runnable example in my answer on a similar question (it uses BasicRNNCell, but you'll get the same result with LSTMCell).
Yes, the second element of the state is the same as the output.
From https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/LSTMStateTuple
Stores two elements: (c, h), in that order. Where c is the hidden state and h is the output.
Also to verify experimentally:
import tensorflow as tf
from numpy import random as rng
lstm = tf.nn.rnn_cell.LSTMCell(10)
inp = tf.placeholder(tf.float32, shape=(1, 10))
stt = tf.placeholder(tf.float32, shape=(1, 10))
hdd = tf.placeholder(tf.float32, shape=(1, 10))
out = lstm(inp, (stt, hdd))
sess = tf.InteractiveSession()
init = tf.global_variables_initializer()
sess.run(init)
a = rng.randn(1, 10)
b = rng.randn(1, 10)
c = rng.randn(1, 10)
output = sess.run(out, {inp: a, stt: b, hdd: c})
assert (output[0] == output[1][1]).all()
I am trying to execute this numpy code in tensor flow. The reason for this is because I want to make binary predictions in a customized way (not using a softmax) and use that in the loss for my network later. Output1 is what the network outputs, an array of size (1, batch_size). Here is the numpy code:
predictions = []
for j in range(batch_size):
if output1[0, j] >= output2[0] and output1[0, j] <= output2[1]:
predictions.append(1)
else:
predictions.append(0)
In Tensorflow, I have tried to do something like this, using tf.cond since I want to evaluate the value of the output of the network and do something based on that:
predictions = []
for j in range(batch_size):
condResult = tf.cond(output1[0, j] >= output2[0], lambda: predictions.append(1), lambda: predictions.append(0))
condResultFalse = tf.cond(output1[0, j] <= output2[1], lambda: predictions.append(1), lambda: predictions(0))
However, this has some problems. First, if both conditions are true, it will append 1 to the list twice, which I don't want. Second, it throws an error saying ValueError: true_fn must have a return value. Apparently, I must return a tensor, but I'm not sure how to do this since I just want to append to a list.
Any help in translating this to Tensorflow would be great!
Thanks
A good solution would be to use logical functions directly, saying tf.less_equal, or '<=', as follow using broadcasting:
It's gonna be '1' where your condition is True.
import tensorflow as tf
import numpy as np
output1 = tf.constant(np.random.randn(1, 200), dtype='float32')
output2 = tf.constant([0.1, 0.5], dtype='float32')
a = output2[0] <= output1[0]
b = output1[0] <= output2[1]
c = tf.cast(tf.logical_and(a, b), tf.int64)
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
res = sess.run(c)
print res
Edit: Actually working with int64
The way to do this is close to the answer, but that didn't work for me.
predictions = []
for j in range(batch_size):
condition = tf.less_equal(output1[0, j], output2[1])
condition2 = tf.greater_equal(output1[0, j], output2[0])
resultingCondition = tf.where(condition, True, False)
resultingCondition2 = tf.where(condition2, True, False)
finalResultingCondition = tf.cast(tf.logical_not(tf.logical_and(resultingCondition, resultingCondition2)), tf.float32)
predictions.append(finalResultingCondition)
This will append 1 and 0 instead of 1 and -1.
I am having difficulty applying tf.scatter_nd_add() to 2D tensors. The documentation is a bit unclear and has does not contain an example for sparse update but only for full slice updates.
My case is the following:
updates - 2D tensor of shape [None, 6]
indices - 2D tensor of shape [None, 6]
ref - 2D Variable of zeros of shape [None, 6]
It is guaranteed that updates, indices and ref will always have their first dimension equal, but the size of that dimension can be varying. The update I want to perform looks like
for i, j:
k = indices[i][j]
ref[i][k] += updates[i][j]
Note that indices contains duplicates. tf.scatter_nd_add(ref, indices, updates) complains about shape mismatch and I cannot figure out how I need to restructure the tensors in order to performs the update.
I figured it out. Each 2D entry in indices must actually specify the absolute location that will get updated in ref. This means that indices must be 3D and then the non-vectorized update looks like:
for i, j:
r, k = indices[i][j]
ref[r][k] += updates[i][j]
In the above question it just happens that r is always equal to i.
Here is a full Tensorflow implementation with varying shapes. For clarity, in the following example, col_indices corresponds to indices from the original question:
import tensorflow as tf
import numpy as np
updates = tf.placeholder(dtype=tf.float32, shape=[None, 6])
col_indices = tf.placeholder(dtype=tf.int32, shape=[None, 6])
row_indices = tf.cumsum(tf.ones_like(col_indices), axis=0, exclusive=True)
indices = tf.concat([tf.expand_dims(row_indices, axis=-1),
tf.expand_dims(col_indices, axis=-1)], axis=-1)
tmp_var = tf.Variable(0, trainable=False, dtype=tf.float32, validate_shape=False)
ref = tf.assign(tmp_var, tf.zeros_like(updates), validate_shape=False)
# This makes sure that ref is always 0 before scatter_nd_add() runs
with tf.control_dependencies([target_var]):
result = tf.scatter_nd_add(ref, indices, updates)
sess = tf.Session()
sess.run(tf.global_variables_initializer())
# Create example input data
np_input = np.arange(0, 6, 1, dtype=np.int32)
np_input = np.tile(np_input[None,:], [10, 1])
res = sess.run(result, feed_dict={updates: np_input, col_indices: np_input})
print(res)
I am new to Tensorflow. I am trying to write a function in python using Tensorflow that operates on a sparse matrix input. Normally I would define a tensorflow placeholder, but apparently there is no placeholder for sparse matrices.
What is the proper way to define a function that operates on sparse data in tensorflow and pass values into it?
Specifically, I am trying to rewrite the fundamental example of a multilayer perceptron, found here https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/multilayer_perceptron.py, to accept sparse input instead of dense.
As a dummy example, how would you write a function that looks something like this?
import tensorflow as tf
x = tf.placeholder("sparse")
y = tf.placeholder("float", [None, n_classes])
# Create model
def sparse_multiply(x, y):
outlayer = tf.sparse_tensor_dense_matmul(x, y)
return out_layer
pred = multiply(x, y)
# Launch the graph
with tf.Session() as sess:
result = sess.run(pred, feed_dict={x: x_input, y: y_input})
Someone at the link https://github.com/tensorflow/tensorflow/issues/342 recommended, as a workaround, passing in the elements needed to construct the sparse matrix and then creating the sparse matrix on the fly within the function. That seems a little hacky, and I get errors when I try to construct it that way.
Any help, especially answers with code, would be greatly appreciated!
I think I figured it out. The suggestion I linked to actually did work, I just needed to correct all the inputs to have consistent types. Here is the dummy example I listed in the question, coded correctly:
import tensorflow as tf
import sklearn.feature_extraction
import numpy as np
def convert_csr_to_sparse_tensor_inputs(X):
coo = X.tocoo()
indices = np.mat([coo.row, coo.col]).transpose()
return indices, coo.data, coo.shape
X = ____ #Some sparse 2 x 8 csr matrix
y_input = np.asarray([1, 1, 1, 1, 1, 1, 1, 1])
y_input.shape = (8,1)
x_indices, x_values, x_shape = convert_csr_to_sparse_tensor_inputs(X)
# tf Graph input
y = tf.placeholder(tf.float64)
values = tf.placeholder(tf.float64)
indices = tf.placeholder(tf.int64)
shape = tf.placeholder(tf.int64)
# Create model
def multiply(values, indices, shape, y):
x_tensor = tf.SparseTensor(indices, values, shape)
out_layer = tf.sparse_tensor_dense_matmul(x_tensor, y)
return out_layer
pred = multiply(values, indices, shape, y)
# Launch the graph
with tf.Session() as sess:
result = sess.run(pred, feed_dict={values: x_values, indices: x_indices, shape: x_shape, y: y_input})