I am trying to run multiple convolutions on image in TensorFlow and then concatenate results. Because tf.concat allocates new tensor I sometimes run into ResourceExhaustedError (current solution is to change batch_size to smaller value).
So here is my question. Is there a way to create big tensor (I know all dimensions in advance) and then assign results of convolutions to it (part by part to avoid concatenating and memory allocation)? Or maybe there is other more efficient way of doing this?
Something like:
convs = tf.Variable(tf.zeros([..])
tf.update(convs, [..], tf.nn.conv2d(..) + biases1)
tf.update(convs, [..], tf.nn.conv2d(..) + biases2)
^^^^^^^^^ ^^offsets
There isn't a way to do this - TensorFlow objects are immutable by design.
There might be another way to accomplish what you want (and it'd be interesting to know about cases that are running out of memory for future improvements).
Related
I'm trying to parallelize different tensor operations. I'm aware that tf.vectorized_map and/or tf.map_fn can parallelize input tensor(s) with respect to their first axis, but that's not what I'm looking for. I'm looking for a way to parallelize a for loop on a set of tensors with possibly different shapes.
a = tf.ones((2))
b = tf.ones((2,2))
list_of_tensors = [a,b*2,a*3]
for t in list_of_tensors:
# some operation on t which may vary depending on its shape
Is there a possible way to parallelize this for loop on GPU with TensorFlow? (I'm open to any other library if possible i.e. JAX, numba etc.)
Thanks!
According to the documentation,
The shape and dtype of any intermediate or output tensors in the
computation of fn should not depend on the input to fn.
I'm struggling with this problem myself. I think the answer is one suggested in the comments: If you know the maximum length that your tensor can have, represent the variable length tensor by the maximum length tensor plus an integer which gives the actual length of the tensor. Whether this will be useful at all depends on the meaning of "any intermediate", because at some point you may still need the result of the actual shorter length tensor in your computation. It's a bit of a tail-chasing exercise. This part of Tensorflow is extremely frustrating, it's very very hacky to get things to work that should be easy, especially in the realm of obtaining true parallelism on the GPU for deterministic matrix algorithms, outside of the context of machine learning.
This might work inside the loop:
tf.autograph.experimental.set_loop_options(
shape_invariants=[(v, tf.TensorShape([None]))]
)
What kind of calculation does tf.nn.dynamic_rnn perform? How does it use the parameters cell and inputs (to create the result)?
I have looked up in the documentation, but I have not found an explanation.
tf.nn.static_rnn vs. tf.nn.dynamic_rnn
Internally, tf.nn.static_rnn creates an unrolled graph for a fixed RNN length. That means that, if you call tf.nn.static_rnn with inputs having 200 time-steps you are creating a static graph with 200 RNN steps. First, graph creation is slow. Second, you’re unable to pass in longer sequences (> 200) than you've originally specified.
tf.nn.dynamic_rnn solves this. It uses a tf.while_loop to dynamically construct the graph when it is executed. That means graph creation is faster and you can feed batches of variable size.
What about performance?
You may think the tf.nn.static_rnn is faster than its dynamic counterpart because it pre-builds the graph.
Please note, it is strongly encouraged to use tf.nn.dynamic_rnn.
Reference: http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/
You can get a tensor by name with tf.get_default_graph().get_tensor_by_name("tensor_name:0")
But can you get an operation, such as Optimizer.minimize, or an enqueue operation on a queue?
In my first model I returned all tensors and ops I would need from a build_model function. But the list of tensors got ugly. In later models I tossed all tensors and ops in a dictionary for easier access. This time around I thought I'd just look up tensors by name as I needed them, but I don't know how to do that with ops.
Or is there a better way to do this? I find various tensors and ops are needed all over the place. Training, inference code, test cases, hence the desire for a nice standard way of accessing the various parts of the graph without passing variables all over the place.
You can use the tf.Graph.get_operation_by_name() method to get a tf.Operation by name. For example, to get an operation called "enqueue" from the default graph:
op = tf.get_default_graph().get_operation_by_name("enqueue")
Tensorflow has a great deal of transformations that can be applied to 3D-tensors representing images ([height, width, depth]) like tf.image.rot90() or tf.image.random_flip_left_right() for example.
I know that they are meant to be used with queues hence the fact that they operate on only one image.
But would there be a way to vectorize the ops to transform 4D-tensor ([batch_size,height,width,depth]) to same size tensor with op applied image-wise along the first dimension without explicitely looping through them with tf.while_loop()?
(EDIT : Regarding rot90() a clever hack taken from numpy rot90 would be to do:
rot90=tf.reverse(x,tf.convert_to_tensor((False,False,True,False)))
rot90=tf.transpose(rot90,([0,2,1,3])
EDIT 2: It turns out this question has already been answered quite a few times (one example) it seems map_fn is the way to go if you want an optimized version. I had already seen it but I had forgotten. I guess this makes this question a duplicate...
However for random op or more complex op it would be nice to have a generic method to vectorize existing functions...)
Try tf.map_fn.
processed_images = tf.map_fn(process_fn, images)
Is there a way to make a Tensor iterable without running eval() to get its numpy array?
I am trying to iterate through two parts of a tensor after using split() on it, but it happens within the construction of the hidden layers of my neural network, so it needs to happen before I am able to start a session.
import tensorflow as tf
x = tf.placeholder('float', [None, nbits])
layer = [x]
for i in range(1,numbits):
layer.append(tf.add(tf.matmul(weights[i-1], layer[i-1]), biases[i-1]))
aes, bes = tf.split(1, 2, layer[-1])
if i%2 == 1:
for am, a, b in zip(add_layer, aes, bes):
layer.append(am.ex(a, b))
The problem is that layer[-1] is a tf.placeholder at this point, so aes and bes are both tensors, and I can't iterate through them with zip().
Any ideas would be appreciated.
No, there isn't; not directly.
It's easiest to think about Tensorflow programs as being split into two phases: a building Python phase that builds a computation graph, and a execution phase that runs the computation graph. Nothing actually runs during the building phase; all computation happens during the execution phase. The building phase can't depend on the results of the execution phase, except by running the graph (session.run(), .eval(), etc.)
You can't iterate over a Tensor while building the graph, because it doesn't actually get evaluated to a specific set of values until you call session.run(). Instead it's just a reference to a node in the computation graph.
In general, you have to use Tensorflow functions to manipulate Tensors, not Python primitives (like zip). One way I like to think of it is that it's almost like a Tensor is a radioactive object in a sealed box, and you can only handle it indirectly using a robot that can perform a certain set of actions (Tensorflow library functions) :-) So you likely need to find a way to express your task using Tensorflow primitives.
If you gave a complete example of what you're trying to do, it might be possible to say more (it's not clear to me from your code fragment). One possibility might be to use tf.split to split the tensors up into Python lists of subtensors, and then use something like zip on the lists.
I hope that helps!