You can get a tensor by name with tf.get_default_graph().get_tensor_by_name("tensor_name:0")
But can you get an operation, such as Optimizer.minimize, or an enqueue operation on a queue?
In my first model I returned all tensors and ops I would need from a build_model function. But the list of tensors got ugly. In later models I tossed all tensors and ops in a dictionary for easier access. This time around I thought I'd just look up tensors by name as I needed them, but I don't know how to do that with ops.
Or is there a better way to do this? I find various tensors and ops are needed all over the place. Training, inference code, test cases, hence the desire for a nice standard way of accessing the various parts of the graph without passing variables all over the place.
You can use the tf.Graph.get_operation_by_name() method to get a tf.Operation by name. For example, to get an operation called "enqueue" from the default graph:
op = tf.get_default_graph().get_operation_by_name("enqueue")
Related
I have a tf.data.TFRecordDataset and a (computationally expensive) function, which I want to map to it. I use TensorFlow 1.12 and eager execution, and the function uses NumPy ndarray interpretations of the tensors in my dataset using EagerTensor.numpy(). However, code inside functions that are given to tf.Dataset.map() are not executed eagerly, which is why the .numpy() conversion doesn't work there and .map() is not an option anymore. Is it possible to for-loop through a dataset and modify the examples in it? Simply assigning to them doesn't seem to work.
No, not exactly.
A Dataset is inherently lazily evaluated and cannot be assigned to in that way - conceptually try to think of it as a pipeline rather than a variable: each value is read, passed through any map() operations, batch() ops, etc and surfaced to the model as needed. To "assign" a value would be to write it to disk in the .tfrecord file and just isn't likely to ever be supported (these files are specifically designed to be fast-read not random-accessed).
You could, instead, use TensorFlow to do your pre-processing and use TfRecordWriter to write to a NEW tfrecord with the expensive pre-processing completed then use this new dataset as the input to your model. If you have the disk space avilable this might well be your best option.
Let's say we have some method foo we call during graph construction time that returns some tf.Tensors or a nested structure of them every time is called, and multiple other methods that make use of foo's result. For efficiency and to avoid spamming the TF graph with unnecessary repeated operations, it might be tempting to make foo cache its result (to reuse the subgraph it produces) the first time is called. However, that will fail if foo is ever used in the context of a control flow, like tf.cond, tf.map_fn or tf.while_loop.
My questions are:
When is it safe to cache tf.Tensor objects in such a way that does not cause problems with control flows? Perhaps is there some way to retrieve the control flow under which a tf.Tensor was created (if any), store it and compare it later to see if a cached result can be reused?
How would the answer to the question above apply to tf.Operations?
(Question text updated to make clearer that foo creates a new set of tensors every time is called)
TL;DR: TF already caches what it needs to, don't bother with it yourself.
Every time you call sess.run([some_tensors]) TF's engine find the minimum subgraph needed to compute all tensors in [some_tensors] and runs it from top to bottom (possibly on new data, if you're not feeding it the same data).
That means, caching of results in-between sess.run calls is useless towards saving computation, because they will be recomputed anyway.
If, instead, you're concerned with having multiple tensors using the same data as input in one call of sess.run, don't worry, TF is smart enough. if you have input A and B = 2*A, C = A + 1, as long as you do one sess.run call as sess.run([B,C]) A will be evaluated only once (and then implicitly cached by the TF engine).
What kind of calculation does tf.nn.dynamic_rnn perform? How does it use the parameters cell and inputs (to create the result)?
I have looked up in the documentation, but I have not found an explanation.
tf.nn.static_rnn vs. tf.nn.dynamic_rnn
Internally, tf.nn.static_rnn creates an unrolled graph for a fixed RNN length. That means that, if you call tf.nn.static_rnn with inputs having 200 time-steps you are creating a static graph with 200 RNN steps. First, graph creation is slow. Second, you’re unable to pass in longer sequences (> 200) than you've originally specified.
tf.nn.dynamic_rnn solves this. It uses a tf.while_loop to dynamically construct the graph when it is executed. That means graph creation is faster and you can feed batches of variable size.
What about performance?
You may think the tf.nn.static_rnn is faster than its dynamic counterpart because it pre-builds the graph.
Please note, it is strongly encouraged to use tf.nn.dynamic_rnn.
Reference: http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/
I have a model where I need to assign to the weights (trainable variables) new external values every N iterations.
I can think of a few solutions:
Save and restore
Not good as I would need to serialization, go through a file system calls, etc. (even if I use something like tmpfs)
Using placeholders and assign operations
I would create a placeholder and assign op for each trainable variable. Everytime I want to assign something to the weights, I ran the assign ops.
However, I understand that this means I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph.
In addition I would be using much more memory than necessary..
Use a feed_dict for trainable variable and trigger ops that assign each variable to itself?
Does this work? Is there any drawback?
Before coding something I thought it was a good idea to ask?
What is the recommended way to assign new external values to variables efficiently (memory/timewise)?
Your 3-rd option sounds like the best one.
You can feed values to tensors that aren’t placeholders.
TensorFlow's feed mechanism lets you inject data into any Tensor in a
computation graph. A python computation can thus feed data directly
into the graph.
Any tensors that are feedable can be fed. To check if a tensor is feedable or not, use: tf.Graph.is_feedable(tensor).
In recent versions of Tensorflow Variable class has load method. It does exactly what you want.
https://www.tensorflow.org/api_docs/python/tf/Variable#load
You can use the assign operations with placeholders.
I will be forced to consider these placeholders in every feed_dict and pass dummy values everytime I run any operation in my graph
In addition I would be using much more memory than necessary..
No. You would only need to feed values to the placeholders when you run the assign operations. Don't make the assign operation part of your training graph and only run them when you want to assign new values.
If the assigning turns out to be a bottleneck (for small N it might slow down your program) you can consider other methods of getting data into TensorFlow.
Is there a way to make a Tensor iterable without running eval() to get its numpy array?
I am trying to iterate through two parts of a tensor after using split() on it, but it happens within the construction of the hidden layers of my neural network, so it needs to happen before I am able to start a session.
import tensorflow as tf
x = tf.placeholder('float', [None, nbits])
layer = [x]
for i in range(1,numbits):
layer.append(tf.add(tf.matmul(weights[i-1], layer[i-1]), biases[i-1]))
aes, bes = tf.split(1, 2, layer[-1])
if i%2 == 1:
for am, a, b in zip(add_layer, aes, bes):
layer.append(am.ex(a, b))
The problem is that layer[-1] is a tf.placeholder at this point, so aes and bes are both tensors, and I can't iterate through them with zip().
Any ideas would be appreciated.
No, there isn't; not directly.
It's easiest to think about Tensorflow programs as being split into two phases: a building Python phase that builds a computation graph, and a execution phase that runs the computation graph. Nothing actually runs during the building phase; all computation happens during the execution phase. The building phase can't depend on the results of the execution phase, except by running the graph (session.run(), .eval(), etc.)
You can't iterate over a Tensor while building the graph, because it doesn't actually get evaluated to a specific set of values until you call session.run(). Instead it's just a reference to a node in the computation graph.
In general, you have to use Tensorflow functions to manipulate Tensors, not Python primitives (like zip). One way I like to think of it is that it's almost like a Tensor is a radioactive object in a sealed box, and you can only handle it indirectly using a robot that can perform a certain set of actions (Tensorflow library functions) :-) So you likely need to find a way to express your task using Tensorflow primitives.
If you gave a complete example of what you're trying to do, it might be possible to say more (it's not clear to me from your code fragment). One possibility might be to use tf.split to split the tensors up into Python lists of subtensors, and then use something like zip on the lists.
I hope that helps!