I'm trying to build a fully convolutional neural network. My problem is that at some phase the shape of the tensors no longer match, causing and Exception, and I would like to print the shape of the tensors after each step to be able to pin point the problem. However the problem is that the tf.Print does not seem to print anything if the graph is broken and an Exception is thrown at some point (even the exception occurs after the print statement in the pipeline). I'm using the below code in printing. It is working OK if I have a working graph. So is it really that the tf.Print can be used only with working graphs? If this is the case how could I print the shape of tensors or is the only possibility to use some debugger for example tfdbg?
upsample = custom_layers.crop_center(input_layer, upsample)
upsample_print = tf.Print(upsample, [tf.shape(upsample)], "shape of tensor is ")
logits = tf.reshape(upsample_print, [-1, 2])
...
The error given is
ValueError: Dimension size must be evenly divisible by 2898844 but is 2005644 for 'gradients/Reshape_grad/Reshape' (op: 'Reshape') with input shapes: [1002822,2], [4] and with input tensors computed as partial shapes: input[1] = [?,1391,1042,2].
Try print(upsample.get_shape()).
tf.Print only prints during runtime. It simply adds a node to the graph that upon execution prints something to the console. So, if your graph cannot be constructed, i.e. no computations can be executed, you will never see an output from tf.Print.
At construction time, you can only see static shapes of your tensors (and e.g. print them with the Python native print statement). I am not aware of any way to get the dynamic shape at construction time (the dynamic shape is dependent on the actual input you feed, so there is no way of knowing that before you actually feed something, which only happens during runtime). Knowing the static shapes was often enough for my purposes. If it is not the case for you, try to make the dynamic dimensions static in a toy example and then Python-print all the shapes to track down the problem.
Related
Is it possible to reuse tensors in multiple tf-graphs, even after they are reset?
Problem:
I have a large dataset that I want to evaluate with many different tf-graphs.
For each evaluation, tensorflow is reset with tf.compat.v1.reset_default_graph() and initialized completely from scratch.
Imho, it seems kind of dull and slow to call the data-to-tensor procedure every time, so I thought I could just define the data-tensor once and use it for all future evaluation.
Unfortunately, reusing tensors does not seem to be possible, as 'Tensor must be from the same graph as Tensor'.
ValueError: Tensor("Const:0", shape=(1670,), dtype=float32, device=/device:GPU:0) must be from the same graph as Tensor("Const_1:0", shape=(1670,), dtype=float32).
Is it possible to reuse these tensors somehow?
Check out this answer in another answered on another questio. https://stackoverflow.com/a/42616834/13514201
TensorFlow stores all operations on an operational graph. This graph defines what functions output to where, and it links it all together so that it can follow the steps you have set up in the graph to produce your final output. If you try to input a Tensor or operation on one graph into a Tensor or operation on another graph it will fail. Everything must be on the same execution graph.
Try removing with tf.Graph().as_default():
Note: I already solved my issue, but I'm posting the question in case others have it too and because I don't understand how I solved it.
I was building a Named Entity Classifier (sequence labelling model) in Keras with Tensorflow backend. When I tried to fit the model, I got this error (which, amazingly, returns only 4 Google results):
"If your data is in the form of symbolic tensors, you should specify the `steps_per_epoch` argument (instead of the batch_size argument, because symbolic tensors are expected to produce batches of input data)."
This stackoverflow post discussed the issue, and someone suggested to the op:
one of your data tensors that is being used by Fit() is a symbolic tensor. The one hot label function returns a symbolic tensor. Try something like:
label_onehot = tf.Session().run(K.one_hot(label, 5))
Then I read on this (not related) site:
The Wolfram System also has powerful algorithms to manipulate algebraic combinations of expressions representing [...] arrays. These expressions are called symbolic arrays or symbolic tensors.
These two sources made me think symbolic arrays (at least in TensorFlow) might be something more like arrays of functions that are yet to be evaluated, rather than actual values.
So, using %whos to view all my variables, I saw that my X and Y data were tensors (rather than arrays, like I normally use for my models). The data/info column had quite a complicated description for them, but I lost it once I solved my issue and I can't work out how to get back to the state where I was getting the error.
In any case, I know I solved the problem by changing my data pre-processing so that the X and y data (i.e. X_train and y_train) were of type <class 'numpy.ndarray'> and of dimensions (num sents, max len) for X_train and (num_sents, max len, 1) for y_train (the 1 is necessary because my final layer expects 3D input). Now the model works fine. But I'm still wondering, what are these symbolic tensors and how/why is using steps per epoch instead of batch size supposed to help? I tried that too initially but had no luck.
This can be solved bu using the eval() or numpy() function of your tensors.
Check:
How can I convert a tensor into a numpy array in TensorFlow?
I am using the tf.nn.dynamic_rnn class to create an LSTM. I have trained this model on some data, and now I want to inspect what are the values of the hidden states of this trained LSTM at each time step when I provide it some input.
After some digging around on SO and on TensorFlow's GitHub page, I saw that some people mentioned that I should write my own LSTM cell that returns whatever I want printed as part of the output of the LSTM. However, this does not seem straight forward to me since the hidden states and the output of the LSTM do not have the same shapes.
My output tensor from the LSTM has shape [16, 1] and the hidden state is a tensor of shape [16, 16]. Concatenating them results in a tensor of shape [16, 17]. When I tried to return it, I get an error saying that some TensorFlow op required a tensor of shape [16,1].
Does anyone know an easier work around to this situation? I was wondering if it is possible to use tf.Print to just print the required tensors.
Okay, so the issue was that I was modifying the output but wasn't updating the output_size of the LSTM itself. Hence the error. It works perfectly fine now. However, I still find this method to be extremely annoying. Not accepting my own answer with the hope that somebody will have a cleaner solution.
I am working on neural networks and many times faced problems with shapes.....Tensorflow provides us a keyword None so that we don't have to worry about the size of the tensor.....
Is there any disadvantage of using None in place of known numeric value for shape.
method 1
input_placeholder = tf.placeholder(tf.float32,[None,None])
method 2
input_placeholder = tf.placeholder(tf.float32,[64,100])
Will it make any difference while running the code ?
Tensorflow's tf.placeholder() tensors do not require a fixed shape to be passed to them. This allows you to pass different shapes in later tf.Session.run() call.
So your code will work just fine.
It doesn't have any disadvantage because when we create a placeholder, Tensorflow doesn't allocate any memory. When you feed the placeholder, in the call to tf.Session.run(), TensorFlow will allocate appropriately sized memory for the input tensors.
If you use these input_placeholder in some operation in your code further, defining them with None i.e. an unconstrained shape, can cause Tensorflow to perform some checks, related to the shape of the tensors, while performing those ops dynamically during the Session.run() call. This is because while building the graph Tensorflow doesn't know about the exact shape of your input.
Suppose I have two placeholder quantities in tensorflow: placeholder_1 and placeholder_2. Essentially I would like the following computational functionality: "if placeholder_1 is defined (ie is given a value in the feed_dict of sess.run()), compute X as f(placeholder_1), otherwise, compute X as g(placeholder_2)." Think of X as being a hidden layer in a neural network that can optionally be computed in these two different ways. Eventually I would use X to produce an output, and I'd like to backpropagate error to the parameters of f or g depending on which placeholder I used.
One could accomplish this using the tf.where(condition, x, y) function if there was a way to make the condition "placeholder_1 has a value", but after looking through the tensorflow documentation on booleans and asserts I couldn't find anything that looked applicable.
Any ideas? I have a vague idea of how I could accomplish this basically by copying part of the network, sharing parameters and syncing the networks after updates, but I'm hoping for a cleaner way to do it.
You can create a third placeholder variable of type boolean to select which branch to use and feed that in at run time.
The logic behind it is that since you are feeding in the placholders at runtime anyways you can determine outside of tensorflow which placeholders will be fed.