We Keep Coding

TensorFlow RuntimeError: "Attempting to capture an EagerTensor without building a function"

I am trying to build a neural network in Python for solving PDEs, and, as such, I have had to write custom training steps. My training function looks like this:
...
tf.enable_eager_execution()
class PDENet:
...
def train_step():
input = self.input
with tf.GradientTape() as tape, tf.Session() as sess:
tape.watch(input)
output = self.model(input)
self.loss = self.pde_loss(output) # (network does not use training data)
grad = tape.gradient(self.loss, self.model.trainable_weights)
self.optimizer.apply_gradients([(grad, self.model)])
...
Due to my hardware, I have no choice but to use tensorflow==1.12.0 and keras==2.2.4.
When I run this code, I get "RuntimeError: Attempting to capture an EagerTensor without building a function". I have seen other posts about this, but all of the answers say to update tensorflow/keras, which I can't, use "tf.enable_eager_execution()", which I've already done, and "tf.disable_v2_behavior()", which is nonexistent on older versions of tensorflow. Is there anything else I can do to solve this problem? The error makes me think tensorflow wants me to add #tf.function, but that feature also doesn't seem to exist in tensorflow 1.

Keras + Tensorflow : Debug NaNs

Here is a great question on how to find the first occurence of Nan in a tensorflow graph:
Debugging nans in the backward pass
The answer is quite helpful, here is the code from it:
train_op = ...
check_op = tf.add_check_numerics_ops()
sess = tf.Session()
sess.run([train_op, check_op]) # Runs training and checks for NaNs
Apparently, running the training and the numerical check at the same time will result in an error report as soon as Nan is encountered for the first time.
How do I integrate this into Keras ?
In the documentation, I can't find anything that looks like this.
I checked the code, too.
The update step is executed here:
https://github.com/fchollet/keras/blob/master/keras/engine/training.py
There is a function called _make_train_function where an operation to compute the loss and apply updates is created. This is later called to train the network.
I could change the code like this (always assuming that we're running on a tf backend):
check_op = tf.add_check_numerics_ops()
self.train_function = K.function(inputs,
[self.total_loss] + self.metrics_tensors + [check_op],
updates=updates, name='train_function', **self._function_kwargs)
I'm currently trying to set this up properly and not sure whether the code above actually works.
Maybe there is an easier way ?

I've been running into the exact same problem, and found an alternative to the check_add_numerics_ops() function. Instead of going that route, I use the TensorFlow Debugger to walk through my model, following the example in https://www.tensorflow.org/guide/debugger to figure out exactly where my code produces nans. This snippet should work for replacing the TensorFlow Session that Keras is using with a debugging session, allowing you to use tfdbg.
from tensorflow.python import debug as tf_debug
sess = K.get_session()
sess = tf_debug.LocalCLIDebugWrapperSession(sess)
K.set_session(sess)

How to use masked inputs in Keras?

I am beginner of Machine Learning and Keras. I have a reference code of
Keras. But its style is old and not supported in latest keras.
So I want to reproduce this reference code. But I am not sure how to use a
mask.
Old style code uses 'inputs' and 'merge_mode', so how do I realize this in latest keras?
keras 1.2.2
Python 3.5.2 (Anaconda 4.1.1, 64bit)
updated code
# old style
g = Graph()
# prepare inputs
g.add_node(LSTM(...))
g.add_node(
TimeDistributedDense(out_size, activation='softmax'),
name='h2',
input='h1'
)
# h2.shape and filter.shape are same. (N, )
g.add_node(
Activation(activation=normalize,),
name='action',
inputs=['h2','filter'],
merge_mode='mul',
create_output=True
)
g.compile(...)
# new style
m = Sequential()
m.add(LSTM(...))
m.add(TimeDistributred(Dense(out_size, activation='softmax')))
# how to integrate 'mul merge' and normalization?
m.add_node(BatchNormalization()) # ?
m.compile(...)

where do I find bidirectional_rnn in tensorflow 1.0.0?

I am using some code from here: https://github.com/monikkinom/ner-lstm with tensorflow. I think the code was written for an older version of tensorflow, I am using version 1.0.0. I used tf_upgrade.py to upgrade model.py in that github repos, but I am still getting the error:
output, _, _ = contrib_rnn.bidirectional_rnn(fw_cell, bw_cell,
AttributeError: 'module' object has no attribute 'bidirectional_rnn'
this is after I changed the bidirectional_rnn call to use contrib_rnn which is:
from tensorflow.contrib.rnn.python.ops import core_rnn as contrib_rnn
The old call was
output, _, _ = tf.nn.bidirectional_rnn(fw_cell, bw_cell,
tf.unpack(tf.transpose(self.input_data, perm=[1, 0, 2])),
dtype=tf.float32, sequence_length=self.length)
which also doesn't work.
I had to change the LSTMCell, DroputWrapper, etc. to rnn.LSTMCell, but they seem to work fine. It is the bidirectional_rnn that I can't figure out how to change.

In TensorFlow 1.0, you have the choice of two bidirectional RNN functions:
tf.nn.bidirectional_dynamic_rnn()
tf.contrib.rnn.static_bidirectional_rnn()

Maybe you can try to reimplement a bidirectional RNN by simply wrapping into a single class two monodirectional RNNs with the parameter "go_backwards=True" set on one of them. Then you can also have control over the type of merge done with the outputs. Maybe taking a look at the implementation in https://github.com/fchollet/keras/blob/master/keras/layers/wrappers.py (see the class Bidirectional) could get you started.

Tensorflow - casting from int to float strange behavior

I am working on tensorflow 0.12 and am having problem with casting.
The following snippet of code does a strange thing:
sess = tf.InteractiveSession()
a = tf.constant(1)
b = tf.cast(a, tf.float32)
print b.eval()
I get a value:
6.86574233e-36
I also tried using tf.to_float() and tf.saturate_cast. Both gave the same result.
Please help.

sess = tf.InteractiveSession()
a = tf.constant(1, tf.int64) <--------
b = tf.cast(a, tf.float32)
print b.eval() # 1.0
You need to declare the dtype for your tf.constant: https://www.tensorflow.org/api_docs/python/tf/constant

Since I see that this is still getting some attention, I should mention that the newer versions of tensorflow do not show this behavior, I suggest working with tensorflow version 1.13 or higher

I checked the code in python3 and python2 for the same tensorflow version as well the code seems to be working correctly as in both the cases I got the following output for python2
print b.eval()
1.0
I would suggest checking the tensorflow installation or the virtualenv.

No error in your program.
import tensorflow as tf
sess = tf.InteractiveSession()
a = tf.constant(1)
b = tf.cast(a, tf.float32)
print b.eval()
This is an online environment for TF https://codeenv.com/env/run/gXGpnR/
Test your code there to run, use
click on test_tf.py
add your code
in left side CLI, type ipython test_tf.py