I read the following statement when covering Autographs and Tracing in Tensorflow.
TensorFlow will only capture for loops that iterate over a tensor or a
dataset. So make sure you use for i in tf.range(x) rather than for i
in range(x), or else the loop will not be captured in the graph.
Instead, it will run during tracing.
(This may be what you want if the for loop is meant to build the graph, for example to create each layer in a neural network.)
I am confused as to what exactly happens. If it runs during tracing how it not registered on the graph but also how would the for loop build the graph?
An example which shows the difference between a tf.range loop and a range loop:
for i in tf.range(3):
x = tf.add(x, i)
results in a graph which contains a tf.while_loop that matches the for loop; this is the translation that AutoGraph makes:
def cond(i, x):
return tf.lesss(i, 3)
def body(i, x):
x = tf.add(x, i)
return i, x
tf.while_loop(cond, body, ...)
In turn:
for i in range(3):
x = tf.add(x, i)
results in a graph which contains a three tf.add calls, and i is substituted by constants, without any loop ops:
x = tf.add(x, 0)
x = tf.add(x, 1)
x = tf.add(x, 2)
Related
EDIT: Problem soved, just had to "transform" the input W using W.data ...
Hi guys,
In my code, i am trying to train a model so that it moves a given sample to a given target distribution. The next step is to introduce intermediate distributions and to use a loop so that the particles (the samples) are moved from one distribution to another iteratively. Unfortunately, at the second iteration I get the following Error-Message when running my code:
"Trying to backward through the graph a second time (or directly access saved variables after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved variables after calling backward"
I don't think that retain_graph = True would fit my problem, since I would rather kind of clear the model after every iteraion than retain it. However, i gave it a shot, the result is the following error:
one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1, 2]] is at version 2251; expected version 2250 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Here are the relevant parts of my code:
for k in range(1, K_intermediate+1):
flow = BasicFlow(dim=d, n_flows=n_flows, flow_layer=flow_layer)
ldj = train_flow(
flow, x, W, f_intermediate(x,k-1), lambda x:
f_intermediate(x,k), epochs=2500
)
x, xtransp = flow(x)
x = xtransp.data
And the train_flow function:
def train_flow(flow, sample, weights, f0, f1, epochs=1000):
optim = torch.optim.Adam(flow.parameters(), lr=1e-2)
for i in range(epochs):
x0, xtransp = flow(sample)
ldj = accumulate_kl_div(flow).reshape(sample.size(0))
loss = det_loss(
x_0 = x0,
x_transp = xtransp,
weights = weights,
ldj = ldj,
f0 = f0,
f1 = f1
)
loss.backward(retain_graph = True)
optim.step()
optim.zero_grad()
reset_kl_div(flow)
if i % 250 == 0:
if i > 0 and previous_loss - loss.item() < 1e-06:
break
print(loss.item())
previous_loss = loss.item()
if torch.isnan(loss) == True:
break
return ldj
Note that the problem only arises since I start capturing the ldj-value (log of the determinant jacobian, for those who wonder). Since this value is crucial for further computations i can not just delete this.
In some question and tutorials like below:
Why is an iterable object not an iterator?
Generator "TypeError: 'generator' object is not an iterator"
suggest that a data generator for keras should be a class having __iter__ and __next__ methods in it.
While some other tutorials like:
https://keunwoochoi.wordpress.com/2017/08/24/tip-fit_generator-in-keras-how-to-parallelise-correctly/
https://www.altumintelligence.com/articles/a/Time-Series-Prediction-Using-LSTM-Deep-Neural-Networks
use the normal python function with a yield statement providing data. While I successfully used yield in an LSTM network following the second tutorial above, I am not able to use the normal yield function in a convolutional network and getting the below error in fit_generator:
'method' object is not an iterator
I haven't tried using the __next__ method, but whoever got the above error was advised to use the __next__ method (EDIT: working after a fix suggested by Daniel Möller). Can someone please help me clarify which technique to use when and what is the difference between a function that "yields" the next sample vs a class with __iter__ & __next__?
My working code using yield:
https://github.com/KashyapCKotak/Multidimensional-Stock-Price-Prediction/blob/master/StockTF1_4Sequential.ipynb
My current data generator function using yield (EDIT: working after a fix suggested by Daniel Möller):
def train_images_generator(self):
for epoch in range(0, self.epochs):
print("Current Epoch:",epoch)
cnt = 0
if epoch > 2000:
learning_rate = 1e-5
for ind in np.random.permutation(len(self.train_ids)):
print("provided image with id:",ind)
#get the input image and target/ground truth image based on ind
raw = rawpy.imread(in_path)
input_images = np.expand_dims(pack_raw(raw), axis=0) * ratio # pack the bayer image in 4 channels of RGBG
gt_raw = rawpy.imread(gt_path)
im = gt_raw.postprocess(use_camera_wb=True,
half_size=False,
no_auto_bright=True, output_bps=16)
gt_images = np.expand_dims(np.float32(im / 65535.0),axis=0) # divide by 65535 to normalise (scale between 0 and 1)
# crop
H = input_images.shape[1] # get the image height (number of rows)
W = input_images.shape[2] # get the image width (number of columns)
xx = np.random.randint(0, W - ps) # get a random number in W-ps (W-512)
yy = np.random.randint(0, H - ps) # get a random number in H-ps (H-512)
input_patch = input_images[:, yy:yy + ps, xx:xx + ps, :]
gt_patch = gt_images[:, yy * 2:yy * 2 + ps * 2, xx * 2:xx * 2 + ps * 2, :]
if np.random.randint(2) == 1: # random flip for rows
input_patch = np.flip(input_patch, axis=1)
gt_patch = np.flip(gt_patch, axis=1)
if np.random.randint(2) == 1: # random flip for columns
input_patch = np.flip(input_patch, axis=2)
gt_patch = np.flip(gt_patch, axis=2)
if np.random.randint(2) == 1: # random transpose
input_patch = np.transpose(input_patch, (0, 2, 1, 3))
gt_patch = np.transpose(gt_patch, (0, 2, 1, 3))\
input_patch = np.minimum(input_patch, 1.0)
yield (input_patch,gt_patch)
How I use it:
model.fit_generator(
generator=data.train_images_generator(),
steps_per_epoch=steps_per_epoch,
epochs=epochs,
callbacks=callbacks,
max_queue_size=50
#workers=0
)
Looking carefully at the word 'method', I see you are not "calling" your generator (you are not creating it).
You are passing just the function/method.
Suppose you have:
def generator(...):
...
yield x, y
Instead of something like:
model.fit_generator(generator)
You should do something like:
model.fit_generator(generator(...))
Generator or Sequence
What is the difference between using a generator (a function with yield) and a keras.utils.Sequence?
When using a generator, training will follow the exact loop order, and it will not know when to finish. So.
With a generator:
Cannot shuffle batches because it will always follow the order of the loop
Must inform steps_per_epoch because Keras cannot know when the generator has finished (generators for Keras must be infinite)
If using multiprocessing, the system may not handle the batches correctly because it's impossible to know which process will start or finish before the others.
With a Sequence:
You control the length of the generator. Keras knows the number of batches automatically
You control the indexing of the batches, so Keras can shuffle batches.
You can take whatever batch you want how many times you want (you are not forced to take batches in sequence)
Multiprocessing can use the indices to make sure the batches are not going to be mixed in the end.
I created a neural network with Keras, and added a Lambda layer to perform some calculations, but it is showing a poor performance on inferences.
I was able to make the inferences successfully using a batch of one input and added one more loop to handle multiple inputs. Everything works fine, but the performance is somewhat poor. I figured using a larger batch would make things a lot faster. My question is whether I am handling batches correctly (is it really necessary to use another loop?) as I have not found any keras or tensorflow documentation dealing with this topic in more depth.
Below is a code with a structure similar to the one I'm using in the Lambda layer.
def GenericFunc(x, batch=10, channels=64):
y, group = [], []
for i in range(batch):
for j in range(channels):
y.append(backend.sum(x[0, :, :, j]))
group.append(tf.convert_to_tensor(y, dtype=np.float32))
y = []
yy = backend.stack(group, axis=0)
tensor_stack = backend.reshape(yy, [batch,channels])
return tensor_stack
Any suggestions will be welcome!
Never use loops. Tensors are made for tensor operations.
def GenericFunc(x):
y = backend.sum(x, axis=1)
y = backend.sum(y, axis=1)
return y
Probably also works with
def GenericFunc(x):
return backend.sum(x, axis=[1,2])
I have the following simplified code (actually, unrolled LSTM model):
def func(a, b):
with tf.variable_scope('name'):
res = tf.add(a, b)
print(res.name)
return res
func(tf.constant(10), tf.constant(20))
Whenever I run the last line, it seems that it changes the graph. But I don't want the graph changes. Actually my code is different and is a neural network model but it is too huge, so I've added the above code. I want to call the func without changing the graph of model but it changes. I read about variable scope in TensorFlow but it seems that I've not understand it at all.
You should take a look at the source code of tf.nn.dynamic_rnn, specifically _dynamic_rnn_loop function at python/ops/rnn.py - it's solving the same problem. In order not blow up the graph, it's using tf.while_loop to reuse the same graph ops for new data. But this approach adds several restrictions, namely the shape of tensors that are passing through in a loop must be invariant. See the examples in tf.while_loop documentation:
i0 = tf.constant(0)
m0 = tf.ones([2, 2])
c = lambda i, m: i < 10
b = lambda i, m: [i+1, tf.concat([m, m], axis=0)]
tf.while_loop(
c, b, loop_vars=[i0, m0],
shape_invariants=[i0.get_shape(), tf.TensorShape([None, 2])])
I'm trying to update a two dimensional tensor in a nested while_loop(). When passing the variable to the second loop however, I cannot updated it using tf.assign() as it throws this error:
ValueError: Sliced assignment is only supported for variables
Somehow it works fine if I create the variable outside the while_loop and use it only in the first loop.
How can I modify my 2D tf variable in the second while loop?
(I'm using python 2.7 and TensorFlow 1.2)
My code:
import tensorflow as tf
import numpy as np
tf.reset_default_graph()
BATCH_SIZE = 10
LENGTH_MAX_OUTPUT = 31
it_batch_nr = tf.constant(0)
it_row_nr = tf.Variable(0, dtype=tf.int32)
it_col_nr = tf.constant(0)
cost = tf.constant(0)
it_batch_end = lambda it_batch_nr, cost: tf.less(it_batch_nr, BATCH_SIZE)
it_row_end = lambda it_row_nr, cost_matrix: tf.less(it_row_nr, LENGTH_MAX_OUTPUT+1)
def iterate_batch(it_batch_nr, cost):
cost_matrix = tf.Variable(np.ones((LENGTH_MAX_OUTPUT+1, LENGTH_MAX_OUTPUT+1)), dtype=tf.float32)
it_rows, cost_matrix = tf.while_loop(it_row_end, iterate_row, [it_row_nr, cost_matrix])
cost = cost_matrix[0,0] # IS 1.0, SHOULD BE 100.0
return tf.add(it_batch_nr,1), cost
def iterate_row(it_row_nr, cost_matrix):
# THIS THROWS AN ERROR:
cost_matrix[0,0].assign(100.0)
return tf.add(it_row_nr,1), cost_matrix
it_batch = tf.while_loop(it_batch_end, iterate_batch, [it_batch_nr, cost])
sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())
out = sess.run(it_batch)
print(out)
tf.Variable objects cannot be used as loop variables in a while loop, as loop variables are implemented differently.
So either create your variable outside the loop and update it yourself with tf.assign in each iteration or manually keep track of the updates as you do with loop variables (by returning their updated values from the loop lambdas, and in your case using the value from the inner loop as the new value for the outer loop).
Got this to work, with #AlexandrePassos help, by placing the Variable outside the while_loop. However, I also had to force the execution of the commands using tf.control_dependencies() (as the operations are not directly used on the loop variable). The loop now looks like this:
cost_matrix = tf.Variable(np.ones((LENGTH_MAX_OUTPUT+1, LENGTH_MAX_OUTPUT+1)), dtype=tf.float32)
def iterate_batch(it_batch_nr, cost):
it_rows = tf.while_loop(it_row_end, iterate_row, [it_row_nr])
with tf.control_dependencies([it_rows]):
cost = cost_matrix[0,0]
return tf.add(it_batch_nr,1), cost
def iterate_row(it_row_nr):
a = tf.assign(cost_matrix[0,0], 100.0)
with tf.control_dependencies([a]):
return tf.add(it_row_nr,1)