I am trying to implement dropout in TensorFlow for a simple 3-layer neural network for classification and am running into problems. More specifically, I am trying to apply different dropout parameter pkeep values when I train vs. test.
I am taking the approach as follows:
1) def create_placeholders(n_x, n_y):
X = tf.placeholder("float", [n_x, None])
Y = tf.placeholder("float", [n_y, None])
pkeep = tf.placeholder(tf.float32)
return X,Y,pkeep
2) In the function forward_propagation(X, parameters, pkeep), I perform the following:
Z1 = tf.add(tf.matmul(W1, X), b1)
A1 = tf.nn.relu(Z1)
A1d = tf.nn.dropout(A1, pkeep)
Z2 = tf.add(tf.matmul(W2, A1d),b2)
A2 = tf.nn.relu(Z2)
A2d = tf.nn.dropout(A2, pkeep)
Z3 = tf.add(tf.matmul(W3, A2d),b3)
return Z3
3) Later when tensorflow session is called (in between code lines omitted for clarity):
X, Y, pkeep = create_placeholders(n_x, n_y)
Z3 = forward_propagation(X, parameters, pkeep)
sess.run([optimizer,cost], feed_dict={X:minibatch_X, Y:minibatch_Y, pkeep: 0.75})
Above would run without giving any errors. However, I think that the above would set pkeep value to 0.75 for both training and testing runs. Minibatching is done only on the train data set but I am not setting the pkeep value anywhere else.
I would like to set pkeep = 0.75 for training and pkeep = 1.0 for testing.
4) It does give an error when I do something like this:
x_train_eval = Z3.eval(feed_dict={X: X_train, Y: Y_train, pkeep: 0.75})
x_test_eval = Z3.eval(feed_dict={X: X_test, Y: Y_test, pkeep: 1.0})
The error message I am getting is:
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder_2' with dtype float
[[Node: Placeholder_2 = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
What is the best way to pass different pkeep values for training and testing? Your advice would be much appreciated.
Assuming you have some operation for testing defined as test_op (for instance, something that evaluates accuracy on input data and labels), you can do something like this:
for i in range(num_iters):
# Run your training process.
_, loss = sess.run([optimizer, cost],
feed_dict={X:minibatch_X, Y:minibatch_Y, pkeep: 0.75})
# Test the model after training
test_accuracy = sess.run(test_op,
feed_dict={X:test_X, Y:test_Y, pkeep: 1.0})
Basically you aren't testing the model when you are training it, so you can just call your test_op when you are ready to test, feeding it different hyperparameters like pkeep. The same goes for validating the model periodically during training on a held-out dataset: every so often run your evaluation op on the held-out dataset, passing different hyperparameters than those used during training, and then you can save off configurations or stop early based on your validation accuracy.
Your test_op could return something like the prediction accuracy for a classifier:
correct = tf.equal(tf.argmax(y, 1), tf.argmax(predict, 1))
test_op = tf.reduce_mean(tf.cast(correct, tf.float32))
where y are your target labels and predict is the name of your prediction op.
In neural networks, you forward propagate to compute the logits (probabilities of each label) and compare it with the real value to get the error.
Training involves minimizing the error, by propagating backwards i.e. finding the derivative of error with respect to each weight and then subtracting this value from the weight value.
Applying different dropout parameters is easier when you separate the operations needed for training and evaluating your model.
Training is just minimizing the loss:
def loss(logits, labels):
'''
Calculate cross entropy loss
'''
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
labels=labels, logits=logits)
return tf.reduce_mean(cross_entropy, name='loss_op')
def train(loss, learning_rate):
'''
Train model by optimizing gradient descent
'''
optimizer = tf.train.AdamOptimizer(learning_rate)
train_op = optimizer.minimize(loss, name='train_op')
return train_op
Evaluating is just calculating the accuracy:
def accuracy(logits, labels):
'''
Calculate accuracy of logits at predicting labels
'''
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32),
name='accuracy_op')
return accuracy_op
Then in your Tensorflow session, after generating placeholders and adding necessary operations to the Graph etc. just feed different keep_prob_pl values into the run methods:
# Train model
sess.run(train_op, feed_dict={x_pl: x_train, y_train: y, keep_prob_pl: 0.75}})
# Evaluate test data
batch_size = 100
epoch = data.num_examples // batch_size
acc = 0.0
for i in range(epoch):
batch_x, batch_y = data.next_batch(batch_size)
feed_dict = {x_pl: batch_x, y_pl: batch_y, keep_prob_pl: 1}
acc += sess.run(accuracy_op, feed_dict=feed_dict)
print(("Epoch Accuracy: {:.4f}").format(acc / epoch))
Related
I want to add the test set verification function to my original code while training the model. I called the method of calculating the precision during training and tried to input the test set into the model. But after I run the code that I wrote according to my own understanding, the running result shows that the validation accuracy is larger than the accuracy of the training set. I guess it might be that I mistakenly put the test set into the model for training in one part. This is the result of existing code:
epochs: 0
train acc: 0.09489036 validation acc： 0.125
epochs: 1
train acc: 0.14082506 validation acc: 0.140625
I tried a lot of methods, such as creating two placeholders to hold the test dataset and the tags. But I feel that there is not much chance to solve the problem.
self.sess = tf.Session()
self.x = tf.placeholder(tf.float32, shape=[None, self.img_size,
self.img_size, 3], name="image")
self.y_ = tf.placeholder(tf.float32, shape=[None, 100], name="classes")
self._build_net(self.x)
cross_entropy = tf.losses.softmax_cross_entropy(self.y_, self.predict,
reduction=tf.losses.Reduction.MEAN)
l2 = tf.add_n([tf.nn.l2_loss(var) for var in tf.trainable_variables()])
self.loss = cross_entropy + l2 * 0.0001
self.pred = tf.argmax(self.predict, axis=1)
self.accuracy = tf.reduce_mean(tf.cast(tf.equal(self.pred, tf.argmax(self.y_,
axis=1)), tf.float32))
self.train_op = tf.train.MomentumOptimizer(self.learning_rate,
momentum=.9).minimize(self.loss)
#problem is here (same with self.accuracy)
self.val = tf.reduce_mean(tf.cast(tf.equal(self.pred, tf.argmax(self.y_,
axis=1)), tf.float32))
define _build_net(self, batch_images):
...
CNN model
...
def learn(self, x_batch, y_batch):
_, loss, accuracy = self.sess.run([self.train_op, self.loss,
self.accuracy],feed_dict={self.x:
x_batch, self.y_: y_batch})
return loss, accuracy
#problem is here
def validation(self, x_test_batch, y_test_batch):
validation = self.sess.run(self.val, feed_dict={self.x: x_test_batch,
self.y_: y_test_batch})
return validation
I want to know why I was wrong and expect to get the normal test set output. Or any suggestions that can be verified in my code to validate the test set during the training process.
Thank you very much for any help
I re-added the epoch training and found that the validation accuracy did not increase after more than 45 epoch. Then the output becomes correct. But the reason why the test set is better at the beginning is still unclear.
Actually, you can verify loss of train and test easier if you use a tensorflow keras library. For example: model.fit_generator(train_data, epochs=nb_epochs, steps_per_epoch=nb_train_steps, validation_data=(valid_data, valid_label) where model are created using tf.keras.models
Since you are building a model from scratch, you should use the same intuition from model.fit_generator parameters: "nb_epoch", "train_steps", "batch_size"
In order to verify the test set while training the model, for example, you define:
x_train, y_train, x_test, y_test = tf.placeholder()
batch = 100
iteration = 100000
no_epoch = int(iteration/(len(x_train)/batch))
train_loader = zip(x_train, label)
count = 0 # count to number of epoch
for epoch in range(no_epoch):
for (x, labels) in train_loader:
### your model ###
y_pred = model(x)
loss = error(y_pred, labels)
loss.backward()
optimizer.step()
count += 1
if count % 50 == 0: # for ex train for 50 epoch:
### validation here ###
### you can store training history loss after 400 , 500 epochs etc ###
if __name__ == '__main__':
### load data, prepare x_train_batch, y_train_batch, x_test_batch, y_test_batch###
with tf.Session() as sess:
sess.run(feed_dict={x_train: x_train_batch, y_train: y_train_batch, x_test: x_test_batch, y_test: y_test_batch})
I use pseudo-code for the intuition of using iteration, batch, no_epoch to verify/validate test/valid set while training so don't try to run it. Actually, that was my code implementation from Pytorch, but the same intuition holds for Tensorflow.
About the difference of train accuracy and validation accuracy, there is possibility that the weight initialization at first when you haven't trained enough is better to predict test set than train set. After you train long enough, if the problem still occur, it may be underfitting (bias, variance error)
Hope this helps!
It's really easy to debug and track variable and object values as you run your Python code but in tensorflow, it's really hard to see what's going on behind the scene. I know tensorflow works in graphs and you have to run the session. Is there any simpler way to see the values as you interpret the code? I have attached below screenshot where you can track every variable value but, in tensorflow I am unable to do that. I have tired a lot I have used tf.print() and tf.eval() in session.
Here is the code of tensorflow and I want to see values of Z3 and predict_op
def model(X_train, Y_train, X_test, Y_test, learning_rate=0.01,
ops.reset_default_graph() # to be able to rerun the model without overwriting tf variables
tf.set_random_seed(1) # to keep results consistent (tensorflow seed)
seed = 3 # to keep results consistent (numpy seed)
(m, n_H0, n_W0, n_C0) = X_train.shape
n_y = Y_train.shape[1]
costs = [] # To keep track of the cost
# Create Placeholders of the correct shape
X, Y = create_placeholders(n_H0, n_W0, n_C0, n_y)
# Initialize parameters
parameters = initialize_parameters()
# Forward propagation: Build the forward propagation in the tensorflow graph
Z3 = forward_propagation(X, parameters)
# Cost function: Add cost function to tensorflow graph
cost = compute_cost(Z3, Y)
# Backpropagation: Define the tensorflow optimizer. Use an AdamOptimizer that minimizes the cost.
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
# Initialize all the variables globally
init = tf.global_variables_initializer()
# Start the session to compute the tensorflow graph
with tf.Session() as sess:
# Run the initialization
sess.run(init)
# Do the training loop
for epoch in range(num_epochs):
minibatch_cost = 0.
num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set
seed = seed + 1
minibatches = random_mini_batches(X_train, Y_train, minibatch_size, seed)
for minibatch in minibatches:
# Select a minibatch
(minibatch_X, minibatch_Y) = minibatch
# IMPORTANT: The line that runs the graph on a minibatch.
# Run the session to execute the optimizer and the cost, the feedict should contain a minibatch for (X,Y).
_, temp_cost = sess.run([optimizer, cost], feed_dict={X: minibatch_X, Y: minibatch_Y})
minibatch_cost += temp_cost / num_minibatches
# Print the cost every epoch
if print_cost == True and epoch % 5 == 0:
print("Cost after epoch %i: %f" % (epoch, minibatch_cost))
if print_cost == True and epoch % 1 == 0:
costs.append(minibatch_cost)
# Calculate the correct predictions
predict_op = tf.argmax(Z3, 1)
correct_prediction = tf.equal(predict_op, tf.argmax(Y, 1))
# Calculate accuracy on the test set
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
print(accuracy)
train_accuracy = accuracy.eval({X: X_train, Y: Y_train})
test_accuracy = accuracy.eval({X: X_test, Y: Y_test})
print("Train Accuracy:", train_accuracy)
print("Test Accuracy:", test_accuracy)
return train_accuracy, test_accuracy, parameters
You can try TensorFlow Eager Execution that allows you to run TensorFlow code directly without having to build a graph and run it during a session. It even says in the description that it enables easier debugging.
https://www.tensorflow.org/guide/eager
In the following Keras and Tensorflow implementations of the training of a neural network, how model.train_on_batch([x], [y]) in the keras implementation is different than sess.run([train_optimizer, cross_entropy, accuracy_op], feed_dict=feed_dict) in the Tensorflow implementation? In particular: how those two lines can lead to different computation in training?:
keras_version.py
input_x = Input(shape=input_shape, name="x")
c = Dense(num_classes, activation="softmax")(input_x)
model = Model([input_x], [c])
opt = Adam(lr)
model.compile(loss=['categorical_crossentropy'], optimizer=opt)
nb_batchs = int(len(x_train)/batch_size)
for epoch in range(epochs):
loss = 0.0
for batch in range(nb_batchs):
x = x_train[batch*batch_size:(batch+1)*batch_size]
y = y_train[batch*batch_size:(batch+1)*batch_size]
loss_batch, acc_batch = model.train_on_batch([x], [y])
loss += loss_batch
print(epoch, loss / nb_batchs)
tensorflow_version.py
input_x = Input(shape=input_shape, name="x")
c = Dense(num_classes)(input_x)
input_y = tf.placeholder(tf.float32, shape=[None, num_classes], name="label")
cross_entropy = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits_v2(labels=input_y, logits=c, name="xentropy"),
name="xentropy_mean"
)
train_optimizer = tf.train.AdamOptimizer(learning_rate=lr).minimize(cross_entropy)
nb_batchs = int(len(x_train)/batch_size)
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for epoch in range(epochs):
loss = 0.0
acc = 0.0
for batch in range(nb_batchs):
x = x_train[batch*batch_size:(batch+1)*batch_size]
y = y_train[batch*batch_size:(batch+1)*batch_size]
feed_dict = {input_x: x,
input_y: y}
_, loss_batch = sess.run([train_optimizer, cross_entropy], feed_dict=feed_dict)
loss += loss_batch
print(epoch, loss / nb_batchs)
Note: This question follows Same (?) model converges in Keras but not in Tensorflow , which have been considered too broad but in which I show exactly why I think those two statements are somehow different and lead to different computation.
Yes, the results can be different. The results shouldn't be surprising if you know the following things in advance:
Implementation of corss-entropy in Tensorflow and Keras is different. Tensorflow assumes the input to tf.nn.softmax_cross_entropy_with_logits_v2 as the raw unnormalized logits while Keras accepts inputs as probabilities
Implementation of optimizers in Keras and Tensorflow are different.
It might be the case that you are shuffling the data and the batches passed aren't in the same order. Although it doesn't matter if you run the model for long but initial few epochs can be entirely different. Make sure same batch is passed to both and then compare the results.
I am trying to write a two layer neural network to train a class labeler. The input to the network is a 150-feature list of about 1000 examples; all features on all examples have been L2 normalized.
I only have two outputs, and they should be disjoint--I am just attempting to predict whether the example is a one or a zero.
My code is relatively simple; I am feeding the input data into the hidden layer, and then the hidden layer into the output. As I really just want to see this working in action, I am training on the entire data set with each step.
My code is below. Based on the other NN implementations I have referred to, I believe that the performance of this network should be improving over time. However, regardless of the number of epochs I set, I am getting back an accuracy of about ~20%. The accuracy is not changing when the number of steps are changed, so I don't believe that my weights and biases are being updated.
Is there something obvious I am missing with my model? Thanks!
import numpy as np
import tensorflow as tf
sess = tf.InteractiveSession()
# generate data
np.random.seed(10)
inputs = np.random.normal(size=[1000,150]).astype('float32')*1.5
label = np.round(np.random.uniform(low=0,high=1,size=[1000,1])*0.8)
reverse_label = 1-label
labels = np.append(label,reverse_label,1)
# parameters
learn_rate = 0.01
epochs = 200
n_input = 150
n_hidden = 75
n_output = 2
# set weights/biases
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_output])
b0 = tf.Variable(tf.truncated_normal([n_hidden]))
b1 = tf.Variable(tf.truncated_normal([n_output]))
w0 = tf.Variable(tf.truncated_normal([n_input,n_hidden]))
w1 = tf.Variable(tf.truncated_normal([n_hidden,n_output]))
# step function
def returnPred(x,w0,w1,b0,b1):
z1 = tf.add(tf.matmul(x, w0), b0)
a2 = tf.nn.relu(z1)
z2 = tf.add(tf.matmul(a2, w1), b1)
h = tf.nn.relu(z2)
return h #return the first response vector from the
y_ = returnPred(x,w0,w1,b0,b1) # predict operation
loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=y_,labels=y) # calculate loss between prediction and actual
model = tf.train.GradientDescentOptimizer(learning_rate=learn_rate).minimize(loss) # apply gradient descent based on loss
init = tf.global_variables_initializer()
tf.Session = sess
sess.run(init) #initialize graph
for step in range(0,epochs):
sess.run(model,feed_dict={x: inputs, y: labels }) #train model
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(sess.run(accuracy, feed_dict={x: inputs, y: labels})) # print accuracy
I changed your optimizer to AdamOptimizer (in many cases it performs better than GradientDescentOptimizer).
I also played a bit with the parameters. In particular, I took smaller std for your variable initialization, decreased learning rate (as your loss was unstable and "jumped around") and increased epochs (as I noticed that your loss continues to decrease).
I also reduced the size of the hidden layer. It is harder to train networks with large hidden layer when you don't have that much data.
Regarding your loss, it is better to apply tf.reduce_mean on it so that loss would be a number. In addition, following the answer of ml4294, I used softmax instead of sigmoid, so the loss looks like:
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_,labels=y))
The code below achieves accuracy of around 99.9% on the training data:
import numpy as np
import tensorflow as tf
sess = tf.InteractiveSession()
# generate data
np.random.seed(10)
inputs = np.random.normal(size=[1000,150]).astype('float32')*1.5
label = np.round(np.random.uniform(low=0,high=1,size=[1000,1])*0.8)
reverse_label = 1-label
labels = np.append(label,reverse_label,1)
# parameters
learn_rate = 0.002
epochs = 400
n_input = 150
n_hidden = 60
n_output = 2
# set weights/biases
x = tf.placeholder(tf.float32, [None, n_input])
y = tf.placeholder(tf.float32, [None, n_output])
b0 = tf.Variable(tf.truncated_normal([n_hidden],stddev=0.2,seed=0))
b1 = tf.Variable(tf.truncated_normal([n_output],stddev=0.2,seed=0))
w0 = tf.Variable(tf.truncated_normal([n_input,n_hidden],stddev=0.2,seed=0))
w1 = tf.Variable(tf.truncated_normal([n_hidden,n_output],stddev=0.2,seed=0))
# step function
def returnPred(x,w0,w1,b0,b1):
z1 = tf.add(tf.matmul(x, w0), b0)
a2 = tf.nn.relu(z1)
z2 = tf.add(tf.matmul(a2, w1), b1)
h = tf.nn.relu(z2)
return h #return the first response vector from the
y_ = returnPred(x,w0,w1,b0,b1) # predict operation
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_,labels=y)) # calculate loss between prediction and actual
model = tf.train.AdamOptimizer(learning_rate=learn_rate).minimize(loss) # apply gradient descent based on loss
init = tf.global_variables_initializer()
tf.Session = sess
sess.run(init) #initialize graph
for step in range(0,epochs):
sess.run([model,loss],feed_dict={x: inputs, y: labels }) #train model
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(sess.run(accuracy, feed_dict={x: inputs, y: labels})) # print accuracy
Just a suggestion in addition to the answer provided by Miriam Farber:
You use a multi-dimensional output label ([0., 1.]) for the classification. I suggest to use the softmax cross entropy tf.nn.softmax_cross_entropy_with_logits() instead of the sigmoid cross entropy, since you assume the outputs to be disjoint softmax on Wikipedia. I achieved much faster convergence with this small modification.
This should also improve your performance once you decide to increase your output dimensionality from 2 to a higher number.
I guess you have some problem here:
loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=y_,labels=y) # calculate loss between prediction and actual
It should look smth like that:
loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=y_,labels=y))
Did't look at you code much, so if this would't work out you can check udacity deep learning course or forum they have good samples of that are you trying to do.
GL
I implemented a relatively straightforward logistic regression function. I save all the necessary variables such as weights, bias, x, y, etc. and then I run the training algorithm...
# launch the graph
with tf.Session() as sess:
sess.run(init)
# training cycle
for epoch in range(FLAGS.training_epochs):
avg_cost = 0
total_batch = int(mnist.train.num_examples/FLAGS.batch_size)
# loop over all batches
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(FLAGS.batch_size)
_, c = sess.run([optimizer, cost], feed_dict={x: batch_xs, y: batch_ys})
# compute average loss
avg_cost += c / total_batch
# display logs per epoch step
if (epoch + 1) % FLAGS.display_step == 0:
print("Epoch:", '%04d' % (epoch + 1), "cost=", "{:.9f}".format(avg_cost))
save_path = saver.save(sess, "/tmp/model.ckpt")
The model is saved and the prediction and accuracy of the trained model is displayed...
# list of booleans to determine the correct predictions
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
print(correct_prediction.eval({x:mnist.test.images, y:mnist.test.labels}))
# calculate total accuracy
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("Accuracy:", accuracy.eval({x: mnist.test.images, y: mnist.test.labels}))
This is all fine and dandy. However, now I want to be able to predict any given image using the trained model. For example, I want to feed it picture of say 7 and see what it predicts it to be.
I have another module that restores the model. First we load the variables...
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
# tf Graph Input
x = tf.placeholder(tf.float32, [None, 784]) # mnist data image of shape 28*28=784
y = tf.placeholder(tf.float32, [None, 10]) # 0-9 digits recognition => 10 classes
# set model weights
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
# construct model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax
# minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y * tf.log(pred), reduction_indices=1))
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(FLAGS.learning_rate).minimize(cost)
# initializing the variables
init = tf.global_variables_initializer()
saver = tf.train.Saver()
with tf.Session() as sess:
save.restore(sess, "/tmp/model.ckpt")
This is good. Now I want to compare one image to the model and get a prediction. In this example, I take the first image from the test dataset mnist.test.images[0] and I attempt to compare it to the model.
classification = sess.run(tf.argmax(pred, 1), feed_dict={x: mnist.test.images[0]})
print(classification)
I know this will not work. I get the error...
ValueError: Cannot feed value of shape (784,) for Tensor 'Placeholder:0', which has shape '(?, 784)'
I am at a loss for ideas. This question is rather long, if a straightforward answer is not possible, some guidance as to the steps I may take to do this is appreciated.
Your input placeholder must be of size (?, 784), the question mark meaning variable size which is probably the batch size. You are feeding an input of size (784,) which does not work as the error message states.
In your case, during prediction time, the batch size is just 1, so the following should work:
import numpy as np
...
x_in = np.expand_dims(mnist.test.images[0], axis=0)
classification = sess.run(tf.argmax(pred, 1), feed_dict={x:x_in})
Assuming that the input image is available as a numpy array. If it is already a tensor, the corresponding function is tf.expand_dims(..).