I am currently in the process of planning my first Conv. NN implementation in Tensorflow, and have been reading many of the tutorials available on Tensorflow's website for insight.
It seems that there are essentially two ways to create a custom CNN:
1) Use Tensorflow layers module tf.layers, which is the "high-level API". Using this method, you define a model definition function consisting of tf.layers objects, and in the main function, instantiate a tf.learn.Estimator, passing the model definition function to it. From here, the fit() and evaluate() methods can be called on the Estimator object, which train and validate, respectively. Link: https://www.tensorflow.org/tutorials/layers. Main function below:
def main(unused_argv):
# Load training and eval data
mnist = learn.datasets.load_dataset("mnist")
train_data = mnist.train.images # Returns np.array
train_labels = np.asarray(mnist.train.labels, dtype=np.int32)
eval_data = mnist.test.images # Returns np.array
eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)
# Create the Estimator
mnist_classifier = learn.Estimator(
model_fn=cnn_model_fn, model_dir="/tmp/mnist_convnet_model")
# Set up logging for predictions
# Log the values in the "Softmax" tensor with label "probabilities"
tensors_to_log = {"probabilities": "softmax_tensor"}
logging_hook = tf.train.LoggingTensorHook(
tensors=tensors_to_log, every_n_iter=50)
# Train the model
mnist_classifier.fit(
x=train_data,
y=train_labels,
batch_size=100,
steps=20000,
monitors=[logging_hook])
# Configure the accuracy metric for evaluation
metrics = {
"accuracy":
learn.MetricSpec(
metric_fn=tf.metrics.accuracy, prediction_key="classes"),
}
# Evaluate the model and print results
eval_results = mnist_classifier.evaluate(
x=eval_data, y=eval_labels, metrics=metrics)
print(eval_results)
Full code here
2) Use Tensorflow's "low-level API" in which layers are defined in a definition function. Here, layers are manually defined, and the user must perform many calculations manually. In the main function, the user starts a tf.Session(), and manually configures training/validation using for loop(s). Link: https://www.tensorflow.org/get_started/mnist/pros. Main function below:
def main(_):
# Import data
mnist = input_data.read_data_sets(FLAGS.data_dir, one_hot=True)
# Create the model
x = tf.placeholder(tf.float32, [None, 784])
# Define loss and optimizer
y_ = tf.placeholder(tf.float32, [None, 10])
# Build the graph for the deep net
y_conv, keep_prob = deepnn(x)
with tf.name_scope('loss'):
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=y_,
logits=y_conv)
cross_entropy = tf.reduce_mean(cross_entropy)
with tf.name_scope('adam_optimizer'):
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
with tf.name_scope('accuracy'):
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
correct_prediction = tf.cast(correct_prediction, tf.float32)
accuracy = tf.reduce_mean(correct_prediction)
graph_location = tempfile.mkdtemp()
print('Saving graph to: %s' % graph_location)
train_writer = tf.summary.FileWriter(graph_location)
train_writer.add_graph(tf.get_default_graph())
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(20000):
batch = mnist.train.next_batch(50)
if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict={
x: batch[0], y_: batch[1], keep_prob: 1.0})
print('step %d, training accuracy %g' % (i, train_accuracy))
train_step.run(feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})
print('test accuracy %g' % accuracy.eval(feed_dict={
x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
Full code here
My dilemma is, I like the simplicity of defining the neural network using tf.layers (option 1), but I want the customizability of the training that the "low-level API" (option 2) provides. Specifically, when using the tf.layers implementation, is there a way to report validation accuracy every n iterations of training? Or more generally, can I train/validate using the tf.Session(), or am I confined to using the tf.learn.Estimator's fit() and evaluate() methods?
It seems odd that one would want a final evaluation score after all training is complete, as I thought the whole point of validation is to track network progression during training. Otherwise, what would be the difference between validation and testing?
Any help would be appreciated.
You're nearly right however tf.layers is separate from the Estimator class of functions etc. If you wanted to you could use tf.Layers to define your layers but then build your own training loops or whatever else you like. You can think of tf.Layers just being those functions that you could create in your second option above.
If you are interested in being able to build up a basic model quickly but being able to extend it with other functions, your own training loops etc. then there's no reason you can't use layers to build your model and interact with it however you wish.
tf.Layers - https://www.tensorflow.org/api_docs/python/tf/layers
tf.Estimator - https://www.tensorflow.org/api_docs/python/tf/estimator
Related
I used the tutorial "Classify Flowers with Transfer Learning" to retrain on my own classes.
Unfortunately, no example is provided on how to properly save your model as .pb.
I tried to implement my solution based on the different post I could read. At the end I get to save something as .pb , but I can not reopen the model later. I get the following error:
ValueError: Input 0 of node AssignVariableOp was passed float from
module/InceptionV3/Conv2d_1a_3x3/BatchNorm/beta:0 inco
mpatible with expected resource
Below, in between ** are the parts of the above mentioned colab notebook that I modified to save the model:
def create_model(features):
layer = tf.layers.dense(inputs=features, units=NUM_CLASSES, activation=None)
return layer
logits = create_model(features)
labels = tf.placeholder(tf.float32, [None, NUM_CLASSES])
cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(logits=logits, labels=labels)
cross_entropy_mean = tf.reduce_mean(cross_entropy)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=LEARNING_RATE)
train_op = optimizer.minimize(loss=cross_entropy_mean)
probabilities = tf.nn.softmax(logits)
prediction = tf.argmax(probabilities, 1)
correct_prediction = tf.equal(prediction, tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
**saver = tf.train.Saver()**
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
for i in range(NUM_TRAIN_STEPS):
# Get a random batch of training examples.
train_batch = get_batch(batch_size=TRAIN_BATCH_SIZE)
batch_images, batch_labels = get_images_and_labels(train_batch)
# Run the train_op to train the model.
train_loss, _, train_accuracy = sess.run(
[cross_entropy_mean, train_op, accuracy],
feed_dict={encoded_images: batch_images, labels: batch_labels})
is_final_step = (i == (NUM_TRAIN_STEPS - 1))
if i % EVAL_EVERY == 0 or is_final_step:
# Get a batch of test examples.
test_batch = get_batch(batch_size=None, test=True)
batch_images, batch_labels = get_images_and_labels(test_batch)
# Evaluate how well our model performs on the test set.
test_loss, test_accuracy, test_prediction, correct_predicate = sess.run(
[cross_entropy_mean, accuracy, prediction, correct_prediction],
feed_dict={encoded_images: batch_images, labels: batch_labels})
print('Test accuracy at step %s: %.2f%%' % (i, (test_accuracy * 100)))
**saver.save(sess, BASE_DIR+'/simple_model')
tf.io.write_graph(sess.graph, BASE_DIR+'/graph', 'graph.pbtxt')**
**from tensorflow.python.framework import graph_util
saver = tf.train.import_meta_graph(BASE_DIR+'/simple_model.meta',
clear_devices=True
)
graph = tf.get_default_graph()
input_graph_def = graph.as_graph_def()
sess = tf.Session()
saver.restore(sess, BASE_DIR+"/simple_model")
# this will be seen from the file graph.pbtxt
output_node_names="init"
output_graph_def = graph_util.convert_variables_to_constants(
sess,
input_graph_def,
output_node_names.split(",")
)
output_graph= BASE_DIR+"/simple_model .pb"
print("output_graph...", output_graph)
with tf.gfile.GFile(output_graph, "wb") as f:
f.write(output_graph_def.SerializeToString())**
sess.close()
This doable but can be a little fiddly, which is why tensorflow comes with a tool to do it for you.
See:
freeze_graph.py
You can either dig into how it works or simply import it and use it.
I am trying to implement dropout in TensorFlow for a simple 3-layer neural network for classification and am running into problems. More specifically, I am trying to apply different dropout parameter pkeep values when I train vs. test.
I am taking the approach as follows:
1) def create_placeholders(n_x, n_y):
X = tf.placeholder("float", [n_x, None])
Y = tf.placeholder("float", [n_y, None])
pkeep = tf.placeholder(tf.float32)
return X,Y,pkeep
2) In the function forward_propagation(X, parameters, pkeep), I perform the following:
Z1 = tf.add(tf.matmul(W1, X), b1)
A1 = tf.nn.relu(Z1)
A1d = tf.nn.dropout(A1, pkeep)
Z2 = tf.add(tf.matmul(W2, A1d),b2)
A2 = tf.nn.relu(Z2)
A2d = tf.nn.dropout(A2, pkeep)
Z3 = tf.add(tf.matmul(W3, A2d),b3)
return Z3
3) Later when tensorflow session is called (in between code lines omitted for clarity):
X, Y, pkeep = create_placeholders(n_x, n_y)
Z3 = forward_propagation(X, parameters, pkeep)
sess.run([optimizer,cost], feed_dict={X:minibatch_X, Y:minibatch_Y, pkeep: 0.75})
Above would run without giving any errors. However, I think that the above would set pkeep value to 0.75 for both training and testing runs. Minibatching is done only on the train data set but I am not setting the pkeep value anywhere else.
I would like to set pkeep = 0.75 for training and pkeep = 1.0 for testing.
4) It does give an error when I do something like this:
x_train_eval = Z3.eval(feed_dict={X: X_train, Y: Y_train, pkeep: 0.75})
x_test_eval = Z3.eval(feed_dict={X: X_test, Y: Y_test, pkeep: 1.0})
The error message I am getting is:
InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'Placeholder_2' with dtype float
[[Node: Placeholder_2 = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
What is the best way to pass different pkeep values for training and testing? Your advice would be much appreciated.
Assuming you have some operation for testing defined as test_op (for instance, something that evaluates accuracy on input data and labels), you can do something like this:
for i in range(num_iters):
# Run your training process.
_, loss = sess.run([optimizer, cost],
feed_dict={X:minibatch_X, Y:minibatch_Y, pkeep: 0.75})
# Test the model after training
test_accuracy = sess.run(test_op,
feed_dict={X:test_X, Y:test_Y, pkeep: 1.0})
Basically you aren't testing the model when you are training it, so you can just call your test_op when you are ready to test, feeding it different hyperparameters like pkeep. The same goes for validating the model periodically during training on a held-out dataset: every so often run your evaluation op on the held-out dataset, passing different hyperparameters than those used during training, and then you can save off configurations or stop early based on your validation accuracy.
Your test_op could return something like the prediction accuracy for a classifier:
correct = tf.equal(tf.argmax(y, 1), tf.argmax(predict, 1))
test_op = tf.reduce_mean(tf.cast(correct, tf.float32))
where y are your target labels and predict is the name of your prediction op.
In neural networks, you forward propagate to compute the logits (probabilities of each label) and compare it with the real value to get the error.
Training involves minimizing the error, by propagating backwards i.e. finding the derivative of error with respect to each weight and then subtracting this value from the weight value.
Applying different dropout parameters is easier when you separate the operations needed for training and evaluating your model.
Training is just minimizing the loss:
def loss(logits, labels):
'''
Calculate cross entropy loss
'''
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(
labels=labels, logits=logits)
return tf.reduce_mean(cross_entropy, name='loss_op')
def train(loss, learning_rate):
'''
Train model by optimizing gradient descent
'''
optimizer = tf.train.AdamOptimizer(learning_rate)
train_op = optimizer.minimize(loss, name='train_op')
return train_op
Evaluating is just calculating the accuracy:
def accuracy(logits, labels):
'''
Calculate accuracy of logits at predicting labels
'''
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
accuracy_op = tf.reduce_mean(tf.cast(correct_prediction, tf.float32),
name='accuracy_op')
return accuracy_op
Then in your Tensorflow session, after generating placeholders and adding necessary operations to the Graph etc. just feed different keep_prob_pl values into the run methods:
# Train model
sess.run(train_op, feed_dict={x_pl: x_train, y_train: y, keep_prob_pl: 0.75}})
# Evaluate test data
batch_size = 100
epoch = data.num_examples // batch_size
acc = 0.0
for i in range(epoch):
batch_x, batch_y = data.next_batch(batch_size)
feed_dict = {x_pl: batch_x, y_pl: batch_y, keep_prob_pl: 1}
acc += sess.run(accuracy_op, feed_dict=feed_dict)
print(("Epoch Accuracy: {:.4f}").format(acc / epoch))
I am working through some code to understand how to save and restore checkpoints in tensorflow. To do so, I implemented a simple neural netowork that works with MNIST digits and saved the .ckpt file like so:
from tensorflow.examples.tutorials.mnist import input_data
import numpy as np
learning_rate = 0.001
n_input = 784 # MNIST data input (img shape = 28*28)
n_classes = 10 # MNIST total classes 0-9
#import MNIST data
mnist = input_data.read_data_sets('.', one_hot = True)
#Features and Labels
features = tf.placeholder(tf.float32, [None, n_input])
labels = tf.placeholder(tf.float32, [None, n_classes])
#Weights and biases
weights = tf.Variable(tf.random_normal([n_input, n_classes]))
bias = tf.Variable(tf.random_normal([n_classes]))
#logits = xW + b
logits = tf.add(tf.matmul(features, weights), bias)
#Define loss and optimizer
cost = tf.reduce_mean(\
tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=labels))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)\
.minimize(cost)
# Calculate accuracy
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
import math
save_file = './train_model.ckpt'
batch_size = 128
n_epochs = 100
saver = tf.train.Saver()
# Launch the graph
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(n_epochs):
total_batch = math.ceil(mnist.train.num_examples / batch_size)
# Loop over all batches
for i in range(total_batch):
batch_features, batch_labels = mnist.train.next_batch(batch_size)
sess.run(
optimizer,
feed_dict={features: batch_features, labels: batch_labels})
# Print status for every 10 epochs
if epoch % 10 == 0:
valid_accuracy = sess.run(
accuracy,
feed_dict={
features: mnist.validation.images,
labels: mnist.validation.labels})
print('Epoch {:<3} - Validation Accuracy: {}'.format(
epoch,
valid_accuracy))
# Save the model
saver.save(sess, save_file)
print('Trained Model Saved.')
This part works well, and I get the .ckpt file saved in the correct directory. The problem comes in when I try to restore the model in an attempt to work on it again. I use the following code to restore the model:
saver = tf.train.Saver()
with tf.Session() as sess:
saver.restore(sess, 'train_model.ckpt.meta')
print('model restored')
and end up with the error: ValueError: No variables to save
Not too sure, what the mistake here is. Any help is appreciated. Thanks in advance
A Graph is different to the Session. A graph is the set of operations joining tensors, each of which is a symbolic representation of a set of values. A Session assigns specific values to the Variable tensors, and allows you to run operations in that graph.
The chkpt file saves variable values - i.e. those saved in the weights and biases - but not the graph itself.
The solution is simple: re-run the graph construction (everything before the Session, then start your session and load values from the chkpt file.
Alternatively, you can check out this guide for exporting and importing MetaGraphs.
You should tell the Saver which Variables to restore, default Saver will get all the Variables from the default graph.
As in your case, you should add the constructing graph code before saver = tf.train.Saver()
I am making a neural network in python using the tensorflow library to classify the CIFAR-10 dataset. The issue is that I cannot find a way of converting the batch so that the train step will accept the feed. This is my code, with sections I have verified to work being replaced with comments describing them:
# import statements
# declare some global variables
# also declare filepaths
def read_cifar10(filename_queue):
# parse cifar10 file, return as label with uint8image (Cifar10Record)
def generate_batch(image, label):
# generate shuffled batch, given images and labels
def get_imgs(test = False, fmin = 1, files = 1):
# generate the filename(s) and the filename_queue
# read the input using read_cifar10
# cast it to uint8image and float32
# apply distortions
# set the shape of the image and label
# return the batch, made using generate_batch
# build placeholders for input and output
x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])
# create a weight variable with a given shape
# slight amount of variation
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev = 0.1)
return tf.Variable(initial)
# same but for bias variables
def bias_variable(shape):
initial = tf.constant(0.1, shape = shape)
return tf.Variable(initial)
# convolve it, do some more layers, apply relu, make dropout and readout layers, etc
# do softmax regression
# define the loss function, training step, accuracy function
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = y_, logits = y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
guess = tf.argmax(y_conv, 1)
answer = tf.argmax(y_, 1)
correct_prediction = tf.equal(guess, answer)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
def train_nn():
for i in range(TRAIN_STEPS):
# this is where the error occurs
batch = get_imgs(files = 5).eval()
# every 100 steps output training accuracy
if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict = {x: batch[0], y_: batch[1], keep_prob: 0.5})
print('step %d, training accuracy %g' % (i, train_accuracy))
train_step.run(feed_dict = {x: batch[0], y_: batch[1], keep_prob: 0.5})
def test_nn():
# test it
batch = get_imgs(test = True).eval()
print('test accuracy %g' % accuracy.eval(feed_dict = {x: batch[0], y_: batch[1], keep_prob: 1.0}))
# create session
with tf.Session() as sess:
# initialize global variables
sess.run(tf.global_variables_initializer())
# make queue runners
coord = tf.train.Coordinator()
threads = tf.train.start_queue_runners(sess = sess, coord = coord)
# train it, test it
sess.run(train_nn())
sess.run(test_nn())
coord.request_stop()
coord.join(threads)
Some things I have tried and the results:
batch = get_imgs(args) gives 'TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, numpy ndarrays, or TensorHandles.'
batch = get_imgs(args).eval() gives 'AttributeError: 'tuple' object has no attribute 'eval''
batch = sess.run(get_imgs(args)) causes the program to run indefinitely with no output
printing type(batch) says the batch is a tuple
printing batch gives a description of a tensor
printing batch.eval() or type(batch.eval()) gives 'W tensorflow/core/kernels/queue_base.cc:302] _3_input_producer: Skipping cancelled dequeue attempt with queue not closed'
I suspect the issue is either with the batch conversion, the queueing with tf.train.Coordinator(), or the placeholders. Any help would be greatly appreciated.
I was thinking of trying to choose hyper parameters (like regularization for example) using cross validation or maybe train multiple initializations of a models and then choose the model with highest cross validation accuracy. Implementing k-fold or CV is simple but tedious/annoying (specially if I am trying to train different models in different CPU's, GPU's or even different computers etc). I would expect a library like TensorFlow to have something like this implemented for its user so that we don't have to code the same thing 100 times. Thus, does TensorFlow have a library or something that can help me do Cross Validation?
As an update, it seems one could use scikit learn or something else to do this. If this is the case, then if anyone can provide a simple example of NN training and cross validation with scikit learn it would be awesome! Not sure if this scales to multiple cpus, gpus, clusters etc though.
As already discussed, tensorflow doesn't provide its own way to cross-validate the model. The recommended way is to use KFold. It's a bit tedious, but doable. Here's a complete example of cross-validating MNIST model with tensorflow and KFold:
from sklearn.model_selection import KFold
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
# Parameters
learning_rate = 0.01
batch_size = 500
# TF graph
x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
pred = tf.nn.softmax(tf.matmul(x, W) + b)
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
init = tf.global_variables_initializer()
mnist = input_data.read_data_sets("data/mnist-tf", one_hot=True)
train_x_all = mnist.train.images
train_y_all = mnist.train.labels
test_x = mnist.test.images
test_y = mnist.test.labels
def run_train(session, train_x, train_y):
print "\nStart training"
session.run(init)
for epoch in range(10):
total_batch = int(train_x.shape[0] / batch_size)
for i in range(total_batch):
batch_x = train_x[i*batch_size:(i+1)*batch_size]
batch_y = train_y[i*batch_size:(i+1)*batch_size]
_, c = session.run([optimizer, cost], feed_dict={x: batch_x, y: batch_y})
if i % 50 == 0:
print "Epoch #%d step=%d cost=%f" % (epoch, i, c)
def cross_validate(session, split_size=5):
results = []
kf = KFold(n_splits=split_size)
for train_idx, val_idx in kf.split(train_x_all, train_y_all):
train_x = train_x_all[train_idx]
train_y = train_y_all[train_idx]
val_x = train_x_all[val_idx]
val_y = train_y_all[val_idx]
run_train(session, train_x, train_y)
results.append(session.run(accuracy, feed_dict={x: val_x, y: val_y}))
return results
with tf.Session() as session:
result = cross_validate(session)
print "Cross-validation result: %s" % result
print "Test accuracy: %f" % session.run(accuracy, feed_dict={x: test_x, y: test_y})
As the dataset gets larger cross validation gets more expensive.In deep learning we usually use large data sets.You should be fine with simple training. Tensorflow doesnt have a built in mechanism for cv as it is not usually used in neural networks.In neural networks, the efficiency of the network relies mainly on the data set, number of epochs and the learning rate.
I have used cv in sklearn
You can check the link:
https://github.com/hackmaster0110/Udacity-Data-Analyst-Nano-Degree-Projects/
In that,go to poi_id.py in Identify fraud from enron data(In Project folder)