We Keep Coding

spliting custom binary dataset in train/test subsets using tensorflow io

I am trying to use local binary data to train a network to perform regression inference.
Each local binary data has the following layout:
and the whole data consists of several *.bin files with the layout above. Each file has a variable number of sequences of 403*4 bytes. I was able to read one of those files using the following code:
import tensorflow as tf
RAW_N = 2 + 20*20 + 1
def convert_binary_to_float_array(register):
return tf.io.decode_raw(register, out_type=tf.float32)
raw_dataset = tf.data.FixedLengthRecordDataset(filenames=['mydata.bin'],record_bytes=RAW_N*4)
raw_dataset = raw_dataset.map(map_func=convert_binary_to_float_array)
Now, I need to create 4 datasets train_data, train_labels, test_data, test_labels as follows:
train_data, train_labels, test_data, test_labels = prepare_ds(raw_dataset, 0.8)
and use them to train & evaluate:
model = build_model()
history = model.fit(train_data, train_labels, ...)
loss, mse = model.evaluate(test_data, test_labels)
My question is: how to implement function prepare_ds(dataset, frac)?
def prepare_ds(dataset, frac):
...
I have tried to use tf.shape, tf.reshape, tf.slice, subscription [:] with no success. I realized that those functions doesn't work properly because after the map() call raw_dataset is a MapDataset (as a result of the eager execution concerns).

If the meta-data is suppose to be part of your inputs, which I am assuming, you could try something like this:
import random
import struct
import tensorflow as tf
import numpy as np
RAW_N = 2 + 20*20 + 1
bytess = random.sample(range(1, 5000), RAW_N*4)
with open('mydata.bin', 'wb') as f:
f.write(struct.pack('1612i', *bytess))
def decode_and_prepare(register):
register = tf.io.decode_raw(register, out_type=tf.float32)
inputs = register[:402]
label = register[402:]
return inputs, label
total_data_entries = 8
raw_dataset = tf.data.FixedLengthRecordDataset(filenames=['/content/mydata.bin', '/content/mydata.bin'], record_bytes=RAW_N*4)
raw_dataset = raw_dataset.map(decode_and_prepare)
raw_dataset = raw_dataset.shuffle(buffer_size=total_data_entries)
train_ds_size = int(0.8 * total_data_entries)
test_ds_size = int(0.2 * total_data_entries)
train_ds = raw_dataset.take(train_ds_size)
remaining_data = raw_dataset.skip(train_ds_size)
test_ds = remaining_data.take(test_ds_size)
Note that I am using the same bin file twice for demonstration purposes. After running that code snippet, you could feed the datasets to your model like this:
model = build_model()
history = model.fit(train_ds, ...)
loss, mse = model.evaluate(test_ds)
as each dataset contains the inputs and the corresponding labels.

tensorflow lite model gives very different accuracy value compared to python model

I am using tensorflow 1.10 Python 3.6
My code is based in the premade iris classification model provided by TensorFlow. This means, I am using a Tensorflow DNN premade classifier, with the following difference:
10 features instead 4.
5 classes instead 3.
The test and training files can be downloaded from the following link:
https://www.dropbox.com/sh/nmu8i2i8xe6hvfq/AADQEOIHH8e-kUHQf8zmmDMDa?dl=0
I have made a code to export this classifier to a tflite format, however the accuracy in the python model is higher than 75% but when exported the accuracy decrease approximately to 45% this means approximately 30% Accuracy is lost (This is too much).
I have tried the code with different set of data and in all of them the accuracy after exporting decrease a lot!
This made me think that something is going wrong with the TocoConverter function or that maybe I am exporting to tflite incorrectly, missing a parameter or something like that.
This is the way I generate the model:
classifier = tf.estimator.DNNClassifier(
feature_columns=my_feature_columns,
hidden_units=[100, 500],
optimizer=tf.train.AdagradOptimizer(learning_rate=0.003),
n_classes=num_labels,
model_dir="myModel")
And this is the function I am using to convert to tflite:
converter = tf.contrib.lite.TocoConverter.from_frozen_graph(final_model_path, input_arrays, output_arrays, input_shapes={"dnn/input_from_feature_columns/input_layer/concat": [1, 10]})
tflite_model = converter.convert()
I share the complete code in which I also calculate the accuracy of the resulting .tflite file.
import argparse
import tensorflow as tf
import pandas as pd
import csv
from tensorflow.python.tools import freeze_graph
from tensorflow.python.tools import optimize_for_inference_lib
import numpy as np
parser = argparse.ArgumentParser()
parser.add_argument('--batch_size', default=100, type=int, help='batch size')
parser.add_argument('--train_steps', default=1000, type=int,
help='number of training steps')
features_global = None
feature_spec = None
MODEL_NAME = 'myModel'
def load_data(train_path, test_path):
"""Returns the iris dataset as (train_x, train_y), (test_x, test_y)."""
with open(train_path, newline='') as f:
reader = csv.reader(f)
column_names = next(reader)
y_name = column_names[-1]
train = pd.read_csv(train_path, names=column_names, header=0)
train_x, train_y = train, train.pop(y_name)
test = pd.read_csv(test_path, names=column_names, header=0)
test_x, test_y = test, test.pop(y_name)
return (train_x, train_y), (test_x, test_y)
def train_input_fn(features, labels, batch_size):
"""An input function for training"""
# Convert the inputs to a Dataset.
dataset = tf.data.Dataset.from_tensor_slices((dict(features), labels))
# Shuffle, repeat, and batch the examples.
dataset = dataset.shuffle(1000).repeat().batch(batch_size)
# Return the dataset.
return dataset
def eval_input_fn(features, labels, batch_size):
"""An input function for evaluation or prediction"""
features=dict(features)
if labels is None:
# No labels, use only features.
inputs = features
else:
inputs = (features, labels)
# Convert the inputs to a Dataset.
dataset = tf.data.Dataset.from_tensor_slices(inputs)
# Batch the examples
assert batch_size is not None, "batch_size must not be None"
dataset = dataset.batch(batch_size)
# Return the dataset.
return dataset
def main(argv):
args = parser.parse_args(argv[1:])
train_path = "trainData.csv"
test_path = "testData.csv"
# Fetch the data
(train_x, train_y), (test_x, test_y) = load_data(train_path, test_path)
# Load labels
num_labels = 5
# Feature columns describe how to use the input.
my_feature_columns = []
for key in train_x.keys():
my_feature_columns.append(tf.feature_column.numeric_column(key=key))
# Build 2 hidden layer DNN
classifier = tf.estimator.DNNClassifier(
feature_columns=my_feature_columns,
hidden_units=[100, 500],
optimizer=tf.train.AdagradOptimizer(learning_rate=0.003),
# The model must choose between 'num_labels' classes.
n_classes=num_labels,
model_dir="myModel")
# Train the Model
classifier.train(
input_fn=lambda:train_input_fn(train_x, train_y,
args.batch_size),
steps=args.train_steps)
# Evaluate the model.
eval_result = classifier.evaluate(
input_fn=lambda:eval_input_fn(test_x, test_y,
args.batch_size))
print('\nTest set accuracy: {accuracy:0.3f}\n'.format(**eval_result))
# Export model
feature_spec = tf.feature_column.make_parse_example_spec(my_feature_columns)
serve_input_fun = tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec)
saved_model_path = classifier.export_savedmodel(
export_dir_base="out",
serving_input_receiver_fn=serve_input_fun,
as_text=True,
checkpoint_path=classifier.latest_checkpoint(),
)
tf.reset_default_graph()
var = tf.Variable(0)
with tf.Session() as sess:
# First let's load meta graph and restore weights
sess.run(tf.global_variables_initializer())
latest_checkpoint_path = classifier.latest_checkpoint()
saver = tf.train.import_meta_graph(latest_checkpoint_path + '.meta')
saver.restore(sess, latest_checkpoint_path)
input_arrays = ["dnn/input_from_feature_columns/input_layer/concat"]
output_arrays = ["dnn/logits/BiasAdd"]
frozen_graph_def = tf.graph_util.convert_variables_to_constants(
sess, sess.graph_def,
output_node_names=["dnn/logits/BiasAdd"])
frozen_graph = "out/frozen_graph.pb"
with tf.gfile.FastGFile(frozen_graph, "wb") as f:
f.write(frozen_graph_def.SerializeToString())
# save original graphdef to text file
with open("estimator_graph.pbtxt", "w") as fp:
fp.write(str(sess.graph_def))
# save frozen graph def to text file
with open("estimator_frozen_graph.pbtxt", "w") as fp:
fp.write(str(frozen_graph_def))
input_node_names = input_arrays
output_node_name = output_arrays
output_graph_def = optimize_for_inference_lib.optimize_for_inference(
frozen_graph_def, input_node_names, output_node_name,
tf.float32.as_datatype_enum)
final_model_path = 'out/opt_' + MODEL_NAME + '.pb'
with tf.gfile.FastGFile(final_model_path, "wb") as f:
f.write(output_graph_def.SerializeToString())
tflite_file = "out/iris.tflite"
converter = tf.contrib.lite.TocoConverter.from_frozen_graph(final_model_path, input_arrays, output_arrays, input_shapes={"dnn/input_from_feature_columns/input_layer/concat": [1, 10]})
tflite_model = converter.convert()
open(tflite_file, "wb").write(tflite_model)
interpreter = tf.contrib.lite.Interpreter(model_path=tflite_file)
interpreter.allocate_tensors()
# Get input and output tensors.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()
# Test model on random input data.
input_shape = input_details[0]['shape']
# change the following line to feed into your own data.
input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
resultlist = list()
df = pd.read_csv(test_path)
expected = df.iloc[:, -1].values.tolist()
with open(test_path, newline='') as f:
reader = csv.reader(f)
column_names = next(reader)
for x in range(0, len(expected)):
linea = next(reader)
linea = linea[:len(linea) - 1]
input_data2 = np.array(linea, dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], [input_data2])
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
#print(output_data)
max = 0;
longitud = len(output_data[0])
for k in range(0, longitud):
if (output_data[0][k] > output_data[0][max]):
max = k
resultlist.append(max)
print(resultlist)
coincidences = 0
for pred_dict, expec in zip(resultlist, expected):
if pred_dict == expec:
coincidences = coincidences + 1
print("tflite Accuracy: " + str(coincidences / len(expected)))
if __name__ == '__main__':
tf.logging.set_verbosity(tf.logging.INFO)
tf.app.run(main)
I hope some of you can identify the error, or give a possible solution

This question is answered here might help.
As mentioned in the answer share, doing some
pre-processing
on the image before it is fed into "interpreter.invoke()" solves the issue if that was the problem in the first place.
To elaborate on that here is a block quote from the shared link:
The below code you see is what I meant by pre-processing:
test_image = cv2.imread(file_name)
test_image = cv2.resize(test_image,(299,299),cv2.INTER_AREA)
test_image = np.expand_dims((test_image)/255,axis=0).astype(np.float32)
interpreter.set_tensor(input_tensor_index, test_image)
interpreter.invoke()
digit = np.argmax(output()[0])
#print(digit)
prediction = result[digit]
As you can see there are two crucial commands/pre-processing done on
the image once it is read using "imread()":
i) The image should be resized to the size that is the "input_height"
and "input_width" values of the input image/tensor that was used
during the training. In my case (inception-v3) this was 299 for both
"input_height" and "input_width". (Read the documentation of the model
for this value or look for this variable in the file that you used to
train or retrain the model)
ii) The next command in the above code is:
test_image = np.expand_dims((test_image)/255,axis=0).astype(np.float32)
I got this from the "formulae"/model code:
test_image = np.expand_dims((test_image - input_mean)/input_std, axis=0).astype(np.float32)
Reading the documentation revealed that for my architecture input_mean = 0 and input_std = 255.
Hope this helps.

I have met the same problem. It seems to me that the accuracy problem is mainly caused by failure to detect overlapping objects. I couldn't figure out what part of the code is wrong though.

How to use dataset in TensorFlow session for training

I like to perform image classification on our own large image libary (millions of labeled images) with tensorflow. I´m new to stackoverflow, python and tensorflow and worked myself through a few tutorials (mnist etc.) and got to the point, where i was able to prepare a TensorFlow datset from a dictionary including the absolute path to the images and the according labels. However, i´m stuck at the point using the dataset in a TensorFlow session. Here is my (example) code:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import tensorflow as tf
import numpy as np
import time
import mymodule # I build my module to read the images and labels
from tensorflow.python.framework import ops
from tensorflow.python.framework import dtypes
from tensorflow.contrib.data import Iterator
beginTime = time.time()
batch_size = 100
learning_rate = 0.005
max_steps = 2
NUM_CLASSES = 25
def input_parser(img_path, label):
one_hot = tf.one_hot(label, NUM_CLASSES)
img_file = tf.read_file(img_path)
img_decoded = tf.image.decode_jpeg(img_file, channels = 3)
return img_decoded, one_hot
#Import Training data (returns the dicitonary with paths and labels)
train_dict = mymodule.getFileMap(labelList, imageList)
#Import Test data
test_dict = mymodule.getFileMap(labelList, imageList)
#Get train data
train_file_list, train_label_list = get_file_label_list(train_dict)
train_images_tensor = ops.convert_to_tensor(train_file_list, dtype=dtypes.string)
train_labels_tensor = ops.convert_to_tensor(train_label_list, dtype=dtypes.int64)
#Get test data
test_file_list, test_label_list = get_file_label_list(test_dict)
test_images_tensor = ops.convert_to_tensor(test_file_list, dtype=dtypes.string)
test_labels_tensor = ops.convert_to_tensor(test_label_list, dtype=dtypes.int64)
#Create TensorFlow Datset object
train_data = tf.data.Dataset.from_tensor_slices((train_images_tensor, train_labels_tensor))
test_data = tf.data.Dataset.from_tensor_slices((test_images_tensor, test_labels_tensor))
# Transform the datset so that it contains decoded images
# and one-hot vector labels
train_data = train_data.map(input_parser)
test_data = test_data.map(input_parser)
# Batching --> How to do it right?
#train_data = train_data.batch(batch_size = 100)
#test_data = train_data.batch(batch_size = 100)
#Define input placeholders
image_size = 990*990*3
images_placeholder = tf.placeholder(tf.float32, shape=[None, image_size])
labels_placeholder = tf.placeholder(tf.int64, shape=[None])
# Define variables (these afe the values we want to optimize)
weigths = tf.Variable(tf.zeros([image_size, NUM_CLASSES]))
biases = tf.Variable(tf.zeros([NUM_CLASSES]))
# Define the classifier´s result
logits = tf.matmul(images_placeholder, weigths) + biases
# Define the loss function
loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits = logits, labels = labels_placeholder))
# Define the training operation
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
# Operation comparing prediciton with true label
correct_prediciton = tf.equal(tf.argmax(logits, 1), labels_placeholder)
# Operation calculating the accuracy of our predicitons
accuracy = tf.reduce_mean(tf.cast(correct_prediciton, tf.float32))
#Create TensorFlow Iterator object
iterator = Iterator.from_structure(train_data.output_types,
train_data.output_shapes)
next_element = iterator.get_next()
#Create two initialization ops to switch between the datasets
train_init_op = iterator.make_initializer(train_data)
test_init_op = iterator.make_initializer(test_data)
with tf.Session() as sess:
#Initialize variables
sess.run(tf.global_variables_initializer())
sess.run(train_init_op)
for _ in range(10):
try:
elem = sess.run(next_element)
print(elem)
except tf.errors.OutOfRangeError:
print("End of training datset.")
break
Following this and this tutorial i could not solve the problem of how to use the (image and label) dataset in a tensorflow session for training. I was able to print out the datset by iterating through it, but wasn´t able to use it for learning.
I don´t understand how to access the images and labels seperately after they have been merged in the train_data = tf.data.Dataset.from_tensor_slices((train_images_tensor, train_labels_tensor)) operation, as requried by the 2nd tutorial. Also i don´t know how to implement batching correctly.
What i want to do in the session is basically this (from the 2nd tutorial):
# Generate input data batch
indices = np.random.choice(data_sets['images_train'].shape[0], batch_size)
images_batch = data_sets['images_train'][indices]
labels_batch = data_sets['labels_train'][indices]
# Periodically print out the model's current accuracy
if i % 100 == 0:
train_accuracy = sess.run(accuracy, feed_dict={
images_placeholder: images_batch, labels_placeholder: labels_batch})
print('Step {:5d}: training accuracy {:g}'.format(i, train_accuracy))
# Perform a single training step
sess.run(train_step, feed_dict={images_placeholder: images_batch,
labels_placeholder: labels_batch})
# After finishing the training, evaluate on the test set
test_accuracy = sess.run(accuracy, feed_dict={
images_placeholder: data_sets['images_test'],
labels_placeholder: data_sets['labels_test']})
print('Test accuracy {:g}'.format(test_accuracy))
endTime = time.time()
print('Total time: {:5.2f}s'.format(endTime - beginTime))
If anyone can tell me, how to access images and labels in the dataset sepearately and use it for training, i would be really thankful. Also a tip where and how to do the batching would be appreciated.
Thank you.

In your code, next_element is a tuple of two tensors, matching the structure of your datasets: i.e. it is a tuple whose first element is an image, and second element is a label. To access the individual tensors, you can do the following:
next_element = iterator.get_next()
next_image = next_element[0]
next_label = next_element[1]
# Or, in a single line:
next_image, next_label = iterator.get_next()
To batch a tf.data.Dataset, you can use the Dataset.batch() transformation. Your commented out code for this should simply work:
train_data = train_data.batch(batch_size = 100)
test_data = train_data.batch(batch_size = 100)

How to configure `train_and_evaluate` to also save the metric on training set into tensorboard?

I have the following code that trains a model on the iris data set and uses accuracy as the metric to evaluate the model:
# This is based on the complete code for the following blogpost:
# https://developers.googleblog.com/2017/09/introducing-tensorflow-datasets.html
import tensorflow as tf
import os
from tensorflow.contrib.learn import Experiment, RunConfig
from urllib.request import urlopen
from tensorflow.contrib.learn import RunConfig
()
PATH = "./tf_dataset_and_estimator_apis"
# Fetch and store Training and Test dataset files
PATH_DATASET = PATH + os.sep + "dataset"
FILE_TRAIN = PATH_DATASET + os.sep + "iris_training.csv"
FILE_TEST = PATH_DATASET + os.sep + "iris_test.csv"
URL_TRAIN = "http://download.tensorflow.org/data/iris_training.csv"
URL_TEST = "http://download.tensorflow.org/data/iris_test.csv"
def downloadDataset(url, file):
if not os.path.exists(PATH_DATASET):
os.makedirs(PATH_DATASET)
if not os.path.exists(file):
data = urlopen(url).read()
with open(file, "wb") as f:
f.write(data)
f.close()
downloadDataset(URL_TRAIN, FILE_TRAIN)
downloadDataset(URL_TEST, FILE_TEST)
# The CSV features in our training & test data
feature_names = [
'SepalLength',
'SepalWidth',
'PetalLength',
'PetalWidth']
# Create an input function reading a file using the Dataset API
# Then provide the results to the Estimator API
def my_input_fn(file_path, perform_shuffle=False, repeat_count=1):
def decode_csv(line):
parsed_line = tf.decode_csv(line, [[0.], [0.], [0.], [0.], [0]])
label = parsed_line[-1:] # Last element is the label
del parsed_line[-1] # Delete last element
features = parsed_line # Everything but last elements are the features
d = dict(zip(feature_names, features)), label
return d
dataset = (tf.contrib.data.TextLineDataset(file_path) # Read text file
.skip(1) # Skip header row
.map(decode_csv)) # Transform each elem by applying decode_csv fn
if perform_shuffle:
# Randomizes input using a window of 256 elements (read into memory)
dataset = dataset.shuffle(buffer_size=256)
# dataset = dataset.repeat(repeat_count) # Repeats dataset this # times
dataset = dataset.repeat()
dataset = dataset.batch(32) # Batch size to use
iterator = dataset.make_one_shot_iterator()
batch_features, batch_labels = iterator.get_next()
return batch_features, batch_labels
next_batch = my_input_fn(FILE_TRAIN, True) # Will return 32 random elements
# Create the feature_columns, which specifies the input to our model
# All our input features are numeric, so use numeric_column for each one
feature_columns = [tf.feature_column.numeric_column(k) for k in feature_names]
# Create a deep neural network regression classifier
# Use the DNNClassifier pre-made estimator
config = RunConfig(save_checkpoints_steps=50, save_summary_steps=50)
classifier = tf.estimator.DNNClassifier(
feature_columns=feature_columns, # The input features to our model
hidden_units=[10, 10], # Two layers, each with 10 neurons
n_classes=3,
model_dir=PATH + '/model_dir',
config=config) # Path to where checkpoints etc are stored
experiment = Experiment(
estimator=classifier,
train_input_fn=lambda: my_input_fn(FILE_TRAIN, True, 8),
eval_input_fn=lambda: my_input_fn(FILE_TEST, False, 4),
eval_metrics=None,
train_steps=1000,
min_eval_frequency=50,
eval_delay_secs=0
)
experiment.train_and_evaluate()
This is the tensorboard result:
Is there a way to configure the Experiment or the DNNClassifier class so that the accuracy metric is also computed on the training set and displayed in tensorboard?

You can use tf.summary.scalar to display training accuracy in tensorboard, for example:
accuracy = tf.metrics.mean(some_tensor)
if mode == tf.estimator.ModeKeys.TRAIN:
tf.summary.scalar('accuracy', accuracy[1])

Tensorflow lstm incremental learning and multiple predictions

I am training a tensorflow model and later plan to use it for predictions.
import numpy as np
import pandas as pd
import sys
import tensorflow as tf
from tensorflow.contrib import learn
from sklearn.metrics import mean_squared_error, mean_absolute_error
from lstm_predictor import load_csvdata, lstm_model
import pymysql as mariadb
LOG_DIR = './ops_logs'
K = 1 # history used for lstm.
TIMESTEPS = 65*K
RNN_LAYERS = [{'steps': TIMESTEPS}]
DENSE_LAYERS = [10, 10]
TRAINING_STEPS = 1000
BATCH_SIZE = 1
PRINT_STEPS = TRAINING_STEPS / 10
def train_model(symbol=1,categ='M1',limit=1000,upgrade=False):
MODEL_DIR = 'model/'+str(symbol)+categ
regressor = learn.TensorFlowEstimator(model_fn=lstm_model(TIMESTEPS, RNN_LAYERS, DENSE_LAYERS),
n_classes=0,
verbose=1,
steps=TRAINING_STEPS,
optimizer='Adagrad',
learning_rate=0.03,
continue_training=True,
batch_size=BATCH_SIZE )
X, y = load_csvdata(df, K )
regressor.fit(X['train'], y['train'] , logdir=MODEL_DIR ) #logdir=LOG_DIR)
X['test']=X['train'][-10:]
y['test']=y['train'][-10:]
predicted = regressor.predict(X['test'])
print('actual', 'predictions')
for i,yi in enumerate(y['test']):
print(yi[0], ' ' ,predicted[i])
mse = mean_absolute_error(y['test'], predicted)
print ("mean_absolute_error : %f" % mse)
###############################
regressor.save( LOG_DIR )
train_model()
Then I want to write a predict function which would read the model from model/** and make predictions.
def predict(symbol=1,categ='M1'):
pass
# how to load saved model data ?
But I am unable to load the model using
regressor = learn.TensorFlowEstimator.restore( LOG_DIR )
Since its currently not implemented.
Suggest me how can I do repeated predictions at multiple times in future?
The model checkpoints are saved as:
checkpoint model.ckpt-8001.meta
events.out.tfevents.1476102309.hera.creatory.org model.ckpt-8301-00000-of-00001
events.out.tfevents.1476102926.hera.creatory.org model.ckpt-8301.meta
events.out.tfevents.1476105626.hera.creatory.org model.ckpt-8601-00000-of-00001
events.out.tfevents.1476106521.hera.creatory.org model.ckpt-8601.meta
events.out.tfevents.1476106839.hera.creatory.org model.ckpt-8901-00000-of-00001
events.out.tfevents.1476107001.hera.creatory.org model.ckpt-8901.meta
events.out.tfevents.1476107462.hera.creatory.org model.ckpt-9000-00000-of-00001
graph.pbtxt model.ckpt-9000.meta
model.ckpt-8001-00000-of-00001