I am working on a multilabel classification model where I am trying to combine two models, a CNN and a text-classifier into one model using Keras and train them together, like so:
#cnn_model is a vgg16 model
#text_model looks as follows:
### takes the vectorized text as input
text_model = Sequential()
text_model .add(Dense(vec_size, input_shape=(vec_size,), name='aux_input'))
## merging both models
merged = Merge([cnn_model, text_model], mode='concat')
### final_model takes the combined models and adds a sofmax classifier to it
final_model = Sequential()
final_model.add(merged)
final_model.add(Dense(n_classes, activation='softmax'))
As such, I am working with an ImageDataGenerator to process the images and the respective labels.
For the images I am using a custom helper function that reads images into the model via paths provided by pandas dataframes - one for training (df_train) and one for validation (df_validation). The dataframes also provide the final labels for the model in the "label_vec" column:
# From https://github.com/keras-team/keras/issues/5152
def flow_from_dataframe(img_data_gen, in_df, path_col, y_col, **dflow_args):
base_dir = os.path.dirname(in_df[path_col].values[0])
print('## Ignore next message from keras, values are replaced anyways')
df_gen = img_data_gen.flow_from_directory(base_dir, class_mode = 'sparse', **dflow_args)
df_gen.filenames = in_df[path_col].values
df_gen.classes = numpy.stack(in_df[y_col].values)
df_gen.samples = in_df.shape[0]
df_gen.n = in_df.shape[0]
df_gen._set_index_array()
df_gen.directory = '' # since we have the full path
print('Reinserting dataframe: {} images'.format(in_df.shape[0]))
return df_gen
from keras.applications.vgg16 import preprocess_input
train_datagen = keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input) horizontal_flip=True)
validation_datagen = keras.preprocessing.image.ImageDataGenerator(preprocessing_function=preprocess_input)#rescale=1./255)
train_generator = flow_from_dataframe(train_datagen, df_train,
path_col = 'filename',
y_col = 'label_vec',
target_size=(224, 224), batch_size=128, shuffle=False)
validation_generator = flow_from_dataframe(validation_datagen, df_validation,
path_col = 'filename',
y_col = 'label_vec',
target_size=(224, 224), batch_size=64, shuffle=False)
Now I am trying to provide my one-hot-encoded text vectors (i.e. [0,0,0,1,0,0]) to the model, which are also stored in a pandas dataframe.
Since my train_generator provides me with the image and label data, I am now looking for a solution to combine this generator with a generator which allows me to additionally feed the respective text-vector
You might want to consider writing your own generator (making use of Keras' Sequence object to allow for multiprocessing) instead of modifying the ImageDataGenerator code. From the Keras docs:
class CIFAR10Sequence(Sequence):
def __init__(self, x_set, y_set, batch_size):
self.x, self.y = x_set, y_set
self.batch_size = batch_size
def __len__(self):
return int(np.ceil(len(self.x) / float(self.batch_size)))
def __getitem__(self, idx):
batch_x = self.x[idx * self.batch_size:(idx + 1) * self.batch_size]
batch_y = self.y[idx * self.batch_size:(idx + 1) * self.batch_size]
return np.array([
resize(imread(file_name), (200, 200))
for file_name in batch_x]), np.array(batch_y)
You could have your labels, paths to the images, and paths to the text files in a single pandas dataframe and modify the __getitem__ method from above to have your generator yield all three of them simultaneously: one list of numpy arraysX which contains all the inputs, one numpy array Y which contains the outputs.
Related
I try to train GoogLeNet from scratch in Keras. I build the network architecture, and it is ready to train. Train GoogLeNet with auxiliaries outputs, the data generator should have three output labels. I write my custom data generator using tf.keras.utils.Sequence.
My custom generator is:
from skimage.transform import resize
from skimage.io import imread
import numpy as np
import math
from tensorflow.keras.utils import Sequence
class GoogLeNetDatasetGenerator(Sequence):
def __init__(self, X_train_path, y_train, batch_size):
"""
Initialize the GoogLeNet dataset generator.
:param X_train_path: Path of train images
:param y_train: Labels of train images
:param batch_size:
"""
self.X_train_path = X_train_path
self.y_train = y_train
self.batch_size = batch_size
self.indexes = np.arange(len(self.X_train_path))
np.random.shuffle(self.indexes)
def __len__(self):
"""
Denotes the number of batches per epoch
:return:
"""
return math.ceil(len(self.X_train_path) / self.batch_size)
def __getitem__(self, index):
"""
Get batch indexes from shuffled indexes
:param index:
:return:
"""
indexes = self.indexes[index * self.batch_size:(index + 1) * self.batch_size]
X_batch_names = [self.X_train_path[i] for i in indexes]
y_batch_naive = self.y_train[indexes]
X_batch = np.array([resize(imread(file_name), (224, 224) for file_name in X_batch_names],
dtype='float32')
y_batch = [y_batch_naive, y_batch_naive, y_batch_naive]
return X_batch, y_batch
def on_epoch_end(self):
"""
Updates indexes after each epoch
:return:
"""
self.indexes = np.arange(len(self.X_train_path))
np.random.shuffle(self.indexes)
Also, I compile and train the model with the following codes:
# Compile model
model.compile(loss=[CategoricalCrossentropy(), CategoricalCrossentropy(), CategoricalCrossentropy()],
loss_weights=[1, 0.3, 0.3], optimizer='adam',
metrics=['accuracy'])
# Train model
history = model.fit(train_dataset, validation_data=test_dataset, epochs=100)
While using the GPU version of TensorFlow, loading images in the data generator is time-consuming. It causes the training process slow. Is there any suggestion or other solutions for speeding up the loading data?
P.S.
I search the StackOverflow question such as this, but I did not find any idea.
I found another faster solution. You can use tf.data.Dataset. In the first step, I list all training images directory. Using the map method helped me read the image and properly configure the corresponding label. Here is my sample code to load an image with the ternary label.
image_filenames = tf.constant(image_list)
slices_dataset = tf.data.Dataset.from_tensor_slices(image_filenames)
slices_labels = tf.data.Dataset.from_tensor_slices(label_list)
image_dataset = slices_dataset.map(map_func=process_image)
label_dataset = slices_labels.map(map_func=process_label)
x_dataset = image_dataset.shuffle(buffer_size=Cfg.BUFFER_SIZE, seed=0).\
batch(batch_size=Cfg.BATCH_SIZE)
y_dataset = label_dataset.shuffle(buffer_size=Cfg.BUFFER_SIZE, seed=0).\
batch(batch_size=Cfg.BATCH_SIZE)
dataset = tf.data.Dataset.zip((x_dataset, y_dataset))
I am trying to use local binary data to train a network to perform regression inference.
Each local binary data has the following layout:
and the whole data consists of several *.bin files with the layout above. Each file has a variable number of sequences of 403*4 bytes. I was able to read one of those files using the following code:
import tensorflow as tf
RAW_N = 2 + 20*20 + 1
def convert_binary_to_float_array(register):
return tf.io.decode_raw(register, out_type=tf.float32)
raw_dataset = tf.data.FixedLengthRecordDataset(filenames=['mydata.bin'],record_bytes=RAW_N*4)
raw_dataset = raw_dataset.map(map_func=convert_binary_to_float_array)
Now, I need to create 4 datasets train_data, train_labels, test_data, test_labels as follows:
train_data, train_labels, test_data, test_labels = prepare_ds(raw_dataset, 0.8)
and use them to train & evaluate:
model = build_model()
history = model.fit(train_data, train_labels, ...)
loss, mse = model.evaluate(test_data, test_labels)
My question is: how to implement function prepare_ds(dataset, frac)?
def prepare_ds(dataset, frac):
...
I have tried to use tf.shape, tf.reshape, tf.slice, subscription [:] with no success. I realized that those functions doesn't work properly because after the map() call raw_dataset is a MapDataset (as a result of the eager execution concerns).
If the meta-data is suppose to be part of your inputs, which I am assuming, you could try something like this:
import random
import struct
import tensorflow as tf
import numpy as np
RAW_N = 2 + 20*20 + 1
bytess = random.sample(range(1, 5000), RAW_N*4)
with open('mydata.bin', 'wb') as f:
f.write(struct.pack('1612i', *bytess))
def decode_and_prepare(register):
register = tf.io.decode_raw(register, out_type=tf.float32)
inputs = register[:402]
label = register[402:]
return inputs, label
total_data_entries = 8
raw_dataset = tf.data.FixedLengthRecordDataset(filenames=['/content/mydata.bin', '/content/mydata.bin'], record_bytes=RAW_N*4)
raw_dataset = raw_dataset.map(decode_and_prepare)
raw_dataset = raw_dataset.shuffle(buffer_size=total_data_entries)
train_ds_size = int(0.8 * total_data_entries)
test_ds_size = int(0.2 * total_data_entries)
train_ds = raw_dataset.take(train_ds_size)
remaining_data = raw_dataset.skip(train_ds_size)
test_ds = remaining_data.take(test_ds_size)
Note that I am using the same bin file twice for demonstration purposes. After running that code snippet, you could feed the datasets to your model like this:
model = build_model()
history = model.fit(train_ds, ...)
loss, mse = model.evaluate(test_ds)
as each dataset contains the inputs and the corresponding labels.
I am working on Image Binarization using UNet and have a dataset of 150 images and their binarized versions too. My idea is to augment the images randomly to make them look like they are differentso I have made a function which inserts any of the 4-5 types of Noises, skewness, shearing and so on to an image. I could have easily used
ImageDataGenerator(preprocess_function=my_aug_function) to augment the images but the problem is that my y target is also an image. Also, I could have used something like:
train_dataset = (
train_dataset.map(
encode_single_sample, num_parallel_calls=tf.data.experimental.AUTOTUNE
)
.batch(batch_size)
.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
)
But it has 2 problems:
With larger dataset, it'll blow up the memory as data needs to be already in the memory
This is the crucial part that I need to augment the images on the go to make it look like I have a huge dataset.
Another Solution could be saving augmented images to a directory and making them 30-40K and then loading them. It would be silly thing to do.
Now the idea part is that I can use Sequence as the parent class but How can I keep on augmenting and generating new images on the fly with respective Y binarized images?
I have an idea as the below code. Can somebody help me with the augmentation and generation of y images. I have my X_DIR, Y_DIR where image names for binarised and original are same but stored in different directories.
class DataGenerator(tensorflow.keras.utils.Sequence):
def __init__(self, files_path, labels_path, batch_size=32, shuffle=True, random_state=42):
'Initialization'
self.files = files_path
self.labels = labels_path
self.batch_size = batch_size
self.shuffle = shuffle
self.random_state = random_state
self.on_epoch_end()
def on_epoch_end(self):
'Updates indexes after each epoch'
# Shuffle the data here
def __len__(self):
return int(np.floor(len(self.files) / self.batch_size))
def __getitem__(self, index):
# What do I do here?
def __data_generation(self, files):
# I think this is responsible for Augmentation but no idea how should I implement it and how does it works.
Custom Image Data Generator
load Directory data into dataframe for CustomDataGenerator
def data_to_df(data_dir, subset=None, validation_split=None):
df = pd.DataFrame()
filenames = []
labels = []
for dataset in os.listdir(data_dir):
img_list = os.listdir(os.path.join(data_dir, dataset))
label = name_to_idx[dataset]
for image in img_list:
filenames.append(os.path.join(data_dir, dataset, image))
labels.append(label)
df["filenames"] = filenames
df["labels"] = labels
if subset == "train":
split_indexes = int(len(df) * validation_split)
train_df = df[split_indexes:]
val_df = df[:split_indexes]
return train_df, val_df
return df
train_df, val_df = data_to_df(train_dir, subset="train", validation_split=0.2)
Custom Data Generator
import tensorflow as tf
from PIL import Image
import numpy as np
class CustomDataGenerator(tf.keras.utils.Sequence):
''' Custom DataGenerator to load img
Arguments:
data_frame = pandas data frame in filenames and labels format
batch_size = divide data in batches
shuffle = shuffle data before loading
img_shape = image shape in (h, w, d) format
augmentation = data augmentation to make model rebust to overfitting
Output:
Img: numpy array of image
label : output label for image
'''
def __init__(self, data_frame, batch_size=10, img_shape=None, augmentation=True, num_classes=None):
self.data_frame = data_frame
self.train_len = len(data_frame)
self.batch_size = batch_size
self.img_shape = img_shape
self.num_classes = num_classes
print(f"Found {self.data_frame.shape[0]} images belonging to {self.num_classes} classes")
def __len__(self):
''' return total number of batches '''
self.data_frame = shuffle(self.data_frame)
return math.ceil(self.train_len/self.batch_size)
def on_epoch_end(self):
''' shuffle data after every epoch '''
# fix on epoch end it's not working, adding shuffle in len for alternative
pass
def __data_augmentation(self, img):
''' function for apply some data augmentation '''
img = tf.keras.preprocessing.image.random_shift(img, 0.2, 0.3)
img = tf.image.random_flip_left_right(img)
img = tf.image.random_flip_up_down(img)
return img
def __get_image(self, file_id):
""" open image with file_id path and apply data augmentation """
img = np.asarray(Image.open(file_id))
img = np.resize(img, self.img_shape)
img = self.__data_augmentation(img)
img = preprocess_input(img)
return img
def __get_label(self, label_id):
""" uncomment the below line to convert label into categorical format """
#label_id = tf.keras.utils.to_categorical(label_id, num_classes)
return label_id
def __getitem__(self, idx):
batch_x = self.data_frame["filenames"][idx * self.batch_size:(idx + 1) * self.batch_size]
batch_y = self.data_frame["labels"][idx * self.batch_size:(idx + 1) * self.batch_size]
# read your data here using the batch lists, batch_x and batch_y
x = [self.__get_image(file_id) for file_id in batch_x]
y = [self.__get_label(label_id) for label_id in batch_y]
return tf.convert_to_tensor(x), tf.convert_to_tensor(y)
You can use libraries like albumentations and imgaug, both are good but I have heard there are issues with random seed with albumentations.
Here's an example of imgaug taken from the documentation here:
seq = iaa.Sequential([
iaa.Dropout([0.05, 0.2]), # drop 5% or 20% of all pixels
iaa.Sharpen((0.0, 1.0)), # sharpen the image
iaa.Affine(rotate=(-45, 45)), # rotate by -45 to 45 degrees (affects segmaps)
iaa.ElasticTransformation(alpha=50, sigma=5) # apply water effect (affects segmaps)
], random_order=True)
# Augment images and segmaps.
images_aug = []
segmaps_aug = []
for _ in range(len(input_data)):
images_aug_i, segmaps_aug_i = seq(image=image, segmentation_maps=segmap)
images_aug.append(images_aug_i)
segmaps_aug.append(segmaps_aug_i)
You are going in the right way with the custom generator. In __getitem__, make a batch using batch_x = self.files[index:index+batch_size] and same with batch_y, then augment them using X,y = __data_generation(batch_x, batch_y) which will load images(using any library you like, I prefer opencv), and return the augmented pairs (and any other manipulation).
Your __getitem__ will then return the tuple (X,y)
You can use ImageDataGenerator even if your label is an image.
Here is a simple example of how you can do that:
Code:
# Specifying your data augmentation here for both image and label
image_datagen = tf.keras.preprocessing.image.ImageDataGenerator()
mask_datagen = tf.keras.preprocessing.image.ImageDataGenerator()
# Provide the same seed and keyword arguments to the flow methods
seed = 1
image_generator = image_datagen.flow_from_directory(
data_dir,
class_mode=None,
seed=seed)
mask_generator = mask_datagen.flow_from_directory(
data_dir,
class_mode=None,
seed=seed)
# Combine the image and label generator.
train_generator = zip(image_generator, mask_generator)
Now, if you iterate over it you will get:
for image, label in train_generator:
print(image.shape,label.shape)
break
Output:
(32, 256, 256, 3) (32, 256, 256, 3)
You can use this train_generator with fit() command.
Code:
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=50)
With flow_from_directory your memory won't be cluttered and Imagedatagenerator will take care of the augmentation part.
I have transformed an image database into two TFRecords, one for training and the other for validation. I want to train a simple model with keras using these two files for data input but I obtain an error I can't understand related to the shape of the data.
Here is the code (all-capital variables are defined elsewhere):
def _parse_function(proto):
f = {
"x": tf.FixedLenSequenceFeature([IMG_SIZE[0] * IMG_SIZE[1]], tf.float32, default_value=0., allow_missing=True),
"label": tf.FixedLenSequenceFeature([1], tf.int64, default_value=0, allow_missing=True)
}
parsed_features = tf.parse_single_example(proto, f)
x = tf.reshape(parsed_features['x'] / 255, (IMG_SIZE[0], IMG_SIZE[1], 1))
y = tf.cast(parsed_features['label'], tf.float32)
return x, y
def load_dataset(input_path, batch_size, shuffle_buffer):
dataset = tf.data.TFRecordDataset(input_path)
dataset = dataset.shuffle(shuffle_buffer).repeat() # shuffle and repeat
dataset = dataset.map(_parse_function, num_parallel_calls=16)
dataset = dataset.batch(batch_size).prefetch(1) # batch and prefetch
return dataset.make_one_shot_iterator()
train_iterator = load_dataset(TRAIN_TFRECORDS, BATCH_SIZE, SHUFFLE_BUFFER)
val_iterator = load_dataset(VALIDATION_TFRECORDS, BATCH_SIZE, SHUFFLE_BUFFER)
model = tf.keras.Sequential()
model.add(tf.keras.layers.Flatten(input_shape=(IMG_SIZE[0], IMG_SIZE[1], 1)))
model.add(tf.keras.layers.Dense(1, 'sigmoid'))
model.compile(
optimizer=tf.train.AdamOptimizer(),
loss='binary_crossentropy',
metrics=['accuracy']
)
model.fit(
train_iterator,
epochs=N_EPOCHS,
steps_per_epoch=N_TRAIN // BATCH_SIZE,
validation_data=val_iterator,
validation_steps=N_VALIDATION // BATCH_SIZE
)
And here is the error I obtain:
tensorflow.python.framework.errors_impl.InvalidArgumentError: data[0].shape = [3] does not start with indices[0].shape = [2]
[[Node: training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/DynamicStitch = DynamicStitch[N=2, T=DT_INT32, _class=["loc:#training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/floordiv"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/range, training/TFOptimizer/gradients/loss/dense_loss/Mean_3_grad/Maximum, training/TFOptimizer/gradients/loss/dense_loss/Mean_grad/Shape/_35, training/TFOptimizer/gradients/loss/dense_loss/Mean_3_grad/Maximum/_41)]]
(I know that the model defined here is not a good model for image analysis, I just took the simplest possible architecture that reproduces the error)
Change:
"label": tf.FixedLenSequenceFeature([1]...
into:
"label": tf.FixedLenSequenceFeature([]...
This is unfortunately not explained in the documentation on the website, but some explanation can be found in the docstring of FixedLenSequenceFeature on github. Basically, if your data consists of a single dimension (+ a batch dimension), you don't need to specify it.
You have forget to this line from the example:
parsed_features = tf.parse_single_example(proto, f)
Add it to _parse_function.
Also, you can return just the dataset object. Keras supports iterators as well as instances of the tf.data.Dataset. Also, it looks a bit weird to shuffle and repeat first, and then to parse tfexamples. Here is an example code that works for me:
def dataset(filenames, batch_size, img_height, img_width, is_training=False):
decoder = TfExampleDecoder()
def preprocess(image, boxes, classes):
image = preprocess_image(image, resize_height=img_height, resize_width=img_width)
return image, groundtruth
ds = tf.data.TFRecordDataset(filenames)
ds = ds.map(decoder.decode, num_parallel_calls=8)
if is_training:
ds = ds.shuffle(1000 + 3 * batch_size)
ds = ds.apply(tf.contrib.data.map_and_batch(map_func=preprocess, batch_size=batch_size, num_parallel_calls=8))
ds = ds.repeat()
ds = ds.prefetch(buffer_size=batch_size)
return ds
train_dataset = dataset(args.train_data, args.batch_size,
args.img_height, args.img_width,
is_training=True)
model.fit(train_dataset,
steps_per_epoch=args.steps_per_epoch,
epochs=args.max_epochs,
callbacks=callbacks,
initial_epoch=0)
It seems like an issue with your data or preprocessing pipeline, rather than with Keras. Try to inspect what you are getting out of the dataset with a debugging code like:
ds = dataset(args.data, args.img_height, args.img_width, is_training=True)
image_t, classes_t = ds.make_one_shot_iterator().get_next()
with tf.Session() as sess:
while True:
image, classes = sess.run([image_t, classes_t])
# Do something with the data: display, log etc.
Following is my csv file
file,pt1,pt2,pt3,,pt4,pt5,pt6
object/obj0.png,66.0335639098,39.0022736842,30.2270075188,36.4216781955,59.582075188,39.6474225564
object/obj0.png,66.0335639098,39.0022736842,30.2270075188,36.4216781955,59.582075188,39.6474225564
object/obj0.png,66.0335639098,39.0022736842,30.2270075188,36.4216781955,59.582075188,39.6474225564
How do i load those images and the annotations to train my simple cnn?
I tried using 'ImagedataGenerator' as follows but it didnt help...is there any other alternative?
train_datagen = ImageDataGenerator(
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True)
The ImageDataGenerator object allows to yield data either from numpy arrays or directly from directories. In the latter case, the labels are automatically inferred from the folder structure of your data: each class of images should live in a separate folder. Whenever the label structure is more complex, as in your case, you can opt to write you own custom generator. If you do so, make use of Keras' Sequence object, which allows for safe multiprocessing. The Keras website contains a boilerplate example. In your case, your code would look something like this:
from keras.utils import Sequence
from keras.preprocessing.image import load_img
import pandas as pd
import random
class DataSequence(Sequence):
def __init__(self, csv_path, batch_size, mode='train'):
self.df = pd.read_csv(csv_path) # read your csv file with pandas
self.bsz = batch_size # batch size
self.mode = mode # shuffle when in train mode
# Take labels and a list of image locations in memory
self.labels = self.df[['pt1', 'pt2', 'pt3', 'pt4', 'pt5', 'pt6']].values
self.im_list = self.df['file'].tolist()
def __len__(self):
# compute number of batches to yield
return int(math.ceil(len(self.df) / float(self.bsz)))
def on_epoch_end(self):
# Shuffles indexes after each epoch if in training mode
self.indexes = range(len(self.im_list))
if self.mode == 'train':
self.indexes = random.sample(self.indexes, k=len(self.indexes))
def get_batch_labels(self, idx):
# Fetch a batch of labels
return self.labels[idx * self.bsz: (idx + 1) * self.bsz,:]
def get_batch_features(self, idx):
# Fetch a batch of inputs
return np.array([load_img(im) for im in self.im_list[idx * self.bsz: (1 + idx) * self.bsz]])
def __getitem__(self, idx):
batch_x = self.get_batch_features(idx)
batch_y = self.get_batch_labels(idx)
return batch_x, batch_y
You can use this Sequence object to train your model with model.fit_generator():
sequence = DataSequence('./path_to/csv_file.csv', batch_size)
model.fit_generator(sequence, epochs=1, use_multiprocessing=True)
See also this related question.