We Keep Coding

How to run image classification on multiple images?

I have worked through the image classification tutorial on the tensorflow website here
The tutorial explains how the trained model can be run as a predictor on a new image.
Is there a way to run this on a batch/multiple images? The code is as follows:
sunflower_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/592px-Red_sunflower.jpg"
sunflower_path = tf.keras.utils.get_file('Red_sunflower', origin=sunflower_url)
img = tf.keras.utils.load_img(
sunflower_path, target_size=(img_height, img_width)
)
img_array = tf.keras.utils.img_to_array(img)
img_array = tf.expand_dims(img_array, 0) # Create a batch
predictions = model.predict(img_array)
score = tf.nn.softmax(predictions[0])
print(
"This image most likely belongs to {} with a {:.2f} percent confidence."
.format(class_names[np.argmax(score)], 100 * np.max(score))
)

You can use this code to run your prediction on multiple images at the same time.
import cv2 (pip install opencv-python)
batch_images = [cv2.imread(img) for img in list_images]
predictions = model.predict_on_batch(batch_images)
list_images is a list with paths to your images you want to predict on for example ["path_img1","path_img2",...]
predictions is a list of predictions that your model made on the given batch of images and they are in the same order as the images you used as an input.
so predictions[x] gives you the predictions for the x'th images of the input batch.

Here's a solution that uses Dataset.from_generator to stream images from disk with a generator function.
def read_dir():
files = os.listdir(source_folder)
for file_name in files:
yield keras.utils.load_img(source_folder + file_name, color_mode="rgb")
ds = tf.data.Dataset.from_generator(
lambda: read_dir(),
output_types=(tf.int8),
output_shapes=([128, 128, 3])
)
model = keras.models.load_model('my-model.keras')
predictions = model.predict(ds.batch(64))

tfrecords: encoding images results in distorted images

I am trying to make a tfrecords from image_dataset_from_directory; but when I try to visualize images to check if the encoding was correct, the images turn out to be distorted of some kind.
How I created the tfrecord:
Step 1: create dataset using image_dataset_from_directory
data_dir = 'path to JPG dataset'
load_split = partial(
tf.keras.preprocessing.image_dataset_from_directory,
data_dir,
validation_split=0.2,
shuffle=True,
seed=123,
image_size=(IMG_HEIGHT, IMG_WIDTH),
batch_size=1,
)
ds_train = load_split(subset='training')
ds_valid = load_split(subset='validation')
Step 2: encoding functions
def process_image(image, label):
image = tf.image.convert_image_dtype(image, dtype=tf.uint8)
image = tf.io.encode_jpeg(image)
label = tf.one_hot(label, NUM_CLASSES)
return image, label
def make_example(encoded_image, label):
image_feature = Feature(
bytes_list=BytesList(value=[
encoded_image,
]),
)
label_feature = Feature(
float_list=FloatList(value=label)
)
features = Features(feature={
'image': image_feature,
'label': label_feature,
})
example = Example(features=features)
return example.SerializeToString()
Step 3: encoding and creating tfrecord
ds_train_encoded = (
ds_train
.unbatch()
.map(process_image)
)
ds_valid_encoded = (
ds_valid
.unbatch()
.map(process_image)
)
ds_train_encoded_iter = (
ds_train_encoded
.as_numpy_iterator()
)
with tf.io.TFRecordWriter(path='train.tfrecord') as f: # you can pass gs:// path here :)
for encoded_image, label in ds_train_encoded_iter:
example = make_example(encoded_image, label)
f.write(example)
ds_valid_encoded_iter = (
ds_valid_encoded
.as_numpy_iterator()
)
with tf.io.TFRecordWriter(path='/home/et/medai/images/tfrecords/test.tfrecord') as f:
for encoded_image, label in ds_valid_encoded_iter:
example = make_example(encoded_image, label)
f.write(example)
How I tried to visualize the images in the tfrecords
Step 1: decoding functions
def _parse_image_function(example):
image_feature_description = {
'image': tf.io.FixedLenFeature([], tf.string),
'label': tf.io.FixedLenFeature([40], tf.float32),
}
features = tf.io.parse_single_example(example, image_feature_description)
image = tf.image.decode_jpeg(features['image'], channels=3)
image = tf.image.resize(image, [IMG_SIZE, IMG_SIZE])
# image = features['image']
label = features['label']
return image, label
def read_dataset(filename, batch_size):
dataset = tf.data.TFRecordDataset(filename)
dataset = dataset.map(_parse_image_function, num_parallel_calls=tf.data.experimental.AUTOTUNE)
dataset = dataset.shuffle(500)
dataset = dataset.batch(batch_size, drop_remainder=True)
# dataset = dataset.repeat()
dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
return dataset
Step 2: decode and display
x = read_dataset('/home/et/medai/images/tfrecords/tests_train.tfrecord', 32)
plt.figure(figsize=(10, 10))
batch_size = 32
for images, labels in x.take(1):
for i in range(batch_size):
# display.display(display.Image(data=images[i].numpy()))
ax = plt.subplot(6, 6, i + 1)
plt.imshow(images[i].numpy().astype("uint8"))
plt.axis("off")
The result is something distorted:
https://i.stack.imgur.com/tCAik.jpg
I am not quite sure where this distortion comes from. Original images look like this:
https://i.stack.imgur.com/Zi4HG.png
Any ideas?

I had a similar issue with you. I fixed my problem with image normalization in image processing (for your case, in process_image).
When you use 0~255 as pixel data, it tends to break up while manipulating image data, such as changing into byte and resizing because these manipulations round its pixel value.
So, I want you to try normalizing your image pixel data into 0. to 1. of float values.
I used OpenCV to fix this, and I hope you can figure out your problem in a similar way as did refer to the code I posted below.
# This line distorted my images.
img = cv2.normalize(img, None, alpha=0, beta=255, norm_type=cv2.NORM_MINMAX)
# I changed to this line, and it worked.
img = cv2.normalize(img, None, alpha=0., beta=1., norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)

When I faced that issue, I used:
image = open(content, 'rb').read()
instead of:
image = tf.image.convert_image_dtype(image, dtype=tf.uint8)
image = tf.io.encode_jpeg(image)

How to build a Custom Data Generator for Keras/tf.Keras where X images are being augmented and corresponding Y labels are also images

I am working on Image Binarization using UNet and have a dataset of 150 images and their binarized versions too. My idea is to augment the images randomly to make them look like they are differentso I have made a function which inserts any of the 4-5 types of Noises, skewness, shearing and so on to an image. I could have easily used
ImageDataGenerator(preprocess_function=my_aug_function) to augment the images but the problem is that my y target is also an image. Also, I could have used something like:
train_dataset = (
train_dataset.map(
encode_single_sample, num_parallel_calls=tf.data.experimental.AUTOTUNE
)
.batch(batch_size)
.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
)
But it has 2 problems:
With larger dataset, it'll blow up the memory as data needs to be already in the memory
This is the crucial part that I need to augment the images on the go to make it look like I have a huge dataset.
Another Solution could be saving augmented images to a directory and making them 30-40K and then loading them. It would be silly thing to do.
Now the idea part is that I can use Sequence as the parent class but How can I keep on augmenting and generating new images on the fly with respective Y binarized images?
I have an idea as the below code. Can somebody help me with the augmentation and generation of y images. I have my X_DIR, Y_DIR where image names for binarised and original are same but stored in different directories.
class DataGenerator(tensorflow.keras.utils.Sequence):
def __init__(self, files_path, labels_path, batch_size=32, shuffle=True, random_state=42):
'Initialization'
self.files = files_path
self.labels = labels_path
self.batch_size = batch_size
self.shuffle = shuffle
self.random_state = random_state
self.on_epoch_end()
def on_epoch_end(self):
'Updates indexes after each epoch'
# Shuffle the data here
def __len__(self):
return int(np.floor(len(self.files) / self.batch_size))
def __getitem__(self, index):
# What do I do here?
def __data_generation(self, files):
# I think this is responsible for Augmentation but no idea how should I implement it and how does it works.

Custom Image Data Generator
load Directory data into dataframe for CustomDataGenerator
def data_to_df(data_dir, subset=None, validation_split=None):
df = pd.DataFrame()
filenames = []
labels = []
for dataset in os.listdir(data_dir):
img_list = os.listdir(os.path.join(data_dir, dataset))
label = name_to_idx[dataset]
for image in img_list:
filenames.append(os.path.join(data_dir, dataset, image))
labels.append(label)
df["filenames"] = filenames
df["labels"] = labels
if subset == "train":
split_indexes = int(len(df) * validation_split)
train_df = df[split_indexes:]
val_df = df[:split_indexes]
return train_df, val_df
return df
train_df, val_df = data_to_df(train_dir, subset="train", validation_split=0.2)
Custom Data Generator
import tensorflow as tf
from PIL import Image
import numpy as np
class CustomDataGenerator(tf.keras.utils.Sequence):
''' Custom DataGenerator to load img
Arguments:
data_frame = pandas data frame in filenames and labels format
batch_size = divide data in batches
shuffle = shuffle data before loading
img_shape = image shape in (h, w, d) format
augmentation = data augmentation to make model rebust to overfitting
Output:
Img: numpy array of image
label : output label for image
'''
def __init__(self, data_frame, batch_size=10, img_shape=None, augmentation=True, num_classes=None):
self.data_frame = data_frame
self.train_len = len(data_frame)
self.batch_size = batch_size
self.img_shape = img_shape
self.num_classes = num_classes
print(f"Found {self.data_frame.shape[0]} images belonging to {self.num_classes} classes")
def __len__(self):
''' return total number of batches '''
self.data_frame = shuffle(self.data_frame)
return math.ceil(self.train_len/self.batch_size)
def on_epoch_end(self):
''' shuffle data after every epoch '''
# fix on epoch end it's not working, adding shuffle in len for alternative
pass
def __data_augmentation(self, img):
''' function for apply some data augmentation '''
img = tf.keras.preprocessing.image.random_shift(img, 0.2, 0.3)
img = tf.image.random_flip_left_right(img)
img = tf.image.random_flip_up_down(img)
return img
def __get_image(self, file_id):
""" open image with file_id path and apply data augmentation """
img = np.asarray(Image.open(file_id))
img = np.resize(img, self.img_shape)
img = self.__data_augmentation(img)
img = preprocess_input(img)
return img
def __get_label(self, label_id):
""" uncomment the below line to convert label into categorical format """
#label_id = tf.keras.utils.to_categorical(label_id, num_classes)
return label_id
def __getitem__(self, idx):
batch_x = self.data_frame["filenames"][idx * self.batch_size:(idx + 1) * self.batch_size]
batch_y = self.data_frame["labels"][idx * self.batch_size:(idx + 1) * self.batch_size]
# read your data here using the batch lists, batch_x and batch_y
x = [self.__get_image(file_id) for file_id in batch_x]
y = [self.__get_label(label_id) for label_id in batch_y]
return tf.convert_to_tensor(x), tf.convert_to_tensor(y)

You can use libraries like albumentations and imgaug, both are good but I have heard there are issues with random seed with albumentations.
Here's an example of imgaug taken from the documentation here:
seq = iaa.Sequential([
iaa.Dropout([0.05, 0.2]), # drop 5% or 20% of all pixels
iaa.Sharpen((0.0, 1.0)), # sharpen the image
iaa.Affine(rotate=(-45, 45)), # rotate by -45 to 45 degrees (affects segmaps)
iaa.ElasticTransformation(alpha=50, sigma=5) # apply water effect (affects segmaps)
], random_order=True)
# Augment images and segmaps.
images_aug = []
segmaps_aug = []
for _ in range(len(input_data)):
images_aug_i, segmaps_aug_i = seq(image=image, segmentation_maps=segmap)
images_aug.append(images_aug_i)
segmaps_aug.append(segmaps_aug_i)
You are going in the right way with the custom generator. In __getitem__, make a batch using batch_x = self.files[index:index+batch_size] and same with batch_y, then augment them using X,y = __data_generation(batch_x, batch_y) which will load images(using any library you like, I prefer opencv), and return the augmented pairs (and any other manipulation).
Your __getitem__ will then return the tuple (X,y)

You can use ImageDataGenerator even if your label is an image.
Here is a simple example of how you can do that:
Code:
# Specifying your data augmentation here for both image and label
image_datagen = tf.keras.preprocessing.image.ImageDataGenerator()
mask_datagen = tf.keras.preprocessing.image.ImageDataGenerator()
# Provide the same seed and keyword arguments to the flow methods
seed = 1
image_generator = image_datagen.flow_from_directory(
data_dir,
class_mode=None,
seed=seed)
mask_generator = mask_datagen.flow_from_directory(
data_dir,
class_mode=None,
seed=seed)
# Combine the image and label generator.
train_generator = zip(image_generator, mask_generator)
Now, if you iterate over it you will get:
for image, label in train_generator:
print(image.shape,label.shape)
break
Output:
(32, 256, 256, 3) (32, 256, 256, 3)
You can use this train_generator with fit() command.
Code:
model.fit_generator(
train_generator,
steps_per_epoch=2000,
epochs=50)
With flow_from_directory your memory won't be cluttered and Imagedatagenerator will take care of the augmentation part.

Custom Datagenerator keras model expected 2 arrays but receives 1

So I am trying to get this custom generator working right but seem to have issues with it. It seems like the generator is working fine as I tried with using the gen.next() and it is producing what I want it to. However it may not be making it in the same shape that I think it should.
# Image processing
def preprocess_image(image_path):
img = image.load_img(image_path, target_size=(224, 224))
img = image.img_to_array(img)
img = preprocess_input(img)
return img
def image_generator(data, batch_size):
datagen_args = dict(horizontal_flip=True)
datagen = ImageDataGenerator(**datagen_args)
while True:
for i in range(0, len(data) // batch_size):
# get the label and the imagepath
imgpath, label = data[i]
# Process the image
img = preprocess_image(imgpath)
img = datagen.random_transform(img)
#img = np.expand_dims(img, axis=0)
# add a 0 for a dummy variable
dummy_label = np.zeros(len(label))
x_data = np.array([img, label])
yield x_data, dummy_label
# Prepare data need a array [image, label]
X = [] # hold the data before processing
Y = []
IMAGE_DIR = 'dataset/gt_bbox'
for file in os.listdir(IMAGE_DIR):
file_path = os.path.join(IMAGE_DIR, file)
label = int(file.split('_')[0])
X.append(file_path)
Y.append(label)
# Convert to catigorical
Y = to_categorical(Y)
image_dataset = []
for i in range(0,len(X)):
image_dataset.append([X[i], Y[i]])
# Split to train test data
train, val = train_test_split(image_dataset)
BATCHSIZE = 32
imggen = image_generator(train, BATCHSIZE)
valgen = image_generator(val, BATCHSIZE)
model.fit_generator(imggen,
steps_per_epoch=1000,
epochs=10,
validation_data=valgen,
validation_steps=300,
verbose=1)
My model is set up like this
input_images = Input(shape=(224, 224, 3), name='input_image') # input layer for images
input_labels = Input(shape=(1,), name='input_label') # input layer for labels
embeddings = base_network([input_images]) # output of network -> embeddings
labels_plus_embeddings = Concatenate(axis=-1)([input_labels, embeddings]) # concatenating the labels + embeddings
model = Model(inputs=[input_images, input_labels], outputs=labels_plus_embeddings)
I may be wrong in how I am building the model but it seems right to me.
Error Message
ValueError: Error when checking model input: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays: [array([[array([[[-0.56078434, -0.52156866, -0.4980392 ],
[-0.56078434, -0.52156866, -0.4980392 ],
[-0.56078434, -0.52156866, -0.4980392 ],
...,
[-0.5764706 , -0.545098...

Create tensorflow dataset from image local directory

I have a very huge database of images locally, with the data distribution like each folder cointains the images of one class.
I would like to use the tensorflow dataset API to obtain batches de data without having all the images loaded in memory.
I have tried something like this:
def _parse_function(filename, label):
image_string = tf.read_file(filename, "file_reader")
image_decoded = tf.image.decode_jpeg(image_string, channels=3)
image = tf.cast(image_decoded, tf.float32)
return image, label
image_list, label_list, label_map_dict = read_data()
dataset = tf.data.Dataset.from_tensor_slices((tf.constant(image_list), tf.constant(label_list)))
dataset = dataset.shuffle(len(image_list))
dataset = dataset.repeat(epochs).batch(batch_size)
dataset = dataset.map(_parse_function)
iterator = dataset.make_one_shot_iterator()
image_list is a list where the path (and name) of the images have been appended and label_list is a list where the class of each image has been appended in the same order.
But the _parse_function does not work, the error that I recibe is:
ValueError: Shape must be rank 0 but is rank 1 for 'file_reader' (op: 'ReadFile') with input shapes: [?].
I have googled the error, but nothing works for me.
If I do not use the map function, I just recibe the path of the images (which are store in image_list), so I think that I need the map function to read the images, but I am not able to make it works.
Thank you in advance.
EDIT:
def read_data():
image_list = []
label_list = []
label_map_dict = {}
count_label = 0
for class_name in os.listdir(base_path):
class_path = os.path.join(base_path, class_name)
label_map_dict[class_name]=count_label
for image_name in os.listdir(class_path):
image_path = os.path.join(class_path, image_name)
label_list.append(count_label)
image_list.append(image_path)
count_label += 1

The error is in this line dataset = dataset.repeat(epochs).batch(batch_size) Your pipeline adds batchsize as a dimension to input.
You need to batch your dataset after map function like this
dataset = tf.data.Dataset.from_tensor_slices((tf.constant(image_list), tf.constant(label_list)))
dataset = dataset.shuffle(len(image_list))
dataset = dataset.repeat(epochs)
dataset = dataset.map(_parse_function).batch(batch_size)