We Keep Coding

Evaluate U-Net by layer

I am coming from medical background and a newbie in this machine learning field. I am trying to train my U-Net model using keras and tensorflow for image segmentation. However, my loss value is all NaN and the prediction is all black.
I would like to check the U-Net layer by layer but I don't know how to feed the data and from where to start. What I meant by checking for each layer is that I want to feed my images to first layer for example and see the output from the first layer and then moving on to the second layer and until to the last layer. Just want to see how the output is produced for each layer and to check from where the nan value is started. Really appreciate for your help.
These are my codes.
import os
import matplotlib.pyplot as plt
import tensorflow as tf
from keras_preprocessing.image
import ImageDataGenerator
from tensorflow import keras
#Constants
SEED = 42
BATCH_SIZE_TRAIN = 16
BATCH_SIZE_TEST = 16
IMAGE_HEIGHT = 512
IMAGE_WIDTH = 512
IMG_SIZE = (IMAGE_HEIGHT, IMAGE_WIDTH)
data_dir = 'data'
data_dir_train = os.path.join(data_dir, 'training')
data_dir_train_image = os.path.join(data_dir_train, 'img')
data_dir_train_mask = os.path.join(data_dir_train, 'mask')
data_dir_test = os.path.join(data_dir, 'test')
data_dir_test_image = os.path.join(data_dir_test, 'img')
data_dir_test_mask = os.path.join(data_dir_test, 'mask')
NUM_TRAIN = 1413
NUM_TEST = 210
NUM_OF_EPOCHS = 10
def create_segmentation_generator_train(img_path, mask_path, BATCH_SIZE):
data_gen_args = dict(rescale=1./255)
img_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(*data_gen_args)
img_generator = img_datagen.flow_from_directory(img_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED)
mask_generator = mask_datagen.flow_from_directory(mask_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED)
return zip(img_generator, mask_generator)
def create_segmentation_generator_test(img_path, mask_path, BATCH_SIZE):
data_gen_args = dict(rescale=1./255)
img_datagen = ImageDataGenerator(**data_gen_args)
mask_datagen = ImageDataGenerator(*data_gen_args)
img_generator = img_datagen.flow_from_directory(img_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED)
mask_generator = mask_datagen.flow_from_directory(mask_path, target_size=IMG_SIZE, class_mode=None, color_mode='grayscale', batch_size=BATCH_SIZE, seed=SEED)
return zip(img_generator, mask_generator)
def display(display_list):
plt.figure(figsize=(15,15))
title = ['Input Image', 'True Mask', 'Predicted Mask']
for i in range(len(display_list)):
plt.subplot(1, len(display_list), i+1)
plt.title(title[i])
plt.imshow(tf.keras.preprocessing.image.array_to_img(display_list[i]), cmap='gray')
plt.show()
def show_dataset(datagen, num=1):
for i in range(0,num):
image,mask = next(datagen)
display([image[0], mask[0]])
def unet(n_levels, initial_features=32, n_blocks=2, kernel_size=3, pooling_size=2, in_channels=1, out_channels=1):
#n_blocks = how many conv in each level
inputs = keras.layers.Input(shape=(IMAGE_HEIGHT, IMAGE_WIDTH, in_channels))
x = inputs
convpars = dict(kernel_size=kernel_size, activation='relu', padding='same')
#downstream
skips = {}
for level in range(n_levels):
for _ in range (n_blocks):
x = keras.layers.Conv2D(initial_features * 2 ** level, **convpars)(x)
if level < n_levels - 1:
skips[level] = x
x = keras.layers.MaxPool2D(pooling_size)(x)
#upstream
for level in reversed(range(n_levels-1)):
x = keras.layers.Conv2DTranspose(initial_features * 2 ** level, strides=pooling_size, **convpars)(x)
x = keras.layers.Concatenate()([x, skips[level]])
for _ in range (n_blocks):
x = keras.layers.Conv2D(initial_features * 2 ** level, **convpars)(x)
#output
activation = 'sigmoid' if out_channels == 1 else 'softmax'
x = keras.layers.Conv2D(out_channels, kernel_size=1, activation='sigmoid', padding='same')(x)
return keras.Model(inputs=[inputs], outputs=[x], name=f'UNET-L{n_levels}-F{initial_features}')
EPOCH_STEP_TRAIN = NUM_TRAIN // BATCH_SIZE_TRAIN
EPOCH_STEP_TEST = NUM_TEST // BATCH_SIZE_TRAIN
model = unet(4)
model.compile(optimizer="adam", loss='binary_crossentropy', metrics=['accuracy'])
model.fit_generator(generator=train_generator, steps_per_epoch=EPOCH_STEP_TRAIN, validation_data=test_generator, validation_steps=EPOCH_STEP_TEST, epochs=NUM_OF_EPOCHS)
def show_prediction(datagen, num=1):
for i in range(0,num):
image,mask = next(datagen)
pred_mask = model.predict(image)[0] > 0.5
display([image[0], mask[0], pred_mask])
show_prediction(test_generator, 2)

To investigate your model layer-by-layer please see example how to show summary of the model and also how to save the model:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
#luodaan input
inputs=keras.Input(shape=(1,))
#luodaan kerros
dense=layers.Dense(64,activation="relu")
x=dense(inputs)
x=layers.Dense(64,activation="relu")(x)
outputs=layers.Dense(10)(x)
#Koostetaa
model=keras.Model(inputs=inputs,outputs=outputs,name="Spesiaali")
#Tarkastellaan
model.summary()
#Tallennellaan
model.save(".\model_to_be_investigated_by_someone_else_to_help_you")
...this makes it possible for you to see the whole model structure for "debugging your AI". If you do not find the solution itself, then add the last row of example to your own code, and then put the resulting folder e.g. to github and ask someone other to see the structure of your model to help you in solving the problem.

The blue drawing illustrates the output of command model.summary() and the red line illustrates the output shape of the first dense layer.

How to train imbalanced data with Keras?

I am trying to experiment on ISIC 2019 data as a newbie. Firstly, I downloaded the training data and divided the data into 3 parts as train, test, and validation data, and every dataset folder contains 2 subfolders which are benign and malignant. In short, I just moved all the categories into benign folders except the melenoma category and melanoma images are inside malignant folders. After the division, I get imbalanced data. In the training dataset for benign data, I get 16596 images and for malignant data, I get 3629 images. I tried to train my data and I couldn't get a good result for malignant and my precision value was about 0.18 for malignant. I used ResNet50 to train my model and I would like to ask how can I train my model without data augmentation and oversampling? I am also trying decayed learning metrics at the moment and it seems it won't give a good result too.
import os
import tensorflow as tf
import math
import numpy as np
import matplotlib.pyplot as plt
from tensorflow import keras
from keras.applications.resnet50 import ResNet50, preprocess_input
from keras.layers import Dense, GlobalMaxPooling2D
from keras.models import Model
from keras.optimizers import Adam
from sklearn.metrics import roc_curve
from tensorflow.keras.preprocessing.image import ImageDataGenerator
train_examples = 20225
test_examples = 2551
validation_examples = 2555
img_height = img_width = 224
channel = 3
batch_size = 32
base_model = ResNet50(weights = 'imagenet' , include_top = False, input_shape = (img_height, img_width, channel))
x = base_model.output
x = GlobalMaxPooling2D()(x)
x = Dense(1, activation= 'sigmoid')(x)
model = Model(
inputs = base_model.input,
outputs = x)
model.summary()
train_datagen = ImageDataGenerator(
rotation_range = 20,
width_shift_range=0.10,
height_shift_range=0.10,
zoom_range = 0.10,
horizontal_flip = True,
preprocessing_function = preprocess_input,
fill_mode='nearest'
)
validation_datagen = ImageDataGenerator(
preprocessing_function = preprocess_input,
)
test_datagen = ImageDataGenerator(
preprocessing_function = preprocess_input,
)
train_gen = train_datagen.flow_from_directory(
"dataset/train/",
target_size = (img_height, img_width),
batch_size = batch_size,
color_mode = "rgb",
class_mode = "binary",
shuffle = True,
seed = 123,
)
validation_gen = validation_datagen.flow_from_directory(
"dataset/validation/",
target_size = (img_height, img_width),
batch_size = batch_size,
color_mode = "rgb",
class_mode = "binary",
shuffle = True,
seed = 123,
)
test_gen = test_datagen.flow_from_directory(
"dataset/test/",
target_size =(img_height, img_width),
batch_size = batch_size,
color_mode = "rgb",
class_mode = "binary",
shuffle = True,
seed = 123,
)
METRICS = [
keras.metrics.Precision(name = "precision"),
keras.metrics.Recall(name = "recall"),
keras.metrics.AUC(name = "auc"),
]
model.compile(
optimizer = Adam(lr = 3e-4),
loss = [keras.losses.BinaryCrossentropy(from_logits = False)],
metrics = METRICS,
)
history = model.fit(train_gen,
epochs=50,
verbose=1,
validation_data=validation_gen,
callbacks=[keras.callbacks.ModelCheckpoint("isic_binary_model")],
)

Why I am getting error while using ResNet50 for transfer learning?

So I am trying to use ResNet50 for transfer learning with my keras model(i.e Trinity), but the problem is I am getting an error when I try to execute Trinity.fit().
Exact error "ValueError: Input 0 of layer sequential_8 is incompatible with the layer: expected ndim=4, found ndim=2. Full shape received: [None, 2048]"
What is the reason for this error? How can I solve it?
Here is the python code:
1) from keras.models import Sequential
from keras.layers import Conv2D,MaxPooling2D
from keras.layers import Activation, Dense, Flatten, Dropout
from keras.preprocessing.image import ImageDataGenerator
import os
2) import keras
import wandb
from wandb.keras import WandbCallback
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.applications.resnet50 import ResNet50, decode_predictions,preprocess_input
import matplotlib.pyplot as plt
3) training_dir = '../input/fruits/fruits-360/Training/'
validation_dir = '../input/fruits/fruits-360/Test/'
test_dir = '../input/fruits/fruits-360/test-multiple_fruits/'
4) image_size = 224
data_generator = ImageDataGenerator(preprocessing_function=preprocess_input)
train_generator = data_generator.flow_from_directory(
'../input/fruits/fruits-360/Training/',
target_size=(image_size, image_size),
batch_size=24,
class_mode='categorical')
validation_generator = data_generator.flow_from_directory(
'../input/fruits/fruits-360/Test/',
target_size=(image_size, image_size),
batch_size=24,
class_mode='categorical')
#OUTPUT
#[Found 67692 images belonging to 131 classes.
#Found 22688 images belonging to 131 classes.]
5) resnet_model = ResNet50(weights="imagenet")
6) x_train_preprocessed = train_generator
x_test_preprocessed = validation_generator
7) last_layer = resnet_model.get_layer("avg_pool")
resnet_layers = keras.Model(inputs=resnet_model.inputs, outputs=last_layer.output)
resnet_layers.summary()
8) x_train_features = resnet_layers.predict(x_train_preprocessed)
x_test_features = resnet_layers.predict(x_test_preprocessed)
9) Trinity = Sequential()
Trinity.add(Conv2D(filters = 16, kernel_size = 2,input_shape=(224,224,3),padding='same'))
Trinity.add(Activation('relu'))
Trinity.add(MaxPooling2D(pool_size=2))
Trinity.add(Conv2D(filters = 32,kernel_size = 2,activation= 'relu',padding='same'))
Trinity.add(MaxPooling2D(pool_size=2))
Trinity.add(Conv2D(filters = 64,kernel_size = 2,activation= 'relu',padding='same'))
Trinity.add(MaxPooling2D(pool_size=2))
Trinity.add(Conv2D(filters = 128,kernel_size = 2,activation= 'relu',padding='same'))
Trinity.add(MaxPooling2D(pool_size=2))
Trinity.add(Dropout(0.3))
Trinity.add(Flatten())
Trinity.add(Dense(132))
Trinity.add(Activation('relu'))
Trinity.add(Dropout(0.4))
Trinity.add(Dense(131,activation = 'softmax'))
Trinity.compile(loss='categorical_crossentropy',optimizer='rmsprop',metrics=['accuracy'])
10) Trinity.fit(x_train_features,epochs=50,validation_data=x_test_features)
#OUTPUT OF 7)
#OUTPUT OF 10)

you should add include_top=False to the instance of resnet, then add your model to the top of resnet. that way it should work. Here is an example:
base_model = ResNet50V2(include_top=False, weights="imagenet", input_shape=(224,224,3), pooling="avg")
base_model.summary()
model2 = Sequential()
model2.add(base_model)
model2.add(Dense(64, activation="relu"))
model2.add(Dropout(0.2))
model2.add(Dense(64, activation="relu"))
model2.add(Dropout(0.2))
model2.add(Dense(32, activation="relu"))
model2.add(Dropout(0.2))
model2.add(Dense(32, activation="relu"))
model2.add(Dropout(0.2))
model2.add(Dense(1, activation="sigmoid"))
base_model.trainable = False
model2.summary()
model2.compile(optimizer=Adam(), loss="binary_crossentropy", metrics=["accuracy"])
and here is another example from keras website: https://keras.io/examples/vision/image_classification_efficientnet_fine_tuning/

ValueError: Error when checking target: expected flatten_1 to have shape (2048,) but got array with shape (2,)

I'm trying to run this code, and I have this error:
ValueError: Error when checking target: expected flatten_4 to have shape (2048,) but got array with shape (2,)
NUM_CLASSES = 2
CHANNELS = 3
IMAGE_RESIZE = 224
RESNET50_POOLING_AVERAGE = 'avg'
DENSE_LAYER_ACTIVATION = 'softmax'
OBJECTIVE_FUNCTION = 'categorical_crossentropy'
NUM_EPOCHS = 10
EARLY_STOP_PATIENCE = 3
STEPS_PER_EPOCH_TRAINING = 10
STEPS_PER_EPOCH_VALIDATION = 10
BATCH_SIZE_TRAINING = 100
BATCH_SIZE_VALIDATION = 100
BATCH_SIZE_TESTING = 1
resnet_weights_path = '../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5'
model = Sequential()
train_data_dir = "C:\\Users\\Desktop\\RESNET"
model = ResNet50(include_top=True, weights='imagenet')
model.layers.pop()
model = Model(input=model.input,output=model.layers[-1].output)
model.summary()
sgd = optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=SGD(lr=0.01, momentum=0.9), metrics= ['binary_accuracy'])
data_dir = "C:\\Users\\Desktop\\RESNET"
batch_size = 32
from keras.applications.resnet50 import preprocess_input
from keras.preprocessing.image import ImageDataGenerator
image_size = IMAGE_RESIZE
data_generator = ImageDataGenerator(preprocessing_function=preprocess_input)
def append_ext(fn):
return fn+".jpg"
from os import listdir
from os.path import isfile, join
dir_path = os.path.dirname(os.path.realpath(__file__))
train_dir_path = dir_path + '\data'
onlyfiles = [f for f in listdir(dir_path) if isfile(join(dir_path, f))]
data_labels = [0, 1]
t = []
maxi = 25145
LieOffset = 15799
i = 0
while i < maxi: # t = tuple
if i <= LieOffset:
t.append(label['Lie'])
else:
t.append(label['Truth'])
i = i+1
train_datagenerator = ImageDataGenerator(rescale=1./255,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
validation_split=0.2)
train_generator = train_datagenerator.flow_from_directory(
train_data_dir,
target_size=(image_size, image_size),
batch_size=BATCH_SIZE_TRAINING,
class_mode='categorical', shuffle=False, subset='training')
validation_generator = train_datagenerator.flow_from_directory(
train_data_dir, # same directory as training data kifkif
target_size=(image_size, image_size),
batch_size=BATCH_SIZE_TRAINING,
class_mode='categorical', shuffle=False, subset='validation')
(BATCH_SIZE_TRAINING, len(train_generator), BATCH_SIZE_VALIDATION, len(validation_generator))
from tensorflow.python.keras.callbacks import EarlyStopping, ModelCheckpoint
cb_early_stopper = EarlyStopping(monitor = 'val_loss', patience = EARLY_STOP_PATIENCE)
cb_checkpointer = ModelCheckpoint(filepath = '../working/best.hdf5', monitor = 'val_loss', save_best_only = True, mode = 'auto')
from sklearn.grid_search import ParameterGrid
param_grid = {'epochs': [5, 10, 15], 'steps_per_epoch' : [10, 20, 50]}
grid = ParameterGrid(param_grid)
val_loss as final model
for params in grid:
print(params)
fit_history = model.fit_generator(
train_generator,
steps_per_epoch=STEPS_PER_EPOCH_TRAINING,
epochs = NUM_EPOCHS,
validation_data=validation_generator,
validation_steps=STEPS_PER_EPOCH_VALIDATION,
callbacks=[cb_checkpointer, cb_early_stopper])
model.load_weights("../working/best.hdf5")

The error suggests that your models output layer should have 2 nodes whereas you have 2048 as you are using the output of avg_pool layer of ResNet50 model as your model output. So, you can add a Dense layer having 2 nodes on top of the avg_pool layer to solve the problem.
model = ResNet50(include_top=True, weights='imagenet')
print(model.summary())
x = model.get_layer('avg_pool').output
predictions = Dense(2, activation='sigmoid')(x)
model = Model(input = model.input, output = predictions)
print(model.summary())
As I'm not quite sure about what type of problem you are solving, i assumed that multilabel (2) classification as your data label shape is (2,).
However, if you are solving a binary classification problem then you need to change your label so that it's either 1 or 0. So, Change class_mode='categorical' to class_mode='binary' in both train_generator and validation_generator. In that case the model output layer should have 1 node.
predictions = Dense(1, activation='sigmoid')(x)

shape mismatch model InceptionResNetV2 & weigts

I am using InceptionResNetV2 for image classification & using repective weight. But get error :
ValueError: You are trying to load a weight file containing 449 layers into a model with 448 layers.
img_ht = 96
img_wid = 96
img_chnl = 3
import tensorflow as tf
from tensorflow import keras
from keras_preprocessing.image import ImageDataGenerator
train_generator = train_datagen.flow_from_directory(
directory = "../input/cassava-disease/train/train/",
subset="training",
batch_size = 49,
seed=42,
shuffle=False,
class_mode="categorical",
target_size=(img_ht, img_wid))
valid_generator = train_datagen.flow_from_directory(
directory = "../input/cassava-disease/train/train/",
subset="validation",
batch_size=49,
seed=42,
shuffle=False,
class_mode="categorical",
target_size = (img_ht, img_wid))
from keras.applications import InceptionResNetV2 as InceptionResNetV2
base_model = keras.applications.InceptionResNetV2(input_shape=(img_ht, img_wid, 3),
include_top = False,
weights = "../input/inception/inception_resnet_v2_weights_tf_dim_ordering_tf_kernels.h5")
base_model.trainable = False
print(base_model.summary())

Got the answer. It's because of line --> include_top = False.
Quite new to python & Machine Learning