I see that the imageDataGenerator allows me to specify different styles of data normalization, e.g. featurewise_center, samplewise_center, etc.
I see from the examples that if I specify one of these options, then I need to call the fit method on the generator in order to allow the generator to compute statistics like the mean image on the generator.
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
datagen = ImageDataGenerator(
featurewise_center=True,
featurewise_std_normalization=True,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True)
# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(X_train)
# fits the model on batches with real-time data augmentation:
model.fit_generator(datagen.flow(X_train, Y_train, batch_size=32),
samples_per_epoch=len(X_train), nb_epoch=nb_epoch)
My question is, how does prediction work if I have specified data normalization during training? I can't see how in the framework I would even pass knowledge of the training set mean/std deviation along to predict to allow me to normalize my test data myself, but I also don't see in the training code where this information is stored.
Are the image statistics needed for normalization stored in the model so that they can be used during prediction?
Yes - this is a really huge downside of Keras.ImageDataGenerator that you couldn't provide the standarization statistics on your own. But - there is an easy method on how to overcome this issue.
Assuming that you have a function normalize(x) which is normalizing an image batch (remember that generator is not providing a simple image but an array of images - a batch with shape (nr_of_examples_in_batch, image_dims ..) you could make your own generator with normalization by using:
def gen_with_norm(gen, normalize):
for x, y in gen:
yield normalize(x), y
Then you might simply use gen_with_norm(datagen.flow, normalize) instead of datagen.flow.
Moreover - you might recover the mean and std computed by a fit method by getting it from appropriate fields in datagen (e.g. datagen.mean and datagen.std).
Use the standardize method of the generator for each element. Here is a complete example for CIFAR 10:
#!/usr/bin/env python
import keras
from keras.datasets import cifar10
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
# input image dimensions
img_rows, img_cols, img_channels = 32, 32, 3
num_classes = 10
batch_size = 32
epochs = 1
# The data, shuffled and split between train and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
# Convert class vectors to binary class matrices.
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
model = Sequential()
model.add(Conv2D(32, (3, 3), padding='same', activation='relu',
input_shape=x_train.shape[1:]))
model.add(Conv2D(32, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop',
metrics=['accuracy'])
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
datagen = ImageDataGenerator(zca_whitening=True)
# Compute principal components required for ZCA
datagen.fit(x_train)
# Apply normalization (ZCA and others)
print(x_test.shape)
for i in range(len(x_test)):
# this is what you are looking for
x_test[i] = datagen.standardize(x_test[i])
print(x_test.shape)
# Fit the model on the batches generated by datagen.flow().
model.fit_generator(datagen.flow(x_train, y_train,
batch_size=batch_size),
steps_per_epoch=x_train.shape[0] // batch_size,
epochs=epochs,
validation_data=(x_test, y_test))
I also had the same issue and I solved it using the same functionality, that the ImageDataGenerator used:
# Load Cifar-10 dataset
(trainX, trainY), (testX, testY) = cifar10.load_data()
generator = ImageDataGenerator(featurewise_center=True,
featurewise_std_normalization=True)
# Calculate statistics on train dataset
generator.fit(trainX)
# Apply featurewise_center to test-data with statistics from train data
testX -= generator.mean
# Apply featurewise_std_normalization to test-data with statistics from train data
testX /= (generator.std + K.epsilon())
# Do your regular fitting
model.fit_generator(..., validation_data=(testX, testY), ...)
Note that this is only possible if you have a reasonable small dataset, like CIFAR-10. Otherwise the solution proposed by Marcin sounds good more reasonable.
I am using the datagen.fit function itself.
from keras.preprocessing.image import ImageDataGenerator
train_datagen = ImageDataGenerator(
featurewise_center=True,
featurewise_std_normalization=True)
train_datagen.fit(train_data)
test_datagen = ImageDataGenerator(
featurewise_center=True,
featurewise_std_normalization=True)
test_datagen.fit(train_data)
Ideally with this, test_datagen fitted on training dataset will learn the training datasets statistics. Then it will use these statistics to normalize testing data.
Related
I'm doing a ML model that takes pixel values from a numpy array as training and testing data. I defined a function that divides the dataset into images and labels. My task is to use Image Generator for data augmentation and then train the model. Everything goes smoothly until I am to train the model. It keeps giving me errors about the loss function used. When I use categorical_crossentropy it says I can either use 'sparse_categorical_crossentropy' or use function to_categorical. Well I tried both and there were still errors so I decided to try and use tf.convert_to_tensor() on my labels but now I get a shape error:
ValueError: A target array with shape (126, 25, 2) was passed for an output of shape (None, 3) while using as loss `categorical_crossentropy`. This loss expects targets to have the same shape as the output.
This is my code:
training_labels = tf.convert_to_tensor(training_labels)
testing_labels = tf.convert_to_tensor(testing_labels)
# Create an ImageDataGenerator and do Image Augmentation
train_datagen = ImageDataGenerator(
rescale = 1./255,
rotation_range=40,
width_shift_range=0.2,
height_shift_range=0.2,
shear_range=0.2,
zoom_range=0.2,
horizontal_flip=True,
fill_mode='nearest'
)
validation_datagen = ImageDataGenerator(rescale = 1./255)
train_generator = train_datagen.flow(training_images,
training_labels,
batch_size=126
)
validation_generator = validation_datagen.flow(
testing_images,
testing_labels,
batch_size=126
)
# Keep These
print(training_images.shape)
print(testing_images.shape)
# Their output should be:
# (27455, 28, 28, 1)
# (7172, 28, 28, 1)
And here goes the model:
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(64, (3,3), activation='relu', input_shape=(28, 28, 1)),
tf.keras.layers.MaxPooling2D(2, 2),
tf.keras.layers.Conv2D(64, (3,3), activation='relu'),
tf.keras.layers.MaxPooling2D(2,2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(3, activation='softmax')
])
# Compile Model.
model.compile(loss = 'categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
# Train the Model
history = model.fit_generator(train_generator, validation_data=validation_generator, epochs=2)
model.evaluate(testing_images, testing_labels, verbose=0)
I got stuck with it, I googled for solution but with no success. Can you please help me somehow make a move?
Thanks a lot!
When using categorical cross entropy as the loss function, the labels should be one hot encoded and hence the number of neurons present in the final layer should be equal to the number of classes present in the dataset and that's the error you are getting. Since the number of output neurons is 3, I'm guessing you have 3 classes and hence the shape of training_labels/testing_labels should be (num of images in train/test, 3).
Below is a small snippet for cifar dataset.
from tf.keras.utils import to_categorical
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
num_classes = 10
# convert to one hot encoding
# shape will be (60000, 10) since there are 60000 images and 10 classes in cifar
y_train = to_categorical(y_train, num_classes)
# shape will be (10000, 10) since there are 10000 images and 10 classes in cifar
y_test = to_categorical(y_test, num_classes)
datagen = ImageDataGenerator(
featurewise_center=True,
featurewise_std_normalization=True,
rotation_range=20,
width_shift_range=0.2,
height_shift_range=0.2,
horizontal_flip=True)
# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(x_train)
history = model.fit_generator(datagen.flow(x_train, y_train, batch_size=32), epochs=2)
I'm trying to train my Deep Neural Network to recognize handwritten
numbers but I keep getting the error stated previously in the title It
gives me an error saying: ValueError: Input arrays should have the
same number of samples as target arrays. Found 60000 input samples and
10000 target samples. How can i fix this? (i already tried
train_test_split and transport but nothing worked)
# Imports
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.utils import to_categorical
# Configuration options
feature_vector_length = 784
num_classes = 60000
# Load the data
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
# Reshape the data - MLPs do not understand such things as '2D'.
# Reshape to 28 x 28 pixels = 784 features
X_train = X_train.reshape(X_train.shape[0], feature_vector_length)
X_test = X_test.reshape(X_test.shape[0], feature_vector_length)
# Convert into greyscale
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
# Convert target classes to categorical ones
Y_train = to_categorical(Y_train, num_classes)
Y_test = to_categorical(Y_test, num_classes)
# Load the data
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
# Visualize one sample
import matplotlib.pyplot as plt
plt.imshow(X_train[0], cmap='Greys')
plt.show()
# Set the input shape
input_shape = (feature_vector_length,)
print(f'Feature shape: {input_shape}')
# Create the model
# Using sigmoid instead of relu function
model = Sequential()
model.add(Flatten())
model.add(Dense(350, input_shape=input_shape, activation="sigmoid",
kernel_initializer=init))
model.add(Dense(50, activation="sigmoid", kernel_initializer=init))
model.add(Dense(num_classes, activation="sigmoid",
kernel_initializer=init))
# Configure the model and start training
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=
['accuracy'])
model.fit(X_train, Y_train, epochs=10, batch_size=250, verbose=1,
validation_split=0.2)
# Test the model after training
test_results = model.evaluate(X_test, Y_test, verbose=1)
print(f'Test results - Loss: {test_results[0]} - Accuracy:
{test_results[1]}%')
If you want a solution to your problem, here it is:
# Imports
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Flatten
from keras.utils import to_categorical
# Configuration options
feature_vector_length = 784
num_classes = 10
# Load the data
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
# Reshape the data - MLPs do not understand such things as '2D'.
# Reshape to 28 x 28 pixels = 784 features
# X_train = X_train.reshape(X_train.shape[0], feature_vector_length)
# X_test = X_test.reshape(X_test.shape[0], feature_vector_length)
# Convert into greyscale
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
# Convert target classes to categorical ones
Y_train = to_categorical(Y_train, num_classes)
Y_test = to_categorical(Y_test, num_classes)
# Create the model
# Using sigmoid instead of relu function
model = Sequential()
model.add(Flatten())
model.add(Dense(350, input_shape=input_shape, activation="relu"))
model.add(Dense(50, activation="relu"))
model.add(Dense(num_classes, activation="softmax"))
# Configure the model and start training
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=
['accuracy'])
model.fit(X_train, Y_train, epochs=10, batch_size=250, verbose=1,
validation_split=0.2)
# Test the model after training
test_results = model.evaluate(X_test, Y_test, verbose=1)
But you should really do some research and understand what every line in your code is supposed to do and what every parameter means. For example, the choice of sigmoid activation function being wrong, especially in the final layer is the very first thing to understand. That's one of the many things that you should do research about. Then there is:
understanding why and when to reshape your data,
what is the purpose of flatten layer
and most importantly, understand what is num_classes and why it is 10 and not 1000 or 60000
import tensorflow as tf
import tensorflow.keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D
import pickle
import numpy as np
from keras.models import model_from_json
from keras.models import load_model
import matplotlib.pyplot as plt
# Opening the files about data
X = pickle.load(open("X.pickle", "rb"))
y = pickle.load(open("y.pickle", "rb"))
# normalizing data (a pixel goes from 0 to 255)
X = X/255.0
# Building the model
model = Sequential()
# 3 convolutional layers
model.add(Conv2D(32, (3, 3), input_shape = X.shape[1:]))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Conv2D(64, (3, 3)))
model.add(Activation("relu"))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Dropout(0.9))
# 5 hidden layers
model.add(Flatten())
model.add(Dense(128))
model.add(Activation("relu"))
model.add(Dense(128))
model.add(Activation("relu"))
model.add(Dense(128))
model.add(Activation("relu"))
model.add(Dense(128))
model.add(Activation("relu"))
model.add(Dense(128))
model.add(Activation("relu"))
# The output layer with 7 neurons, for 7 classes
model.add(Dense(13))
model.add(Activation("softmax"))
# Compiling the model using some basic parameters
model.compile(loss="sparse_categorical_crossentropy",
optimizer="adam",
metrics=["accuracy"])
# Training the model, with 40 iterations
# validation_split corresponds to the percentage of images used for the validation phase compared to all the images
print("X = " + str(len(X)))
print("y = " + str(len(y)))
history = model.fit(X, y, batch_size=32, epochs=1000, validation_split=0.1)
# Saving the model
model_json = model.to_json()
with open("model.json", "w") as json_file :
json_file.write(model_json)
model.save_weights("model.h5")
print("Saved model to disk")
model.save('CNN.model')
# Printing a graph showing the accuracy changes during the training phase
print(history.history.keys())
plt.show()
plt.plot(history.history['accuracy'])
plt.plot(history.history['loss'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'validation'], loc='upper left')
plt.show()
The problem is that, I am getting lower training loss but very high validation accuracy. And accuracy of validation is also extremely low. How can I solve this issue? I have tried to increase the drop value up-to 0.9 but still the loss is much higher. I also tried using linear function for activation, but no use.
Please help.
As is already mentioned, it is pretty hard to give a good advice without seeing the data.
What I would try is the following:
- remove the Dropout after the maxpooling layer
- remove some dense layer
- add dropout between dense
If it´s then still overfitting, add dropout between dense layers
Edit:
After I have seen the loss and accuracy plot I would suggest the following:
the highest priority is, to get more data.
then use data augmentation to even increase your dataset
further reduce the complexity of your neural network if additional data doesn’t help (but I think that training will slow down with more data and validation loss will also decrease for a longer period of epochs)
Data Augmentation is the best technique to reduce overfitting. Try data generators for training and validation sets to reduce the loss and increase accuracy.
To learn more about Augmentation, and the available transforms, check out https://github.com/keras-team/keras-preprocessing
# Add our data-augmentation parameters to ImageDataGenerator
train_datagen = ImageDataGenerator(rescale = 1./255.,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
shear_range = 0.2,
zoom_range = 0.2,
horizontal_flip = True)
# Note that the validation data should not be augmented!
test_datagen = ImageDataGenerator(rescale = 1./255.)
# Flow training images in batches of 20 using train_datagen generator
train_generator = train_datagen.flow_from_directory(train_dir,
batch_size = 20,
class_mode = 'binary',
target_size = (150, 150))
# Flow validation images in batches of 20 using test_datagen generator
validation_generator = test_datagen.flow_from_directory(validation_dir,
batch_size = 20,
class_mode = 'binary',
target_size = (150, 150))
# Now fit the training, validation generators to the CNN model
history = model.fit_generator(train_generator,
validation_data = validation_generator,
steps_per_epoch = 100,
epochs = 3,
validation_steps = 50,
verbose = 2,callbacks=[callbacks])
I learned to implement an image classifier from a code which i got from a post and it was very helpful,but i don't know how to predict on an image.I tried but its giving a Value Error.I am still a beginner
Keras Code:-
from __future__ import print_function
from keras.datasets import cifar10
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.optimizers import SGD, Adam
from keras.utils import np_utils
import numpy as np
#seed = 7
#np.random.seed(seed)
batch_size = 50
nb_classes = 10
nb_epoch = 150
data_augmentation = False
# input image dimensions
img_rows, img_cols = 32, 32
# the CIFAR10 images are RGB
img_channels = 3
# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')
# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)
model = Sequential()
model.add(Convolution2D(32, 3, 3, border_mode='same',
input_shape=X_train.shape[1:]))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
# let's train the model using SGD + momentum (how original).
#sgd = SGD(lr=0.001, decay=1e-6, momentum=0.9, nesterov=True)
sgd= Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
model.compile(loss='categorical_crossentropy',
optimizer=sgd,
metrics=['accuracy'])
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
if not data_augmentation:
print('Not using data augmentation.')
model.fit(X_train, Y_train,
batch_size=batch_size,
nb_epoch=nb_epoch,
validation_data=(X_test, Y_test),
shuffle=True)
else:
print('Using real-time data augmentation.')
# this will do preprocessing and realtime data augmentation
datagen = ImageDataGenerator(
featurewise_center=False, # set input mean to 0 over the dataset
samplewise_center=False, # set each sample mean to 0
featurewise_std_normalization=False, # divide inputs by std of the dataset
samplewise_std_normalization=False, # divide each input by its std
zca_whitening=False, # apply ZCA whitening
rotation_range=0, # randomly rotate images in the range (degrees, 0 to 180)
width_shift_range=0.1, # randomly shift images horizontally (fraction of total width)
height_shift_range=0.1, # randomly shift images vertically (fraction of total height)
horizontal_flip=True, # randomly flip images
vertical_flip=False) # randomly flip images
# compute quantities required for featurewise normalization
# (std, mean, and principal components if ZCA whitening is applied)
datagen.fit(X_train)
# fit the model on the batches generated by datagen.flow()
model.fit_generator(datagen.flow(X_train, Y_train,
batch_size=batch_size),
samples_per_epoch=X_train.shape[0],
nb_epoch=nb_epoch,
validation_data=(X_test, Y_test))
model.save('CIFAR10.h5')
My Prediction Code:-
from __future__ import print_function
from keras.models import load_model
from keras.utils import np_utils
import numpy as np
import cv2
img_rows, img_cols = 32, 32
model = load_model('CIFAR10.h5')
img = cv2.imread('D:/Study_Material/Python_3_Tutorial/PythonScripts/Machine_Learning/Project/Images/Deer.jpg')
img = cv2.resize(img,(img_rows,img_cols))
Image = np.array(img)
print(model.predict(Image))
Error:-
Warning (from warnings module):
File "C:\Users\Na462\AppData\Local\Programs\Python\Python35\lib\site-packages\h5py\__init__.py", line 36
from ._conv import register_converters as _register_converters
FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
Using TensorFlow backend.
Traceback (most recent call last):
File "D:\Study_Material\Python_3_Tutorial\PythonScripts\Machine_Learning\Project\Keras(Prediction).py", line 12, in <module>
print(model.predict(Image))
File "C:\Users\Na462\AppData\Local\Programs\Python\Python35\lib\site-packages\keras\models.py", line 1025, in predict
steps=steps)
File "C:\Users\Na462\AppData\Local\Programs\Python\Python35\lib\site-packages\keras\engine\training.py", line 1817, in predict
check_batch_axis=False)
File "C:\Users\Na462\AppData\Local\Programs\Python\Python35\lib\site-packages\keras\engine\training.py", line 113, in _standardize_input_data
'with shape ' + str(data_shape))
ValueError: Error when checking : expected conv2d_1_input to have 4 dimensions, but got array with shape (32, 32, 3)
Please Tell me the right way to Predict in Keras so that i can implement it on different test cases.
The error you are getting is because all the frameworks assume an image input is a batch of images making it a 4d tensor, instead of one image (3d tensor). To just do single image batches, expand the input to be of size (1, 32, 32, 3), the 1 in the beginning being the batch size of 1.
I do not know keras specifically very well, but since you are passing in a numpy array, you can modify your 'Image' object like so (see second to last line):
img_rows, img_cols = 32, 32
model = load_model('CIFAR10.h5')
img = cv2.imread('D:/Study_Material/Python_3_Tutorial/PythonScripts/Machine_Learning/Project/Images/Deer.jpg')
img = cv2.resize(img,(img_rows,img_cols))
Image = np.expand_dims(np.array(img), axis=0)
print(model.predict(Image))
I believe that this is my first question in Stack Overflow, so I apologize in advance if I don't follow all guidelines.
I recently started to use Keras for deep learning, and since I work with HDF5 files using h5py to manage large datasets, I searched for a way to train models using keras on very large HDF5 files. I found out that the most common way would be to use HDF5Matrix found in keras.utils.io_utils.
I modified one of Keras examples (mnist.cnn) as following:
'''Trains a simple convnet on the MNIST dataset.
Gets to 99.25% test accuracy after 12 epochs
(there is still a lot of margin for parameter tuning).
16 seconds per epoch on a GRID K520 GPU.
'''
from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K
# My Imports
from os.path import exists
import h5py
from keras.utils.io_utils import HDF5Matrix
batch_size = 128
num_classes = 10
epochs = 12
# input image dimensions
img_rows, img_cols = 28, 28
# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()
if K.image_data_format() == 'channels_first':
x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
input_shape = (1, img_rows, img_cols)
else:
x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
# convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
#-----------------------------------HDF5 files creation---------------------------------------
sample_file_name = "x.hdf5"
solution_file_name = "y.hdf5"
train_name = "train"
test_name = "test"
#Create dataset
if (not exists(sample_file_name)) and (not exists(solution_file_name)):
samples_file = h5py.File(sample_file_name,mode='a')
solutions_file = h5py.File(solution_file_name,mode='a')
samples_train = samples_file.create_dataset(train_name,data=x_train)
samples_test = samples_file.create_dataset(test_name, data=x_test)
solution_train = solutions_file.create_dataset(train_name, data=y_train)
solution_test = solutions_file.create_dataset(test_name, data=y_test)
samples_file.flush()
samples_file.close()
solutions_file.flush()
solutions_file.close()
x_train = HDF5Matrix(sample_file_name,train_name)
x_test = HDF5Matrix(sample_file_name,test_name)
y_train = HDF5Matrix(solution_file_name,train_name)
y_test = HDF5Matrix(solution_file_name,test_name)
#---------------------------------------------------------------------------------------------
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
# If using HDF5Matrix one needs to disable shuffle
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test),
shuffle=False)
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
However, there is something that concerns me. In segmentation problems \ multi-class problems, where the number of classes is very large, saving the solutions in categorical format is very wasteful. Moreover, doing that means that once you add a new class, the entire dataset should be changed accordingly.
That's why I thought using the normalizer feature of HDF5Matrix as following:
'''Trains a simple convnet on the MNIST dataset.
Gets to 99.25% test accuracy after 12 epochs
(there is still a lot of margin for parameter tuning).
16 seconds per epoch on a GRID K520 GPU.
'''
from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K
# My Imports
from os.path import exists
import h5py
from keras.utils.io_utils import HDF5Matrix
batch_size = 128
num_classes = 10
epochs = 12
# input image dimensions
img_rows, img_cols = 28, 28
# the data, shuffled and split between train and test sets
(x_train, y_train), (x_test, y_test) = mnist.load_data()
if K.image_data_format() == 'channels_first':
x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)
x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)
input_shape = (1, img_rows, img_cols)
else:
x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
#-----------------------------------HDF5 files creation---------------------------------------
sample_file_name = "x.hdf5"
solution_file_name = "y.hdf5"
train_name = "train"
test_name = "test"
#Create dataset
if (not exists(sample_file_name)) and (not exists(solution_file_name)):
samples_file = h5py.File(sample_file_name,mode='a')
solutions_file = h5py.File(solution_file_name,mode='a')
samples_train = samples_file.create_dataset(train_name,data=x_train)
samples_test = samples_file.create_dataset(test_name, data=x_test)
solution_train = solutions_file.create_dataset(train_name, data=y_train)
solution_test = solutions_file.create_dataset(test_name, data=y_test)
samples_file.flush()
samples_file.close()
solutions_file.flush()
solutions_file.close()
x_train = HDF5Matrix(sample_file_name,train_name)
x_test = HDF5Matrix(sample_file_name,test_name)
y_train = HDF5Matrix(solution_file_name,train_name,normalizer=lambda solution: keras.utils.to_categorical(solution,num_classes))
y_test = HDF5Matrix(solution_file_name,test_name,normalizer=lambda solution: keras.utils.to_categorical(solution,num_classes))
#---------------------------------------------------------------------------------------------
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
activation='relu',
input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))
model.compile(loss=keras.losses.categorical_crossentropy,
optimizer=keras.optimizers.Adadelta(),
metrics=['accuracy'])
# If using HDF5Matrix one needs to disable shuffle
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
verbose=1,
validation_data=(x_test, y_test),
shuffle=False)
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
However, this yields an error implying that the shape of the solution should match, and normalizer shouldn't be used that way:
ValueError: Error when checking target: expected dense_2 to have 2, but got array with shape (60000, 1, 10)
So, is there a way to save the data in HDF5 (and if not possible, using some other format), and use Keras in a manner that saves the labels (and not the categorical vector) without turning it into a regression problem?
You are getting this error because of these lines.
Keras checks input shapes before training. The problem is that HDF5Matrix will return the pre-normalized shape if you call .shape, then Keras will believe you have a (60000,) array for y_train and a (10000,) for y_test.
However, when accessing a slice of the matrix, the normalizer is applied so that for example y_train[5:7].shape does have the final expected shape: (2, 10).
This is mainly because the normalizer isn't really expected to change the shape, but Keras could indeed handle this case.
You can fix it by using fit_generator instead of fit so that training only sees normalized data:
def generator(features, labels, size):
while True:
start, end = 0, size
while end < len(features):
s = slice(start, end)
# you can actually do the normalization here if you want
yield features[s], labels[s]
start, end = end, end + size
model.fit_generator(
generator(x_train, y_train, batch_size),
steps_per_epoch=len(x_train) // batch_size,
epochs=1,
verbose=1,
validation_data=generator(x_test, y_test, batch_size),
validation_steps=len(x_test) // batch_size,
shuffle=False)
Note that you could do any kind of normalization inside the generator function and that will be transparent to Keras. And you can use different batch sizes for train and validation.
Also, you have to change the evaluation in the same way:
score = model.evaluate_generator(
generator(x_test, y_test, batch_size),
steps=len(x_test) // batch_size)
I think your solution with the normalizer is a good idea, by the way.