I am currently working on building a neural network from scratch (only using numpy, no tensorflow or similiar libraries) in python. After training with the MNIST training data, the network reached an accuracy of 98%+ on the test data. I then tried creating an user interface with gradio. But the network seems to be performing much worse than expected (probably accuracy of around 70%).
I was wondering if it maybe is because i am not evaluating the model correctly or if the problem is somehow the gradio user interface.
Here is the code for evaluating the network:
def evaluate_network(network):
correct = 0
for test in range(len(x_test)):
if network.output_layer.prediction(network.forwards(x_test[test])) == np.argmax(y_test[test]):
correct += 1
return correct / len(x_test)
And the code for the interface:
import gradio as gr
def predict(image):
image = np.reshape(image, (-1, 784))
image = image.astype('float32') / 255
output = NeuralNetwork.forwards(image)
prediction = NeuralNetwork.output_layer.prediction(output)
return str(prediction)
interface = gr.Interface(fn=predict,inputs="sketchpad",outputs="label", allow_flagging="never")
interface.launch(debug= True, inbrowser=True)
I tried to reshape the drawn image to be like the data the network has been trained with.
Here is how i loaded the mnist data into the network:
from keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
y_train = keras.utils.np_utils.to_categorical(y_train, 10)
y_test = keras.utils.np_utils.to_categorical(y_test, 10)
x_train = np.reshape(x_train, (-1, 784))
x_test = np.reshape(x_test, (-1, 784))
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
So i am just not sure why there seems to be such a big difference in performance, since the data should be (to my understanding) of the same format and shape.
Related
Trying to recognise handwritten digits using simple architecture. Test gives 0.9723 accuracy
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow import keras
from tensorflow.keras.layers import Dense, Flatten
from sklearn.model_selection import train_test_split
# data split
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# normalizing
x_train = x_train / 255
x_test = x_test / 255
y_train_cat = keras.utils.to_categorical(y_train, 10)
y_test_cat = keras.utils.to_categorical(y_test, 10)
# creating model
model = keras.Sequential([
Flatten(input_shape=(28, 28, 1)),
Dense(128, activation='relu'),
Dense(10, activation='softmax')
])
model.compile(optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
x_train_split, x_val_split, y_train_split, y_val_split = train_test_split(x_train, y_train_cat, test_size=0.2)
model.fit(
x_train_split,
y_train_split,
batch_size=32,
epochs=6,
validation_data=(x_val_split, y_val_split))
# saving model
model.save('mnist_model.h5')
# test
model.evaluate(x_test, y_test_cat)
But when I try to recognise my own numbers (0 to 9), some of them aren't recognised correctly:
numbers and prediction above
Trying with this code:
from keras.models import load_model
from tensorflow.keras.datasets import mnist
from tensorflow import keras
import numpy as np
import matplotlib.pyplot as plt
from PIL import Image
model = load_model('mnist_model.h5')
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_test = x_test / 255
y_test_cat = keras.utils.to_categorical(y_test, 10)
model.evaluate(x_test, y_test_cat)
filenames = [
'project_imgs/0.png', 'project_imgs/1.png', 'project_imgs/2.png', 'project_imgs/3.png',
'project_imgs/4.png', 'project_imgs/5.png', 'project_imgs/6.png', 'project_imgs/7.png',
'project_imgs/8.png', 'project_imgs/9.png'
]
data = []
data_eds = []
for file in filenames:
picture = Image.open(file).convert('L')
pic_r = picture.resize((28, 28))
pic = np.array(pic_r)
pic = 255 - pic
pic = pic / 255
pic_eds = np.expand_dims(pic, axis=0)
data.append(pic)
data_eds.append(pic_eds)
plt.figure(figsize=(10, 5))
for i in range(10):
ax = plt.subplot(2, 5, i+1)
ax.set_title(f'Looks like {np.argmax(model.predict(data_eds[i]))}')
plt.xticks([])
plt.yticks([])
plt.imshow(data[i], cmap=plt.cm.binary)
plt.show()
I don't understand why is this happening. Could it be because of the pictures? I've seen that MNIST produces images that are more black and not as grey as mine. Or is it because of the size of the figures in relation to this 28x28 square?
It is probably because of the difference in data sets. MNIST numbers usually has more solid colors and bolder than your own numbers. That is the only thing come up to my mind, because your code looks fine.
The solution is to change your numbers as more similar to MNIST numbers or creating a large enough data set with your numbers to train the model with them.
You are using following line of code while processing the images after loading them:
pic = 255 - pic
This preprocessing step is not present in the training step. Maybe this is the reason why most images are incorrectly classified.
OK, the key was working with images. I wrote code with which I was able to recognise 9 out of 10 images, but the number "9" was still not recognised.
for file in filenames:
img = Image.new('RGBA', size=(28, 28), color='white')
number = Image.open(file).convert('RGBA')
number_res = number.resize((20, 20), resample=Image.ANTIALIAS)\
.rotate(6, expand=1, fillcolor='white')
img.paste(number_res, (4, 4))
img = img.convert('L')
img = np.array(img)
img = 255 - img
img = img / 255
img_eds = np.expand_dims(img, axis=0)
data.append(img)
data_eds.append(img_eds)
Then I worked with it in Photoshop and it worked. "9", as I understood it, was not recognised because there was quite a large horizontal distance between the end of the tail and the loop. Because of this, it was impossible to place the digit in the centre.
FINAL RESULTS:
Dataset:
The PV Yield (kWh) is my output. My model is suppose to predict this.
This is what I have done. I have attached the image of the dataset. From AirTemp to Zenith is my X and Y is PV Yield(KW/H).
df=pd.read_csv("Data1.csv")
X=df.drop(['Date-PrimaryKey','output-PV Yield (kWh)'],axis=1)
Y=df['output-PV Yield (kWh)']
pca = PCA(n_components=9)
pca.fit(X_train)
X_train = pca.transform(X_train)
pca.fit(X_test)
X_test = pca.transform(X_test)
#normalizing the input values to fall in -1 to 1
X_train = X_train/180000000.0
X_test = X_test/180000000.0
#Creating Model
model = Sequential()
model.add(Dense(15, input_shape=(9,)))
model.add(Activation('tanh'))
model.add(Dense(11))
model.add(Activation('tanh'))
model.add(Dense(1))
model.summary()
sgd = optimizers.SGD(lr=0.1,momentum=0.2)
model.compile(loss='mean_absolute_error',optimizer=sgd,metrics=['accuracy'])
#Training
model.fit(X_train, train_y, epochs=20, batch_size = 50, validation_data=(X_test, test_y))
My weights are not getting updated. Accuracy is zero in all epochs.
The model seems OK but there are two problems I can spot fast:
pca = PCA(n_components=9)
pca.fit(X_train)
X_train = pca.transform(X_train)
pca.fit(X_test)
X_test = pca.transform(X_test)
Anything used for transformation of the data must not be fit on testing data. You fit it on train samples and then use it to transform both train and test part. You should assume that you know nothing about data you will be predicting on in production, eg. you know nothing about tomorrows weather, results of sport matches in a month, etc. You wont be able to do so then, so you cant do so during training. Correct way:
pca = PCA(n_components=9)
pca.fit(X_train)
X_train = pca.transform(X_train)
X_test = pca.transform(X_test)
The second very incorrect stuff you have there is here:
#normalizing the input values to fall in -1 to 1
X_train = X_train/180000000.0
X_test = X_test/180000000.0
Of course you want to normalize your data, but this way you will end up with incredibly low decimals in cases where values are low, eg. AlbedoDaily column, and quite high values where are values high, such as SurfacePressure. For such scaling you can use already defined classes such as standard scaler. The code is very simple and each column is treated independently:
from sklearn.preprocessing import StandardScaler
transformer = StandardScaler().fit(X_train)
X_train = transformer.transform(X_train)
X_test = transformer.transform(X_test)
You have not provided or explained what your target variable is and where you get is, there could be other problems in your code I can not see right now.
I have a CNN model built in Keras. I then took out its last layer as a feature and retrained an SVM with it.
Is it possible to now find the gradient of the SVMs output wrt the CNN model's input?
I know of this method (Getting gradient of model output w.r.t weights using Keras) and am able to use it to get the gradient wrt input for the layer i am pulling the features out of. I can also get the numerical gradient of the SVM wrt to its input, albeit at the moment its a bit of a mess. Would appreciate some input here as well actually.
But now I need to somehow combine these two to get the gradient of the SVM to the input of the entire CNN model.
"""
Main CNN script
"""
# Imports ##
# general
import matplotlib.pyplot as plt
import numpy as np
# ML libraries
from tensorflow.keras.datasets import mnist
# ML utilities
from tensorflow.keras.utils import to_categorical
# Python scripts used
import train_CNN
import load_CNN
import train_subSVMs
import load_subSVMs
import train_finalSVM
import load_finalSVM
import joblib
def save_array(array, name):
joblib.dump(array, name+'.pkl', compress = 3)
return
def load_array(array, name):
array = joblib.load(array, name)
return array
def show_data_example(i, dataset):
# show some of the images in the dataset
# call multiple times for multiple images
# squeeze is necessary here to get rid of the extra dimension introduced in rehsaping
print('\nExample Image: %s from selected dataset' %i)
plt.imshow(np.squeeze(dataset[i]), cmap=plt.get_cmap('gray'))
plt.show()
return
def load_and_encode(target_shape):
# load dataset
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train, y_train = X_train[:,:,:],y_train[:]
X_test, y_test = X_test[:,:,:], y_test[:]
print('Loaded Mnist dataset')
print('Train: X=%s, y=%s' % (X_train.shape, y_train.shape))
print('Test: X=%s, y=%s' % (X_test.shape, y_test.shape))
# encode y data
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
# normalise X data (X/255 -> [0,1])
X_train = X_train/255.0
X_test = X_test/255.0
# currently dimensions are (m x 28 x 28)
# making them into (m x 28x28x1) 3Dimensional for convolution networks
X_train = X_train.reshape(X_train.shape[0], target_shape[0], target_shape[1], target_shape[2])
X_test = X_test.reshape(X_test.shape[0], target_shape[0], target_shape[1], target_shape[2])
# show an arbitary example image from training set
show_data_example(12, X_train)
return X_train, y_train, X_test, y_test
image_shape = (28,28,1)
# load and encode mnist data
X_train, y_train, X_test, y_test = load_and_encode(image_shape)
# hyper-parameters
learning_rate = 0.1
momentum = 0.9
dropout = 0.5
batch_size = 128
epochs = 50
decay = 1e-6
number_of_classes = 10
# store required data into a packet to send to various imports
packet = [learning_rate, momentum, dropout, batch_size, epochs, decay,
number_of_classes, image_shape,
X_train, y_train, X_test, y_test]
data = [X_train, y_train, X_test, y_test]
#CNN_model = train_CNN.train_model(packet, save_model = 'True')
CNN_model = load_CNN.load_model(packet) # keras sequential model
#subSVM1, subSVM2, subSVM3, features = train_subSVMs.train(CNN_model, data, c=0.1, save_model = 'True', get_accuracies= 'True')
subSVM1, subSVM2, subSVM3, features = load_subSVMs.load(CNN_model, data, c=0.1, get_accuracies='False')
subSVMs = [subSVM1, subSVM2, subSVM3]
feature1_train, feature1_test,\
feature2_train, feature2_test,\
feature3_train, feature3_test = features
final_SVM = joblib.load('saved_finalSVM.pkl') # sklearn svm trained from features
NUMBER = 48
plt.imshow(np.squeeze(X_train[NUMBER,:,:,:]), cmap=plt.get_cmap('binary'))
# gradients of features wrt to input
import tensorflow.keras.backend as K
gradients = K.gradients(CNN_model.get_layer(name='feature1').output, CNN_model.input) # K.gradients(y,x) for dy/dx
f = K.function([CNN_model.input], gradients)
x = np.expand_dims(X_train[NUMBER,:,:,:],axis=0)
a=f([x])
I'm currently training a Convolutional Neural Network on the MNIST data set using Keras. I'm loading the data set using the format
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
But to reduce iterating through all the data, I want to select only the first 10000 samples from each class 0-9 for X_train and similarly from Y_train. How can I do this?
The MNIST dataset says it returns:
Return:
2 tuples:
X_train, X_test: uint8 array of grayscale image data with shape (nb_samples, 28, 28).
y_train, y_test: uint8 array of digit labels (integers in range 0-9) with shape (nb_samples,).
So you need to slice just the parts you want to keep. I believe the syntax for pandas/numpy is something like:
X_train = X_train[:10000,:,:]
X_test = X_test[:10000,:,:]
y_train = y_train[:10000]
y_test = y_test[:10000]
from keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train[:1000,:,:]
x_test = x_test[:500,:,:]
y_train = y_train[:1000]
y_test = y_test[:500]
print(len(x_train))
print(len(y_train))
print(len(x_test))
print(len(y_test))
#output
> 1000
> 1000
> 500
> 500
I'm trying to do image recognition, so I looked at the CIFAR10 example of Keras.
Before fitting the model to the data, the data (X_train/X_test) needs to be normalize to 0-1 and converted to float32. That's OK when I am using a small data like a CIFAR10. But when the data size increases, it would consume a large amount of memory to convert the data to float32. I do not want to convert all the data to float32.
Can this work (convert data to float32 and normalize) for each mini-batch in keras?
You can do the conversion once and store the normalized, converted data to a file which you load for training, this way you don't need to convert it every time.
For example (normalize.py / python 3):
from keras.datasets import cifar10
import pickle
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
with open('cifar10_normalized.pkl', 'wb') as f:
pickle.dump(((X_train, y_train), (X_test, y_test)), f)
In your code (e.g. train.py) you can then do
import pickle
with open('cifar10_normalized.pkl', 'rb') as f:
(X_train, y_train), (X_test, y_test) = pickle.load(f)
Another possibility is to do the normalization and conversion for each batch. Use model.train_on_batch to run a single batch. For example:
for (x_train,y_train) in yourData:
x_train = x_train.astype(np.float32) / 255
model.train_on_batch(x_train, y_train)
Finally you can also use a python generator for training:
def g():
for (x_train,y_train) in yourData:
x_train = x_train.astype(np.float32) / 255
yield (x_train, y_train)
model.fit_generator(g)