We Keep Coding

Python Keras - Large MSE

I am new to KerasRegressor. I am trying to run neural net following this codes:
import tensorflow as tf
import numpy as np
from sklearn import datasets, linear_model
from sklearn.model_selection import cross_val_score, KFold
from keras.models import Sequential
from sklearn.metrics import accuracy_score
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
seed = 1
def baseline_model():
model = Sequential()
model.add(Dense(10, input_dim=10, activation='relu'))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
return model
estimator = KerasRegressor(build_fn=baseline_model, nb_epoch=100, batch_size=100, verbose=False)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X_train,y_train, cv=kfold)
print("Results: %.2f (%.2f) MSE" % (results.mean(), results.std()))
estimator.fit(X_train, y_train)
prediction = estimator.predict(X_test)
Then I stumbled the following errors:
ValueError: Input 0 of layer sequential_45 is incompatible with the layer: expected axis -1 of input shape to have value 10 but received input with shape [None, 39]
UPDATE: I fixed the error, it was in input_dim, which should be change to X_train.shape[-1]. However, I have a very large error, even after I try normalized X_train:
Results: -1674844.52 (3109620.06) MSE
I wonder how I should solve this? Thanks!

How to get weights from tensorflow fully_connected with Google Colab

I'm trying to extract the weights from a model after training it. Here's a code.
1- how to save the model in the previously uploaded directory path?
2. how to get weights?
3. I have some csv file as the same used for modeling but without class. how can I estimate the classes with this model?
# Preprocess
import pandas as pd
import keras
from sklearn.model_selection import train_test_split
# Prepare data
churn = pd.read_csv('Churn_Modelling3.csv')
# split data into train and test sets
x_train, x_test, y_train, y_test = train_test_split(churn.iloc[:,0:10],churn.iloc[:, 10], test_size=0.2)
# Create a Model
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import MaxPool2D
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
model = Sequential()
# Input layer
model.add(Dense(32, activation=tf.nn.sigmoid,input_dim=x_train.shape[1]))
# Hidden layers
model.add(Dense(64, activation=tf.nn.sigmoid))
model.add(Dense(32, activation=tf.nn.sigmoid))
# Output layer
model.add(Dense(1, activation=tf.nn.sigmoid))
# Compile the Model
from keras.optimizers import SGD
sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile( optimizer= 'sgd', loss='binary_crossentropy', metrics=['accuracy'])
# Fit the model
history = model.fit(x_train, y_train, epochs=20, batch_size=128,validation_data=(x_test, y_test))
# ٍEvaluate the Model
score = model.evaluate(x_test, y_test, batch_size=128)

How to save the model in the previously uploaded directory path? => You can use any of the below commands,
`model.save_weights('./Weights/Weights')` or
`tf.keras.experimental.export_saved_model` or
`model.save`
How to get weights? => We can Load the Weights using any of the below commands:
model.load_weights('my_model_weights.h5') or
keras.models.load_model('my_model.h5') or
tf.keras.experimental.load_from_saved_model
I have some csv file as the same used for modeling but without class. how can I estimate the classes with this model? => One of the ways to do it is to iterate the below code for each instance of CSV data Label.
ynew = model.predict(Xnew)
Fore more information, you can refer to the below links:
https://www.tensorflow.org/tutorials/keras/save_and_restore_models
https://www.tensorflow.org/guide/saved_model
https://www.tensorflow.org/api_docs/python/tf/keras/Model

Why i have a small accuracy?

I have a code to train MNIST dataset to work on the street view house number project, but when I run the code I have acc = 0,1
Import libraries and modules
import numpy as np
np.random.seed(123) # for reproducibility
from keras import backend as K
K.set_image_dim_ordering('th')
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten
from tensorflow.keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras.datasets import mnist
from keras.models import load_model
from keras.utils import CustomObjectScope
from keras.initializers import glorot_uniform
4. Load pre-shuffled MNIST data into train and test sets
(X_train, Y_train), (X_test, Y_test) = mnist.load_data()
X_train = keras.utils.normalize(X_train,axis=1)
X_test = keras.utils.normalize(X_test, axis=1)
7. Define model architecture
model = Sequential()
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(128, activation='relu'))
model.add(Dense(10, activation='softmax'))
8. Compile model
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
9. Fit model on training data
model.fit(np.array(X_train), np.array(Y_train), batch_size=32, epochs=3,verbose=1)

In step 4, are you normalizing the data correctly? If I recall correctly X_train has shape batch, width, height. I don't really know what you would want to normalize, but axis=1 doesn't seem it's supposed to be there. I think you should the normalization.
If you still have low accuracy, try training more epochs than 3. 3 epochs are not that many.

There are several reasons why you have an accuracy like that.
Your data is not normalized correctly.
You are trying to do image recognition with 3 dense layers in 3 epochs, which will not work.
There is no optimization in your code.
Look at the https://keras.io/examples/mnist_cnn/ . It is the Keras documentation on working with the MNIST data using neural network.

How to train a CNN model on 2 classes of 100 samples each and then test it on 200 new samples?

I've got 2 classes for my training set: Birds(100 samples) and no_birds(100) samples. And, the test set is unlabelled consisting of 200 test samples (mixed with birds and no_birds). For every sample in the test set I intend to classify it as bird or no_bird using CNN with Keras.
import numpy as np
import keras
from keras import backend as K
from keras.models import Sequential
from keras.layers import Activation
from keras.layers.core import Dense, Flatten
from keras.optimizers import Adam
from keras.metrics import categorical_crossentropy
from keras.preprocessing.image import ImageDataGenerator
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import *
from sklearn.metrics import confusion_matrix
import itertools
import matplotlib.pyplot as plt
train_path = 'dataset/train_set'
test_path = 'dataset/test_set'
train_batches = ImageDataGenerator().flow_from_directory(train_path, target_size=(224,224), classes=['bird', 'no_bird'], batch_size=10) # bird directory consisting of 100
test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(224,224), classes=['unknown'], batch_size=10)
model = Sequential([
Conv2D(32, (3, 3), activation='relu', input_shape=(224,224,3)),
Flatten(),
Dense(2, activation='softmax'),
])
model.compile(Adam(lr=.0001), loss='categorical_crossentropy', metrics=['accuracy'])
model.fit_generator(train_batches, steps_per_epoch=20, validation_data=test_batches, validation_steps=20, epochs=10, verbose=2)
Error I'm getting at the last step is this:
ValueError: Error when checking target: expected dense_1 to have shape (2,) but got array with shape (1,)
Now, I know it could be probably because of test_set having only 1 directory, since it's unlabelled. Correct me if I'm wrong. What should I do to make this work?

It seems your test set is unlabelled. Remove validation arguments from model.fit. It should be:
model.fit_generator(train_batches, steps_per_epoch=20, epochs=10, verbose=2)
You can't validate without labels.

the line test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(224,224), classes=['unknown'], batch_size=10) is wrong
you should do test_batches = ImageDataGenerator().flow_from_directory(test_path, target_size=(224,224), classes=['bird', 'no_bird'], batch_size=10) still. That way you can score your predictions
folowup information:
when you look at https://keras.io/models/sequential/, it says
validation_data: tuple (x_val, y_val) or tuple (x_val, y_val, val_sample_weights) on which to evaluate the loss and any model metrics at the end of each epoch. The model will not be trained on this data. This will override validation_split.
Your test data must be the same shape as your train data. You'll have to organize the test data directory so it's structured the same as the training data

Why my training speed in Keras with multi_gpu_model is worse than single gpu?

My Keras version is 2.0.9, and using tensorflow backend.
I tried to implement multi_gpu_model in keras. However, training with 4 gpus was even worse than 1 gpu in practice. I got 25sec for 1 gpu, and 50sec for 4 gpus. Could you give me the reason why this happens?
/blog for multi_gpu_model
https://www.pyimagesearch.com/2017/10/30/how-to-multi-gpu-training-with-keras-python-and-deep-learning/
I used this commend for 1 gpu
CUDA_VISIBLE_DEVICES=0 python gpu_test.py
and for 4 gpus,
python gpu_test.py
-Here is source code for training.
from keras.datasets import mnist
from keras.layers import Input, Dense, merge
from keras.layers.core import Lambda
from keras.models import Model
from keras.utils import to_categorical
from keras.utils.training_utils import multi_gpu_model
import time
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
inputs = Input(shape=(784,))
x = Dense(4096, activation='relu')(inputs)
x = Dense(2048, activation='relu')(x)
x = Dense(512, activation='relu')(x)
x = Dense(64, activation='relu')(x)
predictions = Dense(10, activation='softmax')(x)
model = Model(inputs=inputs, outputs=predictions)
'''
m_model = multi_gpu_model(model, 4)
m_model.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
m_model.summary()
a=time.time()
m_model.fit(x_train, y_train, batch_size=128, epochs=5)
print time.time() - a
a=time.time()
m_model.predict(x=x_test, batch_size=128)
print time.time() - a
'''
model.compile(optimizer='rmsprop',
loss='categorical_crossentropy',
metrics=['accuracy'])
model.summary()
a=time.time()
model.fit(x_train, y_train, batch_size=128, epochs=5)
print time.time() - a
a=time.time()
model.predict(x=x_test, batch_size=128)
print time.time() - a
And this is gpu state with running 4 gpus.

I can give you what I think is the answer, but I don't have it fully working myself. I was tipped off onto this by a bug report, but in the source code for multi_gpu_model it says:
# Instantiate the base model (or "template" model).
# We recommend doing this with under a CPU device scope,
# so that the model's weights are hosted on CPU memory.
# Otherwise they may end up hosted on a GPU, which would
# complicate weight sharing.
with tf.device('/cpu:0'):
model = Xception(weights=None,
input_shape=(height, width, 3),
classes=num_classes)
I think this is the problem. I'm still working on making it work myself, though.