We Keep Coding

Training fails if model is saved beforehand

I noticed that saving a Tensorflow model prior to training causes the training to perform poorly. Obviously the solution is to just save the model later, but I'm curious why this is happening in the first place.
The following code runs fine, and produces the following output:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.Sequential([
layers.Flatten(input_shape=(28,28)),
layers.Dense(16, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(10)
])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer='adam',
loss=loss_fn,
metrics=['accuracy'])
### model.save('my_model') ###
model.fit(x_train, y_train, epochs=3, verbose=1)
Epoch 1/3
1875/1875 [==============================] - 3s 1ms/step - loss: 0.4513 - accuracy: 0.8688
Epoch 2/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.2326 - accuracy: 0.9333
Epoch 3/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1974 - accuracy: 0.9432
However when the second last line is commented out, the training fails badly:
INFO:tensorflow:Assets written to: my_model\assets
Epoch 1/3
1875/1875 [==============================] - 3s 1ms/step - loss: 0.4156 - accuracy: 0.0948
Epoch 2/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.2149 - accuracy: 0.1000
Epoch 3/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1840 - accuracy: 0.0998

It's a known bug of TensorFlow. Use metrics=['sparse_categorical_accuracy'] instead of metrics=['accuracy'] when compiling the model to avoid it.

Keras: "loss: nan - accuracy: 0.0000e+00" coming while training the model

I'm making a model (from this dataset: https://www.kaggle.com/karangadiya/fifa19) which guesses the "overall" of a player given some of his statistics. I've done the required data cleaning but the loss and accuracy is coming as "loss: nan - accuracy: 0.0000e+00."
Below is my code for data cleaning:-
%matplotlib notebook
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow import keras
from sklearn.model_selection import train_test_split
data = pd.read_csv("./data.csv")
data.dropna().replace(np.nan, 0)
data.columns = [x.lower() for x in data.columns]
data = data.select_dtypes(include="number").drop(columns = ["id", "unnamed: 0"], axis=1)
y = data["overall"]
X = data[["crossing", "headingaccuracy", "standingtackle", "gkreflexes"]]
X_train, X_test, y_train, y_test = train_test_split(X,y, random_state=45)
X_train = (X_train - np.max(X_train))/(np.max(X_train) - np.min(X_train))
y_train = np.array(y_train, dtype = 'float32')
And here's my model's code:-
model = keras.models.Sequential([
keras.layers.Dense(4, input_shape=(4,)),
keras.layers.Dense(128, activation=tf.nn.relu),
keras.layers.Dense(1, activation=tf.nn.relu)
])
model.compile(optimizer="sgd", loss="mean_squared_error", metrics=["accuracy"])
model.fit(X_train, y_train, epochs=5)
This was the output while model training:-
Epoch 1/5
427/427 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 2/5
427/427 [==============================] - 1s 1ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 3/5
427/427 [==============================] - 1s 1ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 4/5
427/427 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 5/5
427/427 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.0000e+00
<tensorflow.python.keras.callbacks.History at 0x13b16c6e280>
Thank you for your help!

Issue setting up SciKeras model

I have an existing setup using scikit-learn, but am looking into expanding into deep learning with Keras. I am also using Dask, which recommends using SciKeras.
The way the SciKeras KerasClassifier is currently setup, seems to fit as expected (from the verbose output), but the model seems to have learned nothing at all. I have followed the SciKeras docs here, but I might have overlooked something.
With a Scikit-Learn RF Classifier the kappa score is about 0.44, with
Keras it is about 0.55, and with SciKeras it is 0.0 (clearly an
issue). In the 2. Following SciKeras docs to use Keras where
is the implementation error that prevents a similar result compared to
the one achieved using the 3. Exclusively using Keras below?
Below I have listed the current scikit-learn implementation with RF (as expected output), the output with SciKeras (as actual output), and the output using Keras exclusively (as expected result)
1. Current output using scikit-learn random forest:
def default_classifier():
return RandomForestClassifier(oob_score=True, n_jobs=-1)
... ### Preprocessing stuff...
X_train, X_test, y_train, y_test = splits
# Define the Pipeline
## Classification
model = default_classifier()
model.fit(X_train, y_train)
## Evaluation Metrics
from sklearn.model_selection import cross_val_score
score = cross_val_score(model, X_test, y_test, scoring='accuracy', cv=5, n_jobs=-1, error_score='raise')
print('Mean: %.3f (Std: %.3f)' % (np.mean(score), np.std(score)))
# Verbose with results...
columns, report, true_matrix, pred_matrix = cl.classification_metrics(model, splits, score)
Respective sklearn output:
Test Size: 0.2
Split Shapes: [(79997, 96), (20000, 96), (79997, 12), (20000, 12)]
Mean: 0.374 (Std: 0.006)
Overall: 0.510 Kappa: 0.441
Weighted F1-Score: 0.539
2. Following SciKeras docs to use Keras:
from tensorflow import keras
from scikeras.wrappers import KerasClassifier
from sklearn.model_selection import train_test_split
import numpy as np
def fcn_model(hidden_layer_dim, meta):
# note that meta is a special argument that will be
# handed a dict containing input metadata
n_features_in_ = meta["n_features_in_"]
X_shape_ = meta["X_shape_"]
n_classes_ = meta["n_classes_"]
model = keras.models.Sequential()
model.add(keras.layers.Dense(n_features_in_, input_shape=X_shape_[1:]))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(hidden_layer_dim))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(n_classes_))
model.add(keras.layers.Activation("softmax"))
return model
def get_model_fcn(modelargs={}):
return KerasClassifier(fcn_model,
hidden_layer_dim=128,
epochs=10,
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'],
fit__use_multiprocessing=True,
**modelargs)
... ### Preprocessing stuff...
X_train, X_test, y_train, y_test = splits
# Define the Pipeline
## Classification
model = get_model_fcn()
model.fit(X_train, y_train)
## Evaluation Metrics
from sklearn.model_selection import cross_val_score
score = cross_val_score(model, X_test, y_test, scoring='accuracy', cv=5, n_jobs=-1, error_score='raise')
print('Mean: %.3f (Std: %.3f)' % (np.mean(score), np.std(score)))
columns, report, true_matrix, pred_matrix = cl.classification_metrics(model, splits, score)
Respective scikeras output (result not very good):
Test Size: 0.2
Split Shapes: [(79997, 96), (20000, 96), (79997, 12), (20000, 12)]
Epoch 1/10
2500/2500 [==============================] - 4s 1ms/step - loss: 1.6750 - accuracy: 0.3762
Epoch 2/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.3132 - accuracy: 0.5021
Epoch 3/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.2295 - accuracy: 0.5371
Epoch 4/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1651 - accuracy: 0.5599
Epoch 5/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1178 - accuracy: 0.5806
Epoch 6/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0889 - accuracy: 0.5935
Epoch 7/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0845 - accuracy: 0.5922
Epoch 8/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0548 - accuracy: 0.6043
Epoch 9/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0415 - accuracy: 0.6117
Epoch 10/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0316 - accuracy: 0.6172
Mean: 0.000 (Std: 0.000)
625/625 [==============================] - 0s 700us/step # Here it is running model.predict(X_test)
Overall: 0.130 Kappa: 0.000
Weighted F1-Score: 0.030
3. Exclusively using Keras:
# meta copies what SciKeras passes to the Keras model
meta = {
#'classes_': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]),
#'target_type_': 'multilabel-indicator',
'y_dtype_': np.dtype('uint8'),
'y_ndim_': 2,
'X_dtype_': np.dtype('float32'),
'X_shape_': (79997, 96),
'n_features_in_': 96,
#'target_encoder_': ClassifierLabelEncoder(loss='categorical_crossentropy'),
'n_classes_': 12,
'n_outputs_': 1,
'n_outputs_expected_': 1,
#'feature_encoder_': FunctionTransformer()
}
def fcn_model(hidden_layer_dim, meta):
# note that meta is a special argument that will be
# handed a dict containing input metadata
n_features_in_ = meta["n_features_in_"]
X_shape_ = meta["X_shape_"]
n_classes_ = meta["n_classes_"]
model = keras.models.Sequential()
model.add(keras.layers.Dense(n_features_in_, input_shape=X_shape_[1:]))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(hidden_layer_dim))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(n_classes_))
model.add(keras.layers.Activation("softmax"))
return model
def get_model_fcn(modelargs={}):
model = fcn_model(128, meta)
model.compile(optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
return model
... ### Preprocessing stuff...
X_train, X_test, y_train, y_test = splits
# Define the Pipeline
## Classification
model = get_model_fcn()
model.fit(X_train, y_train, epochs=10)
## Evaluation Metrics
#from sklearn.model_selection import cross_val_score
#score = cross_val_score(model, X_test, y_test, scoring='accuracy', cv=5, n_jobs=-1, #error_score='raise')
#print('Mean: %.3f (Std: %.3f)' % (np.mean(score), np.std(score)))
columns, report, true_matrix, pred_matrix = cl.classification_metrics(model, splits, score)
Expected output from using Keras:
Test Size: 0.2
Split Shapes: [(79997, 96), (20000, 96), (79997, 12), (20000, 12)]
Epoch 1/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.6941 - accuracy: 0.3730
Epoch 2/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.3193 - accuracy: 0.5002
Epoch 3/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.2206 - accuracy: 0.5399
Epoch 4/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1585 - accuracy: 0.5613
Epoch 5/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1221 - accuracy: 0.5758
Epoch 6/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0923 - accuracy: 0.5928
Epoch 7/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0682 - accuracy: 0.5984
Epoch 8/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0611 - accuracy: 0.6046
Epoch 9/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0445 - accuracy: 0.6138
Epoch 10/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0236 - accuracy: 0.6186
Overall: 0.601 Kappa: 0.548
Weighted F1-Score: 0.600

Apparently it was a bug with how it handled multi-class one-hot encoded targets, issue handled here

Make a simple KERAS network for classification

I'm trying to do a simple Keras Neural Network but the model doesn't fit:
Train on 562 samples, validate on 188 samples
Epoch 1/20
562/562 [==============================] - 1s 1ms/step - loss: 8.1130 - acc: 0.4911 - val_loss: 7.6320 - val_acc: 0.5213
Epoch 2/20
562/562 [==============================] - 0s 298us/step - loss: 8.1130 - acc: 0.4911 - val_loss: 7.6320 - val_acc: 0.5213
Epoch 3/20
562/562 [==============================] - 0s 295us/step - loss: 8.1130 - acc: 0.4911 - val_loss: 7.6320 - val_acc: 0.5213
Epoch 4/20
562/562 [==============================] - 0s 282us/step - loss: 8.1130 - acc: 0.4911 - val_loss: 7.6320 - val_acc: 0.5213
Epoch 5/20
562/562 [==============================] - 0s 289us/step - loss: 8.1130 - acc: 0.4911 - val_loss: 7.6320 - val_acc: 0.5213
Epoch 6/20
562/562 [==============================] - 0s 265us/step - loss: 8.1130 - acc: 0.4911 - val_loss: 7.6320 - val_acc: 0.5213
The data base is structured in a CSV file like this:
doc venda img1 img2 v1 v2 gt
RG venda1 img123 img12 [3399, 162675, ...] [3399, 162675, ...] 1
My intent its to use the diff between v1 and v2 vector to answer me if img1 and im2 are from the same class.
The code:
from sklearn.model_selection import train_test_split
(X_train, X_test, Y_train, Y_test) = train_test_split(train, train_labels, test_size=0.25, random_state=42)
# create the model
model = Sequential()
model.add(Dense(10, activation="relu", input_dim=10, kernel_initializer="uniform"))
model.add(Dense(6, activation="relu", kernel_initializer="uniform"))
model.add(Dense(1, activation='sigmoid'))
print(model.summary())
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
model.fit(
np.array(X_train),
np.array(Y_train),
shuffle=True,
epochs=20,
verbose=1,
batch_size=5,
validation_data=(np.array(X_test), np.array(Y_test)),
)
What i'm doing wrong?

Divide the difference vector by some constant number so that the feature vector is in range 0 to 1 or -1 to 1. Right now the values are too big and the loss is coming high. Network learns faster if the data is normalized properly.

I have had success normalizing features using this function. I forget exactly why I use the same mu and sigma from train set on the test and val but I am pretty sure I learned it during the deep.ai course on coursera
def normalize_features(dataset):
mu = np.mean(dataset, axis = 0) # columns
sigma = np.std(dataset, axis = 0)
norm_parameters = {'mu': mu,
'sigma': sigma}
return (dataset-mu)/(sigma+1e-10), norm_parameters
# Normal X data; using same mu and sigma from test set;
x_train, norm_parameters = normalize_features(x_train)
x_val = (x_val-norm_parameters['mu'])/(norm_parameters['sigma']+1e-10)
x_test = (x_test-norm_parameters['mu'])/(norm_parameters['sigma']+1e-10)

Keras 2.0.8 only performs 1 epoch with Python 3.x, 10 with 2.x

If I switch this to Python 2.x, it performs 10. Why is that?
Training a logistic regression model
import keras.backend as K
from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import SGD
from sklearn.model_selection import train_test_split, cross_val_score
X_train, X_test, y_train, y_test = train_test_split(X, y,
test_size = 0.3,
random_state = 42)
# NOTE: If I run this in Python 3.x, it only performs 1 Epoch
K.clear_session()
model = Sequential()
model.add(Dense(1, input_shape=(4,), activation='sigmoid'))
model.compile(loss = 'binary_crossentropy',
optimizer= 'sgd',
metrics = ['accuracy'])
# Saved the result of the fitting, to display the history as a data frame & see how the model does
history = model.fit (X_train, y_train)
result = model.evaluate(X_test, y_test)
Output:
Epoch 1/10
960/960 [==============================] - 0s - loss: 0.7943 - acc: 0.5219
Epoch 2/10
960/960 [==============================] - 0s - loss: 0.7338 - acc: 0.5469
Epoch 3/10
960/960 [==============================] - 0s - loss: 0.6847 - acc: 0.5688
Epoch 4/10
960/960 [==============================] - 0s - loss: 0.6446 - acc: 0.6177
Epoch 5/10
960/960 [==============================] - 0s - loss: 0.6113 - acc: 0.6719
Epoch 6/10
960/960 [==============================] - 0s - loss: 0.5832 - acc: 0.7000
Epoch 7/10
960/960 [==============================] - 0s - loss: 0.5591 - acc: 0.7177
Epoch 8/10
960/960 [==============================] - 0s - loss: 0.5381 - acc: 0.7365
Epoch 9/10
960/960 [==============================] - 0s - loss: 0.5196 - acc: 0.7542
Epoch 10/10
960/960 [==============================] - 0s - loss: 0.5031 - acc: 0.7688
32/412 [=>............................] - ETA: 0s

The fit function has parameter epochs with default value 1.
fit(self, x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,
validation_split=0.0, validation_data=None, shuffle=True, class_weight=None,
sample_weight=None, initial_epoch=0, steps_per_epoch=None,
validation_steps=None)
However, the default used to be 10. See the changes to fit in models.py in this commit for example. You most likely have an older version of Keras with Python 2.