Issue setting up SciKeras model - python
I have an existing setup using scikit-learn, but am looking into expanding into deep learning with Keras. I am also using Dask, which recommends using SciKeras.
The way the SciKeras KerasClassifier is currently setup, seems to fit as expected (from the verbose output), but the model seems to have learned nothing at all. I have followed the SciKeras docs here, but I might have overlooked something.
With a Scikit-Learn RF Classifier the kappa score is about 0.44, with
Keras it is about 0.55, and with SciKeras it is 0.0 (clearly an
issue). In the 2. Following SciKeras docs to use Keras where
is the implementation error that prevents a similar result compared to
the one achieved using the 3. Exclusively using Keras below?
Below I have listed the current scikit-learn implementation with RF (as expected output), the output with SciKeras (as actual output), and the output using Keras exclusively (as expected result)
1. Current output using scikit-learn random forest:
def default_classifier():
return RandomForestClassifier(oob_score=True, n_jobs=-1)
... ### Preprocessing stuff...
X_train, X_test, y_train, y_test = splits
# Define the Pipeline
## Classification
model = default_classifier()
model.fit(X_train, y_train)
## Evaluation Metrics
from sklearn.model_selection import cross_val_score
score = cross_val_score(model, X_test, y_test, scoring='accuracy', cv=5, n_jobs=-1, error_score='raise')
print('Mean: %.3f (Std: %.3f)' % (np.mean(score), np.std(score)))
# Verbose with results...
columns, report, true_matrix, pred_matrix = cl.classification_metrics(model, splits, score)
Respective sklearn output:
Test Size: 0.2
Split Shapes: [(79997, 96), (20000, 96), (79997, 12), (20000, 12)]
Mean: 0.374 (Std: 0.006)
Overall: 0.510 Kappa: 0.441
Weighted F1-Score: 0.539
2. Following SciKeras docs to use Keras:
from tensorflow import keras
from scikeras.wrappers import KerasClassifier
from sklearn.model_selection import train_test_split
import numpy as np
def fcn_model(hidden_layer_dim, meta):
# note that meta is a special argument that will be
# handed a dict containing input metadata
n_features_in_ = meta["n_features_in_"]
X_shape_ = meta["X_shape_"]
n_classes_ = meta["n_classes_"]
model = keras.models.Sequential()
model.add(keras.layers.Dense(n_features_in_, input_shape=X_shape_[1:]))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(hidden_layer_dim))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(n_classes_))
model.add(keras.layers.Activation("softmax"))
return model
def get_model_fcn(modelargs={}):
return KerasClassifier(fcn_model,
hidden_layer_dim=128,
epochs=10,
optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'],
fit__use_multiprocessing=True,
**modelargs)
... ### Preprocessing stuff...
X_train, X_test, y_train, y_test = splits
# Define the Pipeline
## Classification
model = get_model_fcn()
model.fit(X_train, y_train)
## Evaluation Metrics
from sklearn.model_selection import cross_val_score
score = cross_val_score(model, X_test, y_test, scoring='accuracy', cv=5, n_jobs=-1, error_score='raise')
print('Mean: %.3f (Std: %.3f)' % (np.mean(score), np.std(score)))
columns, report, true_matrix, pred_matrix = cl.classification_metrics(model, splits, score)
Respective scikeras output (result not very good):
Test Size: 0.2
Split Shapes: [(79997, 96), (20000, 96), (79997, 12), (20000, 12)]
Epoch 1/10
2500/2500 [==============================] - 4s 1ms/step - loss: 1.6750 - accuracy: 0.3762
Epoch 2/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.3132 - accuracy: 0.5021
Epoch 3/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.2295 - accuracy: 0.5371
Epoch 4/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1651 - accuracy: 0.5599
Epoch 5/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1178 - accuracy: 0.5806
Epoch 6/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0889 - accuracy: 0.5935
Epoch 7/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0845 - accuracy: 0.5922
Epoch 8/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0548 - accuracy: 0.6043
Epoch 9/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0415 - accuracy: 0.6117
Epoch 10/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0316 - accuracy: 0.6172
Mean: 0.000 (Std: 0.000)
625/625 [==============================] - 0s 700us/step # Here it is running model.predict(X_test)
Overall: 0.130 Kappa: 0.000
Weighted F1-Score: 0.030
3. Exclusively using Keras:
# meta copies what SciKeras passes to the Keras model
meta = {
#'classes_': array([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]),
#'target_type_': 'multilabel-indicator',
'y_dtype_': np.dtype('uint8'),
'y_ndim_': 2,
'X_dtype_': np.dtype('float32'),
'X_shape_': (79997, 96),
'n_features_in_': 96,
#'target_encoder_': ClassifierLabelEncoder(loss='categorical_crossentropy'),
'n_classes_': 12,
'n_outputs_': 1,
'n_outputs_expected_': 1,
#'feature_encoder_': FunctionTransformer()
}
def fcn_model(hidden_layer_dim, meta):
# note that meta is a special argument that will be
# handed a dict containing input metadata
n_features_in_ = meta["n_features_in_"]
X_shape_ = meta["X_shape_"]
n_classes_ = meta["n_classes_"]
model = keras.models.Sequential()
model.add(keras.layers.Dense(n_features_in_, input_shape=X_shape_[1:]))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(hidden_layer_dim))
model.add(keras.layers.Activation("relu"))
model.add(keras.layers.Dense(n_classes_))
model.add(keras.layers.Activation("softmax"))
return model
def get_model_fcn(modelargs={}):
model = fcn_model(128, meta)
model.compile(optimizer='adam',
loss='categorical_crossentropy',
metrics=['accuracy'])
return model
... ### Preprocessing stuff...
X_train, X_test, y_train, y_test = splits
# Define the Pipeline
## Classification
model = get_model_fcn()
model.fit(X_train, y_train, epochs=10)
## Evaluation Metrics
#from sklearn.model_selection import cross_val_score
#score = cross_val_score(model, X_test, y_test, scoring='accuracy', cv=5, n_jobs=-1, #error_score='raise')
#print('Mean: %.3f (Std: %.3f)' % (np.mean(score), np.std(score)))
columns, report, true_matrix, pred_matrix = cl.classification_metrics(model, splits, score)
Expected output from using Keras:
Test Size: 0.2
Split Shapes: [(79997, 96), (20000, 96), (79997, 12), (20000, 12)]
Epoch 1/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.6941 - accuracy: 0.3730
Epoch 2/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.3193 - accuracy: 0.5002
Epoch 3/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.2206 - accuracy: 0.5399
Epoch 4/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1585 - accuracy: 0.5613
Epoch 5/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.1221 - accuracy: 0.5758
Epoch 6/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0923 - accuracy: 0.5928
Epoch 7/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0682 - accuracy: 0.5984
Epoch 8/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0611 - accuracy: 0.6046
Epoch 9/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0445 - accuracy: 0.6138
Epoch 10/10
2500/2500 [==============================] - 3s 1ms/step - loss: 1.0236 - accuracy: 0.6186
Overall: 0.601 Kappa: 0.548
Weighted F1-Score: 0.600
Apparently it was a bug with how it handled multi-class one-hot encoded targets, issue handled here
Related
Retrain a keras model
I would like to retrain a tensorflow.keras model, and I tried to compare both models: Code 1: keras.backend.clear_session() np.random.seed(42) tf.random.set_seed(42) optimizer = tf.keras.optimizers.Adam(lr=1e-2) model = keras.models.Sequential([keras.layers.Dense(1, input_shape=[8])]) model.compile(loss="mse", optimizer=optimizer) model.fit(X_train_scaled, y_train, epochs=3) model.fit(X_train_scaled, y_train, epochs=3) Code 2: keras.backend.clear_session() np.random.seed(42) tf.random.set_seed(42) optimizer = tf.keras.optimizers.Adam(lr=1e-2) model = keras.models.Sequential([keras.layers.Dense(1, input_shape=[8])]) model.compile(loss="mse", optimizer=optimizer) model.fit(X_train_scaled, y_train, epochs=6) In code 1, I get: Epoch 1/3 363/363 [==============================] - 0s 330us/step - loss: 1.8888 Epoch 2/3 363/363 [==============================] - 0s 342us/step - loss: 0.5772 Epoch 3/3 363/363 [==============================] - 0s 397us/step - loss: 0.5508 Epoch 1/3 363/363 [==============================] - 0s 470us/step - loss: 0.5428 Epoch 2/3 363/363 [==============================] - 0s 486us/step - loss: 0.5338 Epoch 3/3 363/363 [==============================] - 0s 479us/step - loss: 0.5519 In code 2, I get: Epoch 1/6 363/363 [==============================] - 0s 332us/step - loss: 1.8888 Epoch 2/6 363/363 [==============================] - 0s 322us/step - loss: 0.5772 Epoch 3/6 363/363 [==============================] - 0s 333us/step - loss: 0.5508 Epoch 4/6 363/363 [==============================] - 0s 331us/step - loss: 0.5413 Epoch 5/6 363/363 [==============================] - 0s 371us/step - loss: 0.5440 Epoch 6/6 363/363 [==============================] - 0s 356us/step - loss: 0.5318 If you want to check on your own computer, I use this dataset: import tensorflow as tf from tensorflow import keras import numpy as np import os from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler housing = fetch_california_housing() X_train_full, X_test, y_train_full, y_test = train_test_split( housing.data, housing.target.reshape(-1, 1), random_state=42) X_train, X_valid, y_train, y_valid = train_test_split( X_train_full, y_train_full, random_state=42) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_valid_scaled = scaler.transform(X_valid) X_test_scaled = scaler.transform(X_test) I compared both ways of training a model: first is by running 3 epochs, then 3 epochs second is by running 6 epochs It's unclear where the problem is. I use tensorflow 2.2.2 (or 2.3 provides the same results) I don't understand why they don't provide the same result.
Training fails if model is saved beforehand
I noticed that saving a Tensorflow model prior to training causes the training to perform poorly. Obviously the solution is to just save the model later, but I'm curious why this is happening in the first place.
The following code runs fine, and produces the following output:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
model = tf.keras.Sequential([
layers.Flatten(input_shape=(28,28)),
layers.Dense(16, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(10)
])
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer='adam',
loss=loss_fn,
metrics=['accuracy'])
### model.save('my_model') ###
model.fit(x_train, y_train, epochs=3, verbose=1)
Epoch 1/3
1875/1875 [==============================] - 3s 1ms/step - loss: 0.4513 - accuracy: 0.8688
Epoch 2/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.2326 - accuracy: 0.9333
Epoch 3/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1974 - accuracy: 0.9432
However when the second last line is commented out, the training fails badly:
INFO:tensorflow:Assets written to: my_model\assets
Epoch 1/3
1875/1875 [==============================] - 3s 1ms/step - loss: 0.4156 - accuracy: 0.0948
Epoch 2/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.2149 - accuracy: 0.1000
Epoch 3/3
1875/1875 [==============================] - 2s 1ms/step - loss: 0.1840 - accuracy: 0.0998
It's a known bug of TensorFlow. Use metrics=['sparse_categorical_accuracy'] instead of metrics=['accuracy'] when compiling the model to avoid it.
learn a new set of data from existing model for enhancing it (tensorflow - keras - callbacks)
I make a learning on a dataset, and everything is ok. Sometimes, changes occur, and I've got some new data. I'd like to "continue" the learning from my existing model, with the new set of data, without begining from scratch again. Here is a simple example to show the problematic and where I'm stucked. import tensorflow as tf mnist = tf.keras.datasets.mnist # get the data (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0 # split in 2 parts to get 2 set of data x_train1 = x_train[:5000] x_train2 = x_train[5000:10000] y_test1 = y_test[:5000] y_test2 = y_test[5000:] # set the model model = tf.keras.models.Sequential([ tf.keras.layers.Flatten(input_shape=(28, 28)), tf.keras.layers.Dense(128, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(10)]) loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) # Compile the model model.compile(optimizer='adam', loss=loss_fn, metrics=['accuracy']) # set the callback checkpoint_path = "CHECKPOINTS/cp.ckpt" cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path, monitor='accuracy', mode="max", save_best_only=False, save_weights_only=False, save_freq="epoch", verbose=0) Then I fit my first set of data : # fit the 1st dataset model.fit(x_train1, y_test1, epochs=5,callbacks=[cp_callback],verbose=1) Output: Epoch 1/5 157/157 [==============================] - 0s 897us/step - loss: >2.3518 - accuracy: >0.1075 Epoch 2/5 157/157 [==============================] - 0s 752us/step - loss: >2.2833 - accuracy: >0.1438 Epoch 3/5 157/157 [==============================] - 0s 731us/step - loss: >2.2656 - accuracy: >0.1564 Epoch 4/5 157/157 [==============================] - 0s 755us/step - loss: >2.2388 - accuracy: >0.1719 Epoch 5/5 157/157 [==============================] - 0s 759us/step - loss: >2.2117 - accuracy: >0.1901 Then my 2nd one : # fit the 2nd one model.fit(x_train2, y_test2, epochs=5,callbacks=[cp_callback],verbose=1) Output: Epoch 1/5 157/157 [==============================] - 0s 943us/step - loss: >2.3240 - accuracy: >0.0964 Epoch 2/5 157/157 [==============================] - 0s 778us/step - loss: >2.2881 - accuracy: >0.1238 Epoch 3/5 157/157 [==============================] - 0s 805us/step - loss: >2.2688 - accuracy: >0.1514 Epoch 4/5 157/157 [==============================] - 0s 814us/step - loss: >2.2498 - accuracy: >0.1496 Epoch 5/5 157/157 [==============================] - 0s 1ms/step - loss: >2.2289 - accuracy: 0.1704 As you can see, it begins from 0 again the training, without keeping the existing train made before. How could I do that. Thanx by advance.
Keras: "loss: nan - accuracy: 0.0000e+00" coming while training the model
I'm making a model (from this dataset: https://www.kaggle.com/karangadiya/fifa19) which guesses the "overall" of a player given some of his statistics. I've done the required data cleaning but the loss and accuracy is coming as "loss: nan - accuracy: 0.0000e+00."
Below is my code for data cleaning:-
%matplotlib notebook
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow import keras
from sklearn.model_selection import train_test_split
data = pd.read_csv("./data.csv")
data.dropna().replace(np.nan, 0)
data.columns = [x.lower() for x in data.columns]
data = data.select_dtypes(include="number").drop(columns = ["id", "unnamed: 0"], axis=1)
y = data["overall"]
X = data[["crossing", "headingaccuracy", "standingtackle", "gkreflexes"]]
X_train, X_test, y_train, y_test = train_test_split(X,y, random_state=45)
X_train = (X_train - np.max(X_train))/(np.max(X_train) - np.min(X_train))
y_train = np.array(y_train, dtype = 'float32')
And here's my model's code:-
model = keras.models.Sequential([
keras.layers.Dense(4, input_shape=(4,)),
keras.layers.Dense(128, activation=tf.nn.relu),
keras.layers.Dense(1, activation=tf.nn.relu)
])
model.compile(optimizer="sgd", loss="mean_squared_error", metrics=["accuracy"])
model.fit(X_train, y_train, epochs=5)
This was the output while model training:-
Epoch 1/5
427/427 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 2/5
427/427 [==============================] - 1s 1ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 3/5
427/427 [==============================] - 1s 1ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 4/5
427/427 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.0000e+00
Epoch 5/5
427/427 [==============================] - 1s 2ms/step - loss: nan - accuracy: 0.0000e+00
<tensorflow.python.keras.callbacks.History at 0x13b16c6e280>
Thank you for your help!
Keras 2.0.8 only performs 1 epoch with Python 3.x, 10 with 2.x
If I switch this to Python 2.x, it performs 10. Why is that? Training a logistic regression model import keras.backend as K from keras.models import Sequential from keras.layers import Dense, Activation from keras.optimizers import SGD from sklearn.model_selection import train_test_split, cross_val_score X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 42) # NOTE: If I run this in Python 3.x, it only performs 1 Epoch K.clear_session() model = Sequential() model.add(Dense(1, input_shape=(4,), activation='sigmoid')) model.compile(loss = 'binary_crossentropy', optimizer= 'sgd', metrics = ['accuracy']) # Saved the result of the fitting, to display the history as a data frame & see how the model does history = model.fit (X_train, y_train) result = model.evaluate(X_test, y_test) Output: Epoch 1/10 960/960 [==============================] - 0s - loss: 0.7943 - acc: 0.5219 Epoch 2/10 960/960 [==============================] - 0s - loss: 0.7338 - acc: 0.5469 Epoch 3/10 960/960 [==============================] - 0s - loss: 0.6847 - acc: 0.5688 Epoch 4/10 960/960 [==============================] - 0s - loss: 0.6446 - acc: 0.6177 Epoch 5/10 960/960 [==============================] - 0s - loss: 0.6113 - acc: 0.6719 Epoch 6/10 960/960 [==============================] - 0s - loss: 0.5832 - acc: 0.7000 Epoch 7/10 960/960 [==============================] - 0s - loss: 0.5591 - acc: 0.7177 Epoch 8/10 960/960 [==============================] - 0s - loss: 0.5381 - acc: 0.7365 Epoch 9/10 960/960 [==============================] - 0s - loss: 0.5196 - acc: 0.7542 Epoch 10/10 960/960 [==============================] - 0s - loss: 0.5031 - acc: 0.7688 32/412 [=>............................] - ETA: 0s
The fit function has parameter epochs with default value 1. fit(self, x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None, validation_split=0.0, validation_data=None, shuffle=True, class_weight=None, sample_weight=None, initial_epoch=0, steps_per_epoch=None, validation_steps=None) However, the default used to be 10. See the changes to fit in models.py in this commit for example. You most likely have an older version of Keras with Python 2.