We Keep Coding

Keras weighted_metrics does not include sample weights in calculation [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I am training a CNN model with a 2D tensor of shape (400,22) as both input and output. I am using categorical_crossentropy both as loss and metric. However the loss/metrics values are very different.
My model is somewhat like this:
1. Using sample weights, and passing metrics with metrics= in model.compile.
# Imports
import tensorflow as tf
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
from tensorflow.keras.regularizers import *
from tensorflow.keras import *
import numpy as np
# Build the model
X_input = Input(shape=(400,22))
X = Conv1D(filters=32, kernel_size=2, activation='elu',
kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4),
padding='same')(X_input)
X = Dropout(0.2)(X)
X = Conv1D(filters=32, kernel_size=2, activation='elu',
kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4),
padding='same')(X)
X = Dropout(0.2)(X)
y = Conv1D(filters=22, kernel_size=1, activation='softmax',
kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4),
padding='same')(X)
model = Model(X_input, y, name='mymodel')
# Compile and train the model (with metrics=[])
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(20,400,22)
ytrain = np.random.rand(20,400,22)
np.random.seed(0)
sample_weight = np.random.choice([0.01, 0.1, 1], size=20)
history = model.fit(x=Xtrain, y=ytrain, sample_weight=sample_weight, epochs=4)
Epoch 1/4
1/1 [==============================] - 0s 824us/step - loss: 10.2952 - categorical_crossentropy: 34.9296
Epoch 2/4
1/1 [==============================] - 0s 785us/step - loss: 10.2538 - categorical_crossentropy: 34.7858
Epoch 3/4
1/1 [==============================] - 0s 772us/step - loss: 10.2181 - categorical_crossentropy: 34.6719
Epoch 4/4
1/1 [==============================] - 0s 766us/step - loss: 10.1903 - categorical_crossentropy: 34.5797
From the results, it is evident that Keras is not using sample weights in the calculation of metrics, hence it is larger than the loss. If we change the sample weights to ones, we get the following:
2. Sample weights = ones, passing metrics with metrics= in `model.compile.
# Compile and train the model
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(20,400,22)
ytrain = np.random.rand(20,400,22)
np.random.seed(0)
sample_weight = np.ones((20,))
history = model.fit(x=Xtrain, y=ytrain, sample_weight=sample_weight, epochs=4)
Epoch 1/4
1/1 [==============================] - 0s 789us/step - loss: 35.2659 - categorical_crossentropy: 35.2573
Epoch 2/4
1/1 [==============================] - 0s 792us/step - loss: 35.0647 - categorical_crossentropy: 35.0562
Epoch 3/4
1/1 [==============================] - 0s 778us/step - loss: 34.9301 - categorical_crossentropy: 34.9216
Epoch 4/4
1/1 [==============================] - 0s 736us/step - loss: 34.8076 - categorical_crossentropy: 34.7991
Now the metrics and loss are quite close with sample weights of ones. I understand that the loss is slightly larger than metrics due to the effects of dropout, regularization, and the fact that the metric is computed at the end of each epoch, whereas the loss is the average over the batches in the training.
How can I get the metrics to include the sample weights??
3. UPDATED: using sample weights, and passing metrics with weighted_metrics= in model.compile.
It was suggested that I used weighted_metrics=[...] instead of metrics=[...] in model.compile. However, Keras still does not include the sample weights in the evaluation of the metrics.
# Compile and train the model
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
weighted_metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(20,400,22)
ytrain = np.random.rand(20,400,22)
np.random.seed(0)
sample_weight = np.random.choice([0.01, 0.1, 1], size=20)
history = model.fit(x=Xtrain, y=ytrain, sample_weight=sample_weight, epochs=4)
Epoch 1/4
1/1 [==============================] - 0s 764us/step - loss: 10.2581 - categorical_crossentropy: 34.9224
Epoch 2/4
1/1 [==============================] - 0s 739us/step - loss: 10.2251 - categorical_crossentropy: 34.8100
Epoch 3/4
1/1 [==============================] - 0s 755us/step - loss: 10.1854 - categorical_crossentropy: 34.6747
Epoch 4/4
1/1 [==============================] - 0s 746us/step - loss: 10.1631 - categorical_crossentropy: 34.5990
What can be done to ensure that the sample weights are evaluated in the metrics?

Keras does not automatically include sample weights in the evaluation of metrics. That's why there is a huge difference between the loss and the metrics.
If you'll like to include sample weights when evaluating metrics, pass them as weighted_metrics rather than metrics.
model.compile(optimizer=Adam(1e-3),
loss=tf.keras.losses.categorical_crossentropy,
weighted_metrics=[tf.keras.losses.categorical_crossentropy]))

First of all, categorical cross-entropy is usually not used as a metric. Secondly, you are doing some type of seq2seq task, I hope you design the model with that intention.
Finally, in your setup, using sample_weight only works on the loss, it has no effect on the metrics or validation. There are other small bugs in your code too. Here is the fixed working code:
ref: TF 2.3.0 training keras model using tf dataset with sample weights does not apply to metrics (why sample_weight only works on loss)
import tensorflow as tf
from tensorflow.keras.layers import *
from tensorflow.keras.optimizers import *
from tensorflow.keras import *
import numpy as np
X_input = Input(shape=(400,22))
X = Conv1D(filters=32, kernel_size=2, activation='elu', kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4), padding='same')(X_input)
X = Dropout(0.2)(X)
X = Conv1D(filters=32, kernel_size=2, activation='elu', kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4), padding='same')(X)
X = Dropout(0.2)(X)
y = Conv1D(filters=22, kernel_size=1, activation='softmax', kernel_regularizer=L2(1e-4), bias_regularizer=L2(1e-4), padding='same')(X)
model = Model(X_input, y, name='mymodel')
model.compile(optimizer=Adam(1e-3), loss=tf.keras.losses.categorical_crossentropy,
metrics=[tf.keras.losses.categorical_crossentropy])
Xtrain = np.random.rand(10,400,22)
ytrain = np.random.rand(10,400,22)
history = model.fit(Xtrain, ytrain, sample_weight=np.ones(10), epochs=10)
Epoch 1/10
1/1 [==============================] - 1s 719ms/step - loss: 35.4521 - categorical_crossentropy: 35.4437
Epoch 2/10
1/1 [==============================] - 0s 20ms/step - loss: 35.5138 - categorical_crossentropy: 35.5054
Epoch 3/10
1/1 [==============================] - 0s 19ms/step - loss: 35.5984 - categorical_crossentropy: 35.5900
Epoch 4/10
1/1 [==============================] - 0s 19ms/step - loss: 35.6617 - categorical_crossentropy: 35.6533
Epoch 5/10
1/1 [==============================] - 0s 19ms/step - loss: 35.7807 - categorical_crossentropy: 35.7723
Epoch 6/10
1/1 [==============================] - 0s 19ms/step - loss: 35.9045 - categorical_crossentropy: 35.8961
Epoch 7/10
1/1 [==============================] - 0s 18ms/step - loss: 36.0590 - categorical_crossentropy: 36.0505
Epoch 8/10
1/1 [==============================] - 0s 19ms/step - loss: 36.2040 - categorical_crossentropy: 36.1956
Epoch 9/10
1/1 [==============================] - 0s 18ms/step - loss: 36.4169 - categorical_crossentropy: 36.4084
Epoch 10/10
1/1 [==============================] - 0s 32ms/step - loss: 36.6622 - categorical_crossentropy: 36.6538
Here, if you use no sample_weight or 1 for each sample, you will get close/similar categorical cross-entropy.
Use weighted_metrics according to docs.

Using Keras to give a score for combination of words

I have a file with a set of "words" for example:
1a 9( 9j = 2453
3a 4( 6j 0s = 2309
1 7( 8ll = 4934
It looks like random data but it isn't, it has a score for each set of "words". My file consists of about 1million lines and there is definately patterns in it. There are about 3600 unqiue individual words.
The end column contains a score for that particular arrangement of words.
I have encoded each line to ints and padded them with 0's and put them in a file called words.txt
an example of that file would be:
475,12,2495,2934,105,0,0,0,9384 (last column being the output score)
Now I have this code:
When I run it, it's loss/accuracy is very bad, loss is like 70000000.
What am I doing wrong?
from numpy import loadtxt
from itertools import islice
from keras.models import Sequential
from keras.layers import Dense
# load the dataset
dataset = loadtxt('words.txt', delimiter=',')
X = dataset[:,0:8]
y = dataset[:,8]
model = Sequential()
model.add(Dense(12, input_dim=8, activation='relu'))
model.add(Dense(8, activation='relu'))
model.add(Dense(1))
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
model.fit(X, y, epochs=150, batch_size=10000)
_, accuracy = model.evaluate(X, y)
print('Accuracy: %.2f' % (accuracy*100))
My goal is to predict the score of random combinations of words that I generate.
Log of fit:
:\AI>python main.py
Using TensorFlow backend.
2021-02-14 08:52:48.350476: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
C:\Users\fordy\AppData\Local\Programs\Python\Python37\lib\site-packages\tensorflow_core\python\framework\indexed_slices.py:424: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Epoch 1/150
1047711/1047711 [==============================] - 7s 7us/step - loss: 72945595445.1503 - accuracy: 0.2351
Epoch 2/150
1047711/1047711 [==============================] - 3s 3us/step - loss: 72940365091.2725 - accuracy: 0.0016
Epoch 3/150
1047711/1047711 [==============================] - 3s 3us/step - loss: 72922327250.8712 - accuracy: 0.0016
Epoch 4/150
1047711/1047711 [==============================] - 2s 2us/step - loss: 72883151430.7776 - accuracy: 0.0030
Epoch 5/150
1047711/1047711 [==============================] - 2s 2us/step - loss: 72815216732.1170 - accuracy: 0.0041
Epoch 6/150
1047711/1047711 [==============================] - 2s 2us/step - loss: 72711719248.6156 - accuracy: 0.0012
Epoch 7/150
1047711/1047711 [==============================] - 2s 2us/step - loss: 72566884174.8089 - accuracy: 1.5271e-05

Your model is too small. Try adding embedding layer and LSTM:
model = Sequential()
model.add(tf.keras.layers.Embedding(3600, 12, input_length=8)) # <= adjust vocab size
model.add(tf.keras.layers.LSTM(8))
# model.add(tf.keras.layers.Dense(12, input_dim=8, activation='relu'))
# model.add(tf.keras.layers.Dense(8, activation='relu'))
model.add(Dense(1))

Keras loss is not decreasing

I tried to learn my NN with breast Cancer Wisconsin
(I add "id" column as an index and changed "diagnosis" column to 0 and 1 with sklearn.preprocessing.LabelEncoder), but my NN is not reducing Loss.
I tried other optimizers and losses but this isn't working.
That's my NN:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, BatchNormalization, InputLayer
import tensorflow.nn as tfnn
model = Sequential()
model.add(Dense(30, activation = tfnn.relu, input_dim = 30))
model.add(BatchNormalization(axis = 1))
model.add(Dense(60, activation = tfnn.relu))
model.add(BatchNormalization(axis = 1))
model.add(Dense(1, activation = tfnn.softmax))
model.compile(loss = 'binary_crossentropy', optimizer = 'adam', metrics = ['accuracy'])
model.fit(data, target, epochs = 6)
And my output:
Epoch 1/6
569/569 [==============================] - 2s 3ms/sample - loss: 10.0025 - acc: 0.3726
Epoch 2/6
569/569 [==============================] - 0s 172us/sample - loss: 10.0025 - acc: 0.3726
Epoch 3/6
569/569 [==============================] - 0s 176us/sample - loss: 10.0025 - acc: 0.3726
Epoch 4/6
569/569 [==============================] - 0s 167us/sample - loss: 10.0025 - acc: 0.3726
Epoch 5/6
569/569 [==============================] - 0s 163us/sample - loss: 10.0025 - acc: 0.3726
Epoch 6/6
569/569 [==============================] - 0s 169us/sample - loss: 10.0025 - acc: 0.3726
I seems that NN after a few iterations stops learning (look at the time of epochs learning, in the first epoch it's 2s and in others it's 0s and in first epoch speed of processing the data is ms/sample, but in other epochs iits us/sample)
Thank you for your time!

Softmax has sum=1.
You can't use softmax with 1 unit. It will always be 1.
Use 'sigmoid'.
Also be careful with 'relu'. It may (by luck) fall into an "all-zeros" region and stop evolving.
Ideally, the batch normalization should be before it (this way you guarantee that there will always be some positive numbers):
model = Sequential()
model.add(Dense(30, input_dim = 30))
model.add(BatchNormalization(axis = 1))
model.add(Activation(tfnn.relu))
model.add(Dense(60)
model.add(BatchNormalization(axis = 1))
model.add(Activation(tfnn.relu))
model.add(Dense(1, activation = tfnn.sigmoid))

Since you have a binary classification task with a single-unit final layer, you should not use tfnn.softmax as an activation for this layer. Use tfnn.sigmoid instead, i.e.
model.add(Dense(1, activation = tfnn.sigmoid)) # last layer

Keras Neural Network accuracy only 10%

I am learning how to train a keras neural network on the MNIST dataset. However, when I run this code, I get only 10% accuracy after 10 epochs of training. This means that the neural network is predicting only one class, since there are 10 classes. I am sure it is a bug in data preparation rather than a problem with the network architecture, because I got the architecture off of a tutorial (medium tutorial). Any idea why the model is not training?
My code:
from skimage import io
import numpy as np
from numpy import array
from PIL import Image
import csv
import random
from keras.preprocessing.image import ImageDataGenerator
import pandas as pd
from keras.utils import multi_gpu_model
import tensorflow as tf
train_datagen = ImageDataGenerator()
train_generator = train_datagen.flow_from_directory(
directory="./trainingSet",
class_mode="categorical",
target_size=(50, 50),
color_mode="rgb",
batch_size=1,
shuffle=True,
seed=42
)
print(str(train_generator.class_indices) + " class indices")
from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Conv2D
from keras.layers.pooling import MaxPooling2D, GlobalAveragePooling2D
from keras.optimizers import SGD
from keras import backend as K
from keras.layers import Input
from keras.models import Model
import keras
from keras.layers.normalization import BatchNormalization
K.clear_session()
K.set_image_dim_ordering('tf')
reg = keras.regularizers.l1_l2(1e-5, 0.0)
def conv_layer(channels, kernel_size, input):
output = Conv2D(channels, kernel_size, padding='same',kernel_regularizer=reg)(input)
output = BatchNormalization()(output)
output = Activation('relu')(output)
output = Dropout(0)(output)
return output
model = Sequential()
model.add(Conv2D(28, kernel_size=(3,3), input_shape=(50, 50, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten()) # Flattening the 2D arrays for fully connected layers
model.add(Dense(128, activation=tf.nn.relu))
model.add(Dropout(0.2))
model.add(Dense(10, activation=tf.nn.softmax))
from keras.optimizers import Adam
import tensorflow as tf
model.compile(loss='categorical_crossentropy',
optimizer='adam',
metrics=['accuracy'])
from keras.callbacks import ModelCheckpoint
epochs = 10
checkpoint = ModelCheckpoint('mnist.h5', save_best_only=True)
STEP_SIZE_TRAIN=train_generator.n/train_generator.batch_size
model.fit_generator(generator=train_generator,
steps_per_epoch=STEP_SIZE_TRAIN,
epochs=epochs,
callbacks=[checkpoint]
)
The output I am getting is as follows:
Using TensorFlow backend.
Found 42000 images belonging to 10 classes.
{'0': 0, '1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9} class indices
Epoch 1/10
42000/42000 [==============================] - 174s 4ms/step - loss: 14.4503 - acc: 0.1035
/home/ec2-user/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/keras/callbacks.py:434: RuntimeWarning: Can save best model only with val_loss available, skipping.
'skipping.' % (self.monitor), RuntimeWarning)
Epoch 2/10
42000/42000 [==============================] - 169s 4ms/step - loss: 14.4487 - acc: 0.1036
Epoch 3/10
42000/42000 [==============================] - 169s 4ms/step - loss: 14.4483 - acc: 0.1036
Epoch 4/10
42000/42000 [==============================] - 168s 4ms/step - loss: 14.4483 - acc: 0.1036
Epoch 5/10
42000/42000 [==============================] - 169s 4ms/step - loss: 14.4483 - acc: 0.1036
Epoch 6/10
42000/42000 [==============================] - 168s 4ms/step - loss: 14.4483 - acc: 0.1036
Epoch 7/10
42000/42000 [==============================] - 168s 4ms/step - loss: 14.4483 - acc: 0.1036
Epoch 8/10
42000/42000 [==============================] - 168s 4ms/step - loss: 14.4483 - acc: 0.1036
Epoch 9/10
42000/42000 [==============================] - 168s 4ms/step - loss: 14.4480 - acc: 0.1036
Epoch 10/10
5444/42000 [==>...........................] - ETA: 2:26 - loss: 14.3979 - acc: 0.1067
The trainingSet directory contains a folder for each 1-9 digit with the images inside the folders. I am training on an AWS EC2 p3.2xlarge instance with the Amazon Deep Learning Linux AMI.

Here is the list of some weird points that I see :
Not rescaling your images -> ImageDataGenerator(rescale=1/255)
Batch Size of 1 (You may want to increase that)
MNIST is grayscale pictures , therefore color_mode should be "grayscale".
(Also you have several unused part in your code, that you may want to delete from the question)

Adding two more point in answer of #abcdaire,
mnist has image size of (28,28), you have assigned it wrong.
Binarization is another method, which can be used. It also make network to learn fast. It can be done like this.
`
imges_dataset = imges_dataset/255.0
imges_dataset = np.where(imges_dataset>0.5,1,0)

How to approximate the determinant with keras

As an experiment I am building a keras model to approximate the determinant of a matrix. However, when I run it the loss goes down at every epoch and the validation loss goes up! For example:
8s - loss: 7573.9168 - val_loss: 21831.5428
Epoch 21/50
8s - loss: 7345.0197 - val_loss: 23594.8540
Epoch 22/50
13s - loss: 7087.7454 - val_loss: 24718.3967
Epoch 23/50
7s - loss: 6851.8714 - val_loss: 25624.8609
Epoch 24/50
6s - loss: 6637.8168 - val_loss: 26616.7835
Epoch 25/50
7s - loss: 6446.8898 - val_loss: 28856.9654
Epoch 26/50
7s - loss: 6255.7414 - val_loss: 30122.7924
Epoch 27/50
7s - loss: 6054.5280 - val_loss: 32458.5306
Epoch 28/50
Here is the complete code:
import numpy as np
import sys
from scipy.stats import pearsonr
from scipy.linalg import det
from sklearn.model_selection import train_test_split
from tqdm import tqdm
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import math
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from keras import backend as K
def baseline_model():
# create model
model = Sequential()
model.add(Dense(200, input_dim=n**2, kernel_initializer='normal', activation='relu'))
model.add(Dense(1, input_dim=n**2))
# model.add(Dense(1, kernel_initializer='normal'))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam')
return model
n = 15
print("Making the input data using seed 7", file=sys.stderr)
np.random.seed(7)
U = np.random.choice([0, 1], size=(n**2,n))
#U is a random orthogonal matrix
X =[]
Y =[]
# print(U)
for i in tqdm(range(100000)):
I = np.random.choice(n**2, size = n)
# Pick out the random rows and sort the rows of the matrix lexicographically.
A = U[I][np.lexsort(np.rot90(U[I]))]
X.append(A.ravel())
Y.append(det(A))
X = np.array(X)
Y = np.array(Y)
print("Data created")
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=32, verbose=2)))
pipeline = Pipeline(estimators)
X_train, X_test, y_train, y_test = train_test_split(X, Y,
train_size=0.75, test_size=0.25)
pipeline.fit(X_train, y_train, mlp__validation_split=0.3)
How can I stop it overfitting so badly?
Update 1
I tried adding more layers and L_2 regularization. However, it makes little or no difference.
def baseline_model():
# create model
model = Sequential()
model.add(Dense(n**2, input_dim=n**2, kernel_initializer='glorot_normal', activation='relu'))
model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01)))
model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01)))
model.add(Dense(int((n**2)/2.0), kernel_initializer='glorot_normal', activation='relu', kernel_regularizer=regularizers.l2(0.01)))
model.add(Dense(1, kernel_initializer='glorot_normal'))
# Compile model
model.compile(loss='mean_squared_error', optimizer='adam')
return model
I increased the number of epochs to 100 and it finishes with:
19s - loss: 788.9504 - val_loss: 18423.2807
Epoch 97/100
24s - loss: 760.2046 - val_loss: 18305.9273
Epoch 98/100
20s - loss: 806.0941 - val_loss: 18174.8706
Epoch 99/100
24s - loss: 780.0487 - val_loss: 18356.7482
Epoch 100/100
27s - loss: 749.2595 - val_loss: 18331.5859
Is it possible to approximate the determinant of a matrix using keras?

I tested your code and got the same result. But let's go into basic understanding of matrix determinant (DET). DET consists of n! products, so you cannot really approximate it with n*n weights in few layers of neural network. This requires number of weights that would not scale to n=15, since 15! is 1307674368000 terms for multiplication in the DET.