I am currently in the process of displaying precision, recall and fscore. Now my question is how do I do this? What I tried is the following:
num_users, num_items = train_mat.shape
user_input, item_input, labels = get_train_samples(train_mat, num_negatives)
val_user_input, val_item_input, val_labels = get_train_samples(val_mat, num_negatives)
.
.
.
history = model.fit([np.array(user_input), np.array(item_input)], np.array(labels),
epochs=EPOCHS, verbose=VERBOSE, shuffle=True, batch_size = BATCH_SIZE,
validation_data=([np.array(val_user_input), np.array(val_item_input)], np.array(val_labels)),
callbacks=CALLBACKS)
.
.
.
# Precision, recall and fscore
from sklearn.metrics import precision_recall_fscore_support, confusion_matrix, roc_curve, auc
precision, recall, fscore, _ = precision_recall_fscore_support(y_test, y_pred, average='weighted')
print('Precision, recall, and F1 score, averaged and weighted by number of instances in each class:')
print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('f1 score: {}\n'.format(fscore))
precision, recall, fscore, _ = precision_recall_fscore_support(y_test, y_pred)
print('Precision, recall, and F1 score, per class [0 1]:')
print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('f1 score: {}'.format(fscore))
cm = confusion_matrix(y_test, y_pred)
sns.heatmap(cm, annot=True)
Unfortunately I don't know how to get y_test and y_pred. How do I get these values?
you shall have y_test as the test set to test your model and if you dont have such a set you can use sklearn train test split for getting a training set and a test set. Here is the link for how to use it:
sklearn traiin test split
and when you will have your test set you will do this to get y_pred:
y_pred = model.predict(y_test)
Related
I am trying to compute precision, recall, and f1 score on my test dataset. However, I am using ImageDataGenerator format, not using train_test_split (x_train, y_train, x_test and y_test). That's why I couldn't find any references online.
IMAGE_SIZE = 224
BATCH_SIZE = 64
EPOCH = 30
CHANNEL = 3
CLASSES = 10
train_path = "/Users/ba/Documents/mycodes/datasets/DS/train"
valid_path = "/Users/ba/Documents/mycodes/datasets/DS/val"
test_path = "/Users/ba/Documents/mycodes/datasets/DS/test"
train_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet_v3.preprocess_input) \
.flow_from_directory(directory=train_path, target_size=(IMAGE_SIZE,IMAGE_SIZE), batch_size=BATCH_SIZE)
valid_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet_v3.preprocess_input) \
.flow_from_directory(directory=valid_path, target_size=(IMAGE_SIZE,IMAGE_SIZE), batch_size=BATCH_SIZE)
test_batches = ImageDataGenerator(preprocessing_function=tf.keras.applications.mobilenet_v3.preprocess_input) \
.flow_from_directory(directory=test_path, target_size=(IMAGE_SIZE,IMAGE_SIZE), batch_size=BATCH_SIZE, shuffle=False)
Then I tried to calculate precision, recall, and f1 in the following way down below:
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import f1_score
y_pred_logits = model.predict(test_batches)
y_pred = tf.math.argmax(y_pred_logits)
test_classes = test_batches.classes
# accuracy: (tp + tn) / (p + n)
accuracy = accuracy_score(test_classes, y_pred)
print('Accuracy: %f' % accuracy)
# precision tp / (tp + fp)
precision = precision_score(test_classes, y_pred)
print('Precision: %f' % precision)
# recall: tp / (tp + fn)
recall = recall_score(test_classes, y_pred)
print('Recall: %f' % recall)
# f1: 2 tp / (2 tp + fp + fn)
f1 = f1_score(test_classes, y_pred)
print('F1 score: %f' % f1)
Unfortunately it throws this error message:
ValueError: Found input variables with inconsistent numbers of samples: [1887, 10]
Can you help me re-write the code, or any other references using ImageDataGenerator format I used?
So the problem were on y_true (test_classes) and y_pred. With this, one can also calculate the confusion matrix.
#Making prediction
y_true = test_batches.classes
predictions = model.predict(x=test_batches, steps=len(test_batches), verbose=0)
y_pred = predictions.argmax(axis=1)
print("Precision Score: ",precision_score(y_true, y_pred, pos_label='positive', average='micro'))
print("Recall Score: ",recall_score(y_true, y_pred, pos_label='positive', average='micro'))
print("F1 Score: ",f1_score(y_true, y_pred, pos_label='positive', average='micro'))
print("Accuracy Score: ",accuracy_score(y_true, y_pred))
I have a problem. I found this question https://stats.stackexchange.com/questions/56302/what-are-good-rmse-values
Someone wrote:
The RMSE for your training and your test sets should be very similar
if you have built a good model.
and another wrote:
RMSE of test > RMSE of train => OVER FITTING of the data. RMSE of test
< RMSE of train => UNDER FITTING of the data.
I think RMSE of test data it is
y_pred = knn.predict(X_test)
rmse = metrics.mean_squared_error(y_test, y_pred , squared=False)
But how could I get the RMSE (or another metric) of my training data? Perhaps it is
rmse = metrics.mean_squared_error(X_train, X_test, squared=False)
But with that I got
ValueError: Found input variables with inconsistent numbers of samples: [8880, 2220]
So how could I get the RMSE from my training ?
from sklearn.neighbors import (KNeighborsRegressor,)
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.2, random_state=30)
knn = KNeighborsRegressor(algorithm='auto', leaf_size=30, metric='minkowski', metric_params=None, n_jobs=1, n_neighbors=5,
p=2, weights='uniform')
knn.fit(X, y)
y_pred = knn.predict(X_test)
mse = metrics.mean_squared_error(y_test, y_pred)
rmse = metrics.mean_squared_error(y_test, y_pred , squared=False)
print(rmse)
First of all, there's something wrong with your code and it's that you are training your model with the whole data, instead of the training data you've already split, this makes your validation sample useless as the model itself learnt from it.
You should change your fit like so:
knn.fit(X_train, y_train)
Then to get the RMSE of it you should use the predict on your train data and compare it afterwards:
y_train_pred = knn.predict(X_train)
rmse = metrics.mean_squared_error(y_train, y_train_pred, squared=False)
Everything else should stay the same
The context of your question is not clear. However, you should take the predictions and not the model features of the test set, and evaluate the RMSE using the same package that you used:
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html
Pay attention to the squared parameter.
I am trying different classifiers with different parameters and stuff on a dataset provided to us as part of a course project. We have to try and get the best performance on the dataset. The dataset is actually a reduced version of the online news popularity
I have tried the SVM, Random Forest, SVM with cross-validation with k = 5 and they all seem to give approximately 100% training accuracy, while the testing accuracy is between 60-70. I think the testing accuracy is fine, but the training accuracy bothers me.
I would say maybe it was a case of overfitting data but none of my classmates seem to be getting similar results so maybe the problem is with my code.
Here is the code for my cross-validation and random forest classifier. I would be very grateful if you help me find out why I am getting such a high Training accuracy
def crossValidation(X_train, X_test, y_train, y_test, numSplits):
skf = StratifiedKFold(n_splits=5, shuffle=True)
Cs = np.logspace(-3, 3, 10)
gammas = np.logspace(-3, 3, 10)
ACC = np.zeros((10, 10))
DEV = np.zeros((10, 10))
for i, gamma in enumerate(gammas):
for j, C in enumerate(Cs):
acc = []
for train_index, dev_index in skf.split(X_train, y_train):
X_cv_train, X_cv_dev = X_train[train_index], X_train[dev_index]
y_cv_train, y_cv_dev = y_train[train_index], y_train[dev_index]
clf = SVC(C=C, kernel='rbf', gamma=gamma, )
clf.fit(X_cv_train, y_cv_train)
acc.append(accuracy_score(y_cv_dev, clf.predict(X_cv_dev)))
ACC[i, j] = np.mean(acc)
DEV[i, j] = np.std(acc)
i, j = np.argwhere(ACC == np.max(ACC))[0]
clf1 = SVC(C=Cs[j], kernel='rbf', gamma=gammas[i], decision_function_shape='ovr')
clf1.fit(X_train, y_train)
y_predict_train = clf1.predict(X_train)
y_pred_test = clf1.predict(X_test)
print("Train Accuracy :: ", accuracy_score(y_train, y_predict_train))
print("Test Accuracy :: ", accuracy_score(y_test, y_pred_test))
def randomForestClassifier(X_train, X_test, y_train, y_test):
"""
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
y_predict_train = clf.predict(X_train)
y_pred_test = clf.predict(X_test)
print("Train Accuracy :: ", accuracy_score(y_train, y_predict_train))
print("Test Accuracy :: ", accuracy_score(y_test, y_pred_test))
There are two issues about the problem, training accuracy and testing accuracy are significantly different.
Different distribution of training data and testing data.(because of selecting a part of the dataset)
Overfitting of the model to the training data.
Since you apply cross-validation, it seems that you should think about another solution.
I recommend that you apply some feature selection or feature reduction (like PCA) approaches to tackle the overfitting problem.
I have the simple fitting model like this:
lm = linear_model.LinearRegression()
model = lm.fit(X_train, y_train)
predictions = lm.predict(X_test)
print accuracy_score(y_test, predictions)
and with using cross validation I have this:
from sklearn.model_selection import cross_val_score
accuracies = cross_val_score(estimator = model, X = X_train, y = y_train, cv = 7)
from cross validation how can I take the accuracy in order to have the same measure print accuracy_score(y_test, predictions)? Is it accuracies.mean()?
print accuracies will give an array of accuracy on each fold of cross validation
print "Train set score :: {} ".format(accuracies.mean()) will give the mean accuracy on the cross validation and
print "Train set score :: {} +/-{}".format(accuracies.mean(),accuracies.std()*2) will give you the accuracy along with the mean deviation
I'm using cross_val_score from scikit-learn (package sklearn.cross_validation) to evaluate my classifiers.
If I use f1 for the scoring parameter, the function will return the f1-score for one class. To get the average I can use f1_weighted but I can't find out how to get the f1-score of the other class. (precision and recall analogous)
The functions in sklearn.metrics have a labels parameter which does this, but I can't find anything like this in the documentation.
Is there a way to get the f1-score for all classes at once or at least specify the class which should be considered with cross_val_score?
When you create a scorer with make_scorer function you can pass any additional arguments you need, like this:
cross_val_score(
svm.SVC(kernel='rbf', gamma=0.7, C = 1.0),
X, y,
scoring=make_scorer(f1_score, average='weighted', labels=[2]),
cv=10)
But cross_val_score only allows you to return one score. You can't get scores for all classes at once without additional tricks. If you need that please refer to another stack overflow question which covers exactly that: Evaluate multiple scores on sklearn cross_val_score
You may simply try the following:
svm = LinearSVC()
scores = cross_val_score(svm, X, y,
scoring = "f1",
cv = 10)
For individual scores of each class, use this :
f1 = f1_score(y_test, y_pred, average= None)
print("f1 list non intent: ", f1)
To compute F1 score, we can use sklearn.metrics.f1_score
http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html
Sample code
from sklearn import svm
from sklearn import metrics
from sklearn.cross_validation import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import f1_score, accuracy_score
# prepare dataset
iris = load_iris()
X = iris.data[:, :2]
y = iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# svm classification
clf = svm.SVC(kernel='rbf', gamma=0.7, C = 1.0).fit(X_train, y_train)
y_predicted = clf.predict(X_test)
# performance
print "Classification report for %s" % clf
print metrics.classification_report(y_test, y_predicted)
print("F1 micro: %1.4f\n" % f1_score(y_test, y_predicted, average='micro'))
print("F1 macro: %1.4f\n" % f1_score(y_test, y_predicted, average='macro'))
print("F1 weighted: %1.4f\n" % f1_score(y_test, y_predicted, average='weighted'))
print("Accuracy: %1.4f" % (accuracy_score(y_test, y_predicted)))
Sample Output
Classification report for SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
decision_function_shape=None, degree=3, gamma=0.7, kernel='rbf',
max_iter=-1, probability=False, random_state=None, shrinking=True,
tol=0.001, verbose=False)
precision recall f1-score support
0 1.00 0.90 0.95 10
1 0.50 0.88 0.64 8
2 0.86 0.50 0.63 12
avg / total 0.81 0.73 0.74 30
F1 micro: 0.7333
F1 macro: 0.7384
F1 weighted: 0.7381
Accuracy: 0.7333