I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
Everything seems fine, except that some elements have a roc curve that have a slope as follows:
I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained model, instance of Keras, and I have the features X and the binarized labels y, every element in y is an array of length 1000, as it is a multilabel classification problem each element in y might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of binary_crossentropy and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
from sklearn.metrics import roc_curve, auc
#...
for xi, yi in (X_test, y_test):
y_pred = model.predict([xi])[0]
fpr, tpr, _ = roc_curve(yi, y_pred)
plt.plot(fpr, tpr, color='darkorange', lw=0.5)
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why roc curves looks like this?
Asking in the mailing list of scikit-learn, they answered:
Slope usually means there are ties in your predictions.
Which is the case in this problem.
Related
I am running a Convolutional Neural Network. After it finishes running, I use some metrics to evaluate the performance of the model. 2 of the metrics are the auc and roc_auc_score from sklearn
AUC function: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html?highlight=auc#sklearn.metrics.auc
AUROC function: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score
The code I am using is the following:
print(pred)
fpr, tpr, thresholds = metrics.roc_curve(true_classes, pred, pos_label=1)
print("-----AUC-----")
print(metrics.auc(fpr, tpr))
print("----ROC AUC-----")
print(metrics.roc_auc_score(true_classes, pred))
Where true_classes is a table which is of the form : [0 1 0 1 1 0] where 1 is the positive label and 0 the negative.
And pred is the predictions of the model:
prediction = classifier.predict(test_final)
prediction1 = []
predictions = []
for preds in prediction:
prediction1.append(preds[0])
pred = prediction1
However I am getting the same AUC and ROC AUC value no matter how many times I run the test (What I mean by that is that AUC and ROC AUC values in each test are the same. Not that they remain the same on all the tests. For example for test 1 I get AUC = 0.987 and ROC_AUC = 0.987 and for test 2 I get AUC = 0.95 and ROC_AUC = 0.95) . Am I doing something wrong? Or is it normal?
As per documentation linked, metrics.auc is a general case method to calculate area under a curve from points of that curve.
metrics.roc_auc_score is a specific case method used to calculate Area Under Curve for ROC curve.
You would not expect to see different results if you're using the same data to calculate both, as metrics.roc_auc_score will do the same thing as metrics.auc and, most likely, use the metrics.auc method itself, under the hood (i.e. use the general method for the specific task of calculating Area under ROC curve).
I understand that Support Vector Machine algorithm does not compute probabilities, which is needed to find the AUC value, is there any other way to just find the AUC score?
from sklearn.svm import SVC
model_ksvm = SVC(kernel = 'rbf', random_state = 0)
model_ksvm.fit(X_train, y_train)
model_ksvm.predict_proba(X_test)
I can't get the the probability output from the SVM algorithm, without the probability output I can't get the AUC score, which I can get with other algorithm.
You don't really need probabilities for the ROC, just any sort of confidence score. You need to rank-order the samples according to how likely they are to be in the positive class. Support Vector Machines can use the (signed) distance from the separating plane for that purpose, and indeed sklearn does that automatically under the hood when scoring with AUC: it uses the decision_function method, which is the signed distance.
You can also set the probability option in the SVC (docs), which fits a Platt calibration model on top of the SVM to produce probability outputs:
model_ksvm = SVC(kernel='rbf', probability=True, random_state=0)
But this will lead to the same AUC, because the Platt calibration just maps the signed distances to probabilities monotonically.
I built 3 different model in Random Forests for imbalanced binary classification problem. (0 - 369, 1- 1750)
First one is Random Forest (baseline) which uses rf1 fitting on trainX and trainY
Second one is Random Forest (balanced) which uses rf2 that configures class_weight = 'balanced' fitting on trainX and trainY.
Third one is Random Forest (SMOTE) which uses rf1 fitting on trainX_smote and trainY_smote.
I know that in normal practice, area under PR curve should be used for imbalanced data set. However, I am not too sure if I need to use area under PR curve or ROC curve again for 2nd and 3rd case..
Should I still apply area under PR curve for 2nd and 3rd case even if I sort of "balanced" data set using parameter or SMOTE process? I know that ROC AUC is not comparable to PR AUC but my understanding is you use PR AUC for imbalanced data only so I have dilemma.
My goal is to show if oversampling SMOTE or configuring parameter in class_weight = 'balanced' would improve predictive power or not.
Please let me know which one I should use for already balanced dataset and why. Thanks!
trainX, testX, trainY, testY = train_test_split(x,y,test_size=0.20, random_state=2019)
# First 2 models
# baseline model using imbalanced data
rf1 = RandomForestClassifier(n_estimators=2500, max_depth=4)
# balanced model using class_weight = 'balanced'
rf2 = RandomForestClassifier(n_estimators=2500, class_weight = 'balanced', max_depth=4)
# SMOTE
smote = SMOTE(random_state=42)
trainX_smote, trainY_smote = smote.fit_sample(trainX, trainY)
trainX_smote = pd.DataFrame(data = trainX_smote, columns=trainX.columns)
# 3rd model will use rf1 on SMOTEd trainX and trainY and then .fit and will test on original testX.
I have difficulty in plotting OneClassSVM's AUC plot in python (I am using sklearn which generates confusion matrix like [[tp, fp],[fn,tn]] with fn=tn=0.
from sklearn.metrics import roc_curve, auc
fpr, tpr, thresholds = roc_curve(y_test, y_nb_predicted)
roc_auc = auc(fpr, tpr) # this generates ValueError[1]
print "Area under the ROC curve : %f" % roc_auc
plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)
I want to handle error [1] and plot AUC for OneClassSVM.
[1] ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Please see my answer on a similar question. The gist is:
OneClassSVM fundamentally doesn't support converting a decision into a probability score, so you cannot pass the necessary scores into functions that require varying a score threshold, such as for ROC or Precision-Recall curves and scores.
You can approximate this type of score by computing the max value of your OneClassSVM's decision function across your input data, call it MAX, and then score the prediction for a given observation y by computing y_score = MAX - decision_function(y).
Use these scores to pass as y_score to functions such as average_precision_score, etc., which will accept non-thresholded scores instead of probabilities.
Finally, keep in mind that ROC will make less physical sense for OneClassSVM specifically because OneClassSVM is intended for situations where there is an expected and huge class imbalance (outliers vs. non-outliers), and ROC will not accurately up-weight the relative success on the small amount of outliers.
Use the predprobs function to calculate the scores or probabilities/scores as asked in the auc(y_true, y_score), the issue is because of y_score. you can convert it as shown in the following line of code
# Classifier - Algorithm - SVM
# fit the training dataset on the classifier
SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto',probability=True)
SVM.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("SVM Accuracy Score -> ",accuracy_score(predictions_SVM, Test_Y))
probs = SVM.predict_proba(Test_X_Tfidf)
preds = probs[:,1]
fpr, tpr, threshold = roc_curve(Test_Y, preds)
print("SVM Area under curve -> ",auc(fpr, tpr))
see the difference between the accuracy_score and the auc(), you need the scores of predictions.
share edit delete flag
I'm using cross-validation to evaluate the performance of a classifier with scikit-learn and I want to plot the Precision-Recall curve. I found an example on scikit-learn`s website to plot the PR curve but it doesn't use cross validation for the evaluation.
How can I plot the Precision-Recall curve in scikit learn when using cross-validation?
I did the following but i'm not sure if it's the correct way to do it (psudo code):
for each k-fold:
precision, recall, _ = precision_recall_curve(y_test, probs)
mean_precision += precision
mean_recall += recall
mean_precision /= num_folds
mean_recall /= num_folds
plt.plot(recall, precision)
What do you think?
Edit:
it doesn't work because the size of precision and recall arrays are different after each fold.
anyone?
Instead of recording the precision and recall values after each fold, store the predictions on the test samples after each fold. Next, collect all the test (i.e. out-of-bag) predictions and compute precision and recall.
## let test_samples[k] = test samples for the kth fold (list of list)
## let train_samples[k] = test samples for the kth fold (list of list)
for k in range(0, k):
model = train(parameters, train_samples[k])
predictions_fold[k] = predict(model, test_samples[k])
# collect predictions
predictions_combined = [p for preds in predictions_fold for p in preds]
## let predictions = rearranged predictions s.t. they are in the original order
## use predictions and labels to compute lists of TP, FP, FN
## use TP, FP, FN to compute precisions and recalls for one run of k-fold cross-validation
Under a single, complete run of k-fold cross-validation, the predictor makes one and only one prediction for each sample. Given n samples, you should have n test predictions.
(Note: These predictions are different from training predictions, because the predictor makes the prediction for each sample without having been previously seen it.)
Unless you are using leave-one-out cross-validation, k-fold cross validation generally requires a random partitioning of the data. Ideally, you would do repeated (and stratified) k-fold cross validation. Combining precision-recall curves from different rounds, however, is not straight forward, since you cannot use simple linear interpolation between precision-recall points, unlike ROC (See Davis and Goadrich 2006).
I personally calculated AUC-PR using the Davis-Goadrich method for interpolation in PR space (followed by numerical integration) and compared the classifiers using the AUC-PR estimates from repeated stratified 10-fold cross validation.
For a nice plot, I showed a representative PR curve from one of the cross-validation rounds.
There are, of course, many other ways of assessing classifier performance, depending on the nature of your dataset.
For instance, if the proportion of (binary) labels in your dataset is not skewed (i.e. it is roughly 50-50), you could use the simpler ROC analysis with cross-validation:
Collect predictions from each fold and construct ROC curves (as before), collect all the TPR-FPR points (i.e. take the union of all TPR-FPR tuples), then plot the combined set of points with possible smoothing. Optionally, compute AUC-ROC using simple linear interpolation and the composite trapezoid method for numerical integration.
This is currently the best way to plot a Precision Recall curve for an sklearn classifier using cross-validation. Best part is, it plots the PR Curves for ALL classes, so you get multiple neat-looking curves as well
from scikitplot.classifiers import plot_precision_recall_curve
import matplotlib.pyplot as plt
clf = LogisticRegression()
plot_precision_recall_curve(clf, X, y)
plt.show()
The function automatically takes care of cross-validating the given dataset, concatenating all out of fold predictions, and calculating the PR Curves for each class + averaged PR Curve. It's a one-line function that takes care of it all for you.
Precision Recall Curves
Disclaimer: Note that this uses the scikit-plot library, which I built.