I am printing the classification report. the code I am using is printing the AUC value for the ROC curve but not for the precision-recall curve (where it is only plotting a graph). How to get the AUC value for the precision-recall curve?
df_test = pd.read_csv("D:/a.csv")
df_testPred = pd.read_csv("D:/b.csv")
y_true1 = df_test["anomaly"].values[:-1]
y_score1 = df_testPred["anomaly_scores"].values[:-1]
y_pred1 = df_testPred["anomaly"].values[:-1].astype(int)
ap1 = average_precision_score(y_true1, y_score1)
auc1 = roc_auc_score(y_true1, y_score1)
print(f"ap: {ap1}")
print(f"AUC: {auc1}")
print(classification_report(y_true1, y_pred1))
precision1, recall1, thresholds1 = precision_recall_curve(y_true1, y_score1)
#plt.plot([0, 1], [0, 1],'r--')
plt.plot(recall1, precision1)
Since you have already calculated precision1 and recall1, you can simply use the relevant scikit-learn function auc (docs):
from sklearn.metrics import auc
auc_score = auc(recall1, precision1)
See ROC Curves and Precision-Recall Curves for Imbalanced Classification (although, according to my experience, the precision-recall AUC is not as widely used compared to the more usual ROC AUC).
Related
I am running a Convolutional Neural Network. After it finishes running, I use some metrics to evaluate the performance of the model. 2 of the metrics are the auc and roc_auc_score from sklearn
AUC function: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.auc.html?highlight=auc#sklearn.metrics.auc
AUROC function: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score
The code I am using is the following:
print(pred)
fpr, tpr, thresholds = metrics.roc_curve(true_classes, pred, pos_label=1)
print("-----AUC-----")
print(metrics.auc(fpr, tpr))
print("----ROC AUC-----")
print(metrics.roc_auc_score(true_classes, pred))
Where true_classes is a table which is of the form : [0 1 0 1 1 0] where 1 is the positive label and 0 the negative.
And pred is the predictions of the model:
prediction = classifier.predict(test_final)
prediction1 = []
predictions = []
for preds in prediction:
prediction1.append(preds[0])
pred = prediction1
However I am getting the same AUC and ROC AUC value no matter how many times I run the test (What I mean by that is that AUC and ROC AUC values in each test are the same. Not that they remain the same on all the tests. For example for test 1 I get AUC = 0.987 and ROC_AUC = 0.987 and for test 2 I get AUC = 0.95 and ROC_AUC = 0.95) . Am I doing something wrong? Or is it normal?
As per documentation linked, metrics.auc is a general case method to calculate area under a curve from points of that curve.
metrics.roc_auc_score is a specific case method used to calculate Area Under Curve for ROC curve.
You would not expect to see different results if you're using the same data to calculate both, as metrics.roc_auc_score will do the same thing as metrics.auc and, most likely, use the metrics.auc method itself, under the hood (i.e. use the general method for the specific task of calculating Area under ROC curve).
I'm solving a task of multi-class classification and want to estimate the result using roc curve in sklearn. As I know, it allows to plot a curve in this case if I set a positive label.
I tried to plot a roc curve using positive label and got strange results: the bigger the "positive label" of the class was, the closer to the top left corner the roc curve became.
Then I plot a roc curve with a previous binary labeling of the arrays. These 2 plots were different! I think that the second one was built correctly, but in case of binary classes the plot has only 3 points and this is not informative.
I want to understand, why roc curve for binary classes and roc curve with "positive label" look different and how to plot roc curve with positive label correctly.
Here is the code:
from sklearn.metrics import roc_curve, auc
y_pred = [1,2,2,2,3,3,1,1,1,1,1,2,1,2,3,2,2,1,1]
y_test = [1,3,2,2,1,3,2,1,2,2,1,2,2,2,1,1,1,1,1]
fp, tp, _ = roc_curve(y_test, y_pred, pos_label = 2)
from sklearn.preprocessing import label_binarize
y_pred = label_binarize(y_pred, classes=[1, 2, 3])
y_test = label_binarize(y_test, classes=[1, 2, 3])
fpb, tpb, _b = roc_curve(y_test[:,1], y_pred[:,1])
plt.plot(fp, tp, 'ro-', fpb, tpb, 'bo-', alpha = 0.5)
plt.show()
print('AUC with pos_label', auc(fp,tp))
print('AUC binary variant', auc(fpb,tpb))
This is the example of the plot
Red curve represents roc_curve with pos_label, blue curve represents roc_curve in "binary case"
As explained in the comments, ROC curves are not suitable for evaluating thresholded predictions (i.e. hard classes), as your y_pred; moreover, when using AUC, it is useful to keep in mind some limitations that are not readily apparent to many practitioners - see the last part of own answer in Getting a low ROC AUC score but a high accuracy for more details.
Could you give me please some advise, which metrics I can use to evaluate the quality of such a multi-class classification with "hard" classes?
The most straightforward approach would be the confusion matrix and the classification report readily provided by scikit-learn:
from sklearn.metrics import confusion_matrix, classification_report
y_pred = [1,2,2,2,3,3,1,1,1,1,1,2,1,2,3,2,2,1,1]
y_test = [1,3,2,2,1,3,2,1,2,2,1,2,2,2,1,1,1,1,1]
print(classification_report(y_test, y_pred)) # caution - order of arguments matters!
# result:
precision recall f1-score support
1 0.56 0.56 0.56 9
2 0.57 0.50 0.53 8
3 0.33 0.50 0.40 2
avg / total 0.54 0.53 0.53 19
cm = confusion_matrix(y_test, y_pred) # again, order of arguments matters
cm
# result:
array([[5, 2, 2],
[4, 4, 0],
[0, 1, 1]], dtype=int64)
From the confusion matrix, you can extract other quantities of interest, like true & false positives per class etc - for details, please see own answer in How to get precision, recall and f-measure from confusion matrix in Python
I have some data labeled as either a 0 or 1 and I am trying to predict these classes using a random forest. Each instance is labeled with 20 features that are used to train the random forest (~30.000 training instances and ~6000 test instances.
I am plotting the precision-recall and ROC curves using the following code:
precision, recall, _ = precision_recall_curve(y_test, y_pred)
plt.step(recall, precision, color='b', alpha=0.2,where='post')
plt.fill_between(recall, precision, step='post', alpha=0.2, color='b')
fpr, tpr, _ = roc_curve(y_test, y_pred)
roc_auc = auc(fpr, tpr)
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
All the PR and ROC curves I have seen thus far always have a jagged/smooth decline in precision/recall and a smooth/jagged increase in the ROC line. But my PR and ROC curves for some reason always look like this:
For some reason the only have a single point where they change direction. Is this due to a coding error by me or something inherent about the data/classification problem? If so, how can this behavior be explained?
I suspect you used the RandomForestClassifier.predict() method which results in either 0 or 1 depending on the predicted class.
To get the probability, which is the fraction of trees voted for a specific class, you have to use the RandomForestClassifier.predict_proba() method.
Using these probabilities as input for your curve calculations should fix the problem.
EDIT: The curve creation methods of scikit-learn sort the predictions first according to the prediction score, then according to their real/observed value, therefore the curves have these "bends".
Inside the precision_recall_curve, the y_pred must be the probabilities of the target class AND NOT the actual predicted class.
Since you are using a RandomForestClassifier, use predict_proba(X) to get the probabilities.
rf = RandomForestClassifier()
probas_pred = rf.predict_proba(X_test)
precision, recall, _ = precision_recall_curve(y_true, probas_pred)
plt.step(recall, precision, color='b', alpha=0.2,where='post')
plt.fill_between(recall, precision, step='post', alpha=0.2, color='b')
I have a multilabel classifier written in Keras from which I want to compute AUC and plot a ROC curve for every element classified from my test set.
Everything seems fine, except that some elements have a roc curve that have a slope as follows:
I don't know how to interpret the slope in such cases.
Basically my workflow goes as follows, I have a pre-trained model, instance of Keras, and I have the features X and the binarized labels y, every element in y is an array of length 1000, as it is a multilabel classification problem each element in y might contain many 1s, indicating that the element belongs to multiples classes, so I used the built-in loss of binary_crossentropy and my outputs of the model prediction are score probailities. Then I plot the roc curve as follows.
from sklearn.metrics import roc_curve, auc
#...
for xi, yi in (X_test, y_test):
y_pred = model.predict([xi])[0]
fpr, tpr, _ = roc_curve(yi, y_pred)
plt.plot(fpr, tpr, color='darkorange', lw=0.5)
The predict method returns probabilities, as I'm using the functional api of keras.
Does anyone knows why roc curves looks like this?
Asking in the mailing list of scikit-learn, they answered:
Slope usually means there are ties in your predictions.
Which is the case in this problem.
I have difficulty in plotting OneClassSVM's AUC plot in python (I am using sklearn which generates confusion matrix like [[tp, fp],[fn,tn]] with fn=tn=0.
from sklearn.metrics import roc_curve, auc
fpr, tpr, thresholds = roc_curve(y_test, y_nb_predicted)
roc_auc = auc(fpr, tpr) # this generates ValueError[1]
print "Area under the ROC curve : %f" % roc_auc
plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc)
I want to handle error [1] and plot AUC for OneClassSVM.
[1] ValueError: Input contains NaN, infinity or a value too large for dtype('float64').
Please see my answer on a similar question. The gist is:
OneClassSVM fundamentally doesn't support converting a decision into a probability score, so you cannot pass the necessary scores into functions that require varying a score threshold, such as for ROC or Precision-Recall curves and scores.
You can approximate this type of score by computing the max value of your OneClassSVM's decision function across your input data, call it MAX, and then score the prediction for a given observation y by computing y_score = MAX - decision_function(y).
Use these scores to pass as y_score to functions such as average_precision_score, etc., which will accept non-thresholded scores instead of probabilities.
Finally, keep in mind that ROC will make less physical sense for OneClassSVM specifically because OneClassSVM is intended for situations where there is an expected and huge class imbalance (outliers vs. non-outliers), and ROC will not accurately up-weight the relative success on the small amount of outliers.
Use the predprobs function to calculate the scores or probabilities/scores as asked in the auc(y_true, y_score), the issue is because of y_score. you can convert it as shown in the following line of code
# Classifier - Algorithm - SVM
# fit the training dataset on the classifier
SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto',probability=True)
SVM.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("SVM Accuracy Score -> ",accuracy_score(predictions_SVM, Test_Y))
probs = SVM.predict_proba(Test_X_Tfidf)
preds = probs[:,1]
fpr, tpr, threshold = roc_curve(Test_Y, preds)
print("SVM Area under curve -> ",auc(fpr, tpr))
see the difference between the accuracy_score and the auc(), you need the scores of predictions.
share edit delete flag