Receiver Operating Characteristic (ROC) Curve

A ROC curve shows the true positive rate (TPR, or sensitivity) versus the false positive rate (FPR, or 1-specificity) for different thresholds of classification scores.

Each point on a ROC curve corresponds to a pair of TPR and FPR values for a specific threshold value. You can find different pairs of TPR and FPR values by varying the threshold value, and then create a ROC curve using the pairs.

For a multiclass classification problem, you can use the one-versus-all coding design and find a ROC curve for each class. The one-versus-all coding design treats a multiclass classification problem as a set of binary classification problems, and assumes one class as positive and the rest as negative in each binary problem.

A binary classifier typically classifies an observation into a class that yields a larger score, which corresponds to a positive adjusted score for a one-versus-all binary classification problem. That is, a classifier typically uses 0 as a threshold and determines whether an observation is positive or negative. For example, if an adjusted score for an observation is 0.2, then the classifier with a threshold value of 0 assigns the observation to the positive class. You can find a pair of TPR and FPR values by applying the threshold value to all observations, and use the pair as a single point on a ROC curve. Now, assume you use a new threshold value of 0.25. Then, the classifier with a threshold value of 0.25 assigns the observation with an adjusted score of 0.2 to the negative class. By applying the new threshold to all observations, you can find a new pair of TPR and FPR values and have a new point on the a ROC curve. By repeating this process for various threshold values, you find pairs of TPR and FPR values and create a ROC curve using the pairs.

Area Under ROC Curve (AUC)

The area under a ROC curve (AUC) corresponds to the integral of a ROC curve (TPR values) with respect to FPR from FPR = 0 to FPR = 1.

The AUC provides an aggregate performance measure across all possible thresholds. The AUC values are in the range 0 to 1, and larger AUC values indicate better classifier performance.

Plot ROC Curve for Binary Classifier

Compute the performance metrics (FPR and TPR) for a binary classification problem by creating a rocmetrics object, and plot a ROC curve by using the plot function.

Load the ionosphere data set. This data set has 34 predictors (X) and 351 binary responses (Y) for radar returns, either bad ('b') or good ('g').

load ionosphere

Partition the data into training and test sets. Use approximately 80% of the observations to train a support vector machine (SVM) model, and 20% of the observations to test the performance of the trained model on new data. Partition the data using cvpartition.

rng("default") % For reproducibility of the partition
c = cvpartition(Y,Holdout=0.20);
trainingIndices = training(c); % Indices for the training set
testIndices = test(c); % Indices for the test set
XTrain = X(trainingIndices,:);
YTrain = Y(trainingIndices);
XTest = X(testIndices,:);
YTest = Y(testIndices);

Train an SVM classification model.

Mdl = fitcsvm(XTrain,YTrain);

Compute the classification scores for the test set.

[~,Scores] = predict(Mdl,XTest);
size(Scores)

ans = 1×2

    70     2

The output Scores is a matrix of size 70-by-2. The column order of Scores follows the class order in Mdl. Display the class order stored in Mdl.ClassNames.

Mdl.ClassNames

ans = 2x1 cell
    {'b'}
    {'g'}

Create a rocmetrics object by using the true labels in YTest and the classification scores in Scores. Specify the column order of Scores using Mdl.ClassNames.

rocObj = rocmetrics(YTest,Scores,Mdl.ClassNames);

rocObj is a rocmetrics object that stores the performance metrics for each class in the Metrics property. Compute the AUC values using the auc function.

a = auc(rocObj)

a = 1×2

    0.8587    0.8587

For a binary classification problem, the AUC values are equal to each other.

The table in Metrics contains the performance metric values for both classes, vertically concatenated according to the class order. Find the rows for the first class in the table, and display the first eight rows.

idx = strcmp(rocObj.Metrics.ClassName,Mdl.ClassNames(1));
head(rocObj.Metrics(idx,:))

    ClassName    Threshold    FalsePositiveRate    TruePositiveRate
    _________    _________    _________________    ________________

      {'b'}       15.545              0                     0      
      {'b'}       15.545              0                  0.04      
      {'b'}       15.105              0                  0.08      
      {'b'}       11.424              0                  0.16      
      {'b'}       10.077              0                   0.2      
      {'b'}       9.9716              0                  0.24      
      {'b'}       9.9417              0                  0.28      
      {'b'}       9.0338              0                  0.32      

Plot the ROC curve for each class by using the plot function.

plot(rocObj)

For each class, the plot function plots a ROC curve and displays a filled circle marker at the model operating point. The legend displays the class name and AUC value for each curve.

Note that you do not need to examine ROC curves for both classes in a binary classification problem. The two ROC curves are symmetric, and the AUC values are identical. A TPR of one class is a true negative rate (TNR) of the other class, and TNR is 1-FPR. Therefore, a plot of TPR versus FPR for one class is the same as a plot of 1-FPR versus 1-TPR for the other class.

Plot the ROC curve for the first class only by specifying the ClassNames name-value argument.

plot(rocObj,ClassNames=Mdl.ClassNames(1))

Plot ROC Curves for Multiclass Classifier

Compute the performance metrics (FPR and TPR) for a multiclass classification problem by creating a rocmetrics object, and plot a ROC curve for each class by using the plot function. Specify the AverageCurveType name-value argument of plot to create the average ROC curve for the multiclass problem.

Load the fisheriris data set. The matrix meas contains flower measurements for 150 different flowers. The vector species lists the species for each flower. species contains three distinct flower names.

load fisheriris

Train a classification tree that classifies observations into one of the three labels. Cross-validate the model using 10-fold cross-validation.

rng("default") % For reproducibility
Mdl = fitctree(meas,species,Crossval="on");

Compute the classification scores for validation-fold observations.

[~,Scores] = kfoldPredict(Mdl);
size(Scores)

ans = 1×2

   150     3

The output Scores is a matrix of size 150-by-3. The column order of Scores follows the class order in Mdl. Display the class order stored in Mdl.ClassNames.

Mdl.ClassNames

ans = 3x1 cell
    {'setosa'    }
    {'versicolor'}
    {'virginica' }

Create a rocmetrics object by using the true labels in species and the classification scores in Scores. Specify the column order of Scores using Mdl.ClassNames.

rocObj = rocmetrics(species,Scores,Mdl.ClassNames);

rocObj is a rocmetrics object that stores the performance metrics for each class in the Metrics property. Compute the AUC values by using the auc function.

a = auc(rocObj)

a = 1×3

    1.0000    0.9636    0.9636

The table in Metrics contains the performance metric values for all three classes, vertically concatenated according to the class order. Find and display the rows for the second class in the table.

idx = strcmp(rocObj.Metrics.ClassName,Mdl.ClassNames(2));
rocObj.Metrics(idx,:)

ans=13×4 table
      ClassName       Threshold    FalsePositiveRate    TruePositiveRate
    ______________    _________    _________________    ________________

    {'versicolor'}           1              0                    0      
    {'versicolor'}           1           0.01                  0.7      
    {'versicolor'}     0.95455           0.02                  0.8      
    {'versicolor'}     0.91304           0.03                  0.9      
    {'versicolor'}        -0.2           0.04                  0.9      
    {'versicolor'}    -0.33333           0.06                  0.9      
    {'versicolor'}        -0.6           0.08                  0.9      
    {'versicolor'}    -0.86957           0.12                 0.92      
    {'versicolor'}    -0.91111           0.16                 0.96      
    {'versicolor'}    -0.95122           0.31                 0.96      
    {'versicolor'}    -0.95238           0.38                 0.98      
    {'versicolor'}    -0.95349           0.44                 0.98      
    {'versicolor'}          -1              1                    1      

Plot the ROC curve for each class. Specify AverageCurveType="micro" to compute the performance metrics for the average ROC curve using the micro-averaging method.

plot(rocObj,AverageCurveType="micro")

One-Versus-All (OVA) Coding Design

The one-versus-all (OVA) coding design reduces a multiclass classification problem to a set of binary classification problems. In this coding design, each binary classification treats one class as positive and the rest of the classes as negative. rocmetrics uses the OVA coding design for multiclass classification and evaluates the performance on each class by using the binary classification that the class is positive.

For example, the OVA coding design for three classes formulates three binary classifications:

Each row corresponds to a class, and each column corresponds to a binary classification problem. The first binary classification assumes that class 1 is a positive class and the rest of the classes are negative. rocmetrics evaluates the performance on the first class by using the first binary classification problem.

rocmetrics applies the OVA coding design to a binary classification problem as well if you specify classification scores as a two-column matrix. rocmetrics formulates two one-versus-all binary classification problems each of which treats one class as a positive class and the other class as a negative class, and rocmetrics finds two ROC curves. You can use one of them to evaluate the binary classification problem.

Average of Performance Metrics

You can compute metrics for an average ROC curve by using the average function. Alternatively, you can use the plot function to compute the metrics and plot the average ROC curve. For examples, see Find Average ROC Curve (example for average) and Plot Average ROC Curve for Multiclass Classifier (example for plot).

average and plot support three algorithms for computing the average false positive rate (FPR) and average true positive rate (TPR) to find the average ROC curve:

Confusion Matrix

A confusion matrix is defined as

where

For example, the first row of the confusion matrix defines how the classifier identifies instances of the positive class: TP is the count of correctly identified positive instances, and FN is the count of positive instances misidentified as negative.

rocmetrics computes confusion matrices for various threshold values for each one-versus-all binary classification. The one-versus-all binary classification model classifies an observation into a positive class if the score for the observation is greater than or equal to the threshold value.

Prior Class Probabilities

By default, rocmetrics uses empirical probabilities, which are class frequencies in the true labels.

rocmetrics normalizes the 1-by-K prior probability vector π to a 1-by-2 vector for each one-versus-all binary classification, where K is the number of classes.

The prior probabilities for the kth binary classification in which the positive class is the kth class is $$\left[{\pi }_k,1-{\pi }_k\right]$$, where π_k is the prior probability for class k in the multiclass problem.

Scale Vector

rocmetrics defines a scale vector s_k of size 2-by-1 for each one-versus-all binary classification problem:

where P and N represent the total instances of positive class and negative class, respectively. That is, P is the sum of TP and FN, and N is the sum of FP and TN. s_k(1) (first element of s_k) and s_k(2) (second element of s_k) are the scales for the positive class (kth class) and negative class (the rest), respectively.

rocmetrics applies the scale values as multiplicative factors to the counts from the corresponding class. That is, the function multiplies counts from the positive class by s_k(1) and counts from the negative class by s_k(2). For example, to compute the positive predictive value (PPV = TP/(TP+FP)) for the kth binary problem, rocmetrics scales PPV as follows:

Misclassification Cost Matrix

By default, rocmetrics uses a K-by-K cost matrix C, where C(i,j) = 1 if i ~= j, and C(i,j) = 0 if i = j. C(i,j) is the cost of classifying a point into class j if its true class is i (that is, the rows correspond to the true class and the columns correspond to the predicted class).

rocmetrics normalizes the K-by-K cost matrix C to a 2-by-2 matrix for each one-versus-all binary classification:

C_k is the cost matrix for the kth binary classification in which the positive class is the kth class, where costk(N|P) is the cost of misclassifying a positive class as a negative class, and costk(P|N) is the cost of misclassifying a negative class as a positive class.

For class k, let π_k⁺ and π_k^- be K-by-1 vectors with the following values:

π_ki⁺ and π_ki^- are the ith elements of π_k⁺ and π_k^-, respectively.

The cost of classifying a positive-class (class k) observation into the negative class (the rest) is

Similarly, the cost of classifying a negative-class observation into the positive class is

Classification Score Input for rocmetrics

rocmetrics accepts classification scores (Scores) in a matrix of size n-by-K or a vector of length n, where n is the number of observations and K is the number classes. For cross-validated data, Scores can be a cell array of vectors or a cell array of matrices.

You can use rocmetrics with any classifier or any function that returns a numeric score for an instance of input data.

For some classifiers, you can interpret the score as the posterior probability of observing an instance of a class given an observation. An example of such a score is the fraction of observations for a certain class in a leaf of a decision tree. In this case, scores fall into the range from 0 to 1, and scores from all classes add up to 1. Other functions can return scores ranging between minus and plus infinity, without any obvious mapping from the score to the posterior class probability.

rocmetrics does not impose any requirements on the input score range. Because of this lack of normalization, you can use rocmetrics to process scores returned by any classification, regression, or fit functions. rocmetrics does not make any assumptions about the nature of input scores.

rocmetrics is intended for use with classifiers that return scores, not those that return only predicted classes. Consider a classifier that returns only classification labels, 0 or 1, for data with two classes. In this case, the performance curve reduces to a single point because the software can split classified instances into positive and negative categories in one way only.

Adjusted Scores for Multiclass Classification Problem

For each class, rocmetrics adjusts the classification scores (input argument Scores of rocmetrics) relative to the scores for the rest of the classes if you specify Scores as a matrix. Specifically, the adjusted score for a class given an observation is the difference between the score for the class and the maximum value of the scores for the rest of the classes.

For example, if you have [s₁,s₂,s₃] in a row of Scores for a classification problem with three classes, the adjusted score values are [s₁-max(s₂,s₃),s₂-max(s₁,s₃),s₃-max(s₁,s₂)].

rocmetrics computes the performance metrics using the adjusted score values for each class.

For a binary classification problem, you can specify Scores as a two-column matrix or a column vector. Using a two-column matrix is a simpler option because the predict function of a classification object returns classification scores as a matrix, which you can pass to rocmetrics. If you pass scores in a two-column matrix, rocmetrics adjusts scores in the same way that it adjusts scores for multiclass classification, and it computes performance metrics for both classes. You can use the metric values for one of the two classes to evaluate the binary classification problem. The metric values for a class returned by rocmetrics when you pass a two-column matrix are equivalent to the metric values returned by rocmetrics when you specify classification scores for the class as a column vector.

Model Operating Point

The model operating point represents the FPR and TPR corresponding to the typical threshold value.

The typical threshold value depends on the input format of the Scores argument (classification scores) specified when you create a rocmetrics object:

For a binary classification problem, you can specify Scores as a two-column matrix or a column vector. However, if the classification scores are not posterior probabilities, you must specify Scores as a matrix. A binary classifier classifies an observation into a class that yields a larger score, which is equivalent to a class that yields a nonnegative adjusted score. Therefore, if you specify Scores as a matrix for a binary classifier, rocmetrics can find a correct model operating point using the same scheme that it applies to a multiclass classifier. If you specify classification scores that are not posterior probabilities as a vector, rocmetrics cannot identify a correct model operating point because it always uses 0.5 as a threshold for the model operating point.

The plot function displays a filled circle marker at the model operating point for each ROC curve (see ShowModelOperatingPoint). The function chooses a point corresponding to the typical threshold value. If the curve does not have a data point for the typical threshold value, the function finds a point that has the smallest threshold value greater than the typical threshold. The point on the curve indicates identical performance to the performance of the typical threshold value.

For an example, see Find Model Operating Point and Optimal Operating Point.

Thresholds, Fixed Metric, and Fixed Metric Values

rocmetrics finds the ROC curves and other metric values that correspond to the fixed values (FixedMetricValues name-value argument) of the fixed metric (FixedMetric name-value argument), and stores the values in the Metrics property as a table.

The default FixedMetric value is "Thresholds", and the default FixedMetricValues value is "all". For each class, rocmetrics uses all distinct adjusted score values as threshold values, computes the components of the confusion matrix for each threshold value, and then computes performance metrics using the confusion matrix components.

If you use the default FixedMetricValues value ("all"), specifying a nondefault FixedMetric value does not change the software behavior unless you specify to compute confidence intervals. If rocmetrics computes confidence intervals, then it holds FixedMetric fixed at FixedMetricValues and computes confidence intervals for other metrics. For more details, see Pointwise Confidence Intervals.

If you specify a nondefault value for FixedMetricValues, rocmetrics finds the threshold values corresponding to the specified fixed metric values (FixedMetricValues for FixedMetric) and computes other performance metric values using the threshold values.

The Metrics property includes an additional threshold value that replicates the largest threshold value for each class so that a ROC curve starts from the origin (0,0). The additional threshold value represents the reject-all threshold, for which TP = FP = 0 (no positive instances, that is, zero true positive instances and zero false positive instances).

Another special threshold in Metrics is the accept-all threshold, which is the smallest threshold value for which TN = FN = 0 (no negative instances, that is, zero true negative instances and zero false negative instances).

Note that the positive predictive value (PPV = TP/(TP+FP)) is NaN for the reject-all threshold, and the negative predictive value (NPV = TN/(TN+FN)) is NaN for the accept-all threshold.

NaN Score Values

rocmetrics processes NaN values in the classification score input (Scores) in one of two ways:

Consider an example with two rows in the positive class and two rows in the negative class, each pair having a NaN score:

If you discard rows with NaN scores (NaNFlag="omitnan"), then as the score threshold varies, rocmetrics computes performance metrics as shown in the following table. For example, a threshold of 0.5 corresponds to the middle row where rocmetrics classifies rows 1 and 3 correctly and omits rows 2 and 4.

If you add rows with NaN scores to the false category in their respective classes (NaNFlag="includenan"), rocmetrics computes performance metrics as shown in the following table. For example, a threshold of 0.5 corresponds to the middle row where rocmetrics counts rows 2 and 4 as incorrectly classified. Notice that only the FN and FP columns differ between these two tables.