evaluateAnomalyDetection

Evaluate anomaly detection results against ground truth

Syntax

metrics = evaluateAnomalyDetection(predLabels,trueLabels,truePosLabels)

metrics = evaluateAnomalyDetection(predLabels,trueLabels,truePosLabels,Verbose=tf)

Description

metrics = evaluateAnomalyDetection(predLabels,trueLabels,truePosLabels) calculates various metrics to evaluate the quality of the anomaly detection results predLabels against the ground truth trueLabels. truePosLabels lists the class labels in trueLabels that correspond to positive detections.

Note

This function requires the Computer Vision Toolbox™ Automated Visual Inspection Library. You can install the Computer Vision Toolbox Automated Visual Inspection Library from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. To run this function, you will require the Deep Learning Toolbox™.

metrics = evaluateAnomalyDetection(predLabels,trueLabels,truePosLabels,Verbose=tf) also displays the evaluation progress information in the command window.

Examples

collapse all

Calculate Metrics for Trained FCDD Anomaly Detector

This example uses:

Open Live Script

Load test images and corresponding labels, then create a datastore that reads the test data. The data set consists of grayscale images of handwritten digits 0–9.

[Xtest,catLabels] = digitTest4DArrayData;
dsTest = arrayDatastore(Xtest,IterationDimension=4);

Load a pretrained FCDD anomaly detector. This detector has been trained to classify the digit 8 as normal and all other digits as anomalies. Therefore, specify the set of positive labels as the set of digits between 0 and 9, excluding 8.

load("digit8AnomalyDetector.mat");
posLabels = setdiff(string(0:9),"8");

Classify each test image as normal or anomalous.

labelsPred = classify(detector,dsTest);

Calculate metrics for the anomaly detection results.

metrics = evaluateAnomalyDetection(labelsPred,catLabels,posLabels)

Evaluating anomaly detection results
------------------------------------
* Finalizing... Done.
* Data set metrics:

    GlobalAccuracy    MeanAccuracy    Precision    Recall    Specificity    F1Score    FalsePositiveRate    FalseNegativeRate
    ______________    ____________    _________    ______    ___________    _______    _________________    _________________

        0.7662           0.839         0.98971     0.748        0.93        0.85204          0.07                 0.252

metrics = 
  anomalyDetectionMetrics with properties:

              ConfusionMatrix: [2x2 table]
    NormalizedConfusionMatrix: [2x2 table]
               DataSetMetrics: [1x8 table]
                 ClassMetrics: [2x2 table]

Display the confusion matrix.

cm = metrics.ConfusionMatrix

cm=2×2 table
               Normal    Anomaly
               ______    _______

    Normal       465        35  
    Anomaly     1134      3366

Input Arguments

collapse all

`predLabels` — Predicted labels
logical vector

Predicted labels, specified as a logical vector. The label value 0 (false) corresponds to a negative detection (the "Normal" class). The value 1 (true) corresponds to a positive detection (the "Anomaly" class). You can get the predicted labels using the classify function.

Data Types: logical

`trueLabels` — Ground truth labels
numeric vector | logical vector | categorical vector

Ground truth labels for each image, specified as a numeric vector, logical vector, or categorical vector of the same length as predLabels. In order to calculate metrics, the evaluateAnomalyDetection function converts the labels into a logical vector according to the set of positive labels in truePosLabels.

`truePosLabels` — Ground truth positive labels
numeric vector | logical vector | string vector | categorical vector

Ground truth positive labels, specified as a numeric vector, logical vector, string vector, or categorical vector.

When trueLabels is categorical, truePosLabels can be string or categorical.
When trueLabels is numeric or logical, truePosLabels can be numeric or logical.

The evaluateAnomalyDetection function converts all ground truth labels in trueLabels that are in the set of positive labels to a logical true, indicating a positive detection. The function converts all other ground truth labels to a logical false, indicating a negative detection.

`tf` — Display evaluation progress
`1` (default) | `0`

Display evaluation progress information in the command window, specified as a numeric or logical 1 (true) or 0 (false). The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics.

Data Types: logical