Main Content

isanomaly

Find anomalies in data using isolation forest

Since R2021b

collapse all in page

    Syntax

    tf = isanomaly(forest,Tbl)
    tf = isanomaly(forest,X)
    tf = isanomaly(___,Name=Value)
    [tf,scores] = isanomaly(___)

    Description

    tf = isanomaly(forest,Tbl) finds anomalies in the table Tbl using the IsolationForest object forest and returns the logical array tf, whose elements are true when an anomaly is detected in the corresponding row of Tbl. You must use this syntax if you create forest by passing a table to the iforest function.

    example

    tf = isanomaly(forest,X) finds anomalies in the matrix X. You must use this syntax if you create forest by passing a matrix to the iforest function.

    tf = isanomaly(___,Name=Value) specifies options using one or more name-value arguments in addition to any of the input argument combinations in the previous syntaxes. For example, set ScoreThreshold=0.5 to identify observations with scores above 0.5 as anomalies.

    example

    [tf,scores] = isanomaly(___) also returns an anomaly score in the range [0,1] for each observation in Tbl or X. A score value close to 0 indicates a normal observation, and a value close to 1 indicates an anomaly.

    Examples

    collapse all

    Open Live Script

    Create an IsolationForest object for uncontaminated training observations by using the iforest function. Then detect novelties (anomalies in new data) by passing the object and the new data to the object function isanomaly.

    Load the 1994 census data stored in census1994.mat. The data set consists of demographic data from the US Census Bureau to predict whether an individual makes over $50,000 per year.

    load census1994

    census1994 contains the training data set adultdata and the test data set adulttest.

    Train an isolation forest model for adultdata. Assume that adultdata does not contain outliers.

    rng("default") % For reproducibility
[Mdl,tf,s] = iforest(adultdata);

    Mdl is an IsolationForest object. iforest also returns the anomaly indicators tf and anomaly scores s for the training data adultdata. If you do not specify the ContaminationFraction name-value argument as a value greater than 0, then iforest treats all training observations as normal observations, meaning all the values in tf are logical 0 (false). The function sets the score threshold to the maximum score value. Display the threshold value.

    Mdl.ScoreThreshold
    ans = 
0.8600

    Find anomalies in adulttest by using the trained isolation forest model.

    [tf_test,s_test] = isanomaly(Mdl,adulttest);

    The isanomaly function returns the anomaly indicators tf_test and scores s_test for adulttest. By default, isanomaly identifies observations with scores above the threshold (Mdl.ScoreThreshold) as anomalies.

    Create histograms for the anomaly scores s and s_test. Create a vertical line at the threshold of the anomaly scores. 

    histogram(s,Normalization="probability")
hold on
histogram(s_test,Normalization="probability")
xline(Mdl.ScoreThreshold,"r-",join(["Threshold" Mdl.ScoreThreshold]))
legend("Training Data","Test Data",Location="northwest")
hold off

    Figure contains an axes object. The axes object contains 3 objects of type histogram, constantline. These objects represent Training Data, Test Data.

    Display the observation index of the anomalies in the test data. 

    find(tf_test)
    ans = 
15655

    The anomaly score distribution of the test data is similar to that of the training data, so isanomaly detects a small number of anomalies in the test data with the default threshold value. You can specify a different threshold value by using the ScoreThreshold name-value argument. For an example, see Specify Anomaly Score Threshold.

    Open Live Script

    Specify the threshold value for anomaly scores by using the ScoreThreshold name-value argument of isanomaly.

    Load the 1994 census data stored in census1994.mat. The data set consists of demographic data from the US Census Bureau to predict whether an individual makes over $50,000 per year.

    load census1994

    census1994 contains the training data set adultdata and the test data set adulttest.

    Train an isolation forest model for adultdata.

    rng("default") % For reproducibility
[Mdl,tf,scores] = iforest(adultdata);

    Plot a histogram of the score values. Create a vertical line at the default score threshold.

    histogram(scores,Normalization="probability");
xline(Mdl.ScoreThreshold,"r-",join(["Threshold" Mdl.ScoreThreshold]))

    Figure contains an axes object. The axes object contains 2 objects of type histogram, constantline.

    Find the anomalies in the test data using the trained isolation forest model. Use a different threshold from the default threshold value obtained when training the isolation forest model.

    First, determine the score threshold by using the isoutlier function.

    [~,~,U] = isoutlier(scores)
    U = 
0.7449

    Specify the value of the ScoreThreshold name-value argument as U.

    [tf_test,scores_test] = isanomaly(Mdl,adulttest,ScoreThreshold=U);
histogram(scores_test,Normalization="probability")
xline(U,"r-",join(["Threshold" U]))

    Figure contains an axes object. The axes object contains 2 objects of type histogram, constantline.

    Input Arguments

    collapse all

    Trained isolation forest model, specified as an IsolationForest object.

    Predictor data, specified as a table. Each row of Tbl corresponds to one observation, and each column corresponds to one predictor variable. Multicolumn variables and cell arrays other than cell arrays of character vectors are not allowed.

    If you train forest using a table, then you must provide predictor data by using Tbl, not X. All predictor variables in Tbl must have the same variable names and data types as those in the training data. However, the column order in Tbl does not need to correspond to the column order of the training data.

    Data Types: table

    Predictor data, specified as a numeric matrix. Each row of X corresponds to one observation, and each column corresponds to one predictor variable.

    If you train forest using a matrix, then you must provide predictor data by using X, not Tbl. The variables that make up the columns of X must have the same order as the training data.

    Data Types: single | double

    Name-Value Arguments

    Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

    Example: ScoreThreshold=0.75,UseParallel=true sets the threshold for the anomaly score to 0.75 and instructs the function to run computations in parallel.

    Threshold for the anomaly score, specified as a numeric scalar in the range [0,1]. isanomaly identifies observations with scores above the threshold as anomalies.

    The default value is the ScoreThreshold property value of forest.

    Example: ScoreThreshold=0.5

    Data Types: single | double

    Flag to run in parallel, specified as a numeric or logical 1 (true) or 0 (false). If you specify UseParallel=true, the isanomaly function executes for-loop iterations by using parfor. The loop runs in parallel when you have Parallel Computing Toolbox™.

    Example: UseParallel=true

    Data Types: logical

    Output Arguments

    collapse all

    Anomaly indicators, returned as a logical column vector. An element of tf is true when the observation in the corresponding row of Tbl or X is an anomaly, and false otherwise. tf has the same length as Tbl or X.

    isanomaly identifies observations with scores above the threshold (the ScoreThreshold value) as anomalies.

    Anomaly scores, returned as a numeric column vector whose values are in the range [0,1]. scores has the same length as Tbl or X, and each element of scores contains an anomaly score for the observation in the corresponding row of Tbl or X. A score value close to 0 indicates a normal observation, and a value close to 1 indicates an anomaly.

    More About

    collapse all

    Isolation Forest

    The isolation forest algorithm [1] detects anomalies by isolating anomalies from normal points using an ensemble of isolation trees.

    The iforest function creates an isolation forest model (ensemble of isolation trees) for training observations and detects outliers (anomalies in the training data). Each isolation tree is trained for a subset of training observations as follows:

    1. iforest draws samples without replacement from the training observations for each tree.

    2. iforest grows a tree by choosing a split variable and split position uniformly at random. The function continues until every sample reaches a separate leaf node for each tree.

    This algorithm assumes the data has only a few anomalies and they are different from normal points. Therefore, an anomaly reaches a separate leaf node closer to the root node and has a shorter path length (the distance from the root node to the leaf node) than normal points. The iforest function identifies outliers using anomaly scores that are defined based on the average path lengths over all isolation trees.

    The isanomaly function uses a trained isolation forest model to detect anomalies in the data. For novelty detection (detecting anomalies in new data with uncontaminated training data), you can train an isolation forest model with uncontaminated training data (data with no outliers) and use it to detect anomalies in new data. For each observation of the new data, the function finds the corresponding leaf node in each tree, finds the average path length to reach a leaf node from the root node in the trained isolation forest model, and returns an anomaly indicator and score.

    For more details, see Anomaly Detection with Isolation Forest.

    Anomaly Scores

    The isolation forest algorithm computes the anomaly score s(x) of an observation x by normalizing the path length h(x):

    s(x)=2−E[h(x)]c(n),

    where E[h(x)] is the average path length over all isolation trees in the isolation forest, and c(n) is the average path length of unsuccessful searches in a binary search tree of n observations.

    • The score approaches 1 as E[h(x)] approaches 0. Therefore, a score value close to 1 indicates an anomaly.

    • The score approaches 0 as E[h(x)] approaches n – 1. Also, the score approaches 0.5 when E[h(x)] approaches c(n). Therefore, a score value smaller than 0.5 and close to 0 indicates a normal point.

    Algorithms

    isanomaly considers NaN, '' (empty character vector), "" (empty string), <missing>, and <undefined> values in Tbl and NaN values in X to be missing values.

    isanomaly uses observations with missing values to find splits on variables for which these observations have valid values. The function might place these observations in a branch node, not a leaf node. Then isanomaly computes anomaly scores by using the distance from the root node to the branch node. The function places an observation with all missing values in the root node, so the score value becomes 1.

    References

    [1] Liu, F. T., K. M. Ting, and Z. Zhou. "Isolation Forest," 2008 Eighth IEEE International Conference on Data Mining. Pisa, Italy, 2008, pp. 413-422.

    Extended Capabilities

    Version History

    Introduced in R2021b

    See Also

    | | | | |

    Topics