risk.validation.jeffreysTest

Jeffreys test

Since R2026a

Syntax

hJeffreysTest = risk.validation.jeffreysTest(Probability,NumEvents,NumTrials)

hJeffreysTest = risk.validation.jeffreysTest(Probability,NumEvents,NumTrials,ConfidenceLevel=confidenceLevel)

[hJeffreysTest,Output] = risk.validation.jeffreysTest(___)

Description

hJeffreysTest = risk.validation.jeffreysTest(Probability,NumEvents,NumTrials) returns the Jeffreys test result or results, hJeffreysTest, for a given set of probabilities, events, and trials. The output is 1 if the test rejects the null hypothesis at the 95% confidence level, or 0 otherwise. Probability contains numeric values that represent quantities such as probability of default (PD) estimates.

hJeffreysTest = risk.validation.jeffreysTest(Probability,NumEvents,NumTrials,ConfidenceLevel=confidenceLevel) specifies the confidence level for the Jeffreys test.

[hJeffreysTest,Output] = risk.validation.jeffreysTest(___) also returns a structure Output that contains a table of summary metrics.

example

Examples

collapse all

Apply Jeffreys Test on Probability of Default Data

Open Live Script

Perform a Jeffreys test on aggregated probability of default (PD) data. The Jeffreys test determines whether the PD is greater than the default rate. The file PDRatingGradeData.csv contains a table with variables that include PD, RatingGrade, and Default. These variables, respectively, correspond to the PD, customer rating grade, and whether the customer defaulted.

Load the data.

data = readtable("PDRatingGradeData.csv")

data=500×4 table
       PD       RatingGrade    ObservationYear    Default
    ________    ___________    _______________    _______

     0.40736         9              2020             0   
      0.4529        10              2020             0   
    0.063493         2              2020             0   
     0.45669        10              2020             1   
     0.31618         7              2020             0   
     0.04877         1              2020             0   
     0.13925         3              2020             0   
     0.27344         6              2020             0   
     0.47875        10              2020             0   
     0.48244        10              2020             0   
    0.078807         2              2020             0   
      0.4853        10              2020             0   
     0.47858        10              2020             0   
     0.24269         5              2020             0   
     0.40014         9              2020             0   
    0.070943         2              2020             0   
      ⋮

Use the groupsummary function to aggregate the data by rating grade and calculate the sum and the mean of PD and Default for each group.

aggData = groupsummary(data, "RatingGrade", ["sum", "mean"], ["PD", "Default"])

aggData=10×6 table
    RatingGrade    GroupCount    sum_PD    mean_PD     sum_Default    mean_Default
    ___________    __________    ______    ________    ___________    ____________

         1             56        1.5593    0.027845         6           0.10714   
         2             79        6.0743     0.07689        18           0.22785   
         3             68        8.6793     0.12764        16           0.23529   
         4             54        9.4626     0.17523        21           0.38889   
         5             45        10.206      0.2268        11           0.24444   
         6             40        11.001     0.27502        14              0.35   
         7             40         13.08       0.327        10              0.25   
         8             34        12.796     0.37635        13           0.38235   
         9             37        15.526     0.41964        19           0.51351   
        10             47        22.219     0.47274        22           0.46809

aggData is a table that contains the aggregated data.

Perform the Jeffreys test. Use the average probabilities, total number of defaults, and total number of loans as input arguments.

[hJeffreysTest, Output] = risk.validation.jeffreysTest(aggData.mean_PD, aggData.sum_Default, aggData.GroupCount)

hJeffreysTest = 10×1

     1
     1
     1
     1
     0
     0
     0
     0
     0
     0

Output = struct with fields:
    Results: [10×8 table]

hJeffreysTest contains the results of the hypothesis test and Output contains additional statistics. The results of the hypothesis test show that enough evidence exists to reject the null hypothesis for the first four groups in the hJeffreysTest listing.

Display the Results field of Output.

Output.Results

ans=10×8 table
    RejectTest      PValue      Probability    CriticalValue    NumEvents    NumTrials    ObservedProbability    ConfidenceLevel
    __________    __________    ___________    _____________    _________    _________    ___________________    _______________

        1          0.0020852     0.027845        0.053864           6           56              0.10714               0.95      
        1         1.2882e-05      0.07689         0.15815          18           79              0.22785               0.95      
        1          0.0068992      0.12764         0.15973          16           68              0.23529               0.95      
        1          9.566e-05      0.17523         0.28585          21           54              0.38889               0.95      
        0            0.37742       0.2268         0.15256          11           45              0.24444               0.95      
        0            0.14453      0.27502         0.23637          14           40                 0.35               0.95      
        0            0.85089        0.327         0.15239          10           40                 0.25               0.95      
        0            0.46541      0.37635         0.25591          13           34              0.38235               0.95      
        0            0.12403      0.41964          0.3809          19           37              0.51351               0.95      
        0            0.52436      0.47274         0.35209          22           47              0.46809               0.95

The Results table includes the hypothesis test p-values and the critical values. The variable ObservedProbability variable contains the default rate, which is the quotient of NumEvents and NumTrials.

Input Arguments

collapse all

`Probability` — Probability values
numeric vector with values in the range (0,1)

Probability values, specified as a numeric vector with values in the range (0,1). Probability contains values that indicate quantities such as PD estimates.

`NumEvents` — Number of events observed
numeric vector of nonnegative integers

Number of events observed, specified as a numeric vector of nonnegative integers. For PD models, NumEvents contains the number of defaults observed.

`NumTrials` — Number of trials
numeric vector of positive integers

Number of trials, specified as a numeric vector of positive integers. For PD models, NumTrials contains the number of loans.

`confidenceLevel` — Confidence level
`0.95` (default) | numeric scalar in the range (0,1)

Confidence level of the hypothesis test, specified as a numeric scalar in the range (0,1).

Output Arguments

collapse all

`hJeffreysTest` — Hypothesis test results
numeric vector containing values of `1` or `0`

Hypothesis test results, returned as a numeric vector.

A value of 1 rejects the null hypothesis at the specified confidence level.
A value of 0 fails to reject the null hypothesis at the specified confidence level.

`Output` — Output metrics
structure

Output metrics, returned as a structure with a single field Results. Results contains a table with the following columns:

RejectTest — Numeric vector that indicates whether the null hypothesis was rejected. This column represents the same values as hJeffreysTest.
PValue — p-value for the hypothesis test returned as a scalar in the range [0,1]. A small value indicates that the null hypothesis might not be valid.
Probability — Probability values. This column is the test statistic for the hypothesis test.
CriticalValue — Critical value for the hypothesis test, returned as a numeric scalar. The jeffreysTest function rejects the null hypothesis if the probability of an event is less than the critical value.
NumEvents — Number of events observed.
NumTrials — Number of trials.
ObservedProbability — Default rate, which is the ratio of NumEvents to NumTrials.
ConfidenceLevel — Confidence level for the hypothesis test.

More About

collapse all

Jeffreys Test

The Jeffreys test [1] requires you to input the predicted probability of default PD, the number of observed defaults D, and the number of loans N. The null hypothesis is that the probability PD is greater than the default rate. The jeffreysTest function implements a one-sided version of the test, where the alternative hypothesis is that PD underestimates the event frequency. Rejection of the Jeffreys test means that PD is statistically small compared to the observed default rate D/N. The test statistic for Jeffreys test is PD.

The p-value for the test statistic is given by $β_{a, b} (P D)$ , where β is the cumulative distribution function for the beta distribution with shape parameters a = D+1/2, and b = N – D+1/2.

For more information about the Jeffreys test, see Section 2.5.3.1 in [1].

References

[1] European Central Bank, “Instructions for reporting the validation results of internal models.” February, 2019. https://www.bankingsupervision.europa.eu/activities/internal_models/shared/pdf/instructions_validation_reporting_credit_risk.en.pdf.

Version History

Introduced in R2026a

risk.validation.jeffreysTest

Syntax

Description

Examples

Apply Jeffreys Test on Probability of Default Data

Input Arguments

Probability — Probability values numeric vector with values in the range (0,1)

NumEvents — Number of events observed numeric vector of nonnegative integers

NumTrials — Number of trials numeric vector of positive integers

confidenceLevel — Confidence level 0.95 (default) | numeric scalar in the range (0,1)

Output Arguments

hJeffreysTest — Hypothesis test results numeric vector containing values of 1 or 0

Output — Output metrics structure

More About

Jeffreys Test

References

Version History

See Also

`Probability` — Probability values
numeric vector with values in the range (0,1)

`NumEvents` — Number of events observed
numeric vector of nonnegative integers

`NumTrials` — Number of trials
numeric vector of positive integers

`confidenceLevel` — Confidence level
`0.95` (default) | numeric scalar in the range (0,1)

`hJeffreysTest` — Hypothesis test results
numeric vector containing values of `1` or `0`

`Output` — Output metrics
structure