Hi Alberto Azzari,
In MATLAB, there isn't a built-in function that directly computes the p-value to test whether the accuracy of a classifier is significantly better than the no information rate (NIR).
However, you can refer to the workflow below for Permutation Testing:
- Compute the No Information Rate (NIR) by finding the proportion of the largest class in your dataset.
- Compute the “observed accuracy” of the classifier on the original dataset.
- Randomly permute the class labels of your dataset many times, each time calculating the accuracy of your classifier with the permuted labels.
- This will give you a distribution of accuracies under Null Hypothesis.
- The p-value is the proportion of accuracies from the permutation test that are equal to or greater than the observed accuracy.
- A low p-value suggests that the observed accuracy is significantly better than NIR.
To prove that a model is significant, the accuracy should be higher than the NIR.
