Significance test of ROC value with Perfcurve (obtaining the p value)         

5 次查看（过去 30 天）
Farshad
Farshad 2014-8-18
评论： Ilya 2015-12-2
I would like to compare two populations of data points using ROC analysis (calculating area under the curve). The ROC values are between 0.5 and 1:00. I would like to examine whether the ROC value is significantly different from 0.5. One approach is to randomly assign the data points to two populations (say 1000 times) and then check the significance. My question is that whether 'perfcurve' can perform such a significance test and provide the p value. If yes, could you please explain in detail (or with example) how the routine should be written. Thanks

请先登录，再回答此问题。

回答（1 个）

Ilya
Ilya 2014-8-19
I can't explain in detail how the routine should be written because I don't understand how you are comparing the two populations. But I can give you an idea for what you could do. If you have a sufficiently recent version of perfcurve, you can use the 'NBoot' parameter to estimate the curve uncertainty by bootstrap. The 4th output from perfcurve is then a confidence interval for AUC at the test size specified by the 'Alpha' parameter. If this interval does not include 0.5, you can conclude that AUC is statistically different from 0.5 at this value of alpha.
  2 个评论
Marta
Marta 2015-11-5
Hi! Is there any equally simple way to see whether AUCs from different classifiers are significantly different? Eq (3) of http://pubs.rsna.org/doi/pdf/10.1148/radiology.148.3.6878708 suggests a simple method for this that relies on estimates of the standard deviation of the AUC. Would you estimate the std(AUC) using the confidence intervals from the bootstrapping as detailed in your answer?
Many thanks, Marta
Ilya
Ilya 2015-12-2
You can estimate the standard deviation of AUC using confidence intervals from bootstrap. The paper you are referring to prescribes estimating correlation between AUCs produced by two classifiers. This can be done by bootstrap too, but clearly you would need to write your own code for that.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 ROC - AUC 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by