Significance test of ROC value with Perfcurve (obtaining the p value)

Question

Farshad 2014-8-18

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/151403-significance-test-of-roc-value-with-perfcurve-obtaining-the-p-value

评论： Ilya 2015-12-2

I would like to compare two populations of data points using ROC analysis (calculating area under the curve). The ROC values are between 0.5 and 1:00. I would like to examine whether the ROC value is significantly different from 0.5. One approach is to randomly assign the data points to two populations (say 1000 times) and then check the significance. My question is that whether 'perfcurve' can perform such a significance test and provide the p value. If yes, could you please explain in detail (or with example) how the routine should be written. Thanks

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ilya 2014-8-19

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/151403-significance-test-of-roc-value-with-perfcurve-obtaining-the-p-value#answer_149156

I can't explain in detail how the routine should be written because I don't understand how you are comparing the two populations. But I can give you an idea for what you could do. If you have a sufficiently recent version of perfcurve, you can use the 'NBoot' parameter to estimate the curve uncertainty by bootstrap. The 4th output from perfcurve is then a confidence interval for AUC at the test size specified by the 'Alpha' parameter. If this interval does not include 0.5, you can conclude that AUC is statistically different from 0.5 at this value of alpha.

2 个评论
显示无隐藏无

Marta 2015-11-5

Hi! Is there any equally simple way to see whether AUCs from different classifiers are significantly different? Eq (3) of http://pubs.rsna.org/doi/pdf/10.1148/radiology.148.3.6878708 suggests a simple method for this that relies on estimates of the standard deviation of the AUC. Would you estimate the std(AUC) using the confidence intervals from the bootstrapping as detailed in your answer?

Many thanks, Marta

Ilya 2015-12-2

You can estimate the standard deviation of AUC using confidence intervals from bootstrap. The paper you are referring to prescribes estimating correlation between AUCs produced by two classifiers. This can be done by bootstrap too, but clearly you would need to write your own code for that.

请先登录，再进行评论。