Evaluate classification/regression performance against noisy annotation

Question

Kyle 2021-10-14

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1564141-evaluate-classification-regression-performance-against-noisy-annotation

回答： Himanshu 2024-8-9，11:57

Is there a way to evaluate classification/regression performance accounting of noisy annotations.

Let's say I have a cat/dog detector and 1000 cat/dog test images. The 1000 images are human annotated, so it's likely there are annotation errors, e.g. some bounding box may be incorrect and some class label may be wrong. Simply draw a confusion matrix or derive an IoU just compare the detector performance with the noisy data, and I don't think such way is appropriate.

So my questions are

What's the appropriate way to estimate the bbox and class error margin in the data set, given it's not possible to go through each of them?
How to incorporate the above annotation error when reporting the performance of the cat/dog detector?

Thanks.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Himanshu 2024-8-9，11:57

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1564141-evaluate-classification-regression-performance-against-noisy-annotation#answer_1497159

Hi,

I see that you are trying to evaluate classification/regression performance while accounting for noisy annotations.

To estimate the bounding box (bbox) and class error margin, start by manually verifying a small, randomly selected subset of the data to determine the error rate. Additionally, employ statistical methods like bootstrapping to estimate error margins from this subset.

When reporting performance, adjust your metrics to account for the estimated annotation noise and include confidence intervals to reflect the uncertainty due to these errors. You can apply noise-robust algorithms to mitigate the impact of annotation noise on performance evaluation.

Please refer to the below documentations for more information.