It depends on your situation. So I'm presuming you have some ground truth that tells you whether the algorithm gave the correct answer or not. So you have four situations:
- True positive
- True negative
- False positive
- False negative
So which of the 4 is most important depends on the context. Let's say it's a cancer diagnosis. Is the risk of a false negative (you have cancer but we didn't detect it) worse than a false positive (we told you that you have cancer but you really don't)?
You might want to look up ROC Curve : https://en.wikipedia.org/wiki/Receiver_operating_characteristic
Maybe you look at multiple images before making a decision so each point on the ROC curve could be for a different number of images used in the final decision.
but where on the curve you say is the "optimal" number of images to use is a judgment call. Maybe you just simply want to say that if the true positive ("correct") rate is more than 90% you're happy with that. Or maybe if the risk of being wrong is really high you want the true positive rate to be >99%.
