Are you doing semantic segmentation or image classification?
The confusion matrix helps indicate differences in classifications between true positives, true negatives, false positives, false negatives. It does NOT indicate how much total data was used in training. Rather, it can show how well classification of specific classes is performing.
The details on the large plot in this page gives a better explanation: https://www.mathworks.com/help/deeplearning/ref/plotconfusion.html
