Class Activation Mapping gives different scores than classify function?

1 次查看（过去 30 天）
Noa Hoogeweg
Noa Hoogeweg 2022-6-9
回答： aditi bagora 2023-10-27
Hi!
I tried to make a Class Activation Mapping (CAM) code so that I can see the regions on which the classification of my image is based. However, the scores of the CAM don't seem to be in line with the scores I get when I use the classify function. The network I trained has three possible classes: Absent, OnlyMCs, PresentMass.
The code I used, I got from the link below, and I modified it a bit:
https://www.mathworks.com/help/deeplearning/ug/investigate-network-predictions-using-class-activation-mapping.html
See my code here:
load ResNet_3class
dataDir = 'Images';
imds = imageDatastore(dataDir,...
'IncludeSubfolders',true,...
'FileExtensions','.tiff', ...
'LabelSource','foldernames');
inputSize = [512 512 3];
imdsTest_augm = augmentedImageDatastore([inputSize(1) inputSize(2)], imds, 'ColorPreProcessing', 'gray2rgb');
net = ResNet_3class
[LesionPred,scores] = classify(net,imdsTest_augm)
LesionTrue = imds.Labels;
figure(1)
cm=confusionchart(LesionTrue,LesionPred)
%% CAM
classes = net.Layers(end).Classes;
layerName = 'res5b_relu' ;
ImageNumber = 1050;
im = imread(imdsTest_augm.Files{ImageNumber});
im = imresize(im,[inputSize(1),inputSize(1)]);
im = gray2rgb(im);
imageActivations = activations(net,im,layerName);
scores_1 = squeeze(mean(imageActivations,[1 2])); %imAct from size [16 16 512] to [512 1] %= same as GAP layer!!
fcWeights = net.Layers(end-2).Weights; % 'fc1000' layer weights
fcBias = net.Layers(end-2).Bias;
scores_2 = fcWeights*scores_1 + fcBias;
[~, classIds] = maxk(scores_2, 3);
weightVector = shiftdim(fcWeights(classIds(1), :), -1);
classActivationMap = sum(imageActivations.*weightVector, 3);
scores_3 = exp(scores_2)/sum(exp(scores_2));
maxScores = scores_3(classIds);
labels = classes(classIds);
LesionPred(ImageNumber, 1)
scores(ImageNumber,:)
figure(2)
subplot(1, 2, 1)
imshow(im)
TrueLabel = string(LesionTrue(ImageNumber, 1));
title('True Class:' + " " + TrueLabel);
subplot(1, 2, 2)
CAMshow(im, classActivationMap)
title(string(labels) + ", " + string(maxScores));
drawnow
So, for example for a certain image I got with [LesionPred,scores] = classify(net,imdsTest_augm) the scores
0.0021934 0.7136005 0.2842060 . Therefore, the image was classified as an 'OnlyMCs' image by the network. However, the CAM gave me the following outcome:
So the scores 0.99872 0.0012439 4.0788e-05 . Which are very different. Also, the whole image 'heats up', which I don't understand.
I'm really new with deep learning, so maybe it's just a small mistake I'm making. Hopefully someone can help me on my way. Thanks!
(I can't use the GradCAM function because I don't have the license for the newer version of Matlab).

请先登录，再回答此问题。

回答（1 个）

aditi bagora
aditi bagora 2023-10-27
Hi Noa,
I see that you are attempting to generate class activation mapping for your prediction, but the results are not as expected.
Upon reviewing your code, it appears that you have used ResNet for classification into three categories. During my initial review, I noticed that you have created separate variables for scores, and it seems that you are using the incorrect score variable to compute the “classIds”.
To address this issue, I suggest making the following change:
Ensure that the classes are generated before updating the score variable. In your case, the “classIds" should be calculated based on “score_1" rather than "score_2”.
imageActivations = activations(net,im,layerName);
scores_1 = squeeze(mean(imageActivations,[1 2])); %imAct from size [16 16 512] to [512 1] %= same as GAP layer!!
[~, classIds] = maxk(scores_1, 3); % classIds should be calculated from score_1 and not score_2
fcWeights = net.Layers(end-2).Weights; % 'fc1000' layer weights
fcBias = net.Layers(end-2).Bias;
scores_2 = fcWeights*scores_1 + fcBias;
weightVector = shiftdim(fcWeights(classIds(1), :), -1);
classActivationMap = sum(imageActivations.*weightVector, 3);
Please refer to the following link for more information and correct codeflow:
Deep Learning Visualizations: CAM Visualization: https://blogs.mathworks.com/deep-learning/2019/01/31/deep-learning-visualizations-cam-visualization/
Hope this helps and solves your query.
Regards,
Aditi

类别

在 Help Center 和 File Exchange 中查找有关 Recognition, Object Detection, and Semantic Segmentation 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by