Accuracy of ANN confusion matrix is not correct

Question

Keshasni 2024-4-16

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2107506-accuracy-of-ann-confusion-matrix-is-not-correct

回答： Arnav 2024-9-19

ANNMODEL.m

Hi, i been working on ANN to design a prediction model.This is my coding.But after i run the coding the accuracy in confusion matrix is not same as the final accuracy using formula

clear, close all

clc

% help nndatasets

load ('ann.mat');

RandStream.setGlobalStream(RandStream ('mt19937ar','seed',1));

x = [aiparameter.mc1,aiparameter.torque1];

y = aiparameter.output;

a = isnan(y);

[u,~] = find(a==0);

x = x(u,:); y = y(u);

x = x'; y = y';

% x=rows2vars(m);

% y=rows2vars(n);

%[y,n]=vec2ind(y1);%

%y(y == 2) = 0;

len_ = length(y);

temp = rand(1,length(y));

[~, ind] = sort(temp);

x = x(:, ind);

y = y(:, ind);

y_store = categorical(y);

b = 1:4;

for ii = 1:len_

yy(:,ii) = (b == y(ii));

end

y = single(yy);

net = patternnet;

% Divide data into k-folds

kfold = 5; %10

fold = cvpartition(len_, 'kfold', kfold);

% NN train for every fold

for k = 1 : kfold

% Call index of training & testing sets

trainIdx = fold.training(k); testIdx = fold.test(k);

% Call training & testing features and labels

xtrain = x(:,trainIdx);

ytrain = y(:,trainIdx);

xtest = x(:,testIdx);

ytest = y(:,testIdx);

% configure network

net = configure(net,x,y);

trInd = find(trainIdx); tstInd = find(testIdx);

net.divideFcn = 'divideind';

net.divideParam.trainInd = trInd;

net.divideParam.testInd = tstInd;

% Initialize network

net.layers{1}.name='Hidden Layer 1';

net.layers{2}.name='Output Layer';

net.layers{1}.size = 12; % number of hidden neuron (1-10)

net.layers{1}.transferFcn = 'satlin';

net.trainFcn = 'trainlm'; % training algorithm (trainlm, trainscg, etc)

net.performFcn = 'mse';

net.trainParam.epochs = 10000;

% Train the Network

[net, tr] = train(net, x, y);

% Test the Network

yhat = net(x);

% adjust threshold

thr = 0.5;

% for ii = 1:1:len_

% if yhat(:,ii) >= thr

% yhat(:,ii) = 1;

% else

% yhat(:,ii) = 0;

% end

clear Ypred

for ii = 1:len_

[~,Ypred(ii)] = max(yhat(:,ii));

end

Ypred = categorical(Ypred);

% validate trained network

e = gsubtract(y, yhat);

performance = perform(net, y, yhat);

perf_fold(k)=performance;

% Calculate Training and Testing Performance (Confusion Matrix)

x_trn = x(:,tr.trainInd);

x_tst = x(:,tr.testInd);

y_trn = y_store(:,tr.trainInd);

y_tst = y_store(:,tr.testInd);

yhat_trn = Ypred(:,tr.trainInd);

yhat_tst = Ypred(:,tr.testInd);

figure, plotconfusion(y_trn, yhat_trn, 'Training');

figure, plotconfusion(y_tst, yhat_tst, 'Testing');

% for ii = 1:10%length(y_tst)

% [y_class(ii),~] = find(y_tst(:,ii) == 1);

% [yhat_class(ii),~] = find(yhat_tst(:,ii) == 1);

% end

C=confusionmat(y_tst,yhat_tst)

tp(k)=C(1,1);

fn(k)=C(1,2);

tn(k)=C(2,2);

fp(k)=C(2,1);

end

TP=round(mean(tp));

FN=round(mean(fn));

TN=round(mean(tn));

FP=round(mean(fp));

accuracy=(TP+TN)/(TP+FN+TN+FP)*100

sensitivity=TP/(TP+FN)*100

specificity=TN/(TN+FP)*100

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Poorna 2024-4-21

Can you provide the "ann.mat" file?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Arnav 2024-9-19

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2107506-accuracy-of-ann-confusion-matrix-is-not-correct#answer_1519260

在 MATLAB Online 中打开

Hi @Keshasni,

I understand that you are trying to perform k-fold cross-validation training of the ANN model. Here, there are 5 folds used. Also, the model is used to classify between 4 classes. You are trying to find the values of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN) for each fold and then trying to find out the accuracy, sensitivity and specificity.

There are a few issues in the code provided below:

    tp(k)=C(1,1);
    fn(k)=C(1,2);
    tn(k)=C(2,2);
    fp(k)=C(2,1); 

Since there are 4 classes, the confusion matrix for each fold will be a 4X4 matrix and not a 2X2 matrix. Hence, the value of TP (and other values) cannot be simply taken from the Confusion Matrix by indexing.

For a multi-class classification problem, there are 2 ways in which the estimates are defined.

Firstly, it is necessary to calculate TP, TN, FP and FN for each of the classes. It should be noted that these values are vectors holding one value for each of the classes. This can be done as shown below:

    % b is the vector of classes
    for i = b
        TP(i) = C(i, i);
        FN(i) = sum(C(i, :)) - TP(i);
        FP(i) = sum(C(:, i)) - TP(i);
        TN(i) = sum(C(:)) - (TP(i) + FN(i) + FP(i));
    end

Macro Averaging: This method calculates the metric for each class and then averages them. For instance, the macro averaged specificity can be calculated as:

macroSpecificity = mean(TN ./ (TN + FP));

Mirco Averaging: This method sums the confusion matrices for all the classes then it computes the metric. For instance, the micro averaged specificity can be calculated as:

microSpecificity = sum(TN) / (sum(TN) + sum(FP));

Generally, macro-averaging is preferred as it gives equal weight to all the classes as opposed to micro-averaging which gives more weight to majority classes.

It should be noted that the above calculations only make sense for each of the folds separately. An ensemble of the models trained for each fold should be tested separately to find out its performance metrics.

Hope it helped you!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Accuracy of ANN confusion matrix is not correct

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

Community Treasure Hunt

Accuracy of ANN confusion matrix is not correct

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

回答（1 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论