nnstart and test ROC AUC

Question

0 个投票

Hi all,

I'm doing a project now that requires training a large number of pattern recognizing neural nets. It's my first time using them, so I've been using a "simple script" generated from Matlab's neural network toolbox.

during training, I'm able to look at the ROC for the train, test, and validation sets. I'm wondering if it's possible to somehow extract only the test numeric ROC AUC from the neural network after it's final iteration of training.

Any help would be great!

Thanks, J

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Follow Question

Answer 1

Greg Heath 2013-7-2

编辑：Greg Heath 2013-7-2

在 MATLAB Online 中打开

1 个投票

 clear all, close all, clc, plt=0;
 tic
 [ x, t ] = simpleclass_dataset;
 [ I N ] = size(x)        % [ 2 1000 ]
 [ O N ] = size(t)        % [ 4 1000 ]
 Ntst = round(0.15*N)     % 150 default
 Nval = Ntst              % 150 default
 Ntrn = N-Nval-Ntst       % 700 default
 Ntrneq = prod(size(t))   % 4000 No. of training equations
 % H = No. of hidden nodes    ( I-H-O NN )
 % Nw = (I+1)*H+(H+1)*O = No. of unknown weights
 % NO OVERFITTING <==> Nw < Ntrneq 
 % NO OVERFITTING <==>  H <= Hub  (upperbound)
 Hub = -1 + ceil( ( Ntrneq-O) / (I+O+1) )  % 570

% For a robust design desire Nw << Ntrneq. Therefore, if possible, choose H << Hub. The PATTERNNET default is H = 10. However, the smaller H, the more robust the design. Obtaining 110 designs ( 10 trials each for H = 0:10 shows that ANY OF THESE VALUES CAN YIELD EXCELLENT RESULTS PROVIDED THE RANDOM WEIGHT INITIALIZATION IS NOT TOO POOR. The complete result can be duplicated below by choosing Hmax = 10 with Ntrials = 10. However, the result below for Hmax = 0 with Ntrials = 4 is sufficient to illustrate the concept.

 Hmin = 0
 dH = 1
 Hmax = 0
 Ntrials = 4
 rng(0)                           % Initialize RNG
 j=0
 for h = Hmin:dH:Hmax
     j=j+1
     if h==0
         net = patternnet([]);;
         Nw = (I+1)*O
     else
         patternnet(h);
         Nw = (I+1)*h+(h+1)*O
     end
     for i = 1:Ntrials
         s(i,j) = rng              % Store state of RNG
         net = configure(net,x,t);
         [ net tr y ] = train(net,x,t);
         tstind = tr.testInd;
         ttst = t(:,tstind);
         ytst = y(:,tstind);
         plt=plt+1,figure(plt)
         plotconfusion(ttst,ytst)
         title([ ' TEST SET CONFUSION MATRIX. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
         plt=plt+1,figure(plt)
         plotroc(ttst,ytst)
         title([ ' TEST SET ROC. TRIAL =  ', num2str(10*(j-1)+i )] )
         hold off
     end
 end
 toc                                   % 7.3 sec

Hope this helps.

Thank you for formally accepting my answer

Greg

2 个评论
显示无隐藏无

Jordan 2013-7-2

编辑：Jordan 2013-7-2

在 MATLAB Online 中打开

Thanks for the response Greg. Still it's not totally clear to me. All I really need here is some way to extract the test samples being used from the net object. If I had a list of which samples were being used for testing, I could calculate the AUC of the ROC. I don't need full NN code.

This is what was generated for me by nnstart:

______________

inputs = in'
targets = out
hiddenLayerSize = 30;
net = patternnet(hiddenLayerSize);
net.divideParam.trainRatio = 70/100
net.divideParam.valRatio = 15/100
net.divideParam.testRatio = 15/100
[net, tr] = train(net,inputs,targets);
outputs = net(inputs);
errors = gsubtract(targets, outputs);
performance = perform(net, targets, outputs);

______________

I don't really want to deviate from this code.

Greg Heath 2013-7-4

I gave you the expressions for ttst and ytst.

What else do you need??

Greg

请先登录，再进行评论。

nnstart and test ROC AUC

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

2 个评论
显示无隐藏无

更多回答（0 个）

类别

标签

Community Treasure Hunt

nnstart and test ROC AUC

0 个评论 显示 -2更早的评论 隐藏 -2更早的评论

采纳的回答

2 个评论 显示 无 隐藏 无

更多回答（0 个）

类别

标签

另请参阅

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

2 个评论
显示无隐藏无