Different deep learning training behavior between MATLAB 2020a and 2021b

Question

Ashley 2022-2-25

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1658510-different-deep-learning-training-behavior-between-matlab-2020a-and-2021b

评论： Ashley 2022-2-28

I have been using this code to train semantic segmentation networks:

function train_deeplab(pth,classes,classNames,sz)
pthTrain=[pth,'training\'];
pthVal=[pth,'validation\'];
% make training datastore
Trainim=[pthTrain,'im\'];
Trainlabel=[pthTrain,'label\'];
imdsTrain = imageDatastore(Trainim);
pxdsTrain = pixelLabelDatastore(Trainlabel,classNames,classes);
pximdsTrain = pixelLabelImageDatastore(imdsTrain,pxdsTrain);
tbl = countEachLabel(pxdsTrain);
% make validation datastore
Valim=[pthVal,'im\'];
Vallabel=[pthVal,'label\'];
imdsVal = imageDatastore(Valim);
pxdsVal = pixelLabelDatastore(Vallabel,classNames,classes);
pximdsVal = pixelLabelImageDatastore(imdsVal,pxdsVal);
% set training options
options = trainingOptions('adam',...
    'MaxEpochs',8,...
    'MiniBatchSize',5,...
    'Shuffle','every-epoch',...
    'ValidationData',pximdsVal,...
    'ValidationPatience',6,...
    'InitialLearnRate',0.0005,...
    'LearnRateSchedule','piecewise',...
    'LearnRateDropPeriod',1,...
    'LearnRateDropFactor',0.75,...
    'ValidationFrequency',128,...
    'ExecutionEnvironment','gpu',...
    'Plots','training-progress',... 
    'OutputFcn', @(info)savetrainingplot(info,pth)); 
% design network
numclass = numel(classes);
imageFreq = tbl.PixelCount ./ tbl.ImagePixelCount;
classWeights = median(imageFreq) ./ imageFreq;
lgraph = deeplabv3plusLayers([sz sz 3],numclass,"resnet50");
pxLayer = pixelClassificationLayer('Name','labels','Classes',tbl.Name,'ClassWeights',classWeights);
lgraph = replaceLayer(lgraph,"classification",pxLayer);
% train
[net, info] = trainNetwork(pximdsTrain,lgraph,options);
save([pth,'net.mat'],'net','info');
end
% save a png of training progress when finished
function stop=savetrainingplot(info,pthSave)
    stop=false;  
    if info.State=='done'
        exportapp(findall(groot, 'Type', 'Figure'),[pthSave,'training_process_21.png'])
    end
end

Since switching from MATLAB 2020a to 2021b, there is something strange happening with the validation loss. My training and validation accuracy are very similar, but my validation loss is orders of magnitude higher than training loss. Here I include a sample network trained with the code above using identical training & validation datasets in MATLAB 2020a vs 2021b to illustrate the problem.

Trained using MATLAB 2020a (training and validation loss/accuracy are similar):

Trained using MATLAB 2021b (validation loss is much higher than training loss while accuracies remain similar):

I appreciate any help!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

yanqi liu 2022-2-26

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1658510-different-deep-learning-training-behavior-between-matlab-2020a-and-2021b#answer_904835

yes，sir，may be use rgn('default') or rand('seed', 0) to make same run environment

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Ashley 2022-2-28

Hi Yanqi,

I don't think that's the problem. I tried training in 2021b after presetting the random number generator like you suggeted (rng('default');rand('seed',0)) but the validation loss in 2021b is still acting strangely. My validation accuracy is still similar to training accuracy but the validation loss is orders of magnitude higher than training loss:

I don't think this is a discrepency between different initializations of the deep learning training but could be something different between how MATLAB 2020a and 2021b compute the loss function.

Thank you

请先登录，再进行评论。