SSD Object Detector training results in NaN loss and RMSE

Question

Daniel Jampen 2020-5-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/523188-ssd-object-detector-training-results-in-nan-loss-and-rmse

评论： Daniel Jampen 2020-5-12

Hello

I've create an SSD with mobilenetv2 with the example from "Create SSD Object Detection Network". But changed the class count to just 1.

For training I've used the sample from "Object Detection Using SSD Deep Learning".

|=======================================================================================================|
|  Epoch  |  Iteration  |  Time Elapsed  |  Mini-batch  |  Mini-batch  |  Mini-batch  |  Base Learning  |
|         |             |   (hh:mm:ss)   |     Loss     |   Accuracy   |     RMSE     |      Rate       |
|=======================================================================================================|
|       1 |           1 |       00:00:01 |       3.1220 |       50.98% |         3.39 |      1.0000e-05 |
|       4 |          50 |       00:00:27 |          NaN |        0.00% |          NaN |      1.0000e-05 |
|       8 |         100 |       00:00:53 |          NaN |        0.00% |          NaN |      1.0000e-05 |
|      11 |         150 |       00:01:19 |          NaN |        0.00% |          NaN |      1.0000e-05 |

Is there something I'm missing? Is the SSD model created in the first sample not an actual working model?

Best regards

Link Sample 1: https://ch.mathworks.com/help/vision/examples/create-ssd-object-detection-network.html

Link Sample 2: https://ch.mathworks.com/help/deeplearning/ug/object-detection-using-ssd-deep-learning.html

Edit: I've tried decreasing the learning rate with no success.

2 个评论
显示无隐藏无

Sai Bhargav Avula 2020-5-12

can you provide more details like what other changes you made to the code ?

can you share the code that you are working on?

Daniel Jampen 2020-5-12

在 MATLAB Online 中打开

The only thing I've changed is the line in "Create SSD Object detection network"

numClasses = 5;

to

numClasses = 1;

I've done this because the "Object Detection using SSD Deep Learning" uses just one class for training. (Vehicle)

When using more anchor boxes with bigger sizes it trains longer without resulting in NaN. But at somepoint NaN will still show up.

To my knowledge tho, anchorbox size and count should only impact accuracy and not resulting in NaN values?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ryan Comeau 2020-5-10

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/523188-ssd-object-detector-training-results-in-nan-loss-and-rmse#answer_431592

在 MATLAB Online 中打开

Hello,

I do not know the exact thing which may be causing this, but if I had to bed on it, I would check all of the bounding boxes in your data set and make sure they are correctly labelling the objects. If you removed 1 class but left the bounding boxes there it could be finding NaN value in this way. Superimpose the bounding boxed on the image and ensure you have the correct labels and locations of these objects.

Second, the lowest recommended learning rate i've seen in literature(i don't have a specific paper to link her unfortunately) is about 1e-6. Low learning rates like this can cause your network to not converge at all since the weights will never be updated enough. What I recommend is use the learn rate drop schedule that is provided here. Here is a sample of what i've used to achive some satisfactory results.

options = trainingOptions('sgdm',...
    'InitialLearnRate',18.0e-4,...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropFactor',0.7, ...
    'LearnRateDropPeriod',1, ...
    'Verbose',true,...
    'MiniBatchSize',24,...
    'MaxEpochs',8,...
    'Shuffle','every-epoch',...
    'VerboseFrequency',1);

Third, to further gain performance, tune your strides and sizes of convolution kernels, you'll need to adjust this to your specific task i cannot help here.

Hope this helps,

RC