How to avoid NaN in the Mini-batch-loss from traning convolutional neural network?

Question

AlexanderTUE 2017-4-27

2
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network

评论： Kelvin Owusu 2024-1-8

Hello,

I´m working on training a convolutional neural network following the example from https://de.mathworks.com/help/nnet/examples/create-simple-deep-learning-network-for-classification.html. I have 2000 images in each of 8 label categories and use 90% for training and 10% for testing. The images are in .jpg format and have a size 512x512x1. The arciture of the CNN is currently as follows:

          layers = [imageInputLayer([512 512 1])
                   convolution2dLayer(5,15)
                   reluLayer
                   maxPooling2dLayer(2,'Stride',2)
                   fullyConnectedLayer(8)
                   softmaxLayer
                   classificationLayer()];
         options = traningOptions('sgdm','MaxEpochs',15,'InitialLearnRate',0.001, 'ExecutionEnvironment', 'parallel' );

After training the first epoch the mini-batch loss is going to be NaN and the accuracy is around the chance level. The reason for this is probably that the back probagating generates NaN weights.

How can I avoid this problem? Thanks for the answers!

9 个评论
显示 7更早的评论隐藏 7更早的评论

Javier Pinzón 2017-7-25

编辑：Javier Pinzón 2017-7-25

Hello everybody,

Because i have been experienced some issues with PNG format images, I highlight recommend to use JPG/JPEG format, that is because sometimes, due to some layers that a PNG image has, it take the last layer and the image becomes the color of this layer, i.e., all the image is converted to a black or red... image. so, when you send these image to the network, it only will se one color image... nothing related to the rest of the images and the network will not be able to learn the features. Also be careful with the size of your filters. Also Johannes answer might be a solution in some cases.

Hope it helps,

Javier

_____________________

Edit:

Be careful with the size of your input image... When it is really big, as happened with Alexander, using only one convolution will be really difficult to the networ to learn, because will have only one structure of weights for a really big amount of features that the network want to learn. I would recomend use at least 2 or 3 convolution for that size, even a size of 128x128, and to use Pooling layers to reduce the size that will enter to the Fully-conneced layer, because it will help but to classify the features extracted.

Javier Pinzón 2017-9-6

Hello Alexander,

I'm happy that you where able to solve with your problem. Any Question be free to ask.

Greg Heath 2017-9-7

Comment by Ashok kumar on 6 Jun 2017

MOVED FROM AN ACCEPTED ANSWER BOX

What is the mini batch loss in the table in command window and how it is calculated ??

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Javier Pinzón 2017-9-8

4
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network#answer_280767

在 MATLAB Online 中打开

I will provie the best comments as an answer that can help to solve this problem o NaN Accuracy:

Hello everybody,

Because i have been experienced some issues with PNG format images, I highlight recommend to use JPG/JPEG format, that is because sometimes, due to some layers that a PNG image has, it take the last layer and the image becomes the color of this layer, i.e., all the image is converted to a black or red... image. so, when you send these image to the network, it only will se one color image... nothing related to the rest of the images and the network will not be able to learn the features. Also be careful with the size of your filters. Also Johannes answer might be a solution in some cases.

___________________

Be careful with the size of your input image... When it is really big, as happened with Alexander, using only one convolution will be really difficult to the network to learn, because will have only one structure of weights for a really big amount of features that the network want to learn. I would recomend use at least 2 or 3 convolution for that size, even a size of 128x128, and to use Pooling layers to reduce the size that will enter to the Fully-conneced layer, because it will help but to classify the features extracted.

___________________

To initialize the weights, you need to define the convolution layer before the Layer struct:

conv1 = convolution2dLayer(F,D,'Padding',0,...
                     'BiasLearnRateFactor',2,...
                     'name','conv1');
conv1.Weights = gpuArray(single(randn([F F 3 D])*0.0001));
conv1.Bias = gpuArray(single(randn([1 1 D])*0.00001+1));

You can initialize weights and the bias if needed. Remember, D is the amount of Filters to be used and F the size of the filter. Then, call your variable in the layer struct

layers = [ ...
    imageInputLayer([128 128 3]);
    conv1;

and that is all.

Hope it helps,

Javier

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Khalid Babutain 2019-10-18

4
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network#answer_396987

I came across this issue because I had it, and I was able to solve it by only lowering the Initial Learning Rate from ('InitialLearnRate',1e-3) to ('InitialLearnRate',1e-5)

3 个评论
显示 1更早的评论隐藏 1更早的评论

Matthew Luka 2022-2-9

Thanks alot. it worked for me too

Kelvin Owusu 2024-1-8

Yh this works l and helped more than expected. I was getting 29% accuracies changing paramaters, but reset them and reduced the the LearnRate I am getting 100% 👏

请先登录，再进行评论。

Answer 3

Salma Hassan 2017-12-20

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network#answer_297137

i have this line in my code [trainedNet,traininfo] = trainNetwork(trainingimages,Layers,opts); when i opened the structure traininfo i got the values of training accuracy and training loss but in the validation (accuracy , loss) i got only the first value and the rest is nan.. what is the problem in this case ??

2 个评论
显示无隐藏无

Javier Pinzón 2017-12-21

hello As,

Please can you open a new threat with this question, and provide screen shoots of your code and results as posible, to se what may be wrong or what is causing the problem?

Thanks

Salma Hassan 2017-12-31

Mr Javier Pinzón, i did a separate question with title "what is causes NaN values in the validation accuracy and loss from traning convolutional neural network and how to avoid it? " in this link https://www.mathworks.com/matlabcentral/answers/375090-what-is-causes-nan-values-in-the-validation-accuracy-and-loss-from-traning-convolutional-neural-n

请先登录，再进行评论。

Answer 4

Poorya Khanali 2021-2-10

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/337587-how-to-avoid-nan-in-the-mini-batch-loss-from-traning-convolutional-neural-network#answer_619897

I have a ResNet when the image size is 35*60 everything works fine (no NaN during the training), but when I change the image size to 59*60 (for different data) the network at the beginning seems to work, but after some epochs the NaN starts to appear. Could you please help me out!