Why Training Set accuracy decrease dramatically after stopping the trainNetwork?

2018 12 28
1 个回答
更新时间：2019 2 11
3 次查看（30 天）

0 个投票

After stopping manually trainNetworktrainNetwork, the validation error dropped dramatically:
I tested the Training Set accuracy, and got also about 60%:
predY = classify(net,xTrain);
Any ideas what I'am doing wrong?

4 个评论
显示 2更早的评论 隐藏 2更早的评论

Don Mathis
Don Mathis 2019-1-23
What is your network architecture? Does it contain dropoutLayers and later BatchNormlization layers?
Sergy Stepura
Sergy Stepura 2019-2-1
编辑：Sergy Stepura 2019-2-4
The network has simple architecture, 5 fully connected layers with batch normalization + Input layer + Output layer (softmax):
1 '' Image Input 120x1x4 images with 'zerocenter' normalization
2 '' Fully Connected 65 fully connected layer
3 '' Batch Normalization Batch normalization
4 '' ReLU ReLU
5 '' Fully Connected 65 fully connected layer
6 '' Batch Normalization Batch normalization
7 '' ReLU ReLU
8 '' Fully Connected 65 fully connected layer
9 '' Batch Normalization Batch normalization
10 '' ReLU ReLU
11 '' Fully Connected 65 fully connected layer
12 '' Batch Normalization Batch normalization
13 '' ReLU ReLU
14 '' Fully Connected 65 fully connected layer
15 '' Batch Normalization Batch normalization
16 '' ReLU ReLU
17 '' Fully Connected 3 fully connected layer
18 '' Softmax softmax
19 '' Classification Output crossentropyex

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Don Mathis
Don Mathis 2019-2-8

0 个投票

Maybe your minibatch size is too small. The accuracy drop may be due to batchnormalization layers getting finalized, during which time the mean and variance of the incoming activations of each batchnorm layer are computed using the whole training set. If those full-batch statistics don't match the minibatch statistics very well, the finalized batchnorm layers will not be performing a very good normalization.

3 个评论
显示 1更早的评论 隐藏 1更早的评论

Sergy Stepura
Sergy Stepura 2019-2-8
编辑：Sergy Stepura 2019-2-8
I’m training on 30,000 data set and using batches of 500, do you sugest to use 30,000 batch size?
I’m want to see some stable performance before increasing to full 2,000,000 data set.
Don Mathis
Don Mathis 2019-2-11
You could try increasing the batch size iteratively to see whether that fixes the problem. I would try exponentially increasing: 1000, 2000, 4000, 8000, etc. Or you can just try the largest amount that will fit in your GPU memory right away.
Don Mathis
Don Mathis 2019-2-11
Also: Why does your plot show "Iterations per epoch: 1"? Were you using miniBatchSize=30000 in that run?
What are you passing to trainingOptions()?

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

产品

版本

R2018b

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by