Is it normal to stop and save a network for more than 14 hours?

Question

Yi Wei 2020-11-30

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/668483-is-it-normal-to-stop-and-save-a-network-for-more-than-14-hours

编辑： Yi Wei 2020-11-30

I saved my network using the following function when it completed training progress（Fig.1）. But after more than 14 hours, the network still has not finished saving process. The GPU has always been running during the saving process (Fig.2). The training data is about 8T.

net = trainNetwork(trainData,lgraph,options);

save([dir,'Anet'],'net','-v7.3')

Fig. 1

Fig. 2

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Yi Wei 2020-11-30

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/668483-is-it-normal-to-stop-and-save-a-network-for-more-than-14-hours#answer_559903

I do not know if it is because I used the normalization layer. My training data volume is about 8T, since TrainedMean and TrainedVariance are calculated after the training is finished，so this process costs a lot of time.