Neural network: train() behavior with earlier results

Question

Akshay Joshi 2018-1-28

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/379432-neural-network-train-behavior-with-earlier-results

关闭： MATLAB Answer Bot 2021-8-20

I have a very large dataset of around 150GB that I need to process using neural networks. As this data is quite big, I've to break it into chunks, say 5000 elements are sent as 20 batches, each batch containing 250 elements. The following dummy code can be written for this:

for count = 1:num_batches
  inputs = entire_input(1 + (count-1)*num_batches, count * num_batches);
  targets = entire_targets(1 + (count-1)*num_batches, count * num_batches);
  net = train(net, inputs, targets);
end

Will the net again start training with the fresh batch, or will it be able to retain weights calculated for previous batch? As per some of my discussions and findings, with each new batch, the weights start taking shape of current data and may overwrite previous weights.

Please advise if this method works well, or we can use some other method instead of train().

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

此问题已关闭。

Answer 1

Greg Heath 2018-1-29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/379432-neural-network-train-behavior-with-earlier-results#answer_302242

"Need to process" doesn't provide useful information.

What are you trying to design? Curvefitter/Regressor? PatternRecognizer/Unsupervised-Classifier/Supervised-Classifier? Timeseries??

In all cases, training, validation and test data should have similar summary statistics in all run batches. Otherwise training batch n will erase some of what is learned in batches 1 to n-1.

Your response should be far less vague than your original explanation.

Hope this helps.

Greg

Thank you for formally accepting my answer

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Akshay Joshi 2018-1-29

编辑：Akshay Joshi 2018-1-29

在 MATLAB Online 中打开

Hi Greg,

I'm trying to design a supervised classifier with the help of multi layer perceptrons ( feedforwardnet). The input matrix is of 500,000 x 25 dimension, and output matrix 5,000 x 25.

Initially, I tried to train my network using nntool. But I was unable to feed dataset this large (150 GB) into it due to memory constraints, so decided to break data into chunks. For this purpose, I'm writing a Matlab script to create neural network and provide input in chunks.

 In all cases, training, validation and test data should have similar summary statistics in all run batches.
 Otherwise training batch n will erase some of what is learned in batches 1 to n-1.

Can you suggest some method through which we can retain the data of 1 to n-1 batches, and based on that, we calculate the result of say n to n+k batches?

Thanks for the earlier response. Hope I'm clear this time.