MATLAB Answers

incremental training (large data set)

2 views (last 30 days)
Michal Kvasnicka
Michal Kvasnicka on 18 Feb 2020
Edited: Michal Kvasnicka on 26 Mar 2020
Is the following approach suitable for NN incremental learning in general?
for bn = 1:num_batches
inputs = <get batch bn inputs>;
targets = <get batch bn targets>;
net = train(net, inputs, targets);
Is there any recomanded method how to split large data set on a separate batches and how to perform validation and testing?
I will be happy for any relevant reference and/or matlab examples (tutorial).


Sign in to comment.

Answers (1)

Sai Bhargav Avula
Sai Bhargav Avula on 26 Mar 2020
You can try using tall array. Datastore object can also be used for this purpose. You can split the data using cvpartition for the validation and testing
Hope this helps!


Show 4 older comments
Michal Kvasnicka
Michal Kvasnicka on 26 Mar 2020
I am bit confused now. So, all these above mentioned methods are suitable for learning process which realize continuous integration of new information? The incremental learning is something else?
Could you provide any example or tutorial how to integrate new information represented by separate batch file to NN (shallow or deep) in a case of function approximation (regression) problem?
I need to train one NN by several batches of data represented by separate data files. So final NN should integrate information from all separate data batches.
Michal Kvasnicka
Michal Kvasnicka on 26 Mar 2020
OK ... and my type of problem correspond to the "Batch Training" or "Incremental Training with adapt"?
I am still not sure, which method is suitable for integration of new information to one NN. Is the proper method "Batch Training" in my case?
May be, should be better to communicate directly via email.

Sign in to comment.




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by