incremental training (large data set)

2020 2 18
1 个回答
更新时间：2020 3 26
5 次查看（30 天）

0 个投票

Is the following approach suitable for NN incremental learning in general?
for bn = 1:num_batches
inputs = <get batch bn inputs>;
targets = <get batch bn targets>;
net = train(net, inputs, targets);
end
Is there any recomanded method how to split large data set on a separate batches and how to perform validation and testing?
I will be happy for any relevant reference and/or matlab examples (tutorial).

请先登录，再回答此问题。

回答（1 个）

Sai Bhargav Avula
Sai Bhargav Avula 2020-3-26

0 个投票

Hi,
You can try using tall array. Datastore object can also be used for this purpose. You can split the data using cvpartition for the validation and testing
Hope this helps!

7 个评论
显示 5更早的评论 隐藏 5更早的评论

Michal
Michal 2020-3-26
My question is more abut general methodology of inceremental training. I am still not sure if the incremental learning is really correct and reliable method how to train NN for large date sets.
Sai Bhargav Avula
Sai Bhargav Avula 2020-3-26
Iam bit confused what you meant by correct. To the best of my knowledge it depends on the application rather than the data. If you want the NN to adapt to individual environment you need to use incremental learning. If you want to continuously integrate new information then regular batch ML algorithms won't work. These algorithms reconstruct new models from scratch. This is the application of incremental learning.
Michal
Michal 2020-3-26
I am sorry for a bit confusing description of my problem. Yes, I want to continuously integrate new information. Where new information is represented by new batches of data.
Is there any suitable approach how to solve this kind of problem?
Sai Bhargav Avula
Sai Bhargav Avula 2020-3-26
if you are looking for the learning functions, the learngdm and learngd are the two functions that can be used for incremental learning.
If you are using ADAPT to train, BLF can be LEARNGD or LEARNGDM. PF can be MSE, MSEREG, or SSE. BTF is not used when using ADAPT to perform incremental training.
trains can be used for Sequential order incremental training with learning functions.
Hope this what you are looking for!
Michal
Michal 2020-3-26
编辑：Michal 2020-3-26
I am bit confused now. So, all these above mentioned methods are suitable for learning process which realize continuous integration of new information? The incremental learning is something else?
Could you provide any example or tutorial how to integrate new information represented by separate batch file to NN (shallow or deep) in a case of function approximation (regression) problem?
I need to train one NN by several batches of data represented by separate data files. So final NN should integrate information from all separate data batches.
Michal
Michal 2020-3-26
编辑：Michal 2020-3-26
OK ... and my type of problem correspond to the "Batch Training" or "Incremental Training with adapt"?
I am still not sure, which method is suitable for integration of new information to one NN. Is the proper method "Batch Training" in my case?
May be, should be better to communicate directly via email.

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

产品

版本

R2019b

标签

提问:

Michal
Michal
2020-2-18

编辑：

Michal
Michal
2020-3-26

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by