Why does matlab.io.​datastore.​MiniBatcha​ble not support paralle processing (multi-gpu training)

2022 3 11
1 个回答
回答已采纳
更新时间：2022 3 16
3 次查看（30 天）

0 个投票

For my application, I need to arrange data in a specific order when they are fed into the CNN during the training stage. To do so, I implemented a custom mini-batchable datastore called CmapDatastore that inherits from the matlab.io.datastore.MiniBatchable class. It works fine with single GPU in training a CNN model. However, when I try to use multiple GPUs in training by setting trainingOptions as:
trainingOptions( ..........
'ExecutionEnvironment', 'multi-gpu', .......)
The error message is:
The MiniBatchable Datastore CmapDatastore does not support parallel operations.
The code of CmapDatastore.m is attached to this messsage.
I would appreciate your help greatly.
Many thanks,
Yong

请先登录，再回答此问题。

 采纳的回答

Yoann Roth
Yoann Roth 2022-3-14

1 个投票

Hello Yong,
To support parallel training for your custom datastore, you need to select the one of the following option
  • Implement MiniBatchable + PartitionableByIndex (see here)
  • Implement Partitionable. This is what is documented here.
Unfortunately, you implemented MiniBatchable + Partitionable and this is not the correct combination.
Usually, the recommendation is just to stick to datastores that we ship (e.g. fileDatastore), and to use the transform function to modify it appropriately.
In your case, it seems that the choice of a custom datastore is justified because the data seems to have a specific structure and shuffle and partition behave in a specific way.
To support parallel training, you could
  • Implement partitionableByIndex it it is not too much effort. It sort of seems that because of the structure of the data, it might not be possible to index it directly.
  • Otherwise, remove the MiniBatchable interface from your datastore, and modify read to not return a table but just a row of data, like so
function [data,info] = read(ds)
info = struct;
data = {read(ds.Datastore), ds.Labels(ds.CurrentFileIndex)};
ds.CurrentFileIndex = ds.CurrentFileIndex + 1;
end
This will put your datastore in the case Partitionable only and should support the multi-gpu option.

4 个评论
显示 2更早的评论 隐藏 2更早的评论

Yong
Yong 2022-3-14
Yoann: Thank you so much for your help! I will follow your instruction to update my codes.
Yong
Yong 2022-3-15
Hi Yoann,
Thank you again for your help. I read the document of the trainNetwork function (I use it to train a CNN), it seems that in addition to the Matlab built-in datastores, it only allows a custom mini-batchable datastore, Partitional datastore is NOT allowed. If I understand your instruction correctly, the only option I have then will be to implement MiniBatchable + PartitionableByIndex. However, per Matlab document, ParitionableByIndex is Not recommended starting in R2019a. Any thoughts?
Thanks a lot!
Yong
Yoann Roth
Yoann Roth 2022-3-15
Hi Yong,
Your custom datastore does not have to be "MiniBatchable" to be supported.
Having a custom datastore that is only "Partitionable" will work, and support Parallel training.
Yoann
Yong
Yong 2022-3-16
Hi Yoann,
Thank you very much for your response. Yes, you are correct that Paritionable datastore will work with trainNet.
I have another question: because the read function in Partitionable reads only one file at a time, it seems that when multiple-gpus are used in training, the datastore might be partitioned into segments that don't follow my specific requirement, i.e. the data files are read in a gruop of at least 4 files, and the files are in the order of x, x_rotated_90, x_roated_180 and x_rotated_270. I arrange the files in that order in the datastore. Any suggestion to work around?
Many thanks,
Yong

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

产品

版本

R2022a

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by