"splitEachLabel" built-in function does not really randomize the picture distribution?

Question

cui,xingxing 2018-2-26

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/384794-spliteachlabel-built-in-function-does-not-really-randomize-the-picture-distribution

评论： cui,xingxing 2023-10-24

When I use R2017b to do deep learning classification, the imageDatasotre object is divided into training and test set,whether or not to specify the number or proportion, 'splitEachLabel' optional parameters specified as 'randomized', the training set inside the picture is not randomly arranged, and why?

as the document said: https://cn.mathworks.com/help/nnet/examples/create-simple-deep-learning-network-for-classification.html

digitDatasetPath = fullfile(matlabroot,'toolbox','nnet','nndemos', ...
        'nndatasets','DigitDataset');
digitData = imageDatastore(digitDatasetPath, ...
        'IncludeSubfolders',true,'LabelSource','foldernames');
trainingNumFiles = 750;
rng(1) % For reproducibility
[trainDigitData,testDigitData] = splitEachLabel(digitData, ...
        trainingNumFiles,'randomize');

When you open "trainDigitData.Files" and "trainDigitData.Labels" in a workspace, they do not disrupt the order?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Wentao Du 2018-3-1

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/384794-spliteachlabel-built-in-function-does-not-really-randomize-the-picture-distribution#answer_307744

在 MATLAB Online 中打开

Here the order you see will not be completely different because the labels of "digitData" are in order (from 0 to 9). To observe the effect of "randomize" parameter, you can run

[trainDigitData,valDigitData] = splitEachLabel(digitData,trainNumFiles,'randomize');

multiple times and will find the distribution of actual image files keeps changing.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

cui,xingxing 2018-3-1

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/384794-spliteachlabel-built-in-function-does-not-really-randomize-the-picture-distribution#answer_307763

在 MATLAB Online 中打开

thanks a lot! I found the solution to the problem, if you want to disrupt the label, you can use the shuffle function. Example:

imds_new = shuffle(imds)

2 个评论
显示无隐藏无

debojit sharma 2023-7-8

Since,it may be risky to do a standard random train/test split when having strong class imbalance.Because very small number of positive cases, we might end up with a train and test set that have very different class distributions. We may even end up with close to zero positive cases in our test set. So, is there anyfunction to do stratified sampling during train/test split that avoids disturbing class balance in our samples in MatLab @cui @Wentao Du . Like the following code in python:

from sklearn.model_selection import train_test_split

train, test = train_test_split(data, test_size = 0.3, stratify=data.buy)

cui,xingxing 2023-10-24

@debojit sharma ，You can design your own distribution with sample weighting in MATLAB，such as the MATLAB built-in function randsample, and the following example can inspire you.

https://ww2.mathworks.cn/matlabcentral/answers/812345-how-to-make-weighted-random-sampling-for-matrix#answer_690045

Reference:

https://scikit-learn.org/stable/modules/cross_validation.html#stratification

请先登录，再进行评论。