CNN for regression with imageDatastore

Question

Daniele Minotti 2022-12-15

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1878797-cnn-for-regression-with-imagedatastore

评论： Antoni Woss 2023-1-26

Hi, I am facing a CNN regression problem. I have a datastore with 41000 images and the images are 5x16000x1. The task is similar to the matlab example "Train Convolutional Neural Network for Regression" but, instead of angle of rotation, each image as a specific distance associated (for example I have 7000 images with the distance associated equal to 5mm; 7000 images with the distance associated equal to 3mm; 5000 images with the distance associated equal to 7mm; 5000 images with the distance associated equal to 9mm; ecc ecc). I have followed the example indicated, so I created a 4d array of dimension 5x16000x1x41000 and a 41000x1 vector. But the dimensions are very large, so it's not feasible this way. I would like to know how to use in the correct way the imageDatastore for my problem, since when I try to use it this message appear to me: "imageDatastore is not supported for regression tasks".

Thank you for your help in advance!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Ola Ola 2022-12-26

编辑：Ola Ola 2022-12-26

@Daniele Minotti I have a similar problem although with smaller amount of data. Please how did you create 4d array for your images?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Antoni Woss 2022-12-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1878797-cnn-for-regression-with-imagedatastore#answer_1129047

在 MATLAB Online 中打开

You can create a combined datastore to hold the images and scalar distances and pass this into the training routine for the CNN regression task. As an explicit example, run the CNN regression task, "Train Convolutional Neural Network for Regression", from the documentation example - https://uk.mathworks.com/help/deeplearning/ug/train-a-convolutional-neural-network-for-regression.html.

Then save each digit image and corresponding angle of rotation to a "TrainingData" and "ResponseData" directory respectively. In your case, it appears as though you already have the images saved to disk in some format and so these steps can be modified for your use case. This snippet saves the digit images and angles of rotation as MAT files:

for observation = 1:size(XTrain,4)
    X = XTrain(:,:,:,observation);
    save(['TrainingData/X_' num2str(observation) '.mat'],'X');
    X = YTrain(observation);
    save(['ResponseData/Y_' num2str(observation) '.mat'],'X');
end

Next create imageDatastores, loading in the images and responses, and then combine these datastores.

imds = imageDatastore('TrainingData','FileExtensions','.mat','ReadFcn',@matImgRead);
rds = imageDatastore('ResponseData','FileExtensions','.mat','ReadFcn',@matImgRead);
ds = combine(imds,rds);
function data = matImgRead(filename)
img = load(filename);
data = img.X;
end

You then can train the regression CNN using this combined datastore.

net = trainNetwork(ds,layers,options);

The documentation for the imageDatastores give more detail where data is saved in different formats, for example if your images are ".jpg" - https://uk.mathworks.com/help/matlab/ref/matlab.io.datastore.imagedatastore.html

5 个评论
显示 3更早的评论隐藏 3更早的评论

Antoni Woss 2022-12-16

在 MATLAB Online 中打开

Validation and test data can be turned into datastores in the same way. If instead you want to split your original data into training and validation for example, with 80% training and 20% validation, you could create a training datastore and validation datastore in the following way (assuming you have run the previous code snippet and "Train Convolutional Neural Network for Regression" example:

perm = randperm(size(XTrain,4));
trainValidateSplit = 0.8;
imdsTrain = imageDatastore(imds.Files(perm(1:trainValidateSplit*end)),'FileExtensions','.mat','ReadFcn',@matImgRead);
imdsValidate = imageDatastore(imds.Files(perm(trainValidateSplit*end+1:end)),'FileExtensions','.mat','ReadFcn',@matImgRead);
rdsTrain = imageDatastore(rds.Files(perm(1:trainValidateSplit*end)),'FileExtensions','.mat','ReadFcn',@matImgRead);
rdsValidate = imageDatastore(rds.Files(perm(trainValidateSplit*end+1:end)),'FileExtensions','.mat','ReadFcn',@matImgRead);
dsTrain = combine(imdsTrain,rdsTrain);
dsValidate = combine(imdsValidate,rdsValidate);

This creates a random permutation of the data and allocates 80% to training (and response) datastores, the remaining 20% to validation (and response) datastores. You can split three ways to add test data by extending this idea.

You can update the training options to reference this validation datastore, dsValidate, and then train on the dsTrain datastore.

options = trainingOptions('sgdm', ...
    'MiniBatchSize',miniBatchSize, ...
    'MaxEpochs',30, ...
    'InitialLearnRate',1e-3, ...
    'LearnRateSchedule','piecewise', ...
    'LearnRateDropFactor',0.1, ...
    'LearnRateDropPeriod',20, ...
    'Shuffle','every-epoch', ...
    'ValidationData',dsValidate, ...
    'ValidationFrequency',validationFrequency, ...
    'Plots','training-progress', ...
    'Verbose',false);
net = trainNetwork(dsTrain,layers,options);

Ola Ola 2022-12-26

编辑：Ola Ola 2022-12-26

在 MATLAB Online 中打开

@Antoni Woss I am new to MATLAB. I have tried to follow through this snippet you gave but I am have error "Cannot create 'X_1.mat' because 'TrainingData' does not exist.". My images are in a folder (F:\Regression\Images), please I am wondering in the snippet, how do I create XTrain and make MATLAB TrainingData?

for observation = 1:size(XTrain,4)
    X = XTrain(:,:,:,observation);
    save(['TrainingData/X_' num2str(observation) '.mat'],'X');
    X = YTrain(observation);
    save(['ResponseData/Y_' num2str(observation) '.mat'],'X');
end

Antoni Woss 2023-1-26

在 MATLAB Online 中打开

In order to use this workflow with your existing images, you would need to modify the datastore and read function accordingly:

imds = imageDatastore('TrainingData','FileExtensions','.mat','ReadFcn',@matImgRead);
rds = imageDatastore('ResponseData','FileExtensions','.mat','ReadFcn',@matImgRead);
ds = combine(imds,rds);
function data = matImgRead(filename)
img = load(filename);
data = img.X;
end

For example, if your training data is in "F:\Regression\Images", then you need to point to that location. See for example: https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.imagedatastore.html.

You would also need to adjust the matImgRead function to read the correct filenames and point to the data within the file. In the example above, the files were named "X_1.mat" for example (iterating the number for each image). Within the MAT file, the data was saved in the "X" variable.

请先登录，再进行评论。

CNN for regression with imageDatastore

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

采纳的回答

5 个评论
显示 3更早的评论隐藏 3更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

CNN for regression with imageDatastore

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

采纳的回答

5 个评论 显示 3更早的评论隐藏 3更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

5 个评论
显示 3更早的评论隐藏 3更早的评论