Transform and Combine fileDatastores to Train CNN

Question

Mauricio Piraquive 2021-2-19

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/750444-transform-and-combine-filedatastores-to-train-cnn

评论： Mauricio Piraquive 2021-3-26

Hi I would appreciatte your help,

I have a collection of 50x1x12 mat files, that I need to upload into some datastore to subsequently pass into a convolutional neural network, how can I Train my

I am trying to follow this example https://in.mathworks.com/matlabcentral/answers/459161-image-regression-using-mat-files-and-a-datastore#answer_405285

but instead of having .mat file for the labels I am using this function to get the labels from the folders.

function label = readLabel(filename,classNames)
filepath = fileparts(filename);
[~,label] = fileparts(filepath);
label = categorical(string(label),classNames);
end

I am uploading the files into the fileDatastore

inputData=fileDatastore(fullfile('inputData'),'ReadFcn',@load,'FileExtensions','.mat');
%getting the labels from the folders
classNames = string(1:2)
targetData=fileDatastore(fullfile('targetData'),'ReadFcn',@load,'FileExtensions','.mat',readLabel(filename,classNames));

Should I transform before combine the filedatastores? if so, how should my transform function be?

I tried combining without transforming and I got this error when I tried to train my CNN: (trainNetwork)

%combininig
cdsTrain = combine(inputData,targetData);

%Training
net = trainNetwork(cdsTrain,layers,options);

"The training images are of size 1x1x1 but the input layer expects images of size 50x1x12"

Thanks for your help!

@Madhav Thakker

luisa di monaco

2 个评论
显示无隐藏无

luisa di monaco 2021-2-20

Hi, maybe I can help you if you can give some additional information.

What size your input images are?

What size is your target data?

What is 50? What is 1? What is 12? Does any .mat file include an input image and its label?

Mauricio Piraquive 2021-2-21

编辑：Mauricio Piraquive 2021-2-22

Hi @luisa di monaco,

I am working with a "Image" like file when

50 would be the heigth of the "Image"

1 would be the width

12 would be the number of channels. (like 3 for rgb but in this case I have 12 channels).

These file correspond to the reading of a "material" using a lab instrument,

for example, one file could correspond to the results of the reading of material type-A

another file correspond to the results of the reading of material type-B

So I want to create this convolutional neural network, pass a reading and the network should identify the materia and tell me what type of Material that data correspond.

Right now I dont have any target data, I am getting the labels from the folders that contains each file. However, if needed I can create this output data files,

What Size this label files should be?

My current .mat files dont have the label in it.

Really appreciatte your help.

Regards,

Mauricio

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

luisa di monaco 2021-2-22

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/750444-transform-and-combine-filedatastores-to-train-cnn#answer_630524

编辑：luisa di monaco 2021-2-22

在 MATLAB Online 中打开

Hi, this is my code. I know it is not efficient, but it should work! I hope it can help as a starting point to be optimized.

For simplicity, I added information about category inside each .mat file as string variable. However, I'm sure you can get labels from folder names too.

The key idea is to use fileDatastore for input data and tabularTextDatastore for categorical data. Then you can combine them succesfully.

Something similar was done here: https://it.mathworks.com/matlabcentral/answers/586949-multi-input-imagedatastore .

Regards

clear
close all
clc
%% create dummy inputs and targets (.mat file)
% inputs are images of size 50x1x12 [heigth, width, channels]
% target are type of material: categorical array
mkdir trainingData
cd trainingData
n=10; % number of training cases
for i=1:n
    material=ones(50,1,12)*i;
    
    if i<n/2
        matType='typeA';
    else
        matType='typeB';
    end
    mkdir(matType)
    filename=sprintf('material_%d', i);
    save(fullfile(matType,filename),'material', 'matType')
end
cd ..
%% load data
allData=fileDatastore(fullfile('trainingData'),'ReadFcn',@load,'FileExtensions','.mat', 'IncludeSubfolders', true);
%% create inputs
inputData = transform(allData,@(data) rearrange_input(data)); % extract and rearrange input
%% create targets (can be optimized...)
targetData = transform(allData,@(data) rearrange_target(data));
myLabels=targetData.readall;
writematrix(myLabels,'myLabels.txt');
labelStore = tabularTextDatastore('myLabels.txt','TextscanFormats','%C',"ReadVariableNames",false);
read_size=1; % this line is foundamental... I don't know why...
labelStore.ReadSize = read_size;
labelStoreCell = transform(labelStore,@setcat_and_table_to_cell);
%% combininig
cdsTrain = combine(inputData,labelStoreCell);
%% training
% dummy layers and options
numClasses=2;
layers=[
    imageInputLayer([50 1 12],"Name","imageinput")
    batchNormalizationLayer()
    leakyReluLayer(0.1)
    fullyConnectedLayer(numClasses,'Name','fc')
    softmaxLayer('Name','soft')
    classificationLayer('Name','classification')];
    
options = trainingOptions('adam');
net = trainNetwork(cdsTrain,layers,options);
%% functions
function inputData = rearrange_input(data)
inputData=data.material;
inputData= {inputData};
end
function targetData = rearrange_target(data)
targetData=data.matType;
targetData=categorical(cellstr(targetData));
end
function [dataout] = setcat_and_table_to_cell(datain)
validcats = ["typeA", "typeB"];
datain.(1) = setcats(datain.(1),validcats);
dataout = table2cell(datain);
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Mauricio Piraquive 2021-3-26

@luisa di monaco Thank you very much this was very helpful. Really appreciate it.

请先登录，再进行评论。

Answer 2

Mahesh Taparia 2021-2-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/750444-transform-and-combine-filedatastores-to-train-cnn#answer_631899

在 MATLAB Online 中打开

Hi

One possible approach is already suggested by Luisa. In your approach, replace your read function with the following line of code

function label = readLabel(filename)
filepath = fileparts(filename);
[~,label] = fileparts(filepath);
label = categorical(string(label));
end

Also, the fileDatastore of label data is not created properly, replace it with following line of code

targetData=fileDatastore(fullfile('targetData'),'ReadFcn',@readLabel,'FileExtensions','.mat');

Try it, it may work. If it won't work, apply break points in readLabel function and try to create the function readLabel such that it should returns the categorical label. Hope it will help!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Transform and Combine fileDatastores to Train CNN

2 个评论
显示无隐藏无

采纳的回答

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

更多回答（1 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

Community Treasure Hunt

Transform and Combine fileDatastores to Train CNN

2 个评论 显示 无隐藏 无

采纳的回答

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

更多回答（1 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

Community Treasure Hunt

2 个评论
显示无隐藏无

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论