how to Train Network on Image and Feature Data with more then one feature input?

Question

Andrew Brindle 2022-7-19

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1763920-how-to-train-network-on-image-and-feature-data-with-more-then-one-feature-input

评论： Kenneth 2023-10-17

采纳的回答： Ben

In this example: openExample('nnet/TrainNetworkOnImageAndFeatureDataExample')

I want to change numFeatures fro 1 to 3. I have added a 3 element vector to X2Train

>> preview(dsTrain)

ans =

1×3 cell array

{28×28 double} {[-42 0.9891 0.5122]} {[3]}

layers = [

imageInputLayer(imageInputSize,'Normalization','none','Name','images')

convolution2dLayer(filterSize,numFilters,'Name','conv')

reluLayer('Name','relu')

fullyConnectedLayer(50,'Name','fc1')

concatenationLayer(1,3,'Name','concat')

fullyConnectedLayer(numClasses,'Name','fc2')

softmaxLayer('Name','softmax')];

lgraph = layerGraph(layers);

featInput = featureInputLayer(numFeatures,Name="features");

lgraph = addLayers(lgraph,featInput);

lgraph = connectLayers(lgraph,"features","cat/in2");

lgraph = connectLayers(lgraph,"features","cat/in3");

figure

plot(lgraph)

when I run it I keep getting this error:

Error using trainNetwork

Input datastore returned more than one observation per row for network input 2.

Any help would be appreciated!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Ben 2022-7-20

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1763920-how-to-train-network-on-image-and-feature-data-with-more-then-one-feature-input#answer_1011780

The subtle issue here is that the feature data needs to read out of the datastore as a NumFeatures x 1 vector as documented here: https://www.mathworks.com/help/deeplearning/ug/datastores-for-deep-learning.html

So you'll need to transpose your feature data either before it goes into the datastore, or as a transform of your existing datastore (e.g. transformedDsTrain = transform(dsTrain,@(x) [x(1),{x{2}.'},x(3)]);).

However you'll next run into another subtle issue at the concatenationLayer since the output of layer 'fc2' will have size 1(S) x 1(S) x 50(C) x BatchSize(B). This needs squeezing so it can be concatenated with the feature data in shape 3(C) x BatchSize(B). Probably the easiest way to do that is with a functionLayer. Here's some code to get your network running:

imageInputSize = [28,28,1];

filterSize = 3;

numFilters = 8;

numClasses = 10;

numFeatures = 3;

layers = [

imageInputLayer(imageInputSize,'Normalization','none','Name','images')

convolution2dLayer(filterSize,numFilters,'Name','conv')

reluLayer('Name','relu')

fullyConnectedLayer(50,'Name','fc1')

squeezeLayer()

concatenationLayer(1,3,'Name','cat')

fullyConnectedLayer(numClasses,'Name','fc2')

softmaxLayer('Name','softmax')

classificationLayer];

lgraph = layerGraph(layers);

featInput = featureInputLayer(numFeatures,Name="features");

lgraph = addLayers(lgraph,featInput);

lgraph = connectLayers(lgraph,"features","cat/in2");

lgraph = connectLayers(lgraph,"features","cat/in3");

numObservations = 100;

fakeImages = randn([imageInputSize,numObservations]);

imagesDS = arrayDatastore(fakeImages,IterationDimension=4);

fakeFeatures = randn([numObservations,numFeatures]);

featureDS = arrayDatastore(fakeFeatures.',IterationDimension=2);

fakeTargets = categorical(mod(1:numObservations,numClasses));

targetDS = arrayDatastore(fakeTargets,IterationDimension=2);

ds = combine(imagesDS,featureDS,targetDS);

opts = trainingOptions("adam","MaxEpochs",1);

trainNetwork(ds,lgraph,opts);

function layer = squeezeLayer(args)

arguments

args.Name='';

end

layer = functionLayer(@squeezeLayerFcn,"Name",args.Name,"Formattable",true);

end

function x = squeezeLayerFcn(x)

x = squeeze(x);

% Since squeeze will squeeze out some dimensions, we need to relabel x.

% Assumption: x does not have a 'T' dimension.

n = ndims(x);

newdims = [repelem('S',n-2),'CB'];

x = dlarray(x,newdims);

end

As a final note - I notice you're concatenating the feature input layer to itself, alongside the outputs of layer 'fc2'. Maybe that's intentional, it seemed slightly curious to me.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Kenneth 2023-10-13

Hi ben,

I am trying to concatenate image layer (224 x 224 x 3) with feature input layer (4 features) for a regression CNN but am getting the below error:

Error using trainNetwork

Invalid training data. The output size (1) of the last layer does not match the response size (1569).

I have 1569 observations and since I am doing a regression network I have put the last fully connetced layer to be 1 output but it is returning the above error.

Data store prep:

SimTrainDS = arrayDatastore(SimTrain,IterationDimension=4);

aw_SimTrainDS = arrayDatastore(awSimTrain.',IterationDimension=2);

betaTrainDS = arrayDatastore(betaTrain,IterationDimension=2);

All_TrainDs = combine(SimTrainDS,aw_SimTrainDS,betaTrainDS);

layers = [

imageInputLayer([224 224 3],Normalization="none") % coloured image size, height, width, and number of channels respectively

convolution2dLayer(3,8,'Padding','same')

batchNormalizationLayer

reluLayer

averagePooling2dLayer(2,'Stride',2)

convolution2dLayer(3,16,'Padding','same')

batchNormalizationLayer

reluLayer

averagePooling2dLayer(2,'Stride',2)

convolution2dLayer(3,32,'Padding','same')

batchNormalizationLayer

reluLayer

convolution2dLayer(3,32,'Padding','same')

batchNormalizationLayer

reluLayer

dropoutLayer(0.2)

reluLayer

fullyConnectedLayer(50)

flattenLayer

concatenationLayer(1,4,Name="concat")

fullyConnectedLayer(1)

regressionLayer];

%Convert the layers to a layer graph.

lgraph = layerGraph(layers);

% Add a feature input layer to the layer graph and connect it

% to the concatenation layer.

numFeatures=4;

featInput = featureInputLayer(numFeatures,Name="Param_features");

lgraph = addLayers(lgraph,featInput);

lgraph = connectLayers(lgraph,"Param_features","concat/in2");

lgraph = connectLayers(lgraph,"Param_features","concat/in3");

lgraph = connectLayers(lgraph,"Param_features","concat/in4");

%Visualize the network.

figure

set(gcf, 'Color', 'w');

plot(lgraph)

miniBatchSize = 128;

options = trainingOptions('sgdm', ...

'MiniBatchSize',miniBatchSize, ...

'MaxEpochs',30, ...

'InitialLearnRate',1e-3, ...

'LearnRateSchedule','piecewise', ...

'LearnRateDropFactor',0.1, ...

'LearnRateDropPeriod',20, ...

'Shuffle','every-epoch', ...

'Plots','training-progress', ...

'Verbose',false);

Reg_CNNnetSim = trainNetwork(All_TrainDs,lgraph,options);

Ben 2023-10-13

The error is suggestive that the issue is with the datastore setup - trainNetwork thinks that your responses/targets have size 1569, but that's actually the batch/observation dimension.

You can find documentation on datastore inputs for trainNetwork here: https://www.mathworks.com/help/deeplearning/ug/datastores-for-deep-learning.html

If you could call:

data = read(All_TrainDs)

and post information about data, we might be able to debug - in particular we want to check the size of each of the inputs/predictors and the output/response in data.

Kenneth 2023-10-17