I want a small network that contains only a batch normalization layer followed by fully-connected layer, softmax and finally cross entropy loss. To do that, my model is implemented as follows:
function [dlY1,state] = model(parameters,dlX,doTraining,state)
offset = parameters.batchnorm1.Offset;
scale = parameters.batchnorm1.Scale;
trainedMean = state.batchnorm1.TrainedMean;
trainedVariance = state.batchnorm1.TrainedVariance;
[dlY,trainedMean,trainedVariance] = batchnorm(dlX,offset,scale,trainedMean,trainedVariance);
state.batchnorm1.TrainedMean = trainedMean;
state.batchnorm1.TrainedVariance = trainedVariance;
dlY = batchnorm(dlX,offset,scale,trainedMean,trainedVariance);
weights = parameters.fc1.Weights;
bias = parameters.fc1.Bias;
dlY1 = fullyconnect(dlY,weights,bias);
TO calculate the gradients, I have this function here:
function [gradients,state,loss] = modelGradients(parameters,dlX,T1,state)
[dlY1,state] = model(parameters,dlX,doTraining,state);
lossLabels = crossentropy(dlY1,T1);
gradients = dlgradient(loss,parameters,'EnableHigherDerivatives',true);
TO start the program with simple example to calculate the derivatives:
dlX=dlarray(rand(10,5),'BC');
T1=[1;2]; T1=repelem(T1,5);
dlY=onehotencode(categorical(T1'),1);
parameters.batchnorm1.Offset = zeros([5 1]);
parameters.batchnorm1.Scale = ones([5 1]);
state.batchnorm1.TrainedMean = zeros(5,1,'single');
state.batchnorm1.TrainedVariance = ones(5,1,'single');
parameters.fc1.Weights=initializeGlorot(sz,numOut,numIn);
parameters.fc1.Bias=zeros(2,1);
[gradients,state,loss] = dlfeval(@modelGradients, parameters, dlX, dlY, state);
So, the gradients are the thing I am looking for and the output of this process is this following error message:
Error using dlfeval (line 43)
'dlgradient' inputs must be traced dlarray objects or cell arrays,
structures or tables containing traced dlarray objects. To enable tracing,
I tried to use dlfeval instead of dlgradient but another error message appears:
Error using dlfeval (line 43)
Nested dlfeval calls are not supported. To compute higher derivatives, set
the 'EnableHigherDerivatives' option of the dlgradient function to true.
Does anyone got any idea why I recieve these error messages? THanks.
