Specify Custom Weight Initialization Function

Load Data

Load the digit sample data as an image datastore. The imageDatastore function automatically labels the images based on folder names.

unzip("DigitsData.zip");
imds = imageDatastore("DigitsData", ...
    IncludeSubfolders=true, ...
    LabelSource="foldernames");

Divide the data into training and validation data sets, so that each category in the training set contains 750 images, and the validation set contains the remaining images from each label. splitEachLabel splits the datastore into two new datastores for training and validation.

numTrainFiles = 750;
[imdsTrain,imdsValidation] = splitEachLabel(imds,numTrainFiles,"randomize");

Create Custom Weight Initialization Function

The leakyHe function takes the input sz (the size of the layer weights) and returns an array of weights given by the He Initializer for convolution layers followed by a leaky ReLU layer. The function also accepts the optional input argument scale which specifies the scale multiplier for the leaky ReLU layer.

function weights = leakyHe(sz,scale)

% If not specified, then use default scale = 0.1
if nargin < 2
    scale = 0.1;
end

filterSize = [sz(1) sz(2)];
numChannels = sz(3);
numIn = filterSize(1) * filterSize(2) * numChannels;

varWeights = 2 / ((1 + scale^2) * numIn);
weights = randn(sz) * sqrt(varWeights);

end

Define Network Architecture

Define the convolutional neural network architecture:

For each of the convolutional layers, set the weights initializer to the leakyHe function.

inputSize = [28 28 1];
numClasses = 10;

layers = [
    imageInputLayer(inputSize)
    convolution2dLayer(3,8,WeightsInitializer=@leakyHe)
    leakyReluLayer
    convolution2dLayer(3,16,WeightsInitializer=@leakyHe)
    leakyReluLayer
    convolution2dLayer(3,32,WeightsInitializer=@leakyHe)
    leakyReluLayer
    fullyConnectedLayer(numClasses)
    softmaxLayer];

Train Network

Specify the training options and train the network. Train for four epochs. To prevent the gradients from exploding, set the gradient threshold to 2. Validate the network once per epoch. Display the training progress in a plot and monitor the accuracy.

Train the neural network using the trainnet function. For classification, use cross-entropy loss. By default, the trainnet function uses a GPU if one is available. Training on a GPU requires a Parallel Computing Toolbox™ license and a supported GPU device. For information on supported devices, see GPU Computing Requirements (Parallel Computing Toolbox). Otherwise, the trainnet function uses the CPU. To specify the execution environment, use the ExecutionEnvironment training option.

maxEpochs = 4;
miniBatchSize = 128;
numObservations = numel(imdsTrain.Files);
numIterationsPerEpoch = floor(numObservations / miniBatchSize);

options = trainingOptions("sgdm", ...
    MaxEpochs=maxEpochs, ...
    MiniBatchSize=miniBatchSize, ...
    GradientThreshold=2, ...
    ValidationData=imdsValidation, ...
    ValidationFrequency=numIterationsPerEpoch, ...
    Verbose=false, ...
    Metrics="accuracy", ...
    Plots="training-progress");

[netDefault,infoDefault] = trainnet(imdsTrain,layers,"crossentropy",options);

Test Network

Test the neural network using the testnet function. For single-label classification, evaluate the accuracy. The accuracy is the percentage of correct predictions. By default, the testnet function uses a GPU if one is available. To select the execution environment manually, use the ExecutionEnvironment argument of the testnet function.

accuracy = testnet(netDefault,imdsValidation,"accuracy");

Specify Additional Options

The leakyHe function accepts the optional input argument scale. To input extra variables into the custom weight initialization function, specify the function as an anonymous function that accepts a single input sz. To do this, replace instances of @leakyHe with @(sz)leakyHe(sz,scale). Here, the anonymous function accepts the single input argument sz only and calls the leakyHe function with the specified scale input argument.

Create and train the same network as before with the following changes:

scale = 0.01;

layers = [
    imageInputLayer(inputSize)
    convolution2dLayer(3,8,WeightsInitializer=@(sz)leakyHe(sz,scale))
    leakyReluLayer(scale)
    convolution2dLayer(3,16,WeightsInitializer=@(sz)leakyHe(sz,scale))
    leakyReluLayer(scale)
    convolution2dLayer(3,32,WeightsInitializer=@(sz)leakyHe(sz,scale))
    leakyReluLayer(scale)
    fullyConnectedLayer(numClasses)
    softmaxLayer];

[netCustom,infoCustom] = trainnet(imdsTrain,layers,"crossentropy",options);

Test the network using the testnet function.

accuracy = testnet(netCustom,imdsValidation,"accuracy");

Compare Results

Extract the validation accuracy from the information output from the trainnet function.

validationAccuracy = [
    infoDefault.ValidationHistory.Accuracy, ...
    infoCustom.ValidationHistory.Accuracy];

For each of the networks, plot the epoch numbers against the validation accuracy.

figure
epochs = 0:maxEpochs;
plot(epochs,validationAccuracy)
title("Validation Accuracy")
xlabel("Epoch")
ylabel("Validation Accuracy")
legend(["Leaky He (Default)" "Leaky He (Custom)"],'Location','southeast')

Bibliography