Classify Images in Simulink with Imported TensorFlow Network

Load Image Data

Load the digit sample data as an image datastore. The imageDatastore function automatically labels the images based on the folder names, and stores the data as an ImageDatastore object.

digitDatasetPath = fullfile(matlabroot,"toolbox","nnet","nndemos", ...
    "nndatasets","DigitDataset");
imds = imageDatastore(digitDatasetPath, ...
    IncludeSubfolders=true,LabelSource="foldernames");

For reproducibility, specify the seed for the MATLAB random number generator. Randomly select eight images from the image datastore, create the array of images displayIms, and display the selected images by using the function montage (Image Processing Toolbox).

rng("default")
perm = randperm(10000,8);
for i = 1:8
    displayIms(:,:,:,i) = imread(imds.Files{perm(i)});
end
montage(displayIms,size=[1 NaN]);

Convert the images in displayIms to single data type so they can be used as inputs to a dlnetwork object.

inputIms = single(displayIms);

Import Pretrained TensorFlow Network

Specify the model folder that contains the pretrained network digitsDAGnetwithnoise in the saved model format. digitsDAGnetwithnoise can classify images of digits.

if ~exist("digitsDAGnetwithnoise","dir")
    unzip("digitsDAGnetwithnoise.zip")
end
modelFolder = "./digitsDAGnetwithnoise";

Specify the class names.

classNames = {'0','1','2','3','4','5','6','7','8','9'};

Import a TensorFlow network in the saved model format. By default, importNetworkTensorFlow imports the network as a dlnetwork object.

net = importNetworkFromTensorFlow(modelFolder);

Importing the saved model...
Translating the model, this may take a few minutes...
Finished translation. Assembling network...
Import finished.

Analyze the imported network. The analyzeNetwork function displays an interactive plot of the network architecture and a table containing information about the network layers.

analyzeNetwork(net)

The imported network contains layers that are not supported for conversion to built-in MATLAB layers. importNetworkFromTensorFlow automatically generates the custom layers gaussian_noise_1 and gaussian_noise_2. The function saves each generated custom layer to a separate .m file in the package +digitsDAGnetwithnoise in the current folder. For more information about these generated custom layers, see Import TensorFlow Network with Autogenerated Custom Layers.

Save the imported network in a MAT-file.

filename = "digitsNet.mat";
save(filename,"net")

Open or Create Simulink Model

This example provides the Simulink model slexDigitsImportedNetworkPredictExample.slx. You can open the Simulink model (provided in this example) or create a new model by following the steps described in this section.

Open Provided Model

Open the Simulink model slexDigitsImportedNetworkPredictExample.slx.

SimMdlName = "slexDigitsImportedNetworkPredictExample"; 
open_system(SimMdlName)

Create New Model

1. To create a new Simulink model, open the Blank Model template and add the Predict block from the Deep Learning Toolbox™ library. The Predict block predicts responses for the input data by using the trained network that you specify in the Block Parameters dialog box. Typically, the input to the block is an h-by-w-by-c-by-N numeric array, where h, w, and c are the height, width, and number of channels of the images, respectively, and N is the number of images.

Double-click the Predict block to open the Block Parameters dialog box. Select Network from MAT-file for the Network parameter. Click Browse in the File path section and specify the network as the digitsNet.mat network in the current folder. Click OK.

2. Insert the Video From Workspace block from the Computer Vision Toolbox™ library. Double-click the Video From Workspace block to open the Block Parameters dialog box. Specify Signal as inputIms, Sample time as 1, and Form output after final value by as Holding final value. Click OK.

3. Check if the size of the input images matches the network input size. If the sizes do not match, you can resize the input data by adding the Resize block from the Computer Vision Toolbox library to the model.

Display the size of the images and the input size of the network.

size(inputIms)

ans = 1×4

    28    28     1     8

netInputSize = net.Layers(1).InputSize

netInputSize = 1×3

    28    28     1

The input is a sequence of eight grayscale (one-channel) images that are 28-by-28 pixels in size. This size matches the network input size.

4. Add a To Workspace block to the model and change the variable name to yPred. Connect the Video From Workspace block to the input of the Predict block, and connect the To Workspace block to the output of the Predict block.

5. Open the Configuration Parameters dialog box. On the Modeling tab, click Model Settings. Under Solver selection, set Type to Fixed-step, and set Solver to discrete (no continuous states).

Predict Using Simulink Model

Simulate the model and save the simulation output to modelOutput. The field modelOutput.yPred.Data contains the classification results.

modelOutput = sim(SimMdlName)

modelOutput = 
  Simulink.SimulationOutput:

                   tout: [8x1 double] 
                  yPred: [1x1 timeseries] 

     SimulationMetadata: [1x1 Simulink.SimulationMetadata] 
           ErrorMessage: [0x0 char] 

Display the sequence of images and the classification results.

tiledlayout(1,12,TileSpacing="None");
for i = 1:size(inputIms,4)
   nexttile
   imshow(displayIms(:,:,:,i))
   label = modelOutput.yPred.Data(:,:,i)==1;
   title([classNames{label}],FontSize=20)
end