Convert MATLAB Deep Learning Networks to eAI Model Using Qualcomm LPAI SDK

This example uses:

This example shows you how to convert Deep Learning networks developed using MATLAB® built-in layers into eAI models for deployment to eNPU using Qualcomm® LPAI SDK. Deploying to an eNPU (LPAI) requires the Deep Learning network to be in the .eai file format (eAI model), generated using eai_builder tool from the LPAI SDK. This SDK provides a workflow for converting a ONNX/TFLite models to this format. The workflow explained in this example helps you to streamline the process for conversion of MATLAB-based Deep Learning networks to the .eai format.

The procedure for converting a MATLAB Deep Learning network to an eAI model includes these steps:

Download dataset and import pre-trained model: Begin by importing the dataset and the pre-trained floating-point Deep Learning network. Split the dataset into training and validation sets.
Generate quantization information and export to ONNX: Use the training set to determine the quantization ranges for each network layer (used later by the eai_builder tool) and to export the network to the ONNX format.
Convert ONNX network to eAI-model: Provide the ONNX file and the quantization information in the form of a JSON file to the eai_builder to generate an eAI model.
Create confusion matrix chart for validation (optional): Optionally, create a confusion matrix chart to validate the model's performance.

This example uses a Deep Learning network that detects speech commands in the audio. Refer to Train Deep Learning Network for Speech Command Recognition (Audio Toolbox) example for model creation and training, and Speech Command Recognition Using Deep Learning (Audio Toolbox) example for using a pretrained network

Note: This example assumes that you have installed Hexagon SDK v5.5.0.1 and LPAI SDK v2.4.0.0. To use a different SDK version, update the fields in the Convert ONNX network to eAI-model section of this example. Additionally, this example supports converting Deep Learning network to work with eNPU version V3 or V4 only.

Download Dataset and Import Pre-Trained Model

Import the pre-trained model developed using Train Deep Learning Network for Speech Command Recognition (Audio Toolbox) example along with the dataset. In this example, we split the dataset into training and validation sets, similar to the original example. The training set is used for obtaining the quantization information, and the validation set is used for validating the resultant MATLAB network and the final eAI model.

This example uses the Google Speech Commands Dataset [1]. Download and unzip the data set.

% Set useParallel to true to parallelize the computation of data preprocessing
useParallel = true;

dataFolder = tempdir;
dataset = fullfile(dataFolder,"google_speech");

if ~isfolder(dataset)
    downloadFolder = matlab.internal.examples.downloadSupportFile("audio","google_speech.zip");
    unzip(downloadFolder,dataFolder);
end

Import the Speech Command Recognition Network

% Import the trained network 
load('SpeechCommandRecognitionNetwork.mat','trainedNet','labels');

Split the dataset into training data set (used for quantization purpose) and validation set (used for validation).

[adsTrain,adsValidation,commands] = generateTrainAndValidationDataSet(dataset);

Progress = 17 (%)
Progress = 33 (%)

The input layer of this Deep Learning network expects the extracted audio features to be formatted as an image, where the audio samples are transformed as described in the image above.

[XTrain, XValidation] = extractAudioFeatures(adsTrain,adsValidation,useParallel);

Plot the confusion matrix using the validation set for the pre-trained model by using confusionchart (Deep Learning Toolbox).

scores = minibatchpredict(trainedNet,XValidation);
YValidation = scores2label(scores,labels,"auto");
TValidation = adsValidation.Labels;
validationError = mean(YValidation ~= TValidation);
disp("Validation error: " + validationError*100 + " %");
figure(Units="normalized",Position=[0.2,0.2,0.5,0.5]);
cm = confusionchart(TValidation,YValidation, ...
    Title="Confusion Matrix for Validation Data", ...
    ColumnSummary="column-normalized",RowSummary="row-normalized");
sortClasses(cm,labels);

Generate Quantization Information and Export to ONNX

To generate quantization information, this example uses dlupdate (Deep Learning Toolbox), exportONNXNetwork (Deep Learning Toolbox), dlquantizer (Deep Learning Toolbox), and calibrate (Deep Learning Toolbox) functions.

% Convert network for single precision
net =  dlupdate(@single,trainedNet);

% Create a dlquantizer object for calibration
dlquantObj = dlquantizer(net,'ExecutionEnvironment','MATLAB');

% Use exhaustive training set for obtaining quantization ranges of layers
calibData = calibrate(dlquantObj,XTrain);

disp(calibData);
% export the MATLAB DL network as ONNX network
exportedONNXModelName = 'originalNetwork.onnx';
exportONNXNetwork(net,exportedONNXModelName,'OpsetVersion',10);

Convert ONNX Network to eAI Model

Convert the exported ONNX model (obtained from the previous section) to eAI model format. In this example, the eAI model is targeted for eNPU V3 (with eAI Runtime v2.33.0 using LPAI SDK v2.4.0.0). To convert the eAI model for eNPU V4 instead, select 'V4' for eNPUVersion.

enpuVersion = 'V3'

Setup the SDK paths.

updatedONNX = 'rectifiedNetwork.onnx';
templateJSON = 'templateQuantization.json';
dumpFile = fopen('verbose.txt','w');

Note: Update the paths in the below code according to the SDK installation paths on the host machine. This example assumes Windows® as host OS. However, the workflow remains the same for Linux®, and therefore you can modify the paths and the Executables as per Linux OS.

sdkRoot = fullfile('C:','Qualcomm','Hexagon_SDK','5.5.0.1');
if ~isfolder(sdkRoot)
    error('Invalid Hexagon SDK folder. Set the valid Hexagon SDK path. For example: ''C:\Qualcomm\Hexagon_SDK\5.5.0.1'' ');
end

if strcmp(enpuVersion,'V3')
    lpaiRoot = fullfile(sdkRoot,'addons','LPAI','eNPU3','2.33.0');
    eaiBuilder = fullfile(lpaiRoot,'eai_runtime','tools','model_builder', ...
        'eai_builder.exe');
    eaiQuantizer = fullfile(lpaiRoot,'eai_runtime','tools','tensor_quantization', ...
        'eai_quantize.exe');
else
    lpaiRoot = fullfile(sdkRoot,'addons','LPAI','eNPU4','3.18.2'); %#ok<UNRCH>
    eaiBuilder = fullfile(lpaiRoot,'eai_builder','eai_builder.exe');
    eaiQuantizer = fullfile(lpaiRoot,'eai_runtime','utilities','tensor_quantization', ...
        'eai_quantize.exe');
end

if isunix
    % Remove any .exe extensions for Linux hosts
    [p,f,~]=fileparts(eaiBuilder);
    eaiBuilder=fullfile(p,f);
    [p,f,~]=fileparts(eaiQuantizer);
    eaiQuantizer=fullfile(p,f);
end

if ~isfolder(lpaiRoot)
    error(['Invalid LPAI SDK folder. Set a valid LPAI SDK addon path according to the Hexagon SDK recommendations.' ...
        ' For example: ''C:\Qualcomm\Hexagon_SDK\5.5.0.1\addons\eNPU3\2.33.0''']);
end

onnxRectifier = fullfile(sdkRoot,'addons','LPAI','eNPU3','2.33.0','eai_runtime','tools','onnx_rectifier', ...
    'onnx_rectifier.py');

Include Python® to the path. Replace this with the appropriate environment path, if required.

if ~contains(lower(getenv('PATH')),'python')
    error('Python is not found in the environment path. Try including the path using setenv.\nExample: setenv(''PATH'', strcat(getenv(''PATH''), '';C:\Python311''))');
end

Optionally, simplify the exported ONNX network before the conversion.

% Simplify the ONNX network by merging the batch-norm layer and add layer
% into matmul layer
argsToPy = strjoin( {'python',onnxRectifier,exportedONNXModelName, updatedONNX },' ');
cmdExecution(argsToPy,'Failed while rectifiying ONNX',dumpFile);

Create a JSON template file for this network using the eai_builder tools.

cmdExecution(strjoin({eaiBuilder,'--onnx',updatedONNX,'--create_quantization_template',templateJSON},' '), ...
    'Failed while creating the quantization file',dumpFile);

The activation parameters from the JSON file must be updated with the quantization data obtained using the calibration data. For more information on this JSON file, refer to the Quantization Configuration (JSON) section in the LPAI SDK documentation. The bitwidth field from the JSON file determines the bit width of the quantized model, and in this example, it is 8-bit.

Note: Updating the parameters in JSON file is a manual process. To help you skip the manual update part while executing, this example provides a supporting file updatedQuantization.json . Compare the updatedQuantization.json file with the default generated template file templateQuantization.json to learn which parameters are required to be updated from the previously generated quantization information.

The final step involves generating the eAI model.

updatedJSON = 'updatedQuantization.json';
eaiBinary = 'speechCommandRecognitionNetwork.eai';

% Generate the eAI-model for eNPU3 target of adsp environment
cmdExecution(strjoin({eaiBuilder,'--onnx',updatedONNX,'--quant_json',updatedJSON, ...
    '--output', eaiBinary,'--enable_enpu_ver',lower(enpuVersion),'--target_env','adsp'},' '), ...
    'Failed while creating EAI model',dumpFile);

Create Confusion Matrix Chart for Validation (Optional)

You can validate the eAI model (obtained from the previous section) to ensure the numerical accuracy. Deriving the confusion matrix is useful when you want to compare the eAI model predictions with the original MATLAB Deep Learning network predictions.

Note: This process takes a significant amount of time. Therefore, this section is disabled by default. Set eAIValidation to true to generate the confusion matrix.

eAIValidation = false;

This evaluation uses complete SDK utilities like eai_sample_app (for simulating the eAI model on the eNPU host simulation) and eai_quantize (for quantizing the inputs and dequantizing the outputs). The obtained outputs are compared against the MATLAB floating-point Deep Learning network's predictions using the confusion matrix chart.

% Ensure that eai_sample_app exists in the LPAI SDK
if strcmp(enpuVersion,'V3')
    if ispc
        eaiSampleApp = fullfile(lpaiRoot,'windows_fix32','bin','eai_sample_app.exe');
    else
        eaiSampleApp = fullfile(lpaiRoot,'build-fixed32','eai_runtime','eai_sample_app','eai_sample_app');
    end
else
    if ispc %#ok<UNRCH>
        eaiSampleApp = fullfile(lpaiRoot,'binaries','windows','eai_sample_app','fixed32','eai_sample_app.exe');
    else
        eaiSampleApp = fullfile(lpaiRoot,'binaries','linux_x86','eai_sample_app','fixed32','eai_sample_app.exe');
    end
end

if ~isfile(eaiSampleApp)
    error('''eai_sample_app'' binary was not found, follow LPAI SDK documentation to generate this binary.');
end

if eAIValidation == true
    % Generate the scores of the validation data set
    eaiscores = eaiBatchPredict(eaiBinary,sdkRoot,XValidation,labels,dumpFile);

    % Convert scores to labels
    eaiValidation = scores2label(eaiscores,labels,"auto");
    eaivalidationError = mean(eaiValidation ~= TValidation);

    disp("Validation error: " + eaivalidationError*100 + " %");

    figure(Units="normalized",Position=[0.2,0.2,0.5,0.5]);
    
    % Create confusion matrix chart and compare against the MATLAB DL network predictions
    cm = confusionchart(eaiValidation,TValidation, ...
        Title="Confusion Matrix for Validation Data", ...
        ColumnSummary="column-normalized", ...
        RowSummary="row-normalized", ...
        XLabel='Predicted labels by eAI', ...
        YLabel='True label');
    sortClasses(cm,[commands,"unknown","background"]);
end

This chart compares the output of a 8-bit quantized eAI model with the output of a floating-point MATLAB network, and therefore there will be few misclassifications.

Supporting Functions

This example uses these functions, available as supporting files:

generateTrainAndValidationDataSet - Splits the audio data set into training and validation
extractAudioFeatures -Transforms the audio data into the features required for SpeechCommandRecognition model
cmdExecution - Executes the system commands
eaiBatchPredict - Predicts labels on the given XValidation data

References

[1] Warden P. "Speech Commands: A public dataset for single-word speech recognition", 2017. Available from https://storage.googleapis.com/download.tensorflow.org/data/speech_commands_v0.01.tar.gz. Copyright Google 2017. The Speech Commands Dataset is licensed under the Creative Commons Attribution 4.0 license, available here: https://creativecommons.org/licenses/by/4.0/legalcode.