Main Content

Code Generation and Deployment for Acoustic-Based Machine Fault Detection using Deep Learning on ARM Cortex-M Hardware

Since R2024b

This example uses:

Open Live Script

This example demonstrates code generation for Acoustics-Based Machine Fault Recognition (Audio Toolbox) using a long short term memory (LSTM) network and spectral descriptors. This example uses Embedded® Coder™ with deep learning support to generate a processor in the loop (PIL) executable function that leverages C code which is then deployed on ARM Cortex-M™ hardware. The input data consists of acoustics time series recordings from a faulty or healthy air compressors and the output is the state of the mechanical machine predicted by the LSTM network. For details on audio preprocessing and network training, see Acoustics-Based Machine Fault Recognition (Audio Toolbox).

Prepare Input Dataset

Specify a sample rate fs of 16 kHz and a windowLength of 512 samples, as defined in Acoustics-Based Machine Fault Recognition (Audio Toolbox). Set numFrames to 100.

fs = 16000;
windowLength = 512;
numFrames = 100;

To run the example on a test signal, generate a pink noise signal. To test the system on a real dataset, download the air compressor dataset [1].

downloadDataset = true;;

if ~downloadDataset
    pinkNoiseSignal = pinknoise(windowLength*numFrames,'single');
else
    % Download AirCompressorDataset.zip 
    component = 'audio';
    filename = 'AirCompressorDataset/AirCompressorDataset.zip';
    localfile = matlab.internal.examples.downloadSupportFile(component,filename);

    % Unzip the downloaded zip file to the downloadFolder
    downloadFolder = fileparts(localfile);
    if ~exist(fullfile(downloadFolder,'AirCompressorDataset'),'dir')
        unzip(localfile,downloadFolder)
    end
    
    % Create an audioDatastore object dataStore, to manage, the data.
    dataStore = audioDatastore(downloadFolder,IncludeSubfolders=true,LabelSource="foldernames",OutputDataType="single");

    % Use countEachLabel to get the number of samples of each category in the dataset.
    countEachLabel(dataStore)
end
ans=8×2 table
      Label      Count
    _________    _____

    Bearing       225 
    Flywheel      225 
    Healthy       225 
    LIV           225 
    LOV           225 
    NRV           225 
    Piston        225 
    Riderbelt     225

Initialize Test Signal and Load the Network

Create a dsp.AsyncBuffer (DSP System Toolbox) object to read audio in streaming mode and a dsp.AsyncBuffer (DSP System Toolbox) object to accumulate scores for MATLAB and PIL workflows.

audioSource = dsp.AsyncBuffer;
scoreBufferMATLAB = dsp.AsyncBuffer;
scoreBufferPIL = dsp.AsyncBuffer;

Get the labels corresponding to the network outputs.

model = load('AirCompressorFaultRecognitionModel.mat');
labels = string(model.labels);

Initialize signalToBeTested to pinkNoiseSignal or select a signal from the drop-down list to test the file of your choice from the dataset.

if ~downloadDataset
    signalToBeTested = pinkNoiseSignal;
else
    [allFiles,~] = splitEachLabel(dataStore,1);
    allData = readall(allFiles);
    signalToBeTested = allData(3);
    signalToBeTested = cell2mat(signalToBeTested);
end

Detect Machine Fault in MATLAB

To run the streaming classifier in MATLAB, use the pretrained network in AirCompressorFaultRecognition.mat and wrapper file recognizeAirCompressorFault.m developed in Acoustics-Based Machine Fault Recognition (Audio Toolbox).

Stream one audio frame at a time to represent the system as it would be deployed in a real-time embedded system. Use recognizeAirCompressorFault developed in Acoustics-Based Machine Fault Recognition (Audio Toolbox) to compute audio features and perform deep learning classification.

write(audioSource,signalToBeTested);
resetNetworkState = true;

while audioSource.NumUnreadSamples >= windowLength

    % Get a frame of audio data
    x = read(audioSource,windowLength);

    % Apply streaming classifier function
    score = recognizeAirCompressorFault(x,resetNetworkState);
    
    % Store score for analysis
    write(scoreBufferMATLAB,extractdata(score)');
    
    resetNetworkState = false;
end

Compute the recognized fault from scores and display it. 

scoresMATLAB = read(scoreBufferMATLAB);
[~,labelIndex] = max(scoresMATLAB(end,:),[],2);
detectedFault = labels(labelIndex)
detectedFault = 
"Healthy"

Plot the scores of each label for each frame.

plot(scoresMATLAB)
legend("" + labels,Location="northwest") 
xlabel("Time Step")
ylabel("Score")
str = sprintf("Predicted Scores Over Time Steps.\nPredicted Class: %s",detectedFault);
title(str)

Detect Machine Fault on Hardware using PIL

Hardware Requirements:

  • Cortex-M™ hardware such as STM32F769I-Discovery board

  • In this case, the network requires a flash memory of 570 KB. The board used in this example comes equipped with a 2 MB flash memory capacity, which exceeds the network's flash memory requirements. The memory requirement for the network is determined by adding together the inputs and learnables of the network. This can be examined using the analyzeNetwork function.

  • Please ensure that the board's flash size is more than the network's requirements to avoid flash overflow error during code generation.

Generate PIL Executable for Cortex-M Hardware

To generate a PIL MEX function, create a coder.config object for a static library and set the verification mode to 'PIL'. Set the target language to C.

cfg = coder.config('lib','ecoder',true);
cfg.VerificationMode = 'PIL';
cfg.TargetLang = "C";

Create a coder.hardware object for the STM32F769I-Discovery board. Set the hardware property of the coder.config object cfg to coder.hardware object hw. In the following code, replace the STM32F769I-Discovery with the connected hardware and replace COM5 with the port number to which you have connected the Cortex-M hardware.

hardware = "STM32F769I-Discovery";
hw = coder.hardware(hardware);
cfg.Hardware = hw;
cfg.Hardware.PILCOMPort = "COM5";
cfg.HardwareImplementation.TargetHWDeviceType = 'ARM Compatible->ARM Cortex-M';

Create a deep learning configuration object and configure the deep learning config target library to none which generates C Code. Set the code replacement library to 'ARM Cortex-M' for more optimizations.

cfg.DeepLearningConfig = coder.DeepLearningConfig("TargetLibrary","none");
cfg.CodeReplacementLibrary = 'ARM Cortex-M';

Enable the code execution profile flag to obtain timing details.

cfg.CodeExecutionProfiling = true;

Create an audio data frame of length windowLength.

audioFrame = ones(windowLength,1,"single");

Call the codegen function to generate C code for the recognizeAirCompressorFault function. Specify the configuration object and prototype arguments. Check your current folder to see the generated PIL file recognizeAirCompressorFault_pil.

codegen -config cfg recognizeAirCompressorFault -args {audioFrame,resetNetworkState} -report
### Connectivity configuration for function 'recognizeAirCompressorFault': 'STM32 Microcontroller'
### COM port: COM5
### Baud rate: 115200
Code generation successful: View report

Detect Machine Fault on ARM Cortex-M Hardware

Stream one audio frame at a time to represent the system as it would be deployed in a real-time embedded system. Use generated recognizeAirCompressorFault_pil file to compute audio features and perform deep learning classification.

release(audioSource);
write(audioSource,signalToBeTested);
resetNetworkState = true;
while audioSource.NumUnreadSamples >= windowLength

    % Get a frame of audio data
    x = read(audioSource,windowLength);

    % Apply streaming classifier function
    score = recognizeAirCompressorFault_pil(x,resetNetworkState);
    
    % Store score for analysis
    write(scoreBufferPIL,extractdata(score)');

    resetNetworkState = false;
end
### Connectivity configuration for function 'recognizeAirCompressorFault': 'STM32 Microcontroller'
### COM port: COM5
### Baud rate: 115200
### Starting application: 'codegen\lib\recognizeAirCompressorFault\pil\recognizeAirCompressorFault.elf'
    To terminate execution: clear recognizeAirCompressorFault_pil
### Downloading executable to the hardware on Drive: G:
C:\Users\bhpatray\OneDrive - MathWorks\Documents\MATLAB\ExampleManager\bhpatray.latestBdoc\deeplearning_shared-ex42361478\codegen\lib\recognizeAirCompressorFault\pil\recognizeAirCompressorFault.bin 
1 File(s) copied 
    Execution profiling data is available for viewing. Open Simulation Data Inspector.
    Execution profiling report will be available after termination.

Compute the recognized fault from scores and display it. 

scoresPIL = read(scoreBufferPIL);
[~,labelIndex] = max(scoresPIL(end,:),[],2);
detectedFault = labels(labelIndex)
detectedFault = 
"Healthy"

Plot the scores of each label for each frame.

plot(scoresPIL)
legend(labels,Location="northwest")
xlabel("Time Step")
ylabel("Score")
str = sprintf("Predicted Scores Over Time Steps.\nPredicted Class: %s",detectedFault);
title(str)

Evaluate PIL Performance

Obtain the execution profiling report after terminating the PIL.

clear recognizeAirCompressorFault_pil;
    Execution profiling report: coder.profile.show(getCoderExecutionProfile('recognizeAirCompressorFault'))

profileInfo = getCoderExecutionProfile('recognizeAirCompressorFault');

Running the PIL for a single instance takes 1,153,450 ticks, and with the board's clock speed set at 216 MHz, this translates to an approximate time of 5.3 milliseconds.

clockSpeed = profileInfo.TimerTicksPerSecond;
disp(["ClockSpeed :" clockSpeed]);
    "ClockSpeed :"    "216000000"

[initiationTicks, executionTicks, terminationTicks] =  profileInfo.Sections.MaximumExecutionTimeInTicks;
timeTakenForInstance = seconds(single(executionTicks)/single(clockSpeed));
disp(timeTakenForInstance);
   0.0053463 sec

Compare PIL and MATLAB Machine Fault Detection Scores

Compare the outputs of MATLAB and PIL and check if they are within the tolerace value.

checkWithinTol = ismembertol(scoresMATLAB, scoresPIL, 1e-5);
isVerificationPassed = isequal(checkWithinTol, ones(size(scoresMATLAB)))
isVerificationPassed = logical
   1

Further Exploration

  • You can change the signalToBeTested label to 'bearing' or 'piston' and measure the accuracy of other labels.

  • You can also generate the signal using pinkNoise and set the downloadDataset to false to test the example with random signal.

  • The compatible hardware can be set to hardware in "Generate PIL Executable for Cortex-M Hardware" section. Ensure hardware flash memory is more than the memory requirement of network. The memory requirement for the network is determined by adding together the inputs and learnables of the network. This can be examined using the analyzeNetwork function.

  • To attain real time requirements, change the architecture of the network and train the network by referring to Acoustics-Based Machine Fault Recognition (Audio Toolbox) (or) use hardware which can meet network requirements.

References

[1] Verma, Nishchal K., et al. "Intelligent Condition Based Monitoring Using Acoustic Signals for Air Compressors." IEEE Transactions on Reliability, vol. 65, no. 1, Mar. 2016, pp. 291–309. DOI.org (Crossref), doi:10.1109/TR.2015.2459684.