Acoustics-Based Machine Fault Recognition Code Generation on Raspberry Pi

Example Requirements

For supported versions of libraries and for information about setting up environment variables, see Prerequisites for Deep Learning with MATLAB Coder (MATLAB Coder)

Prepare Input Dataset

Specify a sample rate fs of 16 kHz and a windowLength of 512 samples, as defined in Acoustics-Based Machine Fault Recognition (Audio Toolbox). Set numFrames to 100.

fs = 16000;
windowLength = 512;
numFrames = 100;

To run the Example on a test signal, generate a pink noise signal. To test the performance of the system on a real dataset, download the air compressor dataset [1].

downloadDataset = false;

if ~downloadDataset
    pinkNoiseSignal = pinknoise(windowLength*numFrames);
else
    % Download AirCompressorDataset.zip 
    component = 'audio';
    filename = 'AirCompressorDataset/AirCompressorDataset.zip';
    localfile = matlab.internal.examples.downloadSupportFile(component,filename);
    
    % Unzip the downloaded zip file to the downloadFolder
    downloadFolder = fileparts(localfile);
    if ~exist(fullfile(downloadFolder,'AirCompressorDataset'),'dir')
        unzip(localfile, downloadFolder)
    end
    
    % Create an audioDatastore object dataStore, to manage, the data.
    dataStore = audioDatastore(downloadFolder,'IncludeSubfolders',true,'LabelSource','foldernames');

    % Use countEachLabel to get the number of samples of each category in the dataset.
    countEachLabel(dataStore)
end

Recognize Machine Fault in MATLAB

To run the streaming classifier in MATLAB, download and unzip the system developed in Acoustics-Based Machine Fault Recognition (Audio Toolbox).

component = 'audio';
filename = 'AcousticsBasedMachineFaultRecognition/AcousticsBasedMachineFaultRecognition.zip';
localfile = matlab.internal.examples.downloadSupportFile(component,filename);

downloadFolder = fullfile(fileparts(localfile),'system');
if ~exist(downloadFolder,'dir')    
    unzip(localfile,downloadFolder)
end

To access the recognizeAirCompressorFault function of the system, add downloadFolder to the search path.

addpath(downloadFolder)

Create a dsp.AsyncBuffer (DSP System Toolbox) object to read audio in a streaming fashion and a dsp.AsyncBuffer (DSP System Toolbox) object to accumulate scores.

audioSource = dsp.AsyncBuffer;
scoreBuffer = dsp.AsyncBuffer;

Load the pretrained network and extract labels from the network.

airCompNet = coder.loadDeepLearningNetwork('AirCompressorFaultRecognitionModel.mat');
labels = string(airCompNet.Layers(end).Classes);

Initialize signalToBeTested to pinkNoiseSignal or select a signal from the drop-down list to test the file of your choice from the dataset.

if ~downloadDataset
    signalToBeTested = pinkNoiseSignal;
else
    [allFiles,~] = splitEachLabel(dataStore,1);
    allData = readall(allFiles);
    signalToBeTested = allData(1);
    signalToBeTested = cell2mat(signalToBeTested);
end

Stream one audio frame at a time to represent the system as it would be deployed in a real-time embedded system. Use recognizeAirCompressorFault developed in Acoustics-Based Machine Fault Recognition (Audio Toolbox) to compute audio features and perform deep learning classification.

write(audioSource,signalToBeTested);
resetNetworkState = true;

while audioSource.NumUnreadSamples >= windowLength
    
    % Get a frame of audio data
    x = read(audioSource,windowLength);
    
    % Apply streaming classifier function
    score = recognizeAirCompressorFault(x,resetNetworkState);
   
    % Store score for analysis
    write(scoreBuffer,score);
    
    resetNetworkState = false;
end

Compute the recognized fault from scores and display it.

scores = read(scoreBuffer);
[~,labelIndex] = max(scores(end,:),[],2);
detectedFault = labels(labelIndex)

detectedFault = 
"Flywheel"

Plot the scores of each label for each frame.

plot(scores)
legend("" + labels,'Location','northwest') 
xlabel("Time Step")
ylabel("Score")
str = sprintf("Predicted Scores Over Time Steps.\nPredicted Class: %s",detectedFault);
title(str)

Reset the asynchronous buffer audioSource.

reset(audioSource)

Prepare MATLAB Code For Deployment

This example uses the dsp.UDPSender (DSP System Toolbox) System object to send the audio frame to the executable running on Raspberry Pi and the dsp.UDPReceiver (DSP System Toolbox) System object to receive the score vector from the Raspberry Pi. Create a dsp.UDPSender (DSP System Toolbox) system object to send audio captured in MATLAB to your Raspberry Pi. Set the targetIPAddress to the IP address of your Raspberry Pi. Set the RemoteIPPort to 25000. Raspberry Pi receives the input audio frame from the same port using the dsp.UDPReceiver (DSP System Toolbox) system object.

targetIPAddress = '172.31.164.247';
UDPSend = dsp.UDPSender('RemoteIPPort',25000,'RemoteIPAddress',targetIPAddress); 

Create a dsp.UDPReceiver (DSP System Toolbox) system object to receive predicted scores from your Raspberry Pi. Each UDP packet received from the Raspberry Pi is a vector of scores and each vector element is a score for a state of the air compressor. The maximum message length for the dsp.UDPReceiver (DSP System Toolbox) object is 65507 bytes. Calculate the buffer size to accommodate the maximum number of UDP packets.

sizeOfDoubleInBytes = 8;
numScores = 8;
maxUDPMessageLength = floor(65507/sizeOfDoubleInBytes);
numPackets = floor(maxUDPMessageLength/numScores);
bufferSize = numPackets*numScores*sizeOfDoubleInBytes;

UDPReceive = dsp.UDPReceiver("LocalIPPort",21000, ...  
    "MessageDataType","single", ...
    "MaximumMessageLength",numScores, ...
    "ReceiveBufferSize",bufferSize);

Create a supporting function, recognizeAirCompressorFaultRaspi, that receives an audio frame using dsp.UDPReceiver (DSP System Toolbox) and applies the streaming classifier and sends the predicted score vector to MATLAB using dsp.UDPSender (DSP System Toolbox).

type recognizeAirCompressorFaultRaspi

function recognizeAirCompressorFaultRaspi(hostIPAddress)
% This function receives acoustic input using dsp.UDPReceiver and runs a
% streaming classifier by calling recognizeAirCompressorFault, developed in
% the Acoustics-Based Machine Fault Recognition - MATLAB Example. 
% Computed scores are sent to MATLAB using dsp.UDPSender.
%#codegen

%   Copyright 2021 The MathWorks, Inc.

frameLength = 512;

% Configure UDP Sender System Object
UDPSend = dsp.UDPSender('RemoteIPPort',21000,'RemoteIPAddress',hostIPAddress);

% Configure UDP Receiver system object
sizeOfDoubleInBytes = 8;
maxUDPMessageLength = floor(65507/sizeOfDoubleInBytes);
numPackets = floor(maxUDPMessageLength/frameLength);
bufferSize = numPackets*frameLength*sizeOfDoubleInBytes;
UDPReceiveRaspi = dsp.UDPReceiver('LocalIPPort',25000, ...
    'MaximumMessageLength',frameLength, ...
    'ReceiveBufferSize',bufferSize, ...
    'MessageDataType','double');

% Reset network state for first call
resetNetworkState = true;

while true
    % Receive audio frame of size frameLength x 1
    x = UDPReceiveRaspi();

    if(~isempty(x))

        x = x(1:frameLength,1);

        % Apply streaming classifier function
        scores = recognizeAirCompressorFault(x,resetNetworkState);

        %Send output to the host machine
        UDPSend(scores);

        resetNetworkState = false;
    end
end

Generate Executable on Raspberry Pi

Replace the hostIPAddress with your machine's address. Your Raspberry Pi sends the predicted scores to the IP address you specify.

hostIPAddress = coder.Constant('172.18.230.30');

Create a code generation configuration object to generate an executable program. Specify the target language as C++.

cfg = coder.config('exe');
cfg.TargetLang = 'C++';

Create a configuration object for deep learning code generation with the ARM compute library that is on your Raspberry Pi. Specify the architecture of the Raspberry Pi and attach the deep learning configuration object to the code generation configuration object.

dlcfg = coder.DeepLearningConfig('arm-compute');
dlcfg.ArmArchitecture = 'armv7';
dlcfg.ArmComputeVersion = '20.02.1';
cfg.DeepLearningConfig = dlcfg;

Use the Raspberry Pi Support Package function raspi to create a connection to your Raspberry Pi. In the next block of code, replace:

if (~exist('r','var'))
  r = raspi('raspiname','pi','password');
end

Create a coder.hardware (MATLAB Coder) object for Raspberry Pi and attach it to the code generation configuration object.

hw = coder.hardware('Raspberry Pi');
cfg.Hardware = hw;

Specify the build folder on the Raspberry Pi.

buildDir = '~/remoteBuildDir';
cfg.Hardware.BuildDir = buildDir;

Use an autogenerated C++ main file to generate a standalone executable.

cfg.GenerateExampleMain = 'GenerateCodeAndCompile';

Call the codegen (MATLAB Coder) function from MATLAB Coder to generate C++ code and the executable on your Raspberry Pi. By default, the Raspberry Pi executable has the same name as the MATLAB function. You get a warning in the code generation logs that you can disregard because recognizeAirCompressorFaultRaspi has an infinite loop that looks for an audio frame from MATLAB.

codegen -config cfg recognizeAirCompressorFaultRaspi -args {hostIPAddress} -report

 Deploying code. This may take a few minutes. 
Warning: Function 'recognizeAirCompressorFaultRaspi' does not terminate due to an infinite loop.

Warning in ==> recognizeAirCompressorFaultRaspi Line: 1 Column: 1
Code generation successful (with warnings): View report

Perform Machine Fault Recognition Using Deployed Code

Create a command to open the recognizeAirCompressorFaultRaspi application on a Raspberry Pi. Use system to send the command to your Raspberry Pi.

applicationName = 'recognizeAirCompressorFaultRaspi';

applicationDirPaths = raspi.utils.getRemoteBuildDirectory('applicationName',applicationName);
targetDirPath = applicationDirPaths{1}.directory;

exeName = strcat(applicationName,'.elf');
command = ['cd ',targetDirPath,'; ./',exeName,' &> 1 &'];

system(r,command);

Initialize signalToBeTested to pinkNoiseSignal or select a signal from the drop-down list to test the file of your choice from the dataset.

if ~downloadDataset
    signalToBeTested = pinkNoiseSignal;
else
    [allFiles,~] = splitEachLabel(dataStore,1);
    allData = readall(allFiles);
    signalToBeTested = allData(1);
    signalToBeTested = cell2mat(signalToBeTested);
end

Stream one audio frame at a time to represent a system as it would be deployed in a real-time embedded system. Use the generated MEX file recognizeAirCompressorFault_mex to compute audio features and perform deep learning classification.

write(audioSource,signalToBeTested);

while audioSource.NumUnreadSamples >= windowLength
    x = read(audioSource,windowLength);
    UDPSend(x);
    score = UDPReceive();
    if ~isempty(score)    
        write(scoreBuffer,score');
    end
end

Compute the recognized fault from scores and display it.

scores = read(scoreBuffer);
[~,labelIndex] = max(scores(end,:),[],2);
detectedFault = labels(labelIndex)

detectedFault = 
"Flywheel"

Plot the scores of each label for each frame.

plot(scores)
legend("" + labels,'Location','northwest') 
xlabel("Time Step")
ylabel("Score")
str = sprintf("Predicted Scores Over Time Steps.\nPredicted Class: %s",detectedFault);
title(str)

Terminate the standalone executable running on Raspberry Pi.

stopExecutable(codertarget.raspi.raspberrypi,exeName)

Evaluate Execution Time Using Alternative PIL Function Workflow

To evaluate execution time taken by standalone executable on Raspberry Pi, use a PIL (processor-in-loop) workflow. To perform PIL profiling, generate a PIL function for the supporting function recognizeAirCompressorFault.

Create a code generation configuration object to generate the PIL function.

cfg = coder.config('lib','ecoder',true);
cfg.VerificationMode = 'PIL';

Set the ARM compute library and architecture.

dlcfg = coder.DeepLearningConfig('arm-compute');
cfg.DeepLearningConfig = dlcfg ;
cfg.DeepLearningConfig.ArmArchitecture = 'armv7';
cfg.DeepLearningConfig.ArmComputeVersion = '20.02.1';

Set up the connection with your target hardware.

if (~exist('r','var'))
  r = raspi('raspiname','pi','password');
end
hw = coder.hardware('Raspberry Pi');
cfg.Hardware = hw;

Set the build directory and target language.

buildDir = '~/remoteBuildDir';
cfg.Hardware.BuildDir = buildDir;
cfg.TargetLang = 'C++';

Enable profiling and generate the PIL code. A MEX file named recognizeAirCompressorFault_pil is generated in your current folder.

cfg.CodeExecutionProfiling = true;
audioFrame = ones(windowLength,1);
resetNetworkStateFlag = true;
codegen -config cfg recognizeAirCompressorFault -args {audioFrame,resetNetworkStateFlag}

 Deploying code. This may take a few minutes. 
### Connectivity configuration for function 'recognizeAirCompressorFault': 'Raspberry Pi'
Location of the generated elf : /home/pi/remoteBuildDir/MATLAB_ws/R2021b/S/MATLAB/Examples/ExampleManager/sporwal.Bdoc21b.j1720794/deeplearning_shared-ex44063374/codegen/lib/recognizeAirCompressorFault/pil
Code generation successful.

Call the generated PIL function 50 times to get the average execution time.

totalCalls = 50;

for k = 1:totalCalls
    x = pinknoise(windowLength,1);
    score = recognizeAirCompressorFault_pil(x,resetNetworkStateFlag);
    resetNetworkStateFlag = false;
end

### Starting application: 'codegen\lib\recognizeAirCompressorFault\pil\recognizeAirCompressorFault.elf'
    To terminate execution: clear recognizeAirCompressorFault_pil
### Launching application recognizeAirCompressorFault.elf...
    Execution profiling data is available for viewing. Open Simulation Data Inspector.
    Execution profiling report available after termination.

Terminate the PIL execution.

clear recognizeAirCompressorFault_pil

### Host application produced the following standard output (stdout) and standard error (stderr) messages:

### Connectivity configuration for function 'recognizeAirCompressorFault': 'Raspberry Pi'
    Execution profiling report: report(getCoderExecutionProfile('recognizeAirCompressorFault'))

Generate an execution profile report to evaluate execution time.

executionProfile = getCoderExecutionProfile('recognizeAirCompressorFault');
report(executionProfile, ...
       'Units','Seconds', ...
       'ScaleFactor','1e-03', ...
       'NumericFormat','%0.4f');

The average execution time of recognizeAirCompressorFault_pil function is 0.423 ms, which is well below the 32 ms budget for real-time performance. The first call of recognizeAirCompressorFault_pil consumes around 12 times of the average execution time as it includes loading of network and resetting of the states. However, in a real, deployed system, that initialization time is incurred only once. This example ends here. For deploying machine fault recognition on desktops, see Acoustics-Based Machine Fault Recognition Code Generation (Audio Toolbox).

References

[1] Verma, Nishchal K., et al. "Intelligent Condition Based Monitoring Using Acoustic Signals for Air Compressors." IEEE Transactions on Reliability, vol. 65, no. 1, Mar. 2016, pp. 291–309. DOI.org (Crossref), doi:10.1109/TR.2015.2459684.