Classify and Count Fruits Using Arduino Nano 33 BLE Sense, ThingSpeak, and Machine Learning

Overview

This example uses an Arduino Nano BLE 33 Sense board that you can connect to your machine using a micro USB cable. The algorithm in the example completes these steps.

You can use the trained model included with this example instead of training the model on your own.

If you are using the trained model, skip the Create Data Set from RGB Values of Fruits and Export Classification Algorithm section.

Create Data Set from RGB Values of Fruits

Place the Arduino Nano 33 BLE Sense board near the fruit for which you want to capture the RBG values. This image shows a person capturing the RBG data for an apple.

Download the captureRGB.m file by clicking the Open Live Script button. Execute the file and complete the instructions that display in the Command Window. Once you complete executing the file, you can view the trainingData variable of size 2001X3 in the MATLAB workspace. Add a fourth field, fruit, to the trainingData table. Change all the entries in the column under the new field to Apple.

trainingData.fruit(:) = "Apple" ;

Create a training data variable for apples.

trainingDataApple = trainingData ;

The new trainingDataApple variable of size 2001x4 is now visible in the MATLAB workspace.

To capture the RGB values for bananas, execute the captureRBG.m file again. Create a new field called Banana and a training data variable for bananas.

trainingData.fruit(:) = "Banana" ;
trainingDataBanana = trainingData ;

To capture the RGB values for guavas, execute the captureRBG.m file again. Create a new field called Guava and a training data variable for guavas.

trainingData.fruit(:) = "Guava" ;
trainingDataGuava = trainingData ;

The new trainingDataBanana and trainingDataGuava variables of size 2001x4 are now visible in the MATLAB workspace.

Store the training data for the three fruits in a single variable finalTrainingData. The finalTrainingData variable must be of the size 6003x4.

finalTrainingData = [trainingDataApple; trainingDataBanana; trainingDataGuava];

Export Classification Algorithm

Use the Classification Learner app to select a classification model. In the Apps tab on the MATLAB toolstrip, search for the Classification Learner app.

In the app, click the New Session button, and then click From Workspace.

In the Data set pane, select the training variable finalTrainingData and click Start Session.

On the app toolstrip, in the Models section, click All, and in the Train section, click Train All.

.

The app provides you with a list of classification models and their results. For this example, select the model with highest accuracy.

On the app toolstrip, click Export and then click Export Model.

Select Include the training data in the exported model and export the model.

Create ThingSpeak Channel and Add Widgets to Count Fruits

If you are using the trained model included with this example, skip the Create Data Set from RGB Values of Fruits and Export Classification Algorithm sections and start with this section.

To download the trained model included with this example, click the Open Live Script button. To load the trained model, type this command at the MATLAB command line.

load trainedModel.mat

ThingSpeak is an IoT analytics platform service that you can use to aggregate, visualize, and analyze live data streams in the cloud. For more information, see Get Started with ThingSpeak (ThingSpeak).

To track the number of fruits in the box using ThingSpeak™, sign in using your MathWorks® Account credentials, or create a new account. Create a new channel in ThingSpeak to collect and analyze the fruit count data. To create a new channel in ThingSpeak, on the Channels tab, click My Channels. In the My Channels window, click New Channel and enter these values to customize the channel.

ThingSpeak uses the Field 4 field to track if the number of any one of the three types of fruits drops to zero.

Click Save Channel to save the channel settings. In the Channels window, click Add Widgets to add three numeric display widgets to track the number of fruits.

Identify API Keys in ThingSpeak

Before you write the MATLAB code to track the number of fruits, identify the API keys of the ThingSpeak channel. These keys allow you to read data from and write data to your channel.

To identify the API keys, on the Channels tab, click API Keys.

Execute MATLAB Code to Classify and Count Fruits

To track the number of fruits in the box, whenever you take a fruit from the box, place the fruit near the sensor. The sensor then classifies the fruit and updates the fruit count in the ThingSpeak channel.

You can execute the following MATLAB code to read RGB data from the sensor and use this data to predict the type of fruit based on a pretrained model and publish the data to ThingSpeak using the channel ID and the Write API key of the channel. Run these commands, one after another, in the MATLAB Command Window. This example assumes that the box contains 50 guavas, 40 bananas. This example also assumes that 30 apples and the Arduino board is connected to the COM14 port.

% Create the arduino object with required library
arduinoObj = arduino("COM14","Nano33BLE","Libraries","APDS9960");

% Create an APDS9960 object
apds9960Obj = apds9960(arduinoObj, "Bus", 1, "BitRate", 100000);


% Variable to track the number of apples, bananas, and guavas
guavaCount = 50;
bananaCount = 40;
appleCount = 30;

%the variable to track if the fruit count of one of the fruits drops to zero
isEmpty = 0; 


%isEmpty variable changes to 1 if the fruit count of one of the fruits drops to zero
if(guavaCount == 0 || bananaCount == 0 || appleCount == 0)
    isEmpty = 1;
end

% ThingSpeak configurations
channelID = xxxxxxx;
writeAPI = 'xxxxxxxxxxxxxxxx'; %Update this to the writeAPI key identified your Thingspeak Channels page

% Write the value to ThingSpeak channel
thingSpeakWrite(channelID,[appleCount,guavaCount,bananaCount,isEmpty],'WriteKey',writeAPI,'Fields',[1,2,3,4]);

while(1)
    % Continuously read the proximity readings
    proximReadings = readProximity(apds9960Obj);
    flag = 0;
    while(proximReadings>0.1)
        if(flag == 0)
            % Alert to bring the object near the sensor
            disp("Bring object near sensor");
        end
        proximReadings = readProximity(apds9960Obj);
        flag = 1;
    end

    % Turn on the built-in LED when the object is near
    writeDigitalPin(arduinoObj, 'D13', 1);

    % Read the RGB value from apds9960
    colorData = readColor(apds9960Obj);

    % Convert colorData to table format
    colorDataInTableFormat = array2table(colorData);

    % Call predict function
    [predictedFruit,scores] = trainedModel.predictFcn(colorDataInTableFormat);
    message = ["The object detected is ",predictedFruit];
    disp(message);

    % Update the fruit count according to the prediction
    if(predictedFruit=="Apple")
        appleCount = appleCount-1;
    elseif(predictedFruit=="Banana")
        bananaCount = bananaCount-1;
    elseif(predictedFruit=="Guava")
        guavaCount = guavaCount-1;
    end
   
    while(proximReadings <0.1)
        proximReadings = readProximity(apds9960Obj);
    end

    if(guavaCount == 0 || bananaCount == 0 || appleCount == 0)
      isEmpty = 1;
    end

    % Turn off the LED
    writeDigitalPin(arduinoObj, 'D13', 0);

    % Write data to ThingSpeak channel
    thingSpeakWrite(channelID,[appleCount,guavaCount,bananaCount,isEmpty],'WriteKey',writeAPI,'Fields',[1,2,3,4]);

    
end

This GIF shows the channel dashboard when you remove one fruit from the box.

Set Up Email Alert

Configure ThingSpeak to alert you via email if the number of any one of the three types of fruits drops to zero. ThingSpeak sends the email to the email address associated with your ThingSpeak account.

To set up the email alert, on the Apps tab on the ThingSpeak toolstrip, click MATLAB Analysis.

Click New to select a code template.

Select the Custom (no starter code) option and then click Create to create the template.

Set the Name to RefillAlert and enter the code in the MATLAB Code window or copy the sample code from the RefillAlert.m file provided with this example. To access the RefillAlert.m file, click the Open Live Script button and download the file.

Specify the channel ID, Read API key, and Alerts API key corresponding to your channel in code. You can access the alerts key from your ThingSpeak profile page.

To set up an email alert for when the number of any one of the three types of fruits drops to zero, on the Apps tab in ThingSpeak, click Actions, and then click React. In the React window, set these parameters.

Click Save React to save your alert.

When the number of any one of the three types of fruits drops to zero, ThingSpeak sends an email to the registered email address. This is an image of a sample email sent by ThingSpeak.