Collect Coverage for Generated C/C++ Code in Equivalence Tests

This example uses:

When you run C/C++ equivalence tests that build static libraries and execute using software-in-the-loop (SIL) or processor-in-the-loop (PIL) verification, you can collect code coverage information for the generated C/C++ code. Collect coverage interactively by using the MATLAB Test Manager or programmatically by authoring and running a script. When you run tests interactively, you can view the coverage results in a code coverage report or in the Code Quality Dashboard. When you run tests programmatically, you can also view the coverage results in a code coverage report, and can optionally view the results in the Cobertura XML format. Additionally, you can access the coverage results at the MATLAB^® command line. For more information about generated C/C++ equivalence tests, see Generate C/C++ Code and Test for Equivalence.

You can collect these types of coverage:

Statement
Function
Decision
Condition
Modified condition/decision (MC/DC)

For more information, see Types of Coverage for Generated C/C++ Code in Equivalence Tests.

Note

You can only collect coverage for C/C++ equivalence tests on Windows^® and Linux^® platforms.

If you need to deploy C++ shared libraries, you can write equivalence tests that use MATLAB Compiler SDK™ to generate and test deployable C++ shared libraries from MATLAB source code. For more information, see Choose an Approach for Equivalence Testing Generated C++ Code. However, you cannot collect coverage for equivalence tests that generate deployable C++ shared libraries.

Write SIL and PIL Equivalence Tests

You can only collect coverage for C/C++ code generated in an equivalence test if the test uses SIL or PIL verification and builds a static library. For more information, see Execute and Verify in SIL or PIL Mode.

Write Equivalence Tests

Suppose that you want to generate code for a function called myMath, which allows a user to provide two numeric inputs and an operation string to indicate whether to add or subtract the inputs:

function y = myMath(a,b,operation) %#codegen
if operation == "add"
    y = a+b;
elseif operation == "subtract"
    y = b-a;
else
    y = [];
end
end

This SIL equivalence test uses class properties and class-level setup to generate a static library once and specifies that the operation argument can be a variable-sized string. The uses parameterization to execute and verify the generated C code twice, with different inputs each time.

classdef tMyMathSIL < matlabtest.coder.TestCase
    properties
        BuildResults;
    end

    methods (TestClassSetup)        
        function generateCode(testCase)           
            operationSize = coder.typeof("add");
            operationSize.StringLength = inf;
            buildInputs = {1,2,operationSize};
            testCase.BuildResults = build(testCase,"myMath", ...
                Inputs=buildInputs,Configuration="lib");
        end        
    end

    properties (TestParameter)
        runInputs = {{1,2,"add"},{1,2,"subtract"}};
    end
    
    methods(Test)
        function testSILvsMATLAB(testCase,runInputs)
            executionResults = execute(testCase, ...
                testCase.BuildResults,Inputs=runInputs);
            verifyExecutionMatchesMATLAB(testCase,executionResults);
        end        
    end
end

This PIL equivalence test uses the same format as the SIL equivalence test and defines a code generation configuration parameter object to generate code for an ARM^® Cortex^®-M3 board and execute and verify in PIL mode.

classdef tMyMathPIL < matlabtest.coder.TestCase
    properties
        BuildResults;
    end

    methods (TestClassSetup)
        function generateCode(testCase)
            operationSize = coder.typeof("add");
            operationSize.StringLength = inf;
            buildInputs = {1,2,operationSize};

            cfg = coder.config("lib","ecoder",true);
            cfg.Hardware = coder.hardware("ARM Cortex-M3 (QEMU)");
            cfg.VerificationMode = "PIL";
            
            testCase.BuildResults = build(testCase,"myMath", ...
                Inputs=buildInputs,Configuration=cfg);
        end
    end

    properties (TestParameter)
        runInputs = {{1,2,"add"},{1,2,"subtract"}};
    end

    methods(Test)
        function testPILvsMATLAB(testCase,runInputs)
            executionResults = execute(testCase, ...
                testCase.BuildResults,Inputs=runInputs);
            verifyExecutionMatchesMATLAB(testCase,executionResults);
        end
    end
end

Write Equivalence Tests for Multiple Entry-Point Functions

Suppose that you have two functions called myAdd and mySubtract and you want to generate C/C++ code from both functions at the same time.

function y = myAdd(a,b) %#codegen
y = a+b;
end

function y = mySubtract(a,b) %#codegen
y = b-a;
end

This SIL equivalence test uses class properties and class-level setup to generate a static library and specifies both entry-point functions. The test uses parameterization to execute and verify the generated C code twice, executing a different entry-point function each time.

classdef tMyAddMySubtract < matlab.unittest.TestCase
    properties
        Tester;
    end

    methods (TestClassSetup)        
        function generateCode(testCase)
            import matlabtest.coder.MATLABCoderTester
            
            buildInputs = {0,0};
            tester = MATLABCoderTester.forLIBCoderConfiguration( ...
                "myAdd.m",Inputs=buildInputs);
            addEntryPointFunction(tester,"mySubtract.m",buildInputs);
            testCase.Tester = tester;
            build(testCase.Tester,testCase);    
        end
    end
    
    properties (TestParameter)
        entryPoint = {"myAdd","mySubtract"};
    end

    methods(Test)
        function testSILvsMATLAB(testCase,entryPoint)            
            import matlabtest.constraints.ExecutionMatchesMATLAB

            execute(testCase.Tester, ...
                testCase,Inputs={1,2},EntryPoint=entryPoint);
            verifyThat(testCase,testCase.Tester,ExecutionMatchesMATLAB);
        end
    end
end

Interactively Run Tests, Collect, and View Coverage

Since R2024b

You can interactively run equivalence tests with SIL or PIL verification and collect coverage for the generated code by running the tests with generated code coverage enabled in the MATLAB Test Manager. You can view the coverage results by generating a coverage report or view a summary of the coverage results in the Generated Code Coverage section.

Run Tests and Collect Coverage

To interactively run equivalence tests with SIL or PIL verification and collect coverage for the generated code:

Open the project that contains the tests.
Open the MATLAB Test Manager.
Enable generated code coverage by clicking the Code Coverage button and selecting Enable generated code coverage. To configure the test manager to automatically open the generated code coverage report when you run equivalence tests, select Automatically open code coverage report.
Set the coverage metric level to the highest level of coverage that you want to include in the code coverage report.

When you select a metric level, the report includes the metrics up to and including the selected level. For example, if you select Condition, the report includes the statement, decision, and condition coverage metrics.
Run the equivalence tests.
- Run all tests in the project by setting the drop-down menu to the left of the Run button to All Tests in Current Project. Then, click the Run button .
- Run specific equivalence tests by right-clicking the test or test file and selecting Run selected.

View Coverage Results

When you run equivalence tests by using the MATLAB Test Manager, you can view the coverage results in a code coverage report. To open the code coverage report, click the Report button . Under Generated Code Coverage Reports, select the report. The MATLAB Test Manager stores the five most recent generated code coverage reports.

Alternatively, you can view a summary of the coverage results in the Generated Code Coverage section of the Code Quality Dashboard. You can open the associated code coverage report by clicking the Generated Code Coverage section.

The generated code coverage section of the dashboard with results for all metric levels.

Programmatically Run Tests, Collect, and View Coverage

You can programmatically run equivalence tests with SIL or PIL verification and collect coverage for the generated code by authoring and running a script that uses the generated code coverage plugin. You can programmatically create coverage reports or access the results at the MATLAB command line.

Run Tests and Collect Coverage

To programmatically run equivalence tests with SIL or PIL verification and collect coverage for the generated code:

Create a new script. For more information, see Create Scripts.
Define the coverage format by creating an instance of matlab.unittest.plugins.codecoverage.CoverageResult.
Create a plugin for the generated C/C++ code by creating an instance of matlabtest.coder.plugins.GeneratedCodeCoveragePlugin. Specify the desired coverage type for the plugin or use the default settings, which specify statement and function coverage.
Create a test runner by using the testrunner function.
Add the code coverage plugin to the test runner by using the addPlugin method.
Create a test suite for the equivalence tests by using the testsuite function.
Run the tests by using the run method.

This example code creates a test runner with a plugin that programmatically accesses the coverage information for all types of code coverage.

import matlab.unittest.plugins.codecoverage.CoverageResult
import matlabtest.coder.plugins.GeneratedCodeCoveragePlugin

format = CoverageResult;
plugin = GeneratedCodeCoveragePlugin(Producing=format, ...
    MetricLevel="mcdc");
runner = testrunner("textoutput");
addPlugin(runner,plugin);

This example code creates a test suite for the tMyMathSIL equivalence test class and runs the tests. All tests pass.

suite = testsuite("tMyMathSIL.m");
run(runner,suite);

Running tMyMathSIL
..
Done tMyMathSIL
__________

View Coverage Results

To generate a coverage report from the coverage results, pass the Result property of the matlab.unittest.plugins.codecoverage.CoverageResult object to the generateHTMLReport method. For more information about the code coverage report, see Collect Code Coverage Metrics for MATLAB Source Code.

This example code generates an HTML code coverage report with name myCoverageReport for the coverage results collected programmatically from the test in tMyMathSIL.m. The code stores the report in a folder called myArtifacts in the current folder.

covResult = format.Result;
generateHTMLReport(covResult,"myArtifacts", ...
    MainFile="myCoverageReport.html");

Alternatively, to generate a standalone report as a single HTML file, use the generateStandaloneReport method.

You can also generate the coverage report in the Cobertura XML format. For more information, see Generate Cobertura Coverage Reports for Generated C/C++ Code in Equivalence Tests.

To view the coverage results programmatically, pass the Result property of the matlab.unittest.plugins.codecoverage.CoverageResult object to the coverageSummary method.

This example code returns the statement coverage results collected from the test in tMyMathSIL.m.

covResult = format.Result;
covSummary = coverageSummary(covResult,"statement")

covSummary = 5×2    
    33    37
     0    49
    38    45
     0     0
     0     0

Each row in the matrix indicates the coverage results for a file in the static library. The first column indicates the number of executed statements and the second column indicates the number of total statements.