Run Tests in Parallel with Custom Plugin

This example shows how to create a custom plugin that supports running tests in parallel. The custom plugin counts the number of passing and failing assertions for a test suite. To extend a TestRunner instance, the plugin overrides select methods of the matlab.unittest.plugins.TestRunnerPlugin class. Additionally, to support running tests in parallel, the plugin subclasses the matlab.unittest.plugins.Parallelizable interface. To run tests in parallel, you need Parallel Computing Toolbox™.

Create Plugin Class

In a file named AssertionCountingPlugin.m in your current folder, create the parallelizable plugin class AssertionCountingPlugin, which inherits from both the TestRunnerPlugin and Parallelizable classes. For the complete code of AssertionCountingPlugin, see Plugin Class Definition.

To keep track of the number of passing and failing assertions, define four read-only properties within a properties block. Each MATLAB^® worker on the current parallel pool uses NumPassingAssertions and NumFailingAssertions to track the number of passing and failing assertions when running a portion of the TestSuite array. The MATLAB client uses FinalizedNumPassingAssertions and FinalizedNumFailingAssertions to aggregate the results from different workers and to report the total number of passing and failing assertions at the end of the test session.

    properties (SetAccess = private)
        NumPassingAssertions
        NumFailingAssertions
        FinalizedNumPassingAssertions
        FinalizedNumFailingAssertions
    end

Extend Running of Test Session

To extend the running of the entire TestSuite array, override the runSession method in a methods block with protected access. runSession displays information about the total number of Test elements, initializes the properties that the plugin uses to generate text output, and invokes the superclass method to trigger the entire test run. After the testing framework completes evaluating the superclass method, runSession displays the assertion count summary by calling the helper method printAssertionSummary (see Define Helper Methods). The framework evaluates this method one time on the client.

    methods (Access = protected)
        function runSession(plugin,pluginData)
            suiteSize = numel(pluginData.TestSuite);
            fprintf("## Running a total of %d tests\n\n",suiteSize)
            plugin.FinalizedNumPassingAssertions = 0;
            plugin.FinalizedNumFailingAssertions = 0;

            runSession@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData)

            fprintf("## Done running tests\n")
            plugin.printAssertionSummary
        end
    end

Extend Creation of Shared Test Fixtures and Test Cases

Add listeners to AssertionPassed and AssertionFailed events to count the assertions. To add these listeners, extend the methods that the testing framework uses to create the test content. The test content includes TestCase instances for each Test element, class-level TestCase instances for the TestClassSetup and TestClassTeardown methods blocks, and Fixture instances used when a TestCase class has the SharedTestFixtures attribute. Add these creation methods to a methods block with protected access.

Invoke the corresponding superclass method when you override the creation methods. The creation methods return the content that the testing framework creates for each of their respective contexts. When implementing each of these methods using the incrementPassingAssertionsCount and incrementFailingAssertionsCount helper methods, add the listeners required by the plugin to the returned Fixture or TestCase instance.

    methods (Access = protected)
        function fixture = createSharedTestFixture(plugin,pluginData)
            fixture = createSharedTestFixture@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);

            fixture.addlistener("AssertionPassed", ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            fixture.addlistener("AssertionFailed", ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function testCase = createTestClassInstance(plugin,pluginData)
            testCase = createTestClassInstance@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);

            testCase.addlistener("AssertionPassed", ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener("AssertionFailed", ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function testCase = createTestMethodInstance(plugin,pluginData)
            testCase = createTestMethodInstance@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);

            testCase.addlistener("AssertionPassed", ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener("AssertionFailed", ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end
    end

Extend Running of Test Suite Portion

The testing framework divides the entire TestSuite array into groups and assigns them to workers for processing. Each worker can run one or more test suite portions. To customize the behavior of workers, override the runTestSuite method in a methods block with protected access.

Extend the test runner to display the identifier of each test group that a worker runs along with the number of Test elements within the group. Additionally, store the number of passing and failing assertions in a communication buffer so that the client can retrieve these values to produce the finalized results. Like all plugin methods, the runTestSuite method requires you to invoke the corresponding superclass method at an appropriate point. In this case, invoke the superclass method after initializing the properties and before storing the worker data. The testing framework evaluates runTestSuite on the workers as many times as the number of test suite portions.

    methods (Access = protected)
        function runTestSuite(plugin,pluginData)
            suiteSize = numel(pluginData.TestSuite);
            groupNumber = pluginData.Group;
            fprintf("### Running a total of %d tests in group %d\n", ...
                suiteSize,groupNumber)
            plugin.NumPassingAssertions = 0;
            plugin.NumFailingAssertions = 0;

            runTestSuite@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData)

            assertionStruct = struct( ...
                "Passing",plugin.NumPassingAssertions, ...
                "Failing",plugin.NumFailingAssertions);
            plugin.storeIn(pluginData.CommunicationBuffer,assertionStruct)
        end
    end

To store test-specific data, the implementation of runTestSuite contains a call to the storeIn method of the Parallelizable interface. Use storeIn along with retrieveFrom when workers must report to the client. In this example, after returning from the superclass method, NumPassingAssertions and NumFailingAssertions contain the number of passing and failing assertions corresponding to a group of tests. Because storeIn accepts the worker data as only one input argument, the structure assertionStruct groups the assertion counts using two fields.

Extend Reporting of Finalized Test Suite Portion

Extend reportFinalizedSuite to aggregate the assertion counts by retrieving test data for each finalized test suite portion. To retrieve the stored structure for a test suite portion, invoke the retrieveFrom method within the scope of reportFinalizedSuite. To return default data in case the buffer is empty (since R2024a), such as when a fatal assertion failure prevents all workers from writing to the buffer, specify DefaultData when invoking the retrieveFrom method. Add the field values to the corresponding class properties, and invoke the superclass method. The testing framework evaluates this method on the client as many times as the number of test suite portions.

    methods (Access = protected)
        function reportFinalizedSuite(plugin,pluginData)
            assertionStruct = plugin.retrieveFrom( ...
                pluginData.CommunicationBuffer, ...
                DefaultData=struct("Passing",0,"Failing",0));
            plugin.FinalizedNumPassingAssertions = ...
                plugin.FinalizedNumPassingAssertions + assertionStruct.Passing;
            plugin.FinalizedNumFailingAssertions = ...
                plugin.FinalizedNumFailingAssertions + assertionStruct.Failing;

            reportFinalizedSuite@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData)
        end
    end

Define Helper Methods

In a methods block with private access, define three helper methods. These methods increment the number of passing or failing assertions within each running test suite portion and print the assertion count summary.

    methods (Access = private)
        function incrementPassingAssertionsCount(plugin)
            plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1;
        end

        function incrementFailingAssertionsCount(plugin)
            plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1;
        end

        function printAssertionSummary(plugin)
            fprintf("%s\n",repmat('_',1,30))
            fprintf("Total Assertions: %d\n", ...
                plugin.FinalizedNumPassingAssertions + ...
                plugin.FinalizedNumFailingAssertions)
            fprintf("\t%d Passed, %d Failed\n", ...
                plugin.FinalizedNumPassingAssertions, ...
                plugin.FinalizedNumFailingAssertions)
        end
    end

Plugin Class Definition

This code provides the complete contents of the AssertionCountingPlugin class.

classdef AssertionCountingPlugin < ...
        matlab.unittest.plugins.TestRunnerPlugin & ...
        matlab.unittest.plugins.Parallelizable

    properties (SetAccess = private)
        NumPassingAssertions
        NumFailingAssertions
        FinalizedNumPassingAssertions
        FinalizedNumFailingAssertions
    end

    methods (Access = protected)
        function runSession(plugin,pluginData)
            suiteSize = numel(pluginData.TestSuite);
            fprintf("## Running a total of %d tests\n\n",suiteSize)
            plugin.FinalizedNumPassingAssertions = 0;
            plugin.FinalizedNumFailingAssertions = 0;

            runSession@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData)

            fprintf("## Done running tests\n")
            plugin.printAssertionSummary
        end

        function fixture = createSharedTestFixture(plugin,pluginData)
            fixture = createSharedTestFixture@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);

            fixture.addlistener("AssertionPassed", ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            fixture.addlistener("AssertionFailed", ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function testCase = createTestClassInstance(plugin,pluginData)
            testCase = createTestClassInstance@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);

            testCase.addlistener("AssertionPassed", ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener("AssertionFailed", ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function testCase = createTestMethodInstance(plugin,pluginData)
            testCase = createTestMethodInstance@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData);

            testCase.addlistener("AssertionPassed", ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener("AssertionFailed", ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function runTestSuite(plugin,pluginData)
            suiteSize = numel(pluginData.TestSuite);
            groupNumber = pluginData.Group;
            fprintf("### Running a total of %d tests in group %d\n", ...
                suiteSize,groupNumber)
            plugin.NumPassingAssertions = 0;
            plugin.NumFailingAssertions = 0;

            runTestSuite@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData)

            assertionStruct = struct( ...
                "Passing",plugin.NumPassingAssertions, ...
                "Failing",plugin.NumFailingAssertions);
            plugin.storeIn(pluginData.CommunicationBuffer,assertionStruct)
        end

        function reportFinalizedSuite(plugin,pluginData)
            assertionStruct = plugin.retrieveFrom( ...
                pluginData.CommunicationBuffer, ...
                DefaultData=struct("Passing",0,"Failing",0));
            plugin.FinalizedNumPassingAssertions = ...
                plugin.FinalizedNumPassingAssertions + assertionStruct.Passing;
            plugin.FinalizedNumFailingAssertions = ...
                plugin.FinalizedNumFailingAssertions + assertionStruct.Failing;

            reportFinalizedSuite@ ...
                matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData)
        end
    end

    methods (Access = private)
        function incrementPassingAssertionsCount(plugin)
            plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1;
        end

        function incrementFailingAssertionsCount(plugin)
            plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1;
        end

        function printAssertionSummary(plugin)
            fprintf("%s\n",repmat('_',1,30))
            fprintf("Total Assertions: %d\n", ...
                plugin.FinalizedNumPassingAssertions + ...
                plugin.FinalizedNumFailingAssertions)
            fprintf("\t%d Passed, %d Failed\n", ...
                plugin.FinalizedNumPassingAssertions, ...
                plugin.FinalizedNumFailingAssertions)
        end
    end
end

Create Test Class

In a file named ExampleTest.m in your current folder, create the ExampleTest parameterized test class. This class results in 300 tests, 100 of which are assertions to compare pseudorandom integers between 1 and 10.

classdef ExampleTest < matlab.unittest.TestCase
    properties (TestParameter)
        num1 = repmat({@()randi(10)},1,10)
        num2 = repmat({@()randi(10)},1,10)
    end

    methods (Test)
        function testAssert(testCase,num1,num2)
            testCase.assertNotEqual(num1(),num2())
        end
        function testVerify(testCase,num1,num2)
            testCase.verifyNotEqual(num1(),num2())
        end
        function testAssume(testCase,num1,num2)
            testCase.assumeNotEqual(num1(),num2())
        end
    end
end

Add Plugin to Test Runner and Run Tests

Create a test suite from the ExampleTest class.

suite = testsuite("ExampleTest");

Create a test runner with no plugins. This code creates a silent runner that produces no output.

runner = testrunner("minimal");

You can now add any plugins you choose. Add an instance of the AssertionCountingPlugin class to the runner, and run the tests in parallel. (You can also run the same tests in serial mode if you invoke the run method on the runner.)

runner.addPlugin(AssertionCountingPlugin)
runner.runInParallel(suite);

## Running a total of 300 tests

Split tests into 18 groups and running them on 6 workers.
----------------
Finished Group 1
----------------
### Running a total of 20 tests in group 1

----------------
Finished Group 2
----------------
### Running a total of 20 tests in group 2

----------------
Finished Group 3
----------------
### Running a total of 19 tests in group 3

----------------
Finished Group 4
----------------
### Running a total of 19 tests in group 4

----------------
Finished Group 5
----------------
### Running a total of 18 tests in group 5

----------------
Finished Group 6
----------------
### Running a total of 18 tests in group 6

----------------
Finished Group 7
----------------
### Running a total of 18 tests in group 7

----------------
Finished Group 8
----------------
### Running a total of 17 tests in group 8

----------------
Finished Group 9
----------------
### Running a total of 17 tests in group 9

-----------------
Finished Group 10
-----------------
### Running a total of 17 tests in group 10

-----------------
Finished Group 11
-----------------
### Running a total of 16 tests in group 11

-----------------
Finished Group 12
-----------------
### Running a total of 16 tests in group 12

-----------------
Finished Group 13
-----------------
### Running a total of 15 tests in group 13

-----------------
Finished Group 14
-----------------
### Running a total of 15 tests in group 14

-----------------
Finished Group 16
-----------------
### Running a total of 14 tests in group 16

-----------------
Finished Group 15
-----------------
### Running a total of 15 tests in group 15

-----------------
Finished Group 17
-----------------
### Running a total of 14 tests in group 17

-----------------
Finished Group 18
-----------------
### Running a total of 12 tests in group 18

## Done running tests
______________________________
Total Assertions: 100
	88 Passed, 12 Failed