Dynamically Filtered Tests

Assumption failures produce filtered tests. In the matlab.unittest.TestResult class, such a test is marked Incomplete.

Since filtering test content through the use of assumptions does not produce test failures, it has the possibility of creating dead test code. Avoiding this requires monitoring of filtered tests.

Test Methods

If an assumption failure is encountered inside of a TestCase method with the Test attribute, the entire method is marked as filtered, but MATLAB^® runs the subsequent Test methods.

The following class contains an assumption failure in one of the methods in the Test block.

classdef ExampleTest < matlab.unittest.TestCase
    methods(Test)
        function testA(testCase)
            testCase.verifyTrue(true)
        end
        function testB(testCase)
            testCase.assumeEqual(0,1)
            % remaining test code is not exercised
        end
        function testC(testCase)
            testCase.verifyFalse(true)
        end
    end
end

Since the testB method contains an assumption failure, when you run the test, the testing framework filters that test and marks it as incomplete. After the assumption failure in testB, the testing framework proceeds and executes testC, which contains a verification failure.

ts = matlab.unittest.TestSuite.fromClass(?ExampleTest);
res = ts.run;

Running ExampleTest
.
================================================================================
ExampleTest/testB was filtered.
Details
================================================================================
.
================================================================================
Verification failed in ExampleTest/testC.

    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyFalse failed.
    --> The value must evaluate to "false".
    
    Actual logical:
           1

    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testC) at 11
================================================================================
.
Done ExampleTest
__________

Failure Summary:

     Name               Failed  Incomplete  Reason(s)
    ================================================================
     ExampleTest/testB              X       Filtered by assumption.
    ----------------------------------------------------------------
     ExampleTest/testC    X                 Failed by verification.

If you examine the TestResult, you notice that there is a passed test, a failed test, and a test that did not complete due to an assumption failure.

res

res = 

  1×3 TestResult array with properties:

    Name
    Passed
    Failed
    Incomplete
    Duration
    Details

Totals:
   1 Passed, 1 Failed, 1 Incomplete.
   2.4807 seconds testing time.

The testing framework keeps track of incomplete tests so that you can monitor filtered tests for nonexercised test code. You can see information about these tests within the TestResult object.

res([res.Incomplete])

ans = 

  TestResult with properties:

          Name: 'ExampleTest/testB'
        Passed: 0
        Failed: 0
    Incomplete: 1
      Duration: 2.2578
       Details: [1×1 struct]

Totals:
   0 Passed, 0 Failed, 1 Incomplete.
   2.2578 seconds testing time.

To create a modified test suite from only the filtered tests, select incomplete tests from the original test suite.

tsFiltered = ts([res.Incomplete])

tsFiltered = 

  Test with properties:
                  Name: 'ExampleTest/testB'
         ProcedureName: 'testB'
             TestClass: "ExampleTest"
            BaseFolder: 'C:\work'
      Parameterization: [0×0 matlab.unittest.parameters.EmptyParameter]
    SharedTestFixtures: [0×0 matlab.unittest.fixtures.EmptyFixture]
                  Tags: {1×0 cell}

Tests Include:
   0 Parameterizations, 0 Shared Test Fixture Classes, 0 Tags.

Method Setup and Teardown Code

If an assumption failure is encountered inside a TestCase method with the TestMethodSetup attribute, MATLAB filters the method which was to be run for that instance. If a test uses assumptions from within the TestMethodSetup block, consider instead using the assumptions in the TestClassSetup block, which likewise filters all Test methods in the class but is less verbose and more efficient.

One of the methods in the following TestMethodSetup block within ExampleTest.m contains an assumption failure.

    methods(TestMethodSetup)
        function setupMethod1(testCase)
            testCase.assumeEqual(1,0)
            % remaining test code is not exercised
        end
        function setupMethod2(testCase)
            disp('* Running setupMethod2 *')
            testCase.assertEqual(1,1)
        end
    end

Updated ExampleTest Class Definition

When you run the test, you see that the framework completes executes all the methods in the TestMethodSetup block that do not contain the assumption failure, and it marks as incomplete all methods in the Test block.

ts = matlab.unittest.TestSuite.fromClass(?ExampleTest);
res = ts.run;

Running ExampleTest

================================================================================
ExampleTest/testA was filtered.
    Details
================================================================================
* Running setupMethod2 *
.
================================================================================
ExampleTest/testB was filtered.
    Details
================================================================================
* Running setupMethod2 *
.
================================================================================
ExampleTest/testC was filtered.
    Details
================================================================================
* Running setupMethod2 *
.
Done ExampleTest
__________

Failure Summary:

     Name               Failed  Incomplete  Reason(s)
    ================================================================
     ExampleTest/testA              X       Filtered by assumption.
    ----------------------------------------------------------------
     ExampleTest/testB              X       Filtered by assumption.
    ----------------------------------------------------------------
     ExampleTest/testC              X       Filtered by assumption.

The Test methods did not change but all 3 are filtered due to an assumption failure in the TestMethodSetup block. The testing framework executes methods in the TestMethodSetup block without assumption failures, such as setupMethod2. As expected, the testing framework executes setupMethod2 3 times, once before each Test method.

Class Setup and Teardown Code

If an assumption failure is encountered inside of a TestCase method with the TestClassSetup or TestClassTeardown attribute, MATLAB filters the entire TestCase class.

The methods in the following TestClassSetup block within ExampleTest.m contains an assumption failure.

    methods(TestClassSetup)
        function setupClass(testCase)
            testCase.assumeEqual(1,0)
            % remaining test code is not exercised
        end
    end

Updated ExampleTest Class Definition

When you run the test, you see that the framework does not execute any of the methods in the TestMethodSetup or Test.

ts = matlab.unittest.TestSuite.fromClass(?ExampleTest);
res = ts.run;

Running ExampleTest

================================================================================
All tests in ExampleTest were filtered.
    Details
================================================================================

Done ExampleTest
__________

Failure Summary:

     Name               Failed  Incomplete  Reason(s)
    ================================================================
     ExampleTest/testA              X       Filtered by assumption.
    ----------------------------------------------------------------
     ExampleTest/testB              X       Filtered by assumption.
    ----------------------------------------------------------------
     ExampleTest/testC              X       Filtered by assumption.

The Test and TestMethodSetup methods did not change but everything is filtered due to an assumption failure in the TestClassSetup block.