Main Content

matlab.unittest.plugins.StopOnFailuresPlugin Class

Namespace: matlab.unittest.plugins
Superclasses: matlab.unittest.plugins.TestRunnerPlugin

Plugin to debug test failures

expand all in page

Description

The matlab.unittest.plugins.StopOnFailuresPlugin class defines a plugin for debugging test failures. When a test runner with a StopOnFailuresPlugin encounters a qualification failure or uncaught error, it pauses test execution and puts MATLAB® into debug mode. You can then use MATLAB debugging commands, such as dbstep, dbcont, and dbquit, to investigate the cause of the test failure or error.

The matlab.unittest.plugins.StopOnFailuresPlugin class is a handle class.

Creation

Description

p = matlab.unittest.plugins.StopOnFailuresPlugin constructs a plugin to debug test failures.

example

p = matlab.unittest.plugins.StopOnFailuresPlugin("IncludingAssumptionFailures",tf) sets the IncludeAssumptionFailures property to tf. Use this syntax to indicate whether to react to assumption failures. By default, StopOnFailuresPlugin reacts only to uncaught errors, verification failures, assertion failures, and fatal assertion failures. However, when IncludingAssumptionFailures is specified as true, the plugin reacts also to assumption failures.

example

Properties

expand all

Whether to react to assumption failures, specified as a numeric or logical 0 (false) or 1 (true) and stored as a logical value. If the value is true, then the plugin reacts to assumption failures. If the value is false, then the plugin ignores assumption failures.

Attributes:

GetAccess
public
SetAccess
private

Examples

collapse all

Investigate the cause of test failures by adding a StopOnFailuresPlugin instance to the test runner.

In your current folder, create the ExampleTest test class.

classdef ExampleTest < matlab.unittest.TestCase
    methods (Test)
        function testOne(testCase)  % Test fails
            act = 3.1416;
            exp = pi;
            testCase.verifyEqual(act,exp)
        end
        function testTwo(testCase)  % Test does not complete
            testCase.assumeEqual(5,4)
        end
    end
end

At the command prompt, create a test suite from ExampleTest and run the tests. As a result of the qualifications in the test class, the first test fails, and the second test does not complete.

import matlab.unittest.plugins.StopOnFailuresPlugin
suite = testsuite("ExampleTest");
runner = testrunner("textoutput");  
results = runner.run(suite);
Running ExampleTest

================================================================================
Verification failed in ExampleTest/testOne.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqual failed.
    --> The numeric values are not equal using "isequaln".
    --> Failure table:
            Actual        Expected               Error               RelativeError    
            ______    ________________    ____________________    ____________________
        
            3.1416    3.14159265358979    7.34641020683213e-06    2.33843499679617e-06
    
    Actual Value:
       3.141600000000000
    Expected Value:
       3.141592653589793
    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testOne) at 6
================================================================================
.
================================================================================
ExampleTest/testTwo was filtered.
================================================================================
.
Done ExampleTest
__________

Failure Summary:

     Name                 Failed  Incomplete  Reason(s)
    ==================================================================
     ExampleTest/testOne    X                 Failed by verification.
    ------------------------------------------------------------------
     ExampleTest/testTwo              X       Filtered by assumption.

Add a StopOnFailuresPlugin instance to the test runner and rerun the tests. During the test run, when the failure occurs, MATLAB enters debug mode at the source of the failure.

runner.addPlugin(StopOnFailuresPlugin)
result = runner.run(suite);
Running ExampleTest

================================================================================
Verification failed in ExampleTest/testOne.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqual failed.
    --> The numeric values are not equal using "isequaln".
    --> Failure table:
            Actual        Expected               Error               RelativeError    
            ______    ________________    ____________________    ____________________
        
            3.1416    3.14159265358979    7.34641020683213e-06    2.33843499679617e-06
    
    Actual Value:
       3.141600000000000
    Expected Value:
       3.141592653589793
    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testOne) at 6
================================================================================
Test execution paused due to failure. To terminate the test run, use dbquit. To continue, use dbcont.

Investigate the cause of the test failure by using the whos function to examine the variables in the workspace.

whos
  Name          Size            Bytes  Class          Attributes

  act           1x1                 8  double                   
  exp           1x1                 8  double                   
  testCase      1x1                 8  ExampleTest

Try rerunning the failed test with a relative tolerance of 100*eps. The test fails even with the specified tolerance.

testCase.verifyEqual(act,exp,"RelTol",100*eps)
================================================================================
Verification failed in ExampleTest/testOne.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyEqual failed.
    --> The numeric values are not equal using "isequaln".
    --> The error was not within relative tolerance.
    --> Failure table:
            Actual        Expected               Error               RelativeError         RelativeTolerance  
            ______    ________________    ____________________    ____________________    ____________________
        
            3.1416    3.14159265358979    7.34641020683213e-06    2.33843499679617e-06    2.22044604925031e-14
    
    Actual Value:
       3.141600000000000
    Expected Value:
       3.141592653589793
    ------------------
    Stack Information:
    ------------------
    In C:\work\ExampleTest.m (ExampleTest.testOne) at 6
================================================================================

Try rerunning the failed test with an absolute tolerance of 0.001. The failed test passes with the absolute tolerance.

testCase.verifyEqual(act,exp,"AbsTol",0.001)

Use dbquit to end the test run. You also can use dbcont to quit debug mode and run the rest of the tests.

dbquit

To enter debug mode for tests that fail by assumption, such as testTwo in the ExampleTest class, specify IncludingAssumptionFailures as true when you create the plugin. (Make sure to add only one StopOnFailuresPlugin instance to the test runner. Adding more than one StopOnFailuresPlugin instance can result in unexpected behavior.)

When you run the tests, MATLAB enters debug mode in both testOne and testTwo. 

runner.addPlugin(StopOnFailuresPlugin( ...
    "IncludingAssumptionFailures",true))
result = runner.run(suite);

Debug an error in your tests using a StopOnFailuresPlugin instance.

In your current folder, create the ExampleTest test class. Inject an error into your tests by including an invalid function call in the testTwo method.

classdef ExampleTest < matlab.unittest.TestCase
    methods (Test)
        function testOne(testCase)    % Test passes
            act = round(pi);
            exp = 3;
            testCase.verifyEqual(act,exp)
        end
        function testTwo(testCase)    % Test throws an error
            act = cosine(0);    % Invalid function call
            exp = 1;
            testCase.verifyEqual(act,exp)
        end
    end
end

At the command prompt, create a test suite from ExampleTest and run the tests. As a result of the error, the second test fails and does not complete.

import matlab.unittest.plugins.StopOnFailuresPlugin
suite = testsuite("ExampleTest");
runner = testrunner("textoutput");  
results = runner.run(suite);
Running ExampleTest
.
================================================================================
Error occurred in ExampleTest/testTwo and it did not run to completion.
    ---------
    Error ID:
    ---------
    'MATLAB:UndefinedFunction'
    --------------
    Error Details:
    --------------
    Undefined function 'cosine' for input arguments of type 'double'.
    
    Error in ExampleTest/testTwo (line 9)
                act = cosine(0);    % Invalid function call
================================================================================
.
Done ExampleTest
__________

Failure Summary:

     Name                 Failed  Incomplete  Reason(s)
    ====================================================
     ExampleTest/testTwo    X         X       Errored.

Add a StopOnFailuresPlugin instance to the runner and rerun the tests. During the test run, when the error is thrown, MATLAB enters debug mode at the source of the error.

runner.addPlugin(StopOnFailuresPlugin)
result = runner.run(suite);
Running ExampleTest
.9               act = cosine(0);    % Invalid function call
K>>

Fix the error by using the correct function name (that is, cos). When you rerun the tests, both of them pass.

Version History

Introduced in R2013b

expand all

See Also

| |