Main Content

testrunner

Create test runner

Since R2021a

collapse all in page

Syntax

runner = testrunner
runner = testrunner("minimal")
runner = testrunner("textoutput")
runner = testrunner(___,Name=Value)

Description

runner = testrunner creates a default test runner, which is similar to the runner that the testing framework configures by default when you call the runtests function.

The testrunner function returns a matlab.unittest.TestRunner object. You can call the methods on the returned object to run and operate on your test suite and to customize running tests. For example, to run a suite of tests, use run(runner,suite).

example

runner = testrunner("minimal") creates a minimal runner with no plugins installed. The returned test runner is the simplest runner possible and produces no text output. Use this syntax when you want to have complete control over which plugins to add to the runner.

example

runner = testrunner("textoutput") creates a runner that is configured for text output. The output produced includes test progress as well as diagnostics in the event of test failures.

This syntax creates runners that tend to run tests more quickly, because the testing framework does not record diagnostics on test results produced by a nondefault runner. For more information, see Programmatically Access Test Diagnostics.

example

runner = testrunner(___,Name=Value) specifies options using one or more name-value arguments in addition to any of the input argument combinations in previous syntaxes. For example, runner = testrunner("textoutput",OutputDetail="verbose") creates a test runner that displays test output detail at the maximum verbosity level. (since R2024b)

Examples

collapse all

Open Script

Run a suite of tests with a default runner and access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,"Hello World!","llo")
end

Create a test suite from the tests in sampleTest.m. Then, create a default runner and run the tests.

suite = testsuite('SampleTest');
runner = testrunner;
results = run(runner,suite);

Running sampleTest
.
================================================================================
Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --> The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\work\sampleTest.m (testB) at 10
================================================================================
..
Done sampleTest
__________

Failure Summary:

     Name              Failed  Incomplete  Reason(s)
    ===============================================================
     sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

results(2)

ans = 

  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 0.3962
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed, 0 Incomplete.
   0.39619 seconds testing time.

When you run tests with a default runner, the testing framework uses a DiagnosticsRecordingPlugin instance to record diagnostics on test results. Access the recorded diagnostics for the second test using the DiagnosticRecord field in the Details property on the TestResult object.

records = results(2).Details.DiagnosticRecord

records = 

  QualificationDiagnosticRecord with properties:

                          Event: 'VerificationFailed'
                     EventScope: TestMethod
                  EventLocation: 'sampleTest/testB'
          TestDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
     FrameworkDiagnosticResults: [1×1 matlab.automation.diagnostics.DiagnosticResult]
    AdditionalDiagnosticResults: [1×0 matlab.automation.diagnostics.DiagnosticResult]
                          Stack: [1×1 struct]
                         Report: 'Verification failed in sampleTest/testB.↵    ---------------------↵    Framework Diagnostic:↵    ---------------------↵    verifyGreaterThan failed.↵    --> The value must be greater than the minimum value.↵    ↵    Actual Value:↵        13↵    Minimum Value (Exclusive):↵        42↵    ------------------↵    Stack Information:↵    ------------------↵    In C:\work\sampleTest.m (testB) at 10'
Open Live Script

Generate JUnit-style test results by creating a minimal runner and then adding an XMLPlugin instance to the runner.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,'hello, world','llo')
end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a test runner with no plugins. This code creates a silent runner that produces no output. You can now install whatever plugins you like.

runner = testrunner('minimal');

Create an XMLPlugin instance that writes JUnit-style XML output to the file myTestResults.xml.

import matlab.unittest.plugins.XMLPlugin
xmlFile = 'myTestResults.xml';
p = XMLPlugin.producingJUnitFormat(xmlFile);

Add the plugin to the test runner and run the tests.

addPlugin(runner,p)
results = run(runner,suite);

Display the results from the second test.

results(2)
ans = 
  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 0.0723
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed (rerun), 0 Incomplete.
   0.0723 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field. 

records = results(2).Details
records = struct with no fields.

Now, view the contents of the generated artifact.

disp(fileread(xmlFile))
<?xml version="1.0" encoding="UTF-8" standalone="no" ?>
<testsuites>

  <testsuite errors="0" failures="1" name="sampleTest" skipped="0" tests="3" time="0.083731">
    <testcase classname="sampleTest" name="testA" time="0.0085045"/>
    <testcase classname="sampleTest" name="testB" time="0.0723">
      <failure type="VerificationFailure">Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --&gt; The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\TEMP\Examples\matlab-ex97531283\sampleTest.m (testB) at 10</failure>
    </testcase>
    <testcase classname="sampleTest" name="testC" time="0.0029273"/>
  </testsuite>

</testsuites>
Open Live Script

Run a suite of tests with a runner that is configured for text output, and then access the results.

Create a function-based test sampleTest.m in your current folder. The file contains two tests that pass and one test that intentionally fails.

function tests = sampleTest
tests = functiontests(localfunctions);
end

function testA(testCase)      % Test passes
verifyEqual(testCase,2+3,5)
end

function testB(testCase)      % Test fails
verifyGreaterThan(testCase,13,42)
end

function testC(testCase)      % Test passes
verifySubstring(testCase,'hello, world','llo')
end

Create a test suite from the tests in sampleTest.m.

suite = testsuite('sampleTest');

Create a runner that produces text output and use it to run the tests.

runner = testrunner('textoutput');
results = run(runner,suite);
Running sampleTest
.
================================================================================
Verification failed in sampleTest/testB.
    ---------------------
    Framework Diagnostic:
    ---------------------
    verifyGreaterThan failed.
    --> The value must be greater than the minimum value.
    
    Actual Value:
        13
    Minimum Value (Exclusive):
        42
    ------------------
    Stack Information:
    ------------------
    In C:\TEMP\Examples\matlab-ex48684143\sampleTest.m (testB) at 10
================================================================================
..
Done sampleTest
__________

Failure Summary:

     Name              Failed  Incomplete  Reason(s)
    ===============================================================
     sampleTest/testB    X                 Failed by verification.

Display the results from the second test.

results(2)
ans = 
  TestResult with properties:

          Name: 'sampleTest/testB'
        Passed: 0
        Failed: 1
    Incomplete: 0
      Duration: 1.9894
       Details: [1×1 struct]

Totals:
   0 Passed, 1 Failed (rerun), 0 Incomplete.
   1.9894 seconds testing time.

Check for diagnostics recorded on test results. If you were using a default runner, there would be a DiagnosticRecord field at this location. But because you are using a nondefault runner, the framework does not create such a field. 

records = results(2).Details
records = struct with no fields.

Input Arguments

collapse all

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: runner = testrunner("textoutput",OutputDetail="verbose",LoggingLevel="detailed")

Display level of test output, specified as an integer scalar from 0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration.

Numeric RepresentationEnumeration Member NameVerbosity Description
0None

No information

1Terse

Minimal information

2Concise

Moderate amount of information

3Detailed

Some supplemental information

4Verbose

Lots of supplemental information

By default, the test runner displays failing and logged events at the matlab.automation.Verbosity.Detailed level (level 3) and test run progress at the matlab.automation.Verbosity.Concise level (level 2).

Example: runner = testrunner("textoutput",OutputDetail="verbose") creates a test runner that displays test output at the matlab.automation.Verbosity.Verbose level.

Verbosity level of logged diagnostics, specified as an integer scalar from 0 through 4, a matlab.automation.Verbosity enumeration object, or a text representation of the enumeration. The test runner includes diagnostics logged at the specified level and below.

Numeric RepresentationEnumeration Member NameVerbosity Description
0None

No information

1Terse

Minimal information

2Concise

Moderate amount of information

3Detailed

Some supplemental information

4Verbose

Lots of supplemental information

By default, the test runner includes diagnostics logged at the matlab.automation.Verbosity.Terse level (level 1). To exclude logged diagnostics, specify LoggingLevel as matlab.automation.Verbosity.None (level 0).

Logged diagnostics are diagnostics that you supply to the unit testing framework with the log (TestCase) and log (Fixture) methods.

Example: runner = testrunner("textoutput",LoggingLevel="detailed") creates a test runner that includes diagnostics logged at the matlab.automation.Verbosity.Detailed level and below.

Tips

Version History

Introduced in R2021a

expand all

See Also

Apps

Functions

Classes

Topics