TAP results do not show up during execution time, if a TestClassSetup is present

Question

Christian Heigele 2018-8-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/413885-tap-results-do-not-show-up-during-execution-time-if-a-testclasssetup-is-present

回答： Andy Campbell 2018-8-9

Hi there

I have a problem. We use the Matlab testing framework to analyze our codebase. To track the results in our CI system TeamCity we use the TAP-format. Here we have the following problem:

If a test includes a TestClassSetup section, the TAP results show up only at the end, and not during the exection. This results in a few issues for us: - Timestamps might not be correct - If informative output is given per test-case, it is not shown together with the assertion statment.

We use the following (simplified) snippet to identify out TestSuite and execute it:

 testSuite = matlab.unittest.TestSuite.fromFolder('.');
 runner = matlab.unittest.TestRunner.withNoPlugins();
 runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat());
 results = runner.run(testSuite);

With the following two classes the issue is reproducible (the content is of course made up & meaningless...):

 classdef SomeTest < matlab.unittest.TestCase
    properties (TestParameter)
        param = {1, 2};
        param2 = {1, 2};
    end        
    methods (TestClassSetup)
        function someSetup(testCase)
            pause(0.1);
        end
    end    
    methods (Test)
        function testMethod(self, param, param2)
           fprintf('I''m here, with the params: %f/%f\n', param, param2);
           pause(0.1);
           self.assertGreaterThan(param, param2); 
        end
    end    
 end
 classdef SomeOtherTest < matlab.unittest.TestCase
    properties (TestParameter)
        param = {1, 2};
        param2 = {1, 2};
    end    
    methods (Test)
        function testMethod(self, param, param2)
           fprintf('I''m here, with the params: %f/%f\n', param, param2);
           pause(0.1);
           self.assertGreaterThan(param, param2); 
        end
    end
 end

If you copy all three files into one folder, and execute the runner, you'll see the output (assertions are simplified):

 1..8
 I'm here, with the params: 1.000000/1.000000
 not ok 1 - SomeOtherTest/testMethod(param=1,param2=1)
 # ================================================================================
 # Assertion failed in SomeOtherTest/testMethod(param=1,param2=1) and it did not run to completion.
 # ================================================================================
 # 
 I'm here, with the params: 1.000000/2.000000
 not ok 2 - SomeOtherTest/testMethod(param=1,param2=2)
 # ================================================================================
 # Assertion failed in SomeOtherTest/testMethod(param=1,param2=2) and it did not run to completion.
 # ================================================================================
 # 
 I'm here, with the params: 2.000000/1.000000
 ok 3 - SomeOtherTest/testMethod(param=2,param2=1)
 I'm here, with the params: 2.000000/2.000000
 not ok 4 - SomeOtherTest/testMethod(param=2,param2=2)
 # ================================================================================
 # Assertion failed in SomeOtherTest/testMethod(param=2,param2=2) and it did not run to completion.
 # ================================================================================
 # 
 I'm here, with the params: 1.000000/1.000000
 I'm here, with the params: 1.000000/2.000000
 I'm here, with the params: 2.000000/1.000000
 I'm here, with the params: 2.000000/2.000000
 not ok 5 - SomeTest/testMethod(param=1,param2=1)
 # ================================================================================
 # Assertion failed in SomeTest/testMethod(param=1,param2=1) and it did not run to completion.
 # ================================================================================
 # 
 not ok 6 - SomeTest/testMethod(param=1,param2=2)
 # ================================================================================
 # Assertion failed in SomeTest/testMethod(param=1,param2=2) and it did not run to completion.
 # ================================================================================
 # 
 ok 7 - SomeTest/testMethod(param=2,param2=1)
 not ok 8 - SomeTest/testMethod(param=2,param2=2)
 # ================================================================================
 # Assertion failed in SomeTest/testMethod(param=2,param2=2) and it did not run to completion.
 # ================================================================================