Unit tests fail on "even" runs, but pass on "odd" runs (alternating pass/fail)

Question

matquest 2020-4-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/520011-unit-tests-fail-on-even-runs-but-pass-on-odd-runs-alternating-pass-fail

评论： matquest 2020-4-27

I have some unit tests that I wrote for code that is doing some very minor image manipulation (combining several small images into a larger image). When I first run the unit tests, I noticed that three out of four of them failed on the line where they are reading an image from a directory (fails with an index out of bounds error). However, if I run it a second time, they all pass. When I was writing the code as well, I noticed that whenever I set a breakpoint in my code, I'd have to run the unit tests twice because the first time it would run through the tests without hitting any breakpoints.

I have my repo organized like this

src/
    /* source code .m files are in here */
    
unit_tests/
    images/
        squares/
            - img1.png
            - img2.png
            ...
            - imgn.png
    - unit_tests.m

And I have a line in my setup (inside the unit tests) to add paths that I generate:

function tests = unit_tests()
    addpath(genpath('..'));    
    tests = functiontests(localfunctions);
end

At first, I thought that perhaps the unit tests were executing too quickly and it was running the unit tests before the addpath/genpath functions had finished, so I added a pause statement and re-ran the tests, but I had the same issue only this time it would wait for the requisite number of seconds before going ahead and failing. If I run it again, no problem - all my tests pass.

Here's the output of running the unit tests twice (I started on an "odd" run):

>> runtests('unit_tests\unit_tests.m')
Running unit_tests
.......
Done unit_tests
__________
ans = 
  1×7 TestResult array with properties:
    Name
    Passed
    Failed
    Incomplete
    Duration
    Details
Totals:
   7 Passed, 0 Failed, 0 Incomplete.
   0.39979 seconds testing time.
>> runtests('unit_tests')
Running unit_tests
..
================================================================================
Error occurred in unit_tests/testCompositeImage_2x2 and it did not run to completion.
    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'
    --------------
    Error Details:
    --------------
    Index exceeds array bounds.
    
    Error in unit_tests>testCompositeImage_2x2 (line 47)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_2x3 and it did not run to completion.
    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'
    --------------
    Error Details:
    --------------
    Index exceeds array bounds.
    
    Error in unit_tests>testCompositeImage_2x3 (line 71)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_3x2 and it did not run to completion.
    ---------
    Error ID:
    ---------
    'MATLAB:badsubscript'
    --------------
    Error Details:
    --------------
    Index exceeds array bounds.
    
    Error in unit_tests>testCompositeImage_3x2 (line 95)
        img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
...
Done unit_tests
__________
Failure Summary:
     Name                               Failed  Incomplete  Reason(s)
    ==================================================================
     unit_tests/testCompositeImage_2x2    X         X       Errored.
    ------------------------------------------------------------------
     unit_tests/testCompositeImage_2x3    X         X       Errored.
    ------------------------------------------------------------------
     unit_tests/testCompositeImage_3x2    X         X       Errored.
    
ans = 
  1×7 TestResult array with properties:
    Name
    Passed
    Failed
    Incomplete
    Duration
    Details
Totals:
   4 Passed, 3 Failed (rerun), 3 Incomplete.
   0.0040965 seconds testing time.

The fact that it's failing on basically the first line because it's failing to read anything from the folder leads me to suspect that even when the other 4 tests supposedly pass, they are in fact not running at all. I am completely at a loss as to why this is happening; I am using vanilla matlab (R2018a) and don't have anything fancy going on.

2 个评论
显示无隐藏无

Geoff Hayes 2020-4-23

在 MATLAB Online 中打开

matquest - why (in the above output) do you call

runtests('unit_tests\unit_tests.m')

first and then on the second run call as

runtests('unit_tests')

? How is this input used by the runtests function?

matquest 2020-4-23

编辑：matquest 2020-4-23

在 MATLAB Online 中打开

I actually just noticed this and it appears to explain the bizarre output. I am just pressing the "Run Tests" button in the matlab editor tab. I have no idea why, but when I press it, it alternates between running the command:

runtests('unit_tests\unit_tests.m')

and

runtests('unit_tests')

Do you have any ideas as to why it might be doing that?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

matquest 2020-4-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/520011-unit-tests-fail-on-even-runs-but-pass-on-odd-runs-alternating-pass-fail#answer_428185

编辑：matquest 2020-4-24

在 MATLAB Online 中打开

Leaving this here in case someone else runs into the same issue.

Turns out something about the line:

addpath(genpath('..'));

causes the matlab GUI to go into a weird state where pressing the "Run Tests" button alternates between calling runtests('unit_tests\unit_tests.m'); and runtests('unit_tests'); which in turn cases the tests to alternately pass and fail (run/not run). If anyone knows why this is the case or has any insights into using relative paths in matlab, I would be very interested to learn more.

Replacing that line with the following seems to resolve the issue:

pth = fullfile(fileparts(fileparts(mfilename('fullpath'))),'src'); % goes up one dir and into 'src' => ../src
paths = regexp(genpath(pth), ';', 'split');
for idx = 1:length(paths) - 1 % last element is empty
    addpath(paths{idx});
end

Restart matlab and the changes should take effect.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Steven Lord 2020-4-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/520011-unit-tests-fail-on-even-runs-but-pass-on-odd-runs-alternating-pass-fail#answer_428207

You add a directory (or multiple directories) to your path but you never restore the path to the state it was in prior to the test running. Since your test is a function-based test you should create setup and teardown functions. The setup function will run before your test does to store the path and add directories to it. The teardown function will run after your test does and restore the path to its original value afterwards. See this documentation page for a description of how to do this.

When or if you decide to convert your tests (or write new tests) as class-based tests you could use a fixture to handle the setup and teardown once it has been applied (with applyFixture) to your test object.

5 个评论
显示 3更早的评论隐藏 3更早的评论

matquest 2020-4-24

编辑：matquest 2020-4-24

I see. I added a setupOnce function which only includes that call and the tests now run correctly. I did see in the documentation an example for changing directories where the original path is stored in a temp variable and then restored during teardown and I think I will do this for the path.

However, I am still curious as to why this happens. I naively assumed that any changes made within a function would be cleaned up automatically as soon as the function returns. Could you elaborate on how this happens? It appears (based on running some tests and saving the value in path) that path is a global variable whose value is preserved between calls/runs of the unit tests. Which lead me to look into what happens duing the genpath function (as it seemed that it should return the same set of paths between each run). But when I stepped into the genpath function during an erroneous call, I can see that it in fact goes up two directories (equivalent to '../..') and thus give the path some erroneous paths (and yet I cannot see why this should be a problem as the path variable still contains the desired directories - namely, src and unit_tests/images/squares/). This appears to happen in the dir() function, which seems to be a compiled/encrypted file. Could you describe what happens in the dir() function?

Also of note: this behavior does not occur if I run the tests either from outside of the repository or from within the unit_tests directory itself.

Sean de Wolski 2020-4-25

I'm almost certainly willing to bet it's a path issue. I'm also almost certainly willing to bet that using a project to manage the path will fix it.

matquest 2020-4-27

@Sean de Wolski - Projects appear to have been added in R2019a based on the release notes, so I'm not sure if that's an option for me.

请先登录，再进行评论。

Answer 3

Sean de Wolski 2020-4-24

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/520011-unit-tests-fail-on-even-runs-but-pass-on-odd-runs-alternating-pass-fail#answer_428210

You should convert your code files and tests into a MATLAB Project which manages the path for you. Then as long as the project is open (and there's a ProjectFixture in the test framework to ensure this) then everything you need is on the path. Additionally, projects make it easy to run tests because you can run all tests in a project and the test framework makes sure everything is set up.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Unit tests fail on "even" runs, but pass on "odd" runs (alternating pass/fail)

2 个评论
显示无隐藏无

回答（3 个）

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

5 个评论
显示 3更早的评论隐藏 3更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Unit tests fail on "even" runs, but pass on "odd" runs (alternating pass/fail)

2 个评论 显示 无隐藏 无

回答（3 个）

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

5 个评论 显示 3更早的评论隐藏 3更早的评论

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

2 个评论
显示无隐藏无

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

5 个评论
显示 3更早的评论隐藏 3更早的评论

0 个评论
显示 -2更早的评论隐藏 -2更早的评论