Unit tests fail on "even" runs, but pass on "odd" runs (alternating pass/fail)
4 次查看(过去 30 天)
显示 更早的评论
I have some unit tests that I wrote for code that is doing some very minor image manipulation (combining several small images into a larger image). When I first run the unit tests, I noticed that three out of four of them failed on the line where they are reading an image from a directory (fails with an index out of bounds error). However, if I run it a second time, they all pass. When I was writing the code as well, I noticed that whenever I set a breakpoint in my code, I'd have to run the unit tests twice because the first time it would run through the tests without hitting any breakpoints.
I have my repo organized like this
src/
/* source code .m files are in here */
unit_tests/
images/
squares/
- img1.png
- img2.png
...
- imgn.png
- unit_tests.m
And I have a line in my setup (inside the unit tests) to add paths that I generate:
function tests = unit_tests()
addpath(genpath('..'));
tests = functiontests(localfunctions);
end
At first, I thought that perhaps the unit tests were executing too quickly and it was running the unit tests before the addpath/genpath functions had finished, so I added a pause statement and re-ran the tests, but I had the same issue only this time it would wait for the requisite number of seconds before going ahead and failing. If I run it again, no problem - all my tests pass.
Here's the output of running the unit tests twice (I started on an "odd" run):
>> runtests('unit_tests\unit_tests.m')
Running unit_tests
.......
Done unit_tests
__________
ans =
1×7 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Details
Totals:
7 Passed, 0 Failed, 0 Incomplete.
0.39979 seconds testing time.
>> runtests('unit_tests')
Running unit_tests
..
================================================================================
Error occurred in unit_tests/testCompositeImage_2x2 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_2x2 (line 47)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_2x3 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_2x3 (line 71)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
.
================================================================================
Error occurred in unit_tests/testCompositeImage_3x2 and it did not run to completion.
---------
Error ID:
---------
'MATLAB:badsubscript'
--------------
Error Details:
--------------
Index exceeds array bounds.
Error in unit_tests>testCompositeImage_3x2 (line 95)
img = imread([squares(1).folder filesep squares(1).name]); % all same size
================================================================================
...
Done unit_tests
__________
Failure Summary:
Name Failed Incomplete Reason(s)
==================================================================
unit_tests/testCompositeImage_2x2 X X Errored.
------------------------------------------------------------------
unit_tests/testCompositeImage_2x3 X X Errored.
------------------------------------------------------------------
unit_tests/testCompositeImage_3x2 X X Errored.
ans =
1×7 TestResult array with properties:
Name
Passed
Failed
Incomplete
Duration
Details
Totals:
4 Passed, 3 Failed (rerun), 3 Incomplete.
0.0040965 seconds testing time.
The fact that it's failing on basically the first line because it's failing to read anything from the folder leads me to suspect that even when the other 4 tests supposedly pass, they are in fact not running at all. I am completely at a loss as to why this is happening; I am using vanilla matlab (R2018a) and don't have anything fancy going on.
2 个评论
Geoff Hayes
2020-4-23
matquest - why (in the above output) do you call
runtests('unit_tests\unit_tests.m')
first and then on the second run call as
runtests('unit_tests')
? How is this input used by the runtests function?
回答(3 个)
Steven Lord
2020-4-24
You add a directory (or multiple directories) to your path but you never restore the path to the state it was in prior to the test running. Since your test is a function-based test you should create setup and teardown functions. The setup function will run before your test does to store the path and add directories to it. The teardown function will run after your test does and restore the path to its original value afterwards. See this documentation page for a description of how to do this.
When or if you decide to convert your tests (or write new tests) as class-based tests you could use a fixture to handle the setup and teardown once it has been applied (with applyFixture) to your test object.
5 个评论
Sean de Wolski
2020-4-25
I'm almost certainly willing to bet it's a path issue. I'm also almost certainly willing to bet that using a project to manage the path will fix it.
Sean de Wolski
2020-4-24
You should convert your code files and tests into a MATLAB Project which manages the path for you. Then as long as the project is open (and there's a ProjectFixture in the test framework to ensure this) then everything you need is on the path. Additionally, projects make it easy to run tests because you can run all tests in a project and the test framework makes sure everything is set up.
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Results, Reporting, and Test File Management 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!