Automate Testing for Highway Lane Following

Introduction

A highway lane-following system steers a vehicle to travel within a marked lane. It also maintains a set velocity or safe distance from a preceding vehicle in the same lane. The system typically includes lane detection, sensor fusion, decision logic, and controls components. System-level simulation is a common technique for assessing functionality of the integrated components. Simulations are configured to test scenarios based on system requirements. Automatically running these simulations enables regression testing to verify system-level functionality.

The Highway Lane Following example showed how to simulate a system-level model for lane-following. This example shows how to automate testing that model against multiple scenarios using Simulink Test™. The scenarios are based on system-level requirements. In this example, you will:

Testing the system-level model requires a photorealistic simulation environment. In this example, you enable system-level simulation through integration with the Unreal Engine from Epic Games®. This example requires a Windows® 64-bit platform.

if ~ispc
    error(['This example is supported only on Microsoft', char(174),' Windows 64-bit platform.'])
end

To ensure reproducibility of the simulation results, set the random seed.

rng(0);

Review Requirements

Requirements Toolbox™ lets you author, analyze, and manage requirements within Simulink. This example contains ten test scenarios, with high-level testing requirements defined for each scenario. Open the requirement set.

To explore the test bench model, load the highway lane following project.

openProject("HighwayLaneFollowing");

open('HighwayLaneFollowingTestRequirements.slreqx')

Alternatively, you can also open the file from the Requirements tab of the Requirements Manager app in Simulink.

Each row in this file specifies the requirements in textual and graphical formats for testing the lane-following system for a test scenario. The scenarios with the scenario_LF_ prefix enable you to test lane-detection and lane-following algorithms without obstruction by other vehicles. The scenarios with the scenario_LFACC_ prefix enable you to test lane-detection, lane-following, and ACC behavior with other vehicles on the road.

These requirements are implemented as test scenarios with the same names as the scenarios used in the HighwayLaneFollowingTestBench model.

Review Test Bench Model

This example reuses the HighwayLaneFollowingTestBench model from the Highway Lane Following example. Open the test bench model.

open_system("HighwayLaneFollowingTestBench");

This test bench model has Simulation 3D Scenario, Lane Marker Detector, Vehicle Detector, Forward Vehicle Sensor Fusion, Lane Following Decision Logic and Lane Following Controller and Vehicle Dynamics components.

This test bench model is configured using the helperSLHighwayLaneFollowingSetup script. This setup script takes scenarioName as input. scenarioName can be any one of the previously described test scenarios. To run the setup script, use code:

scenarioName = "scenario_LFACC_03_Curve_StopnGo";
helperSLHighwayLaneFollowingSetup("scenarioFcnName",scenarioName);

You can now simulate the model and visualize the results. For more details on the analysis of the simulation results and the design of individual components in the test bench model, see the Highway Lane Following example.

In this example, the focus is more on automating the simulation runs for this test bench model using Simulink Test for the different test scenarios. The Metrics Assessment subsystem enables integration of system-level metric evaluations with Simulink Test. Open the Metrics Assessment subsystem.

open_system("HighwayLaneFollowingTestBench/Metrics Assessment");

Using this example, you can evaluate the system-level behavior using four system-level metrics. Additionally, you can also compute component-level metrics to analyze individual components and their impact on the overall system performance.

System-Level Metrics

Component-Level Metrics

Disable Runtime Visualizations

The system-level test bench model visualizes intermediate outputs during the simulation for the analysis of different components in the model. These visualizations are not required when the tests are automated. You can reduce execution time for the automated testing by disabling them.

Disable runtime visualizations for the Lane Marker Detector subsystem.

load_system('LaneMarkerDetector');
blk = 'LaneMarkerDetector/Lane Marker Detector';
set_param(blk,'EnableDisplays','off');

Disable runtime visualizations for the Vehicle Detector subsystem.

load_system('VisionVehicleDetector');
blk = 'VisionVehicleDetector/Pack Detections/Pack Vehicle Detections';
set_param(blk,'EnableDisplay','off');

Configure the Simulation 3D Scene Configuration block to run the Unreal Engine in headless mode, where the 3D simulation window is disabled.

blk = ['HighwayLaneFollowingTestBench/Simulation 3D Scenario/', ...
      'Simulation 3D Scene Configuration'];
set_param(blk,'EnableWindow','off');

Automate Testing

The Test Manager is configured to automate the testing of the lane-following application. Open the HighwayLaneFollowingTestAssessments.mldatx test file in the Test Manager.

sltestmgr;
testFile = sltest.testmanager.load('HighwayLaneFollowingTestAssessments.mldatx');

Observe the populated test cases that were authored previously in this file. Each test case is linked to the corresponding requirement in the Requirements Editor for traceability. Each test case uses the POST-LOAD callback to run the setup script with appropriate inputs and to configure the output video filename. After the simulation of the test case, it invokes helperTMTestCasePostProcessing from the CLEAN-UP callback to assess performances of the overall system and individual components by generating the plots explained in the Highway Lane Following example.

After simulation of the test case, Simulink Test also invokes these functions from the CUSTOM CRITERIA callback to get additional metrics for lane marker detector and vehicle detector components:

Run and explore results for a single test scenario:

To reduce command-window output, turn off the MPC update messages.

mpcverbosity('off');

To test the system-level model with the scenario_LFACC_03_Curve_StopnGo test scenario from Simulink Test, use this code:

testSuite = getTestSuiteByName(testFile,'Test Scenarios');
testCase = getTestCaseByName(testSuite,'scenario_LFACC_03_Curve_StopnGo');
resultObj = run(testCase);

To generate a report after the simulation, use this code:

sltest.testmanager.report(resultObj,'Report.pdf',...,
    'Title','Highway Lane Following',...
    'IncludeMATLABFigures',true,...
    'IncludeErrorMessages',true,...
    'IncludeTestResults',0,'LaunchReport',true);

Examine the report Report.pdf. Observe that the Test environment section shows the platform on which the test is run and the MATLAB® version used for testing. The Summary section shows the outcome of the test and duration of the simulation in seconds. The Results section shows pass/fail results based on the assessment criteria. This section also shows the plots logged from the helperGenerateFilesForLaneFollowingReport function.

Run and explore results for all test scenarios:

You can simulate the system for all the tests by using sltest.testmanager.run. Alternatively, you can simulate the system by clicking Play in the Test Manager app.

After completion of the test simulations, the results for all the tests can be viewed in the Results and Artifacts tab of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass/fail results.

You can find the generated report in current working directory. This report contains a detailed summary of pass/fail statuses and plots for each test case.

Verify test status in Requirements Editor:

Open the Requirements Editor and select Display. Then, select Verification Status to see a verification status summary for each requirement. Green and red bars indicate the pass/fail status of simulation results for each test.

Automate Testing with Generated Code

The HighwayLaneFollowingTestBench model enables integrated testing of Lane Marker Detector, Vehicle Detector, Forward Vehicle Sensor Fusion, Lane Following Decision Logic, and Lane Following Controller components. It is often helpful to perform regression testing of these components through software-in-the-loop (SIL) verification. If you have Embedded Coder™ Simulink Coder™ license, then you can generate code for these components. This workflow lets you verify that the generated code produces expected results that match the system-level requirements throughout simulation.

Set Lane Marker Detector to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Lane Marker Detector';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Vehicle Detector to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Vehicle Detector';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Forward Vehicle Sensor Fusion to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Forward Vehicle Sensor Fusion';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Lane Following Decision Logic to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Lane Following Decision Logic';
set_param(model,'SimulationMode','Software-in-the-loop');

Set Lane Following Controller to run in Software-in-the-loop mode.

model = 'HighwayLaneFollowingTestBench/Lane Following Controller';
set_param(model,'SimulationMode','Software-in-the-loop');

Now, run sltest.testmanager.run to simulate the system for all the test scenarios. After the completion of tests, review the plots and results in the generated report.

Enable the MPC update messages again.

mpcverbosity('on');

Automate Testing in Parallel

If you have a Parallel Computing Toolbox™ license, then you can configure Test Manager to execute tests in parallel using a parallel pool. To run tests in parallel, save the models after disabling the runtime visualizations using save_system('LaneMarkerDetector'), save_system('VisionVehicleDetector') and save_system('HighwayLaneFollowingTestBench'). Test Manager uses the default Parallel Computing Toolbox cluster and executes tests only on the local machine. Running tests in parallel can speed up execution and decrease the amount of time it takes to get test results. For more information on how to configure tests in parallel from the Test Manager, see Run Tests Using Parallel Execution (Simulink Test).