Automate Testing for Autonomous Emergency Braking

This example uses:

This example shows how to assess the functionality of an autonomous emergency braking (AEB) application by defining scenarios based on requirements and automating testing of components and the generated code for those components. The components include sensor fusion and tracking, decision logic, and controls. This example builds on the Autonomous Emergency Braking with Sensor Fusion example.

Introduction

An AEB system avoids or mitigates collisions, protecting both the ego vehicle and vulnerable road users, such as pedestrians and cyclists. The system typically includes sensor fusion and tracking, decision logic, and controls components. You can use system-level simulation to assess the functionality of the integrated components. In this example, you configure simulations to test scenarios based on system requirements. Automating these simulations enables regression testing to verify system-level functionality. For more information about how to design and simulate a system-level model for AEB, see Autonomous Emergency Braking with Sensor Fusion.

This example shows how to automate testing of the AEB model against multiple scenarios using Simulink® Test™. The scenarios are based on system-level requirements. In this example, you:

Review requirements — Explore the test scenarios and review the requirements that describe the test conditions.
Review the test bench model — Review a system-level autonomous emergency braking test bench model that contains metric assessments. These metric assessments integrate the test bench model with Simulink Test for automated testing.
Automate testing — Configure the Test Manager to simulate each test scenario, assess success criteria, and report the results. You can explore the results dynamically using the Test Manager, and export them to a PDF for external review.
Automate testing with generated code — Configure the sensor fusion, decision logic, and controls components to generate C++ code. Run automated tests on the generated code and verify behavior.
Automate testing in parallel — Increase overall execution speed for the tests by using parallel computing on a multicore computer.

Review Requirements

Requirements Toolbox™ enables you author, analyze, and manage requirements within Simulink. This example provides 26 test scenarios for evaluating the model. Each test scenario also defines high-level testing requirements.

To explore the test requirements and test bench model, load the autonomous emergency braking project. The TestAutomation folder contains the files for automating testing.

openProject("AutonomousEmergencyBraking");
open("AutonomousEmergencyBrakingTestRequirements.slreqx");

Alternatively, you can open the file from the Requirements tab of the Requirements Manager app in Simulink.

The panel displays information about the test scenarios and the test requirements in textual and graphical formats. To test the AEB system, use these scenarios:

scenario_01_AEB_Bicyclist_Longitudinal_25width — The ego vehicle collides with the cyclist in front of it. Before the collision, the cyclist and ego vehicle are traveling in the same direction along the longitudinal axis. At collision time, the bicycle is 25% of the way across the width of the ego vehicle.
scenario_02_AEB_Bicyclist_Longitudinal_50width — The ego vehicle collides with the cyclist in front of it. Before the collision, the cyclist and ego vehicle are traveling in the same direction along the longitudinal axis. At collision time, the bicycle is 50% of the way across the width of the ego vehicle.
scenario_03_AEB_Bicyclist_Longitudinal_75width — The ego vehicle collides with the cyclist in front of it. Before the collision, the cyclist and ego vehicle are traveling in the same direction along the longitudinal axis. At collision time, the bicycle is 75% of the way across the width of the ego vehicle.
scenario_04_AEB_CCRb_2_initialGap_12m — A car-to-car rear braking (CCRb) scenario, where the ego vehicle rear-ends a braking vehicle. The braking vehicle begins to decelerate at a rate of 2 m/s^2. The initial gap between the ego vehicle and the braking vehicle is 12 m.
scenario_05_AEB_CCRb_2_initialGap_40m — A car-to-car rear braking (CCRb) scenario, where the ego vehicle rear-ends a braking vehicle. The braking vehicle begins to decelerate at a rate of 2 m/s^2. The initial gap between the ego vehicle and the braking vehicle is 40 m.
scenario_06_AEB_CCRb_6_initialGap_12m — A car-to-car rear braking (CCRb) scenario, where the ego vehicle rear-ends a braking vehicle. The braking vehicle begins to decelerate at a rate of 6 m/s^2. The initial gap between the ego vehicle and the braking vehicle is 12 m.
scenario_07_AEB_CCRb_6_initialGap_40m — A car-to-car rear braking (CCRb) scenario, where the ego vehicle rear-ends a braking vehicle. The braking vehicle begins to decelerate at a rate of 6 m/s^2. The initial gap between the ego vehicle and the braking vehicle is 40 m.
scenario_08_AEB_CCRm_50overlap — A car-to-car rear moving (CCRm) scenario, where the ego vehicle rear-ends a moving vehicle. At collision time, the ego vehicle overlaps with 50% of the width of the moving vehicle.
scenario_09_AEB_CCRm__50overlap — A car-to-car rear moving (CCRm) scenario, where the ego vehicle rear-ends a moving vehicle. At collision time, the ego vehicle overlaps with –50% of the width of the moving vehicle. When the ego vehicle is to the left of the other vehicle, the percent overlap is negative.
scenario_10_AEB_CCRm_75overlap — A car-to-car rear moving (CCRm) scenario, where the ego vehicle rear-ends a moving vehicle. At collision time, the ego vehicle overlaps with 75% of the width of the moving vehicle.
scenario_11_AEB_CCRm__75overlap — A car-to-car rear moving (CCRm) scenario, where the ego vehicle rear-ends a moving vehicle. At collision time, the ego vehicle overlaps with –75% of the width of the moving vehicle.
scenario_12_AEB_CCRm_100overlap — A car-to-car rear moving (CCRm) scenario, where the ego vehicle rear-ends a moving vehicle. At collision time, the ego vehicle overlaps with 100% of the width of the moving vehicle.
scenario_13_AEB_CCRs_50overlap — A car-to-car rear stationary (CCRs) scenario, where the ego vehicle rear-ends a stationary vehicle. At collision time, the ego vehicle overlaps with 50% of the width of the stationary vehicle.
scenario_14_AEB_CCRs__50overlap — A car-to-car rear stationary (CCRs) scenario, where the ego vehicle rear-ends a stationary vehicle. At collision time, the ego vehicle overlaps with –50% of the width of the stationary vehicle.
scenario_15_AEB_CCRs_75overlap — A car-to-car rear stationary (CCRs) scenario, where the ego vehicle rear-ends a stationary vehicle. At collision time, the ego vehicle overlaps with 75% of the width of the stationary vehicle.
scenario_16_AEB_CCRs__75overlap — A car-to-car rear stationary (CCRs) scenario, where the ego vehicle rear-ends a stationary vehicle. At collision time, the ego vehicle overlaps with –75% of the width of the stationary vehicle. When the ego vehicle is to the left of the other vehicle, the percent overlap is negative.
scenario_17_AEB_CCRs_100overlap — A car-to-car rear stationary (CCRs) scenario, where the ego vehicle rear-ends a stationary vehicle. At collision time, the ego vehicle overlaps with 100% of the width of the stationary vehicle.
scenario_18_AEB_Pedestrian_Farside_50width — The ego vehicle collides with a pedestrian who is traveling from the left side of the road, which Euro NCAP test protocols refer to as the far side. These protocols assume that vehicles travel on the right side of the road. Therefore, the left side of the road is the side farthest from the ego vehicle. At collision time, the pedestrian is 50% of the way across the width of the ego vehicle.
scenario_19_AEB_Pedestrian_Longitudinal_25width — The ego vehicle collides with a pedestrian who is traveling in the direction of ego vehicle on the road. At collision time, the pedestrian is 25% of the way across the width of the ego vehicle.
scenario_20_AEB_Pedestrian_Longitudinal_50width — The ego vehicle collides with a pedestrian who is traveling in the direction of ego vehicle on the road. At collision time, the pedestrian is 50% of the way across the width of the ego vehicle.
scenario_21_AEB_Pedestrian_Nearside_25width — The ego vehicle collides with a pedestrian who is traveling from the right side of the road, which Euro NCAP test protocols refer to as the near side. At collision time, the pedestrian is 25% of the way across the width of the ego vehicle. These protocols assume that vehicles travel on the right side of the road. Therefore, the right side of the road is the side nearest to the ego vehicle.
scenario_22_AEB_Pedestrian_Nearside_75width — The ego vehicle collides with a pedestrian who is traveling from the right side of the road, which Euro NCAP test protocols refer to as the near side. At collision time, the pedestrian is 75% of the way across the width of the ego vehicle
scenario_23_AEB_PedestrianChild_Nearside_50width — The ego vehicle collides with a pedestrian child who is traveling from the right side of the road, which Euro NCAP test protocols refer to as the near side. At collision time, the pedestrian child is 50% of the way across the width of the ego vehicle.
scenario_24_AEB_PedestrianTurning_Farside_10kph — The ego vehicle turns at an intersection and collides with a pedestrian who is traveling parallel with the left side, or far side, of the vehicle at the start of the simulation.
scenario_25_AEB_PedestrianTurning_Nearside_10kph — The ego vehicle turns at an intersection and collides with a pedestrian who is traveling parallel with the right side, or near side, of the vehicle at the start of the simulation.
scenario_26_AEB_CCFtap_VUT_10kph_GVT_30kph — The ego vehicle, while making a turn at an intersection, collides with a target vehicle traveling at a constant speed.

These requirements are implemented as test scenarios with the same names as the scenarios used in the AEBTestBench model.

Review Test Bench Model

This example reuses the AEBTestBench model from the Autonomous Emergency Braking with Sensor Fusion example. Open the test bench model.

open_system("AEBTestBench");

This test bench model has these components:

Sensors and Environment
Sensor Fusion and Tracking
AEB Decision Logic
AEB Controller
Vehicle Dynamics

To configure the test bench model, use the helperSLAEBSetup function. Specify a test scenario as input to the setup function by using the scenarioFcnName input argument. The value for scenarioFcnName must be one of the scenario names specified in the test requirements.

To reduce Command Window output, turn off the MPC update messages.

mpcverbosity("off");

Run the setup function.

helperSLAEBSetup(scenarioFcnName="scenario_25_AEB_PedestrianTurning_Nearside_10kph");

You can now simulate the model and visualize the results. For more details on the analysis of the simulation results, see the Autonomous Emergency Braking with Sensor Fusion example.

In this example, the focus is automating the simulation runs for this test bench model using Simulink Test for the different test scenarios. The Metrics Assessment subsystem enables integration of system-level metric evaluations with Simulink Test. This subsystem uses Check Static Lower Bound (Simulink) blocks for this integration. Open the Metrics Assessment subsystem.

open_system("AEBTestBench/Metrics Assessment");

This example uses these metrics

Check Safety — Verifies that the ego vehicle meets the safetyGoal by reducing its velocity to mitigate the collision risk. Use the helperSLAEBSetup script to specify the safetyGoal parameter.

Check Collision — Verifies that the ego vehicle does not collide with the other actors at any point during the simulation.

Automate Testing

The Test Manager has been configured to automate testing of the AEB application. Open the AutonomousEmergencyBrakingTests.mldatx test file in the Test Manager.

sltestmgr;
testFile = sltest.testmanager.load("AutonomousEmergencyBrakingTests.mldatx");

The test cases in the Test Manager are linked to the test requirements in the Requirements Editor. Each test case uses the POST-LOAD callback to run the setup function with appropriate inputs. After simulating the test case, it invokes helperPlotAEBResults from the CLEANUP callback to generate the plots explained in the Autonomous Emergency Braking with Sensor Fusion example.

Run and Explore Results for a Single Test Scenario

To test the system-level model with the scenario_25_AEB_PedestrianTurning_Nearside_10kph test scenario from Simulink Test, use this code:

testSuite = getTestSuiteByName(testFile,"Test Scenarios");
testCase = getTestCaseByName(testSuite,"scenario_25_AEB_PedestrianTurning_Nearside_10kph");
resultObj = run(testCase);

To generate a report after the simulation, use this code:

sltest.testmanager.report(resultObj,"Report.pdf", ...
    Title="Autonomous Emergency Braking", ...
    IncludeMATLABFigures=true, ...
    IncludeErrorMessages=true, ...
    IncludeTestResults=0, ...
    LaunchReport=true);

Examine Report.pdf. Observe that the Test environment section shows the platform on which the test is run and the MATLAB version used for testing. The Summary section shows the outcome of the test and duration of the simulation in seconds. The Results section shows pass or fail results based on the assessment criteria. This section also shows the plots logged from the helperPlotAEBResults function.

Run and Explore Results for All Test Scenarios

Run a simulation of the system for all the tests by entering run(testFile). Alternatively, you can simulate the system by clicking Run in the Test Manager app.

View the results in the Results and Artifacts pane of the Test Manager. For each test case, the Check Static Lower Bound (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass or fail results.

You can find the generated report in your current working directory. This report contains a detailed summary of pass or fail statuses and plots for each test case.

Verify Test Status in Requirements Editor

Open the Requirements Editor and select Columns. Then, select Verification Status to see a verification status summary for each requirement. Green and red bars indicate the pass or fail status, respectively, for each simulation test result.

Automate Testing with Generated Code

The AEBTestBench model enables integrated testing of the Sensor Fusion and Tracking, AEB Decision Logic, and AEB Controller components. Using this example, you can also perform regression testing of these components through software-in-the-loop (SIL) verification. If you have licenses for Embedded Coder™ and Simulink Coder™, then you can generate code for these components. This workflow enables you to verify that the generated code produces expected results that match the system-level requirements throughout the simulation.

Set Sensor Fusion and Tracking to run in SIL mode.

model = "AEBTestBench/Sensor Fusion and Tracking";
set_param(model,SimulationMode="Software-in-the-loop");

Set AEB Decision Logic to run in SIL mode.

model = "AEBTestBench/AEB Decision Logic";
set_param(model,SimulationMode="Software-in-the-loop");

Set AEB Controller to run in SIL mode.

model = "AEBTestBench/AEB Controller";
set_param(model,SimulationMode="Software-in-the-loop");

Simulate the system for all test scenarios using the run(testFile) command. After completing the tests, review the plots and results in the generated report.

Enable MPC update messages.

mpcverbosity("on");

Automate Testing in Parallel

If you have a Parallel Computing Toolbox™ license, then you can configure Test Manager to execute tests in parallel using a parallel pool. Test Manager uses the default Parallel Computing Toolbox cluster and executes tests on only the local machine. Running tests in parallel can increase execution speed and decrease the amount of time it takes to get test results. For more information on how to configure tests in parallel from the Test Manager, see Run Tests Using Parallel Execution (Simulink Test).