Simulate Construction Vehicles in Unreal Engine for Material Handling

Since R2024b

This example uses:

This example shows how to quickly prototype a Construction Site Scene with standard construction vehicles using ready-made assets. In this example, a backhoe loads bricks into a dump truck. This example uses these toolboxes:

Robotics System Toolbox™— Control the backhoe in Simulink® with rigidBodyTree object and Inverse Kinematics (IK) block.
Simulink 3D Animation™— Provide Unreal Engine® simulation environment.
Robotics System Toolbox Offroad Autonomy Library™— Add the backhoe and dump truck blocks from this support package in the Simulink and simulate in the Unreal Engine.

To use Simulation 3D Physics Dump Truck and Simulation 3D Physics Backhoe blocks, you must download and install the Robotics System Toolbox Offroad Autonomy Library support package from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

This example shows the basic construction scene assembly and simulation. To implement a more advance workflow, see:

Simulate and Control Vehicles in Unreal Engine

This example uses Simulink blocks from the Robotics System Toolbox Offroad Autonomy Library to simulate vehicles in a construction scene. The construction site scene consists of an offroad terrain and contains Unreal assets such as bricks, walls, buildings, and so on. Further, the dump truck and backhoe vehicles are placed side-by-side in the Unreal scene.

Simulation Flow

In this model, you use a scheduler to move the backhoe through four distinct stages:

Initially, the backhoe is in a home configuration, which means that the links, such as the front and rear buckets, are in their default pose.
Once you initiate the trajectory, the rear bucket moves to an open position in front of the bricks.
Next, the bucket moves to a close position, where it scoops the bricks.
Once the bucket picks up the bricks, it moves over the truck bed and deposits the bricks there.

Set Up Simulink Model

The Simulink model consists of these subsystems:

Construction Scene Configuration
Import and Control Construction Vehicles
Backhoe Rear Bucket Motion Scheduler

Construction Scene Configuration

Download Construction Site Scene

To begin, check the maps available in the server.

sim3d.maps.Map.server

             MapName                                        Description                              Version
    __________________________    _______________________________________________________________    _______

    "Suburban scene"              "a suburban area beyond the city's border"                           "1"  
    "Offroad pit mining scene"    "An open pit mine with haul roads and terraced levels"               "1"  
    "Construction site scene"     "A construction site scene with walls, buildings and materials"      "1"

Download the Construction site scene from the server.

sim3d.maps.Map.download("Construction site scene")

Map is successfully downloaded and is up-to-date

Check if the downloaded map is available in your local machine.

sim3d.maps.Map.local

             MapName                                       Description                              Version
    _________________________    _______________________________________________________________    _______

    "Construction site scene"    "A construction site scene with walls, buildings and materials"      "1"

Configure Construction Scene in Unreal Engine

Open the Simulink model.

open_system("ConstructionVehicleUnreal.slx");

Select the Unreal Construction Site Scene using Simulation 3D Scene Configuration block. Next, configure and position the bricks using the Simulation 3D Actor (Simulink 3D Animation) block. The setupworld.m script is called within the Simulation 3D Actor block, which simplifies the process of placing multiple bricks in various locations.

To use downloaded construction scene, the Scene source parameter is set to Unreal Executable and default Unreal executable VehicleSimulation.exe is selected under Project name. Further, Scene field is set to /MathWorksRobotics/Maps/ConstructionSite.

helperSetSceneConfigParam();

Generate the brick locations and use to add target bricks in the scene and target intermediate, pick-up joint configuration calculation.

targetBrickTranslation = helperGenerateTargetLocation();

Import and Control Construction Vehicles

The Simulation 3D Physics Dump Truck and Simulation 3D Physics Backhoe blocks are import the dump truck and backhoe Unreal assets into the construction scene. The SteeringAngle and Throttle inputs are zero as both vehicles remain stationary during simulation and the Simulink model only simulates the motion of the rear bucket of the backhoe. Further, control the backhoe's rear bucket by adjusting the RearBucketAngle, ArmAngle, BoomAngle, and SwingFrameAngle. The backhoe rigidBodyTree object under the backhoe task scheduler computes these parameter values.

Backhoe Rear Bucket Motion Scheduler

The backhoe asset rigidBodyTree controls the rear bucket joints of the backhoe.

RigidBodyTree for Backhoe asset

This model uses few robotics tools to determine poses for the rear bucket of the backhoe. To do that, you need a kinematic model of the backhoe. A rigidBodyTree object relates the joint angles to the poses of the backhoe's components, such as the front bucket, loader frame, swing frame, arm, boom, and rear bucket.

In this example, you import a rigidBodyTree object from a URDF using importrobot. The URDF model was created by exporting backhoe asset from an FBX file using Phobos, a third-party Blender plug-in.

In a rigidBodyTree object, the joint configuration defines joint angles, which specifies the positions of the different bodies in the robot model. In this example, you specify a rigidBodyTree object as an input to an inverse kinematics (IK) solver, which is a tool that accepts a target bucket pose and returns a joint configuration.

Load a precomputed rigid body tree representation of a backhoe.

load("backhoeRigidBodyTree.mat");

Visualize rigid body tree representation of a backhoe.

backhoeRBT.show();

Backhoe Joint Configuration Generation and Joint Angle Controller

The rear bucket motion consists of three states, which are decided based on ground truth target brick location, backhoe location, and truck bed location.

First the rear bucket goes to the intermediate joint configuration from the home joint configuration.
Then moves towards the target brick location with the rear bucket in an open position, so that it can easily collect the target bricks.
After that, the rear bucket closes, which collects a few bricks and moves towards the intermediate joint configuration to avoid a collision with the truck bed.
At the end, the rear bucket moves towards the target drop location, which is determined by the truck bed and backhoe locations.

The solver uses inverse kinematics to find backhoe joint angles from a target location for the rear bucket. The Inverse Kinematics block accepts the backhoe rigidbodytree object and returns the corresponding joint configuration. The intermediate joint configuration places the rear bucket above the target brick location at a higher height.

Similarly, the Target Pick-Up Joint Config Generator subsystem calculates the target pick-up joint angles such that the rear bucket touches the ground at a lower height. Further, the Target Drop Joint Config Generator subsystem calculates the target drop configuration with the truck bed location and backhoe translation.

These tasks details the workflow for picking and dropping the target bricks.

First, the backhoe's rear bucket moves from its starting position to an intermediate joint configuration.
Then, it heads towards the bricks with the bucket open.
The bucket then closes to picks the bricks.
Next, the bucket, now holding the bricks, moves back to the intermediate joint configuration.
After that, it moves over to the truck bed.
Finally, the bucket opens to release the bricks into the truck.

This model uses a simple approach where the system passes a few known ground-truth poses directly to a controller. A more advanced approach requires modeling the simulation world and using a planner to guarantee a collision-free and time-optimal trajectory. For more information on that approach, see the Plan Collision-Free Path for Excavator Arm in MATLAB with Lidar Data example.

Perform Simulation

To perform simulation and visualize the simulation, click Run.

This simulation example demonstrates the power of Unreal Engine combined with Simulink for creating realistic and efficient construction vehicle operations. By following the steps outlined here, you can simulate complex vehicle dynamics and operations, leading to improved design and planning in construction projects.