Simulation 3D Robot

System Overview

The core environment has five components that form the assembly line:

This system is created using CAD data provided by ITQ GmbH. MATLAB® has the imported CAD files, which are preprocessed using Simulink 3D Animation™ functions and then saved to a MAT file for reuse.

During normal execution, Robot 1 picks up cups from a tray in the Robot 1 workcell and places them in an empty shuttle that is waiting for the cup. Placing the cup in the shuttle triggers the filled shuttle to move along the shuttle track to reach Robot 2 and create an instance of the ball in the Robot 2 workcell. The location of this ball in Robot 2 workcell is random. A camera positioned above the area where the ball appears in the Robot 2 workcell captures the image of the ball and communicates it to the Deep Learning network. The network uses this information to detect the position of the ball for Robot 2 to pick it up. Once the shuttle containing the cup stops at Robot 2 workcell, Robot 2 picks up the ball and places it inside the cup. Then the shuttle moves to the unloading station, where it releases the cup onto the slider. The cup then slides into a container placed at the end of the slider. After releasing the cup onto the slider, the shuttle moves back to the Robot 1 workcell and waits to receive another cup.

This example makes use of variant subsystems to provide modularity. One such variant is used for sensing, and has two settings:

Model Overview

This example uses the assembly line CAD data, robot model files and shuttle trajectory data. Download and unzip these data files.

downloadFolder = matlab.internal.examples.downloadSupportFile ... 
    ("R2023a/sl3d/examples/","smart4i_data.zip");
unzip(downloadFolder);

Open the model to view the block diagram:

smart4iModel = "smart4i_model.slx";
open_system(smart4iModel);

Version = 
"Simulation3DRobot"

SensorVersion = 
"ManualDetection"

Warning: Unrecognized function or variable 'CloneDetectionUI.internal.CloneDetectionPerspective.register'.

The model has four main parts, which are detailed below:

Scene Creation & Configuration

This section has two main functions.

In this example, the scene was prepared using these tools:

System Control Logic

The Stateflow chart in the System Control Logic section determines the main system behavior.

Inside the Stateflow chart, there are three main sections, listed from bottom to top:

The overall Stateflow chart has a number of outputs:

While all the system logic is in the Stateflow chart, the actual control of all actors -- the two robots, the shuttles, cups, and balls -- happens outside the chart in the Actor Motion Control section.

Ball Detection

The Ball Detection subsystem identifies the location of the ball to be placed. When the sensor variant is set to manual mode, this value is read by simply querying the ID of the ball actor and reading the pose. However, when the variant is set to Camera-based detection, a Simulation 3D Camera Get block is used to view the ball and identify its position.

The ball position in the Robot 2 workcell is the same but with a small random offset to reflect the real-life variability caused by the ball rolling. The findBallLoc triggered subsystem takes the image data from the camera to detect the ball.

In this subsystem, the Deep Learning Object Detector block from the Deep Learning Toolbox™ takes the image data and outputs the bounding boxes. The bounding boxes provide information about the size and location of the ball in the image. The findBallXY MATLAB function converts the bounding boxes to the XY-positions of the ball and then returns those positions to the Stateflow chart for Robot 2.

The Deep Learning Object Detector was pretrained in MATLAB by creating a video of the ball at random locations and using the Video Labeler app from the Computer Vision Toolbox to create a labeled dataset and train a yolov2ObjectDetector object.

Actor Control

To move the objects in the scene, the system translates motion commands to the desired behavior. The Actor Control section defines the motion of the two robots, the shuttles, and the existence and initial position of the ball. There are two methods for settings robot poses, which can be switched with radio button as mentioned in the Select Variant.

The robot simulation in the Simulink 3D can be performed with Simulation 3D Actor block as well as with Simulation 3D Robot block. The respective block connections are activated based on variant selection from radio button. Once variant is selected, you can update model to see activated connection.

Use a Simulation 3D Robot Block

The Simulink 3D Robot block is designed for position robotic actors. It offers improved performance over a set of transform blocks and allows the user control of proximity-based grasp and release actions. The Robot 1 and Robot 2 subsystem performs robot motion as well as grasp-release target in the Simulink 3D world.

Use a Simulink 3D Actor Block Together with Get Transform Blocks

This mechanism of control allows more complete control, enabling the user to freely position any of the robot bodies regardless of kinematic constraints. However, the complete subsystem requires additional blocks to complete the same joint position task, resulting in a subsystem that is less performant than the Simulink 3D Robot block. Each robot body corresponds to an actor in the Unreal Engine environment. Robot 1 and Robot 2 subsystems control the motion of Robot 1 and Robot 2 respectively, to position the robot actors such that the robots reach the specified poses. This diagram shows the contents of each subsystem:

These steps rely on kinematic computations on the robot which requires a kinematic model of the robot in MATLAB using the rigidBodyTree object. During scene creation, which happens in a model callback, the system imports the rigid body trees for Robot 1 and Robot 2 from their original URDF files to the base MATLAB workspace. Additionally, in the same callback, joint limits derived from the specifications for each robot are applied. This ensures that the robot's feasible joint positions will be incorporated during the inverse kinematics phase of trajectory generation.

show(Robot1);
title("Robot 1 (Comau Racer V3)");

Inspect each robot to see the bodies it contains. These bodies correspond to the actors in the scene. In the Unreal Engine scene, each actor pose is defined relative to the parent body.

showdetails(Robot1)

--------------------
Robot: (6 bodies)

 Idx     Body Name     Joint Name     Joint Type     Parent Name(Idx)   Children Name(s)
 ---     ---------     ----------     ----------     ----------------   ----------------
   1        part_1        joint_1       revolute         base_link(0)   part_2(2)  
   2        part_2        joint_2       revolute            part_1(1)   part_3(3)  
   3        part_3        joint_3       revolute            part_2(2)   part_4(4)  
   4        part_4        joint_4       revolute            part_3(3)   part_5(5)  
   5        part_5        joint_5       revolute            part_4(4)   tool(6)  
   6          tool        joint_6       revolute            part_5(5)   
--------------------

Given the robot models, it is possible to translate a suction gripper pose to actor motion in two mains steps, as specified by the two areas in the model:

Note that the suction gripper does not act on the cup or ball in any way. Instead, cups and balls are picked by reparenting them to the suction gripper, effectively adding the picked part to the robot hierarchy. The supervisory logic in the Stateflow chart sets the events that trigger the Simulink functions calling the attach and detach functions.

Ball Instance

When the Stateflow chart triggers the system to create a new ball, the Simulation 3D Actor block named Ball Instance commands the creation of a new ball for Robot 2 to pick up. The block adds a random offset to the position of the ball to simulate real-world conditions caused by the ball rolling, but the detection subsystem ensures that the control logic gets the exact ball position.

Shuttles

The translation and orientation of the shuttle is set in space to make it move around the track. The Stateflow chart outputs XY-position and orientation about the Z-axis to the Shuttles subsystem to set the poses of the appropriate actors in the Unreal Engine world.

Simulate the Model

Open the model and click Run to start the simulation.

open_system(smart4iModel);

The simulation takes around 30-45 seconds to start in 3D viewer owing to a large number of actors in the scene.

Model Overview

Load the model with the following command:

open_system("SL3DJointSpaceManipulatorTrajectory")

This example uses a Kinova Gen3 manipulator, which is stored in the model workspace. However, load and visualize the robot with the following commands:

gen3 = loadrobot("kinovaGen3","DataFormat","column");
show(gen3);

The model is split into two sections:

Manipulator Trajectory Tracking

The Polynomial Trajectory block generates continuous joint-space trajectories from random sets of waypoints in the range [-0.375*pi 0.375*pi], stopping at each of the waypoints. The Joint-Space Motion Model block simulates the closed-loop tracking of these trajectories for a Kinova Gen3 manipulator with computed-torque control.

Visualization with Simulation 3D Robot block in Simulink 3D™

The Simulation 3D Robot block inserts the manipulator into the Simulink 3D world defined by the RigidBodyTree object input. The Get Transform block is used to get the position of the end effector, which is then converted from a homogeoneous transform matrix to a translation vector.

Simulate the Model

In the model, pacing is active, as indicated by the clock symbol below the run button:

This ensures that the model is slowed down to near real-time speed, so that the visualization can be updated at a realistic pace.

Trajectory Visualization

By default, the model opens the scopes that display velocity and position information. However, if they are closed, the scopes can be opened by double-clicking the associated viewer icons:

Visualize robot motion in Simulink 3D world.

The scopes show the tracking results of the Joint Space Motion Model block. As can be seen on the left in the figures below, the initial configuration of the robot differs from the reference trajectories, but the controlled motion ensures that the trajectory is reached and tracked for the duration of the simulation. The final scope displays the X, Y, and Z position of the end effector in the world frame.