Semi-Structured Intelligent Bin Picking for UR5e Using YOLO and PCA-Based Object Detection

Open the Project

To get started, open the example live script and access the supporting files by either clicking Open Live Script in the documentation or using the openExample function.

openExample('urseries/SemiStructuredIntelligentBinPickingUR5eHardwareProjectExample');

Then, open the Simulink® project file.

prj = openProject('BinPickingUR5eHW/BinPickingApplicationWithUR5eHardware.prj');

Bin Picking Workflow

This diagram shows the operations involved in the one complete execution cycle for the cobot bin picking application.

In this example, we are targeting the semi-structured bin picking, which is a classification based on the bin complexity.

On a high level, this task can be divided into two major modules:

Vision Processing or Perception Module

This workflow can be further divided into two areas:

This diagram shows the steps involved in the object pose estimation using YOLOv4 and PCA-based algorithm.

The perception process involves two main steps:

Motion Planning Module

This workflow can be divided into two areas:

Using the Live Script in the Motion Planning folder available in the MATLAB project of this example, you generate a MEX function using C/C++ code generation for the motion planning of the bin picking application. The motion planning algorithm as explained in Motion Planning and RigidBodyTree Simulation of UR5e for Bin Picking Using manipulatorRRT Algorithm is used to create a MEX function.

Generating a MEX function using C/C++ code generation helps you to reduce the computation time, thereby reducing the pick and place cycle time.

For more information on MEX function creation for the manipulatorRRT algorithm-based planner, refer to Generate Code for Manipulator Motion Planning in Perceived Environment example.

Also, refer to this example to know more about how to generate MEX function to accelerate your MATLAB program execution.

Software Requirements

This example uses:

Hardware Requirements

Interface Used for Universal Robots UR5e

The functionality from Robotics System Toolbox Support Package for Universal Robots UR Series Manipulators is used for trajectory and joint control of the Universal Robots UR5e hardware. The support package offers urROSNode object and its associated functions to enable control over the ROS interface.

Interface Used for Robotiq Epick Suction Gripper

Most of the real-world cobot applications involve operations such as pick and place, dispensing, and so on, and this requires attaching and using any external end-effector tool according to the application. The urROSNode object offers one useful method called handBackControl which allows you to perform end-effector operations using standard URCaps along with the ROS external control.

Interface Used for Intel RealSense D415

For perception purposes, Intel RealSense D415 camera depth module is used in this example. The MATLAB connection is established over the ROS using the IntelRealSense ROS driver.

The dataset used for the training of the YOLOv4 deep learning network has been created using this sensor.

You can follow the detailed steps given for the installation of the required ROS drivers for the connection.

Physical Setup Used in This Example

For the demonstration of the intelligent bin picking workflow using the Universal Robots UR5e, this example uses a cuboid shape whiteboard eraser as the object and a rectangular bin.

The dimensions of the cuboid and bin are provided in the initializeParametersForBinPickingHardware.m script in the Initialize folder of the project.

RigidBodyTree and Actual Environment Setup

In this example, we will create a RigidBodyTree environment for motion planning.

This flowchart shows the complete workflow that uses the Live Scripts available in Perception, Motion Planning, and Integration folders of the MATLAB project.

Parameter Initialization

The initialization script initializeParametersForBinPickingHardware.m runs automatically when you open this MATLAB project. This script defines some of the important parameters used in the perception, motion planning, and integration project module. You can find this script inside the Initialize folder or you can run the below command to open the script. Change the parameters if you are running with some different setup accordingly.

open("initializeParametersForBinPickingHardware.m");

Open Perception Module Script

Run this command to open the script of the perception workflow. This script covers the full perception pipeline from training to the object pose estimation workflow.

Download the pre-trained YOLO v4 network and dataset

You need to download the dataset and pre-trained network from this link. We have used pre-trained network in this script for showing the perception pipeline.

Note: Downloading the content from this link is necessary to run this example as it includes the pre-tranined network and dataset for training the network.

open("DetectionAndPoseEstimationforCuboid.mlx");

Note: This script below demonstrates how to train a YOLOv4 object detector for identifying cuboid objects.

open("trainYoloV4ForCuboidObject.mlx");

Open Motion Planning Module Script

Run this command to open the script for the motion planning workflow. This script covers the RigidBodyTree simulation workflow and MEX function generation steps for the motion planning module.

open("BinPickingMotionPlanningMEXGeneration.mlx");

Open Integration Module Script

Run this command to open the main script of the integrated workflow. This script shows how to use perception and motion planning modules for creating a full bin picking application workflow using the Universal Robots UR5e hardware.

open("UR5eHardwareBinPickingApplicationIntegratedScript.mlx");