Main Content

Label Objects Using Polygons

The Image Labeler, Video Labeler, and Ground Truth Labeler (Automated Driving Toolbox) apps enable you to label images using a variety of interactive drawing tools. Use these labels to create ground truth data for training algorithms.

About Polygon Labels

You can draw polygon shapes around objects in your image or video frame, and use the polygons to label the objects. You can also associate polygon labels with one another, enabling you to label distinct objects as instances of the same class. Exported polygon labels preserve the stack order of the polygons across label definitions and are compatible with the ground truth format required by:

  • Semantic segmentation networks using an H-by-W label matrix. Using polygons maintains the order in which the labels are created or reordered (using the Send to Back and Bring to Front options). Semantic segmentation requires a pixel map of the image with labels. To do this, you must flatten the labels, (losing the layered order of the labels). Imagine turning the opacity of all the polygon labels to maximum and preserving all of the filled areas as a label matrix.

  • Instance segmentation networks using an H-by-W-by-NumObjects mask stack.

  • Raw polygons in a NumObjects-by-1 cell array, where each cell contains M-by-2 points.

In this example, you load an image, create polygon ROI labels, modify preferences related to drawing polygons, and postprocess the exported ground truth object data to use for instance segmentation and semantic segmentation training networks.

Load Unlabeled Data

You can load images stored in a datastore, from a folder, or load a previous labeler session. The images must be readable by imread. In this example, you are loading data that contains a number of vehicles, including boats, a tanker, and a plane.

Load Data Programmatically

Use this process to programmatically load images from a datastore:

  1. Create a datastore from a folder of images. For example, if you had a folder named "stopSignImages" that contained images, you could create a datastore of the images with this code.

    imageFolder = fullfile(toolboxdir("vision"),"visiondata","stopSignImages");
imds = imageDatastore(imageFolder);

  2. Load the datastore into the Image Labeler:

    imageLabeler(imds);

Alternatively, you can load the folder of images directly into Image Labeler:

imageFolder = fullfile(toolboxdir("vision"),"visiondata","stopSignImages");
imageLabeler(imageFolder);

Load Data Directly from the App

Click Import to load images or click Open Session to open a saved session.

Create Polygon Labels

To follow these steps that use an image of sailboats, a tanker, and an airplane, open the Image Labeler app and then import the "boats.png" image from the vision/visiondata folder.

  1. In the ROI Labels pane on the left, click Label.

  2. Select the Polygon label type and name it Sailboat.

  3. Optionally, you can change the label color by clicking the preview color.

  4. Click OK.

    The Sailboat label appears in the ROI Labels pane.

  5. Repeat steps 1 through 4 to create a Tanker label and a Airplane label. You can move a label in the list by left-clicking and dragging the label up or down.

Draw Polygon ROI Labels

  1. Select the Sailboat label, and then using the mouse, draw a polygon ROI around each of the sailboats.

  2. Select the Tanker label and draw a polygon ROI around the tanker ship.

  3. Select the Airplane label and draw a polygon ROI around the airplane.

    Sailboats, tanker, and airplane labeled with polygons

Modify Polygon Preferences and Stacking Order

There are a number of ways to modify the appearance of the ROI labels. You can also set the stacking order for the exported ground truth file. The stacking order indicates whether an object is in front of or behind another object, and is required in order to resolve overlapping areas to build a semantic label map.

In the next steps, you modify the color of a label, set the appearance of label names while labeling, modify the opacity of labels, view instances of a class of labels, and set the stacking order.

  1. To modify the color of the airplane ROI label, right-click the Airplane label in the ROI Labels pane and select Edit Label. Click the color preview and select a color.

  2. To show ROI label names while labeling, select Always from the Show ROI Labels menu in the View section.

  3. To increase the label opacity, use the Polygon slider in the Label Opacity section. Increasing the opacity helps to recognize which labels are in front of or behind other labels in the scene.

    All objects have complete opacity

  4. The tanker label is in front of the sailboat label, but it should be behind it. The position affects the stacking order in the exported ground truth. To modify the stacking order, right-click on the tanker ROI label in the scene and select Send To Back.

    Tanker appears behind sailboat in the scene

  5. To view instances of ROI labels, select By Instance from the ROI Color drop-down menu in the View section.

    Each object is a different color

  6. Export the labeled ground truth data by clicking Export and selecting To Workspace. Name the ground truth data MAT file gTruth.

Postprocess Exported Labels for Instance or Semantic Segmentation Networks

You can use the exported, labeled ground truth for training an instance segmentation network or a semantic segmentation network.

Follow these steps to process the polygon data for either semantic segmentation or instance segmentation.

StepDescriptionProcedure

1 — Display ground truth data

The exported ground truth object contains the data for five objects and three definitions.

Enter gTruth on the MATLAB® command-line.

>> gTruth

gTruth = 

  groundTruth with properties:

          DataSource: [1×1 groundTruthDataSource]
    LabelDefinitions: [3×5 table]
           LabelData: [1×3 table]

2 — Get polygon data

The LabelData property groups the data by label name. The Sailboat label cell array contains three elements, the Tanker cell array contains one element, and the Airplane cell array contains one element.

Type gTruth.LabelData.

>> gTruth.LabelData

ans =

  1×3 table

     Sailboat       Tanker       Airplane 
    __________    __________    __________

    {3×1 cell}    {1×1 cell}    {1×1 cell}

3 — Stack ground truth polygon data

The sailboat and the tanker are overlapping. This information (relative ordering of pixels) is lost in this format.

Use the gatherLabelData object function to group the data by label type. This produces one table containing five objects stacked.

Use the gatherLabelData property and store the output.

>> out = gatherLabelData(gTruth,[labelType.Polygon],'GroupLabelData','LabelType')

out =

  1×1 cell array

    {1×1 table}

Show the contents of the table.

>> out{1}.PolygonData
ans =

  1×1 cell array

    {5×2 cell}

4 — View ground truth data by depth order

The contents of PolygonData, shows the order of the stacked polygons. The order is consistent with the labeling order, including the tanker, which you sent to the back.

Now that the labels are flattened, at the base is the tanker, and the sailboat above it overwrites the tanker where they overlap.

Show the polygon data.

>> out{1}.PolygonData{1}

ans =

  5×2 cell array

    {12×2 double}    {'Airplane'}
    { 6×2 double}    {'Sailboat'}
    { 7×2 double}    {'Sailboat'}
    {13×2 double}    {'Sailboat'}
    { 9×2 double}    {'Tanker'  }

Convert Polygon Data for Semantic Segmentation

StepDescriptionProcedure

1 — Create a label ID map

Create a map associating class names to class IDs.

classNames = {'Airplane','Sailboat','Tanker'};
classIds = 1:numel(classNames);

labelIDMap = containers.Map(classNames,classIds);

2 — Extract polygons

Extract the polygon coordinates and polygon labels from the output structure.

polygons = out{1}.PolygonData{1}(:,1);
polygonLabels = out{1}.PolygonData{1}(:,2);

3 — Flatten polygons to a semantic segmentation map

Convert the label names to IDs.

Convert the polygon coordinates to a semantic segmentation map.

 
polygonLabelIDs = cellfun(@(x)labelIDMap(x),polygonLabels);
 
imageSize = [645 916]; % size(boats_im)
outputIm = poly2label(polygons,polygonLabelIDs,imageSize);

Convert Polygon Data for Instance Segmentation

StepDescriptionProcedure

Preallocate a mask stack for instance segmentation

Preallocate the mask stack with height and width equal to the image dimensions, and the channel depth the same as the number of polygons. 
polygons = out{1}.PolygonData{1}(:,1);
numPolygons = size(polygons,1);

imageSize = [645 916]; % size(boats_im)
maskStack = false([imageSize(1:2) numPolygons]);

Convert polygons to instance masks

Convert each polygon into a separate mask and insert it into the mask stack.

 
for i = 1:numPolygons
    maskStack(:,:,i) = poly2mask(polygons{i}(:,1), ...
                       polygons{i}(:,2),imageSize(1),imageSize(2));
end

See Also

Apps

Functions

Objects

Related Topics