Get Started with the Image Labeler

The Image Labeler app provides an easy way to interactively create a variety of shapes to mark as region of interest (ROI) labels. You can create axis-aligned or rotated rectangular, polyline, pixel, and polygon ROI labels and scene labels in an image or image sequence.

You can use labeled data to validate or train algorithms such as image classifiers, object detectors, and semantic and instance segmentation networks. Consider your application when choosing a labeling drawing tool to create ROI labels. For more details on how to select the right label type and drawing tool for your application, see ROI Labels, Sublabels, and Attributes.

ROI and Scene Label Definitions

An ROI label corresponds to either an axis-aligned or rotated rectangular, polyline, pixel, or polygon region of interest. These labels contain two components: the label name, such as "cars," and the region you create.
A Scene label describes the nature of a scene, such as "sunny." You can associate this label with a frame.

Using This Example

Whether you are working on an individual project or working as part of a team, the workflows outlined in this example are the same. However, if you are working on a team, only a subset of the steps explained here are required. The table shows what steps of the workflow are needed depending on your role. For more details about managing a labeling project using team-based labeling, see Get Started with Team-Based Labeling

Workflow	Individual	Team-based Project Manager	Label Task Owner	Review Task Owner
Load Images
Create Label Definitions
Label Images
Export Labeled Images for an Individual Project

Load Images

To begin a new or open an existing project, select either Open Project, New Individual Project or New Team Project. You can load unlabeled or partially labeled images, or load images that are ready for review.

Images that need to be labeled can be stored in a datastore, or a folder. The images must be readable by imread. A previous labeling session can be loaded from a saved project.

Load Data Programmatically

This code programmatically loads images of book covers from a datastore:

Create a datastore from a folder of images:

imageFolder = fullfile(toolboxdir("vision"),"visiondata","bookcovers")
imds = imageDatastore(imageFolder)

Load the datastore:
```
imageLabeler(imds)
```

To programmatically load a folder of images:

imageFolder = fullfile(toolboxdir("vision"),"visiondata","bookcovers")
imageLabeler(imageFolder)

Load Data Directly from the App

In this example, you load an image interactively from the app. To start, open the app from the Apps tab, under Image Processing and Computer Vision. Click Import to load images from a file or from a datastore in the workspace.

To load the image used for this example, click Import and then select From File. Select the boats.png image:

matlab/toolbox/vision/visiondata/boats.png

Layout of the Image Labeler App

Image labeler app with one image of boats displayed.

These are the sections of the Image Labeler app:

The first image is displayed on the canvas and all of the loaded images are displayed in the Image Browser at the bottom of the app.
The Visual Summary browser displays the distribution of ROI and scene labels for the images in the project.
The app has two tabs which control which set of options are available on the toolbar. The Image Labeler tab shown by default, provides file management, label definition options, automated labeling options, monitoring, a button to view shortcuts and tutorials, and the Export button. The Visualization tab provides options to visualize your work in the app. These options make it easier to work with labels by modifying the app layout, control how to display the ROI labels (for example, on hover, always displayed, or never), how to display ROIs, by color (all ROIs of the same label displayed in the same color) or by instance (each instance of the ROI label is displayed in a different color), and label opacity variation.
The ROI Labels pane displays the ROI label definitions to use for the project.
The Scene Labels pane displays the scene label definitions to use for the project.
The View Labels, Sublabels and Attributes pane display the details of the labeled ROIs and scenes.

Create Label Definitions

Define the labels you intend to draw. In this example, you define labels directly within the app. To define labels from the MATLAB^® command line instead, use the labelDefinitionCreator. All label names must be one word.

Create ROI Label

An ROI label is a label that corresponds to a region of interest (ROI) in an image. You can define these ROI label types.

Rectangle — Draw 2-D axis-aligned rectangular bounding box labels around objects in an image, such as vehicles, boats, buildings.
Rotated Rectangle — Draw 2-D rotated-rectangular bounding box labels around objects in an image, such as vehicles, boats, buildings.
Point — Draw a point to identify an object.
Projected cuboid — Draw 3-D bounding box labels around objects in an image.
Line — Draw linear ROIs to label lines, such as lane boundaries.
Polygon — Draw polygon labels around objects. You can label distinct instances of the same class. For more information on drawing polygon ROI labels for instance and semantic segmentation networks, see Label Objects Using Polygons
Pixel label — Draw pixels manually or automatically to label various classes, such as road or sky, for semantic segmentation:
- Automatically segment objects or regions in a scene and label pixels using the Segment Anything tool. You can select pixel regions of objects and defined regions by marking them with a few clicks, or segment the full image and select regions to label. For more information about automatically selecting pixels to label, see the Automatically Label Ground Truth Using Segment Anything Model example.
- Create a super pixel grid layout over an image and manually label pixels in the grid using the Superpixel tool. Select a predefined pixel label definition and left-click a grid to assign the selected pixel label definition to it. For more information about pixel labeling using manual pixel selection, see Label Pixels for Semantic Segmentation.

For more details about these ROI label definitions, see ROI Labels, Sublabels, and Attributes.

In this example, you define a Boat group for labeling types of boats, and then create a Rectangle ROI label for a Sailboat and a Tanker. To control showing the ROI label names during labeling, select On Hover, Always, or Never from the Show ROI Labels drop-down menu.

Click Label in the Label Definition section of the toolstrip.
Create a Rectangle label type named Sailboat.
Optionally, change the label color by clicking the preview color.
From the Group drop-down menu, select New Group ... and name the group Boats
Click OK.
The Boats group name appears in the ROI Labels pane with the label Sailboat created. You can move a label in the list to a different position or group in the list by left-clicking and dragging the label up or down.
To add a second type of Boats label, click Label and then Rectangle. Name the label Tanker. Click OK.
Select the Sailboat sublabel, then use the mouse to draw rectangular ROIs around the sailboats. Select the Tanker sublabel to draw an ROI around the tanker ship. Click on the image away from the ROI to deselect it. Selected ROIs appear with a yellow outline.

Create Sublabels

A sublabel is a type of ROI label that corresponds to a parent ROI label. Each sublabel must belong to, or be a child of, a specific label defined in the ROI Labels pane. For example, in the ocean scene, a sailboat label might have sublabels for sails. For more details about sublabels, see ROI Labels, Sublabels, and Attributes.

Define a sublabel for sails.

In the ROI Labels pane on the left, click the Sailboat label.
Click Sublabel in the Label Definition section of the toolstrip.
Select Rectangle and name the sublabel sail. Optionally, select a color (by default, sublabels have the same color as its parent label), and write a description. Click OK.
The sail sublabel appears in the ROI Labels pane. The sublabel is nested under the selected ROI label, Sailboat.
You can add multiple sublabels under a label. You can also drag-and-drop the sublabels to reorder them in the list and you can click any label and then click Edit additional edits.
In the ROI Labels pane, select the sail sublabel.
In the image frame, select the Sailboat label. The label turns yellow when selected. You must select the Sailboat label (parent ROI) before you can draw a sublabel for it.
Draw a sail sublabel for each of the sails.

Sublabels cannot have their own sublabels and they can be used with all ROI label types with the exception of pixel labels. Notice that the Labels, Sublabels, and Attributes pane provides a tally and stacking order for the created labels.

Show or Hide Labels and Sublabels

You can show or hide the labels or sublabels in a labeled image by using the icon on the ROI Labels pane. The appears only after you define a label or sublabel. By default, the app displays all the labels and the sublabels.

To hide a label or sublabel, click on the icon along side the label or sublabel name. The app hides the corresponding label or sublabel and displays the icon.

Create ROI Attribute

An ROI attribute specifies additional information about an ROI label or sublabel. For example, in a driving scene, attributes might include the type or color of a vehicle. In the ocean scene, attributes might include the type of sailboat or number of sails. You can define ROI attributes of these types.

Numeric Value — Specify a numeric scalar attribute, such as the number of doors on a labeled vehicle or the number of sails on a sailboat.
String — Specify a string scalar attribute, such as the color of a vehicle or boat.
Logical — Specify a logical true or false attribute, such as whether a vehicle is in motion or a boat is making way.
List — Specify a drop-down list attribute of predefined strings, such as make or model of a vehicle or boat.

For more details about these attribute types, see ROI Labels, Sublabels, and Attributes.

Add an attribute to a Label.

In the ROI Labels pane on the left, select the Sailboat label and then click Attribute in the Label Definition section of the toolstrip. Select List from the options.
In the Attribute Name box, type sailboatType.
In the List Items section, type the three different types of sailboats, typing each one on a new line; Monohull, Catamaran, and Multihull. Optionally, you can give the attribute a description, and click OK. Hover on any label or sublabel ROI to see its name.

You can also add attributes to sublabels. For example, in this example you could an attribute for the sail sublabel to indicate whether it is a foresail or a mainsail. Or, in this step, a logical attribute is added to indicate whether the sail is raised.

In the ROI Labels pane on the left, select the sail sublabel, click Attribute, and then click Logical.
In the Attribute Name box, type sailRaised. Leave the Default Value set to Empty, optionally write a description, and click OK.
Select a sail in the scene, when selected, the ROI appears in yellow. Select the attribute sailRaised and the appropriate logical.

To delete an attribute, right-click an ROI label or sublabel, and select the attribute to delete. Deleting the attribute removes attribute information from all previously created ROI label annotations.

Create Scene Labels

A scene label defines additional information for the scene. Use scene labels to describe conditions, such as lighting, weather, or events, such as lane changes or point of sail.

Create a scene label:

Select Label and then Scene from the Scene Label Definitions section located at the bottom of the drop-down list.
Enter a scene label named dayTime. Create a new group named TimeOfDay.
Create another scene label in the same group named nightTime.
To label the scene as day time, click the dayTime scene label, then click Apply to Image. A check-mark appears for the scene label.

Label Images

You can label images manually, use a suitable built-in automation algorithm, create a new algorithm, or import an algorithm.

Label Ground Truth Manually

The ROI and Scene labels are defined for your entire session and for all of the images in the session. Unless you use an automation algorithm, you must label all the relevant images.

Label Ground Truth Using an Automation Algorithm

To speed up the labeling process, you can use an automation algorithm to label the remainder of your images. Select one of these types of automation algorithms from the app toolstrip Automate Labeling > Select Algorithm section.

Custom Automation Function — Define a custom algorithm.
Use one of the built-in automation algorithms — Select a suitable algorithm. Follow the steps that appear in the right pane.
Add Whole Image Algorithm — You can create a new automation algorithm or import one. For details on both options, see Create Automation Algorithm.
Add Blocked Image Algorithm — You can create a new blocked image automation algorithm or import one. For details on both options, see Label Large Images in the Image Labeler.

After using an automation algorithm you can manually label the remaining frames with sublabel and attribute information.

To further evaluate your labels, you can view a visual summary of the labeled images. From the app toolstrip, select View Label Summary. Use this summary to compare the frames, frequency of labels, and scene conditions. For more details, see View Summary of Ground Truth Labels. This summary does not show sublabels or attributes.

Export Labeled Images for a Team-based Project

To export labeled images when you are working as part of a team-based project, follow the directions outlined in Combine Reviewed Images and Export.

Export Labeled Images for an Individual Project

You can export the labeled ground truth to a MAT-file or to a variable in the MATLAB workspace. In both cases, the labeled ground truth is stored as a groundTruth object.

You can use the groundTruth object to train a deep-learning-based computer vision algorithm. For more details, see Training Data for Object Detection and Semantic Segmentation.

Note

If you export pixel data, the pixel label data and ground truth data are saved in separate files but in the same folder. For considerations when working with exported pixel labels, see How Labeler Apps Store Exported Pixel Labels.

In this example, you export the labeled ground truth to the MATLAB workspace. From the app toolstrip, select Export Labels > To Workspace. The exported MATLAB variable is gTruth.

Display the properties of the exported groundTruth object. The information in your exported object might differ from the information shown here.

gTruth

gTruth = 

  groundTruth with properties:

          DataSource: [1×1 groundTruthDataSource]
    LabelDefinitions: [2x6 table]
           LabelData: [531×3 timetable]

Data Source

DataSource is a groundTruthDataSource object containing the path to the images or video and timestamps. Display the properties of this object.

gTruth.DataSource

ans = 

groundTruthDataSource for a video file with properties

        Source: ...matlab\toolbox\vision\visiondata\visiontraffic.avi
    TimeStamps: [531×1 duration]

Label Definitions

LabelDefinitions is a table containing information about the label definitions. This table does not contain information about the labels that are drawn on the video frames. To save the label definitions in their own MAT-file, from the app toolstrip, select Save > Label Definitions. You can then import these label definitions into another app session by selecting Import Files.

Display the label definitions table. Each row contains information about an ROI label definition or a scene label definition. If you exported pixel label data, the LabelDefinitions table also includes a PixelLabelID column containing the ID numbers for each pixel label definition.

gTruth.LabelDefinitions

ans =
  3×6 table

      Name         Type        LabelColor        Group       Description     Hierarchy  
    _________    _________    ____________    ___________    ___________    ____________

    {'Car'  }    Rectangle    {1×3 double}    {'Vehicle'}    {0×0 char}     {1×1 struct}
    {'Truck'}    Rectangle    {1×3 double}    {'Vehicle'}    {0×0 char}     {0×0 double}
    {'Sunny'}    Scene        {1×3 double}    {'Weather'}    {0×0 char}     {0×0 double}

Within LabelDefinitions, the Hierarchy column stores information about the sublabel and attribute definitions of a parent ROI label.

Display the sublabel and attribute information for the Car label.

gTruth.LabelDefinitions.Hierarchy{1}

ans = 

  struct with fields:
       numDoors: [1×1 struct]
          color: [1×1 struct]
       inMotion: [1×1 struct]
        carType: [1×1 struct]
      headlight: [1×1 struct]
           Type: Rectangle
    Description: ''

Display information about the headlight sublabel.

gTruth.LabelDefinitions.Hierarchy{1}.headlight

ans = 

  struct with fields:
           Type: Rectangle
    Description: ''
          Color: [0.5862 0.8276 0.3103]
           isOn: [1×1 struct]

Display information about the carType attribute.

gTruth.LabelDefinitions.Hierarchy{1}.carType

ans = 

  struct with fields:

      ListItems: {3×1 cell}
    Description: ''

Save App Session in a Team-based Project

Sessions are saved in each step of a team-based project. For more information about each of these steps, see Get Started with Team-Based Labeling.

Save App Session as an Individual Project

From the app toolstrip, select Save Project then Save asto save a MAT-file of the app session. The saved session includes the data source, label definitions, and labeled ground truth. It also includes your session preferences, such as the layout of the app. To change layout options, select Layout.

At any time during a session, you can select New Session to start a new session. You have the option of saving the current session or cancelling.

The app session MAT-file is separate from the ground truth MAT-file that is exported when you select Export. To share labeled ground truth data, as a best practice, share the ground truth MAT-file containing the groundTruth object, not the app session MAT-file. For more details, see Share and Store Labeled Ground Truth Data.