Automatically Label Ground Truth Using Segment Anything Model

Since R2024b

This example uses:

This example shows how to produce pixel labels for semantic segmentation using the Segment Anything Model (SAM) in the Image Labeler app. The SAM is an automatic segmentation technique that you can use to segment object regions to label with just a few clicks, or automatically segment the entire image and instantaneously create labels for selected regions. In this example, you interactively label pixels for semantic segmentation in two ways.

Segment the image into object regions by marking points on the object and optionally, the background, to instantaneously create pixel labels. Use this process when the scene contains a few clearly localized objects.
Automatically segment the entire image into object regions, and create pixel labels by clicking on the segmented regions. Optionally, you can refine pixel labels created this way by marking object or background points. Use this process to label most regions of the image, or when the scene contains numerous similar objects.

To get started with the Image Labeler app, see Get Started with the Image Labeler. To learn more about the SAM, see Get Started with Segment Anything Model for Image Segmentation.

This example requires Image Processing Toolbox™ Model for Segment Anything Model support package. You can install the Image Processing Toolbox Model for Segment Anything Model from the Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. The support package also requires Deep Learning Toolbox™. Processing image data on a GPU requires a supported GPU device and Parallel Computing Toolbox™.

Open the Image Labeler

Open the Image Labeler app. You can also open the app from the Apps tab of the MATLAB® Toolstrip, under Image Processing and Computer Vision.

imageLabeler

On the app menu, select New Individual Project.

After launching the Image Labeler app, select the type of project from the menu.

Load an Image into Image Labeler

To load an image into the Image Labeler app, on the app toolstrip, click Import. Then, under Images, select From File. Browse to the location of visionteam.jpg in the same directory as this example, select the example file, and click Open.

Click Load Image on the main Image Labeler app toolstrip.

Create Label Definitions

A label definition specifies the name, color, and numeric index of a label. To label pixels for semantic segmentation, on the Image Labeler tab of the app toolstrip, select Add Label, and, under ROI Label Definitions, select Pixel.

Select Pixel in ROI Label Definitions menu to label pixel labels.

In the Define New Pixel Label dialog box, specify Label Name as Person. The label name must be a valid MATLAB variable name with no spaces. Select OK to create the new ROI label.

This dialog box enables you to define a new pixel label.

To define another label, select the Add Label on the Image Labeler tab of the app toolstrip again, and select Pixel. Specify Label Name as Shirt, specify New Group in the Group dropdown menu as Clothing, and select OK.

This dialog box enables you to define a new pixel label.

Select Segment Anything Tool

On the Label Pixels tab of the app toolstrip in the Labeling Tools section, select Segment Anything.

Select the Segment Anything Tool in the Labeling Tools setion of te Label Pixels tab.

The toolstrip adds a Segment Anything section, which includes SAM-based automatic segmentation options for labeling pixels. Note that the Mark Object button is automatically selected.

After you select Segment Anything, the toolstrip adds a Segment Anything section with segmentation options.

Label Segmented Object Regions Using Points

If the scene in your image contains only a few different objects, use the segmentation process shown in this section.

To start segmenting an object to assign the Person label to, click on the object. The Image Labeler app automatically generates pixel labels for the object, shading the labeled sections of the object in the color associated with the label. Click on other areas of the object until you are satisfied with the pixel labels.

Click a region to automatically segment it using SAM and add an ROI label.

To add the shaded blue region to the label, press Enter, or click Accept in the Segment Anything section of the app toolstrip.

Continue to label objects in the scene by marking new object points. Label regions object-by-object, and press Enter, or click Accept after you are done segmenting each object.

To remove regions from your segmented object, or refine segmented object boundaries, right-click or select Mark Background, and click, to mark red points on image areas extraneous to the object.

Select Mark Background and click on background regions of the image to remove those regions from the label.

To erase all the points you have marked and restart, press Esc or click Clear Marks.

Label Fully Segmented Image

To rapidly label pixels for many objects or a variety of objects in the scene, or to label regions within or outside of objects (such as walls or clothing), you can automatically segment the entire image. Then, you can assign labels to the various segmented regions.

To automatically segment the entire image into regions and select regions to add to the label, on the Label Pixels tab of the app toolstrip, in the Segment Anything section, select Segment Full Image.

Select Segment Full Image on the Segment Anything toolstrip to segment the entire image and select regions to assign ROI pixel labels to.

Select the Shirt ROI definition in the ROI Label Definitions pane. The Image Labeler app uses the SAM to automatically split the image into regions, whose boundaries appear on the image as thin borders in the color of the selected ROI definition. Pause on a region to preview the region shaded in orange.

This GIF shows the full image segmentation using the SAM-based Segment Full Image tool. Pause on a region to preview a region shaded in orange that you can add the ROI label to.

Click a shirt region to assign the Shirt label. After you click a region and label it, it remains shaded in orange. You can continue then click additional regions or objects to assign the label to them.

Click on each region to label with the ROI definition. Preview the pixel label in the shaded color corresponding to the ROI label color in the ROI Label Definitions pane.

To assign the Person label to additional regions, in the ROI Label Definitions pane, select the Person ROI label definition. The Image Labeler app outlines the borders of the regions in the color of the selected ROI definition. Click the head regions of the people in the image to label them with the Person label, and preview the pixel labels in shaded blue.

While Segment Full Image is selected, select the ROI label definition "Person" and click on people's heads to label them with this ROI label.

Using this process, you can rapidly assign ROI labels to many regions while Segment Full Image is selected. For example, you can create more label definitions, and click on regions to assign the labels.

To accept the assigned pixel labels, press Enter or, on the app toolstrip, click Accept.

Rapidly assign ROI pixel labels for multiple types of regions using the Segment Full Image tool.

Specify Full Image Segmentation Parameters

For most cases, such as in this example, you can use the default SAM parameters for full image segmentation. To change the default parameters of the model, in the Segment Anything section of the app toolstrip, click Settings. The app opens the Settings dialog box.

You can modify these full image segmentation settings, based on your application and available computing resources.

To segment and label small, densely packed objects relative to the image size, increase the Grid Size value.
To improve processing speed for segmentation when you are using a CPU, or have limited computing resources, decrease the Grid Size value.
To improve processing speed, at the expense of higher memory usage when you are using a GPU, increase the Point Batch Size value.
To label large segmented objects as a whole, without segmenting their individual parts, set the Segmentation Regions parameter to Large Regions. By default, the Segment Anything tool segments most identifiable regions, including the inner regions of an object.

To improve processing speed, use an acceptable GPU device. Using a GPU requires a Parallel Computing Toolbox™ license and a CUDA®-enabled NVIDIA® GPU. For more information, see GPU Computing Requirements (Parallel Computing Toolbox).

Refine Pixel Label Using Marked Points

While Segment Full Image is selected, you can refine the labeled segmented regions.

To add pixel labels to a region, click on the region to select it. Click on unlabeled parts of the region to add pixel labels.
To remove pixels from a labeled region, click on the region to select it. Then, right-click on incorrectly labeled parts of the region to remove pixel labels.
To label a region that has not been fully segmented by the Segment Anything tool (such as a part of a wall in the scene background), select the region.

For example, to refine a labeled shirt region, click to select the region. The Image Labeler app highlights the border of the region in yellow, and other region borders in gray.

While Segment Full Image is selected, click a region to refine it. The region is highlighted with a yellow border.

To add the pixels in the shirt collar of the person to the shirt region outlined in yellow, click both the shirt region and the region containing the shirt collar. The app marks the indicated points in green.

To add a shirt color region to the selected shirt region, click on both the shirt region and the color region.

To erase all the refinement points you have marked and restart, press Esc or click Clear Marks. If you are satisfied with the refined yellow-highlighted region, press Enter or click Accept to accept the pixel labels.

Preview the refined label result in the shaded color associated with the ROI label definition.