segmentAnythingModel

Pretrained Segment Anything Model 2 (SAM 2) for image segmentation

Since R2024a

expand all in page

Description

Use the segmentAnythingModel object to interactively segment objects in an image using visual prompts, such as foreground points or bounding boxes.

A segmentAnythingModel object configures a pretrained Segment Anything Model 2 (SAM 2) or Segment Anything Model (SAM) for image segmentation of objects in an image without retraining the model. To learn more about the model and the training data, see the SA-V Dataset and SA-1B Dataset pages.

To segment objects in an image, you must first extract the image embeddings from the SAM image encoder by using the extractEmbeddings function. Then, segment objects from the image embeddings by using the segmentObjectsFromEmbeddings function.

Note

To use any of the SAM 2 models, this functionality requires the Image Processing Toolbox™ Model for Segment Anything Model 2 add-on if you use any of the SAM 2 models. To use the base SAM model, this functionality requires the Image Processing Toolbox Model for Segment Anything Model add-on.

Creation

Syntax

sam = segmentAnythingModel
sam = segmentAnythingModel(modelName)

Description

sam = segmentAnythingModel creates a pretrained Segment Anything Model 2 that was trained on the Segment Anything Video (SA-V) data set.

example

sam = segmentAnythingModel(modelName) specifies the SAM variant to use for segmentation, and sets the ModelName property.

Input Arguments

expand all

Since R2026a

SAM variant, specified as one of these values:

SAM VariantDescription
"sam2-large"Selects the large SAM 2 model trained on the Segment Anything Video (SA-V) data set. This model is the largest, slowest, and most accurate SAM 2 model, suitable for high-accuracy applications such as medical imaging and video post-production. It requires the most computational resources, including a GPU with 40 to 80 GB of VRAM.
"sam2-baseplus"Selects the base SAM 2 model trained on the SA-V data set. It is suitable for general applications such as land cover classification from geospatial imagery. It requires fewer computational resources than the large SAM 2 model, including a GPU with 6 to 32 GB of VRAM.
"sam2-small"Selects the small SAM 2 model trained on the SA-V data set. This model balances size, speed, and accuracy, and is suitable for efficiently segmenting objects across video frames. Use this model for real-time or near-real-time video frame analysis on mid-range GPUs or edge devices.
"sam2-tiny"Selects the tiny SAM 2 model trained on the SA-V data set. This model is the smaller, faster, and less accurate than other SAM 2 models. You can use this model with a standard CPU or in mobile applications, because of its fast inference.
"sam-base"Selects the base SAM ViT-B model trained on the Segment Anything 1 Billion (SA-1B) data set. Use SAM instead of SAM 2 if you require compatibility with legacy systems, have hardware constraints, or need to reproduce results achieved using the base SAM model.

Note

The SAM 2 models require the Image Processing Toolbox Model for Segment Anything Model 2 add-on. The base SAM model requires the Image Processing Toolbox Model for Segment Anything Model add-on.

Data Types: char | string

Properties

expand all

Since R2026a

This property is read-only.

SAM variant, represented as "sam2-large", "sam2-baseplus", "sam2-small", "sam2-tiny", or "sam-base". You can set this property using the modelName input argument.

Object Functions

extractEmbeddingsExtract feature embeddings from Segment Anything Model (SAM) encoder
segmentObjectsFromEmbeddingsSegment objects in image using Segment Anything Model (SAM) feature embeddings

Examples

collapse all

This example uses:

Open Live Script

Create a Segment Anything Model (SAM) for image segmentation. 

sam = segmentAnythingModel;

Read and display an image.

I = imread("pears.png");
imshow(I)

Calculate the image size.

imageSize = size(I);

Extract the feature embeddings from the image.

embeddings = extractEmbeddings(sam,I);

Specify visual prompts corresponding to the object that you want to segment, such as a pear along the bottom edge of the image. This example selects two foreground points within the pear. Refine the segmentation by including one background point outside the object to segment.

fore = [512 400; 480 420];
back = [340 300];

Overlay the foreground points in green and the background point in red.

hold on
plot(fore(:,1),fore(:,2),"g*",back(:,1),back(:,2),"r*",Parent=gca)
hold off

Segment an object in the image using SAM segmentation.

masks = segmentObjectsFromEmbeddings(sam,embeddings,imageSize, ...
    ForegroundPoints=fore,BackgroundPoints=back);

Overlay the detected object mask on the test image.

imMask = insertObjectMask(I,masks);
imshow(imMask)

Limitations

The SAM can have difficulty detecting and accurately segmenting objects of these types:

  • Objects with boundaries that are indistinct or blurry, such as those in low-resolution images.

  • Very small objects within a scene, particularly when they occupy only a few pixels.

References

[1] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, et al. "Segment Anything," April 5, 2023. https://doi.org/10.48550/arXiv.2304.02643.

[2] Ravi, Nikhila, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, et al. “SAM 2: Segment Anything in Images and Videos.” arXiv, October 28, 2024. https://doi.org/10.48550/arXiv.2408.00714.

Version History

Introduced in R2024a

expand all

See Also

Functions

Objects

Apps

Topics

External Websites