Main Content

segmentObjectsFromEmbeddings

Segment objects in medical image using Medical Segment Anything Model (MedSAM) image embeddings

Since R2024b

collapse all in page

    Syntax

    [mask,score] = segmentObjectsFromEmbeddings(medsam,embeddings,imageSize,BoundingBox=boxPrompt)

    Description

    [mask,score] = segmentObjectsFromEmbeddings(medsam,embeddings,imageSize,BoundingBox=boxPrompt) segments objects from an image of size imageSize using the Medical Segment Anything Model (MedSAM) image embeddings embeddings and the bounding box coordinates boxPrompt as a visual prompt, and returns the predicted object mask mask and the corresponding prediction score score.

    Note

    This functionality requires Deep Learning Toolbox™, Computer Vision Toolbox™, and the Medical Imaging Toolbox™ Model for Medical Segment Anything Model. You can install the Medical Imaging Toolbox Model for Medical Segment Anything Model from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

    example

    Examples

    collapse all

    This example uses:

    Open Live Script

    Create a Medical Segment Anything Model (MedSAM) object.

    medsam = medicalSegmentAnythingModel;

    Download and unzip the Medical Segmentation Decathlon data set [1].

    zipFile = matlab.internal.examples.downloadSupportFile("medical","MedicalVolumeNIfTIData.zip");
filepath = fileparts(zipFile);
unzip(zipFile,filepath)

    Load a lung CT volume from the data set into the workspace.

    dataFolder = fullfile(filepath,"MedicalVolumeNIfTIData");
filePath = fullfile(dataFolder,"lung_027.nii.gz");
medVol = medicalVolume(filePath);
vol = medVol.Voxels;

    Extract a transverse slice and visualize it.

    img = extractSlice(medVol,45,"transverse");
figure
imshow(img,[])

    Extract the image embeddings from the CT image.

    embeddings = extractEmbeddings(medsam,img);

    Specify a bounding box that contains an object to segment by interacting with the visualization, such as a bounding box with height and width of 70 pixels each and a top-left corner with xy-coordinates of (310,255).

    boxPrompt = [310 255 70 70];

    To segment the object, run the MedSAM decoder on the image embeddings. Specify the bounding box visual prompt BoundingBox as the position of the rectangle ROI that you have drawn on the image display.

    mask = segmentObjectsFromEmbeddings(medsam,embeddings,size(img),BoundingBox=boxPrompt);
overlayedImg = insertObjectMask(rescale(img),mask);

    Visualize the segmentation mask overlaid on the image in the image display.

    figure
imshow(overlayedImg)

    [1] Medical Segmentation Decathlon. "Lung." Tasks. Accessed May 10, 2018. http://medicaldecathlon.com/.

    The Medical Segmentation Decathlon data set is provided under the CC-BY-SA 4.0 license. All warranties and representations are disclaimed. See the license for details.

    Input Arguments

    collapse all

    Medical Segment Anything Model, specified as a medicalSegmentAnythingModel object.

    Image embeddings, specified as a 64-by-64-by-256 numeric array. Generate the embeddings for an image or a batch of images using the extractEmbeddings object function. To segment an image using the segmentObjectsFromEmbeddings function, you must use the embedding of only one image, along with a visual prompt.

    Data Types: single

    Size of the image to segment, specified as a 1-by-2 vector of positive integers of the form [height width] or a 1-by-3 vector of positive integers of the form [height width channels], in pixels.

    Data Types: single | double

    Rectangular bounding box that contains the object to be segmented, specified as a 1-by-4 vector of the form [x y width height]. The coordinates x and y specify the center of the box, and width and height are the width and height of the box, respectively.

    Data Types: single | double

    Output Arguments

    collapse all

    Object masks, returned as a logical matrix of the same height and width as the input image.

    Prediction score, returned as a numeric scalar.

    Extended Capabilities

    Version History

    Introduced in R2024b

    See Also

    |

    Topics