Can you specify a particular filter kernel such that roifilt2() will preserve the area selected by the mask and discard the region outside? No.
Can you contrive a means to misuse roifilt2() for this task? Yes.
Consider the two images:
% a single-channel image
inpict = imread('cameraman.tif');
% a logical mask
mask = imread('sources/standardmods/cman/cmantifmk.png')>128;
% invert the mask and use a zero-valued kernel
%outpict = roifilt2(0,inpict,~mask);
% invert the mask and use a function handle which returns a
% zero-valued array of the same size and class as the image
outpict = roifilt2(inpict,~mask,@(x) 0*x);
Of course, both of these examples will fail for int16 images.
Should you use roifilt2() to do this job? No.
IPT roifilt2() is a tool to apply a function to an image segment and then composite the filtered segment back into the original image with the ostensible goal of reducing the amount of data that needs to be filtered. It combines a filtering process and a compositing process, and in this case, the filtering process is entirely unnecssary. To use roifilt2() for compositing alone would not only be wasteful and cryptic, it would impose unnecessary restrictions on what you could do.
If you want to do a basic compositng task like filling a masked region with a solid color, just do that.
% presuming the following:
% mask is logical-class
% image is not signed-integer class
% mask and image have same number of channels
outpict = inpict;
outpict(~mask) = 0;
This example requires about 13% as much time as misusing roifilt().
