Just do it on the whole image. Then use your binary mask to assign only those pixels in your new image that are in the mask, to your original image in the same pixel locations:
% Replace original pixels with adjusted pixels, but only under the mask.
I(binaryMask) = newImg(binaryMask);
