It's trivial. Just use bwboundaries to get the list of (x,y) coordinates of the boundary and then put in a for loop, calculating the distance using the Pythagorean Theorem to get the boundary that is the closest. In the for loop, you can vectorize it and use min().
boundaries = bwboundaries(binaryImage);
overallMinDistance = inf;
for b = 1 : length(boundaries)
thisBoundaryX = boundaries{b}(:, 1);
thisBoundaryY = boundaries{b}(:, 2);
distances = sqrt((thisBoundaryX - refx).^2 + (thisBoundaryY - refy).^2);
[minDistance, indexOfMin] = min(distances);
if minDistance < overallMinDistance
% This boundary is the closer one.
.............you finish the rest - it's easy.
end
