Hi Sukuchha,
I think that you just need to do this:
XY = [0 0; 1 0; 1 1; 0 1]; % unclosed XY points
XY_closed = XY([1:end 1],:); % closed XY points
The Douglas Peucker algorithm (at least, the one used by reducem in the Mapping Toolbox) takes account of the shape as a polygon, so its result should be closed (or at least closeable via the code I gave above).
Furthermore, the output from bwboundaries supplies each boundary as closed (ie, the first XY boundary point is coincident with the last XY boundary point), so I think you can simply pass the results of bwboundary to a Douglas Peucker function, and then use that result directly.
BW = false(9)
BW(3:6,3:6) = true
bb = bwboundaries(BW)
bb{1}
ans =
2 2
2 3
2 4
3 4
4 4
4 3
4 2
3 2
2 2 % Note that this last coordinate is the same as the first
[newX, newY] = reducem(bb{1}(:,2),bb{1}(:,1));
[newX, newY]
ans =
2 2
4 2
4 3
4 4
2 4
2 2 % In the reduced result the first/last coords also match
If you want to turn this reduced polygon back into a mask that was the same size as the original BW, you can just use poly2mask() as follows:
newBW = poly2mask(newX,newY,size(BW,1),size(BW,2)) - BW
