Concave hulls are ambiguous, though there may be a tool that will help you that I'll mention after the example. Consider this set of points:
x = [-1 1 1 -1 0];
y = [-1 -1 1 1 0];
plot(x, y, 'o');
axis([-2 2 -2 2]);
axis square
Which would be the concave hull of this data?
figure
subplot(2, 2, 1);
plot(x([1:5 1]), y([1:5 1]), 'o-');
axis([-2 2 -2 2]);
axis square
subplot(2, 2, 2);
plot(x([1 2 5 3 4 1]), y([1 2 5 3 4 1]), 'o-');
axis([-2 2 -2 2]);
axis square
subplot(2, 2, 3);
plot(x([1 2 5 3 4 5 1]), y([1 2 5 3 4 5 1]), 'o-');
axis([-2 2 -2 2]);
axis square
subplot(2, 2, 4);
plot(x([1:4 1]), y([1:4 1]), 'o-');
hold on
plot(x(5), y(5), 'o');
axis([-2 2 -2 2]);
axis square
That being said, the alphaShape function may do what you want.
figure
plot(alphaShape(x.', y.'))
axis([-2 2 -2 2]);
axis square
Choose your alpha value carefully.
