How can you do what R does? You can write the code. You will not find this directly in MATLAB that I know of. (I'm the author of alpha shape codes in 2-d and in 3-d. This variation of alpha shape is not something I've seen in MATLAB. My codes are close to the tessellation based code you will find in MATLAB.) However, it is a direct variation of the basic alpha shape algorithm, where a ball of fixed radius is allowed to roll around the exterior of the point cloud. So those curved segments are just the shape of the alpha ball, as far as it can intrude into the point cloud.
The basic difference is that code appears to use the shape of the alpha ball around the perimeter, so constructing those curved segments. In comparison, the tessellation based codes use the alpha ball to "erode" simplexes from a delaunay tessellation. Then what you see is the perimeter of the simplexes that remain.
Personally, I don't really like this variation. Why not? Because it will ALWAYS result in very spiky boundaries, with those devil shaped horns poking out at all angles. And that does not seem to make sense, when I am trying to represent some domain of interest. The standard tessellation based version of alpha shape is equivalent to the convex hull for sufficiently large alpha, and that seems to be a good idea to me. (So you would never catch me writing it anyway.) Those concave curved arcs may LOOK pretty, but they make little sense to me. (Just personal preference, but I can like what I like.)
If you do write it, then you might be willing to post it on the File Exchange, so others can benefit from it too. I'm sure someone else will like it, even if I don't.
Edit:
Let me explain more clearly why I think this basic idea is wrong, even though it may look pretty. Consider a simple point cloud:
xy(sum(xy.^2,2) > 1,:) = [];
S = alphaShape(xy,1)
S =
alphaShape with properties:
Points: [791×2 double]
Alpha: 1
HoleThreshold: 0
RegionThreshold: 0
plot(xy(:,1),xy(:,2),'.r')
So the alpha shape in MATLAB approximates this point cloud well enough. But if we looked at what the R code would do, each edge around the perimeter would be a tiny little wave, dipping inwards. It would be a wonderful thing, if you happen to be modelling pictures of sea urchins. But in general, I expect that a closed curved surface will be better approximated by straight line segments than by inwardly everywhere concave curved segments.
Next, once you have that alpha shape, computations done with it and using it will be far more efficient IF they are based on a triangulation of the enclosed domain. For example testing to see if a point lies inside the simple triangulation shape are trivial. Not so much when you have this horned thing.
And, yes, this may be biased by my use of alpha shapes to represent color gamuts in some of my prior work, where efficient computations were very much necessary, and where the surfaces usually were fairly smooth.
