To find points within a threshold distance, try the rangesearch function from the Statistics Toolbox.
If you want three-point clusters, one thing you could try is to use knnsearch to find the three closest points to each point. Pass the same array as the first two inputs to this function. Presumably the closest three will consist of the point itself plus the closest two other points.