Hypothesize that norm(M(i,:) - M(pIdx,:)) < Eps is true for many "i" values. You initialize cluster to be mostly 0's, so the second condition in your "if" will usually be true, leading the code to call itself recursively for all those places where the norm satisfied that condition. If you can't be certain that there are no more than 100-ish places that the norm is satisfied, then you are going to blow the normal recursion limit.
You should be considering an iterative algorithm instead of a recursive one.
By the way, a side point about efficiency: there is no point in computing the norm() for any i for which cluster(i) is non-zero, so you might as well change the "for" loop to
for i = find(cluster==0,1) : size(M,1)
There is another efficiency to be gained by re-calculating the start index after the recursive dbrec() call; if it is after the current position then you can skip the iterations until that position. A "while" loop would probably work better than a "for" loop for this efficiency.
