Why do you feel the need to do this? Since it is trivial to use sort to also sort a second vector/array that goes along with your sorted input, this task seems a poor choice to invest your efforts.
And since you actually use a high level tool like ismember, it cannot be that for some reason, sort is not "allowed", as ismember would logically then also be disallowed.
Writing your own basic routines like sort is almost never a good idea. Are you trying to learn basic sorting techniques? If so, then why are you so worried about efficiency?
Anyway, you are then trying to do this using recursive calls to your function. The recursion will probably be quite expensive in itself, as then you cause MATLAB to create multiple instances of your function, all with subtly changing data. This is something I would expect to be not efficient at all, and of course, those constant calls to ismember will add up considerably. I'd even claim that deep recursions are only rarely a good choice of algorithm in MATLAB.
