This is a bit confusing.
x = rand(1,5);
y = rand(10,5);
X = x(3:5)
Y = y(:,3:5);
D = pdist2(X, Y)
[Dsort,I] = sort(D(:))
The sort call sorts the vectors and returns the sorted vector and the original indices of the corresponding element.
I do not see where a matrix of distances would be used here, because this compares one vector to a matrix. A matrix would be appropriate for comparing two matrices, as described in Compute Euclidean Distance.
.