If I understand your latest comment, you begin by finding the unique numbers:
an = unique(a,'stable').'
bn = unique(b,'stable').'
an =
1 3 5 6 7 2 8
bn =
5 2 1 3 7 6 8
Now you take all the 5's in b and replace them by 1's, all the 2's by 3's, and so on, all at once (otherwise in a later step all the 1's will be changed back to 5's). You can do this by initializing a new matrix and putting all the numbers in it:
c = zeros(size(b));
for i=1:length(an)
idx = (b==bn(i));
c(idx) = an(i);
end
c
c =
1 5 7 8 6
2 3 5 8 7
1 6 3 2 8
This seems to be what you're saying, although the result is completely different from the answer you provided.
