The discrepancy occurs because RREF attempts to produce rational solutions, with fairly small integers for the numerators and denominators, to systems which have coefficients that are "close to rational" as determined by approximating them via the function RAT. This behavior allows "nice" systems to have "nice" solutions when verifying hand calculations, thereby supporting the academic nature of RREF.
After performing Gauss-Jordan Elimination on the system, if the original coefficients were approximately rational as described above, then RREF calls RAT to find a simple rational approximation to the system's solution.
The attached MATLAB file, "rrefNoRational.m" is a copy of "rref.m" without the code that performs the rational conversion. Using this algorithm, the code:
A = [87 54 71 34 51;26 33 49 31 23;57 9 107 28 7;30 21 89 27 44;55 5 6 7 8];
b = [29;41;91;0;38];
B = [A b];
C = rrefNoRat(B);
returns
C(:,end) =
0.659729931133364
-0.461822077848194
-0.034894614961433
2.632290333420119
-1.774087558408284
