Storage order in memory of which byte in memory is the most/least significant. See the following link for details...
ADDENDUM
That, of course, is for the syntax of your case 1) which, I now note is NOT the same as that of the subject line of the question;, ie,
fid=fopen(filename,'r','b');
is nothing at all like
fid=fopen(filename,'rb');
--the two 'b' 's are entirely different animals owing to position. In the former, it's the optional third argument in the syntax
[FID, MESSAGE] = fopen(FILENAME,PERMISSION,MACHINEFORMAT);
so it's the MACHINEFORMAT parameter, in the latter it's the second (also optional) argument as in
[FID, MESSAGE] = fopen(FILENAME,PERMISSION)
so it's part of the PERMISSIONs string in which case it is, as Philip notes, indicating a stream ('binary') file format as opposed to ASCII ('text') as given by the 't' substring.
Again, see
and read all the supplementary description for the various parameters for the details. The short story is that the 't' indicates text mode that forces the cr/lf pair for \n on Windows that is superfluous in Unix-like OS platforms.
