A return value of 3 from that statement would be typical. The value returned from fopen() is a File Identifier, which is a number used to identify which file you are referring to when you use other I/O statements.
After your fopen() statement, you will need fread() statements to read the data from the file, and you will need output statements such as fprintf() to display the file content as hex.
For example,
fid = fopen('VCS_5405.rbf','r','l');
while true
this16 = fread(fid, 16, '*uint8');
if isempty(this16); break; end %end of file
thisfmt = [repmat('%02X ', 1, length(this16)-1), '%02X\n'];
fprintf(1, thisfmt, this16);
end
