It calls Java to create a message digest engine, and then it runs through the array or data file feeding chunks of the data to the engine, which updates the MD5 as it goes. Eventually it gets to the end and has the Java message digest engine return the hash, which it converts to hex.
You might prefer to look at http://www.mathworks.com/matlabcentral/fileexchange/5498-md5-signature-of-a-file
