The reason that a vector rather than a matrix is returned is that MATLAB doesn't know that your "mask" is going to still be rectangular. For example, if your input had been
A = [1 2; 3 NaN];
then your code wouldn't be able to return a matrix.
As Walter's question suggests, you'll need to decide what to do about partial-NaN rows. Probably one of the following is what you want:
% Remove rows with all NaN:
A(all(isnan(A),2),:) = []
% Remove row with any NaN:
A(any(isnan(A),2),:) = []
You could do the analogous operations on columns by changing the 2's to 1's (and using the semicolon in the first index instead of the second).
