By "I want to replace p percent of largest values", I assume you mean replace the value of A that are greater than (1-p) * max_value_of_A. If not, you'll have to clarify what you mean.
A = rand(8, 10) %for example
p = 0.1; %for example
maxvalue = max(A(:));
threshold = (1-p) * maxvalue;
A(A >= threshold) = NaN
