First of all, 3 million elements not a large dataset by any modern definition. It's like 1/10th the size of a typical point and shoot camera image. Anyway, you can simply threshold to get a map of where the outlier pixels are:
badElements = displace > 1; % Or whatever value you want.
To remove them from your array, do this:
displace(badElements) = [];
