Do the whole avg_spec calculation at once:
avg_spec = mean(H_cube,3); % mean along third dimension
Use logical indexing to create data_mask and specular_mask
specular_mask = bsxfun(@or,H_cube>0.75, avg_spec>0.30); % apply or with avg_spec to every page of H_cube
data_mask = H_cube > 0.40; % logical indexing directly