Outliers using the quartile method are usually identified as values greater than Q3+1.5*IQR or less than Q1-1.5*IQR where IQR is the interquartile range, Q1 and Q3 are the 25th and 75th percentiles.
What's the logic behind adding 3 to the upper range and adding 1 plus flipping the sign of the lower range? I don't think this is what you want to do.
Nevertheless, you can compute the outliers directly without using rmoutliers by using these simple steps.
1) Compute the interquartile range.
iqrng = iqr(data);
2) Compute Q1 and Q3
Q1Q3 = prctile(data, [25,75]);
3) Create logical array identifying outliers
isOut = data > Q1Q3(2) + 3+1.5*iqrng | data < Q1Q3(1) - 1-1.5*iqrng
% * ^^^ * ^^^^^ ???????
*I highly doubt this is how you want to define the outliers. I can't imagine how it would be useful.
To remove the outliers,
data(isOut) = [];
% or
data(isOut) = NaN;
