How could I remove outliers in really small data?

Question

Juan Manuel Hussein Belda 2021-11-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1591574-how-could-i-remove-outliers-in-really-small-data

评论： Sulaymon Eshkabilov 2021-11-22

I have data that should resemble a parabola when plotted into a figure. However, near the center, there is a "high" value for the data.

x = [-9.0000   -8.0000   -7.0000   -6.0000   -5.0000   -4.0000   -3.0000   -2.0000   -1.0000         0    1.0000 ...
     2.0000    3.0000    4.0000    5.0000    6.0000    7.0000    8.0000    9.0000   10.0000];
y = [0.0173    0.0169    0.0168    0.0166    0.0166    0.0167    0.0165    0.0165    0.0166    0.0167    0.0168 ...
   0.0177    0.0189    0.0173    0.0176    0.0178    0.0180    0.0181    0.0182    0.0185];

The values I would like Matlab to see as an outlier are x = 2 --> y = 0.0177, and x = 3, --> y = 0.0189, because I should not expected the parabola to grow in the middle, and then decrease. However, it does not count this points as outliers because, of course, Matlab does not know that I should be expecting a parabola-like shape. How could I do this? Thank you!

6 个评论
显示 4更早的评论隐藏 4更早的评论

John D'Errico 2021-11-20

编辑：John D'Errico 2021-11-20

在 MATLAB Online 中打开

A picture is sometimes worth a thousand words.

x = [-9.0000 -8.0000 -7.0000 -6.0000 -5.0000 -4.0000 -3.0000 -2.0000 -1.0000 0 1.0000 ...

2.0000 3.0000 4.0000 5.0000 6.0000 7.0000 8.0000 9.0000 10.0000];

y = [0.0173 0.0169 0.0168 0.0166 0.0166 0.0167 0.0165 0.0165 0.0166 0.0167 0.0168 ...

0.0177 0.0189 0.0173 0.0176 0.0178 0.0180 0.0181 0.0182 0.0185];

plot(x,y,'o')

Of course, sometimes even a picture is insufficient. ;-)

So which outliers do you hope to identify? Is the bump down at x==-4 really an outlier? It is not very far out, so perhaps difficult to identify.

And when you have a pair of points that are close to each other that are both apprently outliers, this is more difficult to identify, since they tend to mask each other. And when you have only few data points, things get more tough yet. At least your data does not appear to be real noisy outside of the outliers.

John D'Errico 2021-11-20

What DPB has suggested is a variation of often called an iteratively reweighted least squares. It is the basis for many of the robust fitting tools you will find. Points with large residuals are downweighted, then a weighted fit is redone. In the case of an outlier scheme, you can just decide to just remove them if you wish.

Juan Manuel Hussein Belda 2021-11-22