Outlier Detection of a matrix depending on spatio coordinates

18 次查看（过去 30 天）

CSCh 2023-8-15

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2008757-outlier-detection-of-a-matrix-depending-on-spatio-coordinates

编辑： Alan 2023-9-4

Hi, I have thousands of matrices, each of (130x160) with values (range of -25 up to +25) consisting of outliers.The values of the matrix depending on spatial data x(1:130) and Y(1:160). Outliers are often within the range and sometimes not only single values, clustering across 2 or 3 values across the raws and/or columns (X or Y) of the matrix.

I already tried "filloutliers" and deleteoutliers (https://de.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers)

with rather average level of satisfaction.

Can anyone recommend an appropriate method to detect the outliers? What kind of ML algorithms could I try?

Thanks,

Chris

14 个评论
显示 12更早的评论隐藏 12更早的评论

Bruno Luong 2023-8-25

Because you have another fit somewhere else I guess.

Alan 2023-9-4

I had intended to post my answer in the "Answers" section instead of the "Coments" section. Therefore I shall repost it in the "Answers" section. I hope that the OP and others looking for a solution to a similar problem will find the resources to be helpful.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Alan 2023-9-4

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2008757-outlier-detection-of-a-matrix-depending-on-spatio-coordinates#answer_1300326

编辑：Alan 2023-9-4

在 MATLAB Online 中打开

(This is a repost from the comments section)

If you want to detect outliers within each matrix rather than considering each matrix as datapoints.

In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://www.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2

Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:

x = 1:130;

y = 1:160;

[X Y] = meshgrid(x, y);

% Generating a matrix for this example

Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));

% Fitting surface

xdata = reshape(X, 1, [])';

ydata = reshape(Y, 1, [])';

zdata = reshape(Z, 1, [])';

fitted_surface = fit([xdata ydata], zdata, "poly23");

% Visualize fitted surface against the matrix

plot(fitted_surface, [xdata ydata], zdata)

legend(["Fitted Surface"]);

% Continue to calculate z-score and exclude outliers

In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:

isoutlier() - https://www.mathworks.com/help/matlab/ref/isoutlier.html
rmoutliers() - https://www.mathworks.com/help/matlab/ref/rmoutliers.html
filoutliers() - https://www.mathworks.com/help/matlab/ref/filloutliers.html

Some parameters to experiment with could be:

dim : Dimension along which the outliers should be detected.
movemethod : To detect outliers using a moving window.
percentiles : To identify the extreme percentile of data.

As for machine learning methods, you could check out this resource: https://www.mathworks.com/help/stats/unsupervised-anomaly-detection.html . It contains various supervised methods for multivariate data.