Outlier Detection of a matrix depending on spatio coordinates

9 次查看（过去 30 天）
CSCh
CSCh 2023-8-15
编辑： Alan 2023-9-4
Hi, I have thousands of matrices, each of (130x160) with values (range of -25 up to +25) consisting of outliers.The values of the matrix depending on spatial data x(1:130) and Y(1:160). Outliers are often within the range and sometimes not only single values, clustering across 2 or 3 values across the raws and/or columns (X or Y) of the matrix.
I already tried "filloutliers" and deleteoutliers (https://de.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers)
with rather average level of satisfaction.
Can anyone recommend an appropriate method to detect the outliers? What kind of ML algorithms could I try?
Thanks,
Chris
  14 个评论
显示 12更早的评论
Alan
Alan 2023-9-4
I had intended to post my answer in the "Answers" section instead of the "Coments" section. Therefore I shall repost it in the "Answers" section. I hope that the OP and others looking for a solution to a similar problem will find the resources to be helpful.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Alan
Alan 2023-9-4
编辑：Alan 2023-9-4
Ran in:
(This is a repost from the comments section)
If you want to detect outliers within each matrix rather than considering each matrix as datapoints.
In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://www.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2
Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:
x = 1:130;
y = 1:160;
[X Y] = meshgrid(x, y);
% Generating a matrix for this example
Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));
% Fitting surface
xdata = reshape(X, 1, [])';
ydata = reshape(Y, 1, [])';
zdata = reshape(Z, 1, [])';
fitted_surface = fit([xdata ydata], zdata, "poly23");
% Visualize fitted surface against the matrix
plot(fitted_surface, [xdata ydata], zdata)
legend(["Fitted Surface"]);
% Continue to calculate z-score and exclude outliers
In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:
  1. isoutlier() - https://www.mathworks.com/help/matlab/ref/isoutlier.html
  2. rmoutliers() - https://www.mathworks.com/help/matlab/ref/rmoutliers.html
  3. filoutliers() - https://www.mathworks.com/help/matlab/ref/filloutliers.html
Some parameters to experiment with could be:
  1. dim : Dimension along which the outliers should be detected.
  2. movemethod : To detect outliers using a moving window.
  3. percentiles : To identify the extreme percentile of data.
As for machine learning methods, you could check out this resource: https://www.mathworks.com/help/stats/unsupervised-anomaly-detection.html . It contains various supervised methods for multivariate data.

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Fit Postprocessing 的更多信息

标签

产品


版本

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by