Outlier Detection of a matrix depending on spatio coordinates

18 次查看(过去 30 天)
Hi, I have thousands of matrices, each of (130x160) with values (range of -25 up to +25) consisting of outliers.The values of the matrix depending on spatial data x(1:130) and Y(1:160). Outliers are often within the range and sometimes not only single values, clustering across 2 or 3 values across the raws and/or columns (X or Y) of the matrix.
I already tried "filloutliers" and deleteoutliers (https://de.mathworks.com/matlabcentral/fileexchange/3961-deleteoutliers)
with rather average level of satisfaction.
Can anyone recommend an appropriate method to detect the outliers? What kind of ML algorithms could I try?
  14 个评论
Alan 2023-9-4
I had intended to post my answer in the "Answers" section instead of the "Coments" section. Therefore I shall repost it in the "Answers" section. I hope that the OP and others looking for a solution to a similar problem will find the resources to be helpful.



Alan 2023-9-4
编辑:Alan 2023-9-4
(This is a repost from the comments section)
If you want to detect outliers within each matrix rather than considering each matrix as datapoints.
In that case, you can try to fit a surface to each matrix and evaluate the z-score of the matrix (considering the fitted surface as the mean). You could exclude the points whose absolute z-score crosses a certain threshold. Here is an example in 1-D: https://www.mathworks.com/help/curvefit/removing-outliers.html?s_tid=srchtitle_support_results_2_outliers#RemoveOutliersExample-2
Here is how you can fit a surface using the fit function instead of fitting a curve as shown in the above example:
x = 1:130;
y = 1:160;
[X Y] = meshgrid(x, y);
% Generating a matrix for this example
Z = 25 * sin(2 * pi / 130 / 160 * (X.^2 + Y.^2));
% Fitting surface
xdata = reshape(X, 1, [])';
ydata = reshape(Y, 1, [])';
zdata = reshape(Z, 1, [])';
fitted_surface = fit([xdata ydata], zdata, "poly23");
% Visualize fitted surface against the matrix
plot(fitted_surface, [xdata ydata], zdata)
legend(["Fitted Surface"]);
% Continue to calculate z-score and exclude outliers
In case you are considering each matrix as a datapoint, consider combining the matrices and find the outliers along the z-axis. The following functions could be utilized:
  1. isoutlier() - https://www.mathworks.com/help/matlab/ref/isoutlier.html
  2. rmoutliers() - https://www.mathworks.com/help/matlab/ref/rmoutliers.html
  3. filoutliers() - https://www.mathworks.com/help/matlab/ref/filloutliers.html
Some parameters to experiment with could be:
  1. dim : Dimension along which the outliers should be detected.
  2. movemethod : To detect outliers using a moving window.
  3. percentiles : To identify the extreme percentile of data.
As for machine learning methods, you could check out this resource: https://www.mathworks.com/help/stats/unsupervised-anomaly-detection.html . It contains various supervised methods for multivariate data.

更多回答(0 个)


Help CenterFile Exchange 中查找有关 Preprocessing Data 的更多信息




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by