Removing columns if single value is more than threshold

6 次查看(过去 30 天)
I have a very big data matrix which I am trying to filter a bit. I would like to remove whole columns, if any value is for example 10% greater or less than row average. I checked the rmoutliers function, but I don't know how I can make that work the way I need. Another matter is that some columns are fine for my use, but they are scaled up, so they would get probably filtered out too with that method. That is fine, but could that be avoided by first normalizing the data somehow, and then restoring the filtered data to original scale after that. I would appreciate the help very much

采纳的回答

Adam Danz
Adam Danz 2020-8-24
编辑:Adam Danz 2020-8-24
" I would like to remove whole columns, if any value is for example 10% greater or less than row average"
Demo:
% Create 100x10 matrix
data = rand(100,10) .* linspace(1,100,10);
% Determine which columns have at least 1 values that is
% within +/- 10% of the row's average
rowAverages = mean(data,2);
isNearAvg = abs(data - rowAverages) <= rowAverages * 0.1; % 10% threshold
replaceColumn = any(isNearAvg,1);
% Option 1: Repalce the column with NaNs, thereby preserving the original structure
data(:,replaceColumn) = NaN
% Option 2: Remove the columns (use replaceColumn to see which cols were removed)
data(:,replaceColumn) = []
  2 个评论
Mat P
Mat P 2020-8-25
I think this works well, thank you. I Didn't think the logical way.
Adam Danz
Adam Danz 2020-8-25
Glad I could help!
Functions like rmoutliers come in handy during data exploration and if you are using a well-established method of outlier removal but it's often better to write your own functions when the method requires customization.

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Data Preprocessing 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by