Removing columns if single value is more than threshold
6 次查看(过去 30 天)
显示 更早的评论
I have a very big data matrix which I am trying to filter a bit. I would like to remove whole columns, if any value is for example 10% greater or less than row average. I checked the rmoutliers function, but I don't know how I can make that work the way I need. Another matter is that some columns are fine for my use, but they are scaled up, so they would get probably filtered out too with that method. That is fine, but could that be avoided by first normalizing the data somehow, and then restoring the filtered data to original scale after that. I would appreciate the help very much
0 个评论
采纳的回答
Adam Danz
2020-8-24
编辑:Adam Danz
2020-8-24
" I would like to remove whole columns, if any value is for example 10% greater or less than row average"
Demo:
% Create 100x10 matrix
data = rand(100,10) .* linspace(1,100,10);
% Determine which columns have at least 1 values that is
% within +/- 10% of the row's average
rowAverages = mean(data,2);
isNearAvg = abs(data - rowAverages) <= rowAverages * 0.1; % 10% threshold
replaceColumn = any(isNearAvg,1);
% Option 1: Repalce the column with NaNs, thereby preserving the original structure
data(:,replaceColumn) = NaN
% Option 2: Remove the columns (use replaceColumn to see which cols were removed)
data(:,replaceColumn) = []
2 个评论
Adam Danz
2020-8-25
Glad I could help!
Functions like rmoutliers come in handy during data exploration and if you are using a well-established method of outlier removal but it's often better to write your own functions when the method requires customization.
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Data Preprocessing 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!