how to remove outliers in large data sets?

Question

MUKESH KUMAR 2022-1-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1624185-how-to-remove-outliers-in-large-data-sets

评论： Image Analyst 2022-1-17

N186TSR.png

I am unable to open example code of outliers (openExample('matlab/RemoveOutliersInVectorExample') ) and openExample('matlab/DetermineOutliersWithStandardDeviationExample') also.

I had large datasets of power load for three years at 30min interval, I want to remove the outliers poitns which is affecting my forecasting error.

any help to remove the outliers in such datasets would be appreciated.

Thanks

Reference image is attached which shows the outliers datasets in upper side of image, reference to these points the error is also high (as lower side of image).

Thanks again

2 个评论
显示无隐藏无

Image Analyst 2022-1-7

How large is the data set? How many gigabytes? Can you attach a smaller set (less than 5 MB) in a .zip file?

MUKESH KUMAR 2022-1-8

Copy of data.xlsx

Not in GB but its five years data having outliers data in patters, i am attaching data file here

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Image Analyst 2022-1-8

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1624185-how-to-remove-outliers-in-large-data-sets#answer_870815

在 MATLAB Online 中打开

Try this:

data = readmatrix('Copy of data.xlsx');
x = data(:, 1);
y = data(:, 2);
% Plot just the first cycle.
last = round(70000/3) 
x = x(1:last);
y = y(1:last);
subplot(2, 1, 1);
plot(x, y, 'b-')
grid on;
title('Showing One Cycle Only')
% Smooth the data.
windowWidth = 2001;	% Some large odd number.
smoothY = movmean(y, windowWidth);
hold on;
plot(x, smoothY, 'r-', 'LineWidth', 3)
% Compute difference between actual and smoothed.
diffy = y - smoothY;
subplot(2, 1, 2);
plot(x, diffy, 'b-');
grid on;
% Detect outliers as having a MAD of more than 900
outlierIndexes = abs(diffy) > 900;
% Plot outliers as red dots over the original data.
subplot(2, 1, 1);
hold on
plot(x(outlierIndexes), y(outlierIndexes), 'r.', 'MarkerSize', 7);
% Now remove outliers from x and y
x(outlierIndexes) = [];
y(outlierIndexes) = [];

2 个评论
显示无隐藏无

MUKESH KUMAR 2022-1-17

This is very helpful and I understand the code and applied in my problem

I also tried to fill outliers data like openExample('matlab/DetectOutliersWithSlidingWindowRmoutliersExample')

but the cmd is not working in 2018a version, its applicable on 2018b version.

''rmoutliers" cmd not working

Is there any way to how to fill these outliers with using movmean or movmedian?

Thanks

Image Analyst 2022-1-17

You must have a version so old that rmoutliers was not in it yet. However you can do it manually. Just smooth the curve and subtract it from your data and threshold like I did. I didn't use rmoutliers.

请先登录，再进行评论。