How to smoothen the noisy part of the data?

3 次查看（过去 30 天）
Muha
Muha 2024-9-21
评论： Star Strider 2024-9-23
I have taken out 10 examples of data sets. These data sets show gradual variation except towards the end where they show haphazard behavior. I have tried to correlate the problem with a previous answer related to signal smoothing (link): I have a very surface knowledge of which they have used.
In an ideal situation I would like to retain the original data as long as it varies gradually. Gradually varies refers to data which:
  1. has sudden jump but then it varies smoothly (can be seen initially in plot 3 and 7).
  2. does not have spikes where a spike is sudden up and down.
Keeping in mind the above results, I have adopted the reference code in the link and made some modifications e.g. movmean to movmedian and reduced the sample size of mov to 5 because data set is small. This can mostly work for me but it shows some inconsistent behavior at the locations encircled in the screenshot attached. Sometimes at the start the points do not match. Sometimes in the midway, it chooses the noisy points. I am not much proficient in the data filtering. I would like to smoothen the disturbed data without disturbing the acceptable data at all. I Will appreciate your help in this regard.
If it does not work, I would simply have to remove the noisy data.
clc
clear all
close all
%% Data have been sorted in x and y
load('chk.mat')
x = chkdta(:,1);
dta = chkdta(:,2:end);
figure
for i= 1:10
nexttile
hold on; grid on; box on;
% data
y = dta(:,i);
y_out = y;
% create the low pass filtered / smoothed version of y
k = [5 5];
ys = smoothdata(y,'movmedian',k); % smoothed version of y
% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal
ls = abs(yhp);
k = [5 5];
ls = movmedian(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);
plot(x,y,':.r',DisplayName='original')
plot(x,y_out,'--ob',DisplayName='filtered')
end
  1 个评论
John D'Errico
John D'Errico 2024-9-22
Unfortunately, your eye/brain is very good at knowing what you want those curves to look like. But writing an automatic scheme that willl be robust, and consistently perfect when faced with a variety of outliers is not oing to be trivial. Good luck.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Star Strider
Star Strider 2024-9-22
Ran in:
See if using the filloutliers function will do what you want —
clc
clear all
close all
%% Data have been sorted in x and y
load('chk.mat')
x = chkdta(:,1);
dta = chkdta(:,2:end);
dta = filloutliers(dta, 'linear'); % <— ADDED
figure
for i= 1:10
nexttile
hold on; grid on; box on;
% data
y = dta(:,i);
y_out = y;
% create the low pass filtered / smoothed version of y
k = [5 5];
ys = smoothdata(y,'movmedian',k); % smoothed version of y
% let's detect when we have too much noise content and replace y with ys in
% those sections
[B,A] = butter(2,0.03,'high');
yhp = filtfilt(B,A,y);
% logic signal
ls = abs(yhp);
k = [5 5];
ls = movmedian(ls.^2,k);
ls = ls./max(ls);
idx = (ls>0.1);
y_out(idx) = ys(idx);
plot(x,y,':.r',DisplayName='original')
plot(x,y_out,'--ob',DisplayName='filtered')
end
This runs in MATLAB Online and seems to produce tthe result you want. I can’t run this here, since I get some sort of weird ‘Authentication failed’ error. MathWorks must be doing site maintenance. (The problem didn’t exist a few hours ago.)
.
  2 个评论
Muha
Muha 2024-9-23
I have tried this but with other parameters. The linear seems to work better. Thank you. Should have found this out. Do'h! Thank you for providing the solution.
Cheers

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Smoothing and Denoising 的更多信息

标签

产品


版本

R2024a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by