Filtering and Cleaning Data

Technically, this is not a programming issue. rather, this is an issue about algorithm. It's all depending on what you meant by cleaning. Do you think the spikes are what you want to filtered out? Or do you want to do something else? If the spikes are what you think that shouldn't be filtered out, the simplest way to clean this is by a so called three sigma criterion--anything that is beyong mean+/-3*standard deviation is believed to be outliers. There are other tricks too. So, again, this is about the algorithm not about programming I believe.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Camilo Costa 2020-3-26

You are correct. This is about algorithm. If you have some advices, like functions to clean the outliers, smooth some high frequencies, etc., I will aprecciate. Thx

请先登录，再进行评论。

Answer 2

Peng Li 2020-3-26

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/512997-filtering-and-cleaning-data#answer_422068

在 MATLAB Online 中打开

A simple work around:

b = DADOSUFCS2(:, 2);
bstd = movstd(b, 100);
thre = nanmean(bstd);
bnew = b(bstd <= thre);

3 个评论
显示 1更早的评论隐藏 1更早的评论

Peng Li 2020-3-26

Sorry it's difficult for me to understand what you are trying to ask. What I provided is a simple algorithm based on moving standard deviation. anything whose corresponding moving standard deviation is above a threshold will be treated as outliers in my example.

Peng Li 2020-3-26

How do you know that they are not real? Do you have a specific criterion? If you have, then it is simple. If you don't, you may need to work out a bit more algorithm side as no algorithm is the best for filtering a general data set. You are the best person who knows your data the best.

请先登录，再进行评论。