how to remove data discontinuities, jumps, outliers, and other artifacts that are not noise. Smoothing is bad,

24 次查看（过去 30 天）
Eugene Davis
Eugene Davis 2016-7-22
评论： Walter Roberson 2016-7-24
I'd like to hear of cool ways to process what should nominally be smooth data and detect and remove jumps, single point outliers, and other artifacts that are not noise. The data must be knit together prior to doing a nonlinear regression fit to a model.

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2016-7-23
You do not know what is noise and what is not until you have a model to compare against. You therefore cannot eliminate the noise in order to fit to determine what your model is.
With a non-linear model, a value that is much different than the others is not necessarily an artifact or noise. For example it might turn out that the model was y = 1/(a*x-b)^2 in which case if x turned out to be very close to b/a then the y value could be quite big but just a very short distance away y might be much smaller. Eliminating the large y value would, in such a case, lead to an incorrect model.
  2 个评论
Eugene Davis
Eugene Davis 2016-7-23
i actually know the parametric form for the data. It is a detection system and sometimes there are step changes or single point outliers after which the system returns to the nominal smooth form. Of course trying to model the data without the correction results in an incorrect solution. CUrrently I do detection and correction and it works well. I just wonder what other cool methods folks might have.
Walter Roberson
Walter Roberson 2016-7-24
You might know the parametric form, but we do not know it. In the general case with non-linear systems, you cannot know what the noise is until you have produced a complete model with coefficients.
You want to do "outlier detection" to find points that cannot be produced by any combination of parameters for the form of model that you have. That requires a bunch of numeric analysis.
Perhaps the following might help:
https://www.mathworks.com/matlabcentral/fileexchange/34795-outlier-detection-and-removal--hampel-
https://www.mathworks.com/matlabcentral/fileexchange/37211-remove-outliers
https://www.mathworks.com/matlabcentral/fileexchange/28501-tests-to-identify-outliers-in-data-series http://www.mathworks.com/help/signal/ref/hampel.html

请先登录，再进行评论。

更多回答（1 个）

Image Analyst
Image Analyst 2016-7-23
Let's say you have a step and you remove it. What will be in it's place? It can't be a step anymore because you removed it. So it will be smoother. It must be because there is no longer a step there. How can you remove a step without smoothing the data? Please show an illustration of that.
  1 个评论
Eugene Davis
Eugene Davis 2016-7-23
The step removal process knits the data back together. For a step I estimate the place and size of the step and then shift the data to the right of the step by the estimated size.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Linear Predictive Coding 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by