Suggestions on how to detect anomalies in around 200k time series, maybe with deep learning, maybe in a fast way?

Question

Sim 2022-6-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1743110-suggestions-on-how-to-detect-anomalies-in-around-200k-time-series-maybe-with-deep-learning-maybe-i

编辑： Sim 2022-6-21

I have tried Time Series Anomaly Detection Using Deep Learning, with just a few thousands of time series, but it is very slow...

Any suggestion to get anomalies detected with 200k time series in a faster way?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Eamonn 2022-6-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1743110-suggestions-on-how-to-detect-anomalies-in-around-200k-time-series-maybe-with-deep-learning-maybe-i#answer_988400

Hello

The DAMP algorithm (in Matlab) can do anomlay detecion at 100,000+ Hz

It is the only time series anomlay detecion algorthm to process a dataset with one trillion datapoints.

OH, and it beat deep learning on all the benchmarks.

It will be here https://www.cs.ucr.edu/~eamonn/MatrixProfile.html next week, but in the meantime, if you want the code/paper, I am happy to share it.

6 个评论
显示 4更早的评论隐藏 4更早的评论

Sim 2022-6-20

编辑：Sim 2022-6-20

在 MATLAB Online 中打开

ts.mat

Thanks a lot @Eamonn !

I have read the paper you have suggested and I have watched the youtube video: they are both very very interesting and promising! Given the benchamark tests I would prefer DAMP than deep learning methods, since (way much) simpler, and at the same time, accurate, robust and faster than deep learning... this is what I have understood from that paper (and others ones).

I have already tried DAMP on my machine, with one time series here attached

ts.mat

However, I was not able to use all the DAMP variants. Before accepting the Answer, I would like to make all the DAMP functions work...

It looks like that the function DAMP works....

load('/Users/.../ts.mat', 'w3')
>> DAMP(w3)
Pruning Rate: 0
Predicted discord score/position: 4.017/1021

...as well as the DAMP_X_Lag_Amnesic function...

>> DAMP_X_Lag_Amnesic(w3)
Pruning Rate: 0
Predicted discord score/position: 4.017/1021

...as well as the DAMP_fullMP function....

>> DAMP_fullMP(w3);
Pruning Rate: 0.9274
Predicted discord score/position: 4.9389/844

...but, unfortunately, the DAMP_topK function gives me an error:

>> K = 5
K =
     5
>> DAMP_topK(w3,K)
Index exceeds the number of array elements. Index must not exceed 1056.
Error in DAMP_topK (line 26)
        Left_MP(i) = min( MASS_V2(T(1:i), T(i:i+SubsequenceLength-1)));

Do you know how I could solve that error ?

Again, many thanks for informing me about this algorithm!

Sim 2022-6-21

在 MATLAB Online 中打开

Thanks @Eamonn, I think I will drop you an email later...

Meanwhile, just for a sake of completeness, I would like to inform you that I might have found "a way" to overtake the error coming out when using DAMP_topK. Indeed, I changed

CurrentIndex = 1000+1; % line 9 of DAMP_topK

into

CurrentIndex = 24;

getting this figure

However, if I change that parameter into

CurrentIndex = 50;

I get the following:

By changing CurrentIndex, the predicted scores/positions change a bit (i.e. the first two "peaks" in the previous case, i.e. with CurrentIndex = 24 disappear and, obviously, two "lower peaks" appear with CurrentIndex = 50, in order to always have K = 5)...

Therefore, what is the right value for the variable CurrentIndex ?

Sim 2022-6-21

编辑：Sim 2022-6-21

ts.mat

@Eamonn, email just sent to you :-)

P.S.: just in case I attach here again the ts.mat file (including both "w3" and "w4" time series)

请先登录，再进行评论。

Answer 2

Image Analyst 2022-6-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1743110-suggestions-on-how-to-detect-anomalies-in-around-200k-time-series-maybe-with-deep-learning-maybe-i#answer_988265

Are you talking about training or prediction?

For what it's worth, here is another anomaly detection demo from Mathworks.

https://www.mathworks.com/videos/anomaly-detection-for-industrial-processes-and-machinery-with-matlab-1648135588169.html?s_tid=srchtitle

Anomaly Detection for Industrial Processes and Machinery with MATLAB Video

Learn how to apply statistical and machine learning based anomaly detection techniques to industrial processes and machinery.

www.mathworks.com

3 个评论
显示 1更早的评论隐藏 1更早的评论

Image Analyst 2022-6-19

Well of course you could always get a faster computer with more memory. Get the best video adapter you can afford with the most GPU memory you can afford. And get the Parallel Processing Toolbox, which will speed it up quite a bit (if you end up having to use the CPU because your GPU does not have enough memory and you get an "out of memory" error). The GPU does not know how to use disk space to allocate additional memory (everything happens on the GPU) but the CPU can get as much virtual memory from the disk drive as it needs (or that you allow).

Sim 2022-6-20

Thanks a lot @Image Analyst, in this moment I do not have access to faster machines, but I could try with the parallel processing toolbox. However, by considering the paper/s showed by @Eamonn, it looks like that (for time series) deep learning is outperformed by another algorithm based on the Matrix Profile :-)

请先登录，再进行评论。