Signal processing timetable data in segments defined by time ranges

Kevin Williams

2023 10 18

0 个回答

10 次查看（30 天）

0 个投票

Hi,

I have large recordings stored as tall timetables on which I would like to do signal processing in consecutive time range blocks.

Big data processing is new to me, and would like to ask if there is an efficient way to do this with mapreduce, or some other technique? (I am currently selecting each subset at a time, processing, and storing the result, but this is taking long.)

I've been thinking that writing a "readfcn()" that gets the time period number might be the way, but its not clear how to do this.

Could anyone please give me some advice?

Many thanks, Kevin

7 个评论
显示 5更早的评论隐藏 5更早的评论

Kevin Williams 2023-10-19

Experimenting with mapreduce, I see that the mapper gets blocks of a certain number of rows (I get 65532).

This results in some "leftover" rows at the end of the block that don't necessarily form a complete burst. (I am trying to process my recording in frames of a fixed number of samples.)

I noticed the "CustomDatastore" has a read() method that needs to be implemented, and wondered whether this could be used.

However, I'm not sure this is the best way to do this burst processing? (The statistics I need are things like the peak power, average power, etc. in the burst.)

Sam Marshalik 2023-10-19

If you have a tall array, are you able to use overloaded functions that works on tall to do your analysis? If so, you can leverage Parallel Computing Toolbox or MATLAB Parallel Server to speed up that analysis to run simultenously across multiple cores on one or multiple machines.

请先登录，再进行评论。

请先登录，再回答此问题。

Follow Question