Synchronizing several large timetables

Question

Lutetium 2021-3-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/777052-synchronizing-several-large-timetables

评论： Seth Furman 2021-4-1

Dear community,

while I'm waiting for my synchronization to finish, I'm wondering if there is any faster way perform below synchronization. I had to use tall() to avoid running out of memory.

In my example, I have around 19 timetable with timevector and corrsponding signal. I take my first timetable and put it into variable TT1. Then, I put the second table in TT2 and synchronize TT1 and TT2. The resulting table is then transferred into TT1 and I synchronize the next timetable.

While this certainly works, it will take on the current speed (see screenshot below) approx 15 hours or more. Is this a very inefficient way? Is there a way to speed this up significantly? Any obvious mistakes?

I appreciate your input (This is local parallel pool with 16 workers, Xeon E-2286 2.4 GHz, 32 GB RAM)

TT1=tall(tbl{1,1});
for i=2:sigcount
    
    TT2=tall(tbl{1,i});
    TTSync=synchronize(TT1,TT2,'union','nearest');
    TT1=TTSync;
end
location = 'E:\Matlab\Analysis';
write(location,TT1);
clearvars;

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Seth Furman 2021-3-18

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/777052-synchronizing-several-large-timetables#answer_651917

在 MATLAB Online 中打开

Where are your timetables coming from? Are you reading them from one or more files? If so, it probably makes more sense to create each tall timetable from a datastore rather than an in-memory timetable. See the link below for an example.

https://www.mathworks.com/help/matlab/import_export/tall-arrays.html

Assuming tblTall is a cell array of tall timetables, we can pass all tall timetables to synchronize at the same time as follows

TTSync = synchronize(tblTall{:,:},'union','nearest');

Please let me know if you are able to try this and, if so, whether you see better performance.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Seth Furman 2021-4-1

在 MATLAB Online 中打开

It sounds like you are creating a tall cell array of non-tall timetables, e.g.

>> tblCell = {timetable(1,'SampleRate',1),timetable(2,'SampleRate',1)};
>> tblTall = tall(tblCell)
tblTall =
  1×2 tall cell array
    {1×1 timetable}    {1×1 timetable}
>> tblTall{:,:}
Indexing expressions of the form T{...,...} are not supported for tall arrays.

Instead, what you want is to create a non-tall cell array of tall timetables, e.g.

>> tblTall = cellfun(@tall,tblCell,'UniformOutput',false)
tblTall =
  1×2 cell array
    {1×1 tall}    {1×1 tall}
>> tblTall{:,:}
ans =
  tall timetable
    Time     Var1
    _____    ____
    0 sec     1  
ans =
  tall timetable
    Time     Var1
    _____    ____
    0 sec     2  
    
>> synchronize(tblTall{:,:})
ans =
  1×2 tall timetable
    Time     Var1_1    Var1_2
    _____    ______    ______
    0 sec      1         2   

Please let me know whether you notice any improvement in performance with this workflow.

Seth Furman 2021-4-1

在 MATLAB Online 中打开

Please also clarify, does code like the following work on your machine with your timetables or do you get errors about running out of memory?

>> tblCell = {timetable(1,'SampleRate',1),timetable(2,'SampleRate',1)};
synchronize(tblCell{:})
ans =
  1×2 timetable
    Time     Var1_1    Var1_2
    _____    ______    ______
    0 sec      1         2   