Synchronizing several large timetables
显示 更早的评论
Dear community,
while I'm waiting for my synchronization to finish, I'm wondering if there is any faster way perform below synchronization. I had to use tall() to avoid running out of memory.
In my example, I have around 19 timetable with timevector and corrsponding signal. I take my first timetable and put it into variable TT1. Then, I put the second table in TT2 and synchronize TT1 and TT2. The resulting table is then transferred into TT1 and I synchronize the next timetable.
While this certainly works, it will take on the current speed (see screenshot below) approx 15 hours or more. Is this a very inefficient way? Is there a way to speed this up significantly? Any obvious mistakes?
I appreciate your input (This is local parallel pool with 16 workers, Xeon E-2286 2.4 GHz, 32 GB RAM)
TT1=tall(tbl{1,1});
for i=2:sigcount
TT2=tall(tbl{1,i});
TTSync=synchronize(TT1,TT2,'union','nearest');
TT1=TTSync;
end
location = 'E:\Matlab\Analysis';
write(location,TT1);
clearvars;
采纳的回答
Seth Furman
2021-3-18
Where are your timetables coming from? Are you reading them from one or more files? If so, it probably makes more sense to create each tall timetable from a datastore rather than an in-memory timetable. See the link below for an example.
Assuming tblTall is a cell array of tall timetables, we can pass all tall timetables to synchronize at the same time as follows
TTSync = synchronize(tblTall{:,:},'union','nearest');
Please let me know if you are able to try this and, if so, whether you see better performance.
3 个评论
Lutetium
2021-3-30
Seth Furman
2021-4-1
It sounds like you are creating a tall cell array of non-tall timetables, e.g.
>> tblCell = {timetable(1,'SampleRate',1),timetable(2,'SampleRate',1)};
>> tblTall = tall(tblCell)
tblTall =
1×2 tall cell array
{1×1 timetable} {1×1 timetable}
>> tblTall{:,:}
Indexing expressions of the form T{...,...} are not supported for tall arrays.
Instead, what you want is to create a non-tall cell array of tall timetables, e.g.
>> tblTall = cellfun(@tall,tblCell,'UniformOutput',false)
tblTall =
1×2 cell array
{1×1 tall} {1×1 tall}
>> tblTall{:,:}
ans =
tall timetable
Time Var1
_____ ____
0 sec 1
ans =
tall timetable
Time Var1
_____ ____
0 sec 2
>> synchronize(tblTall{:,:})
ans =
1×2 tall timetable
Time Var1_1 Var1_2
_____ ______ ______
0 sec 1 2
Please let me know whether you notice any improvement in performance with this workflow.
Seth Furman
2021-4-1
Please also clarify, does code like the following work on your machine with your timetables or do you get errors about running out of memory?
>> tblCell = {timetable(1,'SampleRate',1),timetable(2,'SampleRate',1)};
synchronize(tblCell{:})
ans =
1×2 timetable
Time Var1_1 Var1_2
_____ ______ ______
0 sec 1 2
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Tall Arrays 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
