Dealing with large timetables
显示 更早的评论
I generated two timetable [283580428x5] and [21317257x2] and have also written them to a file using writetimetable().
Now I would like to synchronize them using synchronize [table1, table 2] but I get 'out of memory' very soon after executing the command.
Is there a way to process these table blockwise or any advice how I can handle these kind of large timetable efficiently w/o running out of memory? I appreciate any help or guidance
采纳的回答
Star Strider
2020-10-28
0 个投票
The only option I can suggest to you is in the timetable documentation section on Extended Capabilities under Tall Arrays. (I have no personal experience with huge data sets.)
4 个评论
Lutetium
2020-10-28
Thanks, thats actually what I need.
However, it seems that I keep running into problems.
I used the write to writetimetable to write my as *csv. I then used datastore to read it in
ds1 = tabularTextDatastore('file.csv')
convert it to a tall array
tt = tall(ds1)
Thats how the data looks like in the preview which seems fine.
Then I try to get a timetable again but timevector is not recognized. Any idea how the syntax needs to be in this case? I have googled/tried a lot but wasnt able to recover my timetable.
TT = timetable(rowTimes, tt.Var1, tt.Var2, ...)
The conversion from the original variable (in-memory timetable) works perfectly fine
TT = tall(tt).
Star Strider
2020-10-29
As always, my pleasure!
I am not certain what the problem could be. If you are having problems importing the time vector, it could be a problem with the format, and I am not certain how to solve that. It might be better to store it as a more conventional datetime or duration object that MATLAB expects, rather than an unusual format that may require tweaks to the detectImportOptions options structure. That is a useful function, however it can be frustrating (for me at least) to get to work correctly.
For the time being, unless it needs to be read by software other than MATLAB, it might be best to save it as a .mat file, since that eliminates the format problem. Be sure to back it up to another computer or the MathWorks Cloud (see Use Files on the Cloud for more information). I have no experience with MATLAB Drive either, however with a sufficiently large file, it is always best to back it up, and that is a convenient option.
Lutetium
2020-10-29
Star Strider
2020-10-29
As always, my pleasure!
I have obnly used the datastore function once, and forgot to mention it as an option.
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Tall Arrays 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
