Handling Big Data in MatLab

Question

Buddhi Wimarshana 2021-1-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/724023-handling-big-data-in-matlab

评论： Buddhi Wimarshana 2021-1-24

Hi everyone,

I am working with large sized data and the followings are the steps I follow, but need your advice/help to correct the procedure? or make it better...

I receive the data as csv files (data is of mixed types like doubles, date-time ets.) and the whole data set spans across about 150 csv files each having more than 100MB of data. So, I combined them all into one matlab table so the table contains the whole data set. I can't (don't like to really) make matlab tables for each csv file, becuase of the continuity of the data across each csv file so I guess it would be harder to retrieve them later. A part of the table I created looks like below:

This Table is more than 2GB in size (26 columns and millions of raws). Then I saved this Table as a mat file so I can 'load' it back to workspace later and retrieve the necessary data segments as per requirements. For example, I may need to retrive and post-process data of 'CurrentA' and 'VoltageV' for Cycle_Index=1 & Step_Index=30.

My concerens are:

this method is quite time taking, for example loading the mat file takes quite long, so wonder is there a better way? I saw using 'matfiles' we don't need to load the mat file to memory. But I am not sure this will allow me to retrieve data with much flexibility compared to original file loading to workspace method?
not so sure about the total methodology I followed, is there a better way to handle these kind of large data in a better, efficient way ?

Many thanks in advance...

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Steven Lord 2021-1-22

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/724023-handling-big-data-in-matlab#answer_604048

Some of the tools and techniques described in the Large Files and Big Data category in the documentation may be of interest to you.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Buddhi Wimarshana 2021-1-24

Thnk you so much Steven, went through this method before, then had a problem so went with the method I mentioned. I think I got more understanding on datastores now, seems it's the way to go!!

请先登录，再进行评论。