How to deal with a large amount of data in MatLab and create individual data sets each 30000 elements

13 次查看（过去 30 天）
Roderick
Roderick 2020-2-17
评论： Roderick 2020-2-21
Hi everyone,
I have a huge data set in .dat format, of about 5 GB. Of course, my computer takes a lot of time to deal with this archive if I use load to try to read it, as I usually do with shorter files. The thing is: in this file, each 30000 rows constitutes the data from a certain time step. So maybe a good way to deal with this is create through MatLab as much new files as needed, each 30000 rows, to minimize the loading time, maybe using a string in the name of each of the new created data sets to index in which time step you are. From example, the first 30000 rows, corresponding to t1, the next ones to t2... What do you think that I can do? This will reduce the loading time or it is not even to work due to the amount of data, at the end, is the same?
Thank you for your attention!

请先登录，再回答此问题。

采纳的回答

Raunak Gupta
Raunak Gupta 2020-2-21
Hi,
From the size of dataset, I can see that the dataset involves a huge number of rows. I suggest using tall array if you want the data to be loaded without an out-of-memory error and can use gather for getting specific rows. You may look here, for how the out-of-memory data is handled in tall array.
As it is mentioned that each 30000 rows corresponds to a time instance, you may want to create separate data files for these chunks but if it is required to load all the files in a program, tall array would be a better alternative.
  1 个评论
Roderick
Roderick 2020-2-21
Hey! Thank you. I have already find a way, but due to its not very fast computational time, I think that it is not suitable to write it here for future readers. I think that your answer opens the proper way, so I think your comment is the one that close this thread.
Cheers.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Data Import and Analysis 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by