Importing data from a large data set

1 次查看（过去 30 天）
hal9k
hal9k 2020-3-31
评论： Ameer Hamza 2020-4-1
I have a ~30 GB worth of data in a .mat file. The .mat file has m*n (m = ~10^9, n = 10).
I need to run a simulation that derives its value from randomly selected row (from a choice of row 1:m).
The column (1:n) has the simulation parameters.
What is the best way (memory and speed) to do it? Is it possible to do it without loading/importing the entire .mat file in Workspace?

请先登录，再回答此问题。

采纳的回答

Ameer Hamza
Ameer Hamza 2020-3-31
Yes, you can read data from a mat file without loading it completely. See matfile()https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html
  2 个评论
hal9k
hal9k 2020-4-1
Here is what worked for me.
tic
importData = matfile(filename);
data_info = whos('-file',filename);
row = data_info.size(1);
rand_row= randi([1 row],1);
output = importData.filename(rand_row,:)
toc
Elapsed time = 0.02 sec.
This is great.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Data Import and Analysis 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by