Importing data from a large data set

Question

0 个投票

I have a ~30 GB worth of data in a .mat file. The .mat file has m*n (m = ~10^9, n = 10).

I need to run a simulation that derives its value from randomly selected row (from a choice of row 1:m).

The column (1:n) has the simulation parameters.

What is the best way (memory and speed) to do it? Is it possible to do it without loading/importing the entire .mat file in Workspace?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

Answer 1

Ameer Hamza 2020-3-31

0 个投票

Yes, you can read data from a mat file without loading it completely. See matfile()https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html

hal9k 2020-4-1

Here is what worked for me.

tic

importData = matfile(filename);

data_info = whos('-file',filename);

row = data_info.size(1);

rand_row= randi([1 row],1);

output = importData.filename(rand_row,:)

toc

Elapsed time = 0.02 sec.

This is great.

Ameer Hamza 2020-4-1

Glad to be of Help.