any faster way to load part of the mat file?

Question

Yu Li 2017-2-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/325978-any-faster-way-to-load-part-of-the-mat-file

编辑： Urs Hofmann 2019-7-15

hi: I have a large matfile ROP, with the size of 45*156*300000, but I only need part of the data with given i: ROP(i,:,:).

here I have two ways: 1)directly load the file. 2)use matfile commmand.

below is the code I used, and the tic-toc result:

tic
load ROP
toc
tic
m=matfile('ROP.mat');
test=squeeze(m.ROP(1,:,:));
toc
Elapsed time is 39.358792 seconds.
Elapsed time is 32.416581 seconds.

looks it does not make the loading significantly faster than before. and the only positive effect is that it saves the memory.

when I load a matrix same size with the ROP(1,:,:), it only cost about 3 seconds, which is about 10 times shorter.

so my question is: is there anyway to make the loading faster? while keep the same small memory like matfile command?

thanks!

Li

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

José-Luis 2017-2-20

How do you create ROP? The fastest way would probably be to only to save what you need from the start.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Guillaume 2017-2-20

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/325978-any-faster-way-to-load-part-of-the-mat-file#answer_255549

在 MATLAB Online 中打开

To efficiently use the benefits of matfile it is important to understand how matlab stores matrices in memory (and on file). Matlab stores all the rows of the first column of the first page continuously, then all the rows of the the second column of the first page, etc. until the last column of the first page, then it's the same with the second page. So the storage order is

rop(:, 1, 1), rop(:, 2, 1), ..., rop(:, end, 1), rop(:,1, 2), ..., rop(:, end, end)

With that in mind, using rop(x, :, :) is not efficient at all, since matlab needs to skip size(rop, 1) elements between each element you've requested. However, if you'd requested rop(:, :, x), matlab just needs to seek to the right location and then read the whole lot as one consecutive read. Much faster.

All, this to say, that if you want to speed up matfile read, you ought to change the order of the dimensions when you write that rop variable to file:

rop = permute(rop, [3 2 1]);
save('ROP.mat', 'rop');
m=matfile('ROP.mat');
test = m.ROP(:, :, 1)

2 个评论
显示无隐藏无

Yu Li 2017-2-20

在 MATLAB Online 中打开

hi: thanks for your reply, I tried this idea but looks still cost lot of time. below is the code, I load the ROP, re-arrange it and save to ROP1, and tried to load ROP1 with the data interested.

tic
load ROP
toc
tic
ROP1 = permute(ROP, [3 2 1]);
toc
tic
save ROP1 ROP1 -v7.3
toc
tic
m=matfile('ROP1.mat');
test=m.ROP1(:, :, 1);
toc

and here is the result:

Elapsed time is 39.310962 seconds.
Elapsed time is 5.730803 seconds.
Elapsed time is 134.724888 seconds.
Elapsed time is 31.991022 seconds.

however thanks your answer! best! Li

Urs Hofmann 2019-7-15

编辑：Urs Hofmann 2019-7-15

在 MATLAB Online 中打开

For me it decreased the time required to read the matrix from 4.9 s to 2.0 s (on top comes 0.5 s for each loading for permuting and 3.5 s while saving).

% hugeMatrix has dimensions [3000, 1001, 1001] of type 'single')
% size: 11.2 Gb
% cropped version: 1.5 Gb 
% permuted version
hugeMatrix = permute(hugeMatrix, [2, 3, 1]); % 3.5 sec
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all
m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(:, :, 1:400); % 2.0 sec
hugeMatrix = permute(hugeMatrix, [3, 1, 2]); % 0.5 s
% plain version
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all
m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(1:400, :, :); % 4.9 sec