any faster way to load part of the mat file?

15 次查看（过去 30 天）
Yu Li
Yu Li 2017-2-20
编辑： Urs Hofmann 2019-7-15
hi: I have a large matfile ROP, with the size of 45*156*300000, but I only need part of the data with given i: ROP(i,:,:).
here I have two ways: 1)directly load the file. 2)use matfile commmand.
below is the code I used, and the tic-toc result:
tic
load ROP
toc
tic
m=matfile('ROP.mat');
test=squeeze(m.ROP(1,:,:));
toc
Elapsed time is 39.358792 seconds.
Elapsed time is 32.416581 seconds.
looks it does not make the loading significantly faster than before. and the only positive effect is that it saves the memory.
when I load a matrix same size with the ROP(1,:,:), it only cost about 3 seconds, which is about 10 times shorter.
so my question is: is there anyway to make the loading faster? while keep the same small memory like matfile command?
thanks!
Li
  1 个评论
José-Luis
José-Luis 2017-2-20
How do you create ROP? The fastest way would probably be to only to save what you need from the start.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Guillaume
Guillaume 2017-2-20
To efficiently use the benefits of matfile it is important to understand how matlab stores matrices in memory (and on file). Matlab stores all the rows of the first column of the first page continuously, then all the rows of the the second column of the first page, etc. until the last column of the first page, then it's the same with the second page. So the storage order is
rop(:, 1, 1), rop(:, 2, 1), ..., rop(:, end, 1), rop(:,1, 2), ..., rop(:, end, end)
With that in mind, using rop(x, :, :) is not efficient at all, since matlab needs to skip size(rop, 1) elements between each element you've requested. However, if you'd requested rop(:, :, x), matlab just needs to seek to the right location and then read the whole lot as one consecutive read. Much faster.
All, this to say, that if you want to speed up matfile read, you ought to change the order of the dimensions when you write that rop variable to file:
rop = permute(rop, [3 2 1]);
save('ROP.mat', 'rop');
m=matfile('ROP.mat');
test = m.ROP(:, :, 1)
  2 个评论
Yu Li
Yu Li 2017-2-20
hi: thanks for your reply, I tried this idea but looks still cost lot of time. below is the code, I load the ROP, re-arrange it and save to ROP1, and tried to load ROP1 with the data interested.
tic
load ROP
toc
tic
ROP1 = permute(ROP, [3 2 1]);
toc
tic
save ROP1 ROP1 -v7.3
toc
tic
m=matfile('ROP1.mat');
test=m.ROP1(:, :, 1);
toc
and here is the result:
Elapsed time is 39.310962 seconds.
Elapsed time is 5.730803 seconds.
Elapsed time is 134.724888 seconds.
Elapsed time is 31.991022 seconds.
however thanks your answer! best! Li
Urs Hofmann
Urs Hofmann 2019-7-15
编辑：Urs Hofmann 2019-7-15
For me it decreased the time required to read the matrix from 4.9 s to 2.0 s (on top comes 0.5 s for each loading for permuting and 3.5 s while saving).
% hugeMatrix has dimensions [3000, 1001, 1001] of type 'single')
% size: 11.2 Gb
% cropped version: 1.5 Gb
% permuted version
hugeMatrix = permute(hugeMatrix, [2, 3, 1]); % 3.5 sec
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all
m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(:, :, 1:400); % 2.0 sec
hugeMatrix = permute(hugeMatrix, [3, 1, 2]); % 0.5 s
% plain version
save('file.mat', 'hugeMatrix', '-nocompression', '-v7.3');
clear all
m = matfile('file.mat');
hugeMatrix = m.hugeMatrix(1:400, :, :); % 4.9 sec
Meaning: I only save once during a measurement and then load it multiple times, so permuted version is the way to go for me.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Software Development Tools 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by