How to save a huge matrix efficiently?

28 次查看（过去 30 天）
DaWo
DaWo 2016-11-25
评论： DaWo 2016-11-28
Hello,
I want to save a huge matrix into a mat.file.
V = zeros(2000,2000,2000,'uint8');
loop filling the matrix;
save('test.mat','V','-v7.3');
clear all
This works very well! Then I want to get some parts of the variable V in the mat file.
data = matfile('test.mat');
V = data.V(:,:,1); % very slow ~20sec
V = data.V(:,1,:); % very slow ~20sec
V = data.V(1,:,:); % very slow ~20sec
V = data.V(1:500,1:500,1:500); % very fast although more bytes are extracted?! ~ 2sec
So now my question: Is there a possibility to save the matrix in an efficient way to get the slices faster? I already have some huge mat-files, where I can get slices of the volume very fast. This mat-files have the same properties like mine. They are a download, so I have no idea of the making. What I know is the structure of the variable and the type of saving (2182x2182x2242 uint8 and -v7.3). So I guess, it's possible to save the matrix elements in a more efficient way.
I hope you can help me. Best regards and thank you in anticipation!
PS: Why I need that? I'd seen, that normaly it is very fast to get a part of the matfile. So I can reduce working memory. Unfortunately I need this high resolution.
  1 个评论
Walter Roberson
Walter Roberson 2016-11-27
On my system I was able to create a 2000 by 2000 by 2000 uint8 matrix, barely. But saving it with save() was just toooo slow and I had to give it up.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Daniel kiracofe
Daniel kiracofe 2016-11-27
If you have enough memory and are using a 64 bit version of matlab, using load() to load the entire matrix into memory will be quicker. it will take some time to load the file, but after that each slice will be quicker.
If the matrices are sparse (i.e. they contain mostly zeros with a few non-zero values) then saving the file as a sparse matrix would be significantly quicker (https://www.mathworks.com/help/matlab/ref/sparse.html)
Beyond that, I don't really know any way to make matfile() run quicker. In order to get the slices, matfile() basically has to read the entire file. And that's just going to be slow because the file is big.
  1 个评论
DaWo
DaWo 2016-11-28
Thanks for the answer. The load function isn't my favorite solution, but for now it should work. ;)

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Debugging and Analysis 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by