Speed up loading a single variable from a large mat file

Question

Wonsang You 2017-7-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/349277-speed-up-loading-a-single-variable-from-a-large-mat-file

评论： Walter Roberson 2018-9-25

I was trying to load a single variable 'data' from a large .mat file (>2GB) using the following command in MATLAB R2016b on Mac OS.

data = load('db.mat','-mat','data');

However, it was too slow to load the single variable. Note that the variable 'data' is not a matrix but a structure. I already refer to the previous discussion but it didn't give solution. Could anyone please suggest an alternative way to speed up loading a single variable from a large mat file? Thank you for your help in advance.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

per isakson 2017-7-18

编辑：per isakson 2017-7-18

">2GB" means version '-v7.3', I assume.
'-nocompression' improves speed, but not that much
AFAIK: '-v7.3' is slow. (I didn't test that since R2013a.)
Try HDF5 files with 'ChunkSize' Default: Not chunked

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2017-7-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/349277-speed-up-loading-a-single-variable-from-a-large-mat-file#answer_274676

Using a large MAT file and extracting a single variable takes time. There is no magic acceleration. The compression of the data of v7.3 files requires to decompress the complete or at least a large part of the data to extract you variable.

The clean solution would be not to use MAT files for this job.

2 个评论
显示无隐藏无

Rahimeh Rouhi 2018-9-25

I have the same problem. You are right, using Matfile is time-consuming. How can I fix the problem?

Thank you for your guidance.

Walter Roberson 2018-9-25

With -v7.3 mat files there is now a -nocompression option

https://www.mathworks.com/help/matlab/ref/save.html

This applies only for -v7.3 .

It is possible to have a -v7 (not -v7.3) .mat file that is over 2 gigabytes as long as any one variable is not over 2 gigabytes. However, compression would happen automatically for -v7 mat files as well.

Sometimes what helps is to use matfile with -v7.3 files . This can help in accessing members of a struct array. Reading between the lines, it might be able to load a single field of a struct array, but anything below that would be loaded as a whole.

For pure numeric arrays sometimes it makes sense to write them as pure binary files and memmapfile() . This can also be done for struct arrays, but in each case the data written per entry must be fixed size.

请先登录，再进行评论。