Speed up loading a single variable from a large mat file

53 次查看（过去 30 天）
Wonsang You
Wonsang You 2017-7-18
评论： Walter Roberson 2018-9-25
I was trying to load a single variable 'data' from a large .mat file (>2GB) using the following command in MATLAB R2016b on Mac OS.
data = load('db.mat','-mat','data');
However, it was too slow to load the single variable. Note that the variable 'data' is not a matrix but a structure. I already refer to the previous discussion but it didn't give solution. Could anyone please suggest an alternative way to speed up loading a single variable from a large mat file? Thank you for your help in advance.
  1 个评论
per isakson
per isakson 2017-7-18
编辑：per isakson 2017-7-18
  • ">2GB" means version '-v7.3', I assume.
  • '-nocompression' improves speed, but not that much
  • AFAIK: '-v7.3' is slow. (I didn't test that since R2013a.)
  • Try HDF5 files with 'ChunkSize' Default: Not chunked

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Jan
Jan 2017-7-18
Using a large MAT file and extracting a single variable takes time. There is no magic acceleration. The compression of the data of v7.3 files requires to decompress the complete or at least a large part of the data to extract you variable.
The clean solution would be not to use MAT files for this job.
  2 个评论
Rahimeh Rouhi
Rahimeh Rouhi 2018-9-25
I have the same problem. You are right, using Matfile is time-consuming. How can I fix the problem?
Thank you for your guidance.
Walter Roberson
Walter Roberson 2018-9-25
With -v7.3 mat files there is now a -nocompression option
https://www.mathworks.com/help/matlab/ref/save.html
This applies only for -v7.3 .
It is possible to have a -v7 (not -v7.3) .mat file that is over 2 gigabytes as long as any one variable is not over 2 gigabytes. However, compression would happen automatically for -v7 mat files as well.
Sometimes what helps is to use matfile with -v7.3 files . This can help in accessing members of a struct array. Reading between the lines, it might be able to load a single field of a struct array, but anything below that would be loaded as a whole.
For pure numeric arrays sometimes it makes sense to write them as pure binary files and memmapfile() . This can also be done for struct arrays, but in each case the data written per entry must be fixed size.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Structures 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by