Saving large variables with save -v7 instead of -v7.3

403 次查看（过去 30 天）
David
David 2020-3-4
评论： Rik 2020-3-27
Question: Is there a better way to save large variables?
The problem of saving large variables is a rapidly growing problem. In MATLAB, the save function has the option of saving as v7.3, which uses HDF5 to allow the storage of variables larger than 2 GB. However, saving and loading take much longer with this option than with v7 (the default). I know I am not the only one with this problem - I found several other very similar questions (here and here, and more that I couldn't find again quickly).
I have a poor-practice solution, and I would like to know if there is a more elegant solution out there. My code is below, and here is my description. I want to save a cell array of large matricies. Each matrix is below the 2 GB limit, but the cell array is not. I want to make a cell array of handles instead, which would be very small but still permit indexing of the entire set of data. I have done that by making a cell array of variable names and using the infamous eval function to use the character vectors as handles.
My experiment produces data named "img01", "img02", ... Each image is about 300 MB. I would like to have a cell array with temperature, pressure, other variables for the dimensions, and each cell is a handle to the "img" file taken under those conditions. It would also be nice to load the workspace in a reasonable amount of time, i.e. not use -v7.3.
var1 = rand(1e4, 1e4); % 0.8 GB
var2 = rand(1e4, 1e4);
var3 = rand(2e4, 1e4); % 1.6 GB
var4 = rand(1e4, 2e4);
var5 = rand(1e4, 2e4);
data_all = {var1, var2, var3, var4, var5}; % 6.4 GB
data_handles = {'var1', 'var2', 'var3', 'var4', 'var5'};
%%% Check variable size
% whos
slice_data_all = data_all{2}(:,3);
var_data_handles = eval(data_handles{2});
slice_data_handles = var_data_handles(:,3);
%%% Check that the slices are the same
% sum(slice_data_all == slice_data_handles, 'all')
tic
save('SingleLargeVariable.mat', 'data_all', '-v7.3');
toc
tic
save('ManyVariables.mat', 'data_handles', 'var1', 'var2', 'var3', 'var4', 'var5', '-v7');
toc
Elapsed time is 144.252925 seconds.
Elapsed time is 105.970389 seconds.
In this example, saving with -v7 didn't actually save that much time, but I believe it saves much more when the variables are more compressible. I couldn't figure out how to test this easily. My actual data has many repeated values.
Another option would be to save the data in a binary file instead of a MAT file. Is this a good direction to go? I don't know much about this.
Another option is to save the individual variables, as in my example, but instead of the cell array of character vectors, save a string with the command to package all the variables back into a cell array. However, this seems likely to take more time than it saves by avoiding v7.3.
I am using MATLAB 2019b.
  2 个评论
David
David 2020-3-20
Thanks Rik, that looks interesting. Looks like ImageDataStore and datastore were designed to help with RAM problems, but it might help with data storage as well.
It looks like for my purposes, the fastest solution is to run the code that generated my variables every time I want to load that data. It is faster to read the raw data and redo the analysis than it is to open the MAT file with the data analyzed.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Moe_2015
Moe_2015 2020-3-20
Hi David,
In 2019b you can actually choose to use the v7.3 without any compression. If storage is not an issue, you can add the -nocompression flag to your input to save and that should make loading the file faster than just the v7.3 with compression.
save('SingleLargeVariable.mat', 'data_all', '-v7.3','-nocompression');
  4 个评论
显示 2更早的评论
Moe_2015
Moe_2015 2020-3-27
编辑：Moe_2015 2020-3-27
Actually for the -v7.3 option, -nocompression was introduced in R2017a according to the release notes. But for -v7, yes it was R2019b. Either way though, it won't help out Bob, sorry.
Rik
Rik 2020-3-27
Good catch, thanks for the correction. I guess I need to read more carefully next time.

请先登录，再进行评论。

Walter Roberson
Walter Roberson 2020-3-20
Sometimes it can be worth using a serialization tool and fwrite() the results to a binary file. For long term storage, the resulting file could potentially be compressed afterwards.
https://www.mathworks.com/matlabcentral/fileexchange/34564-fast-serialize-deserialize

类别

在 Help Center 和 File Exchange 中查找有关 HDF5 的更多信息

标签

产品


版本

R2019b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by