Slow down of save(..., '-append')

Question

Xinyi Shen 2016-4-12

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/278644-slow-down-of-save-append

回答： Walter Roberson 2016-4-12

Hi there, I am currently using the v7.3 mat file to store hourly hydrologic simulation results. In the simulation we generate a few million variables so that I have to write them to monthly file to maintain a reasonable number files. In each monthly file, I have 24*30*10 variables. I am using

if exist(fileName,'file')~=7
   save (fileName,varName,'-v7.3');
else
   save (fileName,varName,'-append'); 
end

to save the multi-variable file because I figure out it is faster than

S=matfile(fileName,'Writable',true);
cmd=['S.',varName,'=',varName];
eval(cmd);

With the number of variables increasing, the writing speed slows down greatly. I understand that matlab needs to check the existence of a variable in the file before writing and the checking time is longer when the number of variables is larger. However, in my case, this checking is redundant because my model can ensure that each variable I am writing does not exist in the file. I am wondering if there is a way to avoid this time consuming checking when I use the save command with '-append' option.

Thanks!

5 个评论
显示 3更早的评论隐藏 3更早的评论

Walter Roberson 2016-4-12

Xinyi commented to dpb:

I am not generating dynamic variables but dynamic files. Each variable remains unchanged in the file. It is impossible for me to put them all in memory and write them to the disk once for all because the RAM is also limited.

jgg 2016-4-12

编辑：jgg 2016-4-12

Why are you creating so many variables? Why not create a single matrix to store them instead? This would completely avoid your problem. if it's a RAM issue, use several matrices or a struct instead which doesn't use continguous memory.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2016-4-12

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/278644-slow-down-of-save-append#answer_217631

在 MATLAB Online 中打开

Do not use eval() . At most,

S.(varName_as_string) = varName_not_as_string

Secondly: save -append does not remove any content from the file, except possibly if the variable happens to be the very last thing in the file. Instead, it just marks any existing variable of that name as being unused, leaving it in the file. This might not matter to you if you are never replacing variables. Using -append to add new variables might result in a file that is not as efficient to access as if it had all been written at one time. MATLAB does need to examine the entire file to determine whether there is a variable of that name; it will not "take your word for it"

Third: matFile is the proper way to do these kinds of updates of a .mat file . I would recommend you matFile only once and leave it existing until you no longer need it, rather than calling it each time you want to add a variable.

Fourth: If your variables are a consistent size, consider using binary files, possibly together with memmapfile. If you need .mat files specifically for later use, it can still make sense from an efficiency standpoint to write binary files during the processing stage, and then later go back and convert the binary files into .mat files... at which time you already have all the data available and so can use a single save() to put it into the file.