Iteratively save data to mat file

11 次查看（过去 30 天）
MCM
MCM 2018-10-10
评论： Walter Roberson 2018-10-11
I have to read, process and plot data from 700 text files. I want to save the processed data from one text file to a Mat file before moving on to the next text file. This is a precaution in case of error. I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently.
If I use save('myMatFile.m','x') to save the data from each file, then I will need to create 700 variables in the Mat file; not ideal. If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal. Originally, I use dlmwrite to export to a text file but that is slow.
Also, I'd like to know if my approach is wrong and there is a better way to do this. ANYTHING to speed up this incredibly slow IO process.

请先登录，再回答此问题。

回答（2 个）

Stephen23
Stephen23 2018-10-11
编辑：Stephen23 2018-10-11
"I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently."
"If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal."
Use matfile, that is exactly what it was designed for.
  2 个评论
MCM
MCM 2018-10-11
When I said, "if I append a single variable in the Mat file then Matlab partially loads it each time; not ideal", I was referring to the matfile function.
When I append using matfile, Matlab gives me a warning that the function will partially read the MAT file in order to do the append
Walter Roberson
Walter Roberson 2018-10-11
Consider that if you were changing an existing variable, then MATLAB would have to locate the variable in the file, figure out its size, and make a decision about whether to overwrite in place or to delete the variable and write a new one.
When you append a new variable, MATLAB must first search the .mat file to determine that the variable is not already present, after which it makes whatever changes it needs to the .mat file to write the new variable.
The only way that these processes could be done without partially reading the .mat file would be if every change made by matFile were by simply writing new information on the end without looking inside the file to see what was already there. Not impossible, but it would have the consequence that reading the .mat file would require scanning from the beginning to the end for everything, just in case there was an update later on in the file that overwrote the existing information. It could be done that way, but it would make every read inefficient.

请先登录，再进行评论。

gonzalo Mier
gonzalo Mier 2018-10-10
First, use the -nocompression option for the save operator if you have a lot of info.
Maybe you could improve your performance writing the info in 700 different mat files as save("mat_"+num2str(i)+".m") so you don't have to read the info each time.
  1 个评论
Walter Roberson
Walter Roberson 2018-10-11
I recently posted a timing test using matFile with and without compression. For the rand() that the user was working with, turning off compression made a big difference. It is plausible that in some other cases like a mostly 0 matrix that compression might be more efficient due to reduced disk i/o.
Be sure to save with -v7.3 to use matFile efficiently

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Workspace Variables and MAT-Files 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by