Large data file I/O

Question

Paul 2012-1-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/26362-large-data-file-i-o

I am trying to speed up a bottle neck in our code. Currently, the output of one of our FORTRAN modules writes the data to a text file. The text file is saved as .m file. The m files is then loaded into MATLAB. An example would be

function x = my_data(x)
x.time = [LARGE AMOUNT OF DATA]

Now this read operation seems to cause MATLAB to run out of memory. The MATLAB help file suggests storing large amounts of data in MAT files because they are optimized for read/write operations and compress data as well. They say this is better than using low-level file I/O such as fopen. But since our data is being read as an “.m” is it still using such file I/O. My question is should we take the time to try to write that data from FORTRAN as a mat file instead of a “.m”.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2012-1-17

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/26362-large-data-file-i-o#answer_34422

在 MATLAB Online 中打开

It appears to me that you would be better off writing a binary file. The speed might not be as good as a .mat file (because of no compression), but Fortran should have no problem writing a binary file whereas bringing in the .mat format can be a nuisance.

I would also suggest that it would be faster if you used

x.time = my_data(); %no argument

and

function times = my_data
  times = fread(...);
end

This would avoid having to make a copy of x.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Paul 2012-1-17

Thanks for the suggestion. I will give it a try. You are absolutely right about .mat being a nuisance. I don't want to re-write and re-compile our FORTRAN code to output .mat. However, changing the output to binary may work as it is a smaller change in the FOTRAN code.

请先登录，再进行评论。