Changing large matrices by not completely loading them into memory

Question

Moritz 2015-6-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/224395-changing-large-matrices-by-not-completely-loading-them-into-memory

评论： Walter Roberson 2015-6-18

Hi,

I'm attempting to modify very large matrices (single, 50e3 x 50e3), which don't make sense to load into the memory. I was wondering what you could recommend me as a data handling strategy? I thought ideally I could always load a let us say 100x100 square modify it and write it back. My working machine uses a SSD connected via M2 so it should be relatively speedy (however of course not nearly as fast as RAM). What suggestions do you have?

Thanks,

Moritz

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Stephen23 2015-6-18

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/224395-changing-large-matrices-by-not-completely-loading-them-into-memory#answer_183176

编辑：Stephen23 2015-6-18

You should read TMW's own advice on working with big data:

http://www.mathworks.com/discovery/big-data-matlab.html

And in particular you might find memmapfile to be of significant interest to you:

http://www.mathworks.com/help/matlab/ref/memmapfile.html

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Walter Roberson 2015-6-18

Or instead of memmapfile, save the .mat with -v7.3 and then use matFile objects to read in portions of the array.

请先登录，再进行评论。

Answer 2

Alessandro 2015-6-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/224395-changing-large-matrices-by-not-completely-loading-them-into-memory#answer_183173

Did you check the sparse command out?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Moritz 2015-6-18

Yes I did. However, I believe this only works if you have a considerable amount of zero elements. In my case however, the amount of zero elements are < 5%.

请先登录，再进行评论。