Matlab crashes with large matrix

9 次查看(过去 30 天)
I need to create a large matrix F=[F1, F2, ... , FN] where each Fn has the same number of rows (about 12,000) but different number of columns. What I am doing now is to create F as a cell array using a loop with element F{n} = Fn. The problem is that when the number of columns of F is very large, Matlab crashes: at some point it shuts down (I run this on a server, and I find matlab closed). I suspect that it shuts down when I convert the cell array into a single matrix, F=cell2mat(F);
Notice that everything works ok when F has a number of columns less than say, say 1,000,000,
Any help is much appreciated. Thanks!

采纳的回答

Walter Roberson
Walter Roberson 2023-10-25
编辑:Walter Roberson 2023-10-25
Well, that is certainly possible... if you make the arrays large enough then there might just not be enough memory available.
Is it possible to predict the number of columns that will be needed for each Fn before doing the actual callculations? If so then you could consider doing that calculation and totaling them, and pre-allocating the output array, then writing directly into the output array instead of building the cell array. If you pre-build the sizes then you can use cumsum() to create an array that will tell you which indices to write at
As a side note: it is more efficient to use
F = horzcat(F{:});
than F = cell2mat(F); -- but this would not solve your memory problems.
  2 个评论
valentino dardanoni
valentino dardanoni 2023-10-25
Thank you very much Walter. Unfortunatley I cannot predict ex ante the number of columns in each Fn. I guess I could check them afterwrods looking at the cell array, then stop the program, and rerun it by preallocation... It would not be fun, but it is doable... I was wondering, however, if it is normal that Matlab crashes in such a situation, rtaher than giving some sort of warning and stopping....
Bruno Luong
Bruno Luong 2023-10-25
No it's not normal. But when all memory are consumed, many tasks are slowing down and requesred memory get swap to hard drive. The PC are in a "non standard" functioning mode and things can get slippy from there. Even pop up a warning message or windows become a hercules task.

请先登录,再进行评论。

更多回答(1 个)

Bruno Luong
Bruno Luong 2023-10-25
编辑:Bruno Luong 2023-10-25
For 1e6 column the memory required in Gb for one array is
memgGb = 1e6*12e3*8 / 1e9
memgGb = 96
And you have the cell array + the concatenate array (single contiguous emory blockà, so you need 192 Gb of free memory.
You might consider using single-class array, write directly to the allocated matrix as Walter suggests or using tall array assuming you store the transposed of the original intended array.
  2 个评论
valentino dardanoni
valentino dardanoni 2023-10-25
Grazie Bruno, will explore the tall array route... BTW by coincindence the next step in my program is to use "licols", which I understand stands form an idea of yours!
Bruno Luong
Bruno Luong 2023-10-25
licols is Matt's FEX, but yeah I quite participate to popular the idea. Thanks

请先登录,再进行评论。

类别

Help CenterFile Exchange 中查找有关 Logical 的更多信息

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by