How to improve with single 4D arrays when memory limit problem?
1 次查看(过去 30 天)
显示 更早的评论
I am studying the question in the thread here . I am replacing my 2D cell array of matrices with a single 4D array but I am getting now the memory limit problem. My intuitive data structure would be hashed array tree or using internally Java in Matlab for the task.
> Error using zeros Requested 906x1808x2x1692 (36.3GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Code where increase the values of `(a,b)` to get enough big array size
a=66; b=77;
channels=[1 2];
Times=(0+5*60):1:(29*60+50);
% Let's use a single 4d array instead of 2d cell arrays of matrices
Cm = zeros(a,b, numel(channels), numel(Times));
for channel=channels
for k=1:numel(Times)
Cm(:,:, channel, k) = rand(a,b);
end
end
How to improve the internal data structure there? I really would like to avoid such memory problems and/or limits. If Oracle Java is the choice, please propose how to do such a basic processing there from Matlab.
Matlab: 2016b OS Debian: 8.5 64 bit Linux kernel: 4.6 of backports Hardware: Asus Zenbook UX303UA Related: https://se.mathworks.com/matlabcentral/answers/249344-matlab-array-size-limit-error-why
0 个评论
回答(2 个)
Neekar Mohammed
2019-12-29
Have you find any solution for your problem? because I'm currently facing same thing?Thanks
0 个评论
Walter Roberson
2019-12-30
That code as posted does not attempt to produce a 906 x 1808 x 2 x 1692 array, but it would not take a lot of changes to make it do so.
Cm = zeros(a,b, numel(channels), numel(Times), 'single');
would take half of the memory compared to double precision.
Cm = rand(a, b, numel(channels), numel(Times), 'single');
would create the entire array at one time without a loop. However, I am not completely sure at the moment whether it would only create single precision numbers, or if it would try to generate a full double precision array and then convert it to single precision. Using a loop to generate a portion at a time is not unreasonable -- though I would suggest rand(a, b, numel(channels), 'single') at a time rather than rand(a,b,'single') in order to take fewer calls.
My intuitive data structure would be hashed array tree
Unfortunately the original stackoverflow postgres related question no longer exists so we do not know what was being asked at the time. What I can say is that in MATLAB, a 4D array is the most memory efficient way that store "dense" numeric values that are logically 4 dimensional. Random numbers that are uncorrelated are barely compressible (though it would in theory be possible to compress single precision rand() results losslessly at 3 bytes instead of 4.) Hash tables and cell arrays just add overhead. Sometimes techniques such as hash tables are useful, especially in sparse situations, but at the moment I do not see any reason to use them here.
2 个评论
James Tursa
2020-4-23
You might look into this FEX submission by Matt J:
MATLAB does not support anything beyond 2D for the sparse format.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Matrix Indexing 的更多信息
产品
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!