Query regarding matlab maximum allowed size of an array

9 次查看（过去 30 天）

显示更早的评论

yash 2024-4-4

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2102951-query-regarding-matlab-maximum-allowed-size-of-an-array

评论： Rik 2024-4-12

I need to assign more then 16GB array for which it is showing following error. (I am assigning using x = zeros(size)):

Requested 50000x50000 (18.6GB) array exceeds maximum array size preference (6.0GB). This

might cause MATLAB to become unresponsive.

Now I found created three different array with x = zeros(28000); which is approximately 6 GB for each. Even then also on assigning third array it is giving "Out of Memory" error.

I also tried using tall array as follows. But still it is giving of out of memory error

a = tall(zeros(28000));

I checked my system memory using memory command as follows:

>> memory

Maximum possible array: 8600 MB (9.018e+09 bytes) *

Memory available for all arrays: 8600 MB (9.018e+09 bytes) *

Memory used by MATLAB: 3256 MB (3.414e+09 bytes)

Physical Memory (RAM): 6100 MB (6.396e+09 bytes)

* Limited by System Memory (physical + swap file) available.

So, Is there any way to assign and perform calculation on more than 16 GB array?

10 个评论
显示 8更早的评论隐藏 8更早的评论

John D'Errico 2024-4-4

编辑：John D'Errico 2024-4-4

You want to do something that effectively (compared to what you have) requires a supercomputer. Well, again, compared to what you have, it would be one. Do you have a supercomputer? That is, one with probably at least 32 GB of memory? And preferably 3x as much, to be sure. Because you will never just have one such array. Copies will be made on the fly to do stuff. If you do not have that supercomputer, then what do you expect? The little toy thing you do have cannot leap tall buildings in a single bound. It trips over train tracks. You just want it to do more.

So, you can either get more memory, or you can change your requirements, making your problem smaller, or you can find a bigger computer. And that does not mean finding another way to let MATLAB allow you to try to allocate 16GB of memory on your existing machine. You still cannot stuff 10 pounds of crap in a 5 pound bag, no matter how diligently you try. I'm sorry if I have said this too plainly for you, but that is the essential problem.

If you want help, then you need to explain enough about what you are doing, NOT just what you think you want to do at this step, since what you think you need to do now is create a 16GB array on a 6GB computer. You need to explain what you are doing. Go back to the beginning. That may help someone to show you how to do what you need to do, perhaps using sparse matrices. Or possibly you might be able to get away with allocating a uint8 or int8 array instead. That might just barely fit into your memory. (Until next month, when you will decide to make the problem larger. John's rule applies here, that computer problems always expand to be just a little larger than the largest computer resource you have available. The easy problems have already been solved by someone else.)

DGM 2024-4-5

编辑：DGM 2024-4-5

在 MATLAB Online 中打开

The fact that those are your array sizes tells me that your data is stored in uint8 format. If the only reason you're converting it to double is to scale it by some non-integer scaling factor for display purposes, then you can possibly avoid the whole problem and just keep the data in uint8.

I'm assuming the following:

The data is stored in uint8
The scaling is just a simple global linear transformation
The data are just vectors to be plotted with a line/scatter plot or something similar

On those assumptions, consider the example where no rescaling is used:

% these are our uint8-scaled arrays (1x100 vectors)

load uint8_x.mat

load uint8_y.mat

% this is where our xy data are in uint8-scale

xurange = imrange(xu);

yurange = imrange(yu);

% let's say we know these are the extrema of our x,y data after rescaling

% you could also just calculate these by transforming xurange,yurange

% using whatever scaling operation you would have used on the data

xrange = [0 2];

yrange = [0 4];

% plot the data without rescaling anything

hp = plot(xu,yu);

% set plot extents in uint8 scale

xlim(xurange)

ylim(yurange)

% arrange the ticks and ticklabels

numticks = [6 6]; % number of ticks [x y]

% place the tick positions in uint8 scale

xticks(linspace(xurange(1),xurange(2),numticks(1)));

yticks(linspace(yurange(1),yurange(2),numticks(2)));

% label the ticks as if rescaled

xticklabels(linspace(xrange(1),xrange(2),numticks(1)));

yticklabels(linspace(yrange(1),yrange(2),numticks(2)));

That could probably be simplified, but it should demonstrate the concept.

While that keeps the data arrays from ballooning by a factor of 8, you still have 1.9 billion data points plotted in a raster image which is no larger than your screen. For a simple single-valued function like this, you have several orders of magnitude more samples than you can physically display. Graphs and charts are visualizations of data, not stores of data. You might not want to downsample your data, but your display is going to eliminate information whether you want to or not. What's appropriate depends on the data, how it's being plotted, and what information is important.

yash 2024-4-9

Okay, I understood I cannot have full array at a time in my memory due to memory limitation. My idea is I will read data in an in-memory array according to my available memory and rest of data I will dump in a binary file. Now, I need to extract a part of this file as and when needed from particular index. I see there are two options for doing this.

1) data store and tall arrays

2) memmapfile

So, for this application which is the best way of doing this and can you provide me one small example of doing this?

Rik 2024-4-12

You are ignoring what we're telling you. You will need to write code capable of analyzing a small part of your data. Then you can process your full file in chunks. If you need help, you need to be explicit in what data you have and what output you want.

请先登录，再进行评论。

请先登录，再回答此问题。