Bypass maximum matrix size for ones()

3 次查看（过去 30 天）
Sana Ahmed
Sana Ahmed 2020-7-26
评论： Walter Roberson 2020-7-27
I have a program in which I create a large matrix of ones using the ones() function, but apparently the matrix is too large for the function. I get the error 'Maximum variable size allowed by the program is exceeded.' I use this matrix later in my code and overwrite some entries, but most of the entries stay one, so it would be most efficient if I have the matrix be initialized with ones from the start. Is there someway that I can expand the maximum variable size of this function or is there another way to create a large matrix pre-populated with ones. (Just for reference, by large I mean 40,859,280 elements and I think the maximum number of elements is 524,288). Any input would be helpful and thanks in advance!
  2 个评论
James Tursa
James Tursa 2020-7-26
40M elements is not that big for a modern computer. Can you show us the exact syntax you are using to create the matrix? Maybe you are inadvertently trying to create a 40M x 40M matrix.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

Walter Roberson
Walter Roberson 2020-7-26
Should you perhaps be using a sparse() array?
Your array is only about 312 megabytes; I would expect any modern system to be able to handle an array that size; I would only expect trouble with the 32 bit versions of MATLAB (it was common to have only about a gigabyte of free memory for those.)
Anyhow, go into Preferences -> Workspace -> MATLAB Array Size limit.
You can configure the fraction of RAM that you permit, and you can turn off the check.
  8 个评论
显示 6更早的评论
Walter Roberson
Walter Roberson 2020-7-27
>> log2(40000000^2*8)
ans =
53.5069933284231
You would require 54 address bits to address that much memory. However, all publicly known implementations of the Intel x64 architecture have a limit of 48 address bits. The upper bytes of addresses is currently used by the operating system for its internal purposes (including access to privileged memory), so even if you had a chip with more than 48 address lines, you would have problems.
I just tested on my system, and I was able to
foobar=spalloc(40E6, 40E6,2064000000000); %0.00129 occupancy
but not
foobar=spalloc(40E6, 40E6,20640000000000); %0.0129 occupancy
The 0.0129 occupancy was tested because that is the fraction you had asked for earlier when you wanted 524,288 out of 40,859,280.
So if you use sparse matrices, it is not the 40E6 on a side that is the problem, but the amount of memory requested to be allocated.
It appears that the allocated memory according to whos() is
(columns+1) * 8 + nonzeros * 16
and
>> log2(33024320000008)
ans =
44.9085940889452
whereas if I had asked for 10 times as much that would add another 3 to 4 bits... which would take it just over 2^48.
Therefore you can spalloc() requesting up to 2^48 bytes according to (columns+1) * 8 + nonzeros * 16 . But you are going to swap very badly once you go beyond your physical memory.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Performance and Memory 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by