"Error using zeros" maximum variable size allowed by the program is exceeded

Question

Cassandra Wong 2016-6-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/291814-error-using-zeros-maximum-variable-size-allowed-by-the-program-is-exceeded

评论： Najme Behdad 2017-1-4

I was trying to generate a 2^57 x 57 zeros matrix and error occurred. I tried with sparse and the error still persist. Is there anywhere can I generate the matrix efficiently.

Please help. I am student, still exploring how to work with MATLAB. Thank you.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Steven Lord 2016-6-23

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/291814-error-using-zeros-maximum-variable-size-allowed-by-the-program-is-exceeded#answer_226492

在 MATLAB Online 中打开

If you wanted to create that matrix as a logical array, it would require:

numElements = (2^57)*57;
exabytes = numElements/(1000^6)
exabytes =
        8.21456572032379

To put that into perspective, take a look at the Wikipedia page for exabyte. One comparison:

"Therefore, one exabyte could hold a hundred thousand times the printed material, or 500 to 3000 times all content of the Library of Congress."

To process all possible subsets of a set with 57 elements (which is what I assume you're trying to do, based on the dimensions of the matrix you're trying to create) would require quite a bit of time.

numCombinations = 2^57;
processingTime = years(seconds(numCombinations))
processingTime =
              4.5668e+09

You probably want to rethink your approach, unless perhaps you have a VERY large cluster of machines available.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Cassandra Wong 2016-6-24

I guess I have to change my approach then. Anyway, thank you sir you the explanation.

请先登录，再进行评论。

Answer 2

James Tursa 2016-6-23

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/291814-error-using-zeros-maximum-variable-size-allowed-by-the-program-is-exceeded#answer_226435

编辑：James Tursa 2016-6-23

A matrix of size 2^57 x 57 doubles would take about 6.57e19 bytes of memory ... way way more than your computer has available. So no, there is no way to generate this matrix on your computer. Even a sparse matrix of this size would take too much memory just to store the index data.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Cassandra Wong 2016-6-24

Thank you sir for the reply.

请先登录，再进行评论。

Answer 3

Walter Roberson 2016-6-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/291814-error-using-zeros-maximum-variable-size-allowed-by-the-program-is-exceeded#answer_226441

A full array for that would require a virtual memory space more than 3 1/2 times larger than can be addressed with 64 bit memory. I don't think you are going to be able to create something that big. You might not be able to create anything that requires more than 2^48 bytes as all current implementations of the x64 architecture only have at most 48 address pins.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Cassandra Wong 2016-6-24

Thank you sir for the explanation.

请先登录，再进行评论。

Answer 4

Najme Behdad 2017-1-2

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/291814-error-using-zeros-maximum-variable-size-allowed-by-the-program-is-exceeded#answer_249182

编辑：Najme Behdad 2017-1-2

A question comes to me, Is there a way processing for binary large scale matrix in MATLAB 2 ^ 40. Was this possibility added in MATLAB 2016? please help me

10 个评论
显示 8更早的评论隐藏 8更早的评论

Walter Roberson 2017-1-3

The maximum size of a matrix of logical or uint8, of size 2^N by N, that can be processed in MATLAB, is 2^42 by 42. That requires a system with 168 terabytes of memory.

"What is the solution for the processing of this matrix? 40 is the number of features."

You have not described the goal. It sounds as if you are thinking you need to do an exhaustive trial of all the possible combinations of features being present or not, but what are you calculating?

Considering your use of the word "feature", I suspect you might be trying to find the "best" set of features for some kind of classification. That is a topic known as "feature selection". A complete search is probably not going to do a significantly better job than using some approximations: you would run a significant risk of over-fitting -- the answer you came up with would likely involve small statistical accidents that made that particular choice work marginally better on your test data but worse than other possibilities on real-world data.

There are a number of different approaches to feature selection; different approaches have been studied more with respect to different classification schemes.

In some cases a good approach is to run a number of different random approaches, pick the best several, and use an ensemble classification scheme

Walter Roberson 2017-1-3

Do you mean you have a vector of 40 values and you are trying to find a subset of the values that totals exactly 1.0 ? If so, then you have to be very careful about round-off in your calculations.

Finding a subset of values with a particular total is known as the Knapsack problem.

https://www.mathworks.com/matlabcentral/fileexchange/22783-0-1-knapsack

http://enthumema.blogspot.ca/2012/09/knapsack-problem-in-dynamic-programming.html

https://labs.ise.ncsu.edu/fangroup/wp-content/uploads/sites/14/2016/08/TA-Lecture-MATLAB.pdf

http://www.matarka.hu/koz/ISSN_1789-2198/vol_7_no_1_2013_eng/ISSN_1789-2198_vol_7_no_1_2013_eng_013-020.pdf

Najme Behdad 2017-1-4