Array exceeds maximum array size preference

65 次查看（过去 30 天）
luca
luca 2019-7-25
回答： Muniba Ashfaq 2020-9-11
Hi, given the following
v = [1 2 3 4 1 1 1 1 1 2 3 2];
P = unique(perms(v),'rows');
I've obtained
Error using zeros
Requested 479001600x12 (42.8GB) array exceeds maximum array size preference. Creation of arrays
greater than this limit may take a long time and cause MATLAB to become unresponsive. See array
size limit or preference panel for more information.
Do you know how to fix the problem to get all the possible permutations? One idea was to divide in different cases. For example is the total nuber of permutations is 1000, is there a way to modify the command perms to get just the first 200 elements? Is there a more efficient way?
  4 个评论
显示 2更早的评论
Guillaume
Guillaume 2019-7-25
Anyway I'm generating an NP complete algorithm, and I need all the possible permutations.
There's a big difference between what you want and what is possible.
As the error message tells you, generate all the (479001600) permutations of your vector with perms would require 42.8 GB of memory. It's unlikely that your computer has that much memory and even if it had, it would take a loooong time to generate.
Even if you were to generate all these 479001600, tomorrow you'll decide to extend to 13 elements (6227020800 elements = 603 GB), 14 elements (8.8 TB of memory), etc. You need to release that your current algorithm requires an exponential amount of memory and rethink it. Brute-force generating all the combinations is only going to work for up to 10 elements.
Joel Handy
Joel Handy 2019-7-25
There is a lot being asked here with very little info to go on.
We don't have any insight into this algorithm. We don't know what it's supposed to do or how it does it. This makes it hard to suggest ways of generating the permutations in useable chunks.
Regardless, processing 479 billion things is going to take a while. Even if it only takes a microsecond to process each permutation, it will take over 5 days to run.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Hari Krishna Ravuri
I understand that, you are trying to generate all permutations of the array ‘v’ resulting in generating a vector of very large size, which exceeds maximum array size preference. To know more about the variable size preferences, please refer https://www.mathworks.com/help/matlab/matlab_env/set-workspace-and-variable-preferences.html.
As you have mentioned that you are trying to solve NP-complete problem and need all the permutations, I suggest you write a small script which generates all the permutations by rearranging the elements in the array itself i.e. with using constant memory space. Now, in that script, for the first Iteration, create a .mat file and write the permutation to the file and for the remaining iterations, you just need to append every permutation to the file. For writing data to .mat file you can use save function with the syntax save(filename, variables,'-append'). For more information, please refer https://www.mathworks.com/help/matlab/ref/save.html
By writing data to .mat file, you can access all the data without bringing the entire file into the main memory.
“is there a way to modify the command perms to get just the first 200 elements?”
No, as of now, there is no such way mentioned in the documentation. Please refer https://www.mathworks.com/help/matlab/ref/perms.html
“Is there a more efficient way?”
As you are solving an NP-complete problem and need all the permutations for sure to get the optimal solution, the best way is to write all the permutations to a file and accessing them, as the entire array will not fit in the main memory. To know more about handling mat files, please refer to https://blogs.mathworks.com/loren/2011/10/14/new-mat-file-functionality-in-r2011b/
If you need a good solution but not the optimal one, you can use the problem heuristics and try to get a good solution in less time.
  1 个评论
Guillaume
Guillaume 2019-7-29
You could indeed generate all the permutations a few at a time (see for example, nextperm, nextvector, another nextperm, and probably more on the FileExchange) and save these to a file (mat or text), and then process them with tall arrays to avoid loading them in memory all at once, it doesn't change the fact that generating 479 millions permutations is going to take a while and processing them even longer.
And this will get exponentially worse, if the vector length is increased. 6,222 millions permutations for 13, 87,178 millions for 14, etc.

请先登录，再进行评论。

Muniba Ashfaq
Muniba Ashfaq 2020-9-11
I was also getting this error. I deleted all unnecessary variables from workspace to free up memory. I just keep those variables that was needed for my MATLAB code execution.
This worked amazingly.

类别

在 Help Center 和 File Exchange 中查找有关 Logical 的更多信息

标签

产品


版本

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by