Split the combvec resulting matrix?

Question

Zi 2023-3-4

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1923085-split-the-combvec-resulting-matrix

回答： Walter Roberson 2023-3-4

I am using combvec function to create all possible combinations of 28 different numbers. The resulting matrix should have 3,221,225,472 columns (combinations) and 28 rows. I cannto run the code because of the out of memory error. I was wondering if there is anyway I can split the columns into more than one file to tackle this issue.

Here is the code:

a1 = ...=a26 = [1 2];

a27 = a28 = [1 2 3 4];

Mat = combvec(a1,..,a28).

Your help is highly appreciated.

2 个评论
显示无隐藏无

Stephen23 2023-3-4

编辑：Stephen23 2023-3-4

在 MATLAB Online 中打开

"I cannto run the code because of the out of memory error."

That is not a big surprise if you ask for more than seven hundred and twenty gigabytes of contiguous memory:

R = 3221225472;
C = 28;
N = R*C
N = 9.0194e+10
N*64 / 8 % bytes
ans = 7.2155e+11

To perform any operations on it you would then need three our four times that in practice, so does your computer have some three or four terabytes of memory installed? Note this is memory, not drive storage (i.e. not your HDD or SDD).

"I was wondering if there is anyway I can split the columns into more than one file to tackle this issue."

How exactly would splitting data up into separate arrays help "to tackle this issue" ? The amount of memory your computer has installed is constant: why would your computer suddenly have more memory if you ask it to store lots of smaller arrays?

https://www.mathworks.com/help/matlab/matlab_prog/resolving-out-of-memory-errors.html

What does any "file" have to do with this? A file is essentially some data saved to some storage device. What your storage devices have saved on them is unrelated to your computer's memory (unless you use some kind of slow virtual memory). You seem to be confusing storage devices (e.g. HDD, SDD) and memory.

In any case, rather than attempting things that do not make much sense and are very unlikely to work, you would be much better off reading the MATLAB documentation on how to store and manipulate large data:

https://www.mathworks.com/help/matlab/tall-arrays.html

and also reconsidering your algorithm: the best big data processing is the one that avoids the need for big data. If you can tweak your algorith so that you do not need to store all of that data at once in memory, then that is likely a better approach.

Dyuman Joshi 2023-3-4

For splitting an array into columns you would require the array first, which you do not have.

The function combvec returns a matrix as an output, and you can not change how a function returns an output.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2023-3-4

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1923085-split-the-combvec-resulting-matrix#answer_1185310

在 MATLAB Online 中打开

a1 = [1 2];
a2 = a1; a3 = a1; a4 = a1;
a27 = [1 2 3 4];
a28 = a27;
Mat = combvec(a1, a2, a3, a4, a27, a28);
numel(a1)^4 * numel(a27)^2
ans = 256

So with you using 26 vectors each of length 2, and two vectors of length 4, the number of columns should be

count = numel(a1)^26 * numel(a27)^2
count = 1.0737e+09

whereas you are expecting

expected = 3221225472
expected = 3.2212e+09
expected / count
ans = 3

You are expected three times as many columns as should be calculated.

If you were expecting numel(a1)^26 * numel(a27)^2 then there is an easy way to sequence through the values:

for count = uint32(0) : uint32(2^26 - 1)
    bits = bitget(count, 1:26) + 1;
    thisiterelements = a1(bits);  %assuming a1 through a26 are the same
    now combine thisiterelements with all 16 combinations of a27 and a28 which you can prebuild
    now do something with these 16 combinations.
end