How to solve "out of memory error" caused by bsxfun?

Question

Glazio 2017-8-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/351652-how-to-solve-out-of-memory-error-caused-by-bsxfun

编辑： Glazio 2017-8-29

If I try to enlarge the base sample size of 10000 by 500 samples (in the context wie a Sensitivity Analysis) the following error message occurs:

Error using bsxfun
Out of memory. Type HELP MEMORY for your options.
Error in ipdm>distcomp (line 1008)
      d = bsxfun(@minus,set1(:,1),set2(:,1)').^2;
Error in ipdm (line 364)
      d = distcomp(data1,data1,params);
Error in lhcube_extend (line 89)
    dd=(ipdm(Xi,'metric',2)); dd=min(dd(dd>0));
Error in AAT_sampling_extend (line 125)
Uext = lhcube_extend(U,Next,nrep) ;

The script AAT_sampling_extend produces the error. AAT_sampling_extend is integrated in the SAFE-Toolbox (and has already been used several times).

I started this with the following command:

Xext = AAT_sampling_extend(X,DistrFun,DistrPar,2*(N+N2))

with

N = base sample size (N = 10.000)
N2 = extended sample size (N2 = 500)
X = Matrix of size 10.000 * 31
DistrFun = Latin hyper cube sampling
DistrPar = Parameters and their range

How can you solve the problem (without changing the source code too much)?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Glazio 2017-8-28

I have upgraded the RAM (from 4 GB to 12 GB) and the same error message appears.

I have increased the virtual memory to the recommended value (18430 MB).

I have closed other applications (processes), cleaned unused large variables and pre-allocated arrays.

At the time of program-crash, more than 4 GB of RAM were still free. Currently I have no idea what the problem could be.

Does anyone have an idea (RAM, CPU aren't the problem)?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jan 2017-8-5

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/351652-how-to-solve-out-of-memory-error-caused-by-bsxfun#answer_276922

A 10000 by 500 array of type double requires 40MB only. If your computer does not have enough memory to allocate such a small array, then the best idea is to install more RAM. Other ideas:

Close other applications
Increase the virtual memory
Use the type single if precision is sufficient
Clean unused (large!) variables
Pre-allocate arrays (this might not necessarily help, but is a good programming practice in every case)
But nothing will beat real hardware RAM, so install more.

10 个评论
显示 8更早的评论隐藏 8更早的评论

Glazio 2017-8-5

在 MATLAB Online 中打开

@ Jan Simon:

feature memstats:

    Physical Memory (RAM):
        In Use:                             2170 MB (87a10000)
        Free:                               1925 MB (78539000)
        Total:                              4095 MB (fff49000)
    Page File (Swap space):
        In Use:                             2290 MB (8f271000)
        Free:                              10803 MB (2a3307000)
        Total:                             13093 MB (332578000)
    Virtual Memory (Address Space):
        In Use:                              718 MB (2ce73000)
        Free:                               3377 MB (d316d000)
        Total:                              4095 MB (fffe0000)
    Largest Contiguous Free Blocks:
         1. [at 7fff0000]                   2045 MB (7fdf2000)
         2. [at 2c710000]                    953 MB (3b9b0000)
         3. [at 683dd000]                    100 MB ( 64d3000)
         4. [at 770f0000]                     59 MB ( 3bb0000)
         5. [at 6ebdb000]                     20 MB ( 1445000)
         6. [at 7c427000]                     16 MB ( 1089000)
         7. [at 7b58e000]                     14 MB (  e02000)
         8. [at 71ea2000]                     11 MB (  b4e000)
         9. [at 26f70000]                      8 MB (  850000)
        10. [at 7e1ea000]                      8 MB (  826000)
                                            ======= ==========
                                            3238 MB (ca6b9000)

I have tried now to run the function with different sample sizes. In addition, I have increased the swap-memory even further. Now the function runs up to a sample size of 500, without any problems.

However, about 9000 additional samples would be required.

So I'll have to upgrade the RAM.

How big should be the RAM for such a sample?

Glazio 2017-8-6

编辑：Glazio 2017-8-6

在 MATLAB Online 中打开

@ Image Analyst: I called it for both situations.

And as mentioned above: I have tried now to run the function with different sample sizes. In addition, I have increased the swap-memory even further. Now the function runs up to a sample size of 500, without any problems.

So I think the main problem is the RAM (with 2 x 2 GB -> in sum 4 GB).

I can expand the RAM to max. 16 GB (more is not possible with this PC).

So I hope that in the following several runs will be possible.

Feature memstats gives the following result during the run for 501 samples (and breaks):

 Physical Memory (RAM):
        In Use:                             2111 MB (83f1a000)
        Free:                               1984 MB (7c02f000)
        Total:                              4095 MB (fff49000)
    Page File (Swap space):
        In Use:                             2410 MB (96ad9000)
        Free:                              10682 MB (29ba9f000)
        Total:                             13093 MB (332578000)
    Virtual Memory (Address Space):
        In Use:                              761 MB (2f9e3000)
        Free:                               3333 MB (d05fd000)
        Total:                              4095 MB (fffe0000)
    Largest Contiguous Free Blocks:
         1. [at 7fff0000]                   2045 MB (7fde3000)
         2. [at 2e0c0000]                    919 MB (39780000)
         3. [at 6bd0d000]                     71 MB ( 47f3000)
         4. [at 67871000]                     50 MB ( 320f000)
         5. [at 77e30000]                     46 MB ( 2e70000)
         6. [at 7c427000]                     16 MB ( 1089000)
         7. [at 6ab11000]                     15 MB (  fbf000)
         8. [at 7b58e000]                     14 MB (  e02000)
         9. [at 74057000]                     10 MB (  ad9000)
        10. [at 7e1ea000]                      8 MB (  826000)
                                            ======= ==========
                                            3199 MB (c7f1e000)

Jan 2017-8-28

在 MATLAB Online 中打开

Sorry, the question is still not clear. Please stop Matlab when the problem occurs:

dbstop if error

Then tell us the dimensions of the variables "set1" and "set2". Formerly you mentioned a 10000 by 500 matrix, which needs 40 MB only. If you have 4GB of free memory, this cannot be the problem. If bsxfun tries to create a 10'000 x 10'000 matrix, this requires 800MB of RAM.

I hesitate to install the mentioned toolbox only to reproduce the problem, if you can find out the source of the problem so fast. Please check how much memory is required in the failing command.

Finally remember, that some problems demand for a huge RAM. If a code works with 500 samples, the required computation time can explode to millions of years for 1000 samples.

Glazio 2017-8-28

编辑：Glazio 2017-8-28

在 MATLAB Online 中打开

@ Jan Simon: Thanks for your answer,

the input-matrix has a dimension of 10000 x 31.

If I extend it by 10000 samples there are generated three matrices:

XA with 5000 x 31, XB with 5000 x 31 and XC with 155000 x 31.

If I extend it by 500 samples there are generated three matrices:

XA with 200 x 31, XB with 200 x 31, XC with 620 x 31

dbstop if error gives the following result:

Error using  .^ 
Out of memory. Type HELP MEMORY for your options.
Error in ipdm>distcomp (line 1015)
          d = d + bsxfun(@minus,set1(:,i),set2(:,i)').^2;
Error in ipdm (line 364)
      d = distcomp(data1,data1,params);
Error in lhcube (line 88)
    dk=(ipdm(Xk,'metric',2)); dk=min(dk(dk>0));
Error in lhcube_extend (line 86)
    Xext=lhcube(N_new-N,M);
Error in AAT_sampling_extend (line 125)
Uext = lhcube_extend(U,Next,nrep) ;
Error in extended_sample (line 76)
Xext = AAT_sampling_extend(X,DistrFun,DistrPar,(N+N2)) ; % extended sample

请先登录，再进行评论。

Answer 2

Steven Lord 2017-8-28

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/351652-how-to-solve-out-of-memory-error-caused-by-bsxfun#answer_279461

Are you using a 32-bit or 64-bit version of MATLAB and version of your operating system? If you're using a 32-bit system, upgrade to a 64-bit system.

It seems like the function you're using calls this File Exchange submission or something similar and creates a 10000-by-10000 inter-point distance matrix during the course of its execution. That would require a contiguous block of memory about 750 MB in size. Anything you do that requires a new copy of that matrix may require another contiguous block of that same size.

5 个评论
显示 3更早的评论隐藏 3更早的评论

Jan 2017-8-29

编辑：Jan 2017-8-29

Then buy an Apple computer. Sorry, just kidding.

Perhaps the student versions have been limited to 32 bit. Contacting the support directly can clear this directly. Simply use the "Contact us" button on this page.

You can upgrade a student version, see https://de.mathworks.com/matlabcentral/answers/242937-how-can-i-update-my-matlab-student-license-to-a-newer-version.

Glazio 2017-8-29

编辑：Glazio 2017-8-29

@ Jan Simon: Thanks for your answer. I would prefer Linux as a operating system ;-)

2012a is not available for 64-bit Windows system. This means that a new license must be purchased.

请先登录，再进行评论。

How to solve "out of memory error" caused by bsxfun?

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

回答（2 个）

10 个评论
显示 8更早的评论隐藏 8更早的评论

5 个评论
显示 3更早的评论隐藏 3更早的评论

另请参阅

类别

标签

产品

Community Treasure Hunt

How to solve "out of memory error" caused by bsxfun?

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

回答（2 个）

10 个评论 显示 8更早的评论隐藏 8更早的评论

5 个评论 显示 3更早的评论隐藏 3更早的评论

另请参阅

类别

标签

产品

Community Treasure Hunt

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

10 个评论
显示 8更早的评论隐藏 8更早的评论

5 个评论
显示 3更早的评论隐藏 3更早的评论