Out of memory smpd parallel computing toolbox eig on complex matrix

Question

Renee 2017-2-2

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/323094-out-of-memory-smpd-parallel-computing-toolbox-eig-on-complex-matrix

评论： Renee 2017-2-8

Hi, I have an 11x11 matrix from a system of 11 ODEs (hence the complexity). Matlab's eig was unable to solve the matrix without running out of memory, so I'm trying out the parallel computing toolbox. I haven't been able to find any clear instructions, so I may be doing very obvious things wrong. My code is:

parpool
spmd
    tic
    [l1,l2] = eig(D);
    time = gop(@max, toc) % Time for all labs in the pool to complete.
end
delete(gcp)

and D is my matrix. I set up my local profile so that I use 6 workers. My question is: am I using the toolbox/functions incorrectly, or is my system just too complex to solve? In case relevant, I am running Matlab 2016b on win10 64, 4 ghz cpu, 24 gb ram. It gets up to almost 100% memory usage and about 80% cpu usage with this code.

The error:

Starting parallel pool (parpool) using the 'local' profile ... connected to 6 workers.
ans = 
   Pool with properties: 
              Connected: true
             NumWorkers: 6
                Cluster: local
          AttachedFiles: {}
            IdleTimeout: 30 minute(s) (30 minutes remaining)
            SpmdEnabled: true
Warning: An error has occurred during SPMD execution. An attempt has been made to interrupt execution on the workers. If this situation persists, it may be necessary to interrupt execution using CTRL-C and then deleting and
restarting the parallel pool.
The error that occurred on worker 3 is:
Error using symengine
Out of memory.
Error in sym/mupadmexnout (line 981)
        out = mupadmex(fcn,args{:});
Error in sym/eig (line 54)
    [Vsym,Dsym,p] = mupadmexnout('symobj::eigenvectors',A);
Error in eigen_test1 (line 50)
spmd
. 
> In spmdlang.RemoteSpmdExecutor/maybeWarnIfInterruptedAndWaiting (line 300)
  In spmdlang.RemoteSpmdExecutor/isComputationComplete (line 131)
  In spmdlang.spmd_feval_impl (line 19)
  In spmd_feval (line 8)
  In eigen_test1 (line 50) 
Error using eigen_test1 (line 50)
Error detected on worker 3.
Caused by:
    Error using symengine
    Out of memory.
Exception in thread "Explorer NavigationContext request queue" IdleTimeout has been reached.
Parallel pool using the 'local' profile is shutting down.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Edric Ellis 2017-2-3

In this case, unfortunately I think you are simply replicating the same computation on each worker. Inside an spmd block, to get parallelism, you need to use codistributed arrays. But it looks here like you are using symbolic computation.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Prannay Jain 2017-2-7

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/323094-out-of-memory-smpd-parallel-computing-toolbox-eig-on-complex-matrix#answer_253778

I understand that you are running into memory issues even after parallelizing the computation using spmd and parpool.

Moving to Parallel Computing Toolbox (PCT) while remaining on the same machine will never resolve a memory issue. By splitting the computation across multiple workers makes the memory situation worse by having extra MATLABs open. You will either need a bigger machine with more memory or try considering out a cluster where you can appropriately distribute the problem across multiple machines' memory using MATLAB Distributed Computing Server (MDCS) and PCT.