How to diagnose "Out of Memory during deserialization" when running on high core count

49 次查看（过去 30 天）

Adrian 2018-7-25

2
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/412022-how-to-diagnose-out-of-memory-during-deserialization-when-running-on-high-core-count

评论： Zhenhao Gong 2023-11-19

Hello MATLAB Community! I'm trying to figure out a problem with parallel computing at the moment. I'm running the built-in genetic algorithm on a high performance cluster. Now I'm running into the issue that if I assign a certain number of cores the code does not work anymore.

I get the proper reply when starting the parallel pool (Connected to 48 or 280 workers) but when using 280 workers I first get a couple of warnings like this:

[^HWarning: A worker aborted during execution of the parfor loop. The parfor loop
will now run again on the remaining workers.]^H
[^H> In parallel_function (line 599)
In fcnvectorizer (line 16)
In gaminlppenaltyfcn
In gapenalty
In makeState (line 64)
In galincon (line 17)
In gapenalty
In gaminlp
In ga (line 366)

and then it finally crashes with this:

{^HError using fcnvectorizer (line 16)
Out of Memory during deserialization

On my local machine, the same code runs fine and needs about 2 gigabytes of ram per worker. I have assigned 4.5 gigabytes per core on the cluster, so I don't think it's an actual memory issue. However, all solutions I found online regarding this error point to memory issues.

Any input is greatly appreciated.

Cheers.

2 个评论
显示无隐藏无

Lina Koronfel 2022-8-22

@Adrian did you solve this problem? I have a simmilar one

Zhenhao Gong 2023-11-19

Hello, I met the same problem as you, and MATLAB also gave me feedback of "Out of Memory during deser....". But by my calculations, my code uses up to 20GB of memory, and the maximum RAM allowed by the cluster is 168GB. In fact, code does use 160GB.

I think some matrix operations under parfor are wrong.

请先登录，再进行评论。

请先登录，再回答此问题。