Can I share a copy of a CONSTANT large array between workers of a parpool?

Question

Naor Movshovitz 2017-3-20

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/330917-can-i-share-a-copy-of-a-constant-large-array-between-workers-of-a-parpool

回答： Naor Movshovitz 2017-3-21

I have an N-by-N array, data, of constant data that needs to be accessed by each worker of a parallel pool in a parfor loop. The loop will be called several times. Unfortunately each worker does need access to the entire array.

I just learned about the parallel.pool.Constant function which helps avoid multiple transfers of data and this is already helpful. However it seems that each worker keeps its own copy of data so I am limited by available memory to a small parallel pool.

Is there a way to use only a single copy of data in a parfor loop? To make things easier let's assume we will use a local cluster only, i.e. a multi-core desktop.

thanks!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Naor Movshovitz 2017-3-21

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/330917-can-i-share-a-copy-of-a-constant-large-array-between-workers-of-a-parpool#answer_259712

Answering my own question after some more googling and doc search. I am not 100% sure but I think the right answer is: no, this is not possible with a parfor construct. The ease of use of parfor comes with some limitations and this is one of them. For sharing a large array between workers the code would need to be rewritten as a spmd region and use the distributed function. I expect this might not be ideal for small-to-medium arrays though, that we would not want fragmented across memory, but possibly ML is smart enough to optimize this behavior behind the scenes.

As a last resort the sharedmatrix fx submission suggested above might work but it sounds pretty risky. A lot of talk of segmentation faults in the comments ;)