Parallelize calculations on a big cell array without making N input copies

12 次查看(过去 30 天)
Hello,
The existing answers on matlab answers, also due to their age, have not made it clear to me if it is possible to parallelize CPU-intensive calculations done with either cellfun or a for loop over a large (15.000.000x1) cell array with each cell containing a 24x24 matrix, and then writing the calculation results for each into a 15.000.000x1 vector.
Using just parfor is no use since my PC runs out of memory (I have enough memory for 1 copy of the cell array but not 6 copies for the 6 workers). Is there a way to perhaps copy it only once, with each of the 6 workers receiving only a 6th of the total array as copy?
An older comment said there was a userwritten solution with shared memory, but that this would not work anymore with newer Matlab versions (I use 2016a).
Thank you for your help!

采纳的回答

OCDER
OCDER 2017-9-14
编辑:OCDER 2017-9-14
If each calculation is independent (as in, you don't need the entire 15000000x1 cell to make 1 calculation), then you could rewrite the parfor loop to work on a sliced variable. Workers will only receive a "slice" of the large 15000000x1 cell. This prevents passing the large cell as a broadcast variable, which will use too much memory.
The parfor loop should look something like this:
LargeCell = repmat({zeros(24)}, 100, 1); %Represent your 15000000x1 cell
Results = zeros(size(LargeCell)); %Store results here
parfor k = 1:length(LargeCell)
Results(k) = complex_function( LargeCell{k} ); %Do your CPU-intensive calculation here
end
  1 个评论
Flora Feldner
Flora Feldner 2017-9-14
编辑:Flora Feldner 2017-9-14
Yes, each calculation is independent. The sliced variable concept might exactly be what I need. Now I just need to monitor the memory usage and hope for the best. Thank you!

请先登录,再进行评论。

更多回答(0 个)

类别

Help CenterFile Exchange 中查找有关 Loops and Conditional Statements 的更多信息

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by