- Suppress the output arguments you don't need
- Save to a buffer and save that to disk when it becomes too large.
- Have each worker save its own output, not aggregating. You can put all the files together yourself afterwards
- Use sparse arrays
Info
此问题已关闭。 请重新打开它进行编辑或回答。
parfor loops with large outputs
1 次查看(过去 30 天)
显示 更早的评论
Hi,
I'm using MATLAB's parallel toolbox on a shiny server (32 physical cores, 64 logical, 512GB RAM). All my parallelization needs can be classified as 'embarrassingly parallel', e.g.:
parfor i_par = 1:n_tot
[CalcOutputs_array(:,:,i_par), AdvancedCalcOutputs_array(i_par)] = CPU_DemandingFunction(InputsStruct_array(i_par));
end
In order to minimize overheads, i've already split the input such that each worker receives only what it needs.
The output, however, can become quite large (1GB-10GB), and data collection from the workers is vastly over-weighting the calculation itself.
I'm positive that this is the case since by lowering the amount of data outputted from the function (i have a flag that controls it) - the runtime improvement factor approaches the # of cores i'm using.
Can that be improved? Is there any remedy for data-collection overheads?
Thanks a lot!
Yanir
1 个评论
José-Luis
2016-6-30
This is a very general question.
You could:
etc...
回答(0 个)
此问题已关闭。
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!