Nested parfor loop slows down with each iteration

Question

Marc 2024-6-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2128146-nested-parfor-loop-slows-down-with-each-iteration

回答： Jasvin 2024-6-13

I have the following (simplified) code example with a nested outer parfor and inner for loop. Variables are sliced. It takes about 30 minutes for the first 10 outer iterations (running on 10 workers). The second round (iterations 10-20) takes more than an hour and it keeps increasing for later iterations. What is wrong here?

function my_function()
n_iterations_v = cell(1,50);
n_iterations_v = init_iterations(); % each element is an integer
parameters_v = cell(1,50); 
parameters_v = init_params() % element i is a cell array with n_iterations_v{i} elements (where each element is a struct)
% outer loop
parfor (k = 1:50)
    results_v = cell(1,n_iterations_v{k});
    
    % inner loop
    for i=1:n_iterations_v{k}
        results_v{i} = some_other_function(parameters_v{k}{i});
    end
      
    save(filename, "-fromstruct", struct("results_v",results_v));
end
end

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jasvin 2024-6-13

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2128146-nested-parfor-loop-slows-down-with-each-iteration#answer_1471356

Have a couple of suggestions on how to investigate further.

Can you measure whether the same behaviour is observed for a regular "for" loop as well? If that's the case then it would imply that this is expected behaviour and parallelization can only do so much and the performance has to do more with the business logic of your code instead.

I also found this StackOverflow answer that suggests to use "parpool('threads')" which could be applicable in your case since you too seem to have a decent amount of data along with the copy operation involved.

https://stackoverflow.com/a/69955780

You could also create a technical support case with the MATLAB team by contacting support@mathworks.com