Issue with parfor when using containers.Map

4 次查看（过去 30 天）

Shlomo Geva 2023-5-28

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1974689-issue-with-parfor-when-using-containers-map

评论： Shlomo Geva 2023-5-28

Here is the code of a script that calls a function inside a loop. The function is passed a Map object (created previously). I entered in the comment the details of the Map. The function actually does nothing with the map. I run this code with a for loop and it takes little time of course:

"Elapsed time is 0.000417 seconds."

When I change the for to a parfor and run again - it goes into a black hole. I have to kill it. In the Activity Monitor (on a Mac) I can see all 8 processors are 100% busy.

I assume it may have to do with the distribution of a large map, but I tried it also on a server with 1.5TB of memory and it hangs there too.

Any suggestions or explanantions will be greatly appreciated.

Perhaps there is a better way to distribute the map as a shared variable (it is not modified inside the loop)

Here is the script:

% >> myMap
% 
% myMap = 
% 
%   Map with properties:
% 
%         Count: 10344256
%       KeyType: char
%     ValueType: any
% 
% >>
tic
for i=1:8 % if this for loop is cahanged to parfor - it hangs
    [x,y] = foobar(myMap,i);
end
toc;
function [x,y] = foobar(myMap,i)
x=[];
y=[];
end

4 个评论
显示 2更早的评论隐藏 2更早的评论

Walter Roberson 2023-5-28

Yes. parfor() and parfeval() do not use shared memory (or at least are not documented as doing so).

In a discussion within the last few days I showed from the R2023a documentation that as of R2023a it cannot be the case that parpool constants are "broadcast" to all workers in a single packet. I see you are using R2022b, at which time such a "broadcast" was at least a hypothetical possibility.

You might want to experiment with using backgroundPool and parfeval as background pools can use shared memory and are designed to reduce the amount of memory copying required.

Shlomo Geva 2023-5-28