PARFOR behavior sensitive to comments???

Question

Matt J 2014-2-7

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/115254-parfor-behavior-sensitive-to-comments

评论： Matt J 2014-2-19

I am finding that the speed performance of PARFOR is greatly impacted depending on whether certain inconsequential lines in the code are commented out or not. In particular, when running the code below with a pool of 12 workers, I obtain a time of 26 sec (4 times that of a normal for loop).

However, if I comment out either the first line (thus converting the mfile to a script) or if I comment out the inconsequential line

A=zeros(M,N); B=A;

the time drops sharply to less than 1 sec!!

This is under Windows 7 64-bit. Processor is Intel Xeon X5680 @3.33 Ghz, dual hexacore. I've tested with R2011b,R2012b,R2013b, all with similar results.

Can anyone reproduce this? Is there something obvious that I'm not seeing?

function test1  %Comment this out
P=100;
fun=@(X)imrotate(X,30,'crop'); 
Xtypical=zeros(P);
M=P^2;
N=P^2;
   A=zeros(M,N); B=A; %or Comment this out
        A=cell(1,N);
        tic;
        parfor j=1:N
         input=Xtypical;
         input(j)=1;
         T=fun(input); 
         A{j}=sparse(T(:));
        end 
        A=cell2mat(A);
        toc;

2 个评论
显示无隐藏无

amir 2014-2-9

编辑：amir 2014-2-9

with my system, if I comment first line, the result is generated after about 3.5 seconds but if I run it with first line (as a function) my system goes to a bad state, CPU usage is about zero but whole of my system has no response and I must reboot my system.

by smaller values for P (for example 70), my results was like yours, running as script was better than running as function (opposite to my results in here )

Matt J 2014-2-9

Thanks! Guess it's time to make a Bug Report.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Edric Ellis 2014-2-10

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/115254-parfor-behavior-sensitive-to-comments#answer_123825

Unfortunately what I think you're seeing there is a limitation of the way function handles are created. If you use the FUNCTIONS function on your function handle 'fun' just before entering the PARFOR loop, you can see that in the 'workspace' entry, it contains everything in the function workspace. In particular, the large value of 'B' is there - hence why commenting out the creation of that variable 'fixes' things.

Function handles are created differently at the command-line or in scripts, and don't end up grabbing everything in this way - hence why using a script also 'fixes' things.

Perhaps the best workaround is either to use internal functions, or generate the function handles in a scope that cannot see any large variables.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Matt J 2014-2-10

编辑：Matt J 2014-2-10

Pretty subtle pitfall, Edric! But it looks like you're right.

Could you also take a look at this one,

http://www.mathworks.com/matlabcentral/answers/114704-execution-time-with-or-without-parfor

and see if you have more insights into what is happening than we did?

There, we also saw parfor performance change greatly depending on whether running as a script or mfile function. Initially, I thought it might have been caused by reasons related to my post here, but now I guess not.

请先登录，再进行评论。

Answer 2

Eric Sampson 2014-2-10

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/115254-parfor-behavior-sensitive-to-comments#answer_123874

Matt, what happens if you create the anonymous function inside the parfor loop? If that helps, but if in actual code you need to define it before the parfor loop, then could you first create a string like fun_str = '@(X)imrotate(X,30,''crop'')' and then inside the parfor loop do fun = str2func(fun_str); ?

9 个评论
显示 7更早的评论隐藏 7更早的评论

amir 2014-2-16

编辑：amir 2014-2-16

在 MATLAB Online 中打开

I have tested and It seems handle class behaves different in parfor: (number of pools is 8)

%%%%%%%%%%%%%%%%%%%%%%%%%
% file : testclass.m
classdef testclass < handle
    properties
        a = 2000;
  end
end
%%%%%%%%%%%%%%%%%%%%%%%%%
% file: main.m
t1 = testclass();
t1.a = 1000;
parfor i = 1 : 10
  task = getCurrentTask();
  if isempty(task)
    ID = 0;
  else
    ID = task.ID;
  end
  previous_value = t1.a;
  t = t1;
  t.a = task.ID;
  time_check = tic();
  fprintf(1,'%d : %d , %d , %d \n', time_check , previous_value , t1.a , t.a);
end
t1.a
%%%%%%%%%%%%%%%%%%%%%%%%%
% results : 
688005680481 : 1000 , 1 , 1 
688005674577 : 1000 , 4 , 4 
688005592817 : 1000 , 5 , 5 
688005592714 : 1000 , 6 , 6 
688005679832 : 1000 , 8 , 8 
688005592765 : 1000 , 2 , 2 
688005601756 : 1000 , 3 , 3 
688005857083 : 5 , 5 , 5 
688005679596 : 1000 , 7 , 7 
688005856927 : 8 , 8 , 8
ans =
          1000

as you can see they don't share their attributes in each thread. It seems a complete clone is created for each thread. the problem is when we have a big handle class, overhead of creating and destroying these objects is very high. the question is can we change this behavior?

amir 2014-2-17

sorry but I can not find where Edric do that (creating data directly on the workers and making it persistent across multiple calling parfor).

Matt J 2014-2-19

http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper

请先登录，再进行评论。

PARFOR behavior sensitive to comments???

2 个评论
显示无隐藏无

采纳的回答

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

更多回答（1 个）

9 个评论
显示 7更早的评论隐藏 7更早的评论

另请参阅

类别

标签

产品

Community Treasure Hunt

PARFOR behavior sensitive to comments???

2 个评论 显示 无隐藏 无

采纳的回答

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

更多回答（1 个）

9 个评论 显示 7更早的评论隐藏 7更早的评论

另请参阅

类别

标签

产品

Community Treasure Hunt

2 个评论
显示无隐藏无

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

9 个评论
显示 7更早的评论隐藏 7更早的评论