Trouble with accessing variable in parfor loop

16 次查看（过去 30 天）
Quentin
Quentin 2014-3-7
评论： Sergiy Golovin 2019-10-26
Hello all,
I am trying to do batch processing and I am having trouble implementing the parfor loop. I have a variable (a movie frame), that I want to subtract from every frame in a movie after I load in the frames. The basics of my loop look like this.
load('gray_mean_frame.mat');
set(handles.testMovieButton, 'String', 'Please Wait');
poolobj = parpool('local',4);
addAttachedFiles(poolobj,{'gray_mean_frame.mat'});
parfor frameNumber = 1:numFrames
movieFrame = read(readerobj, frameNumber);
ghostFrame = gray_mean_frame - movieFrame;
end
I always get the error
"An UndefinedFunction error was thrown on the workers for 'gray_mean_frame'.
This might be because the file containing 'gray_mean_frame' is not accessible
on the workers. Use addAttachedFiles(pool, files) to specify the required
files to be attached. See the documentation for
'parallel.Pool/addAttachedFiles' for more details."
But I think I am correctly passing the file gray_mean_frame to the workers. gray_mean_frame.mat is a file which contains the variable gray_mean_frame, among a couple others. I've also tried making gray_mean_frame a global variable but it did not help. It is not clear to me what I am doing wrong. Thank you for any help.

请先登录，再回答此问题。

采纳的回答

Edric Ellis
Edric Ellis 2014-3-7
The problem here is that unfortunately the PARFOR analysis is based on the text of your program, and in this case the analysis cannot deduce that gray_mean_frame is a variable. You should be able to fix this by making a minor change:
data = load('gray_mean_frame.mat');
gray_mean_frame = data.gray_mean_frame;
  2 个评论
Andreas Holzhammer
Andreas Holzhammer 2014-9-5
Thanks Edric, you helped me as well! I ran into the same problem because in a function, I used an input parser and then dynamically made variables out of every field.
varargin={};
p = inputParser;
addParameter(p,'autosave',false,@islogical);
addParameter(p,'initialSampleSize',[],@isnumeric);
parse(p,varargin{:});
opts=p.Results;
params=fieldnames(opts);
for i=1:numel(params)
eval(sprintf('%s=opts.(params{i});',params{i}));
end
myZeros=cell(2,1);
parfor i=1:2
myZeros{i}=zeros(initialSampleSize);
end
It suffices to make a dummy statement like above before the par loop, e.g.
initialSampleSize=initialSampleSize;
while just
initialSampleSize;
won't work. Could anybody please explain why that is and whether there is a dynamic workaround? I don't want to have to add that dummy line every time somebody adds a parameter.
Sean de Wolski
Sean de Wolski 2014-9-5
Andreas, I would pass input parsing off to a second function (sub function probably) and have it return the parameters you care about
% First line of your main function
[x,y,z,initialSampleSize] = parse_inputs(varargin{:})
You're going to have to add them to the parse anyway so you might as well add them as outputs which is way cleaner and faster than using eval to poof them into existence.

请先登录，再进行评论。

更多回答（5 个）

Vincenzo De Lipsis
Vincenzo De Lipsis 2019-7-16
I found a solution to a similar issue which might be helpful to somebody.
I had a parfor loop inside a function, after loading a .mat file full of variables. For some reason the workers are not able to see variables loaded from an external .mat file; all variables need to be stated individually. But if the parfor loop part of the code is saved as an external .m Matlab script, which can be called inside the Matlab function, all variables are visible to the workers. I don't know the logic of this, and I didn't find explanation on the web.
  1 个评论
Sergiy Golovin
Sergiy Golovin 2019-10-26
Thank you for pointing it out! This is exactly what helped me. Simply putting the parfor loop in a separate script and then calling it fixes the error and allows the calculations to proceed. (Note: that Matlab is going to complain about the parfoor loop in the script by underlining your statements. Just ignore it.)

请先登录，再进行评论。

Quentin
Quentin 2014-3-8
Thanks Edric, that solved my problem.
Kazutaka
Kazutaka 2014-9-8
编辑：Walter Roberson 2015-8-12
I have a similar issue, but not involving to load any files whose names are confusing with the variable names.
Here is the part of the code and once it reaches
"SpikeData1_1=extractdatapt...."
line, then it seems that parfor thinks I am loading a file and keep getting an error:
Error using COHshuf_SpkSpk_M1S1_allTM_y1_parfor>(parfor body) (line 87)
An UndefinedFunction error was thrown on the workers for 'SpikeData1'. This
might be because the file containing 'SpikeData1' is not accessible on the
workers. Use addAttachedFiles(pool, files) to specify the required files to be
attached. See the documentation for 'parallel.Pool/addAttachedFiles' for more
details.
Error in COHshuf_SpkSpk_M1S1_allTM_y1_parfor (line 86)
parfor i=1:length(sigpairs)
Caused by:
Undefined function or variable 'SpikeData1'.
-----
Any ideas?
pc = parcluster('local')
pc.JobStorageLocation = strcat('/tmp/kazutaka/',getenv('SLURM_JOB_ID'))
matlabpool (pc, 12)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% %NOW get M1S1pairs
m1=size(TMunion_M1,1);
s1=size(TMunion_S1,1);
max_units=max([m1 s1]);
covmat=[];
for r=1:m1
d=[];
d=r*ones(s1,1);
d(:,2)=repmat(1:s1,1,1);
covmat=[covmat;d];
end
frqname={'26';'613';'beta';'gamma'};
frqrng={3:11;12:26;31:63;64:107};
Tm=[];Tm=1.75:0.01:2.74;
%Create a sliced structure for M1 data
for i=1:length(TMunion_M1);
%tmp=[]; tmp=TMunion_M1(covmat(sigpairs(i),1),:);
strtmp1=sprintf('elec%03d_%03d',TMunion_M1(i,1),TMunion_M1(i,2));
eval(['lengthtemp = length(M1spk_MB_',strtmp1,');']);
for ll = 1:lengthtemp
eval(['SpikeData1(',num2str(TMunion_M1(i,1)),',',num2str(TMunion_M1(i,2)),',ll).times=M1spk_MB_',strtmp1,'(ll).times;']);
end
end
%Create a sliced structure for S1 data
for i = 1:length(TMunion_S1);
%tmp=[]; tmp=TMunion_S1(covmat(sigpairs(i),1),:);
strtmp1=sprintf('elec%03d_%03d',TMunion_S1(i,1),TMunion_S1(i,2));
eval(['lengthtemp = length(S1spk_MB_',strtmp1,');']);
for ll = 1:lengthtemp
eval(['SpikeData2(',num2str(TMunion_S1(i,1)),',',num2str(TMunion_S1(i,2)),',ll).times=S1spk_MB_',strtmp1,'(ll).times;']);
end
end
for frq=1:4
%frq = 1;
tmp=[];eval(['tmp=SIGCOHpk' frqname{frq} '(dsnum).pk;']);
inx=[];inx=find(tmp < unique(Cconf(dsnum-1).cf));
sigpairs=[];eval(['sigpairs=SIGCOHinx' frqname{frq} '(dsnum).cinx;']);
sigpairs(inx)=[];%delete the nonsog (1,1(pairs
peaktm=[];eval(['peaktm=SIGCOHpktm' frqname{frq} '(dsnum).ptm;']);
peaktm(inx)=[];
peak=[];eval(['peak=SIGCOHpk' frqname{frq} '(dsnum).pk;']);
peak(inx)=[];
%k=0;%count for sigpeak after permutation
%tic
sortedC = zeros(length(sigpairs),1000);
parfor i=1:length(sigpairs)
size(SpikeData1)
size(SpikeData2)
k = 0; % Need to redefine within parfor loop
% Need to rewrite the whole thing here and create a sliced cell or
% structure that can be used on parfor without using eval and
% sprintf to retrieve variables.
%unit
tmp=[]; tmp=TMunion_M1(covmat(sigpairs(i),1),:);
% strtmp1=sprintf('elec%03d_%03d',tmp(1),tmp(2));
% SpikeData1=sprintf('M1spk_MB_%s',strtmp1);
% %lfp
% tmp=[];tmp=TMunion_S1(covmat(sigpairs(i),2),:);
% strtmp2=sprintf('elec%03d_%03d',tmp(1),tmp(2));
% SpikeData2=sprintf('S1spk_MB_%s',strtmp2);
%
% eval(sprintf('SpikeData1=%s;',SpikeData1));
% eval(sprintf('SpikeData2=%s;',SpikeData2));
SpikeData1_1=extractdatapt(squeeze(SpikeData1(tmp(1),tmp(2),1:100)),[Tm(peaktm(i))-0.25 Tm(peaktm(i))+0.25]);
Joseph MacGregor
Joseph MacGregor 2015-8-12
I am receiving the same error message as the questioner(s) in R2015a for reasons that I don't understand. The answers imply to me that all variables must be explicitly assigned to the pool, but in my 2+ years experience with parfor this was never necessary, so I'm confused as to what changed.
I start the local pool and then initialize a bunch of variables. Because there are so many to initialize, I use a cell containing their names and eval in a for loop to generate them. Because of that approach, I can see why parfor might not recognize them following the above discussion, but it worked fine in R2014b. The same error occurs if I move the pool start until after those initializations. I am in no mood to explicitly call out every variable I need in the parfor loop, given that I never had to before.
Any thoughts? Thanks.
  1 个评论
Walter Roberson
Walter Roberson 2015-8-12
parfor and eval are essentially incompatible.
I suggest you use dynamic field names in a struct.

请先登录，再进行评论。

BHUSHAN MUTHIYAN
BHUSHAN MUTHIYAN 2017-9-5
编辑：Walter Roberson 2017-9-6
Hello,
I am facing similar issue accessing the out variable.
Can some one please help me. I have a function which returns "out" as the output%%
function out = flexible(filter, bias, rowS, input, convModel, sumModel) %#codegen
colS = rowS;
ins = [size(input,1),size(input,2),size(input,3),size(input,4)];
filter = rot90(filter, 2);
fis = [size(filter,1),size(filter,2),size(filter,3),size(filter,4)];
bis = [size(bias,1),size(bias,2),size(bias,3),size(bias,4)];
dimx = ins(1) - fis(1) + 1;
dimy = ins(2) - fis(2) + 1;
dimfox = 1+floor((dimx-1)/rowS);
dimfoy = 1+floor((dimy-1)/colS);
out = zeros(dimfox, dimfoy, fis(4), ins(4), 'like', sumModel);
frac = 32 - (convModel.WordLength - convModel.FractionLength);
H = vision.Convolver('OutputSize', 'Valid', 'CustomProductDataType', numerictype([],32,frac), 'CustomOutputDataType', numerictype([],32,frac));
parfor im = 1:ins(4)
carve = zeros(dimfox, dimfoy, 'like', sumModel);
for f = 1:fis(4)
for ch = 1:fis(3)
carve(:) = carve + step(H, input(:,:,ch,im), filter(:,:,ch,f));
end
carve(:) = carve + bias(1, f);
out(:,:,f,im) = carve;
end
end
end
It gives me below error:
The variable out in a parfor cannot be classified.
See Parallel for Loops in MATLAB, "Overview".
Use help codegen for more information on using this command.
Error using codegen
Also, can someone let me know if we can change value of global variables inside parfor loop?
  1 个评论
Walter Roberson
Walter Roberson 2017-9-6
编辑：Walter Roberson 2017-9-6
You initialize carve in the im loop, and you accumulate into it in the f and ch loops. You write the current value of carve into out in the f loop. That means that you are essentially writing partial calculations into out. Are you sure that is what you want??
fis3 = fis(3);
fis4 = fis(4);
parfor im = 1:ins(4)
carve = zeros(dimfox, dimfoy, fis4, 'like', sumModel);
for f = 1:fis4
if f ~= 1
carve(:, :, f) = carve(:, :, f-1);
end
for ch = 1:fis3
carve(:, :, f) = carve(:, :, f) + step(H, input(:,:,ch,im), filter(:,:,ch,f));
end
carve(:, :, f) = carve(:, :, f) + bias(1, f);
end
out(:,:,:,im) = carve;
end

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by