Can't find file on worker - Parallel Programming

8 次查看（过去 30 天）
Thomas Clifford
Thomas Clifford 2021-11-11
评论： Raymond Norris 2021-11-13
Hi, I'm attempting to parallelize part of my code to speed up the process. But have become stuck when one of the attached files cannot be found by a worker. The file is being opened and modified on each driver for calculation before being reverted back to it's original state. The missing file is the "hydroNet.NetworkFile". Do I have to create a duplicate file for each individual worker?
parpool('local','AttachedFiles',{'epanet2.dll',hydroNet.NetworkFile});
spmd
folder = getAttachedFilesFolder(hydroNet.NetworkFile);
if not(libisloaded('epanet2'))
loadlibrary('epanet2');
end
oldFolder = cd(folder); % Change to that folder
[OK,output] = system(x(hydroNet,pumpCombo,nCluster,pumpCluster,idxSysLink,logger));
cd(oldFolder); % Change back to the original folder
end

请先登录，再回答此问题。

回答（1 个）

Raymond Norris
Raymond Norris 2021-11-11
You mean to say that folder is empty?
At the top of the spmd, add this
spmd
getAttachedFilesFolder
hydroNet.NetworkFile
getAttachedFilesFolder(hydroNet.NetworkFile)
...
end
  4 个评论
显示 2更早的评论
Thomas Clifford
Thomas Clifford 2021-11-12
I believe so. I'm using a toolkit that simulates a particular piece of software. The error i'm getting is that it can't find any data in the file. So I think it succesfully finds the file, but if it is in use by another worker then the file reads blank.
Raymond Norris
Raymond Norris 2021-11-13
You can have many processes reading the same file, but need to have semiphore or another way to have simultaneous writes (e.g. parallel database). I'm not sure why with one worker it will on occasion throw an error.
In regards to your situtation, each worker ought to be able to read from the same file, but then they might write to their own file, followed by a way to "true" up the output. Or maybe send everyone their local part and then write to one file. For example (I'm leaving off error handling)
spmd
% Import data
fid1 = fopen('input.txt','rt');
A = fscanf(fid1,'%s');
% Modify data
len = length(A);
variantB = A(randperm(len));
% Send each local part to everyone
replicatedB = gcat(variantB);
% Write to disk one output file
fid2 = fopen('output.txt','wt');
fprintf(fid2,[repmat('%c',1,len) '\n'],replicatedB);
fclose('all');
end

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel Computing 的更多信息

标签

产品


版本

R2021b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by