Remote workers still access data files from the client despite the files have been copied to the remote host beforehand
1 次查看(过去 30 天)
显示 更早的评论
Hi,
I have two PCs, and I use Parallel Server and Parallel Computing to utilize the cores in both PCs simultaneously. Since the script I'm running load relatively large amount of data files (.mat), I copied the files to the remote host beforehand, and then use the following command:
MJSName = 'AdibMJS';
ScriptName = 'SatelliteDataProcessing';
PathInRemoteHost = 'D:\SatelliteData';
NumWorkers = 8; % 4 from each PC
C = parcluster(MJSName);
Job = batch(C, ScriptName, 'Pool', NumWorkers - 1, 'AdditionalPaths', PathInTheRemoteHost);
parfor i = 1:N
load('SomeData.mat')
% Some codes
end
By doing this, I thought I could avoid the files to be transferred during runtime through WLAN, which is slow in my setup. When I started the job, I could see that, through Task Manager of both PCs, there are Network usage by MATLAB, indicating that there are still some data being transferred through WLAN. This cause the parfor-loop to run slower compared to using less workers (e.g., 4) on just the local cluster. What I want to happen instead is for both PCs to load the data files locally and run the parfor-loop simultaneously. Note that in both PCs, the path that stores the data files is identical.
Please advise where did I do wrong? Thank you!
0 个评论
采纳的回答
Jason Ross
2021-2-10
It sounds like there might be some dependency analysis still going on that's sending the files over. You can turn that off by setting 'AutoAddClientPath' and 'AutoAttachFiles' to 'false' in your batch command.
0 个评论
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!