Remote workers still access data files from the client despite the files have been copied to the remote host beforehand

2 次查看（过去 30 天）
Adib Yusof
Adib Yusof 2021-2-10
回答： Jason Ross 2021-2-10
Hi,
I have two PCs, and I use Parallel Server and Parallel Computing to utilize the cores in both PCs simultaneously. Since the script I'm running load relatively large amount of data files (.mat), I copied the files to the remote host beforehand, and then use the following command:
MJSName = 'AdibMJS';
ScriptName = 'SatelliteDataProcessing';
PathInRemoteHost = 'D:\SatelliteData';
NumWorkers = 8; % 4 from each PC
C = parcluster(MJSName);
Job = batch(C, ScriptName, 'Pool', NumWorkers - 1, 'AdditionalPaths', PathInTheRemoteHost);
parfor i = 1:N
load('SomeData.mat')
% Some codes
end
By doing this, I thought I could avoid the files to be transferred during runtime through WLAN, which is slow in my setup. When I started the job, I could see that, through Task Manager of both PCs, there are Network usage by MATLAB, indicating that there are still some data being transferred through WLAN. This cause the parfor-loop to run slower compared to using less workers (e.g., 4) on just the local cluster. What I want to happen instead is for both PCs to load the data files locally and run the parfor-loop simultaneously. Note that in both PCs, the path that stores the data files is identical.
Please advise where did I do wrong? Thank you!

请先登录，再回答此问题。

采纳的回答

Jason Ross
Jason Ross 2021-2-10
It sounds like there might be some dependency analysis still going on that's sending the files over. You can turn that off by setting 'AutoAddClientPath' and 'AutoAttachFiles' to 'false' in your batch command.
See the batch documentation for more info.

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息

标签

产品


版本

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by