Parallel pool takes extremely long to startup on virtual machines

26 次查看（过去 30 天）
Chris Burgner
Chris Burgner 2022-8-10
回答： Alison Eele 2022-8-12
Hello,
I have spent some time searching the forums about this question and have read many of the threads regarding the parallel pool runtime/startup issues. Where there were suggestions I tested them. Unfortunately we are still running into significant runtime issues.
The general description: starting up the parallel pool and executing a very simple parallel loop on our virtual machines takes an extremely long time. All virtual machines are accessed through Dell VDI thin-clients and are running windows.
In all cases I am using the parallel computing toolbox on my named-user license and not attempting to pull from a license server. I used MATLAB 2021b for these tests.
I ran multiple tests, and in each case I ran the following script:
tic;
nWorker = ...
parpool(nWorker);
a = zeros(100000,1);
parfor ii = 1 : size(a,1)
a(ii) = rand;
end
toc;
In some cases I set nWorker to total available cores. In other cases I limited it. Some benchmark results:
  • Desktop Computer: Dell Precision 5820 | Xeon W-2135 | 6 (of 6) cores @ 3.7GHz | 32GB Memory | Win10. Runtime: 10 seconds
  • Virtual Machine: Xeon Gold 6246 | 8 (of 8) cores @ 3.3GHz | 32GB Memory | Win10. Runtime: 65 seconds
  • Virtual Machine: Xeon Platinum 8280 | 12 (of 28 ) cores @ 2.7GHz | 64GB Memory | Win10. Runtime: 175 seconds
  • Virtual Machine: Xeon Platinum 8280 | 27 (of 28 ) cores @ 2.7GHz | 64GB Memory | Win10. Runtime: 614 seconds
As you can see, in the worst case the virtual machine (whose specs far outmatch, on paper, the desktop computer) runs 61.4x slower.
I followed the advice on this thread:
https://stackoverflow.com/questions/17125785/matlabpool-very-slow-to-open-workers
But that did not have much improvement at all.
I am looking for advice on why this may be happening and what I might be able to do to fix it!
Thank you in advance,
Chris

请先登录，再回答此问题。

回答（1 个）

Alison Eele
Alison Eele 2022-8-12
Hi Chris, there are a unfortunately a few possibilities, your observation that the time is scaling with the number of workers on the machine is especially interesting since that suggests to me some sort of bottleneck.
If you observe the parpool starting on the virtual machine from something like Task Manager (Window) or top (Linux) do you see the additional MATLAB processes for the workers start quickly and reach a memory footprint close to about 800MB (Windows) or are the additional worker processes taking a while to start up?
Slow starting workers would suggest that the disk I/O needed to start a lot of MATLAB processes simultaneously might be the bottleneck or if you're seeing system resources spike close to 100% the machine is needing to work hard to start MATLAB.
If the processes are starting quickly do you see the parallel status indicator in your client MATLAB turn green and stay green for a long time before the pool is available to use?
If so then one of the setup steps for the parpool seems to be taking a while, this can be affected by the number of entries on the user's MATLAB path, especially anything that might include a network directory. I wouldn't expect this to scale with the number of workers so seems unlikely in your case.
If the processes take a long time to appear in Task Manager or top then that suggests to me the delay is in the parpool job creation and I'd be looking to see if the JobStorageLocation of the local profile is slow to write to.
Our support teams may be able to help you further in isolating the cause of the delay. https://www.mathworks.com/support/contact_us.html

类别

在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by