problem performing multiple, independent tasks simultaneously in parfor

2 次查看（过去 30 天）
David L.
David L. 2022-7-26
评论： David L. 2022-7-28
Here's a parfor code snippet that's intended to search for 2 different file types and load a file simultaneously, but fails due to different output types:
% run 3 independent things simultaneously to save time
parfor n 1:3
switch n
case 1
output{n} = dir('pathname\**\*.txt'); % this takes a long time
case 2
output{n} = dir('pathname\**\*.csv'); % this takes a long time
case 3
output{n} = load('pathname\matlab.mat'); % this also takes a long time
end
end
Here's how I'd prefer to write this function, which doesn't crash and runs MUCH faster than each function separately, but variables a, b, and c are not saved.
% run 3 independent things simultaneously to save time
parfor n 1:3
switch n
case 1
a = dir('pathname\**\*.txt'); % this takes a long time
case 2
b = dir('pathname\**\*.csv'); % this takes a long time
case 3
c = load('pathname\matlab.mat'); % this also takes a long time
end
end
I'm not I/O limited, these processes run simultaneously much faster. Run time with the second snippet is the same as the single longest operation. How can I run either method of code snippet with the outputs saved?
  3 个评论
显示 1更早的评论
Bruno Luong
Bruno Luong 2022-7-26
编辑：Bruno Luong 2022-7-26
Redargless it does't work (I don't know why just by reading your code)
Your parfor doesn't do any real computation, just reading, so it is mostly limited by your perepherical HW (reading speed and transfert data through the bus). Not sur parfor will not in contrary to what you would expect slow down even more, since the caching is worse.
David L.
David L. 2022-7-26
Edited above for clarity of my problem. Yes, the parfor loop runs much faster. Each function takes ~5 minutes, sequentially it's 15 minutes. In the parfor, it takes 5 minutes total.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2022-7-27
Ran in:
That code that assigns to output{n} will not fail due to different output types. All three of the outputs (for the code you posted) would be struct(), and besides you are outputting a cell array.
parfor n = 1:3
switch n
case 1
output{n} = dir('*.txt'); % this takes a long time
case 2
output{n} = dir('*.csv'); % this takes a long time
case 3
output{n} = struct('this', 5, 'that', 7); % this also takes a long time
end
end
Starting parallel pool (parpool) using the 'local' profile ... Connected to the parallel pool (number of workers: 2).
output
output = 1×3 cell array
{0×1 struct} {0×1 struct} {1×1 struct}
Have you considered using parfeval?
F(1) = parfeval(@() dir('*.txt'), 1);
F(2) = parfeval(@() dir('*.csv'), 1);
F(3) = parfeval(@() struct('this', 5, 'that', 7), 1);
wait(F)
a = fetchOutputs(F(1))
a = 0×1 empty struct array with fields: name folder date bytes isdir datenum
b = fetchOutputs(F(2))
b = 0×1 empty struct array with fields: name folder date bytes isdir datenum
c = fetchOutputs(F(3))
c = struct with fields:
this: 5 that: 7

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by