Parfor Execution time variation

2 次查看（过去 30 天）
Anshika Goel
Anshika Goel 2024-6-6
评论： Anshika Goel 2024-8-21
Hi,
I am using parfor for reading 600 .raw files.
c=zeros(1536,1536,600,'uint16');
parpool('threads',4);
parfor i=1:600
fileName=[folder,'/',fileList(i).name];
a=fopen(fileName,'r');
Z=fread(a,[1536 1536],'uint16');
fclose(a);
c(:,:,i)=Z;
end
However, I am observing significant variability in the execution time, which ranges from 14 seconds to 110 seconds across different runs.
Why is this discrepancy occurring? Is there a way to achieve more consistent execution times?
  4 个评论
显示 2更早的评论
Christopher Mirfin
Christopher Mirfin 2024-6-12
Do you observe the same variability when running with a standard for-loop, or a process-based pool parpool("Processes",4) ?
Also, are you reading from your local hard drive or a network location?
Anshika Goel
Anshika Goel 2024-6-13
编辑：Anshika Goel 2024-6-13
In the standard for loop, I am not getting any variability it is taking 35-40 sec.
Whereas, in a process based pool, the variability is less (55-63 sec), but it is taking more time than the standard for loop.
And I am reading from a local hard drive,not from network location.
Is there any other solution, where I can reduce this execution time to less than 15 sec.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Swastik Sarkar
Swastik Sarkar 2024-8-21
Hi @Anshika Goel,
I also have an Intel Xeon processor (4 cores) with 16GB RAM. I executed your code after generating 600 files as follows:
matrix = uint16(ones(1536));
folder = 'nums';
for i=1:600
fid = fopen([folder '/' num2str(i)], 'w');
mat = matrix .* i;
fwrite(fid, mat, 'uint16');
fclose(fid);
end
In my tests, the variability in execution time was not as significant as you mentioned; it ranged from 189 seconds to 200 seconds. the main bottleneck is likely due to file I/O operations. 
To optimize execution time, consider performing file reads asynchronously. I developed the following code using “parfeval” to read 600 files asynchronously:
c = zeros(1536, 1536, 600, 'uint16');
folder = "nums";
pool = gcp('nocreate');
if isempty(pool)
pool = parpool('threads');
end
futures = parallel.FevalFuture.empty(600, 0);
for i = 1:600
fileName = fullfile(folder, num2str(i));
futures(i) = parfeval(@readFile, 1, fileName);
end
for i = 1:600
c(:, :, i) = fetchOutputs(futures(i));
end
function Z = readFile(fileName)
a = fopen(fileName, 'r');
Z = fread(a, [1536 1536], 'uint16');
fclose(a);
end
In my tests, this approach reduced the execution time to between 14-16 seconds.
You can learn more about “parfeval” from here:
https://www.mathworks.com/help/releases/R2023a/parallel-computing/parallel.pool.parfeval.html
I hope this helps.

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品


版本

R2023a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by