Parfor Execution time variation
显示 更早的评论
Hi,
I am using parfor for reading 600 .raw files.
c=zeros(1536,1536,600,'uint16');
parpool('threads',4);
parfor i=1:600
fileName=[folder,'/',fileList(i).name];
a=fopen(fileName,'r');
Z=fread(a,[1536 1536],'uint16');
fclose(a);
c(:,:,i)=Z;
end
However, I am observing significant variability in the execution time, which ranges from 14 seconds to 110 seconds across different runs.
Why is this discrepancy occurring? Is there a way to achieve more consistent execution times?
4 个评论
Christopher Mirfin
2024-6-7
Can I ask for a little more information for clarification:
- Are you timing the entire parfor block? e.g. tic, parfor ... end, toc
- What is the spread of timings? Is it usually 14 seconds, and occasionally up to 110 seconds?
- Machine information (operating system), and MATLAB Release.
Anshika Goel
2024-6-10
Christopher Mirfin
2024-6-12
Do you observe the same variability when running with a standard for-loop, or a process-based pool parpool("Processes",4) ?
Also, are you reading from your local hard drive or a network location?
Anshika Goel
2024-6-13
编辑：Anshika Goel
2024-6-13
采纳的回答
Swastik Sarkar
2024-8-21
I also have an Intel Xeon processor (4 cores) with 16GB RAM. I executed your code after generating 600 files as follows:
matrix = uint16(ones(1536));
folder = 'nums';
for i=1:600
fid = fopen([folder '/' num2str(i)], 'w');
mat = matrix .* i;
fwrite(fid, mat, 'uint16');
fclose(fid);
end
In my tests, the variability in execution time was not as significant as you mentioned; it ranged from 189 seconds to 200 seconds. the main bottleneck is likely due to file I/O operations.
To optimize execution time, consider performing file reads asynchronously. I developed the following code using “parfeval” to read 600 files asynchronously:
c = zeros(1536, 1536, 600, 'uint16');
folder = "nums";
pool = gcp('nocreate');
if isempty(pool)
pool = parpool('threads');
end
futures = parallel.FevalFuture.empty(600, 0);
for i = 1:600
fileName = fullfile(folder, num2str(i));
futures(i) = parfeval(@readFile, 1, fileName);
end
for i = 1:600
c(:, :, i) = fetchOutputs(futures(i));
end
function Z = readFile(fileName)
a = fopen(fileName, 'r');
Z = fread(a, [1536 1536], 'uint16');
fclose(a);
end
In my tests, this approach reduced the execution time to between 14-16 seconds.
You can learn more about “parfeval” from here:
I hope this helps.
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
