Parfor Execution time variation

Question

Anshika Goel 2024-6-6

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2125921-parfor-execution-time-variation

评论： Anshika Goel 2024-8-21，11:46

采纳的回答： Swastik

在 MATLAB Online 中打开

Hi,

I am using parfor for reading 600 .raw files.

c=zeros(1536,1536,600,'uint16');
parpool('threads',4);
parfor i=1:600
    fileName=[folder,'/',fileList(i).name];
    a=fopen(fileName,'r');
    Z=fread(a,[1536 1536],'uint16');
    fclose(a);
    c(:,:,i)=Z;
end

However, I am observing significant variability in the execution time, which ranges from 14 seconds to 110 seconds across different runs.

Why is this discrepancy occurring? Is there a way to achieve more consistent execution times?

4 个评论
显示 2更早的评论隐藏 2更早的评论

Christopher Mirfin 2024-6-12

Do you observe the same variability when running with a standard for-loop, or a process-based pool parpool("Processes",4) ?

Also, are you reading from your local hard drive or a network location?

Anshika Goel 2024-6-13

编辑：Anshika Goel 2024-6-13

In the standard for loop, I am not getting any variability it is taking 35-40 sec.

Whereas, in a process based pool, the variability is less (55-63 sec), but it is taking more time than the standard for loop.

And I am reading from a local hard drive,not from network location.

Is there any other solution, where I can reduce this execution time to less than 15 sec.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Swastik 2024-8-21，11:03

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2125921-parfor-execution-time-variation#answer_1502639

在 MATLAB Online 中打开

Hi @Anshika Goel,

I also have an Intel Xeon processor (4 cores) with 16GB RAM. I executed your code after generating 600 files as follows:

matrix = uint16(ones(1536)); 
folder = 'nums'; 
for i=1:600 
    fid = fopen([folder '/' num2str(i)], 'w');  
    mat = matrix .* i; 
    fwrite(fid, mat, 'uint16');  
    fclose(fid);  
end 

In my tests, the variability in execution time was not as significant as you mentioned; it ranged from 189 seconds to 200 seconds. the main bottleneck is likely due to file I/O operations. 

To optimize execution time, consider performing file reads asynchronously. I developed the following code using “parfeval” to read 600 files asynchronously:

c = zeros(1536, 1536, 600, 'uint16');  
folder = "nums"; 
pool = gcp('nocreate'); 
if isempty(pool) 
    pool = parpool('threads'); 
end 
futures = parallel.FevalFuture.empty(600, 0); 
for i = 1:600 
    fileName = fullfile(folder, num2str(i)); 
    futures(i) = parfeval(@readFile, 1, fileName); 
end 
for i = 1:600 
    c(:, :, i) = fetchOutputs(futures(i)); 
end 
function Z = readFile(fileName) 
    a = fopen(fileName, 'r'); 
    Z = fread(a, [1536 1536], 'uint16'); 
    fclose(a); 
end 