Loading mat files within parfor takes long time

4 次查看（过去 30 天）

Dave Lee 2018-12-24

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/437047-loading-mat-files-within-parfor-takes-long-time

评论： Walter Roberson 2018-12-24

Hi,

I am writing the code below to load/process 2627 files under parfor with the size of each file is 100MB, makes it total of 272GB. It is taking very long time to process. I actually use a progress bar for parfor, not shown in this simplified code, and the progress bar even hasn't started yet. I think it is still doing some overhead work. When I use for-loop, I think it is not slower than parfor. Can someone please give me a suggestion? My computer: 40 cores and RAM of 383GB

Merry Xmas,

clear all;
folder_mat_file = '.\Mat_files_full\';
mat_filename_valid = dir([folder_mat_file '*.mat']);
no_mat_files = numel(mat_filename_valid);
if isempty(gcp) 
     parpool;
end 
poolobj  = gcp;
for n=1:no_mat_files
    filename_cell{n} = [folder_mat_file mat_filename_valid(n).name];
end
addAttachedFiles(poolobj,filename_cell)
parfor n=1:no_mat_files
    % data_struct is temporary variable
    data_struct = parload([folder_mat_file mat_filename_valid(n).name]);
    %% Here we do processing
end
%% Below is the function parload use to load file within parfor
function x = parload(filename)
    x = load(filename);
end