parallel processing for readtable function
显示 更早的评论
ps = parallel.Settings;
ps.Pool.AutoCreate = false; %do not autocreate parpool when encountering a |parfor|
ps.Pool.IdleTimeout = Inf; %do not shutdown parpool after Inf idle time
parfor j=1:10
tCOD{j,:}=readtable(full_file_name(j,:),'FileType','text', ...
'headerlines',end_of_header_line(j),'readvariablenames',0);
tCOD{j,:}=[];
end
When using 10 files with a size of about 100mb, only a couple of second differences are observed between for and parfor. Is there something wrong with the above steps for parfor. Is there any more convenient parallel processing approach that can be applied?
If the multiple "full_file_name(j,:)" can be read simultaneously (using multiple core) rather than sequentially, the speed of readtable can be increased significantly.
采纳的回答
Edric Ellis
2021-8-19
You've manually disabled the AutoCreate for parallel pools - I presume you're manually creating a pool with a separate parpool statement.
Whether or not parfor can go faster than a serial for loop depends on your underlying hardware for this case. It might simply be that the limiting factor is your disk drive, and trying to read from it multiple times simultaneously does not allow any speedup.
You could try manually running multiple copies of MATLAB and calling readtable from each of them simultaneously to see if they slow down when there is contention for disk access. I.e. run something like this in multiple copies of MATLAB:
while true
t = tic();
readtable(args..);
toc(t)
end
I suspect that you will see that as you start more copies of MATLAB, the toc time will increase.
1 个评论
Walter Roberson
2021-8-19
As a generalization: if you are not using server-quality hardware components, then optimal utilzation is typically two processes per controller. Not per drive . Any more and you are typically filling the memory channel between the controller and the rest of the hardware.
If you have only one drive per controller, then optimal utilization is typically two processes. Controllers can often be moving drive heads to position for the next read at the same time that the previous data is being transferred, so you want commands from multiple processes to be queued... but not too many processes.
The considerations are notably different for SSD.
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
