can I accelerate parfor loop running external code?

Question

Eungyu Park 2020-4-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/514472-can-i-accelerate-parfor-loop-running-external-code

评论： Walter Roberson 2020-4-4

Here is my question:

if I run following code, the toc time is 0.296749.

tic
lambda=1;
for k=1:lambda
    filename_nam=sprintf('mf2005 test_%03d.nam',k);
    [~,~]=system(filename_nam);
    .
    .
    .
end
toc

Otherwise, if I run following code, the toc time is 12.920165. This result is reasonable because 0.296749 * 43 is approximately 12.9.

tic
lambda=43;
for k=1:lambda
    filename_nam=sprintf('mf2005 test_%03d.nam',k);
    [~,~]=system(filename_nam);
    .
    .
    .
end
toc

Lastly, if I run following code, the toc time is 3.902111. This is obviously improvement by using parfor but I wonder if there is further way of improving the efficiency. In my parallel pool, I have 36 worker (36 core CPU) and the required memory for the exe code is very small compared to my 64Gb memory. The operating system is Windows 10. What is the limiting condition? code? a specific hardware? or any other?

tic
lambda=43;
parfor k=1:lambda
    filename_nam=sprintf('mf2005 test_%03d.nam',k);
    [~,~]=system(filename_nam);
    .
    .
    .
end
toc

3 个评论
显示 1更早的评论隐藏 1更早的评论

Eungyu Park 2020-4-2

The part inside the parfor loop consists of an execution subpart and a subpart that reads the execution result. It looks like the code using .NET framework is trying to read the results before execution ends. This causes an error. If I just run the code without the reading subpart, the computation rate seems to be very fast. However, the problem is that the toc time is much earlier than termination of all executions (The toc time is already displayed in the command window even though it is still being calculated). Also, the shell window popup consumes computer resources and slows down computation. Are there any solutions?

Walter Roberson 2020-4-2

You can test if a process started with .NET is still going, or you can wait for it (with an optional timeout)

https://support.microsoft.com/en-ca/help/305369/how-to-wait-for-a-shelled-application-to-finish-by-using-visual-c

https://stackoverflow.com/questions/3147911/wait-until-a-process-ends

Anyhow, typically the answer to the question of the limiting condition is either communications overhead or use of a limited resource (for example disk I/O bandwidth.)

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Eungyu Park 2020-4-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/514472-can-i-accelerate-parfor-loop-running-external-code#answer_423747

Thank you Mohammad and Walter for your help! It seems that 'system' and 'System.Diagnostics.Process()' take about same time for my parfor loop execution of external code.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Walter Roberson 2020-4-4

Sometimes when multiple processes use a shared resource but do not each need 100% of the resource, then it can be beneficial to start a limited number of the processes and wait for one to finish before starting the next, so that there might be a small number of processes accessing the resource but not too many. This can reduce contention for the resource. This can be especially important for disc access.

请先登录，再进行评论。