Parfor with options = statset('U​seParallel​',true);

5 次查看(过去 30 天)
Hi
I have access to a large cluster many CPU's. I am hoping to prefrom the same operations on several hundred .edf input files with a parfor loop. So far so good. Part of this processing would benifit from an additional pararellel loop for all-vs-one testing.
I am therefore wondering if it is possiable to use both of these options at once. For example if I have a parpool(6) and a classifier with five classes. Therefore 6*5 = 30 cpu cores, which I have access too. I know that nested parfor loops are not generally supported, however I am wondering if there is anyway to 'trick' matlab into preforming as such?
Alternatively I could just use parfor(30) for example, however I am intrested to see what the preformacne gains/losses are of implementing this.
Kind regards,
Christopher

采纳的回答

Edric Ellis
Edric Ellis 2023-11-23
If you are able to "flatten" the parallelism so that it is all exposed to a single parfor loop with enough iterations to keep all your workers busy, that will always be the most efficient. But that's not always possible.
The restriction on nested parfor loops is that you cannot nest a parfor loop directly inside another parfor loop. However, you can "hide" the inner parfor loop inside a function. Today, achieving nested parallelism is a bit tricky. One way to achieve that is to have each worker create its own parpool. Depending on the workload, you might be able to set your cluster to launch one multi-threaded worker per host on the cluster, and then use parpool("Threads") on the cluster workers to launch an inner thread-based pool.
  2 个评论
Christopher McCausland
Hi Edric,
Thank you for the information. I have 'hidden' other parfor incompatibilites in functions before, however I hadn't realsied I could do so with parpool, it does however make a lot of sense.
My other question is about flattening efficency, in this specific use case the second parpool is leveraging the inbuilt function Mdl = fitcecoc(Tbl,ResponseVarName,'Options',options);options = statset('UseParallel',true); function, I haven't had a chance to time it however if anyone is going to know it's you, would there be any significant differacne in compute time vs a flattened parfor loop, and is this just from the overhead of several parpools or something else?
Kind regards,
Christopher
Edric Ellis
Edric Ellis 2023-11-28
The thing you're trying to avoid is having idle workers at the back end of your computation because they've run out of things to do. When you try to use nested parallelism, this might increase the amount of time wasted in that way. The happiest scenario is when you've got plenty of iterations of a top-level parallel construct so that they can be load-balanced by the parfor internals to smooth out any differences in execution times between the iterations etc.

请先登录,再进行评论。

更多回答(1 个)

Walter Roberson
Walter Roberson 2023-11-22
I know that nested parfor loops are not generally supported, however I am wondering if there is anyway to 'trick' matlab into preforming as such?
You would need to start separate MATLAB sessions.
You should perhaps also consider queuing up at lot of parfeval() sessions.
Hmmmm -- I wonder if parfeval() can call parfeval() to queue more work? I don't see why not... though you do risk deadlock if all of the sessions are locked up waiting for other sessions that cannot execute because there are no available cores...
  2 个评论
Christopher McCausland
Hi Walter,
Thank you, I hadn't considered multiple parfeval() calls. I will give this a go tommorrow and come back to you then. I can see the (reasonable) limitation of nested parpools() for most users, however with an HPC it's not as clear cut.
Thank you!
Christopher
Edric Ellis
Edric Ellis 2023-11-23
A worker cannot call parfeval to schedule more work on the pool that it is part of (one reason is to avoid the risk of deadlock). You can use afterEach to have the client schedule more work.

请先登录,再进行评论。

类别

Help CenterFile Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息

产品


版本

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by