Is there a parallel version of splitapply()?

2023 9 15
2 个回答
回答已采纳
更新时间：2023 9 20
7 次查看（30 天）

0 个投票

Group-based computation is naturally sutiable for parallel computation. I wonder why Matlab has not yet a built-in parfor-splitapply? In Github, there is a repository called Matlab-FunUtils, which has cmap.m. It's like a parfor version of arrayfun. I couldn't find a similar function for splitapply. I have been trying to modify Matlab's splitapply, but the codes are very difficult to understand. Any suggesstoins? Should I write from scratch?

请先登录，再回答此问题。

 采纳的回答

Edric Ellis
Edric Ellis 2023-9-15

0 个投票

tall arrays support parallel execution of splitapply. Would that work for your case?

6 个评论
显示 4更早的评论 隐藏 4更早的评论

Simon
Simon 2023-9-15
Thanks for your help. This is what I did:
tallA = tall(A);
g = findgroups(cat1, cat2);
tallg = tall(g);
tallout = splitapply(@mean, tallA, tallg);
shortout = gather(tallout);
I think, then, I can use shortout for 'normal' processings. Was I doing it right?
Simon
Simon 2023-9-15
Very glad I've asked the question. It led me to the awareness of a very important, very useful feature that I had ignored.
note: advanced Matlab utilities are hidden at the not-so-fine-print bottom of documentaion.
Edric Ellis
Edric Ellis 2023-9-15
I think you should apply findgroups to tallA.
Also note that if you want mean by groups, you can probably use some ideas from this example: https://www.mathworks.com/help/matlab/import_export/grouped-statistics-calculations-with-tall-arrays.html
Simon
Simon 2023-9-16
移动：Voss 2023-9-16
@Edric Ellis "I think you should apply findgroups to tallA. "
Great tips! Thank you.
"Also note that if you want mean by groups, you can probably use some ideas from this example: https://www.mathworks.com/help/matlab/import_export/grouped-statistics-calculations-with-tall-arrays.html"
That's a very relevant link. Mean is just for an example. What I really do is to wrangle out a new array/table out of each group.
Simon
Simon 2023-9-20
@Edric Ellis When my codes have nested splitapply, the tall approach fails. The situation occurs when I have 2-layer nested groupping structure, and splitapply() is used to process groups in each layer. For example, like in patients.mat, suppose there is a new grouping variable, 'BloodType', within each Male/Female group in Gender variable. tall will not allow splitapply to work in both level of groups. I guess this is the intrinsic limitation of parallel toolbox? Similarly, it does not allow nested parfor-loop.
Edric Ellis
Edric Ellis 2023-9-20
Ran in:
Ran in:
Yes, typically this sort of algorithm cannot work in a nested way. It isn't quite the same restriction as nested parfor (in fact, with thread-pools, you can get genuine 2-level parallelism with parfor, but there's a separate restriction that you need to "hide" the inner parfor inside a function).
As to your original question - does it work to make the groups based on the combinations of variables, as per the 2nd syntax of findgroups ? I.e. something like:
load patients
G = findgroups(Smoker, Age >= 40)
G = 100×1
3 2 1 2 2 2 3 2 1 1
This divides the data into 4 groups for each combination of Smoker and Age >= 40.

请先登录，再进行评论。

更多回答（1 个）

Bruno Luong
Bruno Luong 2023-9-15

0 个投票

To me there are 2 reasons:
  • Not everyone have parallel toolbox
  • The user function can be multi-threaded and already use efficiently the CPU cores, add parallel on top will then likely reduce the performance. The parallel computation should never be applied automatically anywhere. It should be judged case by case.

2 个评论
显示 无 隐藏 无

Simon
Simon 2023-9-15
I do computations with tall, tall tables. parfor saves huge amount of runtime for me compared with for-looping through rows serially. Now I am doing group-based computations and the number of groups I have is large. For that, I feel that splitapply is a little sluggish.
Simon
Simon 2023-9-16
@Bruno Luong "Not everyone have parallel toolbox"
I know that. I was very hesitant to use parallel toolbox. I thought it would be extremely difficult to adopt parallelization. But I dived in giving it a try. It turned out Matlab has done a wonderful job making a daunting task beginner-easy. I google how parallelization could be done in Python, and again what I find is choices of possible useful packages, new sets of documentations to digest, ... and etc.
IMHO, if one has chosen Matlab, he should seriously consider adopting parallel toolbox. It's easy to use, and combined with group-processing functions and functional programming, to shift the whole coding game to a different paradigm, saving both coding and run time.

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

产品

版本

R2023a

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by