Mapreduce with tall array in matrix multiplication

5 次查看（过去 30 天）
chen zhe
chen zhe 2024-6-26
评论： chen zhe 2024-6-26
Hi,
I am processing huge dataset (billions rows), so I used tall array in my code. While I found the tall array are not support general matrix multiplication. The '*' only works in a limited conditions.
I am wondering if we can combine mapreduce to perform matrix multiplication with tall arrays ? Below is a example shows the error.
A = tall(ones(200,2));
B = tall(ones(2,3));
values = A*B;
gather(values)
Error using tall/mtimes>iVerifyAtLeastOneScalar
Matrix multiplication of two tall arrays requires one of them to be scalar.
Learn more about errors encountered during GATHER.
Error in * (line 31)
[X,Y] = iVerifyAtLeastOneScalar(X,Y,"MATLAB:bigdata:array:MtimesBothTall");

请先登录，再回答此问题。

回答（1 个）

Torsten
Torsten 2024-6-26
编辑：Torsten 2024-6-26
Ran in:
Would this work for you ?
B = ones(2,3);
A = tall(ones(200,2));
values = A*B;
gather(values)
Evaluating tall expression using the Local MATLAB Session: - Pass 1 of 1: 0% complete - Pass 1 of 1: Completed in 0.21 sec Evaluation completed in 0.37 sec
ans = 200x3
2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
<mw-icon class=""></mw-icon>
<mw-icon class=""></mw-icon>
  1 个评论
chen zhe
chen zhe 2024-6-26
Unfortunately no. Actually, A and B ate both tall arrays obtained by some processings.
Your method is equivalent to gather（B），then multiply A and B. Finally need to gather（values） again. （2 gather（） in total）

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Tall Arrays 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by