parfor - Attempt to serialize data which is too large

1 次查看（过去 30 天）
Germán
Germán 2012-4-2
评论： Edric Ellis 2013-11-8
I'm working on a large data set that is is stored as a cell array X, where each cell contains an array of about 2MB, totaling 1GB. I'm running it in parallel on 50 cores that are each on separate nodes. I get the error "Attempt to serialize data which is too large" when I try to run the whole array . When I run a small array X I don't get the error. I check on the nodes and there's plenty of memory left, it seems the bottleneck is data transmission
I'm using the following code
Y1= cell(1,N1);
Y2 = cell(1,N1);
parfor i = 1:N1
XX = X{i};
Np = size(XX,1);
Y1_temp = zeros(400, Np-1);
Y2_temp = zeros(400, Np-1);
for j= 1:(Np-1)
Y1_temp(:,j) = someFunction;
Y2_temp(:,j) = someOtherFunction;
end
Y1{i} = Y1_temp;
Y2{i} = Y2_temp;
end
I don't understand why this is, has anyone ever got this error? I couln't find anything online. I also tried writing it as an spmd block where I operate on the local worker data but I get the same error. Any help will be very appreciated.
Regards Germán

请先登录，再回答此问题。

采纳的回答

Edric Ellis
Edric Ellis 2013-11-8
The 2GB PARFOR transmission limitation was removed from Parallel Computing Toolbox in R2013a.

更多回答（2 个）

Konrad Malkowski
Konrad Malkowski 2012-4-2
There is a 2GB transmission limit between client and workers. Could you provide me with additional values for Np, N1, Nt, etc so that I have a better picture of which variables are causing the problem?
  1 个评论
Germán
Germán 2012-4-2
sure, the values are usually Np=100000, N1=Nt=512.
My understanding is that only the sliced portions of X, Y1, Y2 would be transmitted in each iteration, is that correct? If so the the size of a single cell in those variables is never above 2 MB.
Thanks for your help!

请先登录，再进行评论。

James
James 2013-11-4
I have the same problem. I think it is a common problem, but I couldn't find a solution for this. If it is the 2GB transmission limit, then how can we get around this then?
  2 个评论
Germán
Germán 2013-11-4
I got around it using normal arrays instead of cell arrays since there was a bug with cell arrays, but the issue might have been solved in the latest releases.
Edric Ellis
Edric Ellis 2013-11-8
The 2GB PARFOR transmission limitation was removed from Parallel Computing Toolbox in R2013a.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by