How to use parfor with a range from a very large object and cause memory problems

1 次查看（过去 30 天）
Jon Ward
Jon Ward 2021-11-18
编辑： Jon Ward 2021-11-18
How can I achieve the following without gettting the broadcast variable warning and minimising memory usage by Matlab trying to copy the full variable R?
This is a simplified example of what I am trying to achieve, but the memory requirements explode when I try to do this (I am using 48 processors in parallel on a high spec AWS box with lots of memory).
Thanks!
T = 10000;
S = 100; %S is actually ~1000
P = 10;
M = 60;
R = rand(T,S,P,M); % This is an example, R and C below get populated from another process, but this is to illustrate the size of R.
C = rand(T,M);
resultsM = zeros(T,S,P);
resultsSD = zeros(T,S,P);
parfor t=50:T
r = R(1:t,:,:,:); % Broadcast warning here
% Do some stuff with r
c = C(1:t,:);
for s=1:S
for p=1:P
rr = squeeze(r(:,s,p,:));
tmpResults = sum(c.*rr,2);
resultsM(t,s,p) = mean(tmpResults); % Mean and below sd are examples, I apply other custom functions to tmpResults
resultsSD(t,s,p) = std(tmpResults);
end
end
end

请先登录，再回答此问题。

回答（1 个）

Raymond Norris
Raymond Norris 2021-11-18
Have you considered assigning R in the parfor?
parfor t = 1:T
R = rand(T,S,P,M)
...
end
It's less intuitive for MATLAB serial code, but may be a better practice for MATLAB parallel code. R is now a temporary variable, which can't be referenced after the parfor.
Also look at ticBytes and tocBytes to see how much data is truely getting passed back and forth.
  1 个评论
Jon Ward
Jon Ward 2021-11-18
Unfortunately not, I was using rand to show the size of R only, it actually gets generated from another process. I have edited the code above to show what I am trying to do.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品


版本

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by