Problem using parfor for reading variable sized chunks of data into a larger pre-allocated container
2 次查看(过去 30 天)
显示 更早的评论
Hi,
I have an problem where I have pre-allocated a large matrix or vector, a, and where I will read data blocks from a large number of files that will be inserted at the right indexes in a. The different files, and resulting blocks, will typically have different size.
A simple example:
a = zeros(15,1); % pre-allocated vector
b = [1,10;11,15]; % each row contains the start to stop index for each block
parfor i = 1:size(b,1)
a(b(i,1):b(i,2),:) = i*ones((b(i,2)-b(i,1))+1,1);
end
With 'for' instead of 'parfor' it works as intended.
I have been reading several sites like https://www.mathworks.com/help/parallel-computing/troubleshoot-variables-in-parfor-loops.html and https://www.mathworks.com/matlabcentral/answers/126515-usage-of-sturctures-inside-parfor but I cannot wrap my head around my problem above.
Any tips or solutions which don't reduce the performance I am trying to obtain by using parfor in the first place?
Thanks,
Oyvind
0 个评论
采纳的回答
Edric Ellis
2020-10-21
编辑:Edric Ellis
2020-10-21
There's no simple way to do this without at least some duplication of data. With some duplication of data, you could do something simple like this:
aCell = cell(1, size(b,1));
parfor i = 1:size(b,1)
aCell{i} = <stuff>; % return each block in its entirety
end
a = vertcat(aCell{:}); % concatenate all cell entries into the final result
If that is not sufficiently performant, you could consider using parfeval to give you a little more control, but this is more difficult to code, and may not actually save you much. Here's an untested sketch though:
a = zeros(15,1);
for i = 1:size(b,1)
fut(i) = parfeval(@doStuff, 1, b(i,1), b(i,2)); % invoke doStuff(b(i,1),b(i,2))
end
for i = 1:size(b,1)
[idx, result] = fetchNext(fut); % collect the next result
% (note that 'idx' tells you the index into 'fut' that just
% completed)
a(b(idx,1):b(idx,2),:) = result; % push the result into 'a'
end
更多回答(0 个)
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!