Processing blocks of data in parallel
显示 更早的评论
I have a data set which is 4 million rows by 80 columns wide. I want to split the data up into equal sized blocks so that each block is processsed in Parallel.
Would the Parallel computing tool box allow a function to be written such that each block is processed in parallel?
采纳的回答
Edric Ellis
2011-10-4
Your best bet is to try and restructure your code to use a PARFOR loop. For example, you could do something like this:
N = 4e6;
indata = rand(N, 80);
outdata = zeros(N, 1); % results
parfor ii = 1:N
outdata(ii) = myfcn(indata(N,:));
end
You can run PARFOR loops in standard MATLAB to check they work, but you need Parallel Computing Toolbox to run in parallel.
Another option is to use distributed arrays - this lets you split your input data across the memory of multiple machines (you need MATLAB Distributed Computing Server licences to run workers on multiple machines). That way, you never need the whole of your large data set in a single machine's memory.
Distributed arrays are built on top of MPI - Parallel Computing Toolbox ships with a build of MPICH2.
2 个评论
adam cox
2011-10-4
Walter Roberson
2011-10-4
Please see
http://www.mathworks.com/help/toolbox/distcomp/distributed.rand.html
http://www.mathworks.com/help/toolbox/distcomp/codistributed.rand.html
http://www.mathworks.com/help/toolbox/distcomp/bsic4fr-1.html
更多回答（1 个）
Daniel Shub
2011-10-4
0 个投票
This depends on the processing. If each block can be processed independently from the other blocks, then yes the Parallel computing toolbox can handle that. If the processing for each block depends on the other blocks, I don't think the Parallel computing toolbox has a simple MPI (although I have never tried).
With standard MATLAB you can see if parfor can help.
类别
在 帮助中心 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
