limiting broadcast variables in parfor loops
2 次查看(过去 30 天)
显示 更早的评论
I have some 3D or 4D arrays that I want to run a block processing on. To do this I create a cell object that I can call that returns matricies of position values.
For example, ind_object{1} = [1 2 3; 1 2 3; 1 2 3], ind_object{2} = [1 2 3; 1 2 3; 2 3 4], (i.e. they can be overlapping blocks).
I then call this index object in a parfor loop to pull out blocks of my image (sz_block(n) is the size of the block, on this example [3,3,3])
parfor n=1:numel(index_object)
block_im = pad_im(index_object{n}(1,1:sz_block(1)),index_object{n}(2,1:sz_block(2)),index_object{n}(3,1:sz_block(3)),:);
out(n) = run_some_function(block_im);
end
And the @run_some_fucntion() is usually some sort of optimization problem, for example it could be using fmincon.
The problem with this is that my "pad_im" array ends up being a broadcast variable and it's rather large 512x512x300 so the memory swapping really slows down the calculation.
One solution to this would be to break the "pad_im" array into smaller arrays
pad_im1 = pad_im(:,:,1:149);
pad_im2 = pad_im(:,:,150:end);
then run this parfor loop twice, once on the pad_im1 and another time on the pad_im2, then I could do something smart to stitch where the break was made. However, I don't think this is a very good solution and there has to be a more eligant/inteligent way to do this; something like a que?
Or does anyone know how other programs (like C/C++ or python) do n-dimensional block processing?
Thanks for the help!!
2 个评论
Alvaro
2023-1-23
Have you considered using distributed arrays for this?
Alternatively, what function are you trying to run on those image blocks? Check out functions in the Image Processing Toolbox, there could be one that already does what you want and has built-in parallelization.
采纳的回答
Matt J
2023-1-24
编辑:Matt J
2023-1-24
The block decomposition that you are doing needs more explanation. If the blocks are just fixed-sized, non-overlapping tiles of the original array, then it would be a simple matter to just reshape pad_im into pages consisting of the blocks. Then it can be passed to the parfor loop as a sliced variable. A simple way to do that reshaping is with this FEX download,
sz=[3,3,3]; %block size
N=blkNumel(pad_im,sz);
Pages=blkReshape(pad_im,sz,[1,1,1,N]);
parfor n=1:N
block_im=Pages(:,:,:,n);
out(n) = run_some_function(block_im);
end
更多回答(0 个)
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!