Job management, control flow

2014 10 14
1 个回答
回答已采纳
更新时间：2014 11 26
9 次查看（30 天）

1 个投票

I work with large (0.2 - 1 Tb) sets of 3D image data. I would like to set up a queue within a function that reads in an 'on-deck' image into RAM while data is being processed on the GPU. I can sort out the details, but would be grateful if someone could help point me in the right direction to do the following:
Read in 2 files run loop1 on file1 ; end loop ; clear file 1
run loop1 on file 2; while running, load file3 into memory.
...and so on.
In other words, i need to essentially be able to send a child process to the gpu, and wait on its return value

请先登录，再回答此问题。

 采纳的回答

Jon Boerner
Jon Boerner 2014-10-21
编辑：Jon Boerner 2014-10-21

1 个投票

Hi Benjamin,
The easiest way to do this is probably using either the batch function, or the parfeval function in newer versions. Basically, you would use one of those functions to start a worker which would handle interfacing with the GPU, while the main thread moves on to loading the next file. The code might look something like:
loadedData = loaddata(...);
c = parcluster('local');
c.NumWorkers = 2; % You may want more workers depending on how many GPU's you have, etc.
j = batch(c,@myGPUFcn,1,{loadedData});
while(...)
loadedData = loadnextdata(...);
j.wait(); %Wait for GPU to finish in case file loaded faster
outputs = j.fetchOutputs;
end
function y = myGPUFcn(data)
% make gpuArray and perform operations on it
end
There are some details missing, but that would be the general approach. Using parfeval would be a very similar workflow. loaddata and loadnextdata are just place holders for however you load your files.
Let me know if you have any questions!

1 个评论
显示 -1更早的评论 隐藏 -1更早的评论

Benjamin
Benjamin 2014-11-26
Jon,
After working through a small test case, the idea you suggest seems to work fine, however, on testing with some more realistic data, I ran into some trouble.
It seems as though when passing the {loadeddata} to the worker, the file is written to disk as a -mat file. Not only is this terribly slow for very large images, it also entirely defeats the purpose, which was to deal with the relatively slow read rate of the data in the first place. (effectively 3x the time.)
Am I missing something obvious in how to pass a workspace variable to a worker, that exists in memory?
(This test data is ~ 6Gb / file, which is still not so large.)
Thanks,
Ben

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Big Data Processing 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by