Retrieving data from createtask is very slow

Question

buaa 2014-1-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/113369-retrieving-data-from-createtask-is-very-slow

评论： buaa 2014-1-24

I used createJob and createTask to execute a program on 16 notes, then retrived my data from each tasks. The result is very large, I can not use fetchOutputs or getAllOutputArgument.My network is a Gigabit LAN. However, I found that the speed of retrieving data is only 3M/s.It cost too much time is data transmission. This is my code:

Job=createJob(sched);  set(Job,'PathDependencies',PathDependency},'FileDependencies','Reconstruction_MLOS_SMART_Parallel.m','IntegratingLOS.m','MARTUpdate.m','MARTInitialize.m'});
          for ii=1:cpucorenum
              curTask=newtask{ii,1};
              PartStart=curTask(1);
              PartEnd=curTask(end);
              TaskObj(ii,1)=createTask(Job,@MARTUpdate,1,{VolumeSpace(:,:,PartStart:PartEnd), MFx, MFy, Xmesh,                 Ymesh, xmesh_jiu, ymesh_jiu, m, oriI, newI, PartStart, PartEnd, ParaSet});
          end
          submit(Job);
          wait(Job,'finished');
            for ii=1:cpucorenum
                curTask=newtask{ii,1};
                PartStart=curTask(1);
                PartEnd=curTask(end);
                results_t = get(TaskObj(ii,1), 'OutputArguments');
                VolumeSpace(:,:,PartStart:PartEnd)=results_t{1,1};
                destroy(TaskObj(ii,1));
                clear results_t
           end
            %%clear the temporary value
            clear results_t TaskObj
            destroy(Job);
            clear Job

The problem arised in ‘results_t = get(TaskObj(ii,1), 'OutputArguments')’. Too much time expended in this step.

I holp some one can help me. Thank you!

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Edric Ellis 2014-1-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/113369-retrieving-data-from-createtask-is-very-slow#answer_121912

在 MATLAB Online 中打开

The time taken for fetching outputs is driven by the disk access speed, and the speed of MATLAB's own SAVE command. I ran the following test:

    c = parcluster('local');
    numMBytes = floor(logspace(1, 3, 10));
    times = zeros(size(numMBytes));
    for idx = 1:numel(numMBytes)
        j = batch(c, @randi, 1, {[-10 10], numMBytes(idx), 1024^2, 'int8'});
        wait(j);
        tic
        x = fetchOutputs(j);
        times(idx) = toc;
        delete(j);
    end
    plot(numMBytes, numMBytes ./ times);
    xlabel 'Number of megabytes'
    ylabel 'MB / s'

and saw on my system approximately 140 MB/s. This is R2013b on GLNXA64 using the local cluster which stores the outputs on a network disk.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

buaa 2014-1-24

Thank you very much. I compare the execute time between parallel program(batch or createtask) and original normal program with same function(randi).The former is much slower than the later, almost 4 times than later. Do you know why, and is there better method than createtask?

请先登录，再进行评论。