Retrieving data from createtask is very slow
4 次查看(过去 30 天)
显示 更早的评论
I used createJob and createTask to execute a program on 16 notes, then retrived my data from each tasks. The result is very large, I can not use fetchOutputs or getAllOutputArgument.My network is a Gigabit LAN. However, I found that the speed of retrieving data is only 3M/s.It cost too much time is data transmission. This is my code:
Job=createJob(sched); set(Job,'PathDependencies',PathDependency},'FileDependencies','Reconstruction_MLOS_SMART_Parallel.m','IntegratingLOS.m','MARTUpdate.m','MARTInitialize.m'});
for ii=1:cpucorenum
curTask=newtask{ii,1};
PartStart=curTask(1);
PartEnd=curTask(end);
TaskObj(ii,1)=createTask(Job,@MARTUpdate,1,{VolumeSpace(:,:,PartStart:PartEnd), MFx, MFy, Xmesh, Ymesh, xmesh_jiu, ymesh_jiu, m, oriI, newI, PartStart, PartEnd, ParaSet});
end
submit(Job);
wait(Job,'finished');
for ii=1:cpucorenum
curTask=newtask{ii,1};
PartStart=curTask(1);
PartEnd=curTask(end);
results_t = get(TaskObj(ii,1), 'OutputArguments');
VolumeSpace(:,:,PartStart:PartEnd)=results_t{1,1};
destroy(TaskObj(ii,1));
clear results_t
end
%%clear the temporary value
clear results_t TaskObj
destroy(Job);
clear Job
The problem arised in ‘results_t = get(TaskObj(ii,1), 'OutputArguments')’. Too much time expended in this step.
I holp some one can help me. Thank you!
0 个评论
回答(1 个)
Edric Ellis
2014-1-23
The time taken for fetching outputs is driven by the disk access speed, and the speed of MATLAB's own SAVE command. I ran the following test:
c = parcluster('local');
numMBytes = floor(logspace(1, 3, 10));
times = zeros(size(numMBytes));
for idx = 1:numel(numMBytes)
j = batch(c, @randi, 1, {[-10 10], numMBytes(idx), 1024^2, 'int8'});
wait(j);
tic
x = fetchOutputs(j);
times(idx) = toc;
delete(j);
end
plot(numMBytes, numMBytes ./ times);
xlabel 'Number of megabytes'
ylabel 'MB / s'
and saw on my system approximately 140 MB/s. This is R2013b on GLNXA64 using the local cluster which stores the outputs on a network disk.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 MATLAB Parallel Server 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!