parfeval or batch processing

9 次查看（过去 30 天）
Muna Tageldin
Muna Tageldin 2020-11-2
回答： Swastik Sarkar 2025-5-28
Assuming I want to convert the below code from using parpool to using parcluster. The problem with my code is that the for loop is 10,000 iterations (I get an error in running the code ), however when I run with 100 iterations, the code runs well). It seems the problem is with waiting time for future objects ( I increased the waiting time from 10,000 to 30,000).
The error is "Error using parallel.FEvalFuture/fetchNext. The function completed with an error. Caused by: no workers available for FevalQueue excution.
I figured I should try batch processing following this question and also parcluster as I am running the code on a remote cluster.
My question is how do I share files (function file) to the workers on the clusters ? do I need to pass constants to workers like in parallel.pool.constant. Also, what's the difference between using batch processing and parfeval ?
poolobj=parpool('my_cluster',8);
[up, op]=ndgri(1e-3:1e-2:1,1e-3:1e-2:1);
up=reshape(up, [1,size(up,1)*size(up,2)]);
up=reshape(up, [1,size(up,1)*size(up,2)]);
z=rand(5,5e3);
addAttachedFiles('<path to my function/cores_random.m');%%to add the function files to workers on the pool
C1=parallel.pool.Constant(z);%%use parallel.pool.Constant to copy these variables into workers
U2=parallel.pool.Constant(up);
O2=parallel.pool.Constant(op);
for i=1:size(up,2)
f1(i) = parfeval(poolobj,@cores_random, 3, i, 1,C1,U2,O2); %%line 11
f2(i) = parfeval(poolobj,@cores_random, 3, i, 2,C1,U2,O2);
f3(i) = parfeval(poolobj,@cores_random, 3, i, 3,C1,U2,O2);
f4(i) = parfeval(poolobj,@cores_random, 3, i, 4,C1,U2,O2);
f5(i) = parfeval(poolobj,@cores_random, 3, i, 5,C1,U2,O2);
end
wait(f1,30000);
wait(f2,30000);
wait(f3,30000);
wait(f4,30000);
wait(f5,30000);
for j=1:size(up,2):-1:1
[idx1,u1,o1,ep1]=fetchNext(f1);
[idx2,u2,o2,ep2]=fetchNext(f2);
[idx3,u3,o3,ep3]=fetchNext(f3);
[idx4,u4,o4,ep4]=fetchNext(f4);
[idx5,u5,o5,ep5]=fetchNext(f5);
end
function [uu,oo,ep] = cores_random(i, zidx,C1,U2,O2)
up = U2.Value(i)
op = O2.Value(i);
z = C1.Value(zidx,:);
%%doing some calculations here
%%z is of size 1*1e3
%%up is scalar op is scalar
end

请先登录，再回答此问题。

回答（1 个）

Swastik Sarkar
Swastik Sarkar 2025-5-28
Hi Muna,
For large-scale parallel tasks like 10,000 iterations, using batch is generally better than parfeval because batch submits jobs to the cluster’s scheduler, which manages resources and scales more efficiently. parfeval runs tasks within an existing parallel pool and can run out of resources if too many futures are queued.
To share files with the cluster workers with batch, make sure to include necessary files using the AttachedFiles option so workers have access to your files. More information about files access from workers is present in the below documenation:
https://www.mathworks.com/help/parallel-computing/run-batch-job-and-access-files-from-workers.html
Hope this helps you chosse between parfeval and batch processing

类别

在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息

标签

产品


版本

R2019a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by